I'm using a Linux driver to transfer large blocks of data to/from logic at high speed. The self-written DMA controller is in logic, can queue up several transfers and operates fine.
The trouble I have with the driver is that for the DMA transfers, I want to use a zero-copy style interface, because the data usually consists of video frames, and moving them arouns in memory is prohibitly expensive.
I based my implementation on what IIO (industrial IO) does, and implemented IOCTL calls to the driver to allocate, free, enqueue and dequeue blocks that are owned by the driver. Each block can be mmapped into user space.
Using dma_alloc_coherent to allocate the blocks, and then just using them without any extra measures works just fine. The system can transfer data at 600MB/s to and from DDR into logic with very little CPU intervention. However, the memory returned by dma_mmap_coherent appears to be uncached, because accessing this area from userspace is horribly slow (e.g. reading 200MB byte-by-byte in a simple for loop takes 20 seconds, while this takes about a second in malloced memory)
After reading some documentation, I decided that I should use a streaming DMA interface because my driver knows exactly when logic or CPU "owns" the data blocks. So instead of the "coherent" functions, I just kmalloc these buffers and then use dma_map_single_* to initialize them. Before and after DMA transfers, I call the appropriate dma_sync_single_for_* method.
By mapping the kmalloced memory into user space, I once again have speedy access to this memory, and caching is enabled. Data transfers also work well and extensive testing shows that this works well. However, the in-kernel performance is now completely crippled. The system spends so much time in the dma_sync_single... calls, the the CPU now becomes a limiting factor. This limits the transfer speeds to about only 180MB/s. This method is only about 20% less CPU intensive than just copying the data from the DMA buffer into a user buffer using a copy_to_user call.
