AXI-DMA driver using UIO

[max.giacometti@gmail.com]

Hi everybody,

I am trying to write a driver for AXI-DMA using UIO - code attached.

The problems I encounter are

1)  

sizeFd = fopen("/sys/class/uio/uio0/maps/map0/size", O_RDONLY);

generates a segmentation fault.

2)

I've seen the content of /sys/class/uio/uio0/maps/map0/size is 0x10000, so I removed the fopen and used this value for uioSize.

The execution stops while waiting for the interrupt.

The printf look like this

zynq> ./axidma 
uioSize: 65536
baseAddress: -1226833920
data ready
S2MM: 1
S2MM: 1 <= after resetting S2MM
S2MM: 0 <= after starting S2MM
MM2S: 1
MM2S: 1 <= after resetting MM2S
MM2S: 0 <= after starting MM2S
^C
zynq> 

Could you please help?

Thanks in advance

[achutha]

max.giacometti@gmail.com

I think with the below code you are trying to see if you are able to access the UIO size .i feel sizeFD < 0 may trigger this segmentation.

 sizeFd = fopen("/sys/class/uio/uio0/maps/map0/size", O_RDONLY);
  if (sizeFd < 0) 
    {
      perror("uio size access");
      return errno;
    }

I have seen the similar usage working but a little change in the code:

unsigned int get_memory_size()
{
FILE *size_fp;
unsigned int size;

size_fp = fopen("/sys/class/uio/uio0/maps/map0/size", "r");

if (!size_fp) {
printf("unable to open the uio size file\n");
exit(-1);
}

fscanf(size_fp, "0x%08X", &size);

fclose(size_fp);

return size;
}

Check if this helps.

Regards,

Achutha

[davewarren]

Max,

Had a quick look at your code, the buffers for the DMA are malloced, that means they are in user space and virtual. You can not DMA from a virtual address, need the pyhsical address.

[max.giacometti@gmail.com]

And how to get the physical address?

[achutha]

max.giacometti@gmail.com

mmap should do this for you.

Refer to the attached c-file and let me know if that helps.

Regards,

Achutha

[max.giacometti@gmail.com]

This is what I did for configuring the DMA.

I think the problem underlined by Dave is how I pass user-space data to DMA: I create a buffer with malloc and give the corresponding address to the DMA, and this is an error because I should use the physical address.

Plus, there might be problems since virtual data might not be contiguous.

Is there a way to overcome this issue in user space or do I have to allocate a buffer in kernel space?

[sampatd]

To copy the user-space data to DMA, you can use the function copy_from_user(). This function copies a buffer of bytes from user-space to the kernel space. 

Check the attached documents. 

[davewarren]

Max,

It is possible to write a user space DMA, you can reserve an area of the DRAM by excluding it from Linux's memory map (do this in device tree, use a block somewhere in the middle of mem). This fixed buffer can be mmapped into your user space program and used for DMA.

However I strongly suggest you don't do it this way, the copy from user space to the fixed buffer will be slow. Write a simple device driver and reserve your buffer in kernel space.

good luck

Dave

[guillaumebres]

---

It is possible to write a user space DMA, you can reserve an area of the DRAM by excluding it from Linux's memory map (do this in device tree, use a block somewhere in the middle of mem). This fixed buffer can be mmapped into your user space program and used for DMA.

However I strongly suggest you don't do it this way, the copy from user space to the fixed buffer will be slow. Write a simple device driver and reserve your buffer in kernel space.

---

You're right, that would do the trick but certainly would be too dependent on this configuration.

As suggested above the DMA controller only understands physical addresses (like any hardware). From the Linux DMA API you could use dma_map_single(device, adress, size, direction);

    /*a virtual pointer to handle data*/    
    sdev->buffer = (int32_t*)kmalloc(1024*sizeof(int32_t), GFP_KERNEL);
    if(sdev->buffer == NULL){
         printk(KERN_ERR "failed to allocate memory for dma buffer\n");
         status = -ENOMEM;
         goto ...;
    }

    /* dma engine - phys address for DMA controller*/
    sdev->dma_dir = DMA_FROM_DEVICE;  //defined in API
    
   bus_addr = dma_map_single(sdev->dev, sdev->buffer, 1024*sizeof(int32_t), sdev->dma_dir);
    
    if(dma_mapping_error(NULL, bus_addr)){
         printk(KERN_ERR "failed to allocate dma coherent buffer\n");
         status = -EINVAL;
         goto ...;
    } 
    else  
         sdev->dma_phys = bus_addr; 

now you can receive/send dma coherent data into/from sdev->buffer

[max.giacometti@gmail.com]

Hi,

I am resuming this old topic because after months spent working on different stuff I am back on DMA.
Status is exactly the same: I have to find how to inform the AXIDMA about the physical address where to get and store data.
Physical address cannot be estracted from userspace, so I have to write some kernel space module.
Now, what's the purpose of this module?
Since I can control the DMA from userspace, I'd like the module to only allocate space for DMA TX and RX buffers.
it could use a dma_alloc_coherent to allocate a buffer and would also return the virtual and physical address given by dma_alloc_coherent
("It returns a pointer to the allocated region (in the processor's virtual address space) or NULL if the allocation failed. It also returns a <dma_handle> which may be cast to an unsigned integer the same width as the bus and given to the device as the DMA address base of the region").

Does this make any sense?

[linnj]

That's exactly what I was doing in those PDFs about DMA that were posted (based on what I remember with a quick review). Yes that should work.

Thanks

John

[max.giacometti@gmail.com]

Is the virtual address given by dma_alloc_coherent valid in userspace?

Can I pass it as it is or shall I do something more, like copy-from-user/copy-to-user or mmap-ping /dev/mem using the physical address as offest?

[linnj]

You need to implement the mmap call in the kernel driver to allow that memory to be used from user space. It's not just an address issue but the user space process has to have mmu entries for the memory and that's the job of mmap. The PDF describes that in a high level way.

[max.giacometti@gmail.com]

Understood.

One last question: in my application I might need multiple buffer in parallel.

I can use ioctl to allocate/deallocate a buffer, but what about mmap?

I was thinking about this scenario (from userspace):

- create buffer w/ size 0x10 => the driver returns the ID of that buffer (0)
- mmap buffer with ID=0
- create buffer w/ size 0x20 => the driver returns the ID of that buffer (1)
- mmap buffer with ID=1
- read/write buffers
- free buffer 0
- create new buffer => bufferID = 0
...

Looking at your example

static int dma_proxy_mmap (struct file * filp, struct vm_area_struct *vma)
{
    return dma_common_mmap(dma_device_pointer, vma,
        buffer_pointer, physical_buffer_pointer,
        vma->vm_end-vma->vm_start);
}

I'd like to have more parameters in dma_proxy_mmap, that will determine which (buffer_pointer, physical_buffer_pointer) to use, like

static int dma_proxy_mmap (int ID, struct file * filp, struct vm_area_struct *vma)
{
    /* get buffer_pointer and physical_buffer_pointer based on ID */
    return dma_common_mmap(dma_device_pointer, vma,
        buffer_pointer, physical_buffer_pointer,
        vma->vm_end-vma->vm_start);
}

Is that possible?

And how would then the call from userspace

proxy_interface_p= mmap(0, sizeof(dma_proxy_channel_interface), PROT_READ | PROT_WRITE, MAP_SHARED, proxy_fd, 0);

look like?

[linnj]

I'm not sure why you would want to mmap each smaller buffer.  I would just mmap a big buffer then chop it up in small pieces to use then use offsets into the buffer.  I would think you can do it your way but it seems more complex.  The last argument to the mmap from user space is an offset I believe (but don't remember the details).

[max.giacometti@gmail.com]

One more thing.

I am initializing my module like explained in the example here.

But dma_common_mmap requires a struct device *: How can I get it?

Using cdev doesn't seem to help more.

[max.giacometti@gmail.com]

Looking at this (your?) example for inspiration...
Does it mean that I have to define a struct consisting of a cdev (that will take care of interfacing w/ userspace) and a driver (which will take care of memory allocation)?

[linnj · Accepted solution]

Yes that example provides all you need.  I would suggest that you start with just the basics of a char driver as it seems like you might be trying to build without the basics which is going to be a challenge.  I find that building in layers is best, get to know one layer before trying to build multiple layers at the same time.

[jkwi111]

Hi, 

   I usd  xdev.c and xdev.h to config DMA driver,which communicate the ps and pl, on the linux kernel Version 2014.03 . when insert the zynq_dma.ko ,the log is showing that it didn't find the rx and tx devices,why? do you know how to use the DMA on the linux ? 

 PS  :log is here:

root@zynq:~# insmod zynq_dma.ko
zfdriver: init begin
zfdriver: tx match is 100001
zfdriver: rx match is 100002
_dma_request_channel
_dma_request_channel:chan:0xd563d340
private_candidate: dma0chan0 filter said false
private_candidate: dma0chan1 filter said false
private_candidate: dma0chan2 filter said false
private_candidate: dma0chan3 filter said false
private_candidate: dma0chan4 filter said false
private_candidate: dma0chan5 filter said false
private_candidate: dma0chan6 filter said false
private_candidate: dma0chan7 filter said false
_dma_request_channel:chan:0x0
__dma_request_channel: fail ((null))
_dma_request_channel
_dma_request_channel:chan:0xd563d340
private_candidate: dma0chan0 filter said false
private_candidate: dma0chan1 filter said false
private_candidate: dma0chan2 filter said false
private_candidate: dma0chan3 filter said false
private_candidate: dma0chan4 filter said false
private_candidate: dma0chan5 filter said false
private_candidate: dma0chan6 filter said false
private_candidate: dma0chan7 filter said false
_dma_request_channel:chan:0x0
__dma_request_channel: fail ((null))
zfdriver: Did not find tx device
zfdriver: Did not find rx device

BR,

lawrence

[max.giacometti@gmail.com]

For those who might need it, I attach the solution I used.

Thank you all for supporting!

[mkarwat]

Hello,

All of this was very helpful to me. But I'm still missing one last thing, it's probably easy, but I'm new to embedded linux. I can control DMA usnig uio driver, I used your kernel module, comiled and loaded, it appered in /dev/memalloc. I can open it and mmap in the same way as uio. But how I can get the phisical and virtual addresses in user aplication? So I could for example write something, use dma to loopback transfer it and read it. Like in simple dma example on standalone?

Here is how I'm trying to do it:

volatile unsigned *Memory;

int memory_fd;

const char *memory_dev = "/dev/memalloc";
memory_fd = open(memory_dev, O_RDWR);
Memory = (unsigned *) mmap(NULL, 0x10000, PROT_READ | PROT_WRITE, MAP_SHARED, memory_fd, 0);
Memory[0] = 0x0000FFFF;
Memory[1] = 0xFFFF0000;

XAxiDma_SimpleTransfer(&AxiDma, PHYSICAL_ADDR, PKT_LEN, XAXIDMA_DMA_TO_DEVICE);

XAxiDma_SimpleTransfer(&AxiDma, DIFFERENT_PHYSICAL_ADDR, PKT_LEN, XAXIDMA_DEVICE_TO_DMA);

uint32_t val1 = Memory[DIFFERENT_VIRTUAL_ADDR];
uint32_t val2 = Memory[DIFFERENT_VIRTUAL_ADDR+1];

And each time I try to run it i get "Bus error" while writing to memory ( Memory[0] = 0x0000FFFF; )

Thanks for any help

[max.giacometti@gmail.com]

Hi, I copy paste here bits of my test code, hope it helps

fd_uio0 = open("/dev/uio0", O_RDWR);
XAxiDmaConfigInstance->BaseAddr = (u32) mmap(NULL, BUFFER_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd_uio0, 0);

memAllocFd = open("/dev/memalloc", O_RDWR);

// TX buffer
iBufID = ioctl(memAllocFd, MEMALLOC_RESERVE, BUFFER_SIZE * 2);
source_paddr = (void *)ioctl(memAllocFd, MEMALLOC_GET_PHYSICAL, iBufID);
ioctl(memAllocFd, MEMALLOC_ACTIVATE_BUFFER, iBufID);
source_vaddr = mmap(0, BUFFER_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, memAllocFd, 0);

// fill TX buffer using source_vaddr
	
// RX buffer
oBufID = ioctl(memAllocFd, MEMALLOC_RESERVE, BUFFER_SIZE * 2);
dest_paddr = (void *)ioctl(memAllocFd, MEMALLOC_GET_PHYSICAL, oBufID);
ioctl(memAllocFd, MEMALLOC_ACTIVATE_BUFFER, oBufID);
dest_vaddr = mmap(0, BUFFER_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, memAllocFd, 0);

// enable irq
XAxiDma_IntrEnable(&XAxiDmaInstance, XAXIDMA_IRQ_ALL_MASK, XAXIDMA_DEVICE_TO_DMA);
XAxiDma_IntrEnable(&XAxiDmaInstance, XAXIDMA_IRQ_ALL_MASK, XAXIDMA_DMA_TO_DEVICE);

// start transfer
XAxiDma_SimpleTransfer(&XAxiDmaInstance, dest_paddr, BUFFER_SIZE, XAXIDMA_DEVICE_TO_DMA);
XAxiDma_SimpleTransfer(&XAxiDmaInstance, source_paddr, BUFFER_SIZE, XAXIDMA_DMA_TO_DEVICE);

// wait for irq
read(fd_uio0, (void *) &pending, sizeof(int));

// ack and reset irq
XAxiDma_IntrAckIrq(&XAxiDmaInstance, XAXIDMA_IRQ_ALL_MASK, XAXIDMA_DEVICE_TO_DMA);
XAxiDma_IntrAckIrq(&XAxiDmaInstance, XAXIDMA_IRQ_ALL_MASK, XAXIDMA_DMA_TO_DEVICE);
write(fd_uio0, (void *) &reenable, sizeof(int));
	
// use dest_vaddr as pointer for received data

[linnj]

I'm just wanting to make sure everyone doing DMA knows about the newer Xilinx product called SDSOC that provides the low level drivers and can use DMA  under the hood to move data.  This should help reduce the amount of kernel work and make it easier to get the data into user space.  I realize there may be reasons you are doing some of this work the way you are but my experience showed that it is a lot of work doing it all manually unless you have a lot of experience in this area.

http://www.xilinx.com/products/design-tools/software-zone/sdsoc.html

Thanks

John

[abbascit]

Hi max.giacometti@gmail.com,

Have you used this driver along with your UIO application which you have attached at start of post? How you are able to cofigure DMA ?

[max.giacometti@gmail.com]

Hi,

I didn't use the code posted in my first message.

What I did is using the this driver to allocate memory for the data transfers through DMA, and a UIO driver to control the DMA through XAxiDma_* functions.

To understand more about the UIO driver, you can have a look at the pdf presentations posted in the first page of this thread.

[abbascit]

Hi,

You mean to say you get physical address using kernel driver and then use that address to write at configuration registers of DMA through UIO?. To get interrupt you use uio read.
So you have used kernel module and UIO together as mentioned in PDF presentation.
But how you are able to use XAxiDma functions. As they are for bare metal application nor for Linux.

[max.giacometti@gmail.com]

Is the mmap that allows you to use these functions

fd_uio0 = open("/dev/uio0", O_RDWR);
XAxiDmaConfigInstance->BaseAddr = (u32) mmap(NULL, BUFFER_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd_uio0, 0);

[schardt]

Hi,

i try to use the XAxiVdma functions as you mentioned above.  Do you use the header and libs generated with the standalone bsp ? How do you include it to your application ? I have some linker error when linking my main.o with the libxil.a:

arm-linux-gnueabihf-gcc  -o "ReadDMATest.elf"  ./src/main.o  /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a 
/opt/Xilinx/SDK/2016.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin/../lib/gcc/arm-linux-gnueabihf/5.2.1/../../../../arm-linux-gnueabihf/bin/ld: Warnung: /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(abort.o) nutzt variable-size enums, die Ausgabe nutzt jedoch 32-bit enums; Die Benutztung von enum-Werten in Objekten könnte scheitern
/opt/Xilinx/SDK/2016.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin/../lib/gcc/arm-linux-gnueabihf/5.2.1/../../../../arm-linux-gnueabihf/bin/ld: Warnung: /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(_exit.o) nutzt variable-size enums, die Ausgabe nutzt jedoch 32-bit enums; Die Benutztung von enum-Werten in Objekten könnte scheitern
makefile:36: recipe for target 'ReadDMATest.elf' failed
/opt/Xilinx/SDK/2016.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin/../lib/gcc/arm-linux-gnueabihf/5.2.1/../../../../arm-linux-gnueabihf/bin/ld: Warnung: /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(open.o) nutzt variable-size enums, die Ausgabe nutzt jedoch 32-bit enums; Die Benutztung von enum-Werten in Objekten könnte scheitern
/opt/Xilinx/SDK/2016.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin/../lib/gcc/arm-linux-gnueabihf/5.2.1/../../../../arm-linux-gnueabihf/bin/ld: Warnung: /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(xaxivdma.o) nutzt variable-size enums, die Ausgabe nutzt jedoch 32-bit enums; Die Benutztung von enum-Werten in Objekten könnte scheitern
/opt/Xilinx/SDK/2016.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin/../lib/gcc/arm-linux-gnueabihf/5.2.1/../../../../arm-linux-gnueabihf/bin/ld: Warnung: /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(errno.o) nutzt variable-size enums, die Ausgabe nutzt jedoch 32-bit enums; Die Benutztung von enum-Werten in Objekten könnte scheitern
/opt/Xilinx/SDK/2016.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin/../lib/gcc/arm-linux-gnueabihf/5.2.1/../../../../arm-linux-gnueabihf/bin/ld: Warnung: /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(xil_assert.o) nutzt variable-size enums, die Ausgabe nutzt jedoch 32-bit enums; Die Benutztung von enum-Werten in Objekten könnte scheitern
/opt/Xilinx/SDK/2016.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin/../lib/gcc/arm-linux-gnueabihf/5.2.1/../../../../arm-linux-gnueabihf/bin/ld: Warnung: /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(xaxivdma_channel.o) nutzt variable-size enums, die Ausgabe nutzt jedoch 32-bit enums; Die Benutztung von enum-Werten in Objekten könnte scheitern
/opt/Xilinx/SDK/2016.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin/../lib/gcc/arm-linux-gnueabihf/5.2.1/../../../../arm-linux-gnueabihf/bin/ld: Warnung: /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(xil_printf.o) nutzt variable-size enums, die Ausgabe nutzt jedoch 32-bit enums; Die Benutztung von enum-Werten in Objekten könnte scheitern
/opt/Xilinx/SDK/2016.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin/../lib/gcc/arm-linux-gnueabihf/5.2.1/../../../../arm-linux-gnueabihf/bin/ld: Warnung: /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(outbyte.o) nutzt variable-size enums, die Ausgabe nutzt jedoch 32-bit enums; Die Benutztung von enum-Werten in Objekten könnte scheitern
/opt/Xilinx/SDK/2016.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin/../lib/gcc/arm-linux-gnueabihf/5.2.1/../../../../arm-linux-gnueabihf/bin/ld: Warnung: /home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(xuartps_hw.o) nutzt variable-size enums, die Ausgabe nutzt jedoch 32-bit enums; Die Benutztung von enum-Werten in Objekten könnte scheitern
/home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(errno.o): In Funktion »__errno«:
errno.c:(.text+0x0): Warnung: undefinierter Verweis auf »_impure_ptr«
errno.c:(.text+0x4): Warnung: undefinierter Verweis auf »_impure_ptr«
/home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(xil_printf.o): In Funktion »getnum«:
xil_printf.c:(.text+0xc): Warnung: undefinierter Verweis auf »__ctype_ptr__«
xil_printf.c:(.text+0x14): Warnung: undefinierter Verweis auf »__ctype_ptr__«
/home/schardt/projekte/vivado/PMOD_ADC1/PMOD_ADC1.sdk/standalone_bsp_1/ps7_cortexa9_0/lib/libxil.a(xil_printf.o): In Funktion »xil_printf«:
xil_printf.c:(.text+0x218): Warnung: undefinierter Verweis auf »__ctype_ptr__«
xil_printf.c:(.text+0x220): Warnung: undefinierter Verweis auf »__ctype_ptr__«
collect2: error: ld returned 1 exit status
make: *** [ReadDMATest.elf] Error 1

Thanks in advance

Best regards

Georg

[suppamax]

Hi Georg,

unfortunately quite some time passed and I don't remember the details of the implementation.

I did use the files created by SDK, but I cannot recall how I did generate them - it's possible I had them through the BSP.

I remember there was a wiki page somewhere in Xilinx's universe which was illuminating on this topic, but I cannot find it any more.

[schardt]

Thanks, i will try to find these pages

at least i can talk to the dma controller via register :)