PCIE DMA subsystem AXI4 Streaming c2h transfers

[jadeanderson]

I am trying use the Xilinx PCIE DMA Subsytem with my KCU105 board.  The problem I have is that when I use the Card2Host AXI streaming channel, I get more data than I expect.

I haven’t found a way to get data back to the host without generating an AXI TLAST pulse.  So I have a counter that counts the number of 32byte AXI4-S  Data transfers, then it generates a pulse on the Tlast input of the DMA C2H port.   I must be doing this incorrectly b/c when I send the pulse, the DMA should sense that the transfer is finished, and then stop the request for data.  

I have data flowing from the ADC and it is valid every clock cycle, so I need to get that into the HOST and do so without losing samples.  I have plenty of bandwidth on the FPGA to Host.

I have not figured out how to do this cleanly.  I get the following:   

I have my transfer size and my internal FPGA counter set exactly the same, so for a 131072 byte transfer I should get 1 tlast pulse.

 ./dma_from_device -d /dev/xdma0_c2h_0 -f output_data.bin -s 131072 -c 1

sscanf() = 1, value = 0x00020000

sscanf() = 1, value = 0x00000001

device = /dev/xdma0_c2h_0, address = 0x00000000, size = 0x00020000, offset = 0x00000000, count = 1

host memory buffer = 0x7f9639ad3000

CLOCK_MONOTONIC reports 0.000112370 seconds (total) for last transfer of 131072 bytes

Done

Tail OF THE DMESG LOG on Linux HOST PC

[105773.751115] ch_irq = 0x00000002

[105773.751126] ch_irq = 0x00000002

[105773.751136] ch_irq = 0x00000002

[105773.751146] ch_irq = 0x00000002

[105773.751157] ch_irq = 0x00000002

[105773.751167] ch_irq = 0x00000002

[105773.751177] ch_irq = 0x00000002

[105773.751187] ch_irq = 0x00000002

[105773.751197] ch_irq = 0x00000002

[105773.751208] ENTERING char_sgdma_close,total read=370

[105773.751209] ch_irq = 0x00000002

[105773.751218] ch_irq = 0x00000002

The ILA in my design shows that the c2h_s_axis_tready line is ready for more data,  I thought it should go low and stay low after the transfer was complete. The main problem is that I am blindly generating the TLAST signal based on a hard coded transfer size and I can only request that size in the host.  There must be a better way to send data samples from FPGA to Host with this C2H streaming interface. If not, then at the least, I need clean way to reset the TLAST counter at the beginning of a transfer.  If I knew the number of bytes requested, then I could load the counter value at the beginning of the transfer and then send the tlast pulse at the exact boundary.

Thanks

[venkata]

s_axis_c2h_tready_x is an indication that DMA is ready to accept the data. You have to assert s_axis_c2h_tvalid_x HIGH whenever you have valid data. If you do not have valid data to send, you may assert s_axis_c2h_tvalid_x LOW.

When you finish the transfer of data, you can assert s_axis_c2h_tlast_x HIGH in the last cycle of the data and assert s_axis_c2h_tvalid_x to be LOW from the next cycle.

[jadeanderson]

I tried doing this, but the problem that I face now is that there are so many channel Interrupts that it locks up the host. So can you point me to a simple AXI streaming generator that I can try ?  I built my own, but I don't know how to control the fact that I always have valid data since my application is ADC > FIFO > PCIE_DMA

I was going to mimic this with a simple verliog model.

drive t_data port with a counter

drive t_valid and t_last based on the rising edge of the t_ready from the host.

See attached file.

Also, can I am trying to use the latest Linux driver with version 2016.2 of Vivado, I assume the driver is backward compatible. Is that true?

Thanks

[xiaohu125]

Hi,

Do you mean that you couldn't assert tlast signal because of the ADC data stream is continous? If that, you don't worry because pcie bandwidth is enough high for you to assert it.

I have simulated your verilog file, it seems like that the signal timing is not right.

[jadeanderson]

Yes this is a streaming application where the ADC data is continuous.  I agree that the timing is not correct, I have modified the file and now I have this timing. Which looks correct to me,

Here is the Sim.. 

however it is still not working correctly when I run a test in hardware.  it hangs the linux host b/c the flow of data seems to never stop. I expected that when you run

 ./dma_from_device -d /dev/xdma0_c2h_0 -f out_1.bin -s 8192 -c 1

 There should be a way for the host to know when the data has arrived.  But in this case it doesn't.  

Here is the Hardware ILA.

Also attached is the dmesg log, which was cleared immediately prior to the burst.

[jadeanderson]

Files for reference

[jadeanderson]

Does anyone have any suggestions?

[venkata]

Can you check the option to bypass the descriptor?

[nagabhar · Accepted solution]

HI @jadeanderson

Disabling cyclic buffer  might help your use case.
please replace ther xdma-core.c file in AR65444 driver with the the one attached  and then load the driver and run the tests  once

[jadeanderson]

Hi @nagabhar, 

Thanks for the help.  Turning off the cyclic buffer works much better in my application.  I was able to test with and without the cyclic transfers, and I see a very dramatic difference in the operation. I was seeing the HOST get flooded with interrupts b/c it was creating 256 Descriptors for each DMA transfer.

  By disabling the cyclic buffering, now when I  run the 'dma_from_device'  I see only 2 interrupts. (descriptor complete, and transfer done)  plus I get the exact amount of data that I asked for.

This is much easier to work with.

Thanks

Jade

[jadeanderson]

I tried that @venkata, however there is no documentation on how to use the descriptor bypass in the PG195 doc. For example there is a signal called c2h_dsc_byp_ctl[15:0]
The only description is:
"Input Descriptor control to be loaded". That isn't enough information to even get close. If there was more info on how to use this, I would have investigated further.
Thanks

[jadeanderson]

After further investigation. It looks like there are problems with disabling the cyclic read buffering.

I see entries in dmesg log that shows system memory was corrupted after I do a few c2h transfers.  I am not a software driver guy, so how might one debug this?

Thanks

Jade

[abba@elet.polimi.it]

Hi,

i have the same problem described in the post. Now the driver version in AR65444 is changed and your file wan't compile. Could you update with the last driver or tell me what to change.

I have an ILA in design and i see that even if i transfer just 1024 byte by the command 

sudo ./dma_from_device -d /dev/xdma0_c2h_0 -f /media/loki/file.bin -s 1024 -c 1

the transfer never stops

This is the block diagram

I think that your solution can also help me

Thank you

Andrea

[azbarks]

Has this been added to the newest driver code?  I ask because I don't see a xdma-core.c file in there. 

[r.sarwar87]

@azbarks I attached the file for the driver released on 20180420. It seems to work for me, and should be okay to use. I did not see any corruption upon repeated usage, albeit I have not extensively tested it.

it disables the cyclic sgdma transfer when using c2h streaming.

@nagabhar

thanks  for the hint, can i ask for an explanation of the bypass control ports? It would be most helpful. 

[leomcwong@astri.org]

Hi @nagabhar

I found the latest version "Xilinx_Answer_65444_Linux_Files_rel20180420.zip" still default enable "cyclic buffer transfer" when ST C2H, Would you please:

1. briefly explain what is "cyclic buffer transfer" ?
2. What is the advantage of it ?

Thank you.