XDMA driver - test scripts error messages

[sktpin]

Hello,

I have set up an x86 Linux system to try out XDMA on the platform it is specified for. Connected to the PCIe slot is an Artix7 FPGA which has, among other things, the XDMA IP block in it.
The driver loading script reports to load the driver correctly, and the device was recognized.

Then I started run_test.sh, and got the output below.
What might the "Unknown error 512" be?

Further below, I'll copy the Xilinx section of my "lspci -vv" output, in case that matters.

Edit: note 2nd post with kernel log excerpts, containing "timed out", "abort transfer".

Xilinx_Answer_65444_Linux_Files_rel20180420/tests# ./run_test.sh 
Info: Number of enabled h2c channels = 1
Info: Number of enabled c2h channels = 1
Info: The PCIe DMA core is memory mapped.
Info: Running PCIe DMA memory mapped write read test
      transfer size:  1024
      transfer count: 1
Info: Writing to h2c channel 0 at address offset 0.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_0, W off 0x0, 0xffffffffffffffff != 0x400.
write file: Unknown error 512
Info: Writing to h2c channel 0 at address offset 1024.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_0, W off 0x400, 0xffffffffffffffff != 0x400.
write file: Unknown error 512
Info: Writing to h2c channel 0 at address offset 2048.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_0, W off 0x800, 0xffffffffffffffff != 0x400.
write file: Unknown error 512
Info: Writing to h2c channel 0 at address offset 3072.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_0, W off 0xc00, 0xffffffffffffffff != 0x400.
write file: Unknown error 512
Info: Reading from c2h channel 0 at address offset 0.
Info: Wait for the current transactions to complete.
/dev/xdma0_c2h_0, R off 0x0, 0xffffffffffffffff != 0x400.
read file: Unknown error 512
Info: Reading from c2h channel 0 at address offset 1024.
Info: Wait for the current transactions to complete.
/dev/xdma0_c2h_0, R off 0x0, 0xffffffffffffffff != 0x400.
read file: Unknown error 512
Info: Reading from c2h channel 0 at address offset 2048.
Info: Wait for the current transactions to complete.
/dev/xdma0_c2h_0, R off 0x0, 0xffffffffffffffff != 0x400.
read file: Unknown error 512
Info: Reading from c2h channel 0 at address offset 3072.
Info: Wait for the current transactions to complete.
/dev/xdma0_c2h_0, R off 0x0, 0xffffffffffffffff != 0x400.
read file: Unknown error 512
Info: Checking data integrity.
cmp: EOF on data/output_datafile0_4K.bin
Error: The data written did not match the data that was read.
       address range:   0 - 1024
       write data file: data/datafile0_4K.bin
       read data file:  data/output_datafile0_4K.bin
cmp: EOF on data/output_datafile1_4K.bin
Error: The data written did not match the data that was read.
       address range:   1024 - 2048
       write data file: data/datafile1_4K.bin
       read data file:  data/output_datafile1_4K.bin
cmp: EOF on data/output_datafile2_4K.bin
Error: The data written did not match the data that was read.
       address range:   2048 - 3072
       write data file: data/datafile2_4K.bin
       read data file:  data/output_datafile2_4K.bin
cmp: EOF on data/output_datafile3_4K.bin
Error: The data written did not match the data that was read.
       address range:   3072 - 4096
       write data file: data/datafile3_4K.bin
       read data file:  data/output_datafile3_4K.bin
Error: Test completed with Errors.
Error: Test completed with Errors.

lspci -vv yields:

01:00.0 RAM memory: Xilinx Corporation Device 7011
	Subsystem: Xilinx Corporation Device 0007
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 18
	Region 0: Memory at fe000000 (32-bit, non-prefetchable) [size=8M]
	Region 1: Memory at fe800000 (32-bit, non-prefetchable) [size=64K]
	Region 2: Memory at d0000000 (64-bit, prefetchable) [size=16M]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [9c] MSI-X: Enable+ Count=31 Masked-
		Vector table: BAR=1 offset=00008000
		PBA: BAR=1 offset=00008fe0
	Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
	Kernel driver in use: xdma

[sktpin]

Enabling debug prints in the xdma driver Makefile, I see this in the log:

That looks suspicious (id mismatch, SKIP ?):

kernel: [  139.562592] xdma:__write_register: engine_init_regs: w reg 0x40fdde5285f8(0xffffc90001521090), 0xf83e1e.
kernel: [  139.562599] xdma:probe_for_engine: C2H 1 engine, reg off 0x1100, id mismatch 0x0,0x0,exp 0x1fc1,0x1, SKIP.

here one of the last blocks of when the tests/run_test.sh from the xdma driver folder was runing:
(note "timed out", "abort transfer")

kernel: [ 5895.117268] xdma:xdma_xfer_submit: xfer 0xffff88021fd26918,1024, s 0x1 timed out, ep 0x1000.
kernel: [ 5895.117284] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0xffffc90001521000) = 0x1fc10006 (id).
kernel: [ 5895.117293] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0xffffc90001521040) = 0x00000006 (status).
kernel: [ 5895.117300] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0xffffc90001521004) = 0x00f83e1e (control)
kernel: [ 5895.117307] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0xffffc90001525080) = 0xa43c0000 (first_desc_lo)
kernel: [ 5895.117314] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0xffffc90001525084) = 0x00000000 (first_desc_hi)
kernel: [ 5895.117321] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0xffffc90001525088) = 0x0000000f (first_desc_adjacent).
kernel: [ 5895.117328] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0xffffc90001521048) = 0x00000040 (completed_desc_count).
kernel: [ 5895.117335] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0xffffc90001521090) = 0x00f83e1e (interrupt_enable_mask)
kernel: [ 5895.117343] xdma:engine_status_dump: SG engine 0-C2H0-MM status: 0x00000006: DESC_STOPPED,DESC_COMPL
kernel: [ 5895.117349] xdma:transfer_abort: abort transfer 0xffff88021fd26918, desc 1, engine desc queued 0.
kernel: [ 5895.117354] xdma:xdma_engine_stop: xdma_engine_stop(engine=ffff880222ff8a78)
kernel: [ 5895.117360] xdma:xdma_engine_stop: Stopping SG DMA 0-C2H0-MM engine; writing 0x00f83e1e to 0xffffc90001521004.
kernel: [ 5895.117366] xdma:__write_register: xdma_engine_stop: w reg 0x40fdde52856c(0xffffc90001521004), 0xf83e1e.
kernel: [ 5895.117371] xdma:xdma_engine_stop: xdma_engine_stop(0-C2H0-MM) done

[yanghaofei@ni]

I have exactly the same problem. Have you already solved it?

[sktpin]

No, I have not further looked into it. The x86 system is not my real target platform. I only wanted to try whether the driver worked better on a x86 system than my ARM system, as there is that note on the answer 65444 site saying it's spec'd only for x86.
Then I get this here... ;)

[jrm]

I am having the exact same issue.  Anyone with a solution?  Someone from Xilinx?

[sktpin]

@jrm

Before you invest any further time, I suggest you digest this thread thoroughly and determine how much it applies to you:
https://forums.xilinx.com/t5/PCI-Express/C2H-Streaming-XDMA-Linux-Driver-Broken/m-p/833977

That said, on a x86_64 platform, I got a modified version of the driver to work, called "v2_xdma" by Wojciech Zabolotny:
Ah, here: https://gitlab.com/WZab/Artix-DMA1/tree/master/v2_xdma

On my ARM Cortex-A platform, though, the transfer would always abort after a set of buffers was transferred, I have since not investigated this further.

[yanjie_x]

Have you already solved it? The same issue occurs after adding an axi4_IP. How it comes !!!

[wh1t3w0lf]

I struggled with the "Unknown error 512" for 3 full days until I was ready to give up on the whole PCIe DMA Core + Driver. It happened on two Kintex-7 boards (KC705 + Enclustra one) and on 3 x86 machines (2 x86_64 + 1 x86_32).

The solution was very very simple and I was really stupid for not realising it quickly. I discovered it by luck while viewing the following video (skip to 12:15):

https://www.xilinx.com/video/technology/dma-for-pci-express.html

In that video it is mentioned that the tests try to DMA from 0x00000000 but most example designs and maybe your custom ones have the xdma core mapped on 0x0000000080000000 (0x80000000 for 32-bit) on the AXI bus.

Thus you need to change the scripts and add this offset as the video explains in 12:15. This will make the scripts call the ./dma_to_device and ./dma_from_device with -a (offset) of 0x80000000 and everything will work correctly on the XDMA IP Core + Driver from then on, at least for the normal AXI-MM case.

I suggest that this setup of the address offset gets added on the default script (or mention it on a comment or option) and also add this on the Xilinx Answer Record 71435 (XDMA Debug Guide pdf) and maybe mention it on the driver answer record 65444.

It will help a lot of people here that like me didn't immediately realize that the core aborted the transactions and didn't fire the completion interrupt not because it was stuck or due to a bigger problem with the PCIe link or the driver or the interrups or whatever, but just because the address offset was wrong and it couldn't fullfill the transaction on the wrong address.