AXI DMA changing some value to 0

[nitthin21]

I use the AXI DMA core to read an array from DRAM and sends it to my custom IP which will increment the array values and I use the S2MM channel to copy back the array to the same location in DRAM. I use this in an algorithm which does this operation iteratively. In some iterations, the array value at some position is becoming 0. If I use the same input, it always happens in the same iterations and at the same position within those iterations. Could it be some problem with the DMA?

[demarco]

Hi @nitthin21 ,

Do you have any traces to show the reads and writes to DRAM (M_AXI_S2MM and M_AXI_MM2S)? Ideally, your trace would also include the AXI DMA register interface (the AXI Lite interface) and the M_AXI_SG interface, if you are using it. You could also include the M_AXIS_MM2S and S_AXIS_S2MM to see what data went to your custom IP and what your custom IP returned.

It is unlikely that AXI DMA is writing incorrect data. It is more likely that there is something else going on in your system. Is there a processor that creates the read data for the AXI DMA to consume? How are you maintaining coherency between the processor and AXI DMA?

Regards,

Deanna

[nitthin21]

I'm using the DMA in simple mode, not SG. The array is located in a DRAM buffer allocated using dma_alloc_coherent call from the driver. My custom IP in hw consist of a SIPO shift register and a PISO shift register with adders in between, both shift registers having size 300 (64 bits each). I send 300 array values to the SIPO shift register, transfer them to the PISO shift register while incrementing all the values in parallel using the adders and the sent back the value from PISO to the DRAM. I am doing this iteratively for different arrays in the DRAM buffer. Since I'm using dma_alloc_coherent, I think there is no issue of coherency. Also in most of the iterations, I get the proper behaviour. But sometimes, the returned array has one of it's value set to 0. All other values are correct. There is no problem in any of the array values in the previous iteration or the iteration which comes after that. This is kind of confusing. I'm not able to find out where the problem is.

[nitthin21]

I initialize the DMA and set the interrupts once from the kernel module, when it is probed. Then for each transfer I just write the source/dest address and the length to the corresponding registers. The length is always 2400 as I'm transfering 300 64-bit values. 

[nitthin21]

Unfortunately I cannot provide you with a trace now as I'm currently not in my lab and it's 00:23 here in Germany. But it will be good for me to fix this as soon as possible. I guess you will not be avalable tomorrow, are you? @demarco 

[dgisselq]

@nitthin21,

Some of us still haunt these forums--even on weekends.  ;)

Let me recommend that you try two things.  First, generate a scope configuration that triggers off of writing a zero to the RAM.  Get a trace, examine the trace, etc.  That will prove that the value is getting written to RAM as you are surmising vs something else writing it to RAM.  Second, generate another example that triggers off of a zero getting sent to the S2MM core.  If that one triggers, you know the problem is upstream of the S2MM core.

As I see it, there are three useful places to make such a trace.  1) at the input to the S2MM core, 2) where the S2MM core is sending AXI out, and 3) at the other side of any interconnect where the AXI commands interact with the memory.  If you find a zero entering into the S2MM core, as in 1) above, that would point to a bug in your own code.  If you find no zeros entering into your S2MM core, but 2) still trips, that would point to a bug in the S2MM code.  If 2) doesn't trip but 3) does, that would point to something else writing to the memory.

Dan

[nitthin21]

@dgisselq Thank you Dan. I will try your suggestions and come back with the results.

[nitthin21]

One thing I noticed is that it is happening for specific physical addresses. For example, 0x257FF820 is one physical address where the value is becoming zero. Does this ring any bell? I am using Zynq Ultrascale ZCU102.

[nitthin21]

0x493820, 0x249820, 0x257FF820, 0x4afff820, 0x249b58 are the addresses where I noticed problem so far.

[demarco]

Hi @nitthin21 ,

It's unusual that those 4 of the 5 addresses have a pattern to them. Are the problem addresses consistent across test runs? Can you set up an ILA trigger to look for writes near that address to see what data it is writing?

Is there a chance that you are encountering some kind of alignment issue? Is the DRE enabled in AXI DMA?

Regards,

Deanna

[nitthin21]

I will set up the ILA now and see if it's writing the right value. What is DRE?

This is my current DMA configuration.

[demarco]

Hi @nitthin21 ,

The DRE is the data realignment engine. It handles the case when you tell the AXI DMA to read or write data to an unaligned memory address. It is disabled by default. 

From your screenshot, the DRE is disabled. You can enable it selecting the Allow Unaligned Transfers option under both the read and the write channels.

I'm not sure if this is what you are running into, but it is something to keep in mind.

Regards,

Deanna

[nitthin21]

I have checked 'Allow Unaligned Transfers' and tested earlier. Still, I was getting the error.

[nitthin21 · Accepted solution]

Finally, I was able to fix the issue. I created the AXI Stream Master and Slave interfaces for my custom IP in Verilog by modifying the code generated by Vivado. I was assuming that once the ready signal becomes high, it will stay high until the transfer is complete. But this was not the case, as shown in the waveform. For the addresses mentioned in one of the previous comment, the ready signal becomes low in between. This case was not handled in my custom IP which will send a 0 when the ready signal goes low in between. I modified the IP and now it is working fine.