UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Visitor george_fornt
Visitor
482 Views
Registered: ‎05-18-2017

AXI CDMA gets stuck transmitting forever

Jump to solution

Dear all,

I am working on a Computer Vision project with a Zynq z7020, and I am having some trouble with CDMA transfers.

I have 4 CDMA IPs running in parallel, one per AXI HP port of the PS. They work in Simple mode because my application needs to use all the BRAM in the PL, so I don't have enough memory to write descriptors in Block RAMs.

When I start the Application, the CDMAs run just fine for about a complete iteration of the PS program, until at some point on the 2nd iteration the 4 Blocks just stop generating interrupts. When I read the CDMASR Register of all of them, in all cases it returns 0x0 (no errors, no interrupts and not in idle), as if they were stuck forever in a transfer.

When measuring the AXI waveforms with an ILA Core, I see that the transfers finish as usual, but no interrupt is ever generated at the end by the CDMAs, like if the IP did not acknowledge it has finished.

I am quite sure that the problem comes from the DDR Addresses, because if I set the CDMAs to always point to the same DDR address (for reads and writes) the Application just runs smoothly and without the problem appearing at all. I am accessing to the same Address Range as the PS APU does (from 0x00100000 to 0x1FFFFFFF), and to make sure that the CDMAs do not overwrite the APU data and vice versa, I am defining static arrays with the size of my data and using a pointer to them as Address for the DDR. However, the problem still appears.

Once my CDMAs are stuck in this state of "transferring forever" I try to Soft-Reset them through AXI, but it has no effect.

So, my questions are:

1- Am I "reserving" the Memory Space for PL transfers correctly by defining these static arrays and then using these addresses? I feel that the answer is "no", but I can't seem to figure out how to do it correctly...

2- If I am indeed trying to write/read to an invalid memory region because of this, shouldn't the CDMA acknowledge this as an Error and show it in the CDMASR, or generate an Error Interrupt?

3- Once the CDMA, for whatever reason, is stuck in this behavior, what can I do to bring it back to life? Shouldn't a soft-reset accomplish this?

Thanks a lot,

George

0 Kudos
1 Solution

Accepted Solutions
Xilinx Employee
Xilinx Employee
451 Views
Registered: ‎10-04-2016

Re: AXI CDMA gets stuck transmitting forever

Jump to solution

Hi @george_fornt,

Could you check your AXI CDMA IPs and see if Store and Forward is enabled? If not, please enable it.

 

cdmaStoreAndForward.JPG

 

With its default settings, AXI CDMA can cause deadlock conditions on the AXI interfaces. What it does is issue the AR and the AW command at the same time. That means it is sending out the write command before the read data comes in. 

Let's say there was only two AXI CDMAs in your system. The deadlock situation looks like this:

1. CDMA0 and CDMA1 send out their read commands targeting DDR simultaneously. CDMA0 wins arbitration for the read channel.

2. CDMA0 and CDMA1 send out their write commands targeting DDR simultaneously. CDMA1 wins arbitration for the write channel.

3. DDR starts to return data to CDMA0.

4. CDMA0 can't forward data to the write path because it didn't win arbitration. It drop RREADY on the data read channel.

5. Because of #4, read data can't go to CDMA1. This means the CDMA1 write path that is blocking CDMA0 never completes.

6. Dead lock.

Without a block diagram, I can't be 100% sure that your system would is suseptible to this issue. The HPn ports do share paths to DDR inside the PS. I don't know what your paths to BRAM look like. Just know that this pops up when multiple AXI CDMAs are targeting the same slave memory.

Once you are in the deadlock state, there isn't a great recovery mechanism--I think you'd have to reset the PS to clear the stuck transactions if PS DDR is involved. Sometimes the system will run for awhile before it hits it because there is a timing component to it.

Enabling Store and Forward prevents deadlock because the AXI CDMA does not issue the write command until the read data command has completed. Basically, it never starts a write until it actually has the write data in its buffers. The down side of AXI CDMA in Store and Forward mode is that the IP consumes more FPGA resources since it has to include buffering logic.

Please give this a try and report back.

Regards,

Deanna

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
2 Replies
Xilinx Employee
Xilinx Employee
452 Views
Registered: ‎10-04-2016

Re: AXI CDMA gets stuck transmitting forever

Jump to solution

Hi @george_fornt,

Could you check your AXI CDMA IPs and see if Store and Forward is enabled? If not, please enable it.

 

cdmaStoreAndForward.JPG

 

With its default settings, AXI CDMA can cause deadlock conditions on the AXI interfaces. What it does is issue the AR and the AW command at the same time. That means it is sending out the write command before the read data comes in. 

Let's say there was only two AXI CDMAs in your system. The deadlock situation looks like this:

1. CDMA0 and CDMA1 send out their read commands targeting DDR simultaneously. CDMA0 wins arbitration for the read channel.

2. CDMA0 and CDMA1 send out their write commands targeting DDR simultaneously. CDMA1 wins arbitration for the write channel.

3. DDR starts to return data to CDMA0.

4. CDMA0 can't forward data to the write path because it didn't win arbitration. It drop RREADY on the data read channel.

5. Because of #4, read data can't go to CDMA1. This means the CDMA1 write path that is blocking CDMA0 never completes.

6. Dead lock.

Without a block diagram, I can't be 100% sure that your system would is suseptible to this issue. The HPn ports do share paths to DDR inside the PS. I don't know what your paths to BRAM look like. Just know that this pops up when multiple AXI CDMAs are targeting the same slave memory.

Once you are in the deadlock state, there isn't a great recovery mechanism--I think you'd have to reset the PS to clear the stuck transactions if PS DDR is involved. Sometimes the system will run for awhile before it hits it because there is a timing component to it.

Enabling Store and Forward prevents deadlock because the AXI CDMA does not issue the write command until the read data command has completed. Basically, it never starts a write until it actually has the write data in its buffers. The down side of AXI CDMA in Store and Forward mode is that the IP consumes more FPGA resources since it has to include buffering logic.

Please give this a try and report back.

Regards,

Deanna

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Highlighted
Visitor george_fornt
Visitor
429 Views
Registered: ‎05-18-2017

Re: AXI CDMA gets stuck transmitting forever

Jump to solution

Hello, Deanna,

That worked! The problem is actually gone by enabling Store and Forward.

The downside, as you said, is that now the CDMAs take more FPGA space. Besides being bigger in LUT and FF size they also take up some BRAMs that I don't currently have (1 to 2.5 BRAMs depending on the Burst Size, for what I have seen), but I can work around that and cut some memory from my design.

Thank you very much!

George

0 Kudos