UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Contributor
Contributor
1,161 Views
Registered: ‎09-13-2019

AXI crossbar acknowledges write address operation but does not forward

Jump to solution

I have a design centered around an AXI crossbar. The design has 8 masters and 2 slaves, the slaves being DDR4 controllers.

The the beginning of the simulation the master starts posting awaddr signals which get acknowledged from the interconnect. Problem is the interconnect doesn't propagate the signals on the other side.

The interconnect (xilinx.com:ip:axi_interconnect:2.1) was generated with the IP integrator (vivado 2019.1). 

I briefly connected the master directly to the DDR4 memory and the write operations were posted and completed. It seems similar problems were posted some time back and not resolved:

https://forums.xilinx.com/t5/Processor-System-Design/AXI-Crossbar-problem/td-p/332045

Waveform below.

AXI Interconnect fail.png
0 Kudos
1 Solution

Accepted Solutions
Contributor
Contributor
213 Views
Registered: ‎09-13-2019

Re: AXI crossbar acknowledges write address operation but does not forward

Jump to solution

I found the bug: because the ddr4 slave had been generated with smaller BID, some upper BID bits were temporarily set to '0' ...

That means that the 0-th master (SLAVE PORT S00 in interconnect) will receive the BVALID/BRESP despite the fact that the master (data producer) connected at S06 port made the request ... This is verified by the fact that at the MASTER PORT M00 of 1st interconnect the request with AWID x30 ...

See attached signals.

I will regerenate everything with the same BID length and see if everything checks out and there are any more issues.

With the above observation I doubt the clocks in the interconnect play any role, I don't think there is a bug there. The issue manifested itself all the same. When the interconnect is synchronous, it might just take a different amount of time to show up.

View solution in original post

AXI Interconnect - BID upper bits cut - RESOLVED.jpg
AXI Interconnect - BID of request - RESOLVED.jpg
0 Kudos
11 Replies
Scholar dgisselq
Scholar
1,119 Views
Registered: ‎05-21-2015

Re: AXI crossbar acknowledges write address operation but does forward

Jump to solution

@dimitris78,

Can you also add to your trace the write data channel?  One could arguably avoid forwarding a write address before the data was available as a bus optimization.

Likewise, an interconnect should never forward a request for an address that isn't mapped in the interconnect.  Were this the case, it should wait for WVALID & WREADY & WLAST and return a BVALID & BRESP where BRESP[1] is set.

Finally, it is an AXI protocol violation to wait for the write address to be accepted before asserting WVALID--another reason for asking about the write data channel.

Dan

0 Kudos
Contributor
Contributor
984 Views
Registered: ‎09-13-2019

Re: AXI crossbar acknowledges write address operation but does not forward

Jump to solution

Hi Dan,

Indeed asserting the wvalid after awready is non-compliant, it doesn't seem to be the issue though. A short bit after the AW handshake wvalid is asserted, a few data transactions are accepted, and before the burst completes wready is lowered and stays like that.

Here's the waveforms for the data channel as well. 

 

AXI Interconnect fail - data channel.jpg
0 Kudos
Scholar dgisselq
Scholar
939 Views
Registered: ‎05-21-2015

Re: AXI crossbar acknowledges write address operation but does forward

Jump to solution

@dimitris78,

This is the behavior I would expect from a master attempting to access a memory area that isn't mapped in the interconnect.  My guess is that as soon as you finish writing to the interconnect and then set WLAST (which I never saw set in this trace) that you'd get a BVALID & BRESP[1].  You might want to check what memory addresses have been assigned to the downstream memory by the interconnect.

That said, I'm a bit surprised that I never saw WLAST & WVALID in this trace. Are there further write data beats outstanding?  Or did you master just never set WLAST?

Let me also ask, have you tried formally verifying your master to see if it follows the AXI bus protocol?  There are a lot of AXI pitfalls that are known for locking up the bus if a design is not properly formally verified.

Dan

0 Kudos
Contributor
Contributor
896 Views
Registered: ‎09-13-2019

Re: AXI crossbar acknowledges write address operation but does not forward

Jump to solution

There are 64 write transactions in the programmed burst and something like 15 are acknowledged. Then the bus hangs.
With regards to the addressing scheme it is very simple, two devices sharing a 4GB space half and half.

I didn't know Xilinx had tools available for verification. Can you point to anything ?

0 Kudos
Highlighted
Scholar dgisselq
Scholar
808 Views
Registered: ‎05-21-2015

Re: AXI crossbar acknowledges write address operation but does forward

Jump to solution

@dimitris78 

Looking over your trace, I don't see any cycles with WLAST high.  Given this, your following statement makes no sense:


There are 64 write transactions in the programmed burst and something like 15 are acknowledged. Then the bus hangs.

There should only be one acknowledgement per burst--not per beat.  Hence, for every AWVALID & AWREADY there should also be one WVALID & WREADY & WLAST.  Then for every WVALID & WREADY & WLAST there will be one BVALID & BREADY.  Given that WLAST is never raised in your trace, I becoming convinced that your master isn't following the AXI protocol in the first place.  This could then explain much of what you are seeing above.


With regards to the addressing scheme it is very simple, two devices sharing a 4GB space half and half.

It sure sounds simple to hear you describe in like that.  That said, an interconnect is responsible for determining whether or not an address given to it exists within the set of slave addresses it is responsible for.  This is a required part of the configuration of the interconnect that I believe is in error in your case--a part that you would notice if your master were following protocol.  You may wish to go back and double check how you've customized the AXI interconnect IP as I have been suggesting for some time.


I didn't know Xilinx had tools available for verification. Can you point to anything ?

I have been using SymbiYosys for formal verification, and building my own AXI properties for verifying AXI components both slaves and masters.  I've used both the free version of SymbiYosys for this task as well as the commercial version.  I've also been blogging and tweeting about the results I've come across.  As linked above, and briefed at ORCONF this year, I've found many bugs in AXI interfaces using these tools.  This includes in both Xilinx's and Intel's demonstration designs as well as several of Xilinx's IP components.  I've also posted most of the AXI cores that I've written using these formal properties in an open github repository, where you can see various open cores I've built--to include an AXI interconnect that you might find easier to debug .

Hope this helps,

Dan

P.S.  No, I don't work for Xilinx.

Contributor
Contributor
805 Views
Registered: ‎09-13-2019

Re: AXI crossbar acknowledges write address operation but does forward

Jump to solution

By acknowledgement I meant WVALID/WREADY handshake (not WLAST or BVALID/BREADY).

Thanks, I'm looking over things, still puzzling because of the simplicity of the setup ...

0 Kudos
Contributor
Contributor
415 Views
Registered: ‎09-13-2019

Re: AXI crossbar acknowledges write address operation but does not forward

Jump to solution

There was a bug before, two of the clocks in the interconnect had the wrong frequency, nevertheless fixing it had no effect. 

I went on and removed one of the slaves in the design so the address space is not shared to rule that one out.

The original interconnect had different clocks for various devices but there might be a bug there as was previously documented here:

https://forums.xilinx.com/t5/Processor-System-Design/AXI-Interconnect-Bug-with-different-clock-rates-on-Master-Slave/td-p/935411

So then I set all the clocks in the interconnect to be the same. Since the slave needs to have a different clock, I added a 2nd 1x1 interconnect between the 1st interconnect and the slave, with the sole purpose of clock conversion.

The outcome: the first two bursts are acknowledged and WLAST is produced, but then the bus locks in exactly the previous manner: the AWVALID/AWREADY hadshake happens but the data burst is not completed. See signals in the figure below. In the second figure, the 2nd interconnect signals are shown: the interconnect does not receive any AWVALID signal after the first 2 transactions so it's the 1st interconnect that locked... 

 

AXI Interconnect fail - Partial throughput.jpg
AXI Interconnect fail - 2nd Interconnect.jpg
0 Kudos
Contributor
Contributor
398 Views
Registered: ‎09-13-2019

Re: AXI crossbar acknowledges write address operation but does not forward

Jump to solution

Also, although the first two transactions have WLAST, the don't have BID/BVALID/BREADY handshake. The BID/BVALID/BREADY is generated by the DDR and the 2nd interconnect passes that signal through. It doesn't however go through the 1st interconnect. Signals below: the data producer that is attached to the slave port S06 never receives the BID/BVALID signals.

AXI Interconnec fail - No BVALID transmit to producer.jpg
0 Kudos
Scholar dgisselq
Scholar
301 Views
Registered: ‎05-21-2015

Re: AXI crossbar acknowledges write address operation but does not forward

Jump to solution

@dimitris78,

One last check ... since your schematics are so far zoomed out, it's hard to tell ... but are you setting AWLEN to one less than the length of the burst?  That is, if AWLEN = 15, there should follow 16 WVALID & WREADY's with the last having WLAST enabled--not 15.  Can you double check that this is done properly within your core?

@calebd,

Since you were part of the cited conversation above regarding the interconnect not crossing clock domains properly, do you see anything missing here?  I'm not seeing anything more.

Dan

0 Kudos
Xilinx Employee
Xilinx Employee
249 Views
Registered: ‎01-09-2019

Re: AXI crossbar acknowledges write address operation but does not forward

Jump to solution

I do not see that this design is exactly the same as the other forum post.  @dimitris78 could you provide a full block diagram of your design with the interconnects expanded?

We did not have a recreate of the issue on the other post, so we did not get to an actual root cause of that issue.  With regards to this issue, can you recreate with all the same clock frequency/domain?  That would be a good data point to show whether the clocking is even relevant in this issue.

Thanks,
Caleb
-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Contributor
Contributor
214 Views
Registered: ‎09-13-2019

Re: AXI crossbar acknowledges write address operation but does not forward

Jump to solution

I found the bug: because the ddr4 slave had been generated with smaller BID, some upper BID bits were temporarily set to '0' ...

That means that the 0-th master (SLAVE PORT S00 in interconnect) will receive the BVALID/BRESP despite the fact that the master (data producer) connected at S06 port made the request ... This is verified by the fact that at the MASTER PORT M00 of 1st interconnect the request with AWID x30 ...

See attached signals.

I will regerenate everything with the same BID length and see if everything checks out and there are any more issues.

With the above observation I doubt the clocks in the interconnect play any role, I don't think there is a bug there. The issue manifested itself all the same. When the interconnect is synchronous, it might just take a different amount of time to show up.

View solution in original post

AXI Interconnect - BID upper bits cut - RESOLVED.jpg
AXI Interconnect - BID of request - RESOLVED.jpg
0 Kudos