06-01-2020 05:30 PM - edited 06-01-2020 05:31 PM
Could someone check out my device tree and PL block diagram to see if anything is wrong that I've missed? I can boot linux and use "ifconfnig eth1 up" to bring up the ethernet link and see no errors. I see the following:
I'm using the zedboard and the opsero FMC 4 port ethernet card. I'm also using the axi ethernet subsystem IP in PL.
06-01-2020 05:32 PM - edited 06-01-2020 05:49 PM
06-02-2020 06:42 PM - edited 06-02-2020 06:48 PM
I have figured out that my ethernet link actually isn't working and there is a problem on the receive side. I do (after waiting a while) get an error from the driver saying that my rx buffer length is zero. As you can see I have enabled "Use Rxlength in Status Stream" which I think is correct.
My error is the following:
The 0x15019 shows the status of the DMA status register in SG mode and the bit that represents my error (I think) is bit 4 which conveys the following information:
Here is a screenshot from vivado of the SG PL link that shows the SG read channel
does anyone know what would cause this? I can't find a device tree setting that would cause this.
06-04-2020 02:48 PM
I've finally gotten an ILA connected up to the SG link, the CTRL link and STS link of the DMA. It seems there is a lot of traffic, but the CTRL stream doesn't match what I would expect. It seems fairly asynchronous and might be due to the error I'm seeing. Does the DMAIntError bit mean that the buffer length in the APP section of the buffer descriptor is 0? If I'm not mistaken the APP words should be transmitted over the CTRL/STS links and not the SG link for this configuration.
How do you troubleshoot this from here? What part of the driver is supposed to be filling out this length value (or what part of the driver is supposed to be filling out the desciriptors) so I can try to get to the bottom of why my buffer descriptors are wrong? Below are some ILA screenshots of the traffic I'm seeing.
06-08-2020 07:30 AM
From the error log, it appears that the Rx interrupt handling is failed as the status bit is not set. The below parts of the code actually handles the BD after calling receive function
06-08-2020 12:44 PM - edited 06-08-2020 01:44 PM
It looks like I'm failing out before anything really happens. Does this mean there is a hardware/configuration problem with the DMA? It looks like the first thing the driver rx isr does is check these flags.
Why does this
irqreturn_t __maybe_unused axienet_rx_irq(int irq, void *_ndev)
say "maybe unused"?
Also, what status bit are you referring to in which register?
I didn't realize this was an RX error so I think the internal dma error bit I should be looking at is this
06-10-2020 03:02 PM
I think I may have finally located the rx data / buffer mismatch with the ILA. I can see the DMA reading the buffer descriptors with the read bus shown below. Slot_0 is connected to the DMA SG link. It looks like the buffer size here in my control byte of the SG read (S2MM) side is 1522 bytes. This seems reasonable since the MTU for ethernet is 1500.
The questionable part is the STS link from the ethernet MAC to the DMA. This is showing much different values in the APP fields. Particularly the APP4 fields which shows 0x11120076 which makes no sense to me. This is supposed to represent received bytes, but why would there be so many rx'd bytes? The state of the ethernet link is actually pretty inactive. For this test I just did "ifconfig eth1 up" which brought the eth1 up, but there is no IP address assigned to eth1 yet (no ipv4 ip address).
I also see no SOF/EOF bits set in any of the SG descriptor transactions.
Are APP0-3 fields not used - the data in them meaningless? The DMA users guide doesn't seem to be specific about what exactly they are supposed to contain and whether they are even useful. APP4 seems to just be the total number of bytes rx'd is that correct?
06-11-2020 04:48 AM
Do you see any Tx and RX count other than 0 with ifconfig output?
I will check further to see if there are any issues with your DMA configurations.
I hope you took the reference as XAPP1082 for zynq-7000, which also make use of PL(AXI 1g/2.5g) ethernet with AXI DMA
06-11-2020 09:59 AM
Hi @shabbirk I did try to use xapp1082, but unfortunately there are too many missing details regarding how to implement this with the driver. I will check today, but I am pretty sure there is no TX and when I ping nothing happens. I also don't see any packets with Wireshark. Same with RX - I believe that is always zero with 100% packet loss as well.
I will verify today.
06-11-2020 10:34 AM
Well very much to my surprise I do see a RX/TX packet count. This is very interesting because I can't ping anything (in or out of the zynq) but do see packet counts increase in ifconfig.
I've also noticed that I can't connect an ILA to the receive side of the axi ethernet MAC IP. I was planning on looking at the received data stream along with the status/control links, but when I connect the axi stream rs data from the mac to dma the entire eth1 disappears. It seems that vivado or the axi ethernet mac really doesn't like when you connect an ILA to the RX or TX data stream in and out of the axi ethernet MAC IP block.
06-11-2020 11:09 AM - edited 06-11-2020 01:41 PM
Some other questions that may be helpful for me troubleshooting this:
1. How do I print the buffer descriptors for RX and TX to console. Since these are all preset in software - does the driver allow me to dump the BD contents so I can verify the information is correct?
2. Does the status stream only move/communication the rx byte count? The users guide for the DMA doesn't really say what values APP0-3 should contain.
3. The ethernet link I'm struggling with comes up with IPv6 information as you can see in the screenshots, but no IPv4 info (ip address, netmask etc...). Would it be beneficial for me to disable IPv6 all together and try to get the eth1 link to automatically come up and pull an IP address?
06-15-2020 11:57 PM
The issue in your case seems to be likely with the design. Debugging it from the software/driver point is challenging to catch the issues with DMA configurations, and I would rather suggest to take the reference design as a baseline and develop the custom design. We also have AXI ethernet dma based design targeting zynq-7000 here:
06-17-2020 12:15 PM
Hi @shabbirk thanks for the reply! I have overlooked some things in that example and am still debugging. I have a question that is probably fairly obvious to you, but I'm having a hard time understanding it.
In this file:
is added. This seems to imply a conversion between GMII to RGMII has to happen. Why is this? Is there no native support for RGMII? Also, what is this being applied to - the driver?
06-18-2020 02:03 PM
Hi @shabbirk ,
I've made some substantial progress. There were a few errors in my project when I went back and looked at the example you linked. I'll note them for others:
1. RX/TX checksum hardware offloading not checked
2. Clock sources were connected to some nodes incorrectly - followed BD in example
3. Added additional source files from link petalinux repo. You need the common, port-0123 and zedboard directories.
4. Apply xilinx axi ethernet driver patch found here
5. Added constraints file as well to constrain RGMII path (found in linked git repo - vivado project) and set the "late" property for vivado compilation
With that said, I've managed to get the link up and running and can ping in/out. I am unfortunately still getting the same error, but it looks to happen less frequently. Is there a way I can figure out what is going on by catching this with an ILA in the PL?
Image of what I'm seeing in console attached
06-19-2020 01:40 PM
I've also finally found that the example project from opsero (eth fmc manufacturer) has the same dma rx problem. So I think my project is working as good as the example project in github you linked me to works.
06-24-2020 12:21 AM
yes, this change is applied to the driver. Basically EMIO based Ethernet connection (that has gmii interface) uses the Gmii-to-Rgmii IP between the PS and the Ethernet PHY, as most of the PHY's are rgmii compatible
Do you have any update on the progress of this issue?
06-24-2020 02:35 PM
Hi @shabbirk ,
Thanks for the reply that makes sense. So the PS/EMIO side expects only GMII? This confused me because the PL MAC IP core allows you to configure the MAC interface for any of these.
I don't particularly have a final solution, I've rebuilt the project from the company that makes the card and see the same issue. I got a prebuilt image from them that does not seem to have this issue. I'm further investigating the build process and some constraints, but so far I'm still unsure. There are some HDL constraints I don't understand so maybe there is a timing problem I just haven't been able to figure out yet. The design meets timing, but I noticed there are a couple lines in a constraint file that are not present in my custom project.