cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
clemus
Visitor
Visitor
2,296 Views
Registered: ‎01-31-2018

Zynq-7000 frame(s) stuck in Gigabit Ethernet RX FIFO

Jump to solution

In the Xilinx Zynq-7000-TRM, there is a known issue describing

 

" It is possible to have the last frame(s) stuck in the RX FIFO with the software having no way to get the last frame(s) out of there. The GEM only initiates a descriptor request when it receives another frame. Therefore, when a frame is in the FIFO and a descriptor only becomes available at a later time and no new frames arrive, there is no way to get that frame out or even know that it exists.

This issue does not occur under typical operating conditions. Typical operating conditions are when the system always has incoming Ethernet frames. The above mentioned issue occurs when the MAC stops receiving the Ethernet frames.

Workaround: There is no workaround in the software except to ensure a continual flow of Ethernet frames."

 

 

Has there been any work done or news on this that may solve the issue? Does anyone have any ideas on some alternative solutions?

 

Unfortunately, I am not running in "typical operating conditions". A simple test for example: a simple python script sending 4 Ethernet packets with no delay in between and then a 20ms gap before the next set of messages for 5000 iterations. 

Like the problem is described above, I am not missing any packets. The timing of the messages is off because the message is not received in a C program I wrote until the next set of messages comes through.

 

 

0 Kudos
1 Solution

Accepted Solutions
hbucher
Scholar
Scholar
2,929 Views
Registered: ‎03-22-2016

@clemus@ken.sallings

You can ping your partner from sockets code - need to allow user in sysctl.conf though. 

https://github.com/HFTrader/Forums/tree/master/Xilinx/Zynq-7000-frame-s-stuck-in-Gigabit-Ethernet-RX-FIFO

 

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.

View solution in original post

8 Replies
ken.sallings
Visitor
Visitor
2,294 Views
Registered: ‎11-29-2017

I am not sure if this works or not, but I'd try sending an Ethernet frame to yourself periodically.  Not the most elegant solution, but you could bound the maximum delay to whatever period was chosen.   Since this has been on my mind for a while, I'd appreciate ideas, too.

 

clemus
Visitor
Visitor
2,288 Views
Registered: ‎01-31-2018

Hey thanks for the Idea! 

Unfortunately, I have tried that already. What I've seen and read is that when you try sending Ethernet frames to yourself it skips the data and phy layers of the MAC so it doesn't really act the same way as receiving a message externally. 

 

I have tested sending like an extra message in my testing after the 20ms gap and it has solved the issue (because it's basically simulating continuous Ethernet packets), but this may not be a real solution for me.

To test the above: 

4 messages

20ms 

1 message

1 ms

 

 

0 Kudos
hbucher
Scholar
Scholar
2,930 Views
Registered: ‎03-22-2016

@clemus@ken.sallings

You can ping your partner from sockets code - need to allow user in sysctl.conf though. 

https://github.com/HFTrader/Forums/tree/master/Xilinx/Zynq-7000-frame-s-stuck-in-Gigabit-Ethernet-RX-FIFO

 

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.

View solution in original post

clemus
Visitor
Visitor
2,194 Views
Registered: ‎01-31-2018

@hbucher

 

I'm not sure this code is working still. The ping code doesn't seem to trigger the Ethernet receive the same way that an external source does. Just visually, the receiver light for the Ethernet port doesn't come on with just the ICMP, but it does when it receives messages from the PC. I verified that it worked because I get the ICMP replies and i properly changed the ipv4 ping group. I can verify that the messages are late for the last port (always) on wireshark because I can see there will be a 20ms delay between most packets and then a message will arrive 20ms late so 40ms (which makes sense with the next set of messages come in), then immediately the message that was stuck in the RX FIFO.  It would be too much to  post all my code but here's a pseudo of what I did just to get a picture. 

 

C Receiver with zynq-7000:

void *receiveEthernetThread(void *args){

        if(Receive UDP packet for one port){
                 //get time 
                 if(time between previous packet and current packet  was too fast or 
                   too slow){
                           //post error
                    }
//Send back to PC for wireshark to pick up
} } pingcodethread(){ //Exactly like the example from https://github.com/HFTrader/Forums/blob/master/Xilinx/Zynq-7000-frame-s-stuck-in-Gigabit-Ethernet-RX-FIFO/pingit.c } main(){ //Create 4 receiveEthernetThread with 4 different ports //Run pingcode as another thread //block }

Python script from PC:

for(i < 5000; i++):
    send to port 1
    send to port 2
    send to port 3
    send to port 4
    wait 20ms

It's possible I'm still doing something wrong, or there is no solution.

 

0 Kudos
hbucher
Scholar
Scholar
2,192 Views
Registered: ‎03-22-2016

@clemus You'd need to ping the partner, ie the other computer that you are connected to. Not your own interface.

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.
clemus
Visitor
Visitor
2,186 Views
Registered: ‎01-31-2018

@hbucher Wow! okay, thank you this seems to work now. I'm sorry I missed that. Yeah I just changed the destination of the ping to be that of the server and looking at the wire shark messages received everything seems to be around the 20ms. Do you know if pinging that often will have any effect on the network? Thanks again! 

0 Kudos
hbucher
Scholar
Scholar
2,177 Views
Registered: ‎03-22-2016

@clemus Well you can flood the network. Ping flood is a known form of DDOS (denial of service) attack. So use carefully!

That said, just make some back-of-napkin calculations to make sure you are not exceeding say 10% of the traffic.

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.
0 Kudos
svdmark
Adventurer
Adventurer
969 Views
Registered: ‎08-17-2009

Hi,

I think I'm seeing the same problem with TX:

From time to time, a packet I want so send gets stuck, and is only transmitted if i send another packet (this may be 30 seconds later... there seems to be no time out).

Has anyone else seen this problem with TX?

Thanks,

Stefan

 

0 Kudos