cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
1,145 Views
Registered: ‎01-13-2020

ZYNQ MPSOC GEM External FIFO (no DMA) system configuration

Jump to solution

Dear All

I have been struggling with the configuration and the use of the external FIFO provided in the ZYNQ MPSOC.
I have read the manual and the tickets on this matter, made several design changes, but still have some doubts, and an interface that is not working.

I will start with what i already did and checked, this may be useful to others as well, then i will describe what the problem is and how to reproduce it.

1. The design:

Target : ZCU102  -  Vivado : 2019.2 - Board part :xilinx.com:zcu102:part0:3.3 - MPSOC IP : MPSOC3.3
The MPSOC is configured to use the external FIFO ports on GEM3 channel.
Check 1 : The pinning on the interface: These are automatically filled in because of the board selection.
The GEM uses MIO 64-75 and the MDIO uses pin 76 - 77. I have not changed any of the settings, these should be fine.
Check 2 : Check the box 'external fifo' in the GEM3 configuration

At this point the block design has added a port 'FIFO_ENET3'

Check 3 : The content of the FIFO_ENET3 port matches with the signals described in the TRM.

For debugging a system ILA is added, with GEM port type. The system ILA is clocked on the port 'fmio_gem3_fifo_tx_clk_to_pl_bufg' which is also automatically added.

Custom logic is written in VHDL, such that the TX and RX interface are correctly timed. The waveforms and suggestions from the forum are applied. Data is collected in BRAM and a DMA controller IP is used to get/set data from/to the BRAM.

Additionally a JTAG master is added, to provide access to the GEM3 register space in the MPSOC

The design in compiled, meets timing constraints and the binary is loaded in the FPGA.

2. The setup

I have the ZCU102 wired via a ETH cable to a windows machine. This machine run 'wireshark' for packet sniffing and packETH for small traffic generation. The vivado 2019.2 is used for JTAG interface.

3 The TX test. 

A packet (MAC frame) of 64 bytes is prepared.
To be sure it is received by the windows machine both MAC SRC and MAC DST is set to broadcast.
The GEM controller is configured :

puts "enable GEM TX + RX"
write_verbose 0xFF0E0000 0x0000000C
puts "set FIFO bus width"
write_verbose 0xFF0E0004 0x00080000
puts "enable FIFO interface"
write_verbose 0xFF0E004C 0x00000001

The ILA is triggered, and the DMA transferred.
I can see signaling, but then the fixed_latency comes up, i do not understand from the TRM what this means ?
When I monitor in wireshark, there is no packets received. 


3 The RX test. 

A packet (MAC frame) of 64 bytes is prepared.
To be sure it is received by the ZCU 102 both MAC SRC and MAC DST is set to broadcast.
The GEM controller is configured :

puts "enable GEM TX + RX"
write_verbose 0xFF0E0000 0x0000000C
puts "set FIFO bus width"
write_verbose 0xFF0E0004 0x00080000
puts "enable FIFO interface"
write_verbose 0xFF0E004C 0x00000001

The ILA is triggered, and a packet is send from the PC.
I can see from the wireshark that the full packet is transmitted. When i do a BRAM dump, I can only see 1/2 of the packet.
The packet is 

ff ff ff ff ff ff ff ff ff ff ff ff 00 2e aa aa
03 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77
77 77 77 77 77 77 77 77 77 77 77 77 77 77 77 77
77 77 77 77 77 77 77 77 77 77 77 77 D1 1F 76 8B

The BRAM dump is 
# read_verbose [format 0x%x [expr $addr_base_bram_00 + 0x00]]
RD : 0xa0000000 : 0xffffffff
# read_verbose [format 0x%x [expr $addr_base_bram_00 + 0x04]]
RD : 0xa0000004 : 0xffffffff
# read_verbose [format 0x%x [expr $addr_base_bram_00 + 0x08]]
RD : 0xa0000008 : 0xffffffff
# read_verbose [format 0x%x [expr $addr_base_bram_00 + 0x0C]]
RD : 0xa000000c : 0xaaaa0031
# read_verbose [format 0x%x [expr $addr_base_bram_01 + 0x00]]
RD : 0xa0002000 : 0xffffffff
# read_verbose [format 0x%x [expr $addr_base_bram_01 + 0x04]]
RD : 0xa0002004 : 0xaaa0ffff
# read_verbose [format 0x%x [expr $addr_base_bram_01 + 0x08]]
RD : 0xa0002008 : 0x77777773
# read_verbose [format 0x%x [expr $addr_base_bram_01 + 0x0C]]
RD : 0xa000200c : 0x77777777
# read_verbose [format 0x%x [expr $addr_base_bram_01 + 0x10]]
RD : 0xa0002010 : 0x77777777
# read_verbose [format 0x%x [expr $addr_base_bram_01 + 0x14]]
RD : 0xa0002014 : 0x77777777
# read_verbose [format 0x%x [expr $addr_base_bram_01 + 0x18]]
RD : 0xa0002018 : 0x77777777
# read_verbose [format 0x%x [expr $addr_base_bram_01 + 0x1C]]
RD : 0xa000201c : 0x973d7777

 

When you look at the ILA traces, it is also seen that the packet aborts. I have added the status information registers and these return : 0x160000204020 From the TRM this means that there is a FCS error and a Length error.

 

So i am not sure why the checksum and length check fails, but i guess it is related to the fact that the FIFO interface looses 1/2 of the bytes.
Then why is this ? I have toggled a few bits in the GEM3 configuration space, and am not able to get a full reception.

 

Questions :

Did I overlook something ?
Is there example code that uses the external FIFO interface on the GEM 3 on the ZCU 102 (no DMA) ?
Is it not allowed to test with broadcast SRC and DST address ?
Can some one explain better what the fixed latency signal means , and how to use it ?
Can some one share waveforms (ILA or simulated) from a working TX and RX transmission over GEM external FIFO ?
In my ILA, the captured data is only 1 bit, I guess it should be byte , but there is no config for that , is this a bug ?

 

All inputs and hints are welcome and appreciated.

I can see from the activity on the forums, that there a quite some users that have problems with getting the interface up and running, full duplex. I will update the ticket if i have more findings.

 

Thanks

 

Kris Provoost

TX_ILA.PNG
RX_ILA.PNG
0 Kudos
1 Solution

Accepted Solutions
473 Views
Registered: ‎01-13-2020

Hello

To confirm, the settings in the VIVADO project are correct, and not that difficult.

In the end, the issue was resolved in the configuration space.

On my setup, i had to configure the GEM and the PHY. Unfortunately the PHY boot defaults are no good for me.

For the GEM, there is an online memory map, which lists the registers, there are only a few to change.

I had a little JTAG AHB MASTER script for that :

puts "set FIFO bus width"
write_verbose [format 0x%x [expr $addr_base_cpu_gem3 + 0x004]] 0x010EA45A
# write_verbose 0xFF0E0004 0x048c0413

puts "enable FIFO interface"
write_verbose [format 0x%x [expr $addr_base_cpu_gem3 + 0x04C]] 0x00000001

puts "set clocks"
write_verbose 0xFF180308 0x00000000

source [cal_phy.tcl]

puts "enable GEM TX + RX"
write_verbose [format 0x%x [expr $addr_base_cpu_gem3 + 0x000]] 0x0000001C

After this configuration, if you have an empty cal_phy.tcl,  i would often see bit errors in my received data, therefore i had to calibrate the PHY.

If i had a scope on the TX/RX signals, i could really see poor signal quality.

In my case the setup works in a lab environment, so in stead of calibration i took a best guess for the following two registers, and then ever since it is working like expected:


WR extended register 0x0031

WR extended register 0x0032

the PHY datasheet is available on the web, but depending on your case , you may wan't to calibrate or not.

 

Hope this helps

View solution in original post

8 Replies
nanz
Moderator
Moderator
1,050 Views
Registered: ‎08-25-2009

Hi kris.provoost@qinetiq.be ,

So GEM3 is hard wired to on board TI PHY. Just to confirm your setup  that you are using RJ45 port to connect to the end PC to send/receive data? What has been connected to external FIFO interface to handle the ethernet data?

I am not sure if you have seen this AR:

https://www.xilinx.com/support/answers/69490.html

The FIFO interface should be 8-bit interface, please check this again in your design.

Unfortunately we do not have an example design to show it's working. Our Linux driver does not have the support for this either just FYI.


-------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our Versal Blogs and our Versal Ethernet Sticky Note.

-------------------------------------------------------------------------------------------
1,024 Views
Registered: ‎01-13-2020

Hello

I can further explain.

1. The link between the PC and the ZCU102 is CAT5 cable running from RJ45 to RJ45. There is no additional hardware such as routers or switches in the connection.
2. Yes, the port is using the TI PHY. I also considered if there could be something wrong with the PHY, but then I already can receive 1/2 of the bytes from PC to ZCU102. I guess this would not be possible if the PHY does not work.
3. Yes I have the ports from the interface that is created on the ZYNQ component. I saw that the signal is 8 bit. I can also see that the interface contains all TX/RX signals like described in the manuals. It is just that i do not understand how the manual explains the 'fixed_latency' signal. So I may not use this correct.
4. Both channels (TX and RX) interface with some custom logic. In the design , the logic runs on the provided TX and RX clocks, which are also provided by the ZYNQ component. The logic is written to comply with the waveforms from the manual, and from AR69490. Then the user side is connected to the Xilinx DMA controller. In order to be able to get/set some data in BRAM. Since the signals are sampled by the ILA on the ZYNQ port, I am sure that the connection from BRAM to TX and from RX to BRAM , over DMA is working. If not , then there would not be 1/2 the data in memory.
5. The design is loaded in the FPGA via the USB programmer. So the CPU should be running, but not booting any OS or SW. Just to say that I am in control of the registers, and dataflow (via JTAG console) And that there should not be an 'accidential' configuration from some other driver/layer from the CPU.
6. From the AR, i have made sure that the DMA toggle is responded with a DMA toggle. I can see from the timing report, that both clocks from the ZYNQ are added and recognized as 125 MHz frequency.

Questions:

Just to verify i understand correctly what is in the AR69490:

"In the GEM_CLK_CTRL (0xFF180308) register, * _RX_SRC_SEL and *_REF_SRC_SEL are expected to set to 1 if using GMII, and 0 if using RGMII."
>> I assume that the default (0x0000.0000) is OK. Because we use the RGMII ? Or can you confirm the value for BIT(18:15) ?

"*_FIFO_CLK_SEL is expected to set to 1 as it is recommended to connect _clk_to_pl_bufg back to _rx_clk_from_pl when using an external FIFO interface."
>> Is this a mistake ? I will add a figure on the clocks that are present, but these are both ouputs, or is this an extra configuration step that i have missed ?

 

thanks

 

Kris Provoost

 

EXT_FIFO_GEM3.PNG
0 Kudos
925 Views
Registered: ‎01-13-2020

Hello 

 

I believe that these configuration settings are mostly correct. As I have been able to send / receive data from the FIFO interface.

I did not succeed in full duplex traffic. That is : as soon as I have received data from the GEM block , and can still feed the GEM via the TX FIFO, and I can check via the ILA that these access are correctly. However , there is nothing on transmitting on the interface. As soon as the FPGA/PHY is power cycled and rebooted, I can transmit again, until i received some packets.

 

This seems to be another configuration issue. Anyone has seen this before ?

 

gr. 

0 Kudos
willchenyx
Observer
Observer
520 Views
Registered: ‎12-18-2018

Hi, kris.provoost@qinetiq.be

I also met this question, The pc  Ethernet card received no packets, alougth tx fifo works well from ILA just as your ILA picture . (I tried tx_r_control input set to 0 and 1)

Have you worked it out? And how did you do it?

0 Kudos
474 Views
Registered: ‎01-13-2020

Hello

To confirm, the settings in the VIVADO project are correct, and not that difficult.

In the end, the issue was resolved in the configuration space.

On my setup, i had to configure the GEM and the PHY. Unfortunately the PHY boot defaults are no good for me.

For the GEM, there is an online memory map, which lists the registers, there are only a few to change.

I had a little JTAG AHB MASTER script for that :

puts "set FIFO bus width"
write_verbose [format 0x%x [expr $addr_base_cpu_gem3 + 0x004]] 0x010EA45A
# write_verbose 0xFF0E0004 0x048c0413

puts "enable FIFO interface"
write_verbose [format 0x%x [expr $addr_base_cpu_gem3 + 0x04C]] 0x00000001

puts "set clocks"
write_verbose 0xFF180308 0x00000000

source [cal_phy.tcl]

puts "enable GEM TX + RX"
write_verbose [format 0x%x [expr $addr_base_cpu_gem3 + 0x000]] 0x0000001C

After this configuration, if you have an empty cal_phy.tcl,  i would often see bit errors in my received data, therefore i had to calibrate the PHY.

If i had a scope on the TX/RX signals, i could really see poor signal quality.

In my case the setup works in a lab environment, so in stead of calibration i took a best guess for the following two registers, and then ever since it is working like expected:


WR extended register 0x0031

WR extended register 0x0032

the PHY datasheet is available on the web, but depending on your case , you may wan't to calibrate or not.

 

Hope this helps

View solution in original post

willchenyx
Observer
Observer
438 Views
Registered: ‎12-18-2018

I have a strange issue, my board can TX packets through switch(my pc can receive thest packets, because thay are broadcast), but my pc cannot receive these packets when my board connect my pc 's eth card directly.

The led of my board's eth port is flickering in both cases, which means the packets has been TXed. Do you have some advice?

P.S. When the FIFO Interfaces to PL is on, the ps cannot operate GEM3, right?

Thank you!

0 Kudos
willchenyx
Observer
Observer
432 Views
Registered: ‎12-18-2018

I have solved this strange thing.

How can GEM3 be used by PL and PS? Do you have some advice?

Thanks!

0 Kudos
391 Views
Registered: ‎01-13-2020

You can not run PS and PL Ethernet link on the same GEM channel, simultaneous

 

0 Kudos