UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 

Debugging Tips when using GEM on Zynq MPSoC devices

Moderator
Moderator
4 1 392

Do you need to run Ethernet applications on a Zynq MPSoC device and consider using a Gigabit Ethernet MAC (GEM) core in the PS rather than using the PL logic?

If so, this blog entry will provide guidance and some debugging tips which might help you design with the GEM core. 

Basics

The GEM module implements a 10/100/1000 Mbps Ethernet MAC compatible with the IEEE 802.3 standard.

It can operate in either half or full duplex mode. The network configuration register is used to select the speed, duplex mode and interface type (MII, GMII, RGMII, TBI or SGMII).

GEM is normally used with its own hard-wired DMA block. In system applications where no DMA operation is required, an external FIFO interface can be used to incorporate GEM into an SoC environment.

The GEM block includes the following signal interfaces:

  • GMII, MII, and TBI interfaces to an external PHY
  • An MDIO interface for external PHY management
  • An AMBA Advanced Peripheral Bus (APB) slave interface for accessing the GEM registers
  • An AMBA Advanced High Speed Bus (AHB or AXI4) master interface for memory access
  • An optional FIFO interface in applications where DMA functionality is not required
  • An optional timestamp interface

The AMBA interfaces fully conform to the ARM AMBA Revision 2.0 specification.

 

forum_1.png

Figure 1: GEM Signal Interface

Below is a summary of what is supported in the stand-alone and Linux driver.

Table 1: ZynqMP GEM Baremetal and Linux Driver Support

 

EMACPS

LWIP

MACB

RGMII (MIO)

Yes

Yes

Yes

PS-GTR SGMII

Yes

No

Yes

PS-GTR 1000BASE-X

Yes

No

Yes

PL SGMII

Yes

Yes

Yes

PL 1000BASE-X

Yes

Yes

Yes

PL GMII2RGMII

N/A

No

Yes

PL MII2RMII

No

No

No

Example Designs

Xilinx has provided reference designs to run on the ZCU102 evaluation board.

XAPP1306 provides stand-alone/LWIP examples in SDK, while XAPP1305 provides Linux examples. Users can refer to the page below for DTS examples and Linux build steps:

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841830/PS+and+PL+based+Ethernet+in+Zynq+MPSoC

There is also a fixed link example on the following wiki page:

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842230/Zynq+Ultrascale+Fixed+Link+PS+Ethernet+Demo

Using the above reference designs as a start point is strongly recommended.

Debugging Tips

The following sections are separated based on the different Ethernet modes of the GEM configurations.

RGMII (MIO) and GMII (EMIO)

ZCU102 GEM3 is hard-wired with an on-board TI PHY. The SDK LWIP echo application can be used directly out of the box to test this interface. In Linux, the recommendation is to always use the provided BSP in your specific tool version to build the design in PetaLinux.

  • RGMII Voltage Support

Unlike on Zynq-7000 devices, voltages of 1.8V, 2.5V, and 3.3V are all supported.

In (DS925) table 44, we have listed switching characters under 2.5V test conditions.

forum_2.png

Figure 2: RGMII Interface

This table can also be referred by customers who want to use 3.3V, as it is faster and we have only listed the worst case scenario.

  • Device tree

For details on PHY bindings, please refer to:

https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/macb.txt.

The “compatible” string "cdns,zynqmp-gem" is for ZynqMP.

This compatible string enables the use of jumbo frame sizes, 1588, and hardware  timestamping support, as well as any features exclusive to ZynqMP.

For the “phy-mode” string, users can refer to the following page:

https://github.com/Xilinx/linuxxlnx/blob/master/Documentation/devicetree/bindings/net/ethernet.txt

Here is a DTS example:

                 gem0: ethernet@e000b000 { 

                                        compatible = "cdns,gem";

                                        reg = <0xe000b000 0x1000>;

                                        status = "disabled";

                                        interrupt-parent = <&gic>;

                                        interrupts = <0 22 4>;

                                        clocks = <&clkc 30>, <&clkc 30>, <&clkc 13>;

                                        clock-names = "pclk", "hclk", "tx_clk";

                                        #address-cells = <1>;

                                        #size-cells = <0>;

                                        phy-handle = <&ethernet_phy>;

                                        phy-mode = "rgmii-id";

                                         ethernet_phy: ethernet-phy@7{

                                                                          reg = <7>;

                                                                      };

                                             }

 

  • Shared MDIO

If the design requires use of the shared MDIO bus, a patch will then be needed as per (Xilinx Answer 69132). Shared MDIO support has been added from the 2019.1 release on. 

GEM PS-GTR SGMII

General FAQs on using PS-GTR with SGMII are answered in (Xilinx Answer 66592)

  • PS-GTR SGMII Debugging

If PS-GTR SGMII is not working, we usually start debugging by first checking the relevant registers dump and then running loopback.

Here is a list of registers to dump which will help to isolate the issue.

  • 0xFD401994• GT internal register
  • 0xFD405994• GT internal register
  • 0xFD409994• GT internal register
  • 0xFD40D994• GT internal register (Please note: these 4 registers need to be set to 0x7)
  • 0xFF0B0000 • network_control
  • 0xFF0B0004 • network_config
  • 0xFF0B0008 • network_status
  • 0xFF0B0200 • pcs_control
  • 0xFF0B0204 • pcs_status
  • 0xFF180360 • GEM_CTRL
  • 0xFF180308 • GEM_CLK_CTRL (IOU_SLCR)
  • 0xFF0B0198 • rx_symbol_error
  • 0xFF0B0158 • frames_rxed_ok
  • 0xFF0B0190 • fcs_errors
  • 0xFD410000 • PLL_REF_SEL0
  • 0xFD402860 • L0_L0_REF_CLK_SEL
  • 0xFD410010 • ICM_CFG0
  • 0xFD410040 • TX_PROT_BUS_WIDTH
  • 0xFD410044 • RX_PROT_BUS_WIDTH
  • 0xFD40106C • L0_TM_DIG_6
  • 0xFD4023E4 • L0_PLL_STATUS_READ_1

All registers info can be found in (UG1087).

https://www.xilinx.com/html_docs/registers/ug1087/ug1087-zynq-ultrascale-registers.html

After confirming that the registers are OK, the next step is to run PCS loopback and PS-GTR loopback to check if loopback works.

The PCS loopback can be turned on by setting the pcs_control register, pcs_control[loopback_mode] bit [14]. The PCS loopback should just run as long as the clock is present. It has no dependencies.

If the PCS loopback works, the issue might be related to Auto-Negotiation. When doing PCS loopback, it auto-negotiates to itself. The user can try to disable AN by setting pcs_control[enable_auto_neg] bit[12] to 0.

If the above PCS loopback works, the PS-GTR loopback is the next level loopback right before the SGMII PHY. This is run inside the PS-GTR transceivers.

The PS-GTR loopback registers are internal registers. Below are the details of the registers. Users need to disable PCS loopback first before turning PS-GTR loopback on.

forum_3.png

Figure 3: LPBK_CTRL0 Register (Internal)

forum_4.png

Figure 4: LPBK_CTRL1 Register (Internal)

For example, if the user wants to enable lane3 loopback, they need to write to 0xFD41003C to 0x00000010. 

If this does not work, it might be related to the GTR itself. Users can get support through the Xilinx Service Portal. A GT expert can guide them on how to measure the open eye of the transceiver and investigate on the GT side.

If this works, then there is not an issue with GEM and PS-GTR, so it could be more of a problem on the PHY side. Users will need to make sure that their PHYs/SFPs are working properly and to rule out any cable issues.

If using PS-GTR with GEM, the user also needs to check the polarity of the GT lanes. If TX and RX are reversed, they might not get RX packets on the receive side.

Here are the TX and RX registers to invert polarity.

forum_5.png

Figure 5: PS-GTR TX Polarity Control

forum_6.png

Figure 6: PS-GTR RX Polarity Control

  • Device Tree

A DTS example is provided in (Xilinx Answer 66592)

  • Fixed Link

If connecting SGMII to SGMII directly without PHY, that is a fixed link connection and the patch file in (Xilinx Answer 69769) is needed. 

GEM PS-GTR 1000BASE-X

When using PS-GTR in 1000BASE-SX/LX, there are no changes in the register settings or design in the MAC for 1000BaseX or SGMII when using the PS-GTR. The configuration remains the same. The external PHY will have to be configured for the required mode. In 1000BaseX mode, only a fixed speed of 1G can be used.

Miscellaneous Issues

Here is a list of various issues users might run into. 

U-boot Support

The below wiki page covers Uboot support with regards to Ethernet:

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842124/U-Boot+Ethernet+Driver

In Uboot, you can select the required Ethernet interface using the ethact command.

There is no limitation on using dual Ethernet in u-boot. For example, you can use “setenv ethact eth0” to set eth0 active.

Here is an example of the device tree when using PS-GTR SGMI mode with a Marvell PHY.

                              ethernet@ff0c0000 {

                                             phy-handle = <&phy0>;

                                              is-internal-pcspma;

                                              phy0: phy@1 {

                                                    compatible = "marvell, 88E1518";

                                                    device-type = "ethernet-phy";

                                                    reg = <1>;

                                                    };

                                              };

 

  • Fixed-Link

Here is an example of testing and using fixed-link in Uboot:

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841998/Testing+Fixed+Link+support+in+U-Boot

In versions prior to 2017.2, a patch is required to use fixed-link in Uboot. See (Xilinx Answer 69768)

GEM External FIFO interface

In applications where the DMA operation is not required, an exposed FIFO interface is available for both the transmit and receive data paths.

This interface is enabled by setting external_fifo_interface Register bit [0] to 1.

General FAQs on using an external FIFO interface are answered in (Xilinx Answer 69490):

In the PCW configuration GUI, you can enable the GEM External FIFO Interface by selecting the option shown the example below:

forum_7.png

 

Figure 7: Enable External FIFO Interface in PCW

forum_8.png

Figure 8: FIFO Interface Available

There is a detailed section in (UG1085) on how this interface works.

Please note: currently there is no driver support for using an external FIFO Interface.

GEM TSU and IEEE 1588 Support

Please refer to the “GEM TSU Interface and IEEE 1588 Support” document attached to (Xilinx Answer 67239)

GEM Performance Limitation

As there is no flow control hardware logic in GEM to detect TX/RX FIFO overflow, when you have high bi-directional traffic, there is a higher chance of the FIFO getting overflown and you might see packets drop.

More details can be found in (Xilinx Answer 71168).

Here are the steps to be followed to test the performance:

Pre-requisite: Make sure to have iperf3 binaries in both the Linux host and in the PetaLinux image.

Also, make sure that basic ping functionality is working on both the host and the target.

Run the below bi-directional iperf3 commands across the Zynq MP board and Linux PC to measure the performance.

Board side:

iperf3 -A1 –s

iperf3 -c <host_ip> -t 999

PC side:

iperf3 –s &

iperf3 -c <board_ip> -t 999

Additional Information

(Xilinx Answer 71349) is the Zynq Ultrascale+ GEM known issues and master Answer Record.

The Xilinx wiki contains performance numbers:

XAPP1306:

XAPP1305:

GEM on Zynq-7000

Here is a table of what is supported in the standalone and Linux driver for GEM on Zynq-7000 devices.

Table 2: Zynq GEM Baremetal and Linux Driver Support 

 

EMACPS

LWIP

MACB

RGMII (MIO)

Yes

Yes

Yes

PL SGMII

Yes

Yes

Yes

PL 1000BASE-X

Yes

Yes

Yes

PL GMII2RGMII

Yes

Yes

Yes

PL MII2RMII

No

No

No

 

Most questions regarding GEM on Zynq are related to the Errata, which are documented as known issues in (UG585) and (Xilinx Answer 52028).

There is no PTP support in GEM on Zynq due to the hardware limitations. Zynq GEM has TSU which can detect incoming and outgoing PTP event frames and time-stamp them with its internal PTP timer. The timer registers and timestamps are accessed through registers, and there is no path to the PL.

There is only one FIFO queue for timestamps and its depth is 1. Matching each frame with its time-stamp has to be done quickly so as to not lose any value. There is no PPS signal.

Also the two GEMs have independent PTP timers and there is no way to make them use the same reference timer. Users can search for a third party solution or design their own timer to improve the accuracy, but there is no supported solution from Xilinx. 

(Xilinx Answer 71352) is the known issue and release notes Answer Record for GEM in Zynq-7000 devices.

1 Comment
Participant joe4702
Participant

Thanks, overall this is very helpful.