cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Participant
Participant
746 Views
Registered: ‎08-10-2018

Petalinux GEM Tuning UltraZed-EG

Jump to solution

The board being used an UltraZed-EG with the IO Carrier Card.  We are setting up as a server share for 2 SATA drives. Currently our bottle neck is the ethernet speed. 

Our desire is to use the Vivado 2019.1 and Petalinux 2019.1.

When we follow the tutorial from http://www.ultrazed.org/support/design/17596/131 for the 2018.2 BSP we achieve a UDP throughput of 500-600 Mbps. To contrast that with the 2017.4 BSP for this kit we were able to achieve 957 Mbps.

We are using the GEM3 with the following dtsi definition:

&gem3 {
	status = "okay";
	local-mac-address = [00 0a 35 00 02 90];
	phy-mode = "rgmii-id";
	phy-handle = <&phy0>;
	phy0: phy@9 {
		reg = <0x9>;
		ti,rx-internal-delay = <0x5>;
		ti,tx-internal-delay = <0x5>;
		ti,fifo-depth = <0x1>;
	};
};

 

In the 2019.1 Vivado design we increased the PSU__PSS_REF_CLK_FREQMHZ from 33.333 to 33.33333333333333 and we were able to acheive better UDP throughput of 840Mbps.  

From these observation we feel there is some clock tuning that needs to be done to get the best performance out of the Ethernet.

We also played with the RX/TX internal delay to the GEM and they seemed best left at 0x5.

Are there any other areas we can look at to tune or setup so that we can acheive the 957 Mbps we observed with the 2017.4 BSP?

 

0 Kudos
1 Solution

Accepted Solutions
Highlighted
Participant
Participant
204 Views
Registered: ‎08-10-2018

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution

Currently we getting about 830 MBits/s with UDP and about 939 MBits/s with TCP.

The priority of this has gone down we may look at it after we have our prototype hardware finished, but for now the speeds seem adequate.

Currently the only thing we did that had any affect was adusting the PSU__PSS_REF_CLK_FREQMHZ from 33.333 to 33.33333333333333.

View solution in original post

0 Kudos
10 Replies
Highlighted
Moderator
Moderator
666 Views
Registered: ‎04-24-2017

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution

Hi @stew.hansen ,

We are investigate this issue by upgrading the design to 2019.1 we will keep you posted asap.

Thanks,
Sandeep
PetaLinux Yocto | Embedded SW Support

---------------------------------------------------------------------------
Don’t forget to Reply, Kudo, and Accept as Solution.
---------------------------------------------------------------------------
0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
635 Views
Registered: ‎08-15-2018

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution

Hi @stew.hansen,

I've done some investigation into this issue - it looks like in release 3.1.5 of iperf3, they changed the default UDP send size from 8KB to something smaller.

From the 3.1.5 release notes:

* iperf3 now chooses a more sane default UDP send size (#496, #498).
    The former default (8KB) caused IP packet fragmentation on paths
    having smaller MTUs (including any Ethernet network not configured
    for jumbo frames).  This could have effects ranging from increased
    burstiness, to packet loss, to complete failure of the test.
    iperf3 now attempts to use the MSS of the control connection to
    determine a default UDP send size if no sending length was
    explicitly specified with --length.

 

2018.2 uses iperf3 v3.2:

iperf2018_2.PNG

 

2017.4 uses iperf3 v3.1:

iperf2017_4.PNG

 

The send size can be overridden with the --length flag, and setting it back to the default which was present in pre-3.1.5 versions (8KB) I see performance return to normal in 2018.2:

iperf2018_2Performance.PNG

 

Thanks,

Clayton

0 Kudos
Highlighted
Participant
Participant
582 Views
Registered: ‎08-10-2018

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution

Hi @claytonr,

I have looked at your post and adjusted our testing procdures. The initial tests with the iperf3 version 3.6 with the --length were lower than expected. I did a reboot of the petalinux system to refresh the interface and connected directly to the host ethernet.

I did the same tests with no --length and then did incremental tests that started at 1024 and went up to 8192 and ended with 8000. The results can be seen below.

Initial results:

 

uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 49945 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  98.1 MBytes   823 Mbits/sec  71030  
[  5]   1.00-2.00   sec  97.8 MBytes   821 Mbits/sec  70847  
[  5]   2.00-3.00   sec  97.9 MBytes   821 Mbits/sec  70910  
[  5]   3.00-4.00   sec  98.0 MBytes   822 Mbits/sec  70979  
[  5]   4.00-5.00   sec  98.0 MBytes   822 Mbits/sec  70961  
[  5]   5.00-6.00   sec  99.4 MBytes   833 Mbits/sec  71947  
[  5]   6.00-7.00   sec  99.6 MBytes   835 Mbits/sec  72117  
[  5]   7.00-8.00   sec  99.9 MBytes   838 Mbits/sec  72367  
[  5]   8.00-9.00   sec  99.7 MBytes   837 Mbits/sec  72226  
[  5]   9.00-10.00  sec  99.8 MBytes   837 Mbits/sec  72289  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   988 MBytes   829 Mbits/sec  0.000 ms  0/715673 (0%)  sender
[  5]   0.00-10.00  sec   985 MBytes   826 Mbits/sec  0.018 ms  2238/715673 (0.31%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 1024
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 33350 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  69.8 MBytes   586 Mbits/sec  71509
[  5]   1.00-2.00   sec  69.9 MBytes   586 Mbits/sec  71532
[  5]   2.00-3.00   sec  69.7 MBytes   585 Mbits/sec  71412
[  5]   3.00-4.00   sec  69.7 MBytes   585 Mbits/sec  71359
[  5]   4.00-5.00   sec  69.9 MBytes   586 Mbits/sec  71551
[  5]   5.00-6.00   sec  71.3 MBytes   598 Mbits/sec  73053
[  5]   6.00-7.00   sec  71.3 MBytes   598 Mbits/sec  72995
[  5]   7.00-8.00   sec  71.3 MBytes   598 Mbits/sec  73057
[  5]   8.00-9.00   sec  71.4 MBytes   599 Mbits/sec  73150
[  5]   9.00-10.00  sec  71.4 MBytes   599 Mbits/sec  73112
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   706 MBytes   592 Mbits/sec  0.000 ms  0/722730 (0%)  sender
[  5]   0.00-10.00  sec   704 MBytes   590 Mbits/sec  0.018 ms  2069/722727 (0.29%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 2048
warning: UDP block size 2048 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 49263 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  50.9 MBytes   427 Mbits/sec  26050
[  5]   1.00-2.00   sec  50.9 MBytes   427 Mbits/sec  26050
[  5]   2.00-3.00   sec  50.8 MBytes   426 Mbits/sec  26025
[  5]   3.00-4.00   sec  50.9 MBytes   427 Mbits/sec  26037
[  5]   4.00-5.00   sec  50.9 MBytes   427 Mbits/sec  26040
[  5]   5.00-6.00   sec  50.9 MBytes   427 Mbits/sec  26047
[  5]   6.00-7.00   sec  50.9 MBytes   427 Mbits/sec  26046
[  5]   7.00-8.00   sec  50.9 MBytes   427 Mbits/sec  26045
[  5]   8.00-9.00   sec  50.9 MBytes   427 Mbits/sec  26066
[  5]   9.00-10.00  sec  50.9 MBytes   427 Mbits/sec  26037
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   509 MBytes   427 Mbits/sec  0.000 ms  0/260443 (0%)  sender
[  5]   0.00-10.00  sec   507 MBytes   425 Mbits/sec  0.041 ms  825/260443 (0.32%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 3072
warning: UDP block size 3072 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 40597 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  58.6 MBytes   491 Mbits/sec  19996  
[  5]   1.00-2.00   sec  58.6 MBytes   492 Mbits/sec  20006  
[  5]   2.00-3.00   sec  58.6 MBytes   492 Mbits/sec  20005  
[  5]   3.00-4.00   sec  58.6 MBytes   492 Mbits/sec  20009  
[  5]   4.00-5.00   sec  58.6 MBytes   492 Mbits/sec  20006  
[  5]   5.00-6.00   sec  58.6 MBytes   492 Mbits/sec  20011  
[  5]   6.00-7.00   sec  58.6 MBytes   492 Mbits/sec  20002  
[  5]   7.00-8.00   sec  58.6 MBytes   492 Mbits/sec  20001  
[  5]   8.00-9.00   sec  58.5 MBytes   491 Mbits/sec  19983  
[  5]   9.00-10.00  sec  58.6 MBytes   491 Mbits/sec  19991  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   586 MBytes   492 Mbits/sec  0.000 ms  0/200010 (0%)  sender
[  5]   0.00-10.00  sec   585 MBytes   491 Mbits/sec  0.046 ms  374/200010 (0.19%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 4096
warning: UDP block size 4096 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 34146 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  70.5 MBytes   591 Mbits/sec  18038  
[  5]   1.00-2.00   sec  70.6 MBytes   592 Mbits/sec  18065  
[  5]   2.00-3.00   sec  70.6 MBytes   592 Mbits/sec  18066  
[  5]   3.00-4.00   sec  70.5 MBytes   592 Mbits/sec  18059  
[  5]   4.00-5.00   sec  70.6 MBytes   592 Mbits/sec  18061  
[  5]   5.00-6.00   sec  70.5 MBytes   592 Mbits/sec  18052  
[  5]   6.00-7.00   sec  70.5 MBytes   592 Mbits/sec  18058  
[  5]   7.00-8.00   sec  70.5 MBytes   592 Mbits/sec  18055  
[  5]   8.00-9.00   sec  70.5 MBytes   592 Mbits/sec  18058  
[  5]   9.00-10.00  sec  70.5 MBytes   592 Mbits/sec  18056  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   705 MBytes   592 Mbits/sec  0.000 ms  0/180568 (0%)  sender
[  5]   0.00-10.00  sec   704 MBytes   591 Mbits/sec  0.046 ms  347/180567 (0.19%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 5120
warning: UDP block size 5120 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 53980 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  96.1 MBytes   806 Mbits/sec  19690  
[  5]   1.00-2.00   sec  96.3 MBytes   808 Mbits/sec  19731  
[  5]   2.00-3.00   sec  96.3 MBytes   808 Mbits/sec  19720  
[  5]   3.00-4.00   sec  96.2 MBytes   807 Mbits/sec  19709  
[  5]   4.00-5.00   sec  96.3 MBytes   808 Mbits/sec  19723  
[  5]   5.00-6.00   sec  96.3 MBytes   808 Mbits/sec  19717  
[  5]   6.00-7.00   sec  96.4 MBytes   809 Mbits/sec  19739  
[  5]   7.00-8.00   sec  96.4 MBytes   809 Mbits/sec  19749  
[  5]   8.00-9.00   sec  96.5 MBytes   809 Mbits/sec  19758  
[  5]   9.00-10.00  sec  96.4 MBytes   809 Mbits/sec  19740  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   963 MBytes   808 Mbits/sec  0.000 ms  0/197276 (0%)  sender
[  5]   0.00-10.00  sec   961 MBytes   806 Mbits/sec  0.041 ms  546/197275 (0.28%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 6144
warning: UDP block size 6144 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 36578 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  99.7 MBytes   836 Mbits/sec  17018  
[  5]   1.00-2.00   sec  99.8 MBytes   837 Mbits/sec  17033  
[  5]   2.00-3.00   sec  99.8 MBytes   837 Mbits/sec  17034  
[  5]   3.00-4.00   sec  99.8 MBytes   838 Mbits/sec  17039  
[  5]   4.00-5.00   sec  99.8 MBytes   837 Mbits/sec  17036  
[  5]   5.00-6.00   sec  99.8 MBytes   837 Mbits/sec  17035  
[  5]   6.00-7.00   sec  99.8 MBytes   837 Mbits/sec  17034  
[  5]   7.00-8.00   sec  99.8 MBytes   837 Mbits/sec  17028  
[  5]   8.00-9.00   sec  99.8 MBytes   837 Mbits/sec  17030  
[  5]   9.00-10.00  sec  99.8 MBytes   837 Mbits/sec  17032  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   998 MBytes   837 Mbits/sec  0.000 ms  0/170319 (0%)  sender
[  5]   0.00-10.00  sec   995 MBytes   834 Mbits/sec  0.034 ms  585/170317 (0.34%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 7168
warning: UDP block size 7168 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 60226 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  77.3 MBytes   648 Mbits/sec  11305  
[  5]   1.00-2.00   sec  77.3 MBytes   649 Mbits/sec  11310  
[  5]   2.00-3.00   sec  77.3 MBytes   649 Mbits/sec  11311  
[  5]   3.00-4.00   sec  77.3 MBytes   649 Mbits/sec  11310  
[  5]   4.00-5.00   sec  77.4 MBytes   649 Mbits/sec  11316  
[  5]   5.00-6.00   sec  77.5 MBytes   650 Mbits/sec  11340  
[  5]   6.00-7.00   sec  77.5 MBytes   650 Mbits/sec  11337  
[  5]   7.00-8.00   sec  77.5 MBytes   650 Mbits/sec  11339  
[  5]   8.00-9.00   sec  77.5 MBytes   650 Mbits/sec  11338  
[  5]   9.00-10.00  sec  77.5 MBytes   650 Mbits/sec  11335  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   774 MBytes   649 Mbits/sec  0.000 ms  0/113241 (0%)  sender
[  5]   0.00-10.00  sec   772 MBytes   647 Mbits/sec  0.018 ms  358/113240 (0.32%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 8192
warning: UDP block size 8192 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 43322 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  79.1 MBytes   663 Mbits/sec  10124  
[  5]   1.00-2.00   sec  79.2 MBytes   664 Mbits/sec  10132  
[  5]   2.00-3.00   sec  79.2 MBytes   665 Mbits/sec  10143  
[  5]   3.00-4.00   sec  79.3 MBytes   665 Mbits/sec  10146  
[  5]   4.00-5.00   sec  79.2 MBytes   664 Mbits/sec  10133  
[  5]   5.00-6.00   sec  79.1 MBytes   664 Mbits/sec  10131  
[  5]   6.00-7.00   sec  79.2 MBytes   664 Mbits/sec  10139  
[  5]   7.00-8.00   sec  79.1 MBytes   664 Mbits/sec  10129  
[  5]   8.00-9.00   sec  79.2 MBytes   664 Mbits/sec  10132  
[  5]   9.00-10.00  sec  79.1 MBytes   664 Mbits/sec  10129  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   792 MBytes   664 Mbits/sec  0.000 ms  0/101338 (0%)  sender
[  5]   0.00-10.00  sec   790 MBytes   663 Mbits/sec  0.004 ms  202/101338 (0.2%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 8000
warning: UDP block size 8000 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 35014 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  76.1 MBytes   638 Mbits/sec  9970  
[  5]   1.00-2.00   sec  76.1 MBytes   639 Mbits/sec  9977  
[  5]   2.00-3.00   sec  76.2 MBytes   639 Mbits/sec  9982  
[  5]   3.00-4.00   sec  76.1 MBytes   639 Mbits/sec  9977  
[  5]   4.00-5.00   sec  76.1 MBytes   639 Mbits/sec  9977  
[  5]   5.00-6.00   sec  76.1 MBytes   638 Mbits/sec  9974  
[  5]   6.00-7.00   sec  76.1 MBytes   638 Mbits/sec  9976  
[  5]   7.00-8.00   sec  76.1 MBytes   639 Mbits/sec  9979  
[  5]   8.00-9.00   sec  76.2 MBytes   639 Mbits/sec  9982  
[  5]   9.00-10.00  sec  76.0 MBytes   638 Mbits/sec  9968  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   761 MBytes   638 Mbits/sec  0.000 ms  0/99762 (0%)  sender
[  5]   0.00-10.00  sec   759 MBytes   636 Mbits/sec  0.002 ms  327/99762 (0.33%)  receiver

iperf Done.

I was surprised by the low data rate at length greaters than 6000 leading to lower data rate. I figured something was still not right. Without changing any thing other than rebooting I get the following results.

After reboot:

uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 55968 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  98.0 MBytes   822 Mbits/sec  70941  
[  5]   1.00-2.00   sec  97.7 MBytes   820 Mbits/sec  70759  
[  5]   2.00-3.00   sec  97.8 MBytes   821 Mbits/sec  70840  
[  5]   3.00-4.00   sec  97.8 MBytes   821 Mbits/sec  70839  
[  5]   4.00-5.00   sec  97.9 MBytes   821 Mbits/sec  70874  
[  5]   5.00-6.00   sec  99.6 MBytes   835 Mbits/sec  72115  
[  5]   6.00-7.00   sec  99.7 MBytes   837 Mbits/sec  72227  
[  5]   7.00-8.00   sec  99.6 MBytes   836 Mbits/sec  72148  
[  5]   8.00-9.00   sec  99.6 MBytes   835 Mbits/sec  72115  
[  5]   9.00-10.00  sec  99.5 MBytes   835 Mbits/sec  72059  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   987 MBytes   828 Mbits/sec  0.000 ms  0/714917 (0%)  sender
[  5]   0.00-10.00  sec   985 MBytes   826 Mbits/sec  0.019 ms  1705/714917 (0.24%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 1024
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 51296 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  69.1 MBytes   580 Mbits/sec  35379  
[  5]   1.00-2.00   sec  69.2 MBytes   580 Mbits/sec  35422  
[  5]   2.00-3.00   sec  69.2 MBytes   580 Mbits/sec  35410  
[  5]   3.00-4.00   sec  69.3 MBytes   581 Mbits/sec  35485  
[  5]   4.00-5.00   sec  69.2 MBytes   581 Mbits/sec  35456  
[  5]   5.00-6.00   sec  69.2 MBytes   581 Mbits/sec  35441  
[  5]   6.00-7.00   sec  69.3 MBytes   581 Mbits/sec  35460  
[  5]   7.00-8.00   sec  69.2 MBytes   580 Mbits/sec  35416  
[  5]   8.00-9.00   sec  69.3 MBytes   581 Mbits/sec  35465  
[  5]   9.00-10.00  sec  69.3 MBytes   581 Mbits/sec  35465  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   692 MBytes   581 Mbits/sec  0.000 ms  0/354399 (0%)  sender
[  5]   0.00-10.00  sec   690 MBytes   579 Mbits/sec  0.033 ms  1140/354398 (0.32%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 2048
warning: UDP block size 3072 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 37565 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  79.7 MBytes   669 Mbits/sec  27205  
[  5]   1.00-2.00   sec  79.8 MBytes   669 Mbits/sec  27230  
[  5]   2.00-3.00   sec  79.7 MBytes   669 Mbits/sec  27210  
[  5]   3.00-4.00   sec  79.8 MBytes   669 Mbits/sec  27236  
[  5]   4.00-5.00   sec  79.8 MBytes   669 Mbits/sec  27236  
[  5]   5.00-6.00   sec  79.9 MBytes   670 Mbits/sec  27262  
[  5]   6.00-7.00   sec  79.8 MBytes   670 Mbits/sec  27253  
[  5]   7.00-8.00   sec  79.8 MBytes   669 Mbits/sec  27228  
[  5]   8.00-9.00   sec  79.7 MBytes   669 Mbits/sec  27221  
[  5]   9.00-10.00  sec  79.8 MBytes   670 Mbits/sec  27254  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   798 MBytes   669 Mbits/sec  0.000 ms  0/272335 (0%)  sender
[  5]   0.00-10.00  sec   796 MBytes   667 Mbits/sec  0.038 ms  790/272334 (0.29%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 3072
warning: UDP block size 4096 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 43897 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  92.2 MBytes   773 Mbits/sec  23600  
[  5]   1.00-2.00   sec  92.3 MBytes   774 Mbits/sec  23624  
[  5]   2.00-3.00   sec  92.3 MBytes   774 Mbits/sec  23628  
[  5]   3.00-4.00   sec  92.3 MBytes   774 Mbits/sec  23626  
[  5]   4.00-5.00   sec  92.3 MBytes   774 Mbits/sec  23625  
[  5]   5.00-6.00   sec  92.3 MBytes   774 Mbits/sec  23625  
[  5]   6.00-7.00   sec  92.3 MBytes   774 Mbits/sec  23624  
[  5]   7.00-8.00   sec  92.3 MBytes   774 Mbits/sec  23625  
[  5]   8.00-9.00   sec  92.3 MBytes   774 Mbits/sec  23624  
[  5]   9.00-10.00  sec  92.3 MBytes   774 Mbits/sec  23626  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   923 MBytes   774 Mbits/sec  0.000 ms  0/236227 (0%)  sender
[  5]   0.00-10.00  sec   921 MBytes   773 Mbits/sec  0.039 ms  409/236227 (0.17%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 4096
warning: UDP block size 5120 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 52896 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  93.0 MBytes   780 Mbits/sec  19051  
[  5]   1.00-2.00   sec  93.2 MBytes   781 Mbits/sec  19078  
[  5]   2.00-3.00   sec  93.1 MBytes   781 Mbits/sec  19059  
[  5]   3.00-4.00   sec  93.3 MBytes   782 Mbits/sec  19101  
[  5]   4.00-5.00   sec  93.2 MBytes   782 Mbits/sec  19083  
[  5]   5.00-6.00   sec  93.2 MBytes   782 Mbits/sec  19087  
[  5]   6.00-7.00   sec  93.2 MBytes   782 Mbits/sec  19081  
[  5]   7.00-8.00   sec  93.2 MBytes   782 Mbits/sec  19085  
[  5]   8.00-9.00   sec  93.2 MBytes   782 Mbits/sec  19083  
[  5]   9.00-10.00  sec  93.2 MBytes   782 Mbits/sec  19086  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   932 MBytes   781 Mbits/sec  0.000 ms  0/190794 (0%)  sender
[  5]   0.00-10.00  sec   929 MBytes   779 Mbits/sec  0.043 ms  514/190793 (0.27%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 5120
warning: UDP block size 6144 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 53574 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  99.8 MBytes   837 Mbits/sec  17031  
[  5]   1.00-2.00   sec  99.9 MBytes   838 Mbits/sec  17049  
[  5]   2.00-3.00   sec  99.9 MBytes   838 Mbits/sec  17050  
[  5]   3.00-4.00   sec  99.9 MBytes   838 Mbits/sec  17050  
[  5]   4.00-5.00   sec  99.9 MBytes   838 Mbits/sec  17046  
[  5]   5.00-6.00   sec  99.9 MBytes   838 Mbits/sec  17043  
[  5]   6.00-7.00   sec  99.9 MBytes   838 Mbits/sec  17046  
[  5]   7.00-8.00   sec  99.9 MBytes   838 Mbits/sec  17046  
[  5]   8.00-9.00   sec  99.9 MBytes   838 Mbits/sec  17047  
[  5]   9.00-10.00  sec  99.9 MBytes   838 Mbits/sec  17048  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   999 MBytes   838 Mbits/sec  0.000 ms  0/170456 (0%)  sender
[  5]   0.00-10.00  sec   997 MBytes   836 Mbits/sec  0.034 ms  318/170455 (0.19%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 6144
warning: UDP block size 7168 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 39825 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   108 MBytes   909 Mbits/sec  15860  
[  5]   1.00-2.00   sec   108 MBytes   910 Mbits/sec  15871  
[  5]   2.00-3.00   sec   109 MBytes   910 Mbits/sec  15873  
[  5]   3.00-4.00   sec   109 MBytes   910 Mbits/sec  15877  
[  5]   4.00-5.00   sec   108 MBytes   909 Mbits/sec  15856  
[  5]   5.00-6.00   sec   108 MBytes   909 Mbits/sec  15854  
[  5]   6.00-7.00   sec   108 MBytes   909 Mbits/sec  15855  
[  5]   7.00-8.00   sec   108 MBytes   910 Mbits/sec  15862  
[  5]   8.00-9.00   sec   108 MBytes   909 Mbits/sec  15859  
[  5]   9.00-10.00  sec   108 MBytes   909 Mbits/sec  15857  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.06 GBytes   910 Mbits/sec  0.000 ms  0/158624 (0%)  sender
[  5]   0.00-10.00  sec  1.05 GBytes   906 Mbits/sec  0.037 ms  692/158623 (0.44%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 7168
warning: UDP block size 8192 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 54007 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   109 MBytes   915 Mbits/sec  13967  
[  5]   1.00-2.00   sec   109 MBytes   918 Mbits/sec  14007  
[  5]   2.00-3.00   sec   109 MBytes   918 Mbits/sec  14010  
[  5]   3.00-4.00   sec   109 MBytes   917 Mbits/sec  13992  
[  5]   4.00-5.00   sec   109 MBytes   917 Mbits/sec  13986  
[  5]   5.00-6.00   sec   109 MBytes   917 Mbits/sec  13986  
[  5]   6.00-7.00   sec   109 MBytes   917 Mbits/sec  13998  
[  5]   7.00-8.00   sec   109 MBytes   917 Mbits/sec  13997  
[  5]   8.00-9.00   sec   109 MBytes   917 Mbits/sec  13996  
[  5]   9.00-10.00  sec   109 MBytes   918 Mbits/sec  14014  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.07 GBytes   917 Mbits/sec  0.000 ms  0/139953 (0%)  sender
[  5]   0.00-10.00  sec  1.06 GBytes   915 Mbits/sec  0.025 ms  372/139952 (0.27%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 8192
warning: UDP block size 8000 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 56707 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   107 MBytes   901 Mbits/sec  14071  
[  5]   1.00-2.00   sec   108 MBytes   902 Mbits/sec  14095  
[  5]   2.00-3.00   sec   107 MBytes   902 Mbits/sec  14087  
[  5]   3.00-4.00   sec   108 MBytes   902 Mbits/sec  14091  
[  5]   4.00-5.00   sec   108 MBytes   902 Mbits/sec  14092  
[  5]   5.00-6.00   sec   108 MBytes   906 Mbits/sec  14158  
[  5]   6.00-7.00   sec   108 MBytes   907 Mbits/sec  14170  
[  5]   7.00-8.00   sec   108 MBytes   907 Mbits/sec  14171  
[  5]   8.00-9.00   sec   108 MBytes   907 Mbits/sec  14172  
[  5]   9.00-10.00  sec   108 MBytes   907 Mbits/sec  14177  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.05 GBytes   904 Mbits/sec  0.000 ms  0/141284 (0%)  sender
[  5]   0.00-10.00  sec  1.05 GBytes   901 Mbits/sec  0.026 ms  436/141283 (0.31%)  receiver

iperf Done.
uz_petalinux:~# iperf3 --bandwidth 1000M --client 192.168.1.100 --format m -u --length 8000
warning: UDP block size 8182 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.10 port 52691 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   109 MBytes   918 Mbits/sec  14019  
[  5]   1.00-2.00   sec   110 MBytes   919 Mbits/sec  14038  
[  5]   2.00-3.00   sec   110 MBytes   919 Mbits/sec  14042  
[  5]   3.00-4.00   sec   110 MBytes   919 Mbits/sec  14045  
[  5]   4.00-5.00   sec   110 MBytes   919 Mbits/sec  14036  
[  5]   5.00-6.00   sec   111 MBytes   928 Mbits/sec  14174  
[  5]   6.00-7.00   sec   111 MBytes   928 Mbits/sec  14176  
[  5]   7.00-8.00   sec   111 MBytes   928 Mbits/sec  14182  
[  5]   8.00-9.00   sec   111 MBytes   927 Mbits/sec  14169  
[  5]   9.00-10.00  sec   111 MBytes   928 Mbits/sec  14177  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.07 GBytes   923 Mbits/sec  0.000 ms  0/141058 (0%)  sender
[  5]   0.00-10.00  sec  1.07 GBytes   921 Mbits/sec  0.021 ms  398/141057 (0.28%)  receiver

iperf Done.

I did some longer tests and after about 90 seconds the data rate dropped from 920Mbits/sec to 624Mbits/sec with the --length set at 8192.

I surmize that there may have been a iperf3 tool version issue, and a clock precision issue. My confidence isn't quite settled as the Ethernet speed does throttle down for some reason that I have not been able to find. If we could get repeatable tests with the >900Mbits/sec then our confidence would increase.

Also the Host iperf3 version was 3.09. I downloaded and updated to the same version iperf3 version 3.6 for the host side, and the results were pretty much the same as the second batch of test results above.

 

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
576 Views
Registered: ‎08-15-2018

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution

Hey @stew.hansen,

What does your network topology between the host machine and the board look like? Could there be heavy traffic, or some sort of QoS enforcement happening?

 

I suspect something external to the device, because I just ran the same test as you, but for 10 minutes:

iperf3 --bandwidth 1000M --client 123.234.1.1 --format m -u --length 8000 -t 600

 

Here are my results:

10minUDPtest.PNG

The 957Mbps was very consistent throughout the test, I didn't see any throttling. I'm on a really small testbed network that just consists of my host PC, an 8 port gigabit switch, and a few boards so traffic is very low.

 

I'm using the 2018.2 BSP for the ultrazed w/ IO Carrier, and the only modification I've made is to change the PHY address in the device tree as mine was strapped to 0x8 instead of 0x9.

 

One other question -  do you have other things running in the background on the ARM while you're doing the test?

 

Thanks,

Clayton

0 Kudos
Highlighted
Participant
Participant
549 Views
Registered: ‎08-10-2018

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution

Hey @claytonr,

First initial test was with an 8 port gigabit switch. The only devices connected are the Ultrazed-EG IO board and the host machine.

There isn't anything running in the background on the Arm.

I tested the 2018.2 bsp from http://www.ultrazed.org/sites/default/files/design/uz3eg_iocc_sd_oob_2018_2.zip.

With the prebuilt images and the command:

 

iperf3 -c 192.168.1.100 -b 1000M --format -m -u -t 600

The result were:Screenshot from 2019-10-08 15-08-06.png

 

 

 

 

 

Throughput started around ~950Mbits/sec and then they dropped to ~640 then back up as can be seen above.

I am doing some more tests.

0 Kudos
Highlighted
Participant
Participant
511 Views
Registered: ‎08-10-2018

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution

Hey @claytonr ,

I have captured 5 tests and their results. I had another reply but felt it would be easier to read the results from the table.

The first two tests are from stock BSPs provided by Avnet for 2017.4 and 2018.2.

The next three tests are 2019.1 with varying approaches to getting a similar project for the UZ3EG_IOCC board from avnet.

The tests were performed with the same setup with an ubuntu 18.04 host running Iperf3 version 3.6. The Ethernet goes through a NETGEAR GS108 x8 Gigabit switch. With only the Host and the UltraZed IO kit connected to the switch. There were three flavors of benchmarks performed for each PetaLinux version, one without the -l option and then two with the -l option for the lengths of 8000 and 8192.

2017.4 version command: 

iperf3 -c 192.168.1.100 -b 1000M --format -m -u -t 600

2018.2 and 2019.1 version commands:

iperf3 -c 192.168.1.100 -b 1000M -u -t 600

Each test was 10 minutes in length. Some tests started out with the 958 Mbits/sec then dropped down to 640 Mbits/sec especially in the 2019.1 tests.

PetaLinux Version-l commandiperf versionBitrate (Mbits/sec)Bitrate (Mbits/sec)DatagramsDatagramsNotes
  sendersenderreceiversenderreceiver 
2017.4 3.1958combined183087/8772393 (2.1%)combinedUsing the uz3eg_iocc_oob_2017_4.bsp, with the prebuilt images.
2017.4-l 80003.1956combined65956/8964791 (0.74%)combined 
2017.4-l 81923.1958combined0/2652067 (0%)combined 
        
2018.2 3.28198150/42402804 (0%)168319/42402801 (0.4%)Using the uz3eg_iocc_oob_2018_2.bsp, with the prebuilt images.
2018.2-l 80003.29549480/8942029 (0%)51955/8942027 (0.58%) 
2018.2-l 81923.29589550/8772086 (0%)31300/8772085 (0.36%) 
        
2019.1 3.69129120/8554232 (0%)5462/8554230 (0.064%)Using the uz3eg_iocc_oob_2018_2.bsp grabbed hardware design and upgraded to 2019.1. The BSP was from a template type matched closely to the 2018.2 bsp Petalinux configurations.
2019.1-l 80003.66986980/6394440 (0%)4047/6394440 (0.063%) 
2019.1-l 81923.69319310/8523328 (0%)9097/8523327 (0.11%) 
        
2019.1 3.69117130/47178844 (0%)10227670/47178844 (22%)Using custom built bsp from template and 2019.1 Vivado project designed after the 2018.2 UZ3EG_IOCC hardware design.
2019.1 3.69119090/47192218 (0%)83446/47192215 (0.18%) 
2019.1 3.68958920/46378569 (0%)171680/46378567 (0.37%) 
2019.1-l 80003.67157140/6703005 (0%)4699/6703005 (0.07%) 
2019.1-l 81923.67157150/6546217 (0%)2823/6546217 (0.043%) 
        
2019.1 3.69119100/47196510 (0%)43692/47196506 (0.093%)Using custom built bsp from template and 2019.1 Vivado project designed after the 2018.2 UZ3EG_IOCC hardware design.
The clock was adjusted in Vivado from 33.333 MHz to 33.333333333 MHz.
2019.1 3.69069020/46901657 (0%)201414/46901657 (0.43%) 
2019.1 3.68948940/4631909428589/46319090 (0.062%) 
2019.1-l 80003.67997970/7487634 (0%)18874/7487634 (0.25%) 
2019.1-l 81923.67047030/6440846 (0%)4068/6440846 (0.063%) 

 

0 Kudos
Highlighted
Participant
Participant
509 Views
Registered: ‎08-10-2018

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution
I am unable to hit the higher speeds of the 958 Mbits/sec for some unknown reason. It could be a combination of the Iperf3 version and the PetaLinux/Vivado design.
0 Kudos
Xilinx Employee
Xilinx Employee
304 Views
Registered: ‎08-15-2018

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution

Hey @stew.hansen,

My apologies, I somehow missed your responses to this thread. I wanted to check in with you (even though it's quite late) to see if you had made any progress in hitting rates in the ballpark of 900+Mbps. I do see that you got up to 900Mbps on the 19.1 images.

 

If it's still relevant to you, I would be more than happy to run some more tests on my end, perhaps with the custom images that you built and see what kind of results I get.

 

It's difficult for me to say what exactly this could be caused by since at the time I was unable to reproduce the failure on my end. One thought that crossed my mind is that I was unable to test using the prebuilt images since I had to change the PHY address in the device tree to match my PHY strapping. I'm not sure that this would have any impact on performance, however, since the PetaLinux settings and HW design that generated the pre-built images should be the same as what was in the .bsp, but you never know.

 

Thanks,

Clayton

Highlighted
Participant
Participant
205 Views
Registered: ‎08-10-2018

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution

Currently we getting about 830 MBits/s with UDP and about 939 MBits/s with TCP.

The priority of this has gone down we may look at it after we have our prototype hardware finished, but for now the speeds seem adequate.

Currently the only thing we did that had any affect was adusting the PSU__PSS_REF_CLK_FREQMHZ from 33.333 to 33.33333333333333.

View solution in original post

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
113 Views
Registered: ‎08-15-2018

Re: Petalinux GEM Tuning UltraZed-EG

Jump to solution

Hey @stew.hansen,

Understood - if the priority of the issue changes once your prototype HW is done, ping me and we can work from there.

 

Thanks,

Clayton

0 Kudos