UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Adventurer
Adventurer
1,178 Views
Registered: ‎11-27-2010

Ethernet SMP Linux/tftp/scp slow smp affinity Arm a53 ultrascale MPSoc

Hi Xilinx Team,

                      I found an issue in petalinux 2017.1 interrupt irq handling despite smp_affinity set to 0xF.

 

1. Cadence zynqmp Macb driver + phy device --> all configured in 1G mode [speed checked at /sys/class/net/eth0/speed at 1000]

2. by default SMP linux is enabled  [evident from uname -a].

3. I have attached pictures, please see the cat /proc/interrupts [Before and After]

4. The ethernet irq is 30 and plz look at the smp_affinity [it is 0xF = 4'b1111].

 

interrupts: - 

before [smp affinity 0xf]: - 

before starting tftp, the affinity 0xF and the interrupts are 123 in number. [interrupt numbered 30]

after tftp is done,as you can see the interrupts are 128309 and they are handled in core0 ONLY. This is despite, SMP being set. interrupts or packets reception should have been distributed across cores.

result: - 300 KiloBytes/sec speed

Ack block ignored errors [received twice] in tftpd

lots of tcp retransmission measured via [scp] - another experiment

73 seconds for just 32 MBytes of file.

cause: - all Interrupts are taken on core0 being SMP linux

 

after[smp affinity 0xe]: - [it is 0xE = 4'b1110]

i removed the core0 from being packet receiver.

core0 is removed from the affinity, so core0 does not process any eth packets.

here also, though allocated cores are 3 in number, only core 1 is picked for interrupt processing and since it is free from HIGH frequency ams-irq interrupts, core 1 handles only eth interrupts and handles it well

result: - 3 MegaBytes/sec speed

No Ack block ignored errors 

NO tcp retransmission measured via [scp] 

10 seconds for 32 MBytes of file.

cause: - all Interrupts are taken on core1 being SMP linux

 

 

 

Now questions: - 

1. let me know if i am missing anything in my CONFIG in linux [which distributes interrupts/packet handling across cores]?

if not, definitely there is issue in packet handling in Xilinx kernel/petalinux.

2. Did you/team even test the tcp/udp throughput before each release,if so can you please publish/share results on udp&tcp throughput statistics/curves on 2017.1/2/3 Xilinx linux kernel.?

3. Is this smp failure to distribute interrupts/bottom halves/work handling across all cores applicable to ethernet or other peripherals as well?

 

Thanks

 

 

tftp_slow_before.jpg
tftp_slow_after.jpg
0 Kudos
1 Reply
Adventurer
Adventurer
1,111 Views
Registered: ‎11-27-2010

Re: Ethernet SMP Linux/tftp/scp slow smp affinity Arm a53 ultrascale MPSoc

Based on this post,

 

https://community.arm.com/processors/f/discussions/2591/how-does-the-gic-distribute-interrupts-between-processors

 

i found that irqbalance is MUST to balance interrupts and distributor just does the CONCENTRATOR job [surprisingly]. I built the irqbalance [AARCH64] and i could see it balancing interrupts before and after.[same AFTER results screenshot are still applicable in case of irqbalance also]

 

also, i could find irqbalance in https://www.xilinx.com/support/documentation/sw_manuals/ug1046-ultrafast-design-methodology-guide.pdf [courtesy google search] [Page 56]

 

"In SMP Linux, interrupt processing can be done by either CPU with the default being to run on CPU0. If processing is left to the default it can result in a large load on one CPU. The CPU affinity of each interrupt can be altered from user space to balance the processing load. There are also applications that help balance interrupt loads, such as irqbalance."

 

0 Kudos