11-06-2017 07:06 PM
Hi Xilinx Team,
I found an issue in petalinux 2017.1 interrupt irq handling despite smp_affinity set to 0xF.
1. Cadence zynqmp Macb driver + phy device --> all configured in 1G mode [speed checked at /sys/class/net/eth0/speed at 1000]
2. by default SMP linux is enabled [evident from uname -a].
3. I have attached pictures, please see the cat /proc/interrupts [Before and After]
4. The ethernet irq is 30 and plz look at the smp_affinity [it is 0xF = 4'b1111].
before [smp affinity 0xf]: -
before starting tftp, the affinity 0xF and the interrupts are 123 in number. [interrupt numbered 30]
after tftp is done,as you can see the interrupts are 128309 and they are handled in core0 ONLY. This is despite, SMP being set. interrupts or packets reception should have been distributed across cores.
result: - 300 KiloBytes/sec speed
Ack block ignored errors [received twice] in tftpd
lots of tcp retransmission measured via [scp] - another experiment
73 seconds for just 32 MBytes of file.
cause: - all Interrupts are taken on core0 being SMP linux
after[smp affinity 0xe]: - [it is 0xE = 4'b1110]
i removed the core0 from being packet receiver.
core0 is removed from the affinity, so core0 does not process any eth packets.
here also, though allocated cores are 3 in number, only core 1 is picked for interrupt processing and since it is free from HIGH frequency ams-irq interrupts, core 1 handles only eth interrupts and handles it well
result: - 3 MegaBytes/sec speed
No Ack block ignored errors
NO tcp retransmission measured via [scp]
10 seconds for 32 MBytes of file.
cause: - all Interrupts are taken on core1 being SMP linux
Now questions: -
1. let me know if i am missing anything in my CONFIG in linux [which distributes interrupts/packet handling across cores]?
if not, definitely there is issue in packet handling in Xilinx kernel/petalinux.
2. Did you/team even test the tcp/udp throughput before each release,if so can you please publish/share results on udp&tcp throughput statistics/curves on 2017.1/2/3 Xilinx linux kernel.?
3. Is this smp failure to distribute interrupts/bottom halves/work handling across all cores applicable to ethernet or other peripherals as well?
11-10-2017 04:26 PM
Based on this post,
i found that irqbalance is MUST to balance interrupts and distributor just does the CONCENTRATOR job [surprisingly]. I built the irqbalance [AARCH64] and i could see it balancing interrupts before and after.[same AFTER results screenshot are still applicable in case of irqbalance also]
also, i could find irqbalance in https://www.xilinx.com/support/documentation/sw_manuals/ug1046-ultrafast-design-methodology-guide.pdf [courtesy google search] [Page 56]
"In SMP Linux, interrupt processing can be done by either CPU with the default being to run on CPU0. If processing is left to the default it can result in a large load on one CPU. The CPU affinity of each interrupt can be altered from user space to balance the processing load. There are also applications that help balance interrupt loads, such as irqbalance."