09-03-2020 04:08 PM
09-08-2020 03:44 PM
The image wasn't attached properly so not sure what you were referring to.
The performance issue could be due to link issue as well. Have you checked if LTSSM is dropping to recovery during data transfer?
09-08-2020 03:55 PM
Thanks for your reply.
Yes I have checked the LTSSM and PCIe link status, the link is stable with no disconnection, error or recovery.
The snap shot of the credit related parameters are attached again here.
09-11-2020 07:47 AM
Thanks for posting the image. Those parameters are factory defined and need not be tuned. Regarding the issue you are having, it is difficult to pin-point where the issue might be coming from. I have put below few pointers that might help to debug the issue:
09-16-2020 12:40 AM
Thanks for your reply.
Here is the answer of your recommendations:
I would like to emphasize on the fact that, the performance is around 88% (Read and Write) when we use a 10m optical cable and it only drops on the C2H direction when we use the 100m optical cable.
Here is a two snapshots of the Completer request Axi interface plus the credits counts of the end device for both scenarios (10M and 100M). I hope it could shed some light on this issue:
09-16-2020 01:35 AM
09-21-2020 02:13 PM - edited 09-21-2020 02:14 PM
100m of fiber is around 1 us of delay (2 * 100 m / (0.7c) = 953 ns). I'm not sure offhand how sensitive PCIe is to propagation delay, but it might be something to look in to. I suspect that this would affect reads against host memory more than anything else, but it's possible that the issue is not having sufficient flow control credits to sustain full rate with such a large delay in releasing the credits. And the number of flow control credits in that direction would be dependent on the link partner.
09-21-2020 04:03 PM
Thanks aforencich for giving the most sensible response.
The problem is, because of the long propagation delay the end device runs out of the credits as shown in the figures that I posted earlier.
I don't how to increase the the number of credits manually or increase the depth of the compeletor request buffer which is currently 512 for the XDMA.
Do you know if using QDMA instead of XDMA would help?
09-21-2020 04:44 PM
Is the problem PCIe reads against host memory that are initiated by the card, or PCIe writes against host memory initiated by the card?
I think the PCIe hard IP core has around 32 KB of completion buffer space. If that is exhausted, then it can queue up some number of read requests internally. If that queue fills up, then it blocks on the AXI stream interface into the core. I don't think there is any way to adjust the completion buffer side. So if that's the problem, I think you're pretty much out of luck.
However, for write requests, the limitation is going to be the buffer space on the upstream PCIe port. You'll have to make some adjustment on that end of the link. It's also possible that the device simply doesn't have enough buffer space, in which case you'll have to find some other approach, such as using a different motherboard.
09-23-2020 01:11 AM
Actually, another consideration is the size of the operations. If you're issuing a lot of small operations, you'll eat up all of the available tags and header credits quickly, and then you'll have to wait at least 1 RTT for these to be replenished, which is delay-dependent.