04-23-2012 10:43 PM
Can anybody explain why the completion header and data flow control credits are infinite for endpoints?
Why isn't the endpoint allowed to thottle the completion TLPs from its link mate?
04-24-2012 06:16 AM
I think there are multiple reasons for that.
First, if there was a finite limit for completion credit, completions would have to be buffered by switches or in other parts of the PCI tree, leading to possible deadlocks. The PCI/PCIe TLP ordering assumes that completions can always pass, at least towards the device in DMA read mode. The PCIe specification does not always state the reasons for their decisions, but it states that
Second, infinite completions means that the device is under control of the read request throughput. No additional (large) buffering resources are required for these completions anywhere else than in the completer (i.e. main memory controller) for cache line performance optimization and in the device requesting the DMA read itself.
Third, if you receive completions, you as an endpoint have requested that data by issuing read TLPs. So it is easy for you to avoid this conflict by just releasing as many read requests as completions fit your rx buffers, then you are on the safe side. I think, that’s a fair design decision by the PCI consortium.
I have to admit, I also was curious about this question when I started into PCIe applications, but at some point you just accept that rule. The downside of having infinite completion credits is that you have to implement this mysterious completion credit buffer handling block for managing read requests. Just look at Appendix F at this document for a good description of how to do this calculation properly and efficiently, depending on your application need. If I understand correctly, the STREAM_FC method doesn’t require you to do any completion credit calculation which you buy by guaranteeing that you will never deassert the rx_ready interface signal, i.e. your throat is always wider than your mouth in rx direction. In my designs I have never relied on that property, so I implement DATA_FC together with a full-rate data acceptance interface.
04-24-2012 08:39 AM
Thanks Matthias for your detailed description. It helped me a lot to understand why there is an infinite limit for completion credit. Very interesting are the strategies for avoiding completion buffer overflow ;)
But can you figure out a possible deadlock situation which one can occure, unless there are flow control limits for completions?
04-24-2012 12:39 PM
The famous PCIe table of ordering rules just states that completions must be allowed to pass read requests (and other non-posted requests) to avoid deadlocks. And it states that since this property is not required to be held for some root complex interconnects, they could deadlock. My understanding is that if finite completion credit would be possible, it would result in switches not being able to adhere to that rule.
I think I can remember that I once managed to find a good understanding of said table but since then I just accept the rules. I’m pretty sure the surrounding part of the spec will state some additional reasoning behind the rule, but I just can’t search for it now, sorry.