cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
Observer
Observer
8,491 Views
Registered: ‎09-20-2011

Flow Control Completion TLPs

Hello,

 

Can anybody explain why the completion header and data flow control credits are infinite for endpoints?

Why isn't the endpoint allowed to thottle the completion TLPs from its link mate?

 

Thanks

-----------------------------------------------------------------------------------------
It's always good practice giving feedback, when description/solution/recommendation have been helpful!
0 Kudos
3 Replies
Highlighted
Visitor
Visitor
8,480 Views
Registered: ‎07-07-2009

I think there are multiple reasons for that.

 

First, if there was a finite limit for completion credit, completions would have to be buffered by switches or in other parts of the PCI tree, leading to possible deadlocks. The PCI/PCIe TLP ordering assumes that completions can always pass, at least towards the device in DMA read mode. The PCIe specification does not always state the reasons for their decisions, but it states that

 

Second, infinite completions means that the device is under control of the read request throughput. No additional (large) buffering resources are required for these completions anywhere else than in the completer (i.e. main memory controller) for cache line performance optimization and in the device requesting the DMA read itself.

 

Third, if you receive completions, you as an endpoint have requested that data by issuing read TLPs. So it is easy for you to avoid this conflict by just releasing as many read requests as completions fit your rx buffers, then you are on the safe side. I think, that’s a fair design decision by the PCI consortium.

 

I have to admit, I also was curious about this question when I started into PCIe applications, but at some point you just accept that rule. The downside of having infinite completion credits is that you have to implement this mysterious completion credit buffer handling block for managing read requests. Just look at Appendix F at this document for a good description of how to do this calculation properly and efficiently, depending on your application need. If I understand correctly, the STREAM_FC method doesn’t require you to do any completion credit calculation which you buy by guaranteeing that you will never deassert the rx_ready interface signal, i.e. your throat is always wider than your mouth in rx direction. In my designs I have never relied on that property, so I implement DATA_FC together with a full-rate data acceptance interface.

 

– Matthias

Highlighted
Observer
Observer
8,471 Views
Registered: ‎09-20-2011

Thanks Matthias for your detailed description. It helped me a lot to understand why there is an infinite limit for completion credit. Very interesting are the strategies for avoiding completion buffer overflow ;)

 

But can you figure out a possible deadlock situation which one can occure, unless there are flow control limits for completions?

-----------------------------------------------------------------------------------------
It's always good practice giving feedback, when description/solution/recommendation have been helpful!
0 Kudos
Highlighted
Visitor
Visitor
8,462 Views
Registered: ‎07-07-2009

The famous PCIe table of ordering rules just states that completions must be allowed to pass read requests (and other non-posted requests) to avoid deadlocks. And it states that since this property is not required to be held for some root complex interconnects, they could deadlock. My understanding is that if finite completion credit would be possible, it would result in switches not being able to adhere to that rule.

 

I think I can remember that I once managed to find a good understanding of said table but since then I just accept the rules. I’m pretty sure the surrounding part of the spec will state some additional reasoning behind the rule, but I just can’t search for it now, sorry.

 

– Matthias