Sign In

Don't have a Xilinx account yet?

  • Choose to receive important news and product information
  • Gain access to special content
  • Personalize your web experience on Xilinx.com

Create Account

Username

Password

Forgot your password?
XClose Panel
Xilinx Home
Reply
Regular Visitor
hse07013
Posts: 14
Registered: ‎09-20-2011
0

Flow Control Completion TLPs

Hello,

 

Can anybody explain why the completion header and data flow control credits are infinite for endpoints?

Why isn't the endpoint allowed to thottle the completion TLPs from its link mate?

 

Thanks

-----------------------------------------------------------------------------------------
It's always good practice giving feedback, when description/solution/recommendation have been helpful!
Visitor
mwaechter
Posts: 18
Registered: ‎07-07-2009

Re: Flow Control Completion TLPs

I think there are multiple reasons for that.

 

First, if there was a finite limit for completion credit, completions would have to be buffered by switches or in other parts of the PCI tree, leading to possible deadlocks. The PCI/PCIe TLP ordering assumes that completions can always pass, at least towards the device in DMA read mode. The PCIe specification does not always state the reasons for their decisions, but it states that

 

Second, infinite completions means that the device is under control of the read request throughput. No additional (large) buffering resources are required for these completions anywhere else than in the completer (i.e. main memory controller) for cache line performance optimization and in the device requesting the DMA read itself.

 

Third, if you receive completions, you as an endpoint have requested that data by issuing read TLPs. So it is easy for you to avoid this conflict by just releasing as many read requests as completions fit your rx buffers, then you are on the safe side. I think, that’s a fair design decision by the PCI consortium.

 

I have to admit, I also was curious about this question when I started into PCIe applications, but at some point you just accept that rule. The downside of having infinite completion credits is that you have to implement this mysterious completion credit buffer handling block for managing read requests. Just look at Appendix F at this document for a good description of how to do this calculation properly and efficiently, depending on your application need. If I understand correctly, the STREAM_FC method doesn’t require you to do any completion credit calculation which you buy by guaranteeing that you will never deassert the rx_ready interface signal, i.e. your throat is always wider than your mouth in rx direction. In my designs I have never relied on that property, so I implement DATA_FC together with a full-rate data acceptance interface.

 

– Matthias

Regular Visitor
hse07013
Posts: 14
Registered: ‎09-20-2011
0

Re: Flow Control Completion TLPs

Thanks Matthias for your detailed description. It helped me a lot to understand why there is an infinite limit for completion credit. Very interesting are the strategies for avoiding completion buffer overflow ;)

 

But can you figure out a possible deadlock situation which one can occure, unless there are flow control limits for completions?

-----------------------------------------------------------------------------------------
It's always good practice giving feedback, when description/solution/recommendation have been helpful!
Visitor
mwaechter
Posts: 18
Registered: ‎07-07-2009

Re: Flow Control Completion TLPs

The famous PCIe table of ordering rules just states that completions must be allowed to pass read requests (and other non-posted requests) to avoid deadlocks. And it states that since this property is not required to be held for some root complex interconnects, they could deadlock. My understanding is that if finite completion credit would be possible, it would result in switches not being able to adhere to that rule.

 

I think I can remember that I once managed to find a good understanding of said table but since then I just accept the rules. I’m pretty sure the surrounding part of the spec will state some additional reasoning behind the rule, but I just can’t search for it now, sorry.

 

– Matthias