cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
goghvv
Visitor
Visitor
8,971 Views
Registered: ‎10-02-2015

An example of data flow in PCIe

Jump to solution

I read about TLPs; about how TLPs are routed, about the different types of TLPs, about how they can carry data and so one.

I still have a pretty basic question I'm trying to resolve:

 

Take an ordinary personal computer system.

To my best understanding, the OS on that PC implements both the TCP and the IP layers of the Internet protocol stack, while the Ethernet protocol is handled by the Network Adapter (NIC).

So, suppose that this OS - on behalf of the user, of-course - wants to send some data to the Internet over a TCP/IP connection.

It will first set up some TCP buffers, add a TCP header to the data, and then add IP header, and now this IP packet is ready to be shifted to the network adapter.

Of-course, since this newly assembled IP packet needs to travel through the PCIe fabric, it must be traveled as a TLP - or more accuratly, as the payload of a TLP.

 

My question is: How exaclty is it accomplished? What type of TLP will be sent and where to?

 

The TLPs that can carry data are:

1. Memory Write Request

2. IO Write Request (lets ignore this one, since I understand it is a dying breed)

3. Configuration Type 0/1 Write Request

4. Message Request

5. Completion

 

The only one that makes sense to be used here is the first - Memory Write Request, but where will this be addressed to?

The BARs assigned to that NIC is the only thing that comes to my mind, but then what?

Say it was written to some memory chunk on the System RAM, to which the NIC's address space is mapped to.

How will the NIC know to read this data, and more importantly - how will it know what to do with it?

0 Kudos
Reply
1 Solution

Accepted Solutions
avrumw
Guide
Guide
16,102 Views
Registered: ‎01-23-2009

First, realize that the real (not virtual) memory space of the PC is flat - there is one address space.

 

All system resources are mapped into that address space - this includes all DRAM, all ROM, and all memory mapped peripherals.

 

Each PCIe device has a configuarion space that helps the Root Complex of the PCIe system (which is generally the PCIe controller of the main processor and the associated drivers for it) map the BARs to the address space of the PC. After this process (called enumeration) is done, all PCIe devices have one or more address spaces in the main memory map (there is also an I/O space, but lets assume that the PCIe device doesn't use that).

 

WIthin the BAR requested by the NIC, the address space will be further broken down. Inside this memory space there will be (at least) some number of registers that control the NIC. Once the NIC is up and running, the CPU can write to these registers to initiate operations. For the most part, these operations are done using Direct Memory Access (DMA).

 

So, to generate a TCP/IP packed, the CPU would get some memory within the DRAM address space (it would use a system function like malloc to get the space). It would then create the TCP/IP packet in this space. It would know the address location (in DRAM) of the packet, and the length of the packet, etc..

 

Then a device driver (software) associated with the NIC would be called with the pointer to the memory, and the length of the packet. This driver would then perform register accesses to the NIC's registers. The driver is specifically written for the NIC, so it knows all the registers associated with the NIC. It also knows the address that was assigned to the BARs of the NIC through the operating system.

 

As part of this, the driver would tell the NIC "Here is the packet address and the length, go get it and then transmit it".

 

With this information the NIC would initiate a PCIe memory read for the TCP/IP packet, which it would (probably) buffer internally, add the required low level headers and checks, and send it ont he Ethernet.

 

All of these reads and writes between the CPU and the NIC registers and between the NIC and the DRAM are done using PCIe Memory Read and Write operations. These are converted to TLP packets (most likely) by the PCIe hardware in the root complex and the NIC.

 

Avrum

View solution in original post

3 Replies
avrumw
Guide
Guide
16,103 Views
Registered: ‎01-23-2009

First, realize that the real (not virtual) memory space of the PC is flat - there is one address space.

 

All system resources are mapped into that address space - this includes all DRAM, all ROM, and all memory mapped peripherals.

 

Each PCIe device has a configuarion space that helps the Root Complex of the PCIe system (which is generally the PCIe controller of the main processor and the associated drivers for it) map the BARs to the address space of the PC. After this process (called enumeration) is done, all PCIe devices have one or more address spaces in the main memory map (there is also an I/O space, but lets assume that the PCIe device doesn't use that).

 

WIthin the BAR requested by the NIC, the address space will be further broken down. Inside this memory space there will be (at least) some number of registers that control the NIC. Once the NIC is up and running, the CPU can write to these registers to initiate operations. For the most part, these operations are done using Direct Memory Access (DMA).

 

So, to generate a TCP/IP packed, the CPU would get some memory within the DRAM address space (it would use a system function like malloc to get the space). It would then create the TCP/IP packet in this space. It would know the address location (in DRAM) of the packet, and the length of the packet, etc..

 

Then a device driver (software) associated with the NIC would be called with the pointer to the memory, and the length of the packet. This driver would then perform register accesses to the NIC's registers. The driver is specifically written for the NIC, so it knows all the registers associated with the NIC. It also knows the address that was assigned to the BARs of the NIC through the operating system.

 

As part of this, the driver would tell the NIC "Here is the packet address and the length, go get it and then transmit it".

 

With this information the NIC would initiate a PCIe memory read for the TCP/IP packet, which it would (probably) buffer internally, add the required low level headers and checks, and send it ont he Ethernet.

 

All of these reads and writes between the CPU and the NIC registers and between the NIC and the DRAM are done using PCIe Memory Read and Write operations. These are converted to TLP packets (most likely) by the PCIe hardware in the root complex and the NIC.

 

Avrum

View solution in original post

goghvv
Visitor
Visitor
8,946 Views
Registered: ‎10-02-2015
Wow, thanks again, Avrum!
That clears up all my confusion.
0 Kudos
Reply
aliciachee
Contributor
Contributor
1,560 Views
Registered: ‎10-15-2018

do you have an example code which I can reference to for virtex-7 vc707 fpga board using windows 10?

0 Kudos
Reply