cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
paul12345
Observer
Observer
277 Views
Registered: ‎07-14-2020

Buggy clock configuration RfSoC Ultrascale+ DMA/Bridge Subsystem for PCI Express (DMA mode)

The clocking documentation for the DMA/Bridge PCIe IP is severely lacking. On Ultrascale+ (RfSoC), the sys_clk_gt appears to be the GTY reference clock, and sys_clk is the source of the AXI-stream (and AXI4-lite and AXI4) clock. However, when I set "Reference Clock Frequency (MHz)" to 100MHz and "AXI Clock Frequency" to 250MHz, it appears that the two clock frequencies get swapped. That is, Vivado 2020.1 shows that sys_clk is 100 MHz and sys_clk_gt is 250 MHz. See below:

paul12345_0-1627602105587.png

paul12345_1-1627602133760.png

paul12345_2-1627602147169.png

Isn't this configured backwards by the IP wizard? I have looked at the generated schematic and sys_clk_gt is the reference clock for the GTYs. The GT refclk comes from the motherboard / PCIe connector and should be 100 MHz per PCIe spec. The 250 MHz frequency is a property of the input and output AXI-stream widths and the number of lanes, and it can be 125 MHz or 250 MHz. I would expect that sys_clk_gt corresponds to the "Reference Frequency" setting in the GUI, and should *always* allow 100 MHz in case a user doesn't want to put a PLL between the PCIe connector clock and the GTREFCLK. The reason, of course, is because there is already a PLL in the GT complex that should be able to accept 100MHz and upconvert it to the line rate for the GTs.

Below is the synthesized schematic highlighting sys_clk_gt, showing that it is indeed connected to GTREFCLK.

paul12345_3-1627602460711.png

This bug appears to make it impossible to use this IP while also setting clock frequencies correctly, because 100 MHz needs to be input to the GTY refclk, and that is not an option when configuring the IP, given these swapped port frequencies.

0 Kudos
3 Replies
pvenugo
Moderator
Moderator
238 Views
Registered: ‎07-31-2012

@paul12345 ,

Please share PCIe core .xci file and _bd.tcl to replicate the issue in 2020.2 . Please cross check in 2021.1 if you are seeing this issue.


-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
paul12345
Observer
Observer
199 Views
Registered: ‎07-14-2020

There are about 5 xci files generated by Vivado for the xdma IP, so rather than send all of those, attached is some tcl you can use to recreate the board. The connections are a bit different (and I split sys_clk and sys_clk_gt ports into two separate ports at the moment), but this issue is unrelated to the connections in the board and shows up as a bug in the clock frequencies of the IP block itself.

The forum won't allow me to upload .tcl files or .txt files, so below is the body of these files pasted. From there you can experience what appears to be this clock-frequency swap bug with xdma_0 by modifying the clock frequency settings. The issue is the same in 2021.1.

# This creates ps and boilerplate AXI bus hardware in a bd design.
#
# All things which should be shared across projects can go here; although you
# can also source this and then have per-project customizations of its parameters
# afterwards, if needed.

create_bd_cell -type ip -vlnv xilinx.com:ip:zynq_ultra_ps_e:3.3 zynq_ultra_ps_e_0

set_property -dict [list \
CONFIG.PSU__CRL_APB__PL0_REF_CTRL__SRCSEL {IOPLL} \
CONFIG.PSU__CRL_APB__PL1_REF_CTRL__SRCSEL {IOPLL} \
CONFIG.PSU__CRL_APB__PL2_REF_CTRL__SRCSEL {IOPLL} \
] [get_bd_cells zynq_ultra_ps_e_0]

set_property -dict [list \
CONFIG.PSU__ENET3__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__ENET3__GRP_MDIO__ENABLE {1} \
CONFIG.PSU__I2C0__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__I2C0__PERIPHERAL__IO {MIO 14 .. 15} \
CONFIG.PSU__I2C1__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__I2C1__PERIPHERAL__IO {EMIO} \
CONFIG.PSU__QSPI__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__QSPI__PERIPHERAL__MODE {Dual Parallel} \
CONFIG.PSU__SD0__PERIPHERAL__ENABLE {0} \
CONFIG.PSU__SD1__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__SD1__GRP_CD__ENABLE {1} \
CONFIG.PSU__DDRC__CWL {16} \
CONFIG.PSU__DDRC__BG_ADDR_COUNT {1} \
CONFIG.PSU__DDRC__DEVICE_CAPACITY {8192 MBits} \
CONFIG.PSU__DDRC__DRAM_WIDTH {16 Bits} \
CONFIG.PSU__DDRC__ROW_ADDR_COUNT {16} \
CONFIG.PSU__DDRC__SPEED_BIN {DDR4_2400R} \
CONFIG.PSU__UART0__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__UART0__PERIPHERAL__IO {MIO 18 .. 19} \
CONFIG.PSU_MIO_70_INPUT_TYPE {schmitt} \
CONFIG.PSU_MIO_71_INPUT_TYPE {schmitt} \
CONFIG.PSU_MIO_72_INPUT_TYPE {schmitt} \
CONFIG.PSU_MIO_73_INPUT_TYPE {schmitt} \
CONFIG.PSU_MIO_74_INPUT_TYPE {schmitt} \
CONFIG.PSU_MIO_75_INPUT_TYPE {schmitt} \
CONFIG.PSU_MIO_77_INPUT_TYPE {schmitt} \
CONFIG.PSU_BANK_1_IO_STANDARD {LVCMOS18} \
CONFIG.PSU_BANK_2_IO_STANDARD {LVCMOS18} \
CONFIG.PSU_BANK_3_IO_STANDARD {LVCMOS18} \
CONFIG.PSU__CRF_APB__DDR_CTRL__FREQMHZ {1200} \
CONFIG.PSU__FPGA_PL1_ENABLE {1} \
CONFIG.PSU__CRL_APB__PL1_REF_CTRL__FREQMHZ {500} \
CONFIG.PSU__FPGA_PL2_ENABLE {1} \
CONFIG.PSU__CRL_APB__PL1_REF_CTRL__FREQMHZ {250} \
CONFIG.PSU__CRL_APB__PL2_REF_CTRL__FREQMHZ {400} \
CONFIG.PSU__SWDT0__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__SWDT0__RESET__ENABLE {1} \
CONFIG.PSU__SWDT0__RESET__IO {EMIO} \
CONFIG.PSU__SWDT1__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__SWDT1__RESET__ENABLE {1} \
CONFIG.PSU__SWDT1__RESET__IO {EMIO} \
CONFIG.PSU__TTC0__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__TTC1__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__TTC2__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__TTC3__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__CSU__PERIPHERAL__ENABLE {1} \
CONFIG.PSU__CSU__PERIPHERAL__IO {MIO 31} \
CONFIG.PSU_BANK_0_IO_STANDARD {LVCMOS18} \
] [get_bd_cells zynq_ultra_ps_e_0]

# Additional clock pins
make_bd_pins_external [get_bd_pins zynq_ultra_ps_e_0/pl_clk0]
make_bd_pins_external [get_bd_pins zynq_ultra_ps_e_0/pl_clk1]
make_bd_pins_external [get_bd_pins zynq_ultra_ps_e_0/pl_clk2]
set_property name pl_clk0 [get_bd_ports pl_clk0_0]
set_property name pl_clk1 [get_bd_ports pl_clk1_0]
set_property name pl_clk2 [get_bd_ports pl_clk2_0]

# Add a GPIO block
create_bd_cell -type ip -vlnv xilinx.com:ip:axi_gpio:2.0 axi_gpio_0
set_property -dict [list \
CONFIG.C_GPIO_WIDTH {32} \
CONFIG.C_ALL_OUTPUTS {1} \
] [get_bd_cells axi_gpio_0]

make_bd_intf_pins_external [get_bd_intf_pins axi_gpio_0/GPIO]

apply_bd_automation -rule xilinx.com:bd_rule:axi4 -config { \
Clk_master {Auto} \
Clk_slave {Auto} \
Clk_xbar {Auto} \
Master {/zynq_ultra_ps_e_0/M_AXI_HPM0_LPD} \
Slave {/axi_gpio_0/S_AXI} \
ddr_seg {Auto} \
intc_ip {New AXI Interconnect} \
master_apm {0} \
} [get_bd_intf_pins axi_gpio_0/S_AXI]

# Connect the watchdog timer 0 reset output to reset all peripherals (we want
# the fabric to reset and start fresh if the processor hangs or reboots).
create_bd_cell -type ip -vlnv xilinx.com:ip:util_vector_logic:2.0 wdt0_reset_not
set_property -dict [list CONFIG.C_SIZE {1} CONFIG.C_OPERATION {not} CONFIG.LOGO_FILE {data/sym_notgate.png}] [get_bd_cells wdt0_reset_not]
connect_bd_net [get_bd_pins zynq_ultra_ps_e_0/emio_wdt0_rst_o] [get_bd_pins wdt0_reset_not/Op1]

# Reset pin (use pl_resetn0 -> made synchronous by the reset generator).
create_bd_port -dir O pl_resetn0
connect_bd_net [get_bd_ports pl_resetn0] [get_bd_pins rst_ps8_0_99M/peripheral_aresetn]
connect_bd_net [get_bd_pins wdt0_reset_not/Res] [get_bd_pins rst_ps8_0_99M/aux_reset_in]

# Instantiate PCIe/DMA block
create_bd_cell -type ip -vlnv xilinx.com:ip:xdma:4.1 xdma_0
set_property -dict [list \
CONFIG.pl_link_cap_max_link_width {X8} \
CONFIG.pl_link_cap_max_link_speed {8.0_GT/s} \
CONFIG.ref_clk_freq {250_MHz} \
CONFIG.drp_clk_sel {Internal} \
CONFIG.axi_data_width {256_bit} \
CONFIG.axisten_freq {250} \
CONFIG.pf0_device_id {9038} \
CONFIG.pf0_base_class_menu {Processing_accelerators} \
CONFIG.pf0_class_code_base {12} \
CONFIG.pf0_sub_class_interface_menu {Unknown} \
CONFIG.pf0_class_code_sub {00} \
CONFIG.pf0_class_code_interface {00} \
CONFIG.pf0_class_code {120000} \
CONFIG.axilite_master_en {true} \
CONFIG.axilite_master_size {128} \
CONFIG.axilite_master_scale {Kilobytes} \
CONFIG.axist_bypass_en {false} \
CONFIG.axist_bypass_scale {Gigabytes} \
CONFIG.xdma_rnum_chnl {4} \
CONFIG.xdma_wnum_chnl {4} \
CONFIG.xdma_axilite_slave {true} \
CONFIG.xdma_wnum_rids {32} \
CONFIG.coreclk_freq {500} \
CONFIG.plltype {QPLL1} \
CONFIG.xdma_axi_intf_mm {AXI_Stream} \
CONFIG.xdma_sts_ports {false} \
CONFIG.pf0_msix_cap_table_bir {BAR_1} \
CONFIG.pf0_msix_cap_pba_bir {BAR_1} \
CONFIG.PF0_DEVICE_ID_mqdma {9038} \
CONFIG.PF2_DEVICE_ID_mqdma {9038} \
CONFIG.PF3_DEVICE_ID_mqdma {9038} \
] [get_bd_cells xdma_0]

# AXI4-LITE connections
set_property -dict [list CONFIG.NUM_SI {2} CONFIG.NUM_MI {2}] [get_bd_cells ps8_0_axi_periph]
connect_bd_intf_net [get_bd_intf_pins xdma_0/M_AXI_LITE] -boundary_type upper [get_bd_intf_pins ps8_0_axi_periph/S01_AXI]
connect_bd_net [get_bd_pins xdma_0/axi_aclk] [get_bd_pins ps8_0_axi_periph/S01_ACLK]
connect_bd_net [get_bd_pins xdma_0/axi_aresetn] [get_bd_pins ps8_0_axi_periph/S01_ARESETN]
connect_bd_intf_net -boundary_type upper [get_bd_intf_pins ps8_0_axi_periph/M01_AXI] [get_bd_intf_pins xdma_0/S_AXI_LITE]
connect_bd_net [get_bd_pins xdma_0/axi_aclk] [get_bd_pins ps8_0_axi_periph/M01_ACLK]
connect_bd_net [get_bd_pins xdma_0/axi_aresetn] [get_bd_pins ps8_0_axi_periph/M01_ARESETN]

# GTY data lines
create_bd_intf_port -mode Master -vlnv xilinx.com:interface:pcie_7x_mgt_rtl:1.0 pcie_mgt
connect_bd_intf_net [get_bd_intf_pins xdma_0/pcie_mgt] [get_bd_intf_ports pcie_mgt]

# Clocking - is a huge pain and is poorly documented
# https://www.xilinx.com/support/answers/71730.html)
# https://forums.xilinx.com/t5/PCIe-and-CPM/XDMA-clock-pin-constraints-in-VCU128-board/td-p/1220178
set_property CONFIG.ext_sys_clk_bufg true [get_bd_cells xdma_0]
create_bd_port -dir I -type clk -freq_hz 250000000 pcie_sys_clk_gt
set_property CONFIG.CLK_DOMAIN [get_property CONFIG.CLK_DOMAIN [get_bd_pins xdma_0/sys_clk_gt]] [get_bd_ports pcie_sys_clk_gt]
connect_bd_net [get_bd_pins /xdma_0/sys_clk_gt] [get_bd_ports pcie_sys_clk_gt]
create_bd_port -dir I -type clk -freq_hz 250000000 pcie_sys_clk
set_property CONFIG.CLK_DOMAIN [get_property CONFIG.CLK_DOMAIN [get_bd_pins xdma_0/sys_clk]] [get_bd_ports pcie_sys_clk]
connect_bd_net [get_bd_pins /xdma_0/sys_clk] [get_bd_ports pcie_sys_clk]

# Reset
create_bd_port -dir I -type rst pcie_rst_n
connect_bd_net [get_bd_pins /xdma_0/sys_rst_n] [get_bd_ports pcie_rst_n]

# Loop back each AXI-S output to corresponding AXI-S input (4 DMA workers)
create_bd_cell -type ip -vlnv xilinx.com:ip:axis_data_fifo:2.0 axis_data_fifo_0
set_property -dict [list \
CONFIG.TDATA_NUM_BYTES.VALUE_SRC USER \
CONFIG.HAS_TKEEP.VALUE_SRC USER] [get_bd_cells axis_data_fifo_0]
set_property -dict [list \
CONFIG.TDATA_NUM_BYTES {32} \
CONFIG.FIFO_DEPTH {8192} \
CONFIG.FIFO_MODE {2} \
CONFIG.HAS_TKEEP {1} \
CONFIG.FIFO_MEMORY_TYPE {block}] [get_bd_cells axis_data_fifo_0]
copy_bd_objs / [get_bd_cells {axis_data_fifo_0}]
copy_bd_objs / [get_bd_cells {axis_data_fifo_0}]
copy_bd_objs / [get_bd_cells {axis_data_fifo_0}]
connect_bd_intf_net [get_bd_intf_pins xdma_0/M_AXIS_H2C_0] [get_bd_intf_pins axis_data_fifo_0/S_AXIS]
connect_bd_intf_net [get_bd_intf_pins axis_data_fifo_0/M_AXIS] [get_bd_intf_pins xdma_0/S_AXIS_C2H_0]
connect_bd_intf_net [get_bd_intf_pins xdma_0/M_AXIS_H2C_1] [get_bd_intf_pins axis_data_fifo_1/S_AXIS]
connect_bd_intf_net [get_bd_intf_pins axis_data_fifo_1/M_AXIS] [get_bd_intf_pins xdma_0/S_AXIS_C2H_1]
connect_bd_intf_net [get_bd_intf_pins xdma_0/M_AXIS_H2C_2] [get_bd_intf_pins axis_data_fifo_2/S_AXIS]
connect_bd_intf_net [get_bd_intf_pins axis_data_fifo_2/M_AXIS] [get_bd_intf_pins xdma_0/S_AXIS_C2H_2]
connect_bd_intf_net [get_bd_intf_pins xdma_0/M_AXIS_H2C_3] [get_bd_intf_pins axis_data_fifo_3/S_AXIS]
connect_bd_intf_net [get_bd_intf_pins axis_data_fifo_3/M_AXIS] [get_bd_intf_pins xdma_0/S_AXIS_C2H_3]
connect_bd_net [get_bd_pins xdma_0/axi_aclk] [get_bd_pins axis_data_fifo_1/s_axis_aclk]
connect_bd_net [get_bd_pins xdma_0/axi_aclk] [get_bd_pins axis_data_fifo_3/s_axis_aclk]
connect_bd_net [get_bd_pins xdma_0/axi_aclk] [get_bd_pins axis_data_fifo_2/s_axis_aclk]
connect_bd_net [get_bd_pins xdma_0/axi_aclk] [get_bd_pins axis_data_fifo_0/s_axis_aclk]
connect_bd_net [get_bd_pins xdma_0/axi_aresetn] [get_bd_pins axis_data_fifo_0/s_axis_aresetn]
connect_bd_net [get_bd_pins xdma_0/axi_aresetn] [get_bd_pins axis_data_fifo_2/s_axis_aresetn]
connect_bd_net [get_bd_pins xdma_0/axi_aresetn] [get_bd_pins axis_data_fifo_3/s_axis_aresetn]
connect_bd_net [get_bd_pins xdma_0/axi_aresetn] [get_bd_pins axis_data_fifo_1/s_axis_aresetn]

# Monitor one FIFO input stream with an ILA
create_bd_cell -type ip -vlnv xilinx.com:ip:ila:6.2 ila_0
set_property -dict [list \
CONFIG.C_DATA_DEPTH {16384} \
CONFIG.C_NUM_OF_PROBES {9} \
CONFIG.C_EN_STRG_QUAL {1} \
CONFIG.C_PROBE8_MU_CNT {2} \
CONFIG.C_PROBE7_MU_CNT {2} \
CONFIG.C_PROBE6_MU_CNT {2} \
CONFIG.C_PROBE5_MU_CNT {2} \
CONFIG.C_PROBE4_MU_CNT {2} \
CONFIG.C_PROBE3_MU_CNT {2} \
CONFIG.C_PROBE2_MU_CNT {2} \
CONFIG.C_PROBE1_MU_CNT {2} \
CONFIG.C_PROBE0_MU_CNT {2} \
CONFIG.ALL_PROBE_SAME_MU_CNT {2} \
CONFIG.C_SLOT_0_AXI_PROTOCOL {AXI4S}] [get_bd_cells ila_0]
connect_bd_intf_net [get_bd_intf_pins ila_0/SLOT_0_AXIS] [get_bd_intf_pins axis_data_fifo_0/S_AXIS]
connect_bd_net [get_bd_pins ila_0/clk] [get_bd_pins xdma_0/axi_aclk]

# Address mapping
assign_bd_address [get_bd_addr_segs {xdma_0/S_AXI_LITE/CTL0 }]
assign_bd_address [get_bd_addr_segs {axi_gpio_0/S_AXI/Reg }]
set_property offset 0x80000000 [get_bd_addr_segs {xdma_0/M_AXI_LITE/SEG_axi_gpio_0_Reg}]

0 Kudos
paul12345
Observer
Observer
142 Views
Registered: ‎07-14-2020

PG195 refers you to PG213 for Ultrascale+. But, the only mention of sys_clk_gt in PG213 is in Table 36 which says:

paul12345_0-1627921861393.png

First of all, this doesn't make sense in english, if it's "selectable" there needs to be more than one choice. Secondly, IP integrator does not actually allow 100MHz and gets sys_clk and sys_clk_gt frequencies reversed.

0 Kudos