Shutdown Hardware Manager when I use ILA/VIO on Alveo U280

[toku1938]

Dear,

I am working on Alveo U280 for HBM and 100GbE.
Observing some signals with ILA and VIO on hardware manager,
I have encountered two types of failure to detect device from time to time when I am using ILA/VIO.

1) showing error message and refreshing the window

ERROR: [Labtoolstcl 44-513] HW Target shutdown. Closing target: localhost:3121/xilinx_tcf/Xilinx/XXXXXXXXX

2) shutdown the window of hardware manager itself

I cannot get any log for this.

I tried D32 pin to fix '0' like following link.
https://www.xilinx.com/support/answers/72926.html
ILA and VIO's input clk is memory clock for HBM or MAC clock for CMAC(100G MAC).

So, I am a bit suspicious for these clocks to stop only for a moment.
I have two U280 cards, and I happened this error on both cards.

Any idea? Anybody who encountered these problem?

Best Regards,
toku1938

[mcertosi]

Hi @toku1938 

You want to open the hardware manager after the xclbin is programmed onto the card. 

You can use this video as a reference. 

https://www.xilinx.com/video/software/hardware-debug-of-sdaccel-kernel-interfaces.html

Can you let me know if you've tried this flow and are still running into problems? 

Regards,

M

[toku1938]

Hi @mcertosi ,

Thank you for your quick response!

But, I am developing with RTL-based in Vivado 2019.2, not SDAccel.
So, I basically programmed bit file and probe file(ltx) on hardware manager directly.

I am not sure how often this happens during using ILA/VIO.
I was able to see ILA and VAIO in a while, but suddenly the errors happened.

Do you have any ideas for solving this problem ?

Best Regards,
toku1938

[mcertosi]

Hi @toku1938 

Hmm... that shouldn't happen unless something is going so wrong that the FPGA is momentarily powering down or losing clocks.

Can you check to make sure the clock connected to your debug hub is a free running clock that never disappears? An example of a bad clock to use would be a GT clock. 

This doesn't happen to me when I am using ILAs/VIOs, is there an error or warning message? Does the hardware manager crash? 

Regards,

M

[toku1938]

Hi  @mcertosi ,

Thank you for your feedback.
I tried the simple code which uses free running clock for ILA as you advised.

However, I have still encountered the same problem
like "ERROR: [Labtoolstcl 44-513] HW Target shutdown. Closing target: localhost:3121/xilinx_tcf/Xilinx/XXXXXXXXXX"

The following code is used for the experiment.
Anything incorrect ??
Any idea for this?

Best Regards,
toku1938

module top (
        input sysclk0_p,
        input sysclk0_n,
        output D32_pin
);

// From https://www.xilinx.com/support/answers/72926.html
assign D32_pin = 1'b0;

/* clock infrastracture */
wire sysclk_100;
IBUFDS IBUFDS_sysclk_300 (
        .I(sysclk0_p),
        .IB(sysclk0_n),
        .O(sysclk_100)
);

wire baseclk;
BUFG bufg_baseclk (
        .I  (sysclk_100),
        .O  (baseclk)
);

reg [13:0] cold_counter = 14'd0;
reg sys_rst;
always @ (posedge baseclk) begin
        if (cold_counter != 14'h3fff) begin
                cold_counter <= cold_counter + 14'd1;
                sys_rst <= 1'b1;
        end else begin
                sys_rst <= 1'b0;
        end
end

reg [31:0] sys_time;
wire probe = sys_time[10];

always @ (posedge baseclk) begin
        if (sys_rst) begin
                sys_time <= 0;
        end else begin
                sys_time <= sys_time + 1;
        end
end

ila_0 u_ila_0 (
        .probe0 (cold_counter),
        .probe1 (sys_time),
        .probe2 (probe),
        .clk    (baseclk)
);

endmodule

set_property CONFIG_VOLTAGE 1.8 [current_design]

# SYSCLK
set_property PACKAGE_PIN BJ44 [get_ports sysclk0_n]
set_property IOSTANDARD LVDS [get_ports sysclk0_n]
set_property PACKAGE_PIN BJ43 [get_ports sysclk0_p]
set_property IOSTANDARD LVDS [get_ports sysclk0_p]
create_clock -period  10.000 -name sysclk0 [get_ports sysclk0_p]
set_clock_groups -asynchronous -group [get_clocks sysclk0 -include_generated_clocks]

set_property PACKAGE_PIN D32 [get_ports D32_pin]
set_property IOSTANDARD LVCMOS18 [get_ports STAT_D32_pin]

[mcertosi]

Hi @toku1938 

The clocking looks correct now. 

How are you connecting to the hardware manager? You can either be using a JTAG cable or using XVC - pcie virtual jtag cable. 

Can you open up Vivado and show me the power report for this project? 

Are you using the AUX power connection for the u280? 

Regards,

M

[toku1938]

Dear @mcertosi ,

Thank you for your quickly response.

I opened the hardware manager via USB-JTAG. 
For the power, I use AUX power.

The AU280 which I use is an active cooling card.

Regarding the power report, is the following report  enough?

Best Regards,
toku1938

Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.
---------------------------------------------------------------------------------------------------------------------------------
| Tool Version     : Vivado v.2019.2 (lin64) Build 2700185 Thu Oct 24 18:45:48 MDT 2019
| Date             : Tue Mar 10 17:34:12 2020
| Host             : host running 64-bit Ubuntu 18.04.3 LTS
| Command          : report_power -file /home/aaa/work/validate-u280/boards/au280/build/power_power_1.txt -name power_2
| Design           : top
| Device           : xcu280-fsvh2892-2L-e
| Design State     : routed
| Grade            : extended
| Process          : typical
| Characterization : Production
---------------------------------------------------------------------------------------------------------------------------------

Power Report

Table of Contents
-----------------
1. Summary
1.1 On-Chip Components
1.2 Power Supply Summary
1.3 Confidence Level
2. Settings
2.1 Environment
2.2 Clock Constraints
3. Detailed Reports
3.1 By Hierarchy

1. Summary
----------

+--------------------------+---------------+
| Total On-Chip Power (W)  | 3.142         |
|   FPGA Power (W)         | 2.902         |
|   HBM Power (W)          | 0.241         |
| Design Power Budget (W)  | 160.000       |
| Power Budget Margin (W)  | 156.858 (MET) |
| Dynamic (W)              | 0.013         |
| Device Static (W)        | 3.129         |
| Effective TJA (C/W)      | 0.4           |
| Max Ambient (C)          | 98.6          |
| Junction Temperature (C) | 26.4          |
| Confidence Level         | Medium        |
| Setting File             | ---           |
| Simulation Activity File | ---           |
| Design Nets Matched      | NA            |
+--------------------------+---------------+


1.1 On-Chip Components
----------------------

+--------------------------+-----------+----------+-----------+-----------------+
| On-Chip                  | Power (W) | Used     | Available | Utilization (%) |
+--------------------------+-----------+----------+-----------+-----------------+
| Clocks                   |     0.007 |        4 |       --- |             --- |
| CLB Logic                |     0.001 |     3790 |       --- |             --- |
|   LUT as Logic           |    <0.001 |     1005 |   1303680 |            0.08 |
|   LUT as Shift Register  |    <0.001 |      110 |    600960 |            0.02 |
|   Register               |    <0.001 |     2075 |   2607360 |            0.08 |
|   LUT as Distributed RAM |    <0.001 |       32 |    600960 |           <0.01 |
|   CARRY8                 |    <0.001 |       26 |    162960 |            0.02 |
|   Others                 |     0.000 |      230 |       --- |             --- |
|   F7/F8 Muxes            |     0.000 |        3 |   1303680 |           <0.01 |
| Signals                  |    <0.001 |     2816 |       --- |             --- |
| Block RAM                |     0.001 |      1.5 |      2016 |            0.07 |
| I/O                      |     0.004 |        7 |       624 |            1.12 |
| Static Power             |     3.129 |          |           |                 |
| Total                    |     3.142 |          |           |                 |
+--------------------------+-----------+----------+-----------+-----------------+


1.2 Power Supply Summary
------------------------

+------------+-------------+-----------+-------------+------------+
| Source     | Voltage (V) | Total (A) | Dynamic (A) | Static (A) |
+------------+-------------+-----------+-------------+------------+
| Vccint     |       0.850 |     1.274 |       0.011 |      1.263 |
| Vccint_io  |       0.850 |     0.238 |       0.000 |      0.237 |
| Vccbram    |       0.850 |     0.030 |       0.000 |      0.030 |
| Vccaux     |       1.800 |     0.822 |       0.000 |      0.822 |
| Vccaux_io  |       1.800 |     0.037 |       0.002 |      0.035 |
| Vcco33     |       3.300 |     0.000 |       0.000 |      0.000 |
| Vcco25     |       2.500 |     0.000 |       0.000 |      0.000 |
| Vcco18     |       1.800 |     0.000 |       0.000 |      0.000 |
| Vcco15     |       1.500 |     0.000 |       0.000 |      0.000 |
| Vcco135    |       1.350 |     0.000 |       0.000 |      0.000 |
| Vcco12     |       1.200 |     0.000 |       0.000 |      0.000 |
| Vcco10     |       1.000 |     0.000 |       0.000 |      0.000 |
| Vccadc     |       1.800 |     0.024 |       0.000 |      0.024 |
| VCC_IO_HBM |       1.200 |     0.082 |       0.000 |      0.082 |
| VCC_HBM    |       1.200 |     0.076 |       0.000 |      0.076 |
| VCCAUX_HBM |       2.500 |     0.022 |       0.000 |      0.022 |
| MGTYAVcc   |       0.900 |     0.000 |       0.000 |      0.000 |
| MGTYAVtt   |       1.200 |     0.000 |       0.000 |      0.000 |
| MGTYVccaux |       1.800 |     0.000 |       0.000 |      0.000 |
+------------+-------------+-----------+-------------+------------+


1.3 Confidence Level
--------------------

+-----------------------------+------------+------------------------------------------------+------------------------------------------------------------------------------------------------------------+
| User Input Data             | Confidence | Details                                        | Action                                                                                                     |
+-----------------------------+------------+------------------------------------------------+------------------------------------------------------------------------------------------------------------+
| Design implementation state | High       | Design is routed                               |                                                                                                            |
| Clock nodes activity        | High       | User specified more than 95% of clocks         |                                                                                                            |
| I/O nodes activity          | High       | User specified more than 95% of inputs         |                                                                                                            |
| Internal nodes activity     | Medium     | User specified less than 25% of internal nodes | Provide missing internal nodes activity with simulation results or by editing the "By Resource Type" views |
| Device models               | High       | Device models are Production                   |                                                                                                            |
|                             |            |                                                |                                                                                                            |
| Overall confidence level    | Medium     |                                                |                                                                                                            |
+-----------------------------+------------+------------------------------------------------+------------------------------------------------------------------------------------------------------------+


2. Settings
-----------

2.1 Environment
---------------

+-----------------------+--------------------------+
| Ambient Temp (C)      | 25.0                     |
| ThetaJA (C/W)         | 0.4                      |
| Airflow (LFM)         | 250                      |
| Heat Sink             | medium (Medium Profile)  |
| ThetaSA (C/W)         | 0.5                      |
| Board Selection       | medium (10"x10")         |
| # of Board Layers     | 12to15 (12 to 15 Layers) |
| Board Temperature (C) | 25.0                     |
+-----------------------+--------------------------+


2.2 Clock Constraints
---------------------

+-----------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+-----------------+
| Clock                                                                                               | Domain                                                            | Constraint (ns) |
+-----------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+-----------------+
| dbg_hub/inst/BSCANID.u_xsdbm_id/SWITCH_N_EXT_BSCAN.bscan_inst/SERIES7_BSCAN.bscan_inst/INTERNAL_TCK | dbg_hub/inst/BSCANID.u_xsdbm_id/SWITCH_N_EXT_BSCAN.bscan_inst/in0 |            50.0 |
| sysclk0                                                                                             | sysclk0_p                                                         |            10.0 |
+-----------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+-----------------+


3. Detailed Reports
-------------------

3.1 By Hierarchy
----------------

+--------------------------+-----------+
| Name                     | Power (W) |
+--------------------------+-----------+
| top                      |     0.013 |
|   IBUFDS_sysclk_300      |     0.002 |
|   dbg_hub                |     0.002 |
|     inst                 |     0.002 |
|       BSCANID.u_xsdbm_id |     0.002 |
|   u_ila_0                |     0.007 |
|     inst                 |     0.007 |
|       ila_core_inst      |     0.007 |
+--------------------------+-----------+

[mcertosi]

Hi @toku1938 

I need to collect the JTAG logs for this behavior so I can send them to the hardware manager engineering team. 

To enable JTAG logging, start the hardware server with this command exec hw_server –Lhwserver.log –ljtag2,slave 

Please also make a copy of the TCL console output and send it to us as well. 

Regards,

M