cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Observer
Observer
313 Views
Registered: ‎09-26-2018

VCU1525: MIG calibration timeout after bitstream download

Hi,

I get the "calibrate_mig: MIG calibration timeout after bitstream download" Error after executing

 python run.py --validate --output_dir work --numBatches 10

from the ml-suite example here 

The exact error message is

loading /opt/ml-suite/overlaybins/1525/overlay_4.xclbin

ERROR: Failed to load xclbin. Timeout, see dmesg for details
ERROR: Failed to create compute program from binary -44
Traceback (most recent call last):
File "run.py", line 139, in <module>
Infer(args["output_dir"]+"/xfdnn_auto_cut_train_val.prototxt",args["output_dir"]+"/deploy.caffemodel",args["numBatches"])
File "run.py", line 80, in Infer
net = caffe.Net(prototxt,caffemodel,caffe.TEST)
File "/opt/ml-suite/xfdnn/rt/scripts/framework/caffe/CaffeXFDNN.py", line 23, in setup
raise Exception("Failed to open FPGA handle.")
Exception: Failed to open FPGA handle.

This is the output from xbutil query:

INFO: Found total 1 card(s), 1 are usable
DSA name 
xilinx_vcu1525_dynamic_5_1

Vendor Device SubDevice SubVendor XMC fw version 
10ee 6a8f 4351 10ee 2017403

DDR size DDR count OCL Frequency Clock0 Clock1 
64 GB 4 300 MHz 350 MHz

PCIe DMA bi-directional threads MIG Calibrated 
GEN 3x16 2 false

###############################################################################
Power 
Not support

PCB TOP FRONT PCB TOP REAR PCB BTM FRONT 
Not support Not support Not support

FPGA Temp TCRIT Temp Fan Speed 
31 C Not support Not support

12V PEX 12V AUX 12V PEX Current 12V AUX Current 
Not support Not support Not support Not support

3V3 PEX 3V3 AUX DDR VPP BOTTOM DDR VPP TOP 
Not support Not support Not support Not support

SYS 5V5 1V2 TOP 1V8 TOP 0V85 
Not support Not support Not support Not support

MGT 0V9 12V SW MGT VTT 
Not support Not support Not support

VCCINT VOL VCCINT CURR 
Not support Not support

###############################################################################
Firewall Last Error Status:
Level 0: 0x0 (GOOD)

###############################################################################
Xclbin ID: 
0x5b921d51

Compute Unit Status:
CU[0]: kernelSxdnn_0@0x1800000 (--)
CU[1]: kernelSxdnn_0@0x1810000 (--)

###############################################################################
Mem Topology Device Memory Usage 
Tag Type Temp Size Mem Usage BO nums 
[0] bank0 **UNUSED** Not Supp 16 GB 0 Byte 0 
[1] bank2 MEM_DDR4 Not Supp 16 GB 0 Byte 0 
[2] bank3 **UNUSED** Not Supp 16 GB 0 Byte 0 
[3] bank1 MEM_DDR4 Not Supp 16 GB 0 Byte 0

Total DMA Transfer Metrics:
Chan[0].h2c: 0 Byte
Chan[0].c2h: 0 Byte
Chan[1].h2c: 0 Byte
Chan[1].c2h: 0 Byte

INFO: xbutil query succeeded.

This is the output from dmesg:

[584454.595670] xocl_xdma 0000:05:00.0: xocl_mmap: io_remap_pfn_range ret code: 0
[584454.595684] xclmgmt 0000:05:00.1: char_open: opened file 00000000af1817f8 by pid: 4205
[584454.707238] icap.m icap.m.1281: icap_download_bitstream_axlf: copy-in headers, num sections: 6, size: 736
[584454.707253] icap.m icap.m.1281: icap_download_bitstream_axlf: incoming xclbin ID: 000000005c92e7c7, on device xclbin ID:0000000000000000
[584454.707255] icap.m icap.m.1281: icap_download_bitstream_axlf: finding CLOCK_FREQ_TOPOLOGY section
[584454.707258] icap.m icap.m.1281: get_axlf_section: trying to find section header for axlf section 11
[584454.707260] icap.m icap.m.1281: get_axlf_section: saw section header: 0
[584454.707261] icap.m icap.m.1281: get_axlf_section: saw section header: 2
[584454.707263] icap.m icap.m.1281: get_axlf_section: saw section header: 6
[584454.707265] icap.m icap.m.1281: get_axlf_section: saw section header: 7
[584454.707266] icap.m icap.m.1281: get_axlf_section: saw section header: 8
[584454.707268] icap.m icap.m.1281: get_axlf_section: saw section header: 11
[584454.707270] icap.m icap.m.1281: get_axlf_section: header offset: 67083824, size: 274
[584454.707275] icap.m icap.m.1281: axlf_set_freqscaling: setting clock freq, num: 4, data_freq: 300 , clk_freq: 350, sys_freq[0]: 0, sys_freq[1]: 0
[584454.707279] icap.m icap.m.1281: icap_freeze_axi_gate: freezing AXI gate
[584454.707302] icap.m icap.m.1281: icap_free_axi_gate: freeing AXI gate
[584454.707338] icap.m icap.m.1281: reset_scheduler: calling xocl_reset_scheduler
[584454.707358] icap.m icap.m.1281: icap_download_bitstream_axlf: finding ip layout sections
[584454.707360] icap.m icap.m.1281: get_axlf_section: trying to find section header for axlf section 8
[584454.707362] icap.m icap.m.1281: get_axlf_section: saw section header: 0
[584454.707364] icap.m icap.m.1281: get_axlf_section: saw section header: 2
[584454.707366] icap.m icap.m.1281: get_axlf_section: saw section header: 6
[584454.707367] icap.m icap.m.1281: get_axlf_section: saw section header: 7
[584454.707369] icap.m icap.m.1281: get_axlf_section: saw section header: 8
[584454.707371] icap.m icap.m.1281: get_axlf_section: header offset: 67083656, size: 168
[584454.707400] icap.m icap.m.1281: icap_download_bitstream_axlf: finding bitstream sections
[584454.707402] icap.m icap.m.1281: get_axlf_section: trying to find section header for axlf section 0
[584454.707403] icap.m icap.m.1281: get_axlf_section: saw section header: 0
[584454.707405] icap.m icap.m.1281: get_axlf_section: header offset: 696, size: 67075654
[584454.707407] icap.m icap.m.1281: get_axlf_section: trying to find section header for axlf section 1
[584454.707409] icap.m icap.m.1281: get_axlf_section: saw section header: 0
[584454.707411] icap.m icap.m.1281: get_axlf_section: saw section header: 2
[584454.707413] icap.m icap.m.1281: get_axlf_section: saw section header: 6
[584454.707414] icap.m icap.m.1281: get_axlf_section: saw section header: 7
[584454.707416] icap.m icap.m.1281: get_axlf_section: saw section header: 8
[584454.707418] icap.m icap.m.1281: get_axlf_section: saw section header: 11
[584454.707420] icap.m icap.m.1281: get_axlf_section: could not find section header 1
[584454.707422] icap.m icap.m.1281: icap_download_user: downloading bitstream, length: 67075654
[584454.707424] icap.m icap.m.1281: icap_freeze_axi_gate: freezing AXI gate
[584454.707439] icap.m icap.m.1281: icap_download_clear_bitstream: downloading clear bitstream of length 0x0
[584454.707443] icap.m icap.m.1281: bitstream_parse_header: Design "pfm_top_wrapper;COMPRESS=TRUE;PARTIAL=TRUE;UserID=0XFFFFFFFF;Version=2018.2"
[584454.707444] icap.m icap.m.1281: bitstream_parse_header: Part "xcvu9p-fsgd2104-2-i"
[584454.707446] icap.m icap.m.1281: bitstream_parse_header: Timestamp "18:22:53 2019/03/20"
[584454.707448] icap.m icap.m.1281: bitstream_parse_header: Raw data size 0x3ff7db4
[584454.778336] audit: type=1400 audit(1564125851.218:447771): apparmor="DENIED" operation="file_mmap" profile="snap.nextcloud.nextcloud-fixer" name="/" pid=4356 comm="php" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
[584456.076680] audit: type=1400 audit(1564125852.514:447772): apparmor="DENIED" operation="file_mmap" profile="snap.nextcloud.nextcloud-fixer" name="/" pid=4421 comm="php" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
[584457.057572] icap.m icap.m.1281: wait_for_done: XHWICAP_SR: 5
[584457.089236] icap.m icap.m.1281: icap_ocl_freqscaling: clockwiz waiting for locked signal
[584457.197372] icap.m icap.m.1281: icap_ocl_freqscaling: clockwiz CONFIG(0) 0xc01
[584457.197376] icap.m icap.m.1281: icap_ocl_freqscaling: clockwiz CONFIG(2) 0x4
[584457.229242] icap.m icap.m.1281: icap_ocl_freqscaling: clockwiz waiting for locked signal
[584457.337425] icap.m icap.m.1281: icap_ocl_freqscaling: clockwiz CONFIG(0) 0xe01
[584457.337429] icap.m icap.m.1281: icap_ocl_freqscaling: clockwiz CONFIG(2) 0x4
[584457.337432] icap.m icap.m.1281: icap_free_axi_gate: freeing AXI gate
[584457.337475] icap.m icap.m.1281: reset_scheduler: calling xocl_reset_scheduler
[584457.384211] audit: type=1400 audit(1564125853.822:447773): apparmor="DENIED" operation="file_mmap" profile="snap.nextcloud.nextcloud-fixer" name="/" pid=4487 comm="php" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
[584458.692881] audit: type=1400 audit(1564125855.130:447774): apparmor="DENIED" operation="file_mmap" profile="snap.nextcloud.nextcloud-fixer" name="/" pid=4553 comm="php" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
[584459.992396] audit: type=1400 audit(1564125856.430:447775): apparmor="DENIED" operation="file_mmap" profile="snap.nextcloud.nextcloud-fixer" name="/" pid=4618 comm="php" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
[584461.301683] audit: type=1400 audit(1564125857.742:447776): apparmor="DENIED" operation="file_mmap" profile="snap.nextcloud.nextcloud-fixer" name="/" pid=4683 comm="php" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
[584462.465310] icap.m icap.m.1281: calibrate_mig: MIG calibration timeout after bitstream download
[584462.465331] icap.m icap.m.1281: icap_download_bitstream_axlf: icap_download_bitstream_axlf err: -110

Any help or hint is highly appriciated

0 Kudos