cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Participant
Participant
576 Views
Registered: ‎01-23-2019

[ DNNDK 3.1 ] DPU timeout while execute DPU Task after Swap DPU from B2304FA to B3136FA

Dear community,

Need some help on fixing this error

Error Message: [DNNDK] DPU timeout while execute DPU Task:yolo-2

The main change I did was swapping from a smaller DPU B2304FA to a bigger DPU B3136FA and turning on some switches,

see below, for switches values, before using the bigger DPU(B3136)  all the performances optimization switches were turned off.

Was already using same DPU IP, and Same DNNDK3.1 as decribded below,

Interrupt pin was always connected properly. 

=====================

dexplorer -w

=======================

[DPU IP Spec]
IP Timestamp : 2019-11-21 17:30:00
DPU Core Count : 1

[DPU Core Configuration List]
DPU Core : #0
DPU Enabled : Yes
DPU Arch : B3136
DPU Target Version : v1.4.0

==============================================

dexplorer -v

==============================================

DNNDK version 3.1
Copyright @ 2018-2019 Xilinx Inc. All Rights Reserved.

DExplorer version 2.0
Build Label: Aug 7 2019 18:24:23

DSight version 1.4
Build Label: Aug 7 2019 18:24:23

DDump version 1.0
Build Label: Aug 7 2019 18:24:23

N2Cube Core library version 3.0
Build Label: Aug 7 2019 18:24:18

DPU Driver version 3.0.0
Build Label: Nov 25 2019 20:53:22


DPU Freqency : 324 MHz
Ram Usage : High
DepthwiseConv : Disabled
DepthwiseConv+Relu6 : Disabled
Conv+Leakyrelu : Enabled
Conv+Relu6 : Enabled
Channel Augmentation : Enabled
Average Pool : Disabled

 

--
Luc Chouinard
FPGA Architect & AI Specialist
Teledyne Imaging
Teledyne Lumenera
Tags (3)
0 Kudos
9 Replies
Highlighted
Xilinx Employee
Xilinx Employee
552 Views
Registered: ‎02-17-2011

Re: [ DNNDK 3.1 ] DPU timeout while execute DPU Task after Swap DPU from B2304FA to B3136FA

Hi @lucc1,
My apologies as this maybe a bit basic of a question but did you make sure to recompile your model to target the new dcf/DPU arch?
Thanks,
-JC
0 Kudos
Highlighted
Participant
Participant
550 Views
Registered: ‎01-23-2019

Re: [ DNNDK 3.1 ] DPU timeout while execute DPU Task after Swap DPU from B2304FA to B3136FA

Yes I did, Thanks John for the interest, appreciated
--
Luc Chouinard
FPGA Architect & AI Specialist
Teledyne Imaging
Teledyne Lumenera
0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
543 Views
Registered: ‎02-17-2011

Re: [ DNNDK 3.1 ] DPU timeout while execute DPU Task after Swap DPU from B2304FA to B3136FA

Hi @lucc1,
Understood and thanks. I would try adding the -dump=all and the -mode=debug switches to the compilation process. Then you can also enabletaskdump in the software per the DNNDK user guide. These may lend some clues as to which layer is having problems during execution.
Thanks and hope this helps!

-JC
Highlighted
Participant
Participant
526 Views
Registered: ‎01-23-2019

Re: [ DNNDK 3.1 ] DPU timeout while execute DPU Task after Swap DPU from B2304FA to B3136FA

Will try that, @jcory, will have update shortly Thanks
--
Luc Chouinard
FPGA Architect & AI Specialist
Teledyne Imaging
Teledyne Lumenera
0 Kudos
Highlighted
Participant
Participant
511 Views
Registered: ‎01-23-2019

Re: [ DNNDK 3.1 ] DPU timeout while execute DPU Task after Swap DPU from B2304FA to B3136FA

Adding the following switches to compile.sh: 

--mode=debug \
--dump=all \

dnnc-dpu1.4.0 \
--prototxt=3_model_after_quantize/deploy.prototxt \
--caffemodel=3_model_after_quantize/deploy.caffemodel \
--dcf=../../../proj2019.1B/no_ddr_3kdpu_leaky_relu/dpu-11-21-2019-17-30.dcf \
--cpu_arch=arm64 \
--output_dir=4_model_elf_B3136 \
--net_name=yolo \
--mode=debug \
--dump=all \
--save_kernel

#=========================

./3_compile_B3136_with_dbg.sh
[DNNC][DEBUG] Generate dpu instruction for node [layer0_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer2_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer4_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer6_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer8_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer10_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer11_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer12_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer13_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer17_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer14_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer20_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer21_conv].

DNNC Kernel topology "yolo_kernel_graph.jpg" for network "yolo"
DNNC kernel list info for network "yolo"
Kernel ID : Name
0 : yolo

Kernel Name : yolo
--------------------------------------------------------------------------------
Kernel Type : DPUKernel
Code Size : 0.09MB
Param Size : 8.27MB
Workload MACs : 3220.99MOPS
IO Memory Space : 0.69MB
Mean Value : 0, 0, 0,
Node Count : 14
Tensor Count : 16
Input Node(s)(H*W*C)
layer0_conv(0) : 320*320*3
Output Node(s)(H*W*C)
layer14_conv(0) : 10*10*21
layer21_conv(0) : 20*20*21

#=============================================

#Running On the card

#=============================================

[DNNDK] DPU timeout while execute DPU Task [yolo-1] of Node [layer0_conv]

#=============================================

 dmesg | grep DPU
[ 5.947081] [DPU][2066]Found DPU signature addr = 0x8f000000 in device-tree
[ 5.954061] [DPU][2066]Checking DPU signature at addr = 0x8ff00000,
[ 5.960452] [DPU][2066]DPU signature checking done!
[ 830.421245] [DPU][2340][PID 2340][taskID 1]Core 0 Run timeout,failed to get finish interrupt!
[ 830.429781] [DPU][2340][DPU debug info]
[ 830.435877] [DPU][2340]Core 0 schedule counter: 1
[ 830.440662] [DPU][2340]Core 0 interrupt counter: 0
[ 830.445454] [DPU][2340][DPU Registers]
[ 830.449194] [DPU][2340]VER : 0x0d9c13f1
[ 830.453619] [DPU][2340]RST : 0x000000ff
[ 830.458046] [DPU][2340]ISR : 0x00000000
[ 830.462473] [DPU][2340]IMR : 0x00000000
[ 830.466900] [DPU][2340]IRSR : 0x00000000
[ 830.471327] [DPU][2340]ICR : 0x00000000
[ 830.475752] [DPU][2340]
[ 830.478184] [DPU][2340]DPU Core : 0
[ 830.481656] [DPU][2340]HP_CTL : 0x07070f0f
[ 830.485909] [DPU][2340]ADDR_IO : 0x00000000
[ 830.490162] [DPU][2340]ADDR_WEIGHT : 0x00000000
[ 830.494676] [DPU][2340]ADDR_CODE : 0x0006fc60
[ 830.499016] [DPU][2340]ADDR_PROF : 0x00000000
[ 830.503356] [DPU][2340]PROF_VALUE : 0x00000000
[ 830.507783] [DPU][2340]PROF_NUM : 0x00000000
[ 830.512036] [DPU][2340]PROF_EN : 0x00000000
[ 830.516289] [DPU][2340]START : 0x00000001
[ 830.520543] [DPU][2340]COM_ADDR_L0 : 0x6fe00000
[ 830.525057] [DPU][2340]COM_ADDR_H0 : 0x00000000
[ 830.529570] [DPU][2340]COM_ADDR_L1 : 0x70700000
[ 830.534084] [DPU][2340]COM_ADDR_H1 : 0x00000000
[ 830.538597] [DPU][2340]COM_ADDR_L2 : 0x6fc60000
[ 830.543111] [DPU][2340]COM_ADDR_H2 : 0x00000000
[ 830.547625] [DPU][2340]COM_ADDR_L3 : 0x00000000
[ 830.552138] [DPU][2340]COM_ADDR_H3 : 0x00000000
[ 830.556652] [DPU][2340]COM_ADDR_L4 : 0x00000000
[ 830.561166] [DPU][2340]COM_ADDR_H4 : 0x00000000
[ 830.565679] [DPU][2340]COM_ADDR_L5 : 0x00000000
[ 830.570193] [DPU][2340]COM_ADDR_H5 : 0x00000000
[ 830.574706] [DPU][2340]COM_ADDR_L6 : 0x00000000
[ 830.579220] [DPU][2340]COM_ADDR_H6 : 0x00000000
[ 830.583734] [DPU][2340]COM_ADDR_L7 : 0x00000000
[ 830.588248] [DPU][2340]COM_ADDR_H7 : 0x00000000
[ 830.592760] [DPU][2340]

--
Luc Chouinard
FPGA Architect & AI Specialist
Teledyne Imaging
Teledyne Lumenera
0 Kudos
Highlighted
Participant
Participant
500 Views
Registered: ‎01-23-2019

Re: [ DNNDK 3.1 ] DPU timeout while execute DPU Task after Swap DPU from B2304FA to B3136FA

@jcory  (Adding @terryo  )

I am now using the "RAM usage" High parameter

set in Vivado on the DPU parameter setting:

Could this be the problem?

https://forums.xilinx.com/t5/Machine-Learning/dnnc-dpu1-4-0-or-dnnc-dpu1-4-0-1/m-p/989199

 

Although, I am not sure I have the dnnc 1.4.0.1 executable. Luc

--
Luc Chouinard
FPGA Architect & AI Specialist
Teledyne Imaging
Teledyne Lumenera
Highlighted
Xilinx Employee
Xilinx Employee
475 Views
Registered: ‎02-17-2011

Re: [ DNNDK 3.1 ] DPU timeout while execute DPU Task after Swap DPU from B2304FA to B3136FA

Hi @lucc1

Good find - are you actually using DNNDK 3.1?  If so, this issue should be resolved (the install process is just 'sudo ./install.sh' I believe) and you should be passing in a dcf file to dnnc.  Otherwise, if you are using v3.0, I think it would be worth a try to give the other dnnc executible a shot.  It should be under the host_x86/pkgs/ubuntuxx.0x folder.

Thanks,

-JC

 

0 Kudos
Highlighted
Participant
Participant
471 Views
Registered: ‎01-23-2019

Re: [ DNNDK 3.1 ] DPU timeout while execute DPU Task after Swap DPU from B2304FA to B3136FA

@jcory John, as you can see in the title of the post I am using DNNDK3.1,
Also, I am passing the .dcf file as argument to dnnc.
Agree it should be solved, I do not think it is, at least not for me, now.

So:
1- is there another dnndk that was release after xilinx_dnndk_v3.1_190809.tar.gz,
2- is there 2 binaries for dnnc, ie dnnc-dpu1.4.0 and dnnc-dpu1.4.0.1 or only one
3- If there is only one, are we sure that the functionality that used to be in dnnc-dpu1.4.0.1 has been merged in dnnc-dpu1.4.0

Currently in xilinx_dnndk_v3.1_190809.tar.gz there is only one binary: dnnc-dpu1.4.0

--
Luc Chouinard
FPGA Architect & AI Specialist
Teledyne Imaging
Teledyne Lumenera
0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
366 Views
Registered: ‎02-17-2011

Re: [ DNNDK 3.1 ] DPU timeout while execute DPU Task after Swap DPU from B2304FA to B3136FA

Hi @lucc1,
The two versions of dnnc that you mention are in DNNDK v3.0. You have the latest package for dnndk as far as I know. I will reach out to you directly offline regarding your issue.
Thanks,
-JC
0 Kudos