UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Contributor
Contributor
292 Views
Registered: ‎05-31-2019

Linux Kernel seems to get stuck on running an application [ZCU104]

Hi all,

I am using a ZCU104 board to test an accelerator. I use Vivado (2019.1)to create the .bit and .hdf files. After this, I am using the .hdf file to generate a petalinux (2019.1) project and from there I am getting the BOOT.BIN and image.ub files. Now I am copying these files to an SD card and putting it on the board, it boots up successfully. However, when I run the application to test some loadable files it just gets stuck after some time:

root@nvdla:/media# ./nvdla_runtime --loadable regression/flatbufs/kmd/CONV/CONV_D_L0_0_small_fbuf 
creating new runtime context...
Emulator starting
submitting tasks...
[ 53.416154] Enter:dla_read_network_config
[ 53.421918] Exit:dla_read_network_config status=0
[ 53.426615] Enter: dla_initiate_processors
[ 53.430703] Enter: dla_submit_operation
[ 53.434530] Prepare Convolution operation index 0 ROI 0 dep_count 1
[ 53.440794] Enter: dla_prepare_operation
[ 53.444714] processor:Convolution group:0, rdma_group:0 available
[ 53.450802] Enter: dla_read_config
[ 53.454206] Exit: dla_read_config
[ 53.457506] Exit: dla_prepare_operation status=0
[ 53.462114] Enter: dla_program_operation
[ 53.466028] Program Convolution operation index 0 ROI 0 Group[0]
[ 53.472078] no desc get due to index==-1
[ 53.475991] no desc get due to index==-1
[ 53.479911] no desc get due to index==-1
[ 53.483829] no desc get due to index==-1
[ 53.487744] no desc get due to index==-1
[ 53.491657] no desc get due to index==-1
[ 53.495574] Enter: dla_op_programmed
[ 53.499142] Update dependency operation index 1 ROI 0 DEP_COUNT=1
[ 53.505227] enable SDP in dla_update_dependency as depdency are resolved
[ 53.511918] Enter: dla_enable_operation
[ 53.515752] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[ 53.524792] Exit: dla_enable_operation status=0
[ 53.529314] Exit: dla_op_programmed
[ 53.532794] Exit: dla_program_operation status=0
[ 53.537405] Exit: dla_submit_operation
[ 53.541153] Enter: dla_dequeue_operation
[ 53.545067] exit Convolution as there's no further operation
[ 53.550718] Exit: dla_dequeue_operation
[ 53.554547] Enter: dla_submit_operation
[ 53.558374] Prepare SDP operation index 1 ROI 0 dep_count 0
[ 53.563938] Enter: dla_prepare_operation
[ 53.567855] processor:SDP group:0, rdma_group:0 available
[ 53.573243] Enter: dla_read_config
[ 53.576644] Exit: dla_read_config
[ 53.579954] Exit: dla_prepare_operation status=0
[ 53.584561] Enter: dla_program_operation
[ 53.588478] Program SDP operation index 1 ROI 0 Group[0]
[ 53.593799] no desc get due to index==-1
[ 53.597712] no desc get due to index==-1
[ 53.601625] no desc get due to index==-1
[ 53.605535] no desc get due to index==-1
[ 53.609447] no desc get due to index==-1
[ 53.613361] no desc get due to index==-1
[ 53.617277] Enter: dla_op_programmed
[ 53.620844] Exit: dla_op_programmed
[ 53.624325] Exit: dla_program_operation status=0
[ 53.628933] Enter: dla_enable_operation
[ 53.632763] Enable SDP operation index 1 ROI 0
[ 53.637198] Enter: dla_op_enabled
[ 53.640506] Update dependency operation index 0 ROI 0 DEP_COUNT=1
[ 53.646589] enable Convolution in dla_update_dependency as depdency are resolved
[ 53.653976] Enter: dla_enable_operation
[ 53.657804] Enable Convolution operation index 0 ROI 0
[ 53.662941] Enter: dla_op_enabled
[ 53.666252] Exit: dla_op_enabled
[ 53.669463] Exit: dla_enable_operation status=0
[ 53.673984] Exit: dla_op_enabled
[ 53.677197] Exit: dla_enable_operation status=0
[ 53.681717] Exit: dla_submit_operation
[ 53.685452] Enter: dla_dequeue_operation
[ 53.689364] exit SDP as there's no further operation
[ 53.694319] Exit: dla_dequeue_operation
[ 53.698149] Exit: dla_initiate_processors status=0

--stall here--

I have modified the device tree to use 1GB of memory and my DRAM is of size 2GB. Besides this, I have also lowered the clock frequency of the zynq IP in Vivado. I am not sure about any other possible workarounds for this. I know it is tough without looking at the application but If anyone can provide any suggestions/ideas as to why the kernel generally gets stuck that will be really helpful. Thank you.

0 Kudos
7 Replies
Scholar watari
Scholar
285 Views
Registered: ‎06-16-2013

Re: Linux Kernel seems to get stuck on running an application [ZCU104]

Hi @maju42 

 

Can you observe free memory size by using top command or cat /proc/meminfo ?

 

Best regards,

0 Kudos
Contributor
Contributor
281 Views
Registered: ‎05-31-2019

Re: Linux Kernel seems to get stuck on running an application [ZCU104]

Hi @watari ,

This is what I get:

root@nvdla:~# cat /proc/meminfo 
MemTotal:        1014664 kB
MemFree:          953948 kB
MemAvailable:     932152 kB
Buffers:            2780 kB
Cached:            14264 kB
SwapCached:            0 kB
Active:            14744 kB
Inactive:           7128 kB
Active(anon):       4888 kB
Inactive(anon):      140 kB
Active(file):       9856 kB
Inactive(file):     6988 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                12 kB
Writeback:             0 kB
AnonPages:          4688 kB
Mapped:             5608 kB
Shmem:               200 kB
Slab:              25940 kB
SReclaimable:       7136 kB
SUnreclaim:        18804 kB
KernelStack:        1616 kB
PageTables:          324 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      507332 kB
Committed_AS:      32448 kB
VmallocTotal:   263061440 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:              720 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:         262144 kB
CmaFree:          261464 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
0 Kudos
Scholar watari
Scholar
270 Views
Registered: ‎06-16-2013

Re: Linux Kernel seems to get stuck on running an application [ZCU104]

Hi @maju42 

 

Hmm. It's not free memory issue.

Would you make sure cpu usage by using top command or a couple of times of "cat proc/stat" ?

 

Best regards,

0 Kudos
Contributor
Contributor
267 Views
Registered: ‎05-31-2019

Re: Linux Kernel seems to get stuck on running an application [ZCU104]

Hi @watari ,

Here's the output:

root@nvdla:~# cat /proc/stat 
cpu 111 0 508 20016 132 0 1 0 0 0
cpu0 22 0 189 4856 43 0 1 0 0 0
cpu1 48 0 184 4977 12 0 0 0 0 0
cpu2 21 0 93 5058 32 0 0 0 0 0
cpu3 18 0 41 5123 43 0 0 0 0 0
intr 16219 0 0 0 8446 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 45 0 0 0 1451 104 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 27383
btime 1572306132
processes 2299
procs_running 1
procs_blocked 0
softirq 17963 2 7241 0 0 0 0 787 4956 0 4977


root@nvdla:~# cat /proc/stat
cpu 111 0 508 27264 132 0 1 0 0 0
cpu0 22 0 190 6668 43 0 1 0 0 0
cpu1 48 0 184 6789 12 0 0 0 0 0
cpu2 21 0 93 6871 32 0 0 0 0 0
cpu3 18 0 41 6935 43 0 0 0 0 0
intr 17470 0 0 0 9609 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 45 0 0 0 1451 116 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 29744
btime 1572306132
processes 2300
procs_running 1
procs_blocked 0
softirq 19862 2 8037 0 0 0 0 787 5732 0 5304
0 Kudos
Scholar watari
Scholar
246 Views
Registered: ‎06-16-2013

Re: Linux Kernel seems to get stuck on running an application [ZCU104]

Hi @maju42 

 

Hmm. It's not cpu usage issue, if you got thes results (current and previous results) with running your application.

 

It is hard to investigate the route cause without detail information...

 

Best regards,

0 Kudos
Contributor
Contributor
244 Views
Registered: ‎05-31-2019

Re: Linux Kernel seems to get stuck on running an application [ZCU104]

Hi @watari,

Yea I understand, I was thinking in what situations the kernel gets stuck mostly? If I can find some possible causes for that, then I can decide how to debug the application. 

0 Kudos
Scholar watari
Scholar
210 Views
Registered: ‎06-16-2013

Re: Linux Kernel seems to get stuck on running an application [ZCU104]

Hi @maju42 

 

Have you ever used gprof ?

If no, would you try it to investigate the route cuase ?

 

https://sourceware.org/binutils/docs/gprof/

https://en.wikipedia.org/wiki/Gprof

 

Best regards,

0 Kudos