cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
jorge_2015
Adventurer
Adventurer
359 Views
Registered: ‎12-06-2017

dpuRunSoftmax segmentation error

Hi, I am trying to run the code below using 50000 images,

void runMobilenet(DPUTask *taskMobilenet) {
    assert(taskMobilenet);

    /* Mean value for MobileNet specified in Caffe prototxt */
    vector<string> kinds, images;
    /* Load all image names.*/
    ListImages(baseImagePath, images);
    if (images.size() == 0) {
        cerr << "\nError: No images existing under " << baseImagePath << endl;
        return;
    }
    sort(images.begin(), images.end());

    /* Load all kinds words.*/
    /*LoadWords(baseImagePath + "words.txt", kinds);
    if (kinds.size() == 0) {
        cerr << "\nError: No words exist in file words.txt." << endl;
        return;
    }*/

    /* Get the output Tensor for Mobilenet Task  */
    int8_t *outAddr = (int8_t *)dpuGetOutputTensorAddress(taskMobilenet, OUTPUT_NODE);
    /* Get size of the output Tensor for Mobilenet Task  */
    int size = dpuGetOutputTensorSize(taskMobilenet, OUTPUT_NODE);
    /* Get channel count of the output Tensor for MobileNet Task  */
    int channel = dpuGetOutputTensorChannel(taskMobilenet, OUTPUT_NODE);
    /* Get scale of the output Tensor for Mobilenet Task  */
    float out_scale = dpuGetOutputTensorScale(taskMobilenet, OUTPUT_NODE);

    float *softmax = new float[size];
    
    for (auto &imageName : images) {
        /* Load image and Set image into DPU Task for MobileNet */
        Mat image = imread(baseImagePath + imageName);
        vector<float> mean{104, 117, 123};
        float scale = 0.00390625;
        dpuSetInputImageWithScale(taskMobilenet, CONV_INPUT_NODE, image, mean.data(), scale);

        /* Launch Mobilenet Task */
        dpuRunTask(taskMobilenet);

        /* Calculate softmax on DPU and display TOP-5 classification results */
         dpuRunSoftmax(outAddr, softmax, channel, size/channel, out_scale);
    }

    delete[] softmax;
}

And when arrives to the 29190th image I get a segmentation error that I have debugged with gdb:

 

 "mobilenet" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fbe007150 (LWP 4567)]
__memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:170
170     ../sysdeps/aarch64/multiarch/../memcpy.S: No such file or directory.
(gdb) where
#0  __memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:170
#1  0x0000007fbf66a2d0 in dpuRunSoftmax(signed char*, float*, int, int, float) () from /usr/lib/libn2cube.so
#2  0x000000555555b48c in runMobilenet (taskMobilenet=0x7fb8000b20, images=...) at /home/xilinx/jupyter_notebooks/mobilenet/src/main.cc:206
#3  0x000000555555b82c in <lambda()>::operator() (__closure=<optimized out>) at /home/xilinx/jupyter_notebooks/mobilenet/src/main.cc:248
#4  std::__invoke_impl<void, classifyEntry(DPUKernel*)::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#5  std::__invoke<classifyEntry(DPUKernel*)::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#6  std::thread::_Invoker<std::tuple<classifyEntry(DPUKernel*)::<lambda()> > >::_M_invoke<0> (this=<optimized out>) at /usr/include/c++/7/thread:234
#7  std::thread::_Invoker<std::tuple<classifyEntry(DPUKernel*)::<lambda()> > >::operator() (this=<optimized out>) at /usr/include/c++/7/thread:243
#8  std::thread::_State_impl<std::thread::_Invoker<std::tuple<classifyEntry(DPUKernel*)::<lambda()> > > >::_M_run(void) (this=<optimized out>) at /usr/include/c++/7/thread:186
#9  0x0000007fbed61d24 in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#10 0x0000007fbee40088 in start_thread (arg=0x7ffffff0af) at pthread_create.c:463
#11 0x0000007fbebf94ec in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

 

I am seeing that when I add dpuRunSoftmax function, program memory start increasing while if I remove it from code memory remains normal. Is there any kind of problems with dpuRunSoftmax? Or how can I modify the code above?

 

Best regards,

 

Jorge

0 Kudos
Reply
3 Replies
jasonwu
Moderator
Moderator
323 Views
Registered: ‎03-27-2013

Hi @jorge_2015 ,

 

Sounds strange. Could you please try with the CPU caculated Softmax function here?

https://github.com/gewuek/flower_classification_dnndk_v2/blob/master/arm/flower_classification/src/main.cc

Best Regards,
Jason
-----------------------------------------------------------------------------------------------
Please mark the Answer as "Accept as solution" if the information provided is helpful.

Give Kudos to a post which you think is helpful and reply oriented.
-----------------------------------------------------------------------------------------------
0 Kudos
Reply
MichaelK
Contributor
Contributor
257 Views
Registered: ‎06-25-2020

I have the same problem with the dpuRunSoftmax-function. RAM usage keeps steadily increasing until it crashes.

This doesn't happen with something similar to the mentioned CPUSoftmax.

0 Kudos
Reply
MichaelK
Contributor
Contributor
219 Views
Registered: ‎06-25-2020

Any info on this @jasonwu ?
I'm looking around in smaps and the difference between the smaps of cpuSoftmax and dpuSoftmax-function is, that the dpu one has many (increasing with runtime) entries like:

7f6e1c8000-7f6e228000 rw-s 00000000 00:06 934                            /dev/dri/renderD128
Size:                384 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
VmFlags: rd wr sh mr mw me ms pf io de dd 

with increasing addresses and either 96 kB or 384 kB. Any idea how to fix this?

0 Kudos
Reply