An error about DNNC

[jsi_wmz]

I met this error when use DNNC to generate the model ELF.

[...]
[DNNC][DEBUG] Generate dpu instruction for node [layer96_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer100_conv].
[DNNC][DEBUG] Generate dpu instruction for node [layer97_conv].
dnnc-dpu1.4.0: /tmp/DNNC_V010_Package/dnnc/submodules/asicv2com/src/SlNode/SlNodeConv.cpp:82: void SlNodeConv::generate_convinit_op(const YAggregationType&, const YAggregationType&, uint32_t, uint32_t): Assertion `shift_cut >= 0' failed.
3_compile.sh: line 9: 28800 Aborted                 (core dumped) dnnc-dpu1.4.0 --prototxt=3_model_after_quantize/deploy.prototxt --caffemodel=3_model_after_quantize/deploy.caffemodel --dpu=4096FA --cpu_arch=arm64 --output_dir=4_model_elf --net_name=yolov3_changed --mode=debug --save_kernel --dump=graph

Actually, I have met this error several times before though it was solved by different ways. However, I couldn't find out the cause this time. I upgraded my dnndk from 0531 version to 0624 version but it didn't work. And some other models won't meet this problem. So I just want to know if anyone can tell me what the error EXACTLY means or the causes of the error. I am really confused about it because I didn't see any explanation about the ERROR itself like the meaning of "Assertion shift_cut >= 0 "

SO, please help me

[jcory]

Hi @ jsi_wmz,
In my experience, the "Assertion shift_cut >= 0 " error tends to relate to poor quantization results. I would verify that the calibration samples applied to the model are of good quality (i.e. they represent the training dataset well) and are of sufficient quantity (try increasing the number of samples to 1000).
I would also recommend adding the -auto_test option to decent along with all of the necessary layers in the prototxt for testing the model and observe the quantized model accuracy during these experiments to see if it increases.
If the accuracy of the quantized model is still poor, I would try increasing the -data_bit and -weights_bit to 32 and re-run quantization to see if the accuracy improves. If this improves things, you could try ignoring certain layers such as batchnorm layers during quantization (using the -ignore_layers argument) to try to isolate which layers are an issue.
I have found in recent model developments that batchnorm layers can be a culprit for poor quantization results. Specifically, using combining the batchnorm layers with scale filler parameters seems to produce very good quantized model results whereas keeping BatchNorm and Scale as separate layers or allowing BatchNorm to be standalone (without Scale) tend to produce poor results for some models.
I've included an example below, and if I believe nvcaffe may support the combined BatchNorm with Scale.
e.g. combined BatchNorm with Scale (good quantization results):
layer {
name: "BatchNorm_d0c"
type: "BatchNorm"
bottom: "d0c"
top: "BatchNorm_d0c"
param {
lr_mult: 1
decay_mult: 0
}
param {
lr_mult: 1
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
batch_norm_param {
use_global_stats: true
eps: 1e-05
scale_filler {
type: "constant"
value: 1
}
bias_filler {
type: "constant"
value: 0
}
}
}

e.g. Separate Batchnorm and Scale example:

layer {
name: "BatchNorm_d0c"
type: "BatchNorm"
bottom: "d0c"
top: "BatchNorm_d0c"
batch_norm_param {
use_global_stats: false
eps: 0.0010000000475
}
}
layer {
name: "BatchNorm_d0c_scale"
type: "Scale"
bottom: "BatchNorm_d0c"
top: "BatchNorm_d0c"
scale_param {
bias_term: true
}
}

Thanks and hope this helps,
-JC

[jsi_wmz]

@jcory 

I added parameters "-auto_test" & "-test_iter". And I met a new error which didn't occur without the parameters.

layer {
  name: "layer47-conv"
  type: "ConvolutionFixed"
  bottom: "layer46-shortcut"
  top: "layer47-
F0713 22:06:39.002722 12861 insert_splits.cpp:76] Unknown bottom blob 'data' (layer 'data_fixed', bottom index 0)
*** Check failure stack trace: ***
2_quantize.sh: line 8: 12861 Aborted                 (core dumped) decent-cpu quantize -model 2_model_for_quantize/v3.prototxt -weights 2_model_for_quantize/v3.caffemodel -sigmoided_layers layer81-conv,layer85-conv,layer97-conv,layer101-conv -auto_test -test_iter 50 -output_dir 3_model_after_quantize -method 1

I don't know what happened after I added the params. The only difference is the two params for calibration test.

[jsi_wmz]

@jcory 

I also has a question about the input shape. The model which has that error has an input that height is not equal to width.

So i wonder if the shape may lead to the problem?

[nittax]

Hi, I also have the exact same issue as @jsi_wmz 

layer {
  name: "layer47-conv"
  type: "ConvolutionFixed"
  bottom: "layer46-shortcut"
  top: "layer47-
F0819 02:21:23.565826  2794 insert_splits.cpp:76] Unknown bottom blob 'data' (layer 'data_fixed', bottom index 0)
*** Check failure stack trace: ***
2_quantize.sh: 10 行:  2794 中止

Have you resolve this issue? How did you solve this issue? 

Thanks in advance.

[lihahagang]

Actually, I want to know the meaning and usage of the parameters which is list on DECENT in the UG1327.@jcory Could you give me some advice? 

Thanks for your assistance.

[jcory]

Hi @jsi_wmz ,

The model height does not need to be equal to model width, so this should not be an issue.  For SSD, however, there are implications if you change the input size as all of the priorboxes will need to change as well.

Thanks,

-JC

[jcory]

Hi @lihahagang,

Please start a new forum post on this and also specify what questions you have.  The user guide does list the explanation of each parameter for decent so let us know what portions may be confusing.

Thanks,

-JC