cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
IgnacioA
Visitor
Visitor
211 Views
Registered: ‎04-18-2021

Vitis AI Quantization with YOLOv4-tiny

Jump to solution

I am trying to quantize a YOLOv4-tiny model converted to Caffe. The YOLO cfg and weights have already been converted to .prototxt and .caffemodel respectively. The network inicialization and iterations complete, but i get an error. The process will be listed below.

> vai_q_caffe quantize -model yolov4.prototxt -weights yolov4.caffemodel -calib_iter 50

after calibration iterations i get:

> I0418 19:42:23.903064 406 vai_q.cpp:360] Start Deploy
terminate called after throwing an instance of 'std::out_of_range'
what(): map::at
Aborted (core dumped)

what could be the error here? i tried looking for the vai_q.cpp file within the Vitis-AI main folder but i could not find it anywhere.

0 Kudos
1 Solution

Accepted Solutions
jheaton
Xilinx Employee
Xilinx Employee
124 Views
Registered: ‎03-21-2008

I recommend that you use the Darknet to Caffe Flow outlined in part 2 of the tutorial: https://github.com/Xilinx/Vitis-Tutorials/tree/master/Machine_Learning/Design_Tutorials/07-yolov4-tutorial.  The tutorial is for yolov4, but you should be able to use the same procedure for tiny yolo. 

There is a problem with your config file that you will need to fix first. The 3 route layers layers with group ids are going to cause a problem because they wont be supported on the DPU hw, and will end up having to be run on the CPU. This is going to kill your performance. 

What you will want to do is comment out each of these route layers in cfg file and use a conv instead, and then retrain in darknet.

For example change:

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky


To:

#[route]
#layers=-1
#groups=2
#group_id=1
[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

 

View solution in original post

0 Kudos
4 Replies
zhipengl
Xilinx Employee
Xilinx Employee
175 Views
Registered: ‎03-21-2021

Hello,

"vai_q.cpp" is vai_q_caffe.cpp 

May be some problem with darknet2caffe, Could you please tell us how you converte "route" in yolov4-tiny to caffe?

 

0 Kudos
IgnacioA
Visitor
Visitor
151 Views
Registered: ‎04-18-2021

Apologies, yes, after searching i found the vai_q_caffe.cpp. This still could not reveal the source of my error.

 

To convert they YOLOv4-tiny to Caffe model I used the Vitis AI Docker program built on CPU. I ran the program with:

> ./docker_run.sh xilinx/vitis-ai-cpu:latest

> conda activate vitis-ai-caffe

Now I added the ImageData layer for model training, with the calibration.txt such as the documentation on the official Xilinx User Documentation for Vitis AI

> python convert.py yolov4-tiny-capstone.cfg yolov4-tiny-capstone_best.weights caffe_converted/yolov4.prototxt caffe_converted/yolov4.caffemodel

After running this line, i get the yolov4.prototxt and the yolov4.caffemodel inside the caffe_converted directory

Then I run:

> vai_q_caffe quantize -model yolov4.prototxt -weights yolov4.caffemodel -calib_iter 50

With this line, I get the quantize_train_test.prototxt and quantize_train_test.caffemodel inside the quantization directory.

When I run the deploy command is when i get the error

> vai_q_caffe deploy -model yolov4.prototxt -weights yolov4.caffemodel

> I0418 19:42:23.903064 406 vai_q.cpp:360] Start Deploy
terminate called after throwing an instance of 'std::out_of_range'
what(): map::at
Aborted (core dumped)

### Here is my original CFG file for YOLOv4-tiny ###

[net]
# Testing
#batch=1
#subdivisions=1
# Training
batch=32
subdivisions=2
width=224
height=224
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.00261
burn_in=1000
max_batches = 6000
policy=steps
steps=4800,5400
scales=.1,.1

[convolutional]
batch_normalize=1
filters=32
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky

[route]
layers = -1,-2

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky

[route]
layers = -6,-1

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[route]
layers = -1,-2

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[route]
layers = -6,-1

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[route]
layers = -1,-2

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[route]
layers = -6,-1

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

##################################

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=24
activation=linear

 

[yolo]
mask = 3,4,5
anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319
classes=3
num=6
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=0
resize=1.5
nms_kind=greedynms
beta_nms=0.6

[route]
layers = -4

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1, 23

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=24
activation=linear

[yolo]
mask = 0,1,2
anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319
classes=3
num=6
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=0
resize=1.5
nms_kind=greedynms
beta_nms=0.6

#### Here is my yolov4.prototxt ####

name: "Darkent2Caffe"
#input: "data"
#input_dim: 1
#input_dim: 3
#input_dim: 224
#input_dim: 224

layer {
name: "data"
type: "ImageData"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
mirror: false
yolo_height: 224
yolo_width: 224
}

image_data_param {
source:"/workspace/caffe_converted/quantization/calibration_images/calib.txt"
root_folder:"/workspace/caffe_converted/quantization/calibration_images/"
batch_size: 10
shuffle: false
}
}

layer {
bottom: "data"
top: "layer0-conv"
name: "layer0-conv"
type: "Convolution"
convolution_param {
num_output: 32
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer0-conv"
top: "layer0-conv"
name: "layer0-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer0-conv"
top: "layer0-conv"
name: "layer0-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer0-conv"
top: "layer1-conv"
name: "layer1-conv"
type: "Convolution"
convolution_param {
num_output: 64
kernel_size: 3
pad: 1
stride: 2
bias_term: false
}
}
layer {
bottom: "layer1-conv"
top: "layer1-conv"
name: "layer1-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer1-conv"
top: "layer1-conv"
name: "layer1-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer1-conv"
top: "layer2-conv"
name: "layer2-conv"
type: "Convolution"
convolution_param {
num_output: 64
kernel_size: 3
pad: 0
stride: 1
bias_term: false
}
}
layer {
bottom: "layer2-conv"
top: "layer2-conv"
name: "layer2-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer2-conv"
top: "layer2-conv"
name: "layer2-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer2-conv"
name: "layer3-slice"
type: "Slice"
top: "layer3-slice_aborted"
top: "layer3-slice"
slice_param {
axis: 1
}
}
layer {
bottom: "layer3-slice"
top: "layer4-conv"
name: "layer4-conv"
type: "Convolution"
convolution_param {
num_output: 32
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer4-conv"
top: "layer4-conv"
name: "layer4-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer4-conv"
top: "layer4-conv"
name: "layer4-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer4-conv"
top: "layer5-conv"
name: "layer5-conv"
type: "Convolution"
convolution_param {
num_output: 32
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer5-conv"
top: "layer5-conv"
name: "layer5-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer5-conv"
top: "layer5-conv"
name: "layer5-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer5-conv"
bottom: "layer4-conv"
top: "layer6-concat"
name: "layer6-concat"
type: "Concat"
}
layer {
bottom: "layer6-concat"
top: "layer7-conv"
name: "layer7-conv"
type: "Convolution"
convolution_param {
num_output: 64
kernel_size: 1
pad: 0
stride: 1
bias_term: false
}
}
layer {
bottom: "layer7-conv"
top: "layer7-conv"
name: "layer7-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer7-conv"
top: "layer7-conv"
name: "layer7-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer2-conv"
bottom: "layer7-conv"
top: "layer8-concat"
name: "layer8-concat"
type: "Concat"
}
layer {
bottom: "layer8-concat"
top: "layer9-maxpool"
name: "layer9-maxpool"
type: "Pooling"
pooling_param {
kernel_size: 2
stride: 2
pool: MAX
}
}
layer {
bottom: "layer9-maxpool"
top: "layer10-conv"
name: "layer10-conv"
type: "Convolution"
convolution_param {
num_output: 128
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer10-conv"
top: "layer10-conv"
name: "layer10-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer10-conv"
top: "layer10-conv"
name: "layer10-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer10-conv"
name: "layer11-slice"
type: "Slice"
top: "layer11-slice_aborted"
top: "layer11-slice"
slice_param {
axis: 1
}
}
layer {
bottom: "layer11-slice"
top: "layer12-conv"
name: "layer12-conv"
type: "Convolution"
convolution_param {
num_output: 64
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer12-conv"
top: "layer12-conv"
name: "layer12-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer12-conv"
top: "layer12-conv"
name: "layer12-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer12-conv"
top: "layer13-conv"
name: "layer13-conv"
type: "Convolution"
convolution_param {
num_output: 64
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer13-conv"
top: "layer13-conv"
name: "layer13-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer13-conv"
top: "layer13-conv"
name: "layer13-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer13-conv"
bottom: "layer12-conv"
top: "layer14-concat"
name: "layer14-concat"
type: "Concat"
}
layer {
bottom: "layer14-concat"
top: "layer15-conv"
name: "layer15-conv"
type: "Convolution"
convolution_param {
num_output: 128
kernel_size: 1
pad: 0
stride: 1
bias_term: false
}
}
layer {
bottom: "layer15-conv"
top: "layer15-conv"
name: "layer15-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer15-conv"
top: "layer15-conv"
name: "layer15-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer10-conv"
bottom: "layer15-conv"
top: "layer16-concat"
name: "layer16-concat"
type: "Concat"
}
layer {
bottom: "layer16-concat"
top: "layer17-maxpool"
name: "layer17-maxpool"
type: "Pooling"
pooling_param {
kernel_size: 2
stride: 2
pool: MAX
}
}
layer {
bottom: "layer17-maxpool"
top: "layer18-conv"
name: "layer18-conv"
type: "Convolution"
convolution_param {
num_output: 256
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer18-conv"
top: "layer18-conv"
name: "layer18-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer18-conv"
top: "layer18-conv"
name: "layer18-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer18-conv"
name: "layer19-slice"
type: "Slice"
top: "layer19-slice_aborted"
top: "layer19-slice"
slice_param {
axis: 1
}
}
layer {
bottom: "layer19-slice"
top: "layer20-conv"
name: "layer20-conv"
type: "Convolution"
convolution_param {
num_output: 128
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer20-conv"
top: "layer20-conv"
name: "layer20-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer20-conv"
top: "layer20-conv"
name: "layer20-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer20-conv"
top: "layer21-conv"
name: "layer21-conv"
type: "Convolution"
convolution_param {
num_output: 128
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer21-conv"
top: "layer21-conv"
name: "layer21-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer21-conv"
top: "layer21-conv"
name: "layer21-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer21-conv"
bottom: "layer20-conv"
top: "layer22-concat"
name: "layer22-concat"
type: "Concat"
}
layer {
bottom: "layer22-concat"
top: "layer23-conv"
name: "layer23-conv"
type: "Convolution"
convolution_param {
num_output: 256
kernel_size: 1
pad: 0
stride: 1
bias_term: false
}
}
layer {
bottom: "layer23-conv"
top: "layer23-conv"
name: "layer23-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer23-conv"
top: "layer23-conv"
name: "layer23-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer18-conv"
bottom: "layer23-conv"
top: "layer24-concat"
name: "layer24-concat"
type: "Concat"
}
layer {
bottom: "layer24-concat"
top: "layer25-maxpool"
name: "layer25-maxpool"
type: "Pooling"
pooling_param {
kernel_size: 2
stride: 2
pool: MAX
}
}
layer {
bottom: "layer25-maxpool"
top: "layer26-conv"
name: "layer26-conv"
type: "Convolution"
convolution_param {
num_output: 512
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer26-conv"
top: "layer26-conv"
name: "layer26-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer26-conv"
top: "layer26-conv"
name: "layer26-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer26-conv"
top: "layer27-conv"
name: "layer27-conv"
type: "Convolution"
convolution_param {
num_output: 256
kernel_size: 1
pad: 0
stride: 1
bias_term: false
}
}
layer {
bottom: "layer27-conv"
top: "layer27-conv"
name: "layer27-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer27-conv"
top: "layer27-conv"
name: "layer27-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer27-conv"
top: "layer28-conv"
name: "layer28-conv"
type: "Convolution"
convolution_param {
num_output: 512
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer28-conv"
top: "layer28-conv"
name: "layer28-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer28-conv"
top: "layer28-conv"
name: "layer28-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer28-conv"
top: "layer29-conv"
name: "layer29-conv"
type: "Convolution"
convolution_param {
num_output: 24
kernel_size: 1
pad: 0
stride: 1
bias_term: true
}
}
layer {
bottom: "layer27-conv"
top: "layer32-conv"
name: "layer32-conv"
type: "Convolution"
convolution_param {
num_output: 128
kernel_size: 1
pad: 0
stride: 1
bias_term: false
}
}
layer {
bottom: "layer32-conv"
top: "layer32-conv"
name: "layer32-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer32-conv"
top: "layer32-conv"
name: "layer32-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer32-conv"
top: "layer33-upsample"
name: "layer33-upsample"
type: "DeephiResize"
deephi_resize_param {
scale_h: 2
scale_w: 2
}
}
layer {
bottom: "layer33-upsample"
bottom: "layer23-conv"
top: "layer34-concat"
name: "layer34-concat"
type: "Concat"
}
layer {
bottom: "layer34-concat"
top: "layer35-conv"
name: "layer35-conv"
type: "Convolution"
convolution_param {
num_output: 256
kernel_size: 3
pad: 1
stride: 1
bias_term: false
}
}
layer {
bottom: "layer35-conv"
top: "layer35-conv"
name: "layer35-bn"
type: "BatchNorm"
batch_norm_param {
use_global_stats: true
}
}
layer {
bottom: "layer35-conv"
top: "layer35-conv"
name: "layer35-act"
type: "ReLU"
relu_param {
negative_slope: 0.1
}
}
layer {
bottom: "layer35-conv"
top: "layer36-conv"
name: "layer36-conv"
type: "Convolution"
convolution_param {
num_output: 24
kernel_size: 1
pad: 0
stride: 1
bias_term: true
}
}

0 Kudos
jheaton
Xilinx Employee
Xilinx Employee
125 Views
Registered: ‎03-21-2008

I recommend that you use the Darknet to Caffe Flow outlined in part 2 of the tutorial: https://github.com/Xilinx/Vitis-Tutorials/tree/master/Machine_Learning/Design_Tutorials/07-yolov4-tutorial.  The tutorial is for yolov4, but you should be able to use the same procedure for tiny yolo. 

There is a problem with your config file that you will need to fix first. The 3 route layers layers with group ids are going to cause a problem because they wont be supported on the DPU hw, and will end up having to be run on the CPU. This is going to kill your performance. 

What you will want to do is comment out each of these route layers in cfg file and use a conv instead, and then retrain in darknet.

For example change:

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky


To:

#[route]
#layers=-1
#groups=2
#group_id=1
[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

 

View solution in original post

0 Kudos
IgnacioA
Visitor
Visitor
86 Views
Registered: ‎04-18-2021

Thank you all for your replies. I have found my solution, and thank you jheaton for the advice, i took it and it worked well.

0 Kudos