How to know the accuracy of quantised model using Deephi tool

[anz162112]

Hi,

I am using the decent tool for quantisation of Resnet model to 8 bit int.

The ouput is generated as deploy.prototxt and deploy.caffemodel.

I want to know the accuracy of the quantised model like the screenshot attached.

To see the accuracy, the user guide says that in decent tool, I need to set the auto_test option to TRUE.

It also says that I need to provide the test dataset to know the accuracy. Can you tell me where to set the path for the test dataset or how to provide test dataset to model so that it can provide me accuracy for quantised model?

[anz162112]

Hi,

I am using the decent tool for quantisation of Resnet model to 8 bit int.

The ouput is generated as deploy.prototxt and deploy.caffemodel.

I want to know the accuracy of the quantised model like the screenshot attached.

[anz162112]

To see the accuracy, the user guide says that in decent tool, I need to set the auto_test option to TRUE.

[anz162112]

It also says that I need to provide the test dataset to know the accuracy. Can you tell me where to set the path for the test dataset or how to provide test dataset to model so that it can provide me accuracy for quantised model?

[qhall · Accepted solution]

@anz162112This tutorial illustrates that the calibration images directory is actually a parameter which is set in the float.prototxt file. 

Ultra96_ML_Embedded_Workshop

This example also uses the auto_test function.  You should be able to refer to the ResNet example as a guide if you have either a ZCU102, 104 or Ultra96 as a starting point, and have loaded the latest stretch image and board files from the Developer Hub:

https://www.xilinx.com/products/design-tools/ai-inference/ai-developer-hub.html#edge

--Quenton

[anz162112]

I could get this working using the link you mentioned.

Thanks

Shikha Goel

[bmanoj19]

Hello,

I followed step specified in the below link to check the accuracy for decent [quantization]

https://github.com/jimheaton/Ultra96_ML_Embedded_Workshop

Without auto_test, we are able to generate deploy.prototxt and deploy.caffemodel.

With auto_test, we are facing error, Please find the attachment. Kindly provide the solution 

Regards

Manoj

[qhall]

@bmanoj19You will need to add a data layer for TEST as shown:

name: "ResNet-50"
layer {
name: "data"
type: "ImageData"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
mirror: true
crop_size: 224
mean_value: 104
mean_value: 107
mean_value: 123
}
image_data_param {
source: "./data/imagenet_val/calibration.txt"
root_folder: "./data/imagenet_val/calibration_images/"
batch_size: 10
shuffle: false
}
}
layer {
name: "data"
type: "ImageData"
top: "data"
top: "label"
include {
phase: TEST
}
transform_param {
mirror: false
crop_size: 224
mean_value: 104
mean_value: 107
mean_value: 123
}
image_data_param {
source: "./data/imagenet_val/calibration.txt"
root_folder: "./data/imagenet_val/calibration_images/"
batch_size: 20
shuffle: false
}
}
layer {
name: "conv1"
type: "Convolution"

.............

--Quenton

[bmanoj19]

I could able to run and get the accuracy for resent50. Thank you for Inputs

Regards

Manoj B