cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
Contributor
Contributor
1,047 Views
Registered: ‎07-05-2018

Quantisation with different bit widths - DECENT tool

Hi,

I am quantizing the resnet model to 8 bit int using Decent tool.

I have few questions regarding the quantisation.

1. Can we have 8 bit fixed point quantisation instead of 8 bit int using Decent tool.

2. Also, I tried quantising the resnet model to  16 bit int. The output is generated as deploy.prototxt and deploy.caffemodel. The problem is that when I read weights in deploy.caffemodel, they are still float 32 bit. What does this means? I supposed the weights to be int 16 bit. Also, when I see the size of input caffemodel(float.caffemodel) and output caffemodel(deploy.caffemodel), both are exactly of same size i.e. 102.1 MB.

I want to use this quantised caffemodel(8 bit or 16 bit) as input for some network but the model generated by decent is still 32 bit float. What is the problem in this case? Is this the tool problem?

Regards,
Shikha Goel
(Ph.D. , IIT Delhi)
0 Kudos
Reply
10 Replies
Highlighted
Explorer
Explorer
978 Views
Registered: ‎10-24-2008

@anz162112 

Decent only supports INT8 quantization.  INT16 is not supported by Decent, nor by the DPU.  Does this help?

--Quenton

0 Kudos
Reply
Highlighted
Contributor
Contributor
966 Views
Registered: ‎07-05-2018

Actually user guide says that it supports quantisation to any bit width and we can set it in decent using data_bit and weights_bit.

 

Thanks

Shikha

Regards,
Shikha Goel
(Ph.D. , IIT Delhi)
0 Kudos
Reply
Highlighted
Explorer
Explorer
956 Views
Registered: ‎10-24-2008

@anz162112Yes, I don't disagree.  And it does seem likely that the quantizer was designed to be parameterizable.  However, only INT8 is supported for deployment today. 

--Quenton

0 Kudos
Reply
Highlighted
Contributor
Contributor
944 Views
Registered: ‎07-05-2018

Thank you. I got your point.

Regards,
Shikha Goel
(Ph.D. , IIT Delhi)
0 Kudos
Reply
Highlighted
Contributor
Contributor
902 Views
Registered: ‎03-05-2019

Hi,

What about the size of the float.caffemodel and the deploy.caffemodel? As far as I understood deploy.caffemodel should be smaller in size, no? All the 32 bit weights have been reduced to 8 bit, so the file size should be smaller. It remains the same for me.

Best regards

Arthur

0 Kudos
Reply
Highlighted
Contributor
Contributor
891 Views
Registered: ‎07-05-2018

Inside Decent, 32-bit float point values are used to represent quantized values, thats why the output files of decent are still float and are of the same size of the float models.

Regards,
Shikha Goel
(Ph.D. , IIT Delhi)
0 Kudos
Reply
Highlighted
Contributor
Contributor
889 Views
Registered: ‎03-05-2019

Thank you very much!

0 Kudos
Reply
Highlighted
Adventurer
Adventurer
711 Views
Registered: ‎04-10-2019

I'm using DNNDK v3.0 which was released after this thread. I tried running DECENT with 12-bit widths. The tool didn't complain, but my network didn't behave at all properly.

Are different bit widths supposed to be supported in the current DECENT? If not, any idea when this support will be available?

0 Kudos
Reply
Highlighted
Adventurer
Adventurer
699 Views
Registered: ‎04-10-2019

I tried the quantization using both methods (non-overflow and min-diffs). They behave differently. One does better on some images and the other on different ones. But neither gets me close to the performance I get with my Caffe model.

It looks like adding bits (especially if I can specify how many bits after the decimal point) would be the most likely way to improve the performance. Is there any way with DNNDK v3.0 to do this?

0 Kudos
Reply
Highlighted
686 Views
Registered: ‎05-17-2019

So I have quantized a custom network with 12 bits of precission and then compiled it with dnnc. I have also compiled my hybrid executable application. There is nowhere a warning that the parameters of my model are 8 bits.

As I read above, the actual model that is used has 8 bit precission?

If arbitrary bit width is not supported for deployment, at least state it somewhere in the user guide. My network is useless with 8 bits and I have lost a week trying to learn the DNNDK framework.

Thanks.

0 Kudos
Reply