UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Visitor m.walz
Visitor
1,323 Views
Registered: ‎02-08-2018

SSD support in DNNC

Hi,

 

I try to compile classic SSD+VGG16 with DNNDK 2.06 beta. The slides and also the User-Guide claim to have "One-click compilation support" for SSD and VGG16.

However when I try to compile it I get:


[DNNC][Error] Unrecognized layer type [Normalize], Maybe you can delete it in deploy.prototxt and try again.

Is it planned to add support for the Normalize-Layer in future releases? Or is there some kind of replacement?

 

Best regards

0 Kudos
10 Replies
Xilinx Employee
Xilinx Employee
1,236 Views
Registered: ‎02-18-2013

Re: SSD support in DNNC

Hi m.Walz, 

 

  We made some modification with the original caffe-trained SSD network as follows,

 

  a) Change the normalize layer to BatchNorm + Scale in the train.prototxt and test.prototxt, then retraining or finetune.

  b) Remove the MultiBoxLoss layer in  train.prototxt before Decent. 

  c) For the convenience of accuracy test, train.protxt and test.prototxt are merged to train_test.prototxt. 

      - Copy train.prototxt to train_test.prototxt

      - Copy & paste the first layer of test.prototxt (AnnonateData layer) after the data layer of train_test.prototxt

      - Copy & paste the last 5 layers starting from mbox_conf_reshape layer from test.prototxt to the end of train_test.prototxt,

         then add the following parameter in these layers,

         include {

         phase: TEST

         }

 

  I hope this answers your question.

 

Regards,

Andy

0 Kudos
Visitor m.walz
Visitor
1,113 Views
Registered: ‎02-08-2018

Re: SSD support in DNNC

Hey Andy,

thank you very much for your efforts. Sorry for my late reply, I was really stuck with other projects.

I tried your changes, but there are still issues:

1) With c), you refer to the accuracy test in decent during quantization? I still get "Calibration iter: 1/100 ,loss: 0" here. Could you please post a train_test.prototxt for reference?

2) After decent tool, I get a deplay.prototxt/caffemodel and a fix_train_test.prototxt/caffemodel. When I try to feed them into dnnc, I get those messages:

deploy.prototxt
are you kidding me? I spent 24 hours to dig such bugs, please don't do such stupid things ever again.12 bits cannot represent value larger than 4095, but 6143 is given
But probably deploy.prototxt is intended to be the pruned network and should not be fed into dnnc, isn't it?


fix_train_test.prototxt:
Message type "caffe.FixedParameter" has no field named "follow_data_layer".
But there is a follow_data_layer:true parameter in the generated FixedNeuron layer.
When I remove the follow_data_layer parameter, I get
Check failed for condition [layer_id < model.layer_size()] in [/tmp/DNNDK_All_V010_Package/dnnc/src/parser/caffe/caffeparser.cc:129] :Failed to find parameter for [mbox_conf_reshape] in caffemodel, make sure name in prototxt is consistent with that in caffemodel.And when I remove those layers, I get
Invalid caffemodel, fixinfo for layers are missing.

It is a quite difficult for me to "debug" here. Could you please provide some basic example for SSD (or a modified version without normalize etc)? A tutorial similar to the resnet50 tutorial would be really great!

 

 

0 Kudos
Highlighted
Visitor lixun123
Visitor
953 Views
Registered: ‎09-29-2018

Re: SSD support in DNNC

Hi,

    Could you post me a train_test.prototxt file? I also need it.My Email:daichong@hrbeu.edu.cn

Thanks!

@m.walz@aluo

0 Kudos
Xilinx Employee
Xilinx Employee
901 Views
Registered: ‎02-18-2013

Re: SSD support in DNNC

@m.walzPls refer to the tutorial below, 

1) Download the pre-trained VGG_ILSVRC_16_layers_fc_reduced_deploy.prototxt (description) and VGG_ILSVRC_16_layers_fc_reduced.caffemodel (weights) files of the VGG network. These will be used
as the backbone (feature extraction layers) for our SSD network and since they are pre-trained, it will help to reduce training time.
1.a) Copy these files into the $CAFFE_ROOT/models/VGGNet/ folder.
2) Download the PASCAL VOC dataset from the following three links:
http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
Now create a folder named $CAFFE_ROOT/data/VOC0712 and copy the 3 tar files above mentioned to that destination. Extract all files from the 3 archives and merge them under a single VOCdevkit folder. At the end of this process you should have a new folder named $CAFFE_ROOT/data/VOC0712/VOCdevkit which contains the VOC2007 and VOC2012 subfolders. Note that this dataset contains the training, validation, and testing images as well as the annotations which are the bounding box locations contained in .xml format.
3) For the next step, we will be processing the annotations (labels) and the training images into LMDB files which can then be directly used by Caffe for the training process.
For this step, you need two shell scripts create_list.sh and create_data.sh which are available from this repository https://github.com/intel/caffe/tree/master/data/VOC0712. I have already downloaded these for you and placed them into this subfolder dataset_files/voc0712 along with one other file named labelmap_voc.prototxt.
Note that I have already edited the two shell scripts in order to point to the $CAFFE_ROOT/data/VOC0712 subfolder such that the first command in the scripts is as follows:
export DATAPATH=$CAFFE_ROOT/data/VOC0712
Since this uses the environment variable previously set for $CAFFE_ROOT, you can launch them from any folder in your Linux file system, though in the next instruction, we'll copy them to a specific directory to make the instructions easy.
Copy all these 3 files into $CAFFE_ROOT/data/VOC0712. If such directories do not exist you have to create them with the mkdir Linux command.
4) Execute the following commands: and change directory to go there:
cd $CAFFE_ROOT/data/VOC0712
source create_list.sh
source create_data.sh
cp -r examples/* ../../examples
At the end of it you should see something similar to what illustrated in this screenshot (my local directories might be different from yours):
Note that a new directory named examples has been created together with 3 txt files, as illustrated in the below screenshot:
Note also that this process has created in $CAFFE_ROOT/examples/VOC0712 two soft links to the real LMDB databases actually stored in $CAFFE_ROOT/data/VOC/VOCdevkit/VOC0712/lmdb, respectively of size ~1.8GB the training validation database and 445MB the test database, as shown in this screenshot:
At this point, the training dataset has been prepared and is ready for use to train the SSD model and you can proceed to the next step which is training the model.
3.0 Training the SSD model
3.0 PART 1: Preparing the SSD Prototxt Files
1) Open with a text editor the file $CAFFE_ROOT/examples/ssd/ssd_pascal.py and change line 332 to provide the ID numbers for GPUs which are available for training. For more than one GPU, the line would read gpus = "0,1,2,etc..."
gpus = "0"
Notice that this python script adds layers to the existing VGGNet: these layers are the SSD framework layers, while the VGGNet that we downloaded previously will be the feature extraction layers or “backbone”.
Also note that this python script will creates solver.prototxt, deploy.prototxt, test.prototxt and train.prototxt files for the SSD network with the all necessary parameters. These will all be created under $CAFFE_ROOT/models/VGGNet/VOC0712/SSD_300x300. Have a look at the prototxt files and note that they all assume to have the LDMB database placed in $CAFFE_ROOT/examples/VOC0712/VOC0712_test_lmdb folder, therefore the previous steps in dataset preparation ensure that you have some soft links that point to the real place into you might have placed the database when you created it in step (described by ssd2_VOC_dataset.md).
To create the prototxt files and start the training, use the command:
cd $CAFFE_ROOT
python examples/ssd/ssd_pascal.py
Output snapshots will be created incrementally under the following directory: $CAFFE_ROOT/models/VGGNet/VOC0712/SSD_300x300, though at this point you will need to halt the training process early on by entering CTRL+C in the terminal. The reason to halt the training process is that we now need to make some manual modifications to the prototxt files for compatibility with the DPU and DNNDK tools.
3.0 PART 2: Modifying the SSD Prototxt Files for Compatibility with the DPU/DNNDK
1) At this point you should have prototxt and solver files under $CAFFE_ROOT/jobs/VGGNet/VOC0712/SSD_300x300. At this point, you need to make a copy of the "train.prototxt" and rename it to "train_test.protoxt"
2) Next open the test.prototxt file and copy the first layer (named “data”) to train_test.prototxt as the second layer. Make sure to omit copying the 1st line of the test.prototxt which is the name. You should be inserting this layer between lines 137 and 138 of the "train_test.prototxt" file.
3) Next copy the last 5 layers from "test.prototxt" (starting at layer named "mbox_conf_reshape" which should be around line 1591, ending at the last layer) and append them to the end of "train_test.prototxt"
4) Next, you need to add a parameter to the mbox_conf_reshape, mbox_conf_softmax, and mbox_conf_flatten layers that were just added to ensure these layers are only used for testing. The entry that needs to be added is as follows:
include{
phase: TEST
}
This can be inserted in each of the layers just after the top: declaration and before the layers other parameters.
5) In the "train_test.prototxt" we now need to replace the Normalize layer with BatchNorm and Scale layers because the DPU does not support Normalize. In the original SSD model, this is called "conv4_3_norm". The following steps can be used to replace the layer:
5.a) Delete the Normalize layer named "conv4_3_norm" which should start at around line 972 and end around line 985
5.b) Insert the Batchnorm layer and scale layer between "conv4_3" and "relu4_3" layers
5.c) Replace the other bottom layers in the prototxt named "conv4_3_norm" with "conv4_3" (there should be something like three of these) and they appear as bottom: "conv4_3_norm".
The following illustration should help with these modifications:
6) Next modify the "fc6" layer by changing the num_output to 682 instead of 1024. The reason for this is that the DPU only supports 12 bits to describe the number of parameters. With a dilation of 6, the number of output parameters will be 6143 which is too many for the DPU to capture.
Note that I have included a pre-modified version of the train_test.prototxt for your reference.
7) Next open the "solver.prototxt" which exists in this same directory. This file has the training hyper-parameters and also points to the net which is being trained. In this file, delete the first two lines:
train_net: "models/VGGNet/VOC0712/SSD_300x300/train.prototxt"
test_net: "models/VGGNet/VOC0712/SSD_300x300/test.prototxt"
and replace these with the following line:
net: "jobs/VGGNet/VOC0712/SSD_300x300/train_test.prototxt"
Notice the snapshot_prefix location which is where the training will produce output models. This can be modified as desired by the user to place the snapshot output model in a desired directory, though for this tutorial, I recommend leaving the snapshots at this location with the recommended prefix as it will allow for easier evaluation of the floating point model without modifying other scripts.
snapshot_prefix: "models/VGGNet/VOC0712/SSD_300x300/VGG_VOC0712_SSD_300x300"
The number/frequency of snapshots are determined based on the parameters specified in the "solver.prototxt" and note that by default the first snapshot will occur after 80K iterations.
8) Next open the "VGG_VOC0712_SSD_300x300.sh" script in a text editor and modify the --solver path to point to the solver.prototxt file in the jobs directory.
--solver="jobs/VGGNet/VOC0712/SSD_300x300/solver.prototxt"
9) Now cd to $CAFFE_ROOT and run the following:
./jobs/VGGNet/VOC0712/SSD_300x300/VGG_VOC0712_SSD_300x300.sh
This will call the ./build/tools/caffe train command pointing to the newly created "train_test.prototxt" file.

 

0 Kudos
Xilinx Employee
Xilinx Employee
883 Views
Registered: ‎02-17-2011

Re: SSD support in DNNC

Attached is a reference train_test.prototxt for SSD using VGG feature leayers after the modifications mentioned by @aluo.  Please refer to @aluo's post for detailed instructions.

 

 

 

0 Kudos
Visitor lixun123
Visitor
829 Views
Registered: ‎09-29-2018

Re: SSD support in DNNC

      Hi! Thank you for your help,I still have some problems, I really need your help. I modified the train_test.prototxt file according to the method you mentioned above and then I trained an SSD model. This model has only two labels: background and motorboat. I tested the caffemodel on the computer and it was OK.
     Before decent, I removed the MultiBox Loss layer from train_test. prototxt,but when I run decent.sh, the calibrated loss is always 0, as shown in Figure 1,I don't know if this is a normal phenomenon or what the problem is. There still generated four files in the decent_output folder, I continue to run dnnc.sh, then I can get dpu_ssd.elf. When running on the board(ZCU104), a lot of boxes appear in the screen, but the target motorboat can not be marked out,as shown in Figure 2.
     Below is my train_test.prototxt, deploy.prototxt(generated by dencent.sh) ,main.cc,decent.sh,dnnc.sh. Can you help me solve the problem? Thank you very much! @aluo @jcory

1.png2.jpg

0 Kudos
Xilinx Employee
Xilinx Employee
749 Views
Registered: ‎02-17-2011

Re: SSD support in DNNC

 Hi @lixun123,

What was the input resolution you used for your SSD model?  Was it 480x360 or other?  The example software that comes with the DNNDK image assumes 480x360.  If you are using a resolution other than 480x360 for your SSD caffemodel, then I expect that all of your priorboxes (CreatePriors()) will need to be update in the software.

For example, here is a version that has been updated from the 480x360 model to 300x300:


void CreatePriors(vector<shared_ptr<vector<float>>> *priors) {
vector<float> variances{0.1, 0.1, 0.2, 0.2};
vector<PriorBoxes> prior_boxes;

prior_boxes.emplace_back(PriorBoxes{
300, 300, 38, 38, variances, {30}, {60}, {2}, 0.5, 8.0, 8.0});
prior_boxes.emplace_back(PriorBoxes{
300, 300, 19, 19, variances, {60.0}, {111.0}, {2, 3}, 0.5, 16, 16});
prior_boxes.emplace_back(PriorBoxes{
300, 300, 10, 10, variances, {111.0}, {162.0}, {2, 3}, 0.5, 32, 32});
prior_boxes.emplace_back(PriorBoxes{
300, 300, 5, 5, variances, {162.0}, {213.0}, {2, 3}, 0.5, 64, 64});
prior_boxes.emplace_back(PriorBoxes{
300, 300, 3, 3, variances, {213.0}, {264.0}, {2}, 0.5, 100, 100});
prior_boxes.emplace_back(PriorBoxes{
300, 300, 1, 1, variances, {264.0}, {315.0}, {2}, 0.5, 300, 300});


int num_priors = 0;
for (auto &p : prior_boxes) {
num_priors += p.priors().size();
}

priors->clear();
priors->reserve(num_priors);
for (auto i = 0U; i < prior_boxes.size(); ++i) {
priors->insert(priors->end(), prior_boxes[i].priors().begin(),
prior_boxes[i].priors().end());
}
}

 

Note that you will also likely need to change the doImg() function to match your reduced number of classes.

 

Thanks and hope this helps!

JC

0 Kudos
Visitor lixun123
Visitor
702 Views
Registered: ‎09-29-2018

Re: SSD support in DNNC

Thank you for your reply, I have solved this problem based on the help you provided.

0 Kudos
Visitor lixun123
Visitor
675 Views
Registered: ‎09-29-2018

Re: SSD support in DNNC

Sorry, I still have problems and need your help. I reposted a new post https://forums.xilinx.com/t5/Deephi-DNNDK/Crashed-when-running-SSD-video-analysis/td-p/926185 Can you help me? Thank you!


@jcory @aluo
0 Kudos
461 Views
Registered: ‎01-17-2019

Re: SSD support in DNNC

0 Kudos