i dont know should be in the --calibration_directory. Imagenet dataset is different from the dataset which can be used for yolov2.
In theory , there should be images and labels. Yolov2 is for detection network and you have to draw the boxes of the objects. In darknet(the yolo original framework), i use VOCdekit to do the training and validation. But, for caffemodel, i dont know the image format for calibration.