cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
sitting
Voyager
Voyager
487 Views
Registered: ‎05-04-2014

tf2_resnet50_imagenet_224_224_7.76G_1.3 evalute fail(Your input ran out of data)

Jump to solution

Hi,

I have tried to run run_eval_by_images_h5.sh and I got an error as below:

(vitis-ai-tensorflow2) Vitis-AI /workspace/Tool-Example/zu6eg_custom/tf2_resnet50_imagenet_224_224_7.76G_1.3/code/test > bash run_eval_by_images_h5.sh
2021-01-26 05:29:48.383313: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-26 05:29:49.253108: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-01-26 05:29:49.276101: E tensorflow/stream_executor/cuda/cuda_driver.cc:314] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable de(vitis-ai-tensorflow2) Vitis-AI /workspace/Tool-Example/zu6eg_custom/tf2_resnet50_imagenet_224_224_7.76G_1.3/code/test > bash run_eval_by_images_h5.sh
2021-01-26 05:37:39.805139: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-26 05:37:40.601079: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-01-26 05:37:40.624212: E tensorflow/stream_executor/cuda/cuda_driver.cc:314] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2021-01-26 05:37:40.624253: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: hill213-pc
2021-01-26 05:37:40.624256: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: hill213-pc
2021-01-26 05:37:40.624377: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 450.102.4
2021-01-26 05:37:40.624394: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 450.102.4
2021-01-26 05:37:40.624398: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 450.102.4
2021-01-26 05:37:40.624712: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-26 05:37:40.629135: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2799925000 Hz
2021-01-26 05:37:40.629607: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5633eaa91b10 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-26 05:37:40.629614: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
1/1000 [..............................] - ETA: 0s - loss: 1.1056 - sparse_categorical_accuracy: 0.7000 - sparse_top_k_categorical_accuracy: 0. 2/1000 [..............................] - ETA: 13:21 - loss: 0.7940 - sparse_categorical_accuracy: 0.7600 - sparse_top_k_categorical_accuracy: 3/1000 [..............................] - ETA: 18:06 - loss: 1.0012 - sparse_categorical_accuracy: 0.7333 - sparse_top_k_categorical_accuracy: 4/1000 [..............................] - ETA: 20:44 - loss: 0.9979 - sparse_categorical_accuracy: 0.7400 - sparse_top_k_categorical_accuracy: 5/1000 [..............................] - ETA: 22:17 - loss: 1.0498 - sparse_categorical_accuracy: 0.7360 - sparse_top_k_categorical_accuracy: 6/1000 [..............................] - ETA: 23:04 - loss: 1.1561 - sparse_categorical_accuracy: 0.7133 - sparse_top_k_categorical_accuracy: 7/1000 [..............................] - ETA: 23:46 - loss: 1.1744 - sparse_categorical_accuracy: 0.7229 - sparse_top_k_categorical_accuracy: 8/1000 [..............................] - ETA: 24:09 - loss: 1.1376 - sparse_categorical_accuracy: 0.7300 - sparse_top_k_categorical_accuracy: 9/1000 [..............................] - ETA: 24:23 - loss: 1.0895 - sparse_categorical_accuracy: 0.7311 - sparse_top_k_categorical_accuracy: 10/1000 [..............................] - ETA: 24:35 - loss: 1.0222 - sparse_categorical_accuracy: 0.7460 - sparse_top_k_categorical_accuracy: 11/1000 [..............................] - ETA: 24:44 - loss: 1.0116 - sparse_categorical_accuracy: 0.7527 - sparse_top_k_categorical_accuracy: 12/1000 [..............................] - ETA: 24:52 - loss: 1.0217 - sparse_categorical_accuracy: 0.7500 - sparse_top_k_categorical_accuracy: 13/1000 [..............................] - ETA: 24:59 - loss: 0.9909 - sparse_categorical_accuracy: 0.7554 - sparse_top_k_categorical_accuracy: 14/1000 [..............................] - ETA: 25:02 - loss: 1.0162 - sparse_categorical_accuracy: 0.7514 - sparse_top_k_categorical_accuracy: 15/1000 [..............................] - ETA: 25:05 - loss: 1.0039 - sparse_categorical_accuracy: 0.7533 - sparse_top_k_categorical_accuracy: 16/1000 [..............................] - ETA: 25:06 - loss: 1.0128 - sparse_categorical_accuracy: 0.7513 - sparse_top_k_categorical_accuracy: 17/1000 [..............................] - ETA: 25:11 - loss: 1.0231 - sparse_categorical_accuracy: 0.7471 - sparse_top_k_categorical_accuracy: 18/1000 [..............................] - ETA: 25:13 - loss: 1.0157 - sparse_categorical_accuracy: 0.7489 - sparse_top_k_categorical_accuracy: 19/1000 [..............................] - ETA: 25:15 - loss: 1.0050 - sparse_categorical_accuracy: 0.7495 - sparse_top_k_categorical_accuracy: 20/1000 [..............................] - ETA: 25:16 - loss: 0.9997 - sparse_categorical_accuracy: 0.7510 - sparse_top_k_categorical_accuracy: 21/1000 [..............................] - ETA: 25:17 - loss: 0.9922 - sparse_categorical_accuracy: 0.7571 - sparse_top_k_categorical_accuracy: 22/1000 [..............................] - ETA: 25:15 - loss: 0.9908 - sparse_categorical_accuracy: 0.7564 - sparse_top_k_categorical_accuracy: 0.9327WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 1000.0 batches). You may need to use the repeat() function when building your dataset.
22/1000 [..............................] - 34s 2s/step - loss: 0.9908 - sparse_categorical_accuracy: 0.7564 - sparse_top_k_categorical_accuracy: 0.9327

 

How do I fix "tensorflow:Your input ran out of data;" ?

 

Thanks

Sitting

 

 

0 Kudos
1 Solution

Accepted Solutions
wangxd-xlnx
Xilinx Employee
Xilinx Employee
367 Views
Registered: ‎01-27-2021

Hi @sitting ,

Do you want to reproduce the accuracy performance in the readme? If so, you need a complete imagenet validation set ~50000 images.

I noticed that the number of your images is around 1100 so the script cannot complete the testing process.

View solution in original post

6 Replies
zhijiexu
Moderator
Moderator
462 Views
Registered: ‎09-29-2020

hi, @sitting 

You can check the batch size in you project.

Hoping it can help you!

-----------------------------------------------------------------------------

Don't forget to reply, give kudos and accept as a splution!

-----------------------------------------------------------------------------

 

0 Kudos
sitting
Voyager
Voyager
457 Views
Registered: ‎05-04-2014

Hi @zhijiexu ,

Is it to reduce eval_batch_size in evaluate step? 

flags.DEFINE_integer('eval_batch_size', 50, 'Evaluate batch size')

 

Thanks

Sitting

0 Kudos
wangxd-xlnx
Xilinx Employee
Xilinx Employee
389 Views
Registered: ‎01-27-2021

 Hi, @sitting 

Yes. If you just need to verify the training scripts and process, you could reduce the batch_size to adapt your own data.

 

 

sitting
Voyager
Voyager
377 Views
Registered: ‎05-04-2014

Hi

1092/50000 [..............................] - ETA: 31:30 - loss: 0.9892 - sparse_categoric

1094/50000 [..............................] - ETA: 31:30 - loss: 0.9927 - sparse_categoric

1096/50000 [..............................] - ETA: 31:30 - loss: 0.9910 - sparse_categoric

1098/50000 [..............................] - ETA: 31:30 - loss: 0.9926 - sparse_categoric

1100/50000 [..............................] - ETA: 31:30 - loss: 0.9908 - sparse_categoric

al_accuracy: 0.7564 - sparse_top_k_categorical_accuracy: 0.9327

WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 50000.0 batches). You may need to use the repeat() function when building your dataset.
1100/50000 [..............................] - 43s 39ms/step - loss: 0.9908 - sparse_categorical_accuracy: 0.7564 - sparse_top_k_categorical_accuracy: 0.9327

I have changed eval_batch_size to 1, but I still get the "ran out of data message"

 

Thanks

Sitting

0 Kudos
wangxd-xlnx
Xilinx Employee
Xilinx Employee
368 Views
Registered: ‎01-27-2021

Hi @sitting ,

Do you want to reproduce the accuracy performance in the readme? If so, you need a complete imagenet validation set ~50000 images.

I noticed that the number of your images is around 1100 so the script cannot complete the testing process.

View solution in original post

sitting
Voyager
Voyager
351 Views
Registered: ‎05-04-2014

Hi,

I solved this problem. Thank you so much.

 

Thanks

Sitting

 

0 Kudos