Hi, I'm trying to use openvino_tensorflow on my core i7-6700 CPU but if I use openvino tensorflow is around 10x slower than using only tf. OS : ubuntu 20.04 CPU: core i7-6700

I just installed openvino_tensorflow and if I run my code using only TF I got an inference time of 0.10 s. Then I add the line

import openvino_tensorflow openvino_tensorflow.set_backend('CPU')

My inference time is 1.5 s, log says 2022-07-11 09:50:48.690843: OVTF Summary -> 90 out of 624 nodes in the graph (14%) are now running with OpenVINO™ backend

After installing openvino_tensorflow, the message "Entry Point not Found" shows up on the command python "import openvino_tensorflow as ovtf"

Installation: python 3.9,windows10 install with to pip3 install -U pip pip3 install tensorflow==2.8.0 pip3 install openvino-tensorflow==2.0.0

Test under python: import openvino_tensorflow See image for error message

Did I missed something simple?

Greetings,

When I tried to install openvino_tensorlow, the following error came out:

ERROR: tensorflow-2.9.1-cp39-cp39-win_amd64.whl is not a supported wheel on this platform.

Any clue? Thanks in advance

my system:

windows 10 python 3.8, in the venv Im trying to install pip command: pip install

I try changing 3.8 with 3.9, NOTHING....

Had anyone installed openvino_tensorflow in Windows 10 x64?

Thanks in advance

• Remove a few unused imports in init.py
• Enable OneDNN when OVTF is imported, unless the environmental variable TF_ENABLE_ONEDNN_OPTS is already set
• Add APIs to enable or disable OneDNN through OVTF

when I pip installed the latest version of of tensorflow 2.91 along with openvino-tensorflow 2.1.0 I get this errow that is thrown when importing openvino_tensorflow.
libtbb.so.2: cannot open shared object file: No such file or directory the libopenvino is looking for libtbb.so.2 whereas the latest libtbb is libtbb.so.12 which is found in the lib folder I am surprised this has not been tested !!!

while running following command after installing openvino_tensorflow.

python -c "import openvino_tensorflow; print(openvino_tensorflow.version)"

tensorflow.python.framework.errors_impl.NotFoundError: XXXXXXXXXXXXXXXXX \lib\site-packages\openvino_tensorflow\openvino_tensorflow.dll not found

no issues with tensorflow:

python -c "import tensorflow as tf; print('TensorFlow version: ',tf.version)" 2022-04-07 08:00:06.091197: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found 2022-04-07 08:00:06.091694: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. TensorFlow version: 2.8.0

We observed that the program fails whenever the same image has been passed to sess.run() more than once. This error only happens with the SSD object detection models. For example:

Error occurs here:

OVTF Summary -> 2093 out of 4245 nodes in the graph (49%) are now running with OpenVINO™ backend
Predict Image [10]
Predict Image [89]
Predict Image [10] --> Segmentation fault

Error doesn't occur if we reinitialise the session:

OVTF Summary -> 2093 out of 4245 nodes in the graph (49%) are now running with OpenVINO™ backend
Predict Image [10]
Predict Image [89]
OVTF Summary -> 2093 out of 4245 nodes in the graph (49%) are now running with OpenVINO™ backend
Predict Image [10] 
...

Here is a code snippet for where the error was at:

Model: ssd_inception_v2,ssd_mobilenet_v1,ssd_mobilenet_v1_fpn,ssd_mobilenet_v2,ssd_resnet_50_fpn, ssdlite_mobilenet_v2 TensorFlow version: 2.6.0 openvino-tensorflow version: 1.0.0

class BackendTensorflowOpenvino(...)
  ...
  def load(self, model_path, inputs=None, outputs=None):
      # there is no input/output meta data i the graph so it need to come from config.
      if not inputs:
          raise ValueError("BackendTensorflow needs inputs")
      if not outputs:
          raise ValueError("BackendTensorflow needs outputs")
      self.outputs = outputs
      self.inputs = inputs

      infer_config = tf.compat.v1.ConfigProto()
      infer_config.intra_op_parallelism_threads = int(os.environ['TF_INTRA_OP_PARALLELISM_THREADS']) \
              if 'TF_INTRA_OP_PARALLELISM_THREADS' in os.environ else os.cpu_count()
      infer_config.inter_op_parallelism_threads = int(os.environ['TF_INTER_OP_PARALLELISM_THREADS']) \
              if 'TF_INTER_OP_PARALLELISM_THREADS' in os.environ else os.cpu_count()
      infer_config.use_per_session_threads = 1

      graph_def = tf.compat.v1.GraphDef()
      with tf.compat.v1.gfile.FastGFile(model_path, "rb") as f:
          graph_def.ParseFromString(f.read())
          g = tf.compat.v1.import_graph_def(graph_def, name='')
          
      self.sess = tf.compat.v1.Session(graph=g, config=infer_config)
      return self

  def predict(self, feed):
      ans = self.sess.run(self.outputs, feed_dict=feed)
      return ans

import openvino_tensorflow as ovtf
ovtf.set_backend('CPU')
model = BackendTensorflowOpenvino(...)
model.predict({"image_tensor:0": img1})
model.predict({"image_tensor:0": img1})

The program fails at the second model.predict() call. Error Message:

./tmp-e9_0_ipe.sh: line 42: 65995 Segmentation fault      (core dumped)

Also, the above code works with the original tensorflow backend and other models ( e.g. faster_rcnn_inception_v2_coco, faster_rcnn_resnet50_coco) from TensorFlow Object Detection Model Zoo and yolo-v3.

To reproduce the error, we can use ck - an automated workflow for designing ML systems.

pip install ck

Then, pull the relevant program:

ck pull repo --url=https://github.com/krai/ck-mlperf.git
ck pull repo --url=https://github.com/krai/ck-object-detection.git

Then, follow the building instruction here. And finally, run the following command

time docker run -it --rm ${CK_IMAGE} \
"ck run program:mlperf-inference-vision --cmd_key=direct --skip_print_timers \
  --env.CK_LOADGEN_SCENARIO=SingleStream \
  --env.CK_LOADGEN_MODE='--accuracy' \
  --env.CK_LOADGEN_EXTRA_PARAMS='--count 50' \
  \
  --env.CK_MODEL_PROFILE=default_tf_object_det_zoo \
  --dep_add_tags.weights=ssd_mobilenet_v1_coco \
  \
  --env.CK_INFERENCE_ENGINE=tensorflow \
  --env.CK_INFERENCE_ENGINE_BACKEND=openvino-cpu \
  --env.CUDA_VISIBLE_DEVICES=-1"

On my machine, the classification_sample example returns this error :

_Create kernel failed: Not found: No registered 'nGraphEncapsulate' OpKernel for 'GPU' devices compatible with node

I am able to successfully run this command :

python3 -c "import tensorflow as tf; print('TensorFlow version: ',tf.version);
import openvino_tensorflow; print(openvino_tensorflow.version)"

The full error message is :

$ python3 examples/classification_sample.py 2021-09-27 14:45:03.705017: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 Available Backends: CPU MYRIAD 2021-09-27 14:45:04.998781: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-09-27 14:45:04.999214: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1 2021-09-27 14:45:05.005135: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-27 14:45:05.005512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: pciBusID: 0000:01:00.0 name: Quadro M2200 computeCapability: 5.2 coreClock: 1.036GHz coreCount: 8 deviceMemorySize: 3.94GiB deviceMemoryBandwidth: 82.08GiB/s 2021-09-27 14:45:05.005547: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 2021-09-27 14:45:05.008327: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11 2021-09-27 14:45:05.008417: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11 2021-09-27 14:45:05.009473: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10 2021-09-27 14:45:05.009687: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10 2021-09-27 14:45:05.010126: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11 2021-09-27 14:45:05.010675: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11 2021-09-27 14:45:05.010834: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8 2021-09-27 14:45:05.010936: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-27 14:45:05.011675: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-27 14:45:05.012200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0 2021-09-27 14:45:05.012250: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 2021-09-27 14:45:05.344618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-09-27 14:45:05.344663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0 2021-09-27 14:45:05.344670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N 2021-09-27 14:45:05.344869: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-27 14:45:05.345291: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-27 14:45:05.345681: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-27 14:45:05.346057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3113 MB memory) -> physical GPU (device: 0, name: Quadro M2200, pci bus id: 0000:01:00.0, compute capability: 5.2) 2021-09-27 14:45:05.380470: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3000000000 Hz 2021-09-27 14:45:06.246353: OVTF Summary -> 476 out of 480 nodes in the graph (99%) are now running with OpenVINO™ backend 2021-09-27 14:45:06.289576: E tensorflow/core/framework/op_segment.cc:54] Create kernel failed: Not found: No registered '_nGraphEncapsulate' OpKernel for 'GPU' devices compatible with node {{node ovtf_cluster_0}} . Registered: device='CPU'

Connecting to pjreddie.com (pjreddie.com)|128.208.4.108|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 248007048 (237M) [application/octet-stream] Saving to: ‘weights/yolov3.weights’

weights/yolov3.weights 100%[=================================================>] 236.52M 40.8MB/s in 6.9s

2021-09-03 10:17:16 (34.3 MB/s) - ‘weights/yolov3.weights’ saved [248007048/248007048]

./convert_yolov3.sh: line 17: python: command not found ./convert_yolov3.sh: line 18: python: command not found cp: cannot stat 'weights/yolo_v3_darknet_2.pb': No such file or directory cp: cannot stat 'weights/darknet53.h5': No such file or directory

DO NOT MERGE YET

• Cluster splitting if dynamic to static data flow exists
• Deassigning clusters for dynamic to static data flow removed
• Deassigning clusters due to Prod op removed
• StridedSlice translation throws error if it produces zero dimension output.

I find it will crash when I use transpose_sinking pass through a ngraph function which only have one Conv op and two inserted transpose op which is used to handle the nhwc input and output. I suppose it should do nothing in this situation instead of crash.

I was able to complete the example on 12th Gen Core Processor using CPU backend. But when I change to GPU backend, the example SegFault.

I have verified that OpenVINO 2022.1 was correctly installed by running the OpenVINO benchmark_app against the GPU device.

2022-07-21 15:47:12.926336: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2022-07-21 15:47:12.929013: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/openvino_2022/extras/opencv/lib:/opt/intel/openvino_2022/tools/compile_tool:/opt/intel/openvino_2022/runtime/3rdparty/tbb/lib::/opt/intel/openvino_2022/runtime/3rdparty/hddl/lib:/opt/intel/openvino_2022/runtime/lib/intel64
2022-07-21 15:47:12.929026: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-07-21 15:47:14.114683: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/opt/intel/openvino_2022.1.0.643/extras/opencv/python/cv2/../../bin:/opt/intel/openvino_2022/extras/opencv/lib:/opt/intel/openvino_2022/tools/compile_tool:/opt/intel/openvino_2022/runtime/3rdparty/tbb/lib::/opt/intel/openvino_2022/runtime/3rdparty/hddl/lib:/opt/intel/openvino_2022/runtime/lib/intel64
2022-07-21 15:47:14.114703: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-07-21 15:47:14.114715: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (my-adl-s): /proc/driver/nvidia/version does not exist
2022-07-21 15:47:14.114839: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Available Backends:
CPU
GPU
2022-07-21 15:47:20.447067: OVTF Summary -> 59 out of 1326 nodes in the graph (4%) are now running with OpenVINO™ backend
2022-07-21 15:47:21.487357: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
Inference time in ms: 205.22
person 0.98
tie 0.81
Output image is saved in detections.jpg

2022-07-21 15:47:23.855600: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2022-07-21 15:47:23.858267: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/openvino_2022/extras/opencv/lib:/opt/intel/openvino_2022/tools/compile_tool:/opt/intel/openvino_2022/runtime/3rdparty/tbb/lib::/opt/intel/openvino_2022/runtime/3rdparty/hddl/lib:/opt/intel/openvino_2022/runtime/lib/intel64
2022-07-21 15:47:23.858281: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-07-21 15:47:25.036728: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/opt/intel/openvino_2022.1.0.643/extras/opencv/python/cv2/../../bin:/opt/intel/openvino_2022/extras/opencv/lib:/opt/intel/openvino_2022/tools/compile_tool:/opt/intel/openvino_2022/runtime/3rdparty/tbb/lib::/opt/intel/openvino_2022/runtime/3rdparty/hddl/lib:/opt/intel/openvino_2022/runtime/lib/intel64
2022-07-21 15:47:25.036750: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-07-21 15:47:25.036761: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (my-adl-s): /proc/driver/nvidia/version does not exist
2022-07-21 15:47:25.036889: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Available Backends:
CPU
GPU
2022-07-21 15:47:31.407075: OVTF Summary -> 59 out of 1326 nodes in the graph (4%) are now running with OpenVINO™ backend
Segmentation fault (core dumped)