Every so often I have found my tensorflow server killing itself due to memory usage hitting capacity. I can't quite figure out how or why this happening because it only happens occasionally and my process to use the server to infer files sequentially is identical between multiple servers and runtimes.

My guess is the issue is related to the files itself (the only non-constant), has anybody run into similar problems? For the sake of context, I am using my tf server to translate text in these files.

Trying bazel build -c opt tensorflow_serving/... doesn't work so, it always gives me a SIMD not enabled (maybe define GEMMLOWP_ALLOW_SLOW_SCALAR_FALLBACK in the source code when x86_64) error, so I had to use bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 tensorflow_serving/... and that built wihtout any errors. But now I always get the same error when trying to test the running model, in fact I get the error trying to run anything in the container. I had to export the model myself by adding savedmodelbuilder code into the retrain script for inception. I'm on an Ubunutu PC, but I tried running tensorflow serving on a mac pro and got the same error when trying bazel build -c opt tensorflow_serving/... so again I had to use bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 tensorflow_serving/..., and again the same exact error listed below

root@42fd2c27af9d:/serving# bazel-bin/tensorflow_serving/example/inception_client --server=localhost:9000 --image=./Xiang_Xiang_panda.jpg
Traceback (most recent call last):
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/tf_serving/tensorflow_serving/example/inception_client.py", line 56, in <module>
    tf.app.run()
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/python/platform/app.py", line 129, in run
    _sys.exit(main(argv))
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/tf_serving/tensorflow_serving/example/inception_client.py", line 50, in main
    tf.contrib.util.make_tensor_proto(data, shape=[1]))
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 53, in __getattr__
    module = self._load()
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 42, in _load
    module = importlib.import_module(self.__name__)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/contrib/__init__.py", line 81, in <module>
    from tensorflow.contrib.eager.python import tfe as eager
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/contrib/eager/python/tfe.py", line 75, in <module>
    from tensorflow.contrib.eager.python.datasets import Iterator
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/contrib/eager/python/datasets.py", line 23, in <module>
    from tensorflow.contrib.data.python.ops import prefetching_ops
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/contrib/data/python/ops/prefetching_ops.py", line 25, in <module>
    resource_loader.get_path_to_datafile("../../_prefetching_ops.so"))
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/contrib/util/loader.py", line 55, in load_op_library
    ret = load_library.load_op_library(path)
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/python/framework/load_library.py", line 56, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/contrib/data/python/ops/../../_prefetching_ops.so: undefined symbol: _ZN6google8protobuf8internal26fixed_address_empty_stringB5cxx11E

When I check the running log I can see that my model is running:

2017-12-09 01:24:00.397485: I tensorflow_serving/model_servers/main.cc:147] Building single TensorFlow model file config:  model_name: inception model_base_path: /tmp/new5
2017-12-09 01:24:00.397670: I tensorflow_serving/model_servers/server_core.cc:439] Adding/updating models.
2017-12-09 01:24:00.397696: I tensorflow_serving/model_servers/server_core.cc:490]  (Re-)adding model: inception
2017-12-09 01:24:00.498119: I tensorflow_serving/core/basic_manager.cc:705] Successfully reserved resources to load servable {name: inception version: 1}
2017-12-09 01:24:00.498154: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: inception version: 1}
2017-12-09 01:24:00.498169: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: inception version: 1}
2017-12-09 01:24:00.498189: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:360] Attempting to load native SavedModelBundle in bundle-shim from: /tmp/new5/1
2017-12-09 01:24:00.498203: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:236] Loading SavedModel from: /tmp/new5/1
2017-12-09 01:24:00.623487: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2 FMA
2017-12-09 01:24:00.743901: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:155] Restoring SavedModel bundle.
2017-12-09 01:24:00.798587: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running LegacyInitOp on SavedModel bundle.
2017-12-09 01:24:00.805405: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:284] Loading SavedModel: success. Took 307196 microseconds.
2017-12-09 01:24:00.805517: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: inception version: 1}
2017-12-09 01:24:00.810840: I tensorflow_serving/model_servers/main.cc:288] Running ModelServer at 0.0.0.0:9000 ...

I have trained nmt models, but I can not understand how to export the models to tensorflow serving. I read the documentations of MNIST and Inception, but I think these models are different with nmt models. Can you add a demo to show how to export the nmt models? This would be a great help to beginners like me, thanks!

I am trying to convert my Keras graph to a TF graph.

I managed to run the provided tensorflow_serving examples, but I'm having issues to run my custom model.

Here is my code:

` import tensorflow as tf from keras import backend as K from tensorflow.contrib.session_bundle import exporter def export_model_to_tf(model): K.set_learning_phase(0) # all new operations will be in test mode from now on # serialize the model and get its weights, for quick re-building

export_path = "./tmp" # where to save the exported graph
export_version = "1" # version number (integer)

print('Exporting trained model to %s' % export_path)

saver = tf.train.Saver(sharded=True)
with tf.Session() as sess:
    model_exporter = exporter.Exporter(saver)

    signature = exporter.classification_signature(input_tensor=model.input, scores_tensor=model.output)

    model_exporter.init(sess.graph.as_graph_def(),
                default_graph_signature=signature)

    model_exporter.export(export_path, tf.constant(export_version), sess)

`

This is the error I am getting:

root@566d926360d6:/serving# bazel-bin/tensorflow_serving/example/main ./tmp/main_model Using TensorFlow backend. Exporting trained model to ./tmp 2017-02-03 22:32:37.001202: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2017-02-03 22:32:37.001260: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. <tensorflow.contrib.session_bundle.exporter.Exporter object at 0x7f617023db50> Traceback (most recent call last): File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/tf_serving/tensorflow_serving/example/main.py", line 100, in main() File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/tf_serving/tensorflow_serving/example/main.py", line 98, in main save_model.export_model_to_tf(model) File "/serving/tensorflow_serving/example/save_model.py", line 25, in export_model_to_tf model_exporter.export(export_path, tf.constant(export_version), sess) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/contrib/session_bundle/exporter.py", line 275, in export meta_graph_suffix=constants.EXPORT_SUFFIX_NAME) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/training/saver.py", line 1390, in save {self.saver_def.filename_tensor_name: checkpoint_file}) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 767, in run run_metadata_ptr) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 965, in _run feed_dict_string, options, run_metadata) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1015, in _do_run target_list, options, run_metadata) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1035, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value dense_1_W [[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](save/ShardedFilename, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, dense_1_W, dense_1_b, dense_2_W, dense_2_b)]]

Caused by op u'save/SaveV2', defined at: File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/tf_serving/tensorflow_serving/example/main.py", line 100, in main() File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/tf_serving/tensorflow_serving/example/main.py", line 98, in main save_model.export_model_to_tf(model) File "/serving/tensorflow_serving/example/save_model.py", line 13, in export_model_to_tf saver = tf.train.Saver(sharded=True) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/training/saver.py", line 1067, in init self.build() File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/training/saver.py", line 1097, in build restore_sequentially=self._restore_sequentially) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/training/saver.py", line 685, in build save_tensor = self._AddShardedSaveOps(filename_tensor, per_device) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/training/saver.py", line 361, in _AddShardedSaveOps return self._AddShardedSaveOpsForV2(filename_tensor, per_device) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/training/saver.py", line 335, in _AddShardedSaveOpsForV2 sharded_saves.append(self._AddSaveOps(sharded_filename, saveables)) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/training/saver.py", line 276, in _AddSaveOps save = self.save_op(filename_tensor, saveables) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/training/saver.py", line 219, in save_op tensors) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/ops/gen_io_ops.py", line 780, in save_v2 tensors=tensors, name=name) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 768, in apply_op op_def=op_def) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 2402, in create_op original_op=self._default_original_op, op_def=op_def) File "/serving/bazel-bin/tensorflow_serving/example/main.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 1264, in init self._traceback = _extract_stack()

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value dense_1_W [[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](save/ShardedFilename, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, dense_1_W, dense_1_b, dense_2_W, dense_2_b)]]

Do you know what could cause the Saver to fail?

Thanks!

We are trying to build Tensorflow Serving 0.5.1 with TensorFlow 1.0.0@07bb8ea

Basing on CUDA 7.5, cuDNN 5. Bazel 0.4.4

cd serving && bazel build -c opt --config=cuda tensorflow_serving/...

ERROR: /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe0160
8c/external/org_tensorflow/tensorflow/contrib/nccl/BUILD:23:1: C++ c
ompilation of rule '@org_tensorflow//tensorflow/contrib/nccl:python/
ops/_nccl_ops.so' failed: crosstool_wrapper_driver_is_not_gcc failed
: error executing command external/local_config_cuda/crosstool/clang
/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTI
FY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-paramete
r ... (remaining 76 argument(s) skipped): com.google.devtools.build.
lib.shell.BadExitStatusException: Process exited with status 1.
In file included from external/org_tensorflow/tensorflow/contrib/ncc
l/kernels/nccl_manager.cc:15:0:
external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager
.h:23:44: fatal error: external/nccl_archive/src/nccl.h: No such fil
e or directory
 #include "external/nccl_archive/src/nccl.h"
                                            ^
compilation terminated.
INFO: Elapsed time: 147.378s, Critical Path: 107.11s

I'm able to find nccl.h, but it can't be found during bazel build. Any suggestions? Thanks in advanced.

find / -name nccl.h
/root/.cache/bazel/_bazel_root/5071e8dca1385fb776f72b33971bf157/exte
rnal/nccl_archive/src/nccl.h
/root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/exte
rnal/nccl_archive/src/nccl.h

The issue is for a single model (not multi-model)

1. What is the best way to serve requests from TF Serving in a multi-GPU box. TF is able to see all the GPUs but uses only one (Default Inception v3 model - from ~ 03/2016 I believe). (Is there some configuration in TF Serving that I am not aware of.)

Is this dependent on how the model is loaded and exported?

For inference (and not training) - Is there an example of a saved model being loaded onto multiple GPUs (with a single CPU) where the CPU splits the GPU load among multiple GPUs instead of using only one GPU.

2. If the model is trained on a single GPU machine, when it is being used for inference in TF Serving, can it be used on a multi GPU box where all GPUs are being used?

Are tensorflow_text ops currently fully supported by TensorFlow Serving? I have some models perfectly running using e.g. Wordpiece tokenization, but I'm failing when trying to serve this new version of the Universal Sentence Encoder due to the following error:

2019-11-14 13:34:17.976685: W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at partitioned_function_ops.cc:113 : Not found: Op type not registered 'SentencepieceOp' in binary running on 08305347fd52. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.

I'm experiencing this with tensorflow/serving:nightly.

Hi Team,

Recently we noticed that TFServing is using only single CPU core while testing object detection model, even though we set TENSORFLOW_SESSION_PARALLELISM=14. OMP_NUM_THREADS=56 is related to intel mkl-dnn.

docker run \
 -p 8501:8501 \
 --name=tfserving_rfcn \
 -v "$(pwd)/rfcn:/models/rfcn" \
 -e MODEL_NAME=rfcn \
 -e OMP_NUM_THREADS=56 \
 -e TENSORFLOW_INTRA_OP_PARALLELISM=14 \
 -e TENSORFLOW_INTER_OP_PARALLELISM=14 \
 tensorflow/serving:mkl

htop looks as below.

Is this something to do with how model script is written ? or any other configuration to do at TFServing end.?

What would be the best way to load and serve multiple models into the system?

Looking at the code that exists today, I couldn't see anything directly usable. If there's nothing that does this today, what would be a good set of things to look at to implement something like the following,

Some thoughts,

• Lets assume each model resided in a directory called /mnt/modelid/0000001
• The current FileSystemStoragePathSource only looks under a given path for the aspired version. So ideally, there are some methods somewhere that can instead help look for modelid rather than within it.
• The output of this should be a hashmap of servableId type objects, and I ought to be able to do a Get() from such an object to get the right modelId, and then run classify() on it.

Lastly - do you guys have use cases where there are multiple models spread out across multiple machines, and using some sort of a DHT implementation to access them?

Hi, I use this Dockerfile

# https://github.com/tensorflow/serving/blob/master/tensorflow_serving/tools/docker/Dockerfile.devel-gpu

FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04

RUN apt-get update && apt-get install -y \
        automake \
        bash-completion \
        build-essential \
        curl \
        git \
        g++ \
        libfreetype6-dev \
        libpng12-dev \
        libtool \
        libzmq3-dev \
        mlocate \
        pkg-config \
        python-dev \
        python-numpy \
        python-pip \
        software-properties-common \
        swig \
        unzip \
        zip \
        zlib1g-dev \
        libcurl3-dev \
        openjdk-8-jdk\
        openjdk-8-jre-headless \
        wget \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Set up grpc
RUN pip install mock grpcio

# Bazel
# required by TensorFlow
# sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python
# https://github.com/bazelbuild/bazel/releases/
ENV BAZEL_VERSION=0.14.1
RUN wget https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel_$BAZEL_VERSION-linux-x86_64.deb \
        && dpkg -i bazel_$BAZEL_VERSION-linux-x86_64.deb \
        && rm      bazel_$BAZEL_VERSION-linux-x86_64.deb


# Build TensorFlow with the CUDA configuration
ENV CI_BUILD_PYTHON python
ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH
ENV TF_NEED_CUDA 1
ENV TF_CUDA_COMPUTE_CAPABILITIES=3.0,3.5,5.2,6.0,6.1,7.0
ENV TF_CUDA_VERSION=9.0
ENV TF_CUDNN_VERSION=7

# Fix paths so that CUDNN can be found: https://github.com/tensorflow/tensorflow/issues/8264

WORKDIR /
RUN mkdir /usr/lib/x86_64-linux-gnu/include/ && \
  ln -s /usr/lib/x86_64-linux-gnu/include/cudnn.h /usr/lib/x86_64-linux-gnu/include/cudnn.h && \
  ln -s /usr/include/cudnn.h /usr/local/cuda/include/cudnn.h && \
  ln -s /usr/lib/x86_64-linux-gnu/libcudnn.so /usr/local/cuda/lib64/libcudnn.so && \
  ln -s /usr/lib/x86_64-linux-gnu/libcudnn.so.$TF_CUDNN_VERSION /usr/local/cuda/lib64/libcudnn.so.$TF_CUDNN_VERSION


# Fix paths so that NCCL can be found
# https://github.com/tensorflow/serving/issues/327
ENV TF_NCCL_VERSION=2.2.12
RUN echo "deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 /" > /etc/apt/sources.list.d/nvidia-ml.list
RUN apt-get update && apt-get install -y --no-install-recommends \
        libnccl2=${TF_NCCL_VERSION}-1+cuda${TF_CUDA_VERSION} \
        libnccl-dev=${TF_NCCL_VERSION}-1+cuda${TF_CUDA_VERSION} \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

ENV NCCL_INSTALL_PATH=/usr/lib/nccl/
WORKDIR /
RUN mkdir /usr/lib/nccl && \
  mkdir /usr/lib/nccl/include/ && \
  mkdir /usr/lib/nccl/lib/ && \
  ln -s /usr/include/nccl.h /usr/lib/nccl/include/nccl.h && \
  ln -s /usr/lib/x86_64-linux-gnu/libnccl.so /usr/lib/nccl/lib/libnccl.so && \
  ln -s /usr/lib/x86_64-linux-gnu/libnccl.so.2 /usr/lib/nccl/lib/libnccl.so.2 && \
  ln -s /usr/lib/x86_64-linux-gnu/libnccl.so.$TF_NCCL_VERSION /usr/lib/nccl/lib/libnccl.so.$TF_NCCL_VERSION


# Download, build, and install TensorFlow Serving
ARG TF_SERVING_VERSION=1.8.0
WORKDIR /tensorflow-serving

RUN apt-get update && apt-get install -y --no-install-recommends \
       libevent-dev \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN git clone --recurse-submodules https://github.com/tensorflow/serving \
    && cd serving \
    && git checkout $TF_SERVING_VERSION \
    && bazel build --jobs 16 -c opt --config=cuda -k --verbose_failures \
        --crosstool_top=@local_config_cuda//crosstool:toolchain \
        tensorflow_serving/model_servers:tensorflow_model_server \
    && cp bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server /usr/local/bin/ \
    && bazel clean --expunge \
    && cd / && rm -rf /tensorflow-serving

and I hit this error:

[4,514 / 4,517] Compiling external/org_tensorflow/tensorflow/core/kernels/tile_functor_gpu.cu.cc; 505s local
ERROR: /tensorflow-serving/serving/tensorflow_serving/model_servers/BUILD:270:1: Couldn't build file tensorflow_serving/model_servers/tensorflow_model_server: Linking of rule '//tensorflow_serving/model_servers:tensorflow_model_server' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command
  (cd /root/.cache/bazel/_bazel_root/86e62be83a53cf1af5b8032777534537/execroot/tf_serving && \
  exec env - \
    LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 \
    PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/usr/bin/python \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -o bazel-out/k8-opt/bin/tensorflow_serving/model_servers/tensorflow_model_server '-Wl,-rpath,$ORIGIN/../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccusolver___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../_solib_local/_U@local_Uconfig_Unccl_S_S_Cnccl___Uexternal_Slocal_Uconfig_Unccl_Snccl_Slib' '-Wl,-rpath,$ORIGIN/../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccublas___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccuda_Udriver___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudnn___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccufft___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccurand___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudart___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' -Lbazel-out/k8-opt/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccusolver___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/k8-opt/bin/_solib_local/_U@local_Uconfig_Unccl_S_S_Cnccl___Uexternal_Slocal_Uconfig_Unccl_Snccl_Slib -Lbazel-out/k8-opt/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccublas___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/k8-opt/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccuda_Udriver___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/k8-opt/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudnn___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/k8-opt/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccufft___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/k8-opt/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccurand___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/k8-opt/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudart___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -Wl,-z,notext -Wl,-z,muldefs -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -Wl,-z,notext -pthread -Wl,-rpath,../local_config_cuda/cuda/lib64 -Wl,-rpath,../local_config_cuda/cuda/extras/CUPTI/lib64 -Wl,-no-as-needed -B/usr/bin/ -pie -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes '-Wl,--build-id=md5' '-Wl,--hash-style=gnu' -Wl,--gc-sections -Wl,@bazel-out/k8-opt/bin/tensorflow_serving/model_servers/tensorflow_model_server-2.params)
/usr/bin/ld: bazel-out/k8-opt/genfiles/external/com_github_libevent_libevent/libevent/lib/libevent.a(buffer.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
bazel-out/k8-opt/genfiles/external/com_github_libevent_libevent/libevent/lib/libevent.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
Target //tensorflow_serving/model_servers:tensorflow_model_server failed to build
INFO: Elapsed time: 1579.845s, Critical Path: 603.27s
INFO: 3721 processes, local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

I tried to add --copt="-fPIC" to the bazel command as recommended, to no success. I tried to apt install libevent-dev beforehand, to no success. Any ideas?

Well, unfortunately, seems here we can not visit googlesource site , is there a workaround so I can bazel build tensorflow serving? I'm using tensorflow right now. Thanks

no such package '@boringssl_git//': Error cloning repository: https://boringssl.googlesource.com/boringssl: cannot open git-upload-pack and referenced by '//external:libssl'.

Bug Report

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• TensorFlow Serving installed from (source or binary): source
• TensorFlow Serving version: master

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow Serving.

Following README in remote predict op. The git commit and sha256 are the same as in master. I run:

tools/run_in_docker.sh bazel run tensorflow_serving/experimental/example:half_plus_two_with_rpop -- --target_address=localhost:8500

I get ModuleNotFoundError: No module named 'tensorflow.compat' in remote_predict_ops.py file. Then I try to remove compat and write everything with v2 api which gives abi issues:

ImportError: /home/rasulkarimov/serving/.cache/_bazel_root/0bdcf8b08a5256a78e00fcc7b2c20a7c/execroot/tf_serving/bazel-out/k8-opt/bin/tensorflow_serving/experimental/example/half_plus_two_with_rpop.runfiles/org_tensorflow/tensorflow/python/../libtensorflow_framework.so.2: undefined symbol: _ZTIN10tensorflow7SessionE

Is there a commit that I can checkout and reproduce the example, would appreciate it.

Additionally I have to manually symlink libtensorflow_framework.2.11.0 to libtensorflow_framework.2.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

tensorflow/ops/remote_predict/python/ops/remote_predict_ops.py", line 23, in <module>
    import tensorflow.compat.v1 as tf
ModuleNotFoundError: No module named 'tensorflow.compat'

I have updated this broken link (https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/saved_model_warmup) which was under this hyperlink (model warmup) to new hyperlink (https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/saved_model_warmup.md)

Bug Report

If this is a bug report, please fill out the following form in full:

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): cloud.google.com/gke-os-distribution: cos kubernetes.io/os: linux
• TensorFlow Serving installed from (source or binary): binary
• TensorFlow Serving version: upgrading from 2.4.1 to 2.11.0

Describe the problem

I have a trained artifact of a simple neural net with semi-complex feature transformations made with tensorflow transforms. I've been running it for quite some time in 2.4.1 and recently looked into running the model with 2.11.0. With 2 different models, I've seen the image bump take the model from ~20ms p90 to >200ms as measured by the historgrams of :tensorflow:core:graph_run_time_usecs_histogram_bucket. One was trained with an earlier version and the 2nd trained with TF2.9.

Server side batching is not enabled and cpu utilization is at 6% with no cpu throttling (container_cpu_cfs_throttled_periods_total). All configurations for running are defaults.

In the startup logs, I see external/org_tensorflow/tensorflow/core/grappler/optimizers/loop_optimizer.cc:906] Skipping loop optimization for Merge node with control input in 2.4.1 and see no grappler log lines in 2.11.0 startup.

Ideally, we would not see a drastic performance degradation with a minor version upgrade. I'm looking to see if there was any changes in graph execution-related code in the recent releases or what remediations are possible to achieve performance parity.

Exact Steps to Reproduce

Cannot provide the model, but this has happened on 2 models.

Profiling info:

In general it seems like in the newer version there is more time in feature preprocessing wheras previously the dense layers dominated the profile. This profile was taken from the running a TFServing instance in our cluster under typical load.

2.11.0 top ops TensorListFromTensor StringSplitV2 PadV2 StatefulPartitionedCall RaggedTensorToTensor

2.4.1 top ops _FusedMatMul StaticRegexReplace Mul Less StringSplitV2

Signed-off-by: Abolfazl Shahbazi [email protected]

Looks like TF Serving + MKL now looks for libiomp5.so and if not found we get an error as follows:

ERROR: /root/.cache/bazel/_bazel_root/e53bbb0b0da4e26d24b415310219b953/external/org_tensorflow/tensorflow/cc/BUILD:681:22: Executing genrule @org_tensorflow//tensorflow/cc:candidate_sampling_ops_genrule failed: (Exit 127): bash failed: error executing command 
  (cd /root/.cache/bazel/_bazel_root/e53bbb0b0da4e26d24b415310219b953/execroot/tf_serving && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /bin/bash bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/cc/candidate_sampling_ops_genrule.genrule_script.sh)
# Configuration: 165ec08304a7eacebdd9f7b599483e991b9b7887aabba1f02a76b171a0e1db4c
# Execution platform: @local_execution_config_platform//:platform
bazel-out/host/bin/external/org_tensorflow/tensorflow/cc/ops/candidate_sampling_ops_gen_cc: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory
ERROR: /root/.cache/bazel/_bazel_root/e53bbb0b0da4e26d24b415310219b953/external/org_tensorflow/tensorflow/cc/BUILD:681:22: Executing genrule @org_tensorflow//tensorflow/cc:nn_ops_genrule failed: (Exit 127): bash failed: error executing command 
  (cd /root/.cache/bazel/_bazel_root/e53bbb0b0da4e26d24b415310219b953/execroot/tf_serving && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /bin/bash bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/cc/nn_ops_genrule.genrule_script.sh)
# Configuration: 165ec08304a7eacebdd9f7b599483e991b9b7887aabba1f02a76b171a0e1db4c
# Execution platform: @local_execution_config_platform//:platform
bazel-out/host/bin/external/org_tensorflow/tensorflow/cc/ops/nn_ops_gen_cc: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory
ERROR: /root/.cache/bazel/_bazel_root/e53bbb0b0da4e26d24b415310219b953/external/org_tensorflow/tensorflow/compiler/tf2xla/cc/BUILD:9:21: Executing genrule @org_tensorflow//tensorflow/compiler/tf2xla/cc:xla_ops_gen_genrule failed: (Exit 127): bash failed: error executing command 
  (cd /root/.cache/bazel/_bazel_root/e53bbb0b0da4e26d24b415310219b953/execroot/tf_serving && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/external/org_tensorflow/tensorflow/compiler/tf2xla/cc/ops/xla_ops_gen_cc bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/compiler/tf2xla/cc/ops/xla_ops.h bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/compiler/tf2xla/cc/ops/xla_ops.cc 0 ,')
# Configuration: 165ec08304a7eacebdd9f7b599483e991b9b7887aabba1f02a76b171a0e1db4c
# Execution platform: @local_execution_config_platform//:platform
bazel-out/host/bin/external/org_tensorflow/tensorflow/compiler/tf2xla/cc/ops/xla_ops_gen_cc: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory

This PR fixes that issue.

As before to build the TensorFlow Serving with MKL please see my notes here: https://github.com/tensorflow/serving/pull/2077#issuecomment-1307535892

Bug Report

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 22.04.1 LTS (running in GKE)
• TensorFlow Serving installed from (source or binary): binary (from tensorflow/serving:2.11.0 docker image)
• TensorFlow Serving version: 2.11.0

Describe the problem

When running tensorflow serving with a custom memory allocator (tcmalloc), after a period of time in the event loop (generally less than 1 minute as long as there is load) tensorflow serving will crash due to a segmentation fault.

Similar issues (std::bad_alloc) were present in tensorflow serving starting with 2.9+ when using tcmalloc.

The issue is not present in 2.8.3.

Exact Steps to Reproduce

No reproduction steps at this time.

Source code / logs

Here are backtraces generated when the segmentation fault occurs.

#0  0x00007fbea81dbe93 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int) () from /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4
#1  0x00007fbea81dc1fe in tcmalloc::ThreadCache::Scavenge() () from /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4
#2  0x000055a1440aa6a2 in dnnl_primitive_desc_destroy ()
#3  0x000055a13b8db316 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#4  0x000055a13ec9658e in std::_Sp_counted_ptr<dnnl::inner_product_forward::primitive_desc*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ()
#5  0x000055a13b8db316 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#6  0x000055a1401d3dbb in tensorflow::MklDnnMatMulFwdPrimitive<float, float, float, float, float>::Setup(tensorflow::MklDnnMatMulFwdParams const&) ()
#7  0x000055a1401d5b5a in tensorflow::MklDnnMatMulFwdPrimitiveFactory<float, float, float, float, float>::Get(tensorflow::MklDnnMatMulFwdParams const&, bool) ()
#8  0x000055a1401d6e18 in tensorflow::MklFusedMatMulOp<Eigen::ThreadPoolDevice, float, true>::Compute(tensorflow::OpKernelContext*) ()
#9  0x000055a142530ac8 in tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) ()
#10 0x000055a142585820 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) ()
#11 0x000055a14258698c in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) ()
#12 0x000055a148210621 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) ()
#13 0x000055a14820e573 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#14 0x000055a14803e8a5 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) ()
#15 0x00007fbea7cedb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#16 0x00007fbea7d7fa00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

#0  0x000055640ec953aa in tensorflow::MklDnnMatMulFwdPrimitive<float, float, float, float, float>::Execute(float const*, float const*, float const*, float*, void*, std::shared_ptr<dnnl::stream>) ()
#1  0x000055640ec9b7f5 in tensorflow::MklFusedMatMulOp<Eigen::ThreadPoolDevice, float, true>::Compute(tensorflow::OpKernelContext*) ()
#2  0x0000556410ff4ac8 in tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) ()
#3  0x0000556411049820 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) ()
#4  0x000055641104a98c in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) ()
#5  0x0000556416cd4621 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) ()
#6  0x0000556416cd2573 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#7  0x0000556416b028a5 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) ()
#8  0x00007f04b9a8eb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x00007f04b9b20a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

#0  0x00007f2c6c1d0eeb in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int) () from /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4
#1  0x00007f2c6c1d11fe in tcmalloc::ThreadCache::Scavenge() () from /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4
#2  0x000055e25aac33f6 in Eigen::internal::TensorBlockScratchAllocator<Eigen::ThreadPoolDevice>::~TensorBlockScratchAllocator() ()
#3  0x000055e25d2f1734 in Eigen::internal::TensorExecutor<Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<float, 2, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorCwiseBinaryOp<Eigen::internal::scalar_product_op<float, float>, Eigen::TensorBroadcastingOp<Eigen::IndexList<long, Eigen::type2index<1l> > const, Eigen::TensorReshapingOp<Eigen::IndexList<Eigen::type2index<1l>, long> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const> const> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const> const> const, Eigen::ThreadPoolDevice, true, (Eigen::internal::TiledEvaluation)1>::run(Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<float, 2, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorCwiseBinaryOp<Eigen::internal::scalar_product_op<float, float>, Eigen::TensorBroadcastingOp<Eigen::IndexList<long, Eigen::type2index<1l> > const, Eigen::TensorReshapingOp<Eigen::IndexList<Eigen::type2index<1l>, long> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const> const> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const> const> const&, Eigen::ThreadPoolDevice const&) ()
#4  0x000055e25d318523 in tensorflow::BinaryOp<Eigen::ThreadPoolDevice, tensorflow::functor::mul<float> >::Compute(tensorflow::OpKernelContext*) ()
#5  0x000055e260ccaac8 in tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) ()
#6  0x000055e260d1f820 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) ()
#7  0x000055e260d2098c in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) ()
#8  0x000055e2669aa621 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) ()
#9  0x000055e2669a8573 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#10 0x000055e2667d88a5 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) ()
#11 0x00007f2c6bce2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x00007f2c6bd74a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

#0  0x00007f40588a4ebb in tc_memalign () from /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4
#1  0x00007f40588a4fda in tc_posix_memalign () from /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4
#2  0x0000562df89f5cf4 in tsl::port::AlignedMalloc(unsigned long, int) ()
#3  0x0000562df856d688 in tensorflow::Tensor::Tensor(tsl::Allocator*, tensorflow::DataType, tensorflow::TensorShape const&) ()
#4  0x0000562dee05fcf8 in tensorflow::Tensor& std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >::emplace_back<tensorflow::DataType, tensorflow::TensorShape>(tensorflow::DataType&&, tensorflow::TensorShape&&) ()
#5  0x0000562df84e1972 in tensorflow::example::FastParseExample(tensorflow::example::FastParseExampleConfig const&, absl::lts_20220623::Span<tsl::tstring const>, absl::lts_20220623::Span<tsl::tstring const>, tsl::thread::ThreadPool*, tensorflow::example::Result*) ()
#6  0x0000562dedc562a6 in tensorflow::ParseExampleOp::Compute(tensorflow::OpKernelContext*) ()
#7  0x0000562df2d05ac8 in tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) ()
#8  0x0000562df2d5a820 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) ()
#9  0x0000562df2d5b98c in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) ()
#10 0x0000562df89e5621 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) ()
#11 0x0000562df89e3573 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#12 0x0000562df88138a5 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) ()
#13 0x00007f40583afb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x00007f4058441a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

#0  0x00007fbf95329717 in tc_newarray () from /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4
#1  0x0000562785fcf579 in std::unordered_map<int, dnnl::memory, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, dnnl::memory> > >& std::vector<std::unordered_map<int, dnnl::memory, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, dnnl::memory> > >, std::allocator<std::unordered_map<int, dnnl::memory, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, dnnl::memory> > > > >::emplace_back<std::unordered_map<int, dnnl::memory, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, dnnl::memory> > > >(std::unordered_map<int, dnnl::memory, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, dnnl::memory> > >&&) ()
#2  0x0000562787a31bb6 in tensorflow::MklDnnMatMulFwdPrimitive<float, float, float, float, float>::Setup(tensorflow::MklDnnMatMulFwdParams const&) ()
#3  0x0000562787a33b5a in tensorflow::MklDnnMatMulFwdPrimitiveFactory<float, float, float, float, float>::Get(tensorflow::MklDnnMatMulFwdParams const&, bool) ()
#4  0x0000562787a34e18 in tensorflow::MklFusedMatMulOp<Eigen::ThreadPoolDevice, float, true>::Compute(tensorflow::OpKernelContext*) ()
#5  0x0000562789d8eac8 in tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) ()
#6  0x0000562789de3820 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) ()
#7  0x0000562789de498c in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) ()
#8  0x000056278fa6e621 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) ()
#9  0x000056278fa6c573 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#10 0x000056278f89c8a5 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) ()
#11 0x00007fbf94e34b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x00007fbf94ec6a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81