Sorry for making duplicates. Im not sure bug is in vx-delegate or in TIM-VX. https://github.com/VeriSilicon/tflite-vx-delegate/issues/32

Hi, I think I found a bug in the vx-delegate runtime.

setup: A311D + Android 9 + TensorFlow Lite with vx-delegate

Model: detector, that outputs multiple things: BBoxes, Landmarks, Probability scores and features vectors.

Problem: Landmarks outputs are garbage. How model works for landmarks:

input image -> backbone -> FPN -> Conv layers that produces features (OUTPUT 1) -> Conv layers that produces landmarks (OUTPUT 2)

so, if I have two outputs, that are downstream one after another, second outputs is not calculated and I get garbage output. The problem is only with INT8 graph, if I use FP32 graph, it works fine with vx-delegate.

On x86 with standard TFLite (and xnnpack) everything works fine with both INT8 and FP32 graphs.

update: Downstream is not important, even if in landmarks branch of graph I have only landmarks as outputs, I get garbage. I don't know why, but part of the graph with landmarks is not calculated on the NPU.

What could be the problem? Thanks.

I try to run SpatialTransformer(mxnet) operator on rv1126 using Tengine, but the Status of "SpatialTransformer" operator in tim-vx is "InternalOnly", and no TIM-VX API implementation.So i can't find a way to add SpatialTransformer npu support to Tengine, i wondered that：

1. Is SpatialTransformer(mxnet) operator on rv1126 npu supported?
2. If it is supported, how can i add it to Tengine?

Hi, I have A311D board with ubuntu on board (from khadas). My end goal is to compile tflite with vx-delegate support. What is the best way to do this?

Also Im trying to compile TIM-VX (libtim-vx.so) it seems Bazel building is broken, so I try CMake. CMake works fine: compiles and links all targets, but when I try to run unit tests (under gdb), I get segfault from *** _LoadStates() from libGAL.so what causes this?

I don't know if it's the right place to ask questions about your TVM fork, but I cannot raise issues in that repo.

I followed the guide from README.VSI.md to build TVM (on host, using x86_64_linux simulation drivers provided here) and TVM runtime (on target, using vendor-provided VIP NPU drivers), and ran the tests in test_vsi_npu, but I got these results:

logs from TVM C++ RPC tool:

VsiNpuModule::LoadFromBinary
LoadFromBinary: nbg size = 593344
LoadFromBinary: input size = 1
LoadFromBinary: output size = 1
VsiNpuModule : DeSerializeTensorSpec
VsiNpuModule : DeSerializeTensorSpec2
VsiNpuModule : DeSerializeTensorSpec
VsiNpuModule : DeSerializeTensorSpec2
[22:31:35] /home/nullko/Documents/tvm-vsi_npu/apps/cpp_rpc/rpc_env.cc:130: Load module from /home/ubuntu/workspace/rpc/model.so ...
VsiNpuModule::GetFunction: _lookup_linked_param
VsiNpuModule::GetFunction: return early
VsiNpuModule::GetFunction: _lookup_linked_param
VsiNpuModule::GetFunction: return early
VsiNpuModule::GetFunction: _lookup_linked_param
VsiNpuModule::GetFunction: return early
VsiNpuModule::GetFunction: _lookup_linked_param
VsiNpuModule::GetFunction: return early
VsiNpuModule::GetFunction: tvmgen_default_vsi_npu_0
[     1] PLS isn't existed
E [compute_node:379]Create node[0] NBG fail
Process Graph: 0 ms or 0 us

It seemed that TVM is able to compile the NBG on the host, but the target runtime cannot execute it. I wonder what caused the "PLS isn't existed" issue, is it because I didn't set some environment variables on the target platform?

Or maybe your tvm fork is still under development and is not ready to be used yet?

I try to use the vsi_nn_AddTensorFromHandle to create a tensor using buffer allocated by cv::Mat, every time the program exits, there will be a double free error issued by OpenCV. It seems that the passed buffer is freed by the OpenVX driver (OpenVX driver is not supposed to free the buffer since it's a handle ) when the context is deinitialized, so when the OpenCV try to free the same buffer, it causes a double free error.

I only encountered this problem with the x86_64 simulator driver, when the program runs on the target device using vendor-provided NPU driver, everything works well.

Here is a short program to reproduce this error:

#include <iostream>
#include <opencv2/core.hpp>
#include <vsi_nn_pub.h>

int main(int argc, char* argv[]) {
    vsi_status err = VSI_SUCCESS;

    auto matIn = cv::Mat(4, 4, CV_32F);
    auto matOut = cv::Mat(4, 4, CV_32F);

    cv::randu(matIn, -1.0F, 1.0F);
    matOut.setTo(0.0F);

    auto context = vsi_nn_CreateContext();
    auto graph = vsi_nn_CreateGraph(context, 2, 1);
    vsi_nn_SetGraphInputs(graph, nullptr, 1);
    vsi_nn_SetGraphOutputs(graph, nullptr, 1);

    vsi_nn_tensor_attr_t attr = {};
    attr.dtype.fmt = VSI_NN_DIM_FMT_NCHW;
    attr.dim_num = 4;
    attr.size[0] = 4;
    attr.size[1] = 4;
    attr.size[2] = 1;
    attr.size[3] = 1;
    attr.dtype.vx_type = VSI_NN_TYPE_FLOAT32;
    attr.dtype.qnt_type = VSI_NN_QNT_TYPE_NONE;
    attr.is_const = 0;
    attr.vtl = 0;

    auto tensorIn = vsi_nn_AddTensorFromHandle(
        graph, VSI_NN_TENSOR_ID_AUTO, &attr, matIn.data);
    auto tensorOut = vsi_nn_AddTensorFromHandle(
        graph, VSI_NN_TENSOR_ID_AUTO, &attr, matOut.data);

    auto nodeReLU = vsi_nn_AddNode(graph, VSI_NN_OP_RELU, 1, 1, nullptr);
    nodeReLU->uid = 100;
    nodeReLU->input.tensors[0] = tensorIn;
    nodeReLU->output.tensors[0] = tensorOut;

    graph->input.tensors[0] = tensorIn;
    graph->output.tensors[0] = tensorOut;

    err = vsi_nn_SetupGraph(graph, vx_false_e);
    err = vsi_nn_VerifyGraph(graph);
    err = vsi_nn_rnn_RunGraph(graph);

    std::cout << "[Input]\n" << matIn << std::endl;
    std::cout << "[Output]\n" << matOut << std::endl;

    vsi_nn_ReleaseGraph(&graph);
    vsi_nn_ReleaseContext(&context);

    return err;
}

callstack:

raise (raise:49)
abort (abort:60)
__libc_message (__libc_message:173)
malloc_printerr (malloc_printerr:0)
_int_free (_int_free:455)
__libc_free (__libc_free:28)
cv::StdMatAllocator::deallocate(cv::UMatData*) const (cv::StdMatAllocator::deallocate(cv::UMatData*) const:17)
cv::Mat::~Mat() (cv::Mat::~Mat():26)
main (/home/nullko/Documents/tim-vx/samples/ncc/test_handle_tensor.cpp:54)
__libc_start_main (__libc_start_main:53)
_start (_start:13)

I am trying to cross-compile a pytorch model for imx8 with vsi_npu with below code but I am getting the errors noted below, I have also attached a working code for normal linux targets

compilation code:

mod, params = relay.frontend.from_pytorch(scripted_model, [(input_name, input_shape)])
target_string = "llvm  -mtriple=aarch64-linux-gnu"              # imx8 host-triple
kwargs = {"cc": "aarch64-linux-gnu-gcc", 'fcompile': False}
disabled_passes = ["AlterOpLayout"]                                     # same error with None
with tvm.transform.PassContext(opt_level=3, disabled_pass=disabled_passes):
        mod = vsi_npu.partition_for_vsi_npu(mod, params)           # runs fine
        lib = relay.build(mod, target_string, params=params)        # error #
lib.export_library(build_dir / 'deploy.so',  **kwargs)

error at compile step: (relay taken from quantized pytorch trace)

This is important----> name_node.value() == tvmgen_default_vsi_npu_593
GraphMakerImpl::Create
TensorMakerImpl::InferCall: qnn.quantize
E [GetMapedTensor:140]Tensor has not beed inserted in tensor map.
python: /code/tim-vx-lib/TIM-VX/src/tim/transform/layout_inference.cc:141: std::shared_ptr<tim::vx::Tensor> tim::transform::layout_inference_impl::LayoutInferContext::GetMapedTensor(const std::shared_ptr<tim::vx::Tensor>&) const: Assertion `false' failed.
Aborted (core dumped)

error at compile step: (relay taken from simple pytorch trace)

This is important----> name_node.value() == tvmgen_default_vsi_npu_57
GraphMakerImpl::Create
TensorMakerImpl::InferCall: nn.conv2d
TensorMakerImpl::InferCall: add
TensorMakerImpl::InferCall: image.resize2d
TensorMakerImpl::InferCall: add
TensorMakerImpl::InferCall: image.resize2d
TensorMakerImpl::InferCall: add
TensorMakerImpl::InferCall: image.resize2d
W [vsi_nn_SortGraphNode:1378]Unprocessed node 7
W [vsi_nn_SetupGraph:706]Sort graph nodes failure.
Fatal error: compile to binary failed

Working Code without vsi_npu: (quantized/normal/tuned/notune all cases)

mod, params = relay.frontend.from_pytorch(scripted_model, [(input_name, input_shape)])
target_string = "llvm -mtriple=x86_64-linux-gnu"              # linux host-triple
with tvm.transform.PassContext(opt_level=3, disabled_pass=None):
        lib = relay.build(mod, target_string, params=params)      # no error #
lib.export_library(build_dir / 'deploy.so')

I have installed using cmake instructions from https://github.com/VeriSilicon/tvm/blob/vsi_npu/README.VSI.md Please guide me on the error

Hi, I've learned that it's possible to use Tensorflow Lite NNAPI on vim3 Android device,

I have /system/lib/libneuralnetworks.so file in my Android 9 OS. How to make sure NPU is used? I benchmarked my model and it seems NPU is not used during TFLite 8bit inference, because speed is 10x slower and there is no difference between channel and tensor quantised models.

also in dmesg after I run my bench:

[ 4907.441064] type=1400 audit(1293888104.720:419): avc: denied { read } for pid=7157 
comm="benchmar" path="/data/local/tmp/build/model/model.tflite" 
dev="mmcblk0p20" ino=261748 scontext=u:r:hal_neuralnetworks_default:s0 tcontext=u:object_r:shell_data_file:s0 
tclass=file permissive=1

Hello, I'm going to use TIM-VX to use the VIM3 development board. I have some questions because it's my first time dealing with applications related to ovxlib and npu. I'm going to use TIM-VX for research purposes. The goal of the study is utilizing NPU effectively. I wonder that,

1. if the NPU of VIM3 can be used through TIM-VX,
2. if it can be used, do all layers defined at TIM-VX can be run on NPU,
3. Is there any method to check to know that I am utilizing NPU when run time,
4. and how much cpu usage of tim-vx is. TIM-VX seems like a very interesting subject. Thank you for reading it!!

Hi, could you provide minimal example, for tflite inference with vx-delegate applied? Im not sure how to create and apply vx-delegate to tflite model. Thanks!

Hello, I am using tim-vx for tensorflow lite delegate and tengine, And I encountered graph verification fail error. What can I try to do for verification error? what may I check for this error Thank you!

Intent: Up to now TIM-VX allocates necessary tensor buffer on host memory, and vsi_nn_AddTensorFromHandle does accept a non-null data argument, so this PR enables a new usage of reusing data buffers from outsides for input and output tensors.

This PR extends and replaces #297. And I've run a test using a yolov4-tiny-uint8 model (from tengine) on both x86_64_linux (sim) and aarch64 (hardware), and the test succeeded stably.

API changes

Add such public APIs:

1. virtual bool Tensor::FlushCacheForHandle() = 0;
2. virtual bool Tensor::InvalidateCacheForHandle() = 0;
3. virtual void* Tensor::map(bool invalidate_cpu_cache = false) = 0;
4. virtual void Tensor::unmap() = 0;
5. virtual std::shared_ptr<Tensor> Graph::CreateIOTensor(const TensorSpec& spec, void* data = nullptr) = 0;

And add:

1. corresponding member functions in TensorImpl and GraphImpl classes
2. TensorImpl::TensorImpl(Graph* graph, const TensorSpec& spec, void* data = nullptr);

Also redefined TensorImpl::data_ from const void * to void *, which is to indicate this may work as a cache area and be updated by Tensor::RefillCacheFromHandle and Tensor::map(true).

Running valgrind on our implemented software, we found there were many errors coming from Tim-VX with regards to the graph and context instances. To verify if this was an issue with our software or something happening internally, we ran it against the lenet example provided in this repo and saw the same output. I've attached the valgrind log here. It's hard for me to tell if this is a tim-vx issue or an openvx issue (or potentially an us issue) so I'm hoping this log can help figure out what may be happening.

The trend I see in the log is that it happens on all tim-vx functions: creating, initalizing, validating (compile), executing (run), and destroying.

The command we ran to get this output: valgrind --tool=memcheck --leak-check=full --error-limit=no --log-file="{filename}" ./{program executable filename}

valgrindOutput3.txt

While debugging my own quantized models created with tim-vx, I came across the issue of binary classifiers failing the validation step. This seemed to happen across different models I had although everything was set up properly. Further, my models worked ok with a non tim-vx implementation.

I am able to reproduce the issue seen in my models with the lenet example provided in this repo by doing the following:

1. Run example - note that it works.
2. Modify the following shapes (change 10's to 3's) so it becomes: tim::vx::ShapeType fc4_weight_shape({500, 3}); tim::vx::ShapeType fc4_bias_shape({3}); tim::vx::ShapeType output_shape({3, 1});
3. Save and run example - note that it works.
4. Repeat step 2 but change the 3's to 2's.
5. Save and run example - note that it does not pass the validation step (graph->Compile()).

I have tested this with models of varying amounts of classes and it only has a problem with 2-class classifiers. I noted there was a similar comment to this in issue 167 but it looks like the issue was resolved without discussing this part further.

The output error codes I get are: D [operator():134]vsi_nn_SetupGraph Returned 0 D [operator():140]vsi_nn_VerifyGraph Returned -1

Some notes:

1. If I remove the softmax layer, the model passes validation.
2. If I leave all layers in but run a non-quantized version of the model, the model passes validation.
3. If I leave the network as is (all layers and quantized), the model doesn't pass validation.

Hello, I try to use galcore.ko from libs in release https://github.com/VeriSilicon/TIM-VX/releases/tag/v1.1.42, but have next output:

/alexrak/vim3l_aarch64/lib # insmod galcore.ko
insmod: can't insert 'galcore.ko': invalid module format
/alexrak/vim3l_aarch64/lib # dmesg
[1128771.137591] galcore: module PLT section(s) missing
[1128771.161401] galcore: module PLT section(s) missing

Can you help me to find out from error with PLT section missing?

Change type of shape from int to uint reason is from https://github.com/VeriSilicon/TIM-VX/issues/376 added ut for negative dimension

Type: Code Improvement Signed-off-by: Feiyue Chen [email protected]