Hi,

I’m currently using CUDNN to write a deep learning super-resolution sample. But the function cudnnConvolutionBiasActivationForward will return status CUDNN_STATUS_EXECUTION_FAILED. I check out the API Reference DOC and only find the description "The function failed to launch on the GPU." for this state.

I also found a similar post in the NVIDIA forum, but there is no reply in this post. I found that the memory occupied by the graphics card will rise rapidly when running the sample. But I can't debug without specific diagnostic information. Could you give any suggestions for this issue?

Thanks a lot in advance!

Hi,

I’m currently using CUDNN to write a deep learning super-resolution sample. But I found that support for LeakyReLU is not mentioned in the document. But LeakyReLU is in my pre-trained model. Could you please give any suggestions or sample code to solve this problem?

Thanks a lot in advance!

A Dockerfile to reproduce:

FROM nvidia/cuda:11.4.2-cudnn8-devel-ubuntu20.04
ENV TZ=Europe/London
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ >/etc/timezone
RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y build-essential git git-lfs wget vim software-properties-common unzip python3-pip
RUN update-alternatives --install /usr/bin/python python $(which python3) 10
RUN pip install --upgrade numpy astor
WORKDIR /workdir
RUN wget https://github.com/bazelbuild/bazel/releases/download/0.26.1/bazel-0.26.1-installer-linux-x86_64.sh
RUN chmod +x bazel-0.26.1-installer-linux-x86_64.sh && ./bazel-0.26.1-installer-linux-x86_64.sh

RUN git clone https://github.com/NVIDIA/cudnn-frontend.git
RUN git clone --branch r1.15.5+nv21.10 --single-branch https://github.com/NVIDIA/tensorflow.git
WORKDIR /workdir/tensorflow
ENV TF_ENABLE_XLA=0 \
    TF_NEED_OPENCL_SYCL=0 \
    TF_NEED_ROCM=0 \
    TF_NEED_CUDA=1 \
    TF_NEED_TENSORRT=0 \
    TF_CUDA_VERSION=11 \
    TF_CUBLAS_VERSION=11 \
    TF_NCCL_VERSION=2 \
    TF_CUDNN_VERSION=8 \
    TF_CUDA_PATHS="/usr/include,/usr/lib/x86_64-linux-gnu,/usr/local/cuda/include,/usr/local/cuda/lib64,/usr/local/cuda/bin,/usr/local/cuda" \
    TF_CUDA_COMPUTE_CAPABILITIES=3.5,5.0,5.2,6.1,7.0,7.5,8.6 \
    CC_OPT_FLAGS="-march=sandybridge -mfma -mfpmath=both -fopenmp"
RUN PYTHON_BIN_PATH=$(which python) ./configure
RUN bazel build --config=opt --config=noaws --config=nogcp --config=nohdfs --config=noignite --config=nokafka //tensorflow/tools/pip_package:build_pip_package

Error message:

ERROR: /workdir/tensorflow/tensorflow/stream_executor/cuda/BUILD:343:1: C++ compilation of rule '//tensorflow/stream_executor/cuda:cudnn_plugin' failed (Exit 1)
In file included from bazel-out/host/bin/external/cudnn_frontend_archive/_virtual_includes/cudnn_frontend/third_party/cudnn_frontend/include/cudnn_frontend.h:116,
                 from tensorflow/stream_executor/cuda/cuda_dnn.cc:50:
bazel-out/host/bin/external/cudnn_frontend_archive/_virtual_includes/cudnn_frontend/third_party/cudnn_frontend/include/cudnn_frontend_ExecutionPlanCache.h:29:10: fatal error: cudnn_frontend_OperationGraph.h: No such file or directory
   29 | #include <cudnn_frontend_OperationGraph.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Target //tensorflow/tools/pip_package:build_pip_package failed to build

while using git clone --branch v0.4.1 --single-branch https://github.com/NVIDIA/cudnn-frontend.git seems to work.

Hi, I'm checking mix precision programming: https://developer.nvidia.com/blog/mixed-precision-programming-cuda-8/ Does cudnn frontend has any support for half2 type convolution?

Can give samples about int8x32 convolution calculation using cudnn backend API? 1、How to create xTensor, wTensor and so on? 2、How to create conv_op node? 3、Other creation.

This resolves the downstream issue in pytorch when we try to enable this check. More information can be found at https://github.com/pytorch/pytorch/pull/81012

hello, thx for your great work @YangXu1990uiuc

i am wondering if we can find a mapping relationship between the execution_engine with the original algorithm like CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD/CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM/CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_​PRECOMP_GEMM.

Hi there,

I'm not sure where to open this issue, so I'm dropping it here since I cannot find the appropriate channel.

I have been trying to crunch some numbers with several python packages. but none of those works with the RTX A4000. After debugging for days, I found the issue is not my environment but the GPU.

More specific, I have been trying to use the following packages:

MXNet CuPy

My first impression was that there was something wrong with my environment. Then I tried to just run a docker container to test the GPU along with MXNet package:

docker pull mxnet/python:gpu

Trying to run either mxnet or cupy hangs... it does not compute anything...

Then I tried the default nvidia container:

docker run --rm --gpus all nvidia/cuda nvidia-smi

The output seems ok:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.44       Driver Version: 495.44       CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A4000    Off  | 00000000:01:00.0  On |                  Off |
| 41%   49C    P2    37W / 140W |    787MiB / 16109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

It is very confusing because I can run Pytorch, TensoRT etc in this machine...

Then I tried to run the installer over again for the following theNVIDIA Guide for CUDNN:

$cp -r /usr/src/cudnn_samples_v8/ $HOME
$ cd  $HOME/cudnn_samples_v8/mnistCUDNN
$make clean && make
$ time ./mnistCUDNN

Then what I get is:

Executing: mnistCUDNN
cudnnGetVersion() : 8300 , CUDNN_VERSION from cudnn.h : 8300 (8.3.0)
Host compiler version : GCC 7.5.0

There are 1 CUDA capable devices on your machine :
device 0 : sms 48  Capabilities 8.6, SmClock 1560.0 Mhz, MemSize (Mb) 16109, MemClock 7001.0 Mhz, Ecc=0, boardGroupID=0
Using device 0

Testing single precision
Loading binary file data/conv1.bin
Loading binary file data/conv1.bias.bin
Loading binary file data/conv2.bin
Loading binary file data/conv2.bias.bin
Loading binary file data/ip1.bin
Loading binary file data/ip1.bias.bin
Loading binary file data/ip2.bin
Loading binary file data/ip2.bias.bin
Loading image data/one_28x28.pgm
Performing forward propagation ...
Testing cudnnGetConvolutionForwardAlgorithm_v7 ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: -1.000000 time requiring 178432 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: -1.000000 time requiring 184784 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 7: -1.000000 time requiring 2057744 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
Testing cudnnFindConvolutionForwardAlgorithm ...

^^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.017408 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.018432 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.027648 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: 0.060416 time requiring 178432 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: 0.096256 time requiring 184784 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_EXECUTION_FAILED for Algo 7: -1.000000 time requiring 2057744 memory
Testing cudnnGetConvolutionForwardAlgorithm_v7 ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: -1.000000 time requiring 2450080 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 7: -1.000000 time requiring 1433120 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: -1.000000 time requiring 2000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: -1.000000 time requiring 128000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: -1.000000 time requiring 4656640 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
Testing cudnnFindConvolutionForwardAlgorithm ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.037888 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.046080 time requiring 2000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: 0.055296 time requiring 2450080 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.057344 time requiring 128000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: 0.078848 time requiring 4656640 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_EXECUTION_FAILED for Algo 7: -1.000000 time requiring 1433120 memory
Resulting weights from Softmax:
0.0000000 0.9999399 0.0000000 0.0000000 0.0000561 0.0000000 0.0000012 0.0000017 0.0000010 0.0000000 
Loading image data/three_28x28.pgm
Performing forward propagation ...
Testing cudnnGetConvolutionForwardAlgorithm_v7 ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: -1.000000 time requiring 178432 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: -1.000000 time requiring 184784 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 7: -1.000000 time requiring 2057744 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
Testing cudnnFindConvolutionForwardAlgorithm ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.016384 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.016384 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.017408 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: 0.055296 time requiring 178432 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: 0.060416 time requiring 184784 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_EXECUTION_FAILED for Algo 7: -1.000000 time requiring 2057744 memory
Testing cudnnGetConvolutionForwardAlgorithm_v7 ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: -1.000000 time requiring 2450080 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 7: -1.000000 time requiring 1433120 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: -1.000000 time requiring 2000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: -1.000000 time requiring 128000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: -1.000000 time requiring 4656640 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
Testing cudnnFindConvolutionForwardAlgorithm ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.037888 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.041984 time requiring 2000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: 0.050176 time requiring 2450080 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.063488 time requiring 128000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: 0.076800 time requiring 4656640 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_EXECUTION_FAILED for Algo 7: -1.000000 time requiring 1433120 memory
Resulting weights from Softmax:
0.0000000 0.0000000 0.0000000 0.9999288 0.0000000 0.0000711 0.0000000 0.0000000 0.0000000 0.0000000 
Loading image data/five_28x28.pgm
Performing forward propagation ...
Resulting weights from Softmax:
0.0000000 0.0000008 0.0000000 0.0000002 0.0000000 0.9999820 0.0000154 0.0000000 0.0000012 0.0000006 

Result of classification: 1 3 5

Test passed!

Testing half precision (math in single precision)
Loading binary file data/conv1.bin
Loading binary file data/conv1.bias.bin
Loading binary file data/conv2.bin
Loading binary file data/conv2.bias.bin
Loading binary file data/ip1.bin
Loading binary file data/ip1.bias.bin
Loading binary file data/ip2.bin
Loading binary file data/ip2.bias.bin
Loading image data/one_28x28.pgm
Performing forward propagation ...
Testing cudnnGetConvolutionForwardAlgorithm_v7 ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: -1.000000 time requiring 28800 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: -1.000000 time requiring 178432 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: -1.000000 time requiring 184784 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 7: -1.000000 time requiring 2057744 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
Testing cudnnFindConvolutionForwardAlgorithm ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.009216 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.031744 time requiring 28800 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: 0.061440 time requiring 184784 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.187392 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: 0.235520 time requiring 178432 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_EXECUTION_FAILED for Algo 7: -1.000000 time requiring 2057744 memory
Testing cudnnGetConvolutionForwardAlgorithm_v7 ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: -1.000000 time requiring 64000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: -1.000000 time requiring 4656640 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: -1.000000 time requiring 2450080 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 7: -1.000000 time requiring 1433120 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
Testing cudnnFindConvolutionForwardAlgorithm ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.066560 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.080864 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.116736 time requiring 64000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: 0.120832 time requiring 2450080 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: 0.121856 time requiring 4656640 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_EXECUTION_FAILED for Algo 7: -1.000000 time requiring 1433120 memory
Resulting weights from Softmax:
0.0000001 1.0000000 0.0000001 0.0000000 0.0000563 0.0000001 0.0000012 0.0000017 0.0000010 0.0000001 
Loading image data/three_28x28.pgm
Performing forward propagation ...
Testing cudnnGetConvolutionForwardAlgorithm_v7 ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: -1.000000 time requiring 28800 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: -1.000000 time requiring 178432 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: -1.000000 time requiring 184784 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 7: -1.000000 time requiring 2057744 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
Testing cudnnFindConvolutionForwardAlgorithm ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.024576 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.029696 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.041984 time requiring 28800 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: 0.076800 time requiring 184784 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: 0.078848 time requiring 178432 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_EXECUTION_FAILED for Algo 7: -1.000000 time requiring 2057744 memory
Testing cudnnGetConvolutionForwardAlgorithm_v7 ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: -1.000000 time requiring 64000 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: -1.000000 time requiring 4656640 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: -1.000000 time requiring 2450080 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 7: -1.000000 time requiring 1433120 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
Testing cudnnFindConvolutionForwardAlgorithm ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.037888 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.057344 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: 0.091136 time requiring 2450080 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 5: 0.106496 time requiring 4656640 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.108544 time requiring 64000 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 6: -1.000000 time requiring 0 memory
^^^^ CUDNN_STATUS_EXECUTION_FAILED for Algo 7: -1.000000 time requiring 1433120 memory
Resulting weights from Softmax:
0.0000000 0.0000000 0.0000000 1.0000000 0.0000000 0.0000714 0.0000000 0.0000000 0.0000000 0.0000000 
Loading image data/five_28x28.pgm
Performing forward propagation ...
Resulting weights from Softmax:
0.0000000 0.0000008 0.0000000 0.0000002 0.0000000 1.0000000 0.0000154 0.0000000 0.0000012 0.0000006 

Result of classification: 1 3 5

Test passed!

real	7m18.842s
user	7m17.724s
sys	0m0.516s

This takes a LONG time to execute.

Then, for a final test, I just swapped the RTX A4000 by an old NVidia Tesla K80... Suddenly everything works like a charm...

I'm almost sure there is something wrong either with the driver or with the GPU firmware. I tried the code I have in Google Colab, the old K80, and even a Jetson AGX and it works, but not with the RTX A4000

The system information for CuPy(python) output which also takes a long time is:

import cupy as cp
print(cp.show_config())

OS                           : Linux-5.4.0-42-generic-x86_64-with-Ubuntu-18.04-bionic
Python Version               : 3.6.9
CuPy Version                 : 9.6.0
CuPy Platform                : NVIDIA CUDA
NumPy Version                : 1.19.5
SciPy Version                : 1.5.4
Cython Build Version         : 0.29.22
Cython Runtime Version       : 0.29.22
CUDA Root                    : /usr/local/cuda
nvcc PATH                    : /usr/local/cuda/bin/nvcc
CUDA Build Version           : 10020
CUDA Driver Version          : 11050
CUDA Runtime Version         : 10020
cuBLAS Version               : (available)
cuFFT Version                : 10102
cuRAND Version               : 10102
cuSOLVER Version             : (10, 3, 0)
cuSPARSE Version             : (available)
NVRTC Version                : (10, 2)
Thrust Version               : 100907
CUB Build Version            : 100800
Jitify Build Version         : 60e9e72
cuDNN Build Version          : 8204
cuDNN Version                : 8300
NCCL Build Version           : 21104
NCCL Runtime Version         : 21104
cuTENSOR Version             : None
cuSPARSELt Build Version     : None
Device 0 Name                : NVIDIA RTX A4000
Device 0 Compute Capability  : 86
Device 0 PCI Bus ID          : 0000:01:00.0
None

Apologies if this is not the right place for this issue, I really appreciate your help.

Hi all,

when building pytorch it treats warnings as errors and it barfs here:

In file included from /opt/pytorch/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_Operation.h:36,
                 from /opt/pytorch/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_OperationGraph.h:36,
                 from /opt/pytorch/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_Heuristics.h:30,
                 from /opt/pytorch/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend.h:101,
                 from /opt/pytorch/pytorch/aten/src/ATen/native/cudnn/Conv_v8.cpp:10:
/opt/pytorch/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h: In member function ‘int64_t cudnn_frontend::PointWiseDesc_v8::getPortCount() const’:
/opt/pytorch/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h:120:5: error: control reaches end of non-void function [-Werror=return-type]
  120 |     }

I think a default value should be returned to fix this.

[Enhancement] Added a CUDNN_FRONTEND_VERSION macro to cudnn_frontend. [Enhancement] Added the inline keyword to the get_plan functions to enable inclusion in multiple compilation units. [Bug fix] Replace CUDNN with CUDNN_VERSION as the right macro names.

[Enhancement] Added a CUDNN_FRONTEND_VERSION macro to cudnn_frontend. [Enhancement] Added the inline keyword to the get_plan functions to enable inclusion in multiple compilation units. [Bug fix] Replace CUDNN with CUDNN_VERSION as the right macro names.

I have try to fuse dgrad with some pointwise operation like relu, but when cudnnBackendFinalize(plan), there is a error. CUDNN_STATUS_NOT_SUPPORTED.

I am not sure if my code error. I want know, is dgrad + pointwise support now in cudnn 8.6 for Turing Arch? Could you give me some sample about dgrad fusion? In this project, only one case for dgrad, but it it only support for Ampere Arch.

I tried to run the matmul op and follow the codes in function "void run_matmul_bias_gelu" in fusion_sample.cpp. My code is as below:

int main() {
  auto ha = loadBinary<float>("/workspace/features.bin");
  auto hb = loadBinary<float>("/workspace/weight.bin");
  auto hc = loadBinary<float>("/workspace/output.bin");

  float *a_ptr;
  cudaMalloc((void **)&a_ptr, ha.size() * sizeof(float));
  cudaMemcpy(a_ptr, ha.data(), ha.size() * sizeof(float),
             cudaMemcpyHostToDevice);
  float *b_ptr;
  cudaMalloc((void **)&b_ptr, hb.size() * sizeof(float));
  cudaMemcpy(b_ptr, hb.data(), hb.size() * sizeof(float),
             cudaMemcpyHostToDevice);

  float *c_ptr;
  cudaMalloc((void **)&c_ptr, hc.size() * sizeof(float));

  const int m = ha.size() / 96;
  const int n = 96;
  const int k = 96;

  int64_t stride[3];
  int64_t a_dim[3] = {1, m, k};
  int64_t b_dim[3] = {1, k, n};
  int64_t c_dim[3] = {1, m, n};
  generateStrides(a_dim, stride, 3, CUDNN_TENSOR_NCHW);
  auto aMatrixTensor =
      cudnn_frontend::TensorBuilder()
          .setDim(3, a_dim)
          .setStride(3, stride)
          .setId('a')
          .setAlignment(
              16)  // 16B alignment is needed to run a tensor core engine
          .setDataType(CUDNN_DATA_FLOAT)
          .build();

  generateStrides(b_dim, stride, 3, CUDNN_TENSOR_NCHW);
  auto bMatrixTensor = cudnn_frontend::TensorBuilder()
                           .setDim(3, b_dim)
                           .setStride(3, stride)
                           .setId('b')
                           .setAlignment(16)
                           .setDataType(CUDNN_DATA_FLOAT)
                           .build();

  generateStrides(c_dim, stride, 3, CUDNN_TENSOR_NCHW);
  auto afterMatMulTensor = cudnn_frontend::TensorBuilder()
                               .setDim(3, c_dim)
                               .setStride(3, stride)
                               .setId('A')  // after matmul
                               .setAlignment(16)
                               .setVirtual()
                               .setDataType(CUDNN_DATA_FLOAT)
                               .build();

  std::cout << aMatrixTensor.describe() << std::endl;
  std::cout << bMatrixTensor.describe() << std::endl;
  std::cout << afterMatMulTensor.describe() << std::endl;

  // Define the matmul desc
  auto matmulDesc = cudnn_frontend::MatMulDescBuilder()
                        .setComputeType(CUDNN_DATA_FLOAT)
                        .build();
  std::cout << matmulDesc.describe() << std::endl;

  // Create a matmul Node
  auto matmul_op = cudnn_frontend::OperationBuilder(
                       CUDNN_BACKEND_OPERATION_MATMUL_DESCRIPTOR)
                       .setaMatDesc(aMatrixTensor)
                       .setbMatDesc(bMatrixTensor)
                       .setcMatDesc(afterMatMulTensor)
                       .setmatmulDesc(matmulDesc)
                       .build();
  std::cout << matmul_op.describe() << std::endl;

  std::array<cudnn_frontend::Operation const *, 1> ops = {&matmul_op};
  cudnnHandle_t handle_;
  checkCudnnErr(cudnnCreate(&handle_));

  auto opGraph = cudnn_frontend::OperationGraphBuilder()
                     .setHandle(handle_)
                     .setOperationGraph(ops.size(), ops.data())
                     .build();
  auto plan =
      get_execplan_from_heuristics_else_fall_back(std::move(opGraph), handle_);

  auto workspace_size = plan.getWorkspaceSize();
  std::cout << plan.describe() << " requires workspace " << workspace_size
            << std::endl;

  void *workspace_ptr = nullptr;
  if (workspace_size > 0) {
    cudaMalloc(&workspace_ptr, (size_t)workspace_size);
  }

  void *data_ptrs[] = {a_ptr, b_ptr, c_ptr};
  int64_t uids[] = {'a', 'b', 'c'};
  auto variantPack = cudnn_frontend::VariantPackBuilder()
                         .setWorkspacePointer(workspace_ptr)
                         .setDataPointers(3, data_ptrs)
                         .setUids(3, uids)
                         .build();

  cudnnStatus_t status = cudnnBackendExecute(handle_, plan.get_raw_desc(),
                                             variantPack.get_raw_desc());

  if (workspace_size > 0) {
    (cudaFree(workspace_ptr));
  }
  checkCudnnErr(cudnnDestroy(handle_));
  cudnn_frontend::throw_if(
      [status]() { return (status != CUDNN_STATUS_SUCCESS); },
      "Plan execute error", status);
}

I run it under cuda-memcheck and got logs as below, is there anything g wrong with my codes? I use docker:nvcr.io/nvidia/pytorch:21.10-py3 and my driver version is Driver Version: 515.65.01

root@3c261e44c675:/workspace/trt_inference/bin# cuda-memcheck ./trtexc ========= CUDA-MEMCHECK ha size is 5164704 hb size is 9216 hc size is 5164704 CUDNN_BACKEND_TENSOR_DESCRIPTOR : Datatype: CUDNN_DATA_FLOAT Id: 97 Alignment: 16 nDims 3 VectorCount: 1 vectorDimension -1 Dim [ 1,53799,96 ] Str [,5164704,96,1] isVirtual: 0 isByValue: 0 CUDNN_BACKEND_TENSOR_DESCRIPTOR : Datatype: CUDNN_DATA_FLOAT Id: 98 Alignment: 16 nDims 3 VectorCount: 1 vectorDimension -1 Dim [ 1,96,96 ] Str [,9216,96,1] isVirtual: 0 isByValue: 0 CUDNN_BACKEND_TENSOR_DESCRIPTOR : Datatype: CUDNN_DATA_FLOAT Id: 65 Alignment: 16 nDims 3 VectorCount: 1 vectorDimension -1 Dim [ 1,53799,96 ] Str [,5164704,96,1] isVirtual: 1 isByValue: 0 CUDNN_BACKEND_MATMUL_DESCRIPTOR : Math precision 0 CUDNN_BACKEND_OPERATION : OpMode: 19 X 0 Y 0 W 0 B 0 T 0 DW 0 DY 0 DX 0 C 0 A Mtrix 0x557468e919e0 B Mtrix 0x5574c343d5d0 C Mtrix 0x5574c35f37b0 P 0 MatMul 0x5574c35f39c0 Reduction 0 alphabetaType 4 Alpha: 1 1 Alpha2: 1 1 Beta: 0 0 Heuristic has 3 configurations CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR : Matmul_eng0_k24=27, numeric_notes:[CUDNN_NUMERICAL_NOTE_TENSOR_CORE,] behavior_notes:[CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION,] workSpaceSize: 0 requires workspace 0 terminate called after throwing an instance of 'cudnn_frontend::cudnnException' what(): Plan execute error ========= Error: process didn't terminate successfully ========= No CUDA-MEMCHECK results found

Hope to get help! Thanks.

I tried cuda 11.2 ,11.4 and cudnn 8.1 , 8.2, 8.4, the Sample ConvScaleBiasAct_int8 didn't work in any combination of these env, and it will return the error : "[ERROR] Exception CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED" my device type is A100, is there anything wrong?

I'm familiar with how to use CUDA and more specifically cuDNN., and I'm now trying to get started with the cuDNN backend API by experimenting with the samples in this repository. However, many of them don't seem to work for me.

My environment

• Windows 11 version 21H2 (OS Build 22000.613)
• Cuda 1.5 V11.5.119
• Cudnn 8.3.3
• Visual Studio 2019 version 16.0
• (the machine I'm compiling this on only has a GTX 1060 MaxQ, but I don't think that's very relevant yet)

General errors

• ConvBiasAct: Setting a relu clip slope doesn't work, the result is always [ERROR] Exception CUDNN_BACKEND_POINTWISE_DESCRIPTOR: SetAttribute CUDNN_ATTR_POINTWISE_RELU_LOWER_CLIP_SLOPE, Failed cudnn_status: CUDNN_STATUS_BAD_PARAM

• ConvBiasAct: After commenting out the previous code, the next issue is [ERROR] Exception CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: GetAttribute CUDNN_ATTR_ENGINE_BEHAVIOR_NOTE Failed cudnn_status: CUDNN_STATUS_BAD_PARAM. This code is gated behind a cuDNN version check, but my version is high enough so I don't know why this fails. After commenting out this and the previous issues this sample is running fine.

• ConvBiasScaleAct, ConvBiasScaleAct_int8, ConvScaleBiasAddAct sample, ConvScaleBiasAddAct sample_float and fail with the exact same [ERROR] Exception CUDNN_BACKEND_ENGINEHEUR_DESCRIPTOR: cudnn Finalize failed cudnn_status: CUDNN_STATUS_BAD_PARAM. No idea what could cause this.

• Multihead attention sample, MatmulBiasAct sample and MatmulBiasAct sample_float, all fail with [ERROR] Exception CUDNN_BACKEND_MATMUL_DESCRIPTOR: cudnnCreate Failed cudnn_status: CUDNN_STATUS_ALLOC_FAILED. I'm not sure what could cause this, I have plenty of free CPU and GPU memory.

• ConvDrelu and DgradDrelu fail with [ERROR] Exception CUDNN_BACKEND_OPERATION: SetAttribute CUDNN_ATTR_OPERATION_POINTWISE_DYDESC Failed cudnn_status: CUDNN_STATUS_BAD_PARAM Maybe pointwise derivatives are properly supported yet? There is no documentation in the API reference for this attribute name yet.

Hard-crash in BN Finalize:

• Creation of the third tensor here fails: https://github.com/NVIDIA/cudnn-frontend/blob/43709ab96c47e26eebcdac72f93f946d44ceffa8/samples/fusion_sample.cpp#L2370-L2372. I assume this is because there is some issue with the CUDNN_DATA_INT64 datatype? Specifically the cudnnBackendFinalize of the tensor descriptor crashes internally (it's not just a non-success return value, the call itself actually generates an exception).

Conclusion

In conclusion, everything seems to be very brittle at the moment. Is this because I am using some wrong versions? Or is the cuDNN backend/frontend API still too much of a work-in-progress for end users? Or is Windows support still lacking?

Hi,

I’m currently using CUDNN to write a deep learning super-resolution sample. But I found that the inference result of the model is all NAN. Then I tried to print the output of the middle layer and found that the output would be enlarged after one block, and the data exceeded the representation range after more than 20 blocks. As shown in the figure below.

But I compare it with the python version and confirmed that the input image and weight of convolutional layers are consistent. So I think I made some mistakes while building the model using the CUDNN API. Could you please take a look at my code to see if I made any obvious mistakes and give me some debugging suggestions?

Thanks a lot in advance!