First time trying this, the following script creates test.mp4.ffindex then ends, no error message:

import vapoursynth as vs
core = vs.core
clip = core.ffms2.Source(source='test.mp4')

clip = core.resize.Bicubic(clip, format=vs.RGBS, matrix_in_s='709')
clip = core.rife.RIFE(clip)
print('not reached')

using Arch Linux, following the compilation instructions. using list_gpu doesn't work either, the plain cli works.

1st: thanks for porting RIFE to Vapoursynth. :)

I'm using a Geforce GTX 1070ti 8GB VRAM (driver version 466.47) on Windows 10 64bit. Using:

clip = core.misc.SCDetect(clip=clip,threshold=0.100)
# adjusting color space from YUV420P8 to RGBS for vsRIFE
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="709", range_s="limited")
# adjusting frame count&rate with RIFE
clip = core.rife.RIFE(clip, sc=True) # new fps: 48

everything works fine, using:

clip = core.misc.SCDetect(clip=clip,threshold=0.100)
# adjusting color space from YUV420P8 to RGBS for vsRIFE
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="709", range_s="limited")
# adjusting frame count&rate with RIFE
clip = core.rife.RIFE(clip, tta=True,sc=True) # new fps: 48

I always get: vkWaitForFences failed -4 and then a black frame and vkQueueSubmit failed -4 for each interpolated frame, after a few frames. Happens a few frames later when using model=1. Also happens with sc=False or fpp32=True. Okay, problems does not happen for SD content, but only for HD content. Lowering gpu_thread to 1, doesn't help either. First I thought that tta would consume to much VRAM, but there are still nearly 6GB free,

Does tta have some special system requirements for HD content (normal 1920x1080)?

Cu Selur

I get stuck at meson trying to find the vs runtime. I tried specifying the sdk lib folder and binary folders manually in meson.build but no luck. What am I missing?

Hi I'm trying to build the project on Ubuntu 18.04, but

Cloning into 'build'...
Submodule 'subprojects/ncnn' (https://github.com/Tencent/ncnn.git) registered for path 'subprojects/ncnn'
Cloning into '/home/selur/workspace/hybrid-vapoursynth-addon/build/build/subprojects/ncnn'...
Submodule path 'subprojects/ncnn': checked out '57ad2c138cd334760cddeacb6876544f8a46e224'
Submodule 'glslang' (https://github.com/KhronosGroup/glslang) registered for path 'subprojects/ncnn/glslang'
Submodule 'python/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'subprojects/ncnn/python/pybind11'
Cloning into '/home/selur/workspace/hybrid-vapoursynth-addon/build/build/subprojects/ncnn/glslang'...
Cloning into '/home/selur/workspace/hybrid-vapoursynth-addon/build/build/subprojects/ncnn/python/pybind11'...
Submodule path 'subprojects/ncnn/glslang': checked out '4afd69177258d0636f78d2c4efb823ab6382a187'
Submodule path 'subprojects/ncnn/python/pybind11': checked out 'd587a2fd17dc89c0b2d4b6dc7715594bed45fb58'
The Meson build system
Version: 0.60.2
Source dir: /home/selur/workspace/hybrid-vapoursynth-addon/build/build
Build dir: /home/selur/workspace/hybrid-vapoursynth-addon/build/build/build
Build type: native build
Project name: RIFE-ncnn-Vulkan
Project version: 2
C++ compiler for the host machine: /usr/bin/g++-11 (gcc 11.1.0 "g++-11 (Ubuntu 11.1.0-1ubuntu1~18.04.1) 11.1.0")
C++ linker for the host machine: /usr/bin/g++-11 ld.bfd 2.30
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: /usr/bin/pkg-config (0.29.1)
Run-time dependency vapoursynth found: YES 57

Executing subproject ncnn method cmake 

ncnn| Found CMake: /home/selur/opt/vapoursynth/bin/cmake (3.14.6)

| Configuring the build directory with CMake version 3.14.6
| Running CMake with: -G Ninja -DCMAKE_INSTALL_PREFIX=/usr/local -DNCNN_OPENMP=False -DNCNN_INSTALL_SDK=False -DNCNN_PIXEL_ROTATE=False -DNCNN_PIXEL_AFFINE=False -DNCNN_PIXEL_DRAWING=False -DNCNN_VULKAN=True -DNCNN_VULKAN_ONLINE_SPIRV=True -DNCNN_BUILD_BENCHMARK=False -DNCNN_BUILD_TESTS=False -DNCNN_BUILD_TOOLS=False -DNCNN_BUILD_EXAMPLES=False -DNCNN_INT8=False -DWITH_LAYER_absval=False -DWITH_LAYER_argmax=False -DWITH_LAYER_batchnorm=False -DWITH_LAYER_bias=False -DWITH_LAYER_bnll=False -DWITH_LAYER_concat=True -DWITH_LAYER_convolution=True -DWITH_LAYER_crop=True -DWITH_LAYER_deconvolution=True -DWITH_LAYER_dropout=False -DWITH_LAYER_eltwise=False -DWITH_LAYER_elu=False -DWITH_LAYER_embed=False -DWITH_LAYER_exp=False -DWITH_LAYER_flatten=True -DWITH_LAYER_innerproduct=True -DWITH_LAYER_input=True -DWITH_LAYER_log=False -DWITH_LAYER_lrn=False -DWITH_LAYER_memorydata=False -DWITH_LAYER_mvn=False -DWITH_LAYER_pooling=True -DWITH_LAYER_power=False -DWITH_LAYER_prelu=True -DWITH_LAYER_proposal=False -DWITH_LAYER_reduction=False -DWITH_LAYER_relu=True -DWITH_LAYER_reshape=False -DWITH_LAYER_roipooling=False -DWITH_LAYER_scale=False -DWITH_LAYER_sigmoid=True -DWITH_LAYER_slice=True -DWITH_LAYER_softmax=False -DWITH_LAYER_split=True -DWITH_LAYER_spp=False -DWITH_LAYER_tanh=False -DWITH_LAYER_threshold=False -DWITH_LAYER_tile=False -DWITH_LAYER_rnn=False -DWITH_LAYER_lstm=False -DWITH_LAYER_binaryop=True -DWITH_LAYER_unaryop=True -DWITH_LAYER_convolutiondepthwise=False -DWITH_LAYER_padding=True -DWITH_LAYER_squeeze=False -DWITH_LAYER_expanddims=False -DWITH_LAYER_normalize=False -DWITH_LAYER_permute=False -DWITH_LAYER_priorbox=False -DWITH_LAYER_detectionoutput=False -DWITH_LAYER_interp=True -DWITH_LAYER_deconvolutiondepthwise=False -DWITH_LAYER_shufflechannel=False -DWITH_LAYER_instancenorm=False -DWITH_LAYER_clip=True -DWITH_LAYER_reorg=False -DWITH_LAYER_yolodetectionoutput=False -DWITH_LAYER_quantize=False -DWITH_LAYER_dequantize=False -DWITH_LAYER_yolov3detectionoutput=False -DWITH_LAYER_psroipooling=False -DWITH_LAYER_roialign=False -DWITH_LAYER_packing=True -DWITH_LAYER_requantize=False -DWITH_LAYER_cast=True -DWITH_LAYER_hardsigmoid=False -DWITH_LAYER_selu=False -DWITH_LAYER_hardswish=False -DWITH_LAYER_noop=False -DWITH_LAYER_pixelshuffle=True -DWITH_LAYER_deepcopy=False -DWITH_LAYER_mish=False -DWITH_LAYER_statisticspooling=False -DWITH_LAYER_swish=False -DWITH_LAYER_gemm=False -DWITH_LAYER_groupnorm=False -DWITH_LAYER_layernorm=False -DWITH_LAYER_softplus=False -DWITH_LAYER_gru=False -DWITH_LAYER_multiheadattention=False -DWITH_LAYER_gelu=False
| - build directory:          /home/selur/workspace/hybrid-vapoursynth-addon/build/build/build/subprojects/ncnn/__CMake_build
| - source directory:         /home/selur/workspace/hybrid-vapoursynth-addon/build/build/subprojects/ncnn
| - toolchain file:           /home/selur/workspace/hybrid-vapoursynth-addon/build/build/build/subprojects/ncnn/__CMake_build/CMakeMesonToolchainFile.cmake
| - preload file:             /home/selur/workspace/hybrid-vapoursynth-addon/build/build/build/meson-private/data/preload.cmake
| - trace args:               --trace --trace-expand --no-warn-unused-cli
| - disabled policy warnings: [CMP0025, CMP0047, CMP0056, CMP0060, CMP0065, CMP0066, CMP0067, CMP0082, CMP0089, CMP0102]

| Running with trace output on.
| Running with expanded trace output on.
| Not searching for unused variables given on the command line.
| -- CMAKE_TOOLCHAIN_FILE = /home/selur/workspace/hybrid-vapoursynth-addon/build/build/build/subprojects/ncnn/__CMake_build/CMakeMesonToolchainFile.cmake
| -- CMAKE_INSTALL_PREFIX = /usr/local
| -- NCNN_VERSION_STRING = 1.0.20211127
| -- The C compiler identification is GNU 11.1.0
| -- The CXX compiler identification is GNU 11.1.0
| -- Check for working C compiler: /usr/bin/gcc-11
| -- Check for working C compiler: /usr/bin/gcc-11 -- works
| -- Detecting C compiler ABI info
| -- Detecting C compiler ABI info - done
| -- Detecting C compile features
| -- Detecting C compile features - done
| -- Check for working CXX compiler: /usr/bin/g++-11
| -- Check for working CXX compiler: /usr/bin/g++-11 -- works
| -- Detecting CXX compiler ABI info
| -- Detecting CXX compiler ABI info - done
| -- Detecting CXX compile features
| -- Detecting CXX compile features - done
| -- Target arch: x86
| -- Looking for pthread.h
| -- Looking for pthread.h - found
| -- Looking for pthread_create
| -- Looking for pthread_create - not found
| -- Check if compiler accepts -pthread
| -- Check if compiler accepts -pthread - yes
| -- Found Threads: TRUE
| -- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
| -- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
| -- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
| -- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
| -- Performing Test COMPILER_HAS_DEPRECATED_ATTR
| -- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
| -- === CMAKE_SYSTEM_NAME is: Linux
| -- Configuring incomplete, errors occurred!
| See also "/home/selur/workspace/hybrid-vapoursynth-addon/build/build/build/subprojects/ncnn/__CMake_build/CMakeFiles/CMakeOutput.log".
| See also "/home/selur/workspace/hybrid-vapoursynth-addon/build/build/build/subprojects/ncnn/__CMake_build/CMakeFiles/CMakeError.log".

ncnn| CMake configuration: FAILED

meson.build:136:0: ERROR: Failed to configure the CMake subproject

A full log can be found at /home/selur/workspace/hybrid-vapoursynth-addon/build/build/build/meson-logs/meson-log.txt

Looking into the CMakeError.log I get:

Determining if the pthread_create exist failed with the following output:
Change Dir: /home/selur/workspace/hybrid-vapoursynth-addon/build/build/build/subprojects/ncnn/__CMake_build/CMakeFiles/CMakeTmp

Run Build Command(s):/home/selur/.local/bin/ninja cmTC_cd395 
[1/2] Building C object CMakeFiles/cmTC_cd395.dir/CheckSymbolExists.c.o
[2/2] Linking C executable cmTC_cd395
FAILED: cmTC_cd395 
: && /usr/bin/gcc-11 -pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -I/home/selur/opt/vapoursynth/include/vapoursynth -I/home/selur/opt/vapoursynth/include -I/usr/include/compute  -L/home/selur/opt/vapoursynth/lib   -rdynamic CMakeFiles/cmTC_cd395.dir/CheckSymbolExists.c.o  -o cmTC_cd395   && :
CMakeFiles/cmTC_cd395.dir/CheckSymbolExists.c.o: In function `main':
CheckSymbolExists.c:(.text.startup+0x3): undefined reference to `pthread_create'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

File /home/selur/workspace/hybrid-vapoursynth-addon/build/build/build/subprojects/ncnn/__CMake_build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:
/* */
#include <pthread.h>

int main(int argc, char** argv)
{
  (void)argv;
#ifndef pthread_create
  return ((int*)(&pthread_create))[argc];
#else
  (void)argc;
  return 0;
#endif
}

any idea what I can do to make it work?

Trying to build rife, I'm stuck at:

...
FAILED: librife.so.p/RIFE_plugin.cpp.o 
c++ -Ilibrife.so.p -I. -I.. -Isubprojects/ncnn/__CMake_build -I../subprojects/ncnn/__CMake_build -Isubprojects/ncnn -I../subprojects/ncnn -I../subprojects/ncnn/glslang -Isubprojects/ncnn/__CMake_build/include -I../subprojects/ncnn/__CMake_build/include -I../subprojects/ncnn/src/layer/x86 -I../subprojects/ncnn/src -Isubprojects/ncnn/__CMake_build/src -I../subprojects/ncnn/__CMake_build/src -I../subprojects/ncnn/src/layer -I/usr/include -I/home/selur/opt/vapoursynth/include/vapoursynth -I/home/selur/opt/vapoursynth/include -I/usr/include/compute -fvisibility=hidden -flto=auto -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -O3 -std=c++20 -mfpmath=sse -msse2 -fno-math-errno -fno-trapping-math -pipe -O3 -Wno-attributes -fPIC -fvisibility=hidden -fno-strict-aliasing -Wno-reorder -fPIC -pthread -MD -MQ librife.so.p/RIFE_plugin.cpp.o -MF librife.so.p/RIFE_plugin.cpp.o.d -o librife.so.p/RIFE_plugin.cpp.o -c ../RIFE/plugin.cpp
../RIFE/plugin.cpp:30:10: fatal error: semaphore: Datei oder Verzeichnis nicht gefunden
   30 | #include <semaphore>
      |          ^~~~~~~~~~~
compilation terminated.
[333/339] Compiling C++ object subprojects/ncnn/libcm_SPIRV.a.p/glslang_SPIRV_SpvBuilder.cpp.o
[334/339] Compiling C++ object subprojects/ncnn/libcm_SPIRV.a.p/glslang_SPIRV_SpvPostProcess.cpp.o
[335/339] Compiling C++ object librife.so.p/RIFE_warp.cpp.o
../RIFE/warp.cpp:110: warning: ignoring ‘#pragma omp parallel’ [-Wunknown-pragmas]
  110 |     #pragma omp parallel for num_threads(opt.num_threads)
      | 
../RIFE/warp.cpp: In member function ‘virtual int Warp::destroy_pipeline(const ncnn::Option&)’:
../RIFE/warp.cpp:82:42: warning: unused parameter ‘opt’ [-Wunused-parameter]
   82 | int Warp::destroy_pipeline(const Option& opt)
      |                            ~~~~~~~~~~~~~~^~~
../RIFE/warp.cpp: In member function ‘virtual int Warp::forward(const std::vector<ncnn::Mat>&, std::vector<ncnn::Mat>&, const ncnn::Option&) const’:
../RIFE/warp.cpp:96:100: warning: unused parameter ‘opt’ [-Wunused-parameter]
   96 | int Warp::forward(const std::vector<Mat>& bottom_blobs, std::vector<Mat>& top_blobs, const Option& opt) const
      |                                                                                      ~~~~~~~~~~~~~~^~~
[336/339] Compiling C++ object librife.so.p/RIFE_rife.cpp.o
../RIFE/rife.cpp: In member function ‘int RIFE::process_cpu(const ncnn::Mat&, const ncnn::Mat&, float, ncnn::Mat&) const’:
../RIFE/rife.cpp:1214:15: warning: unused variable ‘channels’ [-Wunused-variable]
 1214 |     const int channels = 3;//in0image.elempack;
      |               ^~~~~~~~
[337/339] Compiling C++ object subprojects/ncnn/libcm_SPIRV.a.p/glslang_SPIRV_GlslangToSpv.cpp.o
../subprojects/ncnn/glslang/SPIRV/GlslangToSpv.cpp: In member function ‘spv::Id {anonymous}::TGlslangToSpvTraverser::getSampledType(const glslang::TSampler&)’:
../subprojects/ncnn/glslang/SPIRV/GlslangToSpv.cpp:3698:34: warning: this statement may fall through [-Wimplicit-fallthrough=]
 3698 |             builder.addCapability(spv::CapabilityFloat16ImageAMD);
      |             ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../subprojects/ncnn/glslang/SPIRV/GlslangToSpv.cpp:3699:9: note: here
 3699 |         case glslang::EbtUint64:     return builder.makeUintType(64);
      |         ^~~~
../subprojects/ncnn/glslang/SPIRV/GlslangToSpv.cpp:3701:34: warning: this statement may fall through [-Wimplicit-fallthrough=]
 3701 |             builder.addCapability(spv::CapabilityFloat16ImageAMD);
      |             ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../subprojects/ncnn/glslang/SPIRV/GlslangToSpv.cpp:3703:9: note: here
 3703 |         default:
      |         ^~~~~~~
ninja: build stopped: subcommand failed.

I attached the whole log: rife.log My guess is that I'm missing some dependency, but no clue which.

Any idea how to fix this?

RIFE r6 works fine.

Vapoursynth R59+Python3.10.5

Here is the test.vpy

import vapoursynth as vs
from vapoursynth import core

clip = video_in

clip = clip.resize.Bilinear(format=vs.RGBS, matrix_in_s="709")

#clip = core.rife.RIFE(clip=clip, model=9, multiplier=2)
#R6 —— No crash

#clip = core.rife.RIFE(clip=clip, model=9, fps_num=48, fps_den=1)
#R7

clip = core.rife.RIFE(clip=clip, model=9, factor_num=2, factor_den=1)
#R9

clip = clip.resize.Bilinear(format=vs.YUV420P8, matrix_s="709")

clip.set_output()

Seeing that https://github.com/HolyWu/vs-rife has an option: 'scale: Controls the process resolution for optical flow model. Try scale=0.5 for 4K video. Must be 0.25, 0.5, 1.0, 2.0, or 4.0.' I was wondering whether this option could be supported here too? (I prefer this filter since it doesn't require a 5.4GB PyTorch installaiton.)

If I run RIFE twice in a script, and open the script multiple times, sometimes I get random junk output from RIFE. Also, the process fails to terminate properly.

Interestingly enough, if RIFE is set only once in the script, neither of these problems occur.

Also for general purposes, model 2.4 gives better output than model 3.1, and according to some, 2.3 gives better output than 2.4. Any chance to have model 2.3 available?

Finally, I have dual-graphics in my laptop. By default it's using the integrated Intel. If I set gpu_id=1, it says invalid GPU id. How can I use the dedicated graphics card?

First, I'm not sure this has to do with this Vapoursynth wrapper around RIFE-ncnn-Vulkan or if the problem is upstream; I have no current means to test RIFE-ncnn-vulkan nor RIFE iself.

If I create a 50 fps movie of a single circle moving in a straight line across the screen at a constant speed, then use VapourSynth-RIFE-ncnn-Vulkan to interpolate it to 60 fps, I often get weird frames.

Motion for frames X₀ … Xₙ is smooth, with the circle travelling a smaller distance than on the original video — that's what I expect, as the frame display time is shorter.

Then, on frame Xₙ₊₁, the circle does a larger leap than usual, on frame Xₙ₊₂ the circle is slightly behind the one on frame Xₙ₊₁. Then it resumes creating the correct frames at Xₙ₊₃, until a couple of frames later the problem reoccurs.

I see no regular pattern, it's not like I can predict where the bad frames will be. In my case, got the problem on frame numbers (0-based) {6,7}, {12,13}, {24,25}, {42,43}, {48,49}, {54,55}, {78,79}, {84,85}, {90,91}… so it happens on some multiples of 6 (1, 2, 4, 7, 8, 9, 13, 14, 15), but not all of them.

I join the scripts to demonstrate, if that can help diagnose the issue. vs-rife-50-to-60.sh creates the video images, converts them to a 50 fps H.264 video, then uses vspipe to process them via vs-rife-50-to-60.py that uses VapourSynth-RIFE-ncnn-Vulkan to interpolate to 60 fps; vspipe then outputs to ffmpeg that spits out an image for every frame. Note that the ffmpeg output image numbers are 1-based.

vs-rife-50-to-60.sh.txt vs-rife-50-to-60.py.txt

failed to load it in vpy with following error logs

>>> core.std.LoadPlugin(".\\vapoursynth64\\plugins\\libmvtools.dll")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src\cython\vapoursynth.pyx", line 2723, in vapoursynth.Function.__call__
vapoursynth.Error: Plugin .\vapoursynth64\plugins\libmvtools.dll already loaded (com.nodame.mvtools) from D:/Programs/mpvnet/vapoursynth64/plugins/libmvtools.dll
>>> core.std.LoadPlugin(".\\vapoursynth64\\plugins\\RIFE.dll")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src\cython\vapoursynth.pyx", line 2723, in vapoursynth.Function.__call__
vapoursynth.Error: Failed to load .\vapoursynth64\plugins\RIFE.dll. GetLastError() returned 126. The file you tried to load or one of its dependencies is probably missing.

Note that libmvtools.dll can be loaded successfully...

Can you provide an example script (or documentation) of how to use this filter with mpv?

(I've already configured mpv with vapoursynth and it's working, but I'm unsure how to use this filter)

EDIT: I think I've figured it out but the current release only supports 32-bit. Can you compile a build of the latest version that supports 8-bit for me to test?

styler00dollar forked your filter and added:

10 = rife-v4.3 (ensemble=False / fast=True) 11 = rife-v4.3 (ensemble=True / fast=False) 12 = rife-v4.4 (ensemble=False / fast=True) 13 = rife-v4.4 (ensemble=True / fast=False) https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan

and described them in this commentary: https://github.com/nihui/rife-ncnn-vulkan/issues/50#issuecomment-1229203013

Please add both models with two variants each: fastest and highest quality, so 4 new options in total.

I think that with the new graphics card: NVIDIA GeForce RTX 4090 many people might be interested in the highest quality version of the models. Also for the interpolation of lower resolution files the highest quality would also be very useful.