Import C++ files directly from Python!

Related tags

Scripting cppimport
Overview
If you've used cppimport version 0.0.*, some new features for you! Compiler arguments, multiple source files, bug fixes! Read on.

Import C or C++ files directly from Python!

Let's try it out. First, if you're on Linux or OS X, install with the terminal command pip install cppimport.

Most cppimport users combine it with pybind11, but you can use a range of methods to create your Python extensions. Raw C extensions, Boost.Python, SWIG all work. Let's look at a simple C++ extension:

#include <pybind11/pybind11.h>

namespace py = pybind11;

int square(int x) {
    return x * x;
}


PYBIND11_MODULE(somecode, m) {
    m.def("square", &square);
}
/*
<%
setup_pybind11(cfg)
%>
*/

Save this code as somecode.cpp.

Open a python interpreter and run these lines [1]:

>>> import cppimport
>>> somecode = cppimport.imp("somecode") #This will pause for a moment to compile the module
>>> somecode.square(9)
81

I'm a big fan of the workflow that this enables, where you can edit both C++ files and Python and recompilation happens transparently.

I want things to be even easier! (Python import hook)

Add a comment containing the string "cppimport" on the first line of the file. This MUST be on the first line. This is explained further down.

// cppimport

Then import the file using the import hook:

>>> import cppimport.import_hook
>>> import somecode #This will pause for a moment to compile the module
>>> somecode.square(9)
81

What's actually going on?

The technical description: cppimport looks for a C or C++ source file that matches the requested module. If such a file exists, the file is first run through the Mako templating system. The compilation options produced by the Mako pass are then use to compile the file as a Python extension. The extension (shared library) that is produced is placed in the same folder as the C++ source file. Then, the extension is loaded.

Simpler language please: Sometimes Python just isn't fast enough. Or you have existing code in a C++ library. So, you write a Python extension module, a library of compiled code. I recommend pybind11 for C++ to Python bindings or cffi for C to Python bindings. I've done this a lot over the years. But, I discovered that my productivity goes through the floor when my development process goes from Edit -> Test in just Python to Edit -> Compile -> Test in Python plus C++. So, cppimport combines the process of compiling and importing an extension in Python so that you can type modulename = cppimport.imp("modulename") and not have to worry about multiple steps. Internally, cppimport looks for a file modulename.cpp. If one is found, it's run through the Mako templating system to gather compiler options, then it's compiled and loaded as an extension module.

Note that because of the Mako pre-processing, the comments around the configuration block may be omitted. Putting the configuration block at the end of the file, while optional, ensures that line numbers remain correct in compilation error messages.

Recompilation only happens when necessary:

Compilation should only happen the first time the module is imported. The C++ source is compared with a checksum on each import to determine if the file has changed. Additional dependencies (header files!) can be tracked by adding to the Mako header:

cfg['dependencies'] = ['file1.h', 'file2.h']

I need to set the compiler or linker args!

cfg['linker_args'] = ['...']
cfg['compiler_args'] = ['...']
cfg['libraries'] = ['...']
cfg['include_dirs'] = ['...']

For example, to use C++11, add:

<%
cfg['compiler_args'] = ['-std=c++11']
%>

I want multiple source files for one extension!

cfg['sources'] = ['...']

I need more output!

Calling cppimport.set_quiet(False) will result in output that will be helpful in debugging compile errors.

Sometimes I need to force a rebuild even when the checksum matches

Call cppimport.force_rebuild() before running cppimport.imp(...).

I want incremental compiles on extensions with multiple sources.

(For the uninitiated, incremental compilation involves only recompiling those source files that have changed or include headers that have changed.)

cppimport is built on top of the setuptools and distutils, the standard library for python packaging and distribution. Unfortunately, setuptools does not support incremental compilation. I recommend following the suggestions on this SO answer. That is:

  1. Use ccache to (massively) reduce the cost of rebuilds
  2. Enable parallel compilation. This can be done with cfg['parallel'] = True in the C++ file's configuration header.

I need information about filepaths in my module configuration code!

The module name is available as the fullname variable and the C++ module file is available as filepath. For example,

<%
module_dir = os.path.dirname(filepath)
%>

Why does the import hook need "cppimport" on the first line of the .cpp file?

Modifying the Python import system is a global modification and thus affects all imports from any other package. As a result, to avoid accidentally breaking another package, the import hook uses an "opt in" system where C and C++ files can specify they are meant to be used with cppimport by having a comment including the phrase "cppimport" on the first line of the file.

Windows?

I've used cppimport with MinGW-w64 and Python 3.6 and had good success. I've also had reports that cppimport works on Windows with Python 3.6 and Visual C++ 2015 Build Tools. The main challenge is making sure that distutils is aware of your available compilers. Try out the suggestion here.

cppimport uses the MIT License

Comments
  • fatal error: pybind11/pybind11.h: No such file or directory

    fatal error: pybind11/pybind11.h: No such file or directory

    I tried following the example in the README and got the following error:

    /tmp/tmp826vncgj/somecode.cpp:2:36: fatal error: pybind11/pybind11.h: No such file or directory
         #include <pybind11/pybind11.h>
    
    opened by chadgh 27
  • Fail to run `python -m cppimport build`

    Fail to run `python -m cppimport build`

    Hi there~ When try to run %sh python -m cppimport build in databricks, it returns: No module named cppimport.__main__; 'cppimport' is a package and cannot be directly executed

    What the reason could be? Thanks very much!

    cppimport==21.3.7 Python==3.8.10

    opened by VungleTienan 13
  • Adds support for setting libraries and external sources

    Adds support for setting libraries and external sources

    I have the following use cases:

    • I want to wrap an existing library (libvpx and libwebm) for Python and I need add the libraries to link to
    • I have some autocompiled protobuf sources that I need to add to the compilation
    opened by fran6co 13
  • Manipulate sys.path to find the just built modules

    Manipulate sys.path to find the just built modules

    .cpp files built with cppimport can be located anywhere, so let's not assume that python knows where they are located. Append the module path to sys.path to ensure loading works.

    (This fixes python3 -m cppimport build ... for me at $dayjob.)

    opened by bjornfor 8
  • Indentation error in macosx

    Indentation error in macosx

    I keep getting the following indentation error in macosx 10.15. However, the same code runs fine in linux.

    <% cfg['compiler_args'] = ['-std=c++11'] cfg['include_dirs'] = ['/Users/amritpoudel/Desktop/HPC_LIB/HOMEBREW/opt/eigen/include/eigen3', '/Users/amritpoudel/Desktop/HPC_LIB/HOMEBREW/include/python3.7m', '/Users/amritpoudel/Desktop/HPC_LIB/HOMEBREW/opt/python3/Frameworks/Python.framework/Versions/3.7/include/python3.7m'] cfg['libraries'] = ['/Users/amritpoudel/Desktop/HPC_LIB/HOMEBREW/opt/python3/Frameworks/Python.framework/Versions/3.7/lib'] setup_pybind11(cfg) %>

    #############################################

    return compile(expr, filename, mode, PyCF_ONLY_AST) File "", line 2 cfg['include_dirs'] = ['/Users/amritpoudel/Desktop/HPC_LIB/HOMEBREW/opt/eigen/include/eigen3', ^ IndentationError: unexpected indent

    I don't quite understand why it complains in macosx. The same copy-pasted code runs fine in linux machine, so indentation shouldn't be an issue.

    opened by amrit-poudel 7
  • cppimport for compiling in release mode

    cppimport for compiling in release mode

    I have recently started using cppimport (a great library, very useful!) along with pybind11. However, I notice that cppimport seems to compile C++ code in debug mode using -g flag. Also, it does not seem to look for libraries provided in cfg['libraries']. Is there a way to compile in release mode?

    <% cfg['compiler_args'] = ['-std=c++11'] cfg['include_dirs'] = ['/hpc_lib/eigen', '/hpc_lib/pybind11/include', '/hpc_lib/python-3.6.2/include', '/hpc_lib/openblas/include'] cfg['libraries'] = ['/hpc_lib/python-3.6.2/lib', '/hpc_lib/openblas/lib'] cfg['linker_args'] = ['-lopenblas', '-lgfortran', '-lpython3.6m'] setup_pybind11(cfg) %>

    I get the following warning and error during link time:

    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/hpc_lib/eigen -I/hpc_lib/pybind11/include -I/hpc_lib/python-3.6.2/include -I/hpc_lib/openblas/include -I/hpc_lib/python-3.6.2/include/python3.6m -c /src/.rendered.dwf_solver_cpp.cpp -o /tmp/tmpc8iscbum/src/.rendered.dwf_solver_cpp.o -std=c++11 -std=c++11 cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ warning: no library file corresponding to '/hpc_lib/python-3.6.2/lib' found (skipping)

    g++ -pthread -shared -rdynamic /tmp/tmpc8iscbum/src/.rendered.dwf_solver_cpp.o -lpython3.6m -o /tmp/tmpc8iscbum/dwf_solver_cpp.cpython-36m-x86_64-linux-gnu.so -lopenblas -lgfortran -lpython3.6m warning: no library file corresponding to '/hpc_lib/openblas/lib' found (skipping) /binutils/bin/ld: cannot find -lopenblas collect2: error: ld returned 1 exit status

    Why does the compiler not pick up the libraries? It seems to pick up fine the header files though.

    I have those libraries correctly set in LD_LIBRARY_PATH.

    opened by amrit-poudel 7
  • Not working on cpp file that uses #include with angualar brackets

    Not working on cpp file that uses #include with angualar brackets

    I have the following source cpp file. Creating a binding for this file with cppimport work perfectedly.

    #include "../build/example2.h"
    
    int add(int, int);
    
    int add(int i, int j) {
        int a = MAX_VAL;
        return i*j + a;
    }
    

    if I use #include <build/example2.h> instead of #include "../build/example2.h" in the above program. I get the following error.

    fatal error: /build/example2.h: No such file or directory
        1 | #include </build/example2.h>
          |          ^~~~~~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-Linux-gnu-gcc' failed with exit status 1
    

    NOTE: the binding.cpp for this file using cppimort is present in a different directory.

    I think cppimport fails if the source file includes angular brackets (ex: #include<build/example.h>) instead of double-quotes. Is there a way to make cppimport work for angular brackets too?

    opened by siv2r 6
  • [Windows] Compilation warnings: unknown option '-std=c++11'

    [Windows] Compilation warnings: unknown option '-std=c++11'

    Hello, first of all, great work on this tool! it works like a charm!

    I'm a windows user, and the default example raises 2 warnings because of those flags I believe: https://github.com/tbenthompson/cppimport/blob/9b3326424c90e7023e34a1ca413221f6509049a3/cppimport/templating.py#L30

    cl : Command line warning D9002 : ignoring unknown option '-std=c++11'
    cl : Command line warning D9002 : ignoring unknown option '-fvisibility=hidden'
    

    I think msvc has its own set of compiler flags (like /std:c++11) and using those on windows might fix those warnings. Thanks again for the good work!

    opened by kokounet 6
  • Compiler logs have no correlation to actual lines at which errors occur.

    Compiler logs have no correlation to actual lines at which errors occur.

    When I compile and import a .cpp file using cppimport into python, if an error occurs, the lines given for where in the .cpp file the error occurred has no correlation to the actual lines in either the original file or the .rendered.NAME.cpp files. For example, if I get an error message saying there is an error at line 126, I can open both the original and .rendered files and find that line 126 is completely empty.

    I am using the MSVC compiler with pybind11, and this happens with any error in any c++ file.

    opened by 2kai2kai2 6
  • Make setup_pybind11 include -fvisibility=hidden in extra_compile_args

    Make setup_pybind11 include -fvisibility=hidden in extra_compile_args

    Without that flag, one gets warnings per https://pybind11.readthedocs.io/en/stable/faq.html#someclass-declared-with-greater-visibility-than-the-type-of-its-field-someclass-member-wattributes.

    It's probably just a matter of adding it after -std=c++11 in https://github.com/tbenthompson/cppimport/blob/a2f85f355897badc1a94042802da916820e0657e/cppimport/templating.py#L25 (but before + cfg['compiler_args'] to allow a user to overwrite that if they really, really want to).

    opened by anntzer 6
  • Race condition when multiple processes try to compile a module at once

    Race condition when multiple processes try to compile a module at once

    Hi,

    Great package by the way!

    I've encountered an issue when multiple processes are spawned that all race to compile the same module. This can also occur when multiple processes are spawned on different hosts and share the same network filesystem. Such a situation is common when distributing work between multiple processes or hosts for AI or data analytics.

    Here is a demonstration (in the shell):

    echo '// cppimport
    #include <pybind11/pybind11.h>
    
    namespace py = pybind11;
    
    int square(int x) {
        return x * x;
    }
    
    PYBIND11_MODULE(somecode, m) {
        m.def("square", &square);
    }
    /*
    <%
    setup_pybind11(cfg)
    %>
    */' > somecode.cpp
    
    echo 'import cppimport.import_hook
    import somecode
    somecode.square(9)' > test.py
    
    rm somecode.cpython-*
    
    for i in {1..100}; do python3 test.py & done
    

    On my system around 4 out of 100 processes exit in an error. The shell output includes:

    error: could not delete '/localdata/joshl/sandbox/somecode.cpython-36m-x86_64-linux-gnu.so': No such file or directory
    ...
    Exit 1                  python3 test.py
    ...
    Bus error               (core dumped) python3 test.py
    

    These errors don't appear when the binary already exists.


    To mitigate this issue in our applications we have used a file lock so that only one process attempts to compile the module at one time. A process first checks if the binary file exists, otherwise attempts to obtain the file lock. If it can't obtain the lock it waits until either the binary exists, can obtain the file lock or times out. Here is an example how it can be done (app code):

    from cppimport.checksum import is_checksum_valid
    
    binary_path = module_data['ext_path']
    lock_path = binary_path + '.lock'
    
    t = time()
    
    while not (os.path.exists(binary_path) and is_checksum_valid(module_data)) and time() - t < timeout:
        try:
            with FileLock(lock_path, timeout=1):
                if os.path.exists(binary_path) and is_checksum_valid(module_data_new_path):
                    break
                # BUILD BINARY
                template_and_build(filepath, module_data)
        except Timeout:
            logging.debug(f'{os.getpid()}: Could not obtain lock')
            sleep(1)
    
    if not (os.path.exists(binary_path) and is_checksum_valid(module_data_new_path)):
        raise Exception(
            f'Could not compile binary as lock already taken and timed out. Lock file will be deleted: {lock_path}')
    
    if os.path.exists(lock_path):
        with suppress(OSError):
            os.remove(lock_path)
    

    It would be great if we could upstream the above to cppimport to prevent the race condition errors. If you are happy with this solution I could contribute the above to the appropriate place in cppimport.

    opened by joshlk 5
  • Additional automation in the release actions workflow

    Additional automation in the release actions workflow

    remaining tasks to automate in the publish CI workflow:

    • publish to test pypi first
    • then, check the published package
    • then, republish to real pypi
    • then, run the automatic release action.
    opened by tbenthompson 3
  • Dependency version check

    Dependency version check

    Hey,

    I have a source file that has a dependency and I want to force cppimport to recompile the source when the dependency version changes. Unfortunately the header files of the dependency can remain the same when the version changes so I can't rely on that.

    I have a Python function that obtains the version of the dependency, what's the best way to force cppimport too recompile given this information? Would it be possible to add functionality so you can pass a string to be included in the checksum calculation (cppimport.checksum._calc_cur_checksum)?

    opened by joshlk 8
  • Additional commands for `--delete-sources` and `clean`.

    Additional commands for `--delete-sources` and `clean`.

    From @mityax

    I was thinking of a --delete-sources flag for the build command and a clean command deleting all generated/compiled files - maybe an idea for the future, if you'd also think these features are useful.

    opened by tbenthompson 0
  • IDEA: plugin system

    IDEA: plugin system

    I really like cppimport for quick tests and local work. However, there's one irritating part I always have to look up:

    /*
    <%
    setup_pybind11(cfg)
    %>
    */
    

    This has two issues - one it's a special, parsed magic that requires changing the C++ code, and two, it's hard-coded into cppimport; if someone wanted to support something besides pybind11, it would need a new function like this. I'd like to propose a plugin system that allows pybind11 to declare what it needs for setup that could also be used elsewhere. I can help get the pybind11 part into the pybind11 package. :)

    The idea I'm currently thinking of is the following. Users could use:

    # Global
    cppimport.plugin.pybind11.import_hook
    import foobar
    
    # Local
    foobar = cppimport.plugin.pybind11.imp("foobar")
    foobar = cppimport.plugin.pybind11.imp_from_filepath("src/foobar.cpp")
    

    It would be also available in cpp mode to support "classic" usage:

    /*
    <%
    cppimport.plugin.pybind11.setup(cfg)
    %>
    */
    

    A package (pybind11 in this case) would implement an entrypoint, cppimport.setup: pybind11 = .... When you access cppimport.plugin.<attr>, it looks for <attr> item in the cppimport.setup entry points. If found, it calls it with some to-be-determined API, possibly just "cfg" like the current setup_pybind11. No modification or special magic comments needed in the source code.

    Thoughts? Also, a way to set config options would be important, which I haven't addressed above. I think import_hook could be replaced with something callable, like setup_import_hook, which would take cfg options. imp* could take configuration options too.

    opened by henryiii 1
  • Add Windows to continuous integration system.

    Add Windows to continuous integration system.

    Simple, see the title!

    The issue is not quite as easy as appending windows-latest to the list of OS options in the CI workflow. For some reason, the tests fail silently on Windows. Perhaps there is some kind of segfault occuring? I think further testing will require replicating something very similar to the Windows CI environment.

    opened by tbenthompson 0
Releases(22.08.02)
  • 22.08.02(Aug 2, 2022)

    What's Changed

    • Add version string to init.py by @tbenthompson in https://github.com/tbenthompson/cppimport/pull/78

    Full Changelog: https://github.com/tbenthompson/cppimport/compare/22.07.17...22.08.02

    Source code(tar.gz)
    Source code(zip)
  • 22.07.17(Jul 15, 2022)

    What's Changed

    • Feature/concurrent build fixes #67 by @joshlk in https://github.com/tbenthompson/cppimport/pull/71
    • Implement relative imports, clean up Manifest.in. by @tbenthompson in https://github.com/tbenthompson/cppimport/pull/75

    New Contributors

    • @joshlk made their first contribution in https://github.com/tbenthompson/cppimport/pull/71

    Full Changelog: https://github.com/tbenthompson/cppimport/compare/22.05.11...22.07.17 PyPI: https://pypi.org/project/cppimport/22.7.17/

    Source code(tar.gz)
    Source code(zip)
  • 22.05.10(May 11, 2022)

    What's Changed

    • Add cli to pre-compile c sources for deployments by @mityax in https://github.com/tbenthompson/cppimport/pull/65
    • continuous integration: Remove python 3.6, add python 3.10 by @tbenthompson in https://github.com/tbenthompson/cppimport/pull/68

    New Contributors

    • @mityax made their first contribution in https://github.com/tbenthompson/cppimport/pull/65

    Full Changelog: https://github.com/tbenthompson/cppimport/compare/21.03.07...22.05.10

    Source code(tar.gz)
    Source code(zip)
    cppimport-22.5.10.tar.gz(1.65 MB)
  • 21.03.07(Mar 8, 2021)

  • 21.01.11(Jan 12, 2021)

  • 17.09.18(Sep 18, 2017)

  • 16.06.1(Jun 6, 2016)

Owner
Ben Thompson
Ben Thompson
Seamless operability between C++11 and Python

pybind11 — Seamless operability between C++11 and Python Setuptools example • Scikit-build example • CMake example Warning Combining older versions of

pybind 11.6k Oct 2, 2022
Structy is an irresponsibly dumb and simple struct serialization/deserialization library for C, Python, and vanilla JavaScript.

Structy Structy is an irresponsibly dumb and simple struct serialization/deserialization library for C, Python, and vanilla JavaScript. You can think

Stargirl Flowers 48 Sep 13, 2022
Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and deploy without Python.

Python Inference Script(PyIS) Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and d

Microsoft 11 Aug 10, 2022
PyAerotech: An Opensource Python Library or interfacing directly with the Aerotech A3200 Controlle

PyAerotech: An Opensource Python Library or interfacing directly with the Aerotech A3200 Controller pyAerotech is an additional Opensource Python libr

Dr Luke Parry 3 Aug 22, 2022
A foobar2000 component which allows you to load and play ncm files directly.

Play NCM files directly with our favourite How to setup and build project Download foobar2000 SDK and extract into vendor/sdk Download WTL from source

null 36 Sep 18, 2022
Embed image data directly to HTML files.

compact_html Welcome! Embed image data directly to HTML files. Thanks: cpp-base64: Base64 encoding and decoding with c++. cpprestsdk: The C++ REST SDK

Frost Sigh 43 Aug 21, 2022
Tinygettext - A simple gettext replacement that works directly on .po files

tinygettext tinygettext is a minimal replacement for gettext written in C++. It can read .po files directly and doesn't need .mo files generated from

null 57 Sep 2, 2022
Import of the DIY Dynamic Template v2, retrieved from the Internet Archive

Dynamic Templates This is a copy of the D*I*Y Planner Dynamic Template application that was posted to diyplanner.com/node/6210 back in 2009,

Trammell Hudson 21 Aug 7, 2022
fix vmprotect import function used unicorn-engine.

Vm2Import fix vmprotect import function used unicorn-engine. it can repair functions such as call [module.function] or jmp [module.function] or reg(mo

共产主义接班人 76 Oct 1, 2022
VMPImportFixer is a tool aimed to resolve import calls in a VMProtect'd (3.x) binary.

VMPImportFixer VMPImportFixer is a tool aimed to resolve import calls in a VMProtect'd (3.x) binary. Information VMPImportFixer attempts to resolve al

null 252 Sep 23, 2022
api & source menu base import imgui from imgui-js

onetap v4 crack https://discord.gg/AXCtxVH4PB people asking me for otv4 source "bin2h" (meaning binary to hex) large hexadecimal array deleted all the

h4xr0x#1337 9 Sep 6, 2022
Gigaleak | Import HMS file to GEO file for sm64 decomp

Convert HMS to GEO This is a conventer HMS to GEO for Super Mario 64. Requires SM64 decomp and a knowledge of how levels work. NOTE: This is super eas

Swip 2 Dec 26, 2021
Import GIF/WebP animated image as a new AnimatedTexture asset type.

Animated Texture Plugin for Unreal Engine 5 This plugin allows you to import animated pictures into your Unreal Engine 5 project as a new AnimatedText

房燕良 30 Sep 17, 2022
A blender import/export system for Defold

defold-blender-export A blender import/export system for Defold Setup Notes There are no exhaustive documents for this tool yet. Its just not complete

David Lannan 24 Sep 21, 2022
Vmpfix - Universal x86/x64 VMProtect 2.0-3.X Import fixer

vmpfix VMPfix is a dynamic x86/x64 VMProtect 2.0-3.x import fixer. The main goal of this project was to build correct and reliable tool to fix imports

Pavel 189 Sep 23, 2022
WifSolverCuda - Tool for solving misspelled or damaged Bitcoin Private Key in Wallet Import Format (WIF)

WifSolverCuda Tool for solving misspelled or damaged Bitcoin Private Key in Wallet Import Format (WIF) Usage: WifSolverCuda [-d deviceId] [-b NbBlocks

null 19 Sep 21, 2022
A PoC for requesting HWIDs directly from hardware, skipping any potential hooks or OS support.

PCIBan A PoC for requesting HWIDs directly from hardware, skipping any potential hooks or OS support. This is probably very unsafe, not supporting edg

null 58 Sep 12, 2022
Control Hörmann doors drives directly via MQTT from Home Assistant

hoermann_door Control Hörmann doors drives directly via MQTT from Home Assistant

null 59 Sep 21, 2022
An extremely hacky VNC server for WebOS - Works by reading directly from the GPU's framebuffer.

webos-vncserver An extremely hacky VNC server for WebOS - Works by reading directly from the GPU's framebuffer. Requires root privileges.

David Buchanan 16 Sep 27, 2022
Assembly HellGate implementation that directly calls Windows System Calls and displays the PPID of the explorer.exe process

Custom HellsGate Implementation Assembly HellGate implementation that directly calls Windows System Calls and displays the PPID of the explorer.exe pr

Bobby Cooke 89 Sep 19, 2022