Python to CLike language transpiler

Overview

Python to many CLike language transpiler

Currently supports C++ and Rust. Preliminary support for Julia, Kotlin, Nim, Go and Dart.

The main idea is that python is popular, easy to program in, but has poor runtime performance. We can fix that by transpiling a subset of the language into a more performant, statically typed language

Based on Julian Konchunas' pyrs http://github.com/konchunas/pyrs

Based on Lukas Martinelli Py14 (https://github.com/lukasmartinelli/py14) and Py14/python-3 (https://github.com/ProgVal/py14/tree/python-3) branch by Valentin Lorentz.

Example

Original Python version.

def fib(i: int) -> int:
    if i == 0 or i == 1:
        return 1
    return fib(i - 1) + fib(i - 2)

Transpiled Rust code:

fn fib(i: i32) -> i32 {
    if i == 0 || i == 1 {
        return 1;
    }
    return (fib((i - 1)) + fib((i - 2)));
}

Transpiled code for other languages:

https://github.com/adsharma/py2many/tree/main/tests/expected (fib*)

Trying it out

Requirements:

  • python 3
  • clang
  • rustc

Transpiling:

./py2many --cpp=1 /tmp/fib.py
./py2many --rust=1 /tmp/fib.py
./py2many --julia=1 /tmp/fib.py
./py2many --kotlin=1 /tmp/fib.py
./py2many --nim=1 /tmp/fib.py
./py2many --dart=1 /tmp/fib.py
./py2many --go=1 /tmp/fib.py

Compiling:

clang fib.cpp
rustc fib.rs
...

Monkeytype

Python 3.5+ type annotations can be transpiled:

def cut(apple: Fruit, knife: Tool) -> Slice:
fn cut(apple: Fruit, knife: Tool) -> Slice {

Todo list

  • Basic type inference for struct declaration
  • Use constructors for guessing struct member types
  • Return type inference
  • Mutability based on usage

Working Features

Only bare functions using the basic language features are supported. Some of them work partially.

  • classes
  • functions
  • lambdas
  • list comprehensions
  • inheritance
  • operator overloading
  • function and class decorators
  • getter/setter function decorators
  • yield (generator functions)
  • function calls with *args and **kwargs

Language Keywords

  • global, nonlocal
  • while, for, continue, break
  • if, elif, else
  • try, except, raise
  • def, lambda
  • new, class
  • from, import
  • as
  • pass, assert
  • and, or, is, in, not
  • return
  • yield

Builtins

  • dict
  • list
  • tuple
  • int
  • float
  • str
  • round
  • range
  • range_step
  • sum
  • len
  • map
  • filter

Data Structures

  • list
  • Set
  • String
  • Dict
Comments
  • Transpiling a small python program to c++ fails with a traceback

    Transpiling a small python program to c++ fails with a traceback

    Hi.

    Thanks for py2many. It's a really cool idea.

    I'm trying to transpile this program:

    #!/usr/local/cpython-3.9/bin/python3

    import math import time import random

    def main(): a = []

    random.seed(time.time())
    t1 = time.time()
    
    for i in range(0, 5000000):
        b = random.random()
        if b > 0.5:
            c = math.pow(b, 1.546)
        else:
            c = math.pow(b, 0.434232)
        d = math.cos(c + i)
        a.append(math.atan(d))
    
    t2 = time.time()
    print(t2 - t1)
    

    main()

    ...but when I try to transpile it to C++, I get a long traceback ending in: File "/home/dstromberg/.local/lib/python3.9/site-packages/py2many/tracer.py", line 150, in visit_Name return self.visit(var.assigned_from.value) File "/usr/local/cpython-3.9/lib/python3.9/ast.py", line 407, in visit return visitor(node) File "/home/dstromberg/.local/lib/python3.9/site-packages/py2many/tracer.py", line 160, in visit_Call return "{0}({1})".format(node.func.id, params) AttributeError: 'Attribute' object has no attribute 'id'

    I tried both the pypi py2many, as well as tonight's github py2many - and I tried both CPython 3.8 and CPython 3.9. I didn't scrutinize the output tremendously carefully, but I think it was the same traceback for all 4.

    opened by dstromberg 19
  • v0.3

    v0.3

    Once https://github.com/adsharma/py2many/pull/350 is merged, I think we're due for a 0.3 release.

    Any features wanted before a new release?

    ping @PXshadow @dstromberg @john-michaelburke @dumblob

    @nsauzede , I notice you have an active fork with some changes. Any you feel are close to being ready - we can help you get them rebased and merged if you let us know which branches we should be investigating.

    opened by jayvdb 17
  • V language - join efforts with py2v

    V language - join efforts with py2v

    py2many looks like an interesting endeavour (I've just discovered it through https://news.ycombinator.com/item?id=27165366 ).

    I wonder whether there would be some benefits from sharing some knowledge, experience, abstractions, code, templates, etc. with https://github.com/vlang/py2v .

    opened by dumblob 13
  • PyPI name

    PyPI name

    After getting the package ready for release, please upload to PyPI to reserve the name and so it can be easily re-used via a permanent name.

    https://github.com/adsharma/py2many/pull/4 does a lot of the prep needed. README likely needs a few minor adjustments also.

    opened by jayvdb 13
  • C++ headers not found on GitHub Actions Ubuntu 20210517.1

    C++ headers not found on GitHub Actions Ubuntu 20210517.1

    Last main commit which was working used

    Environment: ubuntu-20.04 Version: 20210510.0 Included Software: actions/[email protected]/20210510.0/images/linux/Ubuntu2004-README.md Image Release: actions/[email protected]%2F20210510.0 (release)

    First broken commit was

    Environment: ubuntu-20.04 Version: 20210517.1 Included Software: actions/[email protected]/20210517.1/images/linux/Ubuntu2004-README.md Image Release: actions/[email protected]%2F20210517.1 (release)

    https://github.com/actions/virtual-environments/blob/ubuntu20/20210510.0/images/linux/Ubuntu2004-README.md vs https://github.com/actions/virtual-environments/blob/ubuntu20/20210517.1/images/linux/Ubuntu2004-README.md has clang version changes, which are likely the cause.

    opened by jayvdb 7
  • Reduce golint min_confidence to default 0.8

    Reduce golint min_confidence to default 0.8

    The current golint level of 1.0 causes several test cases to become errors, mostly due to capitalisation and docstrings.

    However the default of 0.8 causes many more errors.

    opened by jayvdb 7
  • "a" in ["a"] broken

    "a" in ["a"] in the ast is an ast.Compare and an ast.In.

    This works for lists and sets when transpiling to Rust, however it does not work for dicts in Rust, and it fails for all container types in the other target languages.

    Also for target cpp, it causes an exception:

    py2many/py2many/cli.py:185: in main
        _process_once(settings, source, outdir)
    py2many/py2many/cli.py:133: in _process_once
        f.write(transpile(source_data, settings.transpiler))
    py2many/py2many/cli.py:45: in transpile
        code = transpiler.visit(tree) + "\n"
    py2many/py2many/clike.py:80: in visit
        return super().visit(node)
    /usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py:407: in visit
        return visitor(node)
    py2many/py14/transpiler.py:285: in visit_Module
        buf = [self.visit(b) for b in node.body]
    py2many/py14/transpiler.py:285: in <listcomp>
        buf = [self.visit(b) for b in node.body]
    py2many/py2many/clike.py:80: in visit
        return super().visit(node)
    /usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py:407: in visit
        return visitor(node)
    py2many/py14/transpiler.py:97: in visit_FunctionDef
        body = "\n".join([self.visit(n) for n in node.body])
    py2many/py14/transpiler.py:97: in <listcomp>
        body = "\n".join([self.visit(n) for n in node.body])
    py2many/py2many/clike.py:80: in visit
        return super().visit(node)
    /usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py:407: in visit
        return visitor(node)
    py2many/py14/transpiler.py:353: in visit_Assert
        return "assert({0});".format(self.visit(node.test))
    py2many/py2many/clike.py:80: in visit
        return super().visit(node)
    /usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py:407: in visit
        return visitor(node)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    self = <py14.transpiler.CppTranspiler object at 0x10c2b3640>, node = <ast.Compare object at 0x10c2b3ac0>
    
        def visit_Compare(self, node):
            if isinstance(node.ops[0], ast.In):
    >           return self.visit_Any(node)
    E           AttributeError: 'CppTranspiler' object has no attribute 'visit_Any'
    
    opened by jayvdb 7
  • Python test cases should be runnable scripts

    Python test cases should be runnable scripts

    In addition to being compile-able, the generated files should also be runnable and their output should be the same as the Python script. That adds an extra layer of validation.

    opened by jayvdb 7
  • Add a directory mode

    Add a directory mode

    py2many.py --dir=/some/python/module <opts>
    

    The idea is that all of the python files in the module are parsed and type inference works across file boundaries and we end up with module-py2many which contains transpiled code for the target language.

    opened by adsharma 6
  • FileNotFoundError [Errno 2] No such file or directory: 'format.jl

    FileNotFoundError [Errno 2] No such file or directory: 'format.jl

    I am unable to execute the Julia conversion code. I am getting the following error:

    Error: Could not format: try.jl Due to: FileNotFoundError [Errno 2] No such file or directory: 'format.jl'

    Here, try.py is the file I want to convert to Julia (It is in the same directory)

    Julia 
    opened by ankushsoni96 5
  • Overwrite cli flag

    Overwrite cli flag

    Currently the cli will happy overwrite an existing transpiled file.

    e.g. py2many.py --rust=1 /tmp/foo.py will overwrite /tmp/foo.rs

    The same occurs if --outdir is used.

    And in directory mode, this can cause a "target language project build phase" to go haywire, as there may be some stale generated files in the directory, left over from a previous transpilation.

    https://github.com/adsharma/py2many/pull/350#pullrequestreview-694194375 makes this even more troublesome, as it would overwrite the input file.

    Often forced overwrite is a --force flag. Sometimes a shortopt of -f is allowed.

    opened by jayvdb 5
  • 'If' object has no attribute 'body_vars'

    'If' object has no attribute 'body_vars'

    Line https://github.com/folkertvanheusden/monkeycoder/blob/master/processor_z80.py#L83

    width = random.choice([8, 16]) if sub_type in [processor.Instruction.i_add, processor.Instruction.i_add_carry, processor.Instruction.i_sub_carry] else 8
    

    triggers the following error in py2many --cpp=1:

    processor_z80.py:83:0: py2many.exceptions.AstNotImplementedError: 'If' object has no attribute 'body_vars'
    
    opened by folkertvanheusden 0
  • test: extend test suite for stdlib

    test: extend test suite for stdlib

    Sed script used as a helper:

    s/self.assertEqual(\(.*\), \(.*\))/assert \1 == \2/g
    s/self.assertTrue(\(.*\))/assert \1/g
    s/self.assertFalse(\(.*\))/assert not \1/g
    s/self.fail(.*)/assert False/g
    s/self.assertIsNaN/assert math.isnan/g
    s/self.assertLessEqual(\(.*\), \(.*\))/assert \1 <= \2/g
    s/self.assertLess(\(.*\), \(.*\))/assert \1 < \2 \2/g
    s/self.assertGreater(\(.*\), \(.*\))/assert \1 > \2/g
    s/self.assertAlmostEqual(\(.*\), \(.*\))/assert abs(\1 - \2) < 1e-6/g
    
    /self.assertRaises/d
    /self.assertIs/d
    /self.subTest/d
    /@[a-zA-Z0-9_]*/d
    
    # math specific
    s/self.ftest('.*', \(.*\), \(.*\))/assert ftest(\1, \2)/g
    
    opened by div72 2
  • Not opening non-utf8 files

    Not opening non-utf8 files

    Here is another bug I found in py2many:

    The transpiler seems to fail to open files encoding with an encoding other than utf-8. I tried transpiling a python file with a Chinese character ("丐") and encountered a generic python UnicodeDecodeError: UnicodeDecodeError: 'charmap' codec can't decode byte ... Removing the Chinese character allowed the program to compile fine.

    If this is the issue, I have a fix: Try using try-except blocks to make sure the file is not opened with the wrong encoding. Here is how I do this in my code:

    try: file = open(path, str(opentype),encoding = "utf8") except UnicodeDecodeError: try: file = open(path, str(opentype),encoding = "utf16") except UnicodeDecodeError: try: file = open(path, str(opentype),encoding = "utf32") except UnicodeDecodeError: fileencodingunknownerror(path) file = None

    opened by HorridModz 2
  • py2many.exceptions.AstUnrecognisedBinOp: Union[Union[int,Union[str,int]],Union[str,int]] <class 'ast.Sub'> int Error

    py2many.exceptions.AstUnrecognisedBinOp: Union[Union[int,Union[str,int]],Union[str,int]] int Error

    Hi, thanks for this awesome python transpiler! I am very excited to use this. I did find a couple bugs in it, which were expected.

    Here is the issue I got:

    Writing to: .
    Code.py ... Code.cpp
    Code.py:680:14: py2many.exceptions.AstUnrecognisedBinOp: Union[Union[int,Union[str,int]],Union[str,int]] <class 'ast.Sub'> int
    Error: Could not format: Code.cpp
    

    I was getting the error in this line of code: startpos = indexofstr(startstr,fullstr,casesensitive) + 1 Changing the code to this fixed the issue: startpos = indexofstr(startstr,fullstr,casesensitive)

    I am assuming that the transpiler does not recognize I am trying to use the return of the function for path. To double-check, I tested this concept in c++ (what I was compiling to) and it worked:

    `#include

    int a(int a1,int a2,int a3){ return (a1 + a2 + a3); }

    int b(){ int c; c = a(1,2,3) + 1; return c; }

    int main(){ std::cout << b(); return 0; }`

    It would be awesome if this issue could be fixed! For now, I will have to work around it.

    opened by HorridModz 5
  • Discoverability

    Discoverability

    There is a new transpiler (pycom) that's getting good engagement on reddit and hackernews. It's written by a 15yo in two weeks.

    https://www.reddit.com/r/Python/comments/w7vlim/i_made_a_python_compiler_that_can_compile_python/

    Where people are comparing it to cython, nuitka, shedskin etc.

    It appears to me that awareness of py14, pyrs and py2many family of work is still minimal. Someone searching google for "python to rust transpiler" will not discover this repo very easily. I don't know what the solution is. But making it more discoverable could attract more contributors.

    opened by adsharma 2
Owner
null
Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and deploy without Python.

Python Inference Script(PyIS) Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and d

Microsoft 11 Aug 10, 2022
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to m

The Apache Software Foundation 20.1k Sep 26, 2022
Tensors and Dynamic neural networks in Python with strong GPU acceleration

PyTorch is a Python package that provides two high-level features: Tensor computation (like NumPy) with strong GPU acceleration Deep neural networks b

null 59.2k Sep 30, 2022
Implement yolov5 with Tensorrt C++ api, and integrate batchedNMSPlugin. A Python wrapper is also provided.

yolov5 Original codes from tensorrtx. I modified the yololayer and integrated batchedNMSPlugin. A yolov5s.wts is provided for fast demo. How to genera

weiwei zhou 43 Sep 26, 2022
Upload and changes to Python 0.9.1 release (from 1991!) so that it would compile

This is Python, an extensible interpreted programming language that combines remarkable power with very clear syntax. This is version 0.9 (the first

Skip Montanaro 134 Aug 31, 2022
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

Website | Documentation | Tutorials | Installation | Release Notes CatBoost is a machine learning method based on gradient boosting over decision tree

CatBoost 6.7k Oct 3, 2022
ParaMonte: Plain Powerful Parallel Monte Carlo and MCMC Library for Python, MATLAB, Fortran, C++, C.

Overview | Installation | Dependencies | Parallelism | Examples | Acknowledgments | License | Authors ParaMonte: Plain Powerful Parallel Monte Carlo L

Computational Data Science Lab 173 Sep 10, 2022
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

eXtreme Gradient Boosting Community | Documentation | Resources | Contributors | Release Notes XGBoost is an optimized distributed gradient boosting l

Distributed (Deep) Machine Learning Community 23.2k Sep 28, 2022
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Chao Ma 3k Oct 2, 2022
An open source python library for automated feature engineering

"One of the holy grails of machine learning is to automate more and more of the feature engineering process." ― Pedro Domingos, A Few Useful Things to

alteryx 6.3k Oct 4, 2022
In-situ data analyses and machine learning with OpenFOAM and Python

PythonFOAM: In-situ data analyses with OpenFOAM and Python Using Python modules for in-situ data analytics with OpenFOAM 8. NOTE that this is NOT PyFO

Argonne Leadership Computing Facility - ALCF 111 Sep 6, 2022
TensorFlow Lite, Coral Edge TPU samples (Python/C++, Raspberry Pi/Windows/Linux).

TensorFlow Lite, Coral Edge TPU samples (Python/C++, Raspberry Pi/Windows/Linux).

Nobuo Tsukamoto 85 Sep 26, 2022
Inference framework for MoE layers based on TensorRT with Python binding

InfMoE Inference framework for MoE-based models, based on a TensorRT custom plugin named MoELayerPlugin (including Python binding) that can run infere

Shengqi Chen 32 Aug 16, 2022
Python and C++ implementation of "MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation". Accepted at LXCV Workshop @ CVPR 2021.

MarkerPose: Robust Real-time Planar Target Tracking for Accurate Stereo Pose Estimation This is a PyTorch and LibTorch implementation of MarkerPose: a

Jhacson Meza 44 Aug 9, 2022
Header-only C++/python library for fast approximate nearest neighbors

Hnswlib - fast approximate nearest neighbor search Header-only C++ HNSW implementation with python bindings. NEWS: Hnswlib is now 0.5.2. Bugfixes - th

null 2.2k Oct 2, 2022
PyCppAD — Python bindings for CppAD Automatic Differentiation library

PyCppAD is an open source framework which provides bindings for the CppAD Automatic Differentiation(CppAD) C++ library in Python. PyCppAD also includes support for the CppADCodeGen (CppADCodeGen), C++ library, which exploits CppAD functionality to perform code generation.

SimpleRobotics 12 Jun 15, 2022
An Out-of-the-Box TensorRT-based Framework for High Performance Inference with C++/Python Support

An Out-of-the-Box TensorRT-based Framework for High Performance Inference with C++/Python Support

手写AI 1.3k Oct 1, 2022