A foreign function interface for bash.

Related tags

Scripting ctypes.sh
Overview

ctypes.sh

This is ctypes.sh, a foreign function interface for bash.

ctypes.sh is a bash plugin that provides a foreign function interface directly in your shell. In other words, it allows you to call routines in shared libraries from within bash.

A (very) simple example will help illustrate:

$ dlcall puts "hello, world"
hello, world

# A more complex example, use libm to calculate sin(PI/2)
$ dlopen libm.so.6
0x172ebf0
$ dlcall -r double sin double:1.57079632679489661923
double:1.000000

ctypes.sh can extend bash scripts to accomplish tasks that were previously impossible, or would require external helpers to be written.

ctypes.sh makes it possible to use GTK+ natively in your shell scripts, or write a high-performance http daemon.

See more examples here

prerequisites

ctypes.sh is dependent on the following libraries and programs:

  • libffi
  • bash
  • libelf (optional)
  • elfutils (optional)
  • libdwarf / libdw (optional)

Fedora

For recent Fedora, this should be enough:

sudo yum install elfutils-devel dnf-utils

Now you can use the debuginfo-install command to install debugging symbols for automatic structure support.

Ubuntu

For recent Ubuntu, this should be enough:

sudo apt install autoconf libltdl-dev libffi-dev libelf-dev elfutils libdw-dev

If you want to use automatic struct support (recommended), you should also make you have ddebs available.

install

ctypes.sh can be installed from source like this:

$ git clone https://github.com/taviso/ctypes.sh.git
$ cd ctypes.sh
$ ./autogen.sh
$ ./configure
$ make
$ [sudo] make install

By default ctypes.sh is installed into /usr/local/bin and /usr/local/lib. You can overload the prefix path by defining the PREFIX environment variable before installing.

$ PREFIX=$HOME make install

example

source ctypes.sh
puts () {
  dlcall puts "[email protected]"
  return $?
}

puts "hello, world"

Here is what people have been saying about ctypes.sh:

  • "that's disgusting"
  • "this has got to stop"
  • "you've gone too far with this"
  • "is this a joke?"
  • "I never knew the c could stand for Cthulhu."

You can read more about ctypes.sh and see it in action on the Wiki

Comments
  • Implement automatic structure definitions

    Implement automatic structure definitions

    Let's fix the ugly problem of accessing system structures for version 1.1.

    In issue #27 @cemeyer suggests automatic creation of standard structures by parsing the dwarf data. I think that could work.

    The utility pahole from the dwarvespackage almost does what we want. Here's an example of pahole output:

    $ ./pahole -C stat /usr/lib/debug/lib64/libc.so.6.debug 
    struct stat {
        __dev_t                    st_dev;               /*     0     8 */
        __ino_t                    st_ino;               /*     8     8 */
        __nlink_t                  st_nlink;             /*    16     8 */
        __mode_t                   st_mode;              /*    24     4 */
        __uid_t                    st_uid;               /*    28     4 */
        __gid_t                    st_gid;               /*    32     4 */
        int                        __pad0;               /*    36     4 */
        __dev_t                    st_rdev;              /*    40     8 */
        __off_t                    st_size;              /*    48     8 */
        __blksize_t                st_blksize;           /*    56     8 */
        /* --- cacheline 1 boundary (64 bytes) --- */
        __blkcnt_t                 st_blocks;            /*    64     8 */
        struct timespec            st_atim;              /*    72    16 */
        struct timespec            st_mtim;              /*    88    16 */
        struct timespec            st_ctim;              /*   104    16 */
        __syscall_slong_t          __glibc_reserved[3];  /*   120    24 */
        /* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
    
        /* size: 144, cachelines: 3, members: 15 */
        /* last cacheline: 16 bytes */
    };
    

    it can also expand types into primitive types (like long, int, etc) with --expand-types. I think if we import it we can make builtin command struct that does something like this:

    # make foo an array of the right size and type prefixes populated correctly
    $ struct stat foo 
    $ echo ${foo[@]}
    long long unsigned unsigned int ..etc
    
    # struct command can also define a bunch of integers like _structname_membername so this works:
    $ echo ${foo[_stat_st_uid]}
    int:0
    $ echo ${foo[_stat_sts_size]}
    long:8192
    
    

    I quickly converted pahole to a builtin and I think it's going to work:

    $ enable -f .libs/ctypes.so struct
    $ struct stat /usr/lib/debug/usr/lib64/libc-2.22.so.debug
    struct stat {
        long unsigned int          st_dev;                                               /*     0     8 */
        long unsigned int          st_ino;                                               /*     8     8 */
        long unsigned int          st_nlink;                                             /*    16     8 */
        unsigned int               st_mode;                                              /*    24     4 */
        unsigned int               st_uid;                                               /*    28     4 */
        unsigned int               st_gid;                                               /*    32     4 */
        int                        __pad0;                                               /*    36     4 */
        long unsigned int          st_rdev;                                              /*    40     8 */
        long int                   st_size;                                              /*    48     8 */
        long int                   st_blksize;                                           /*    56     8 */
        long int                   st_blocks;                                            /*    64     8 */
        struct timespec {
            long int           tv_sec;                                               /*    72     8 */
            long int           tv_nsec;                                              /*    80     8 */
        } st_atim; /*    72    16 */
        struct timespec {
            long int           tv_sec;                                               /*    88     8 */
            long int           tv_nsec;                                              /*    96     8 */
        } st_mtim; /*    88    16 */
        struct timespec {
            long int           tv_sec;                                               /*   104     8 */
            long int           tv_nsec;                                              /*   112     8 */
        } st_ctim; /*   104    16 */
        long int                   __glibc_reserved[3];                                  /*   120    24 */
    };
    $ type struct
    struct is a shell builtin
    
    

    So I just need to use bind_array_variable instead of printing out the types and I think it will work. We will also need a sizeof builtin, then a script like this should work:

    #!/bin/bash
    source ctypes.sh
    
    # create a bash version of the stat structure
    struct stat passwd
    
    # Allocate some space for native stat buffer
    dlcall -n statbuf -r pointer malloc $(sizeof stat)
    
    # Call stat()
    dlcall -r int stat "/etc/passwd" $statbuf
    
    # parse the native struct into bash struct
    unpack $statbuf passwd
    
    # dont need the native version any more
    dlcall free $statbuf
    
    printf "/etc/passwd\n"
    printf "\tuid:  %u\n" ${passwd[st_uid]##*:}
    printf "\tgid:  %u\n" ${passwd[st_gid]##*:}
    printf "\tmode: %o\n" ${passwd[st_mode]##*:}
    printf "\tsize: %u\n" ${passwd[st_size]##*:}
    
    

    So TODO:

    • Parse .gnu_debuglink so user doesn't have to specify any files
    • Automatically check all dlopen'd libraries for a matching structure name
    • Add a sizeof builtin so user can find out the size of a structure when using malloc
    • Find out if this approach is going to work on FreeBSD @cemeyer ?

    We can't use associative arrays for this (unfortunately) because internally bash stores associative arrays as hash tables and doesn't maintain the order of elements, so we will need to define the indexes.

    This means we can't make ${stat[st_size]} work without defining the variable $st_size. It would be nice if we could do stat[stat.st_size], but I don't think bash will parse that correctly. I think the best we can do is:

    echo ${stat[_stat_st_size]}

    But I think I can live with that.

    enhancement 
    opened by taviso 69
  • Port ctypes.sh to FreeBSD

    Port ctypes.sh to FreeBSD

    Fix a number of platform agnostic Clang warnings:

    • ffi_prep_closure_loc()'s callback returns void; fix execute_bash_trampoline to match
    • Fix a few instances of the wrong pointer type being passed to check_parse_ulong()
    • Pull decode_element closures (GCC extension) out of pack_prefixed_array(), unpack_prefixed_array()

    Fix a number of Glibcisms and Linuxisms:

    • FreeBSD doesn't need -ldl
    • RTLD_DEEPBIND is a Linuxism
    • Implement vaguely portable strndupa() and mempcpy
    opened by cemeyer 21
  • Doesn't work on Linux Mint

    Doesn't work on Linux Mint

    I couldn't get it to work on Linux Mint. Probably because it is not properly installed; the installation instructions need work.

    Some of it is not your fault. It is specified that it depends on libffi - and installing that "out of the box" didn't work, either, producing some bizarre errors about missing macros. I figured out that compiling libffi depends on two things that were not installed by default on my machine:

    • libtool
    • makeinfo (in order to install that one, you actually have to install "texinfo")

    After installing them, libffi compiled and installed. After running autogen.sh, configure and make, the things compiles, albeit with a lot of warnings. Most of them are about ignoring the return value of asprintf(), but there are others:

    types.c:51:13: warning: format '%lu' expects argument of type 'long unsigned int', but argument 2 has type 'size_t' [-Wformat=] builtin_error("cannot handle size %lu", type->size);

    types.c:79:9: warning: passing argument 2 of 'check_parse_long' from incompatible pointer type [enabled by default] if (check_parse_long(parameter, &n)) {

    util.h:4:6: note: expected 'long int *' but argument is of type 'intmax_t *' bool check_parse_long(const char *number, long *result);

    unpack.c:83:25: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'arrayind_t' [-Wformat=] element->ind);

    unpack.c:237:25: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'arrayind_t' [-Wformat=] element->ind);

    The installed program simply doesn't work - trying the various examples in the test directory results in segmentation faults and other errors.

    opened by bontchev 17
  • Addition of a build system generator

    Addition of a build system generator

    opened by elfring 12
  • fix Makefile.am in src dir

    fix Makefile.am in src dir

    1. I am not sure if this is a fix, I am on Crux Linux, this is a fix for me. (Thanks to @condy0919 )
    2. Also you guys said installing libffi-devel can fix, I am curious about how that did the trick.
    3. it's a great project, thanks,

    @taviso Can you amswer my question and review this PR? Thanks for your time.

    opened by ghost 9
  • bsd's and darwin

    bsd's and darwin

    I'm not asking for mingw or cygwin yet :) Should not be hard, as libffi does the hard parts.

    link.h and RTLD_DEEPBIND are missing on those. also mempcpy, strchrnul, strndupa

    I'm working on that in my non-linux branch https://github.com/rurban/ctypes.sh/tree/non-linux

    opened by rurban 8
  • installation issue on LinuxMint LMDE 2 Betsy

    installation issue on LinuxMint LMDE 2 Betsy

    $ ./autogen.sh

    autoreconf: Entering directory `.'
    autoreconf: configure.ac: not using Gettext
    autoreconf: running: aclocal --force -I m4
    autoreconf: configure.ac: tracing
    autoreconf: running: libtoolize --copy --force
    libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `build-aux'.
    libtoolize: copying file `build-aux/ltmain.sh'
    libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
    libtoolize: copying file `m4/libtool.m4'
    libtoolize: copying file `m4/ltoptions.m4'
    libtoolize: copying file `m4/ltsugar.m4'
    libtoolize: copying file `m4/ltversion.m4'
    libtoolize: copying file `m4/lt~obsolete.m4'
    autoreconf: running: /usr/bin/autoconf --force
    configure.ac:15: error: possibly undefined macro: LT_SYS_DLSEARCH_PATH
          If this token and others are legitimate, please use m4_pattern_allow.
          See the Autoconf documentation.
    configure.ac:16: error: possibly undefined macro: LT_SYS_MODULE_EXT
    autoreconf: /usr/bin/autoconf failed with exit status: 1
    

    bash and libffi-dev are installed.

    Any idea, why this occurs?

    opened by eriksank 6
  • ubuntu 32-bit support might be broken

    ubuntu 32-bit support might be broken

    I cannot use handler ${DLHANDLERS[libc.so.6]} to call libc functions, shell breaks. Array handling works for me for every other library opened, but libc.so.6. $RTLD_DEFAULT seems to work as handler for any libc function.

    opened by Ismas 5
  • Support for musl libc?

    Support for musl libc?

    Would you consider support for musl libc? (or be open to PRs that added support for it?)

    (I actually love using ctypes.sh - it's.. actually really good.)

    opened by rhencke 4
  • Wondering if you know how to call gtk_init.

    Wondering if you know how to call gtk_init.

    The sign is void gtk_init (int *argc, char ***argv). Both args are useless to me. I tried dlcall gtk_init 'pointer:0x0' 'pointer:0x0' and got

    malloc: unknown:0: assertion botched
    free: start and end chunk sizes differ
    Aborting...
    

    Also I tried replace "'pointer:0x0'" with "NULL", I got segmentation fault.

    opened by Magicloud 4
  • SHA_CTX from libcrypto doesn't work with automatic struct support

    SHA_CTX from libcrypto doesn't work with automatic struct support

    I don't understand why this doesn't work:

    $ dlopen libcrypto.so
    0x26ae7d0
    $ sizeof -am ctx SHA_CTX
    bash: sizeof: warning: SHA_CTX could not be found; check `help struct` for more
    

    (Note that I need -a because it's a typedef, not a struct)

    But if I do this:

    $ cat test.c 
    #include <openssl/sha.h>
    SHA_CTX x;
    $ gcc -o test.so test.c -ggdb3 -shared -fPIC
    $ dlopen ./test.so 
    0x27ed980
    $ sizeof -am ctx SHA_CTX
    96
    

    I suspect some dwarf cu import logic that I haven't seen before is in there, I'll have to go through the dwarfdump output and track it down.

    opened by taviso 4
  • mempcpy needs a configure test

    mempcpy needs a configure test

    Hi there,

    Not only glibc and newlib provide mempcpy, so the build fails e.g. on recent FreeBSD due to a duplicate function mempcpy. To fix this, I've written a patch that adds a configure test to check for mempcpy, only using your own implementation if it is not available.

    Unfortunately your code has a custom config.h header which shadows the config.h header generated by autotools. In my patch, I have simply removed the other header and defined the single macro in it elsewhere. Check if that's the solution you want.

    opened by clausecker 0
  • Associative Array Rehashing

    Associative Array Rehashing

    I think in bash 5.1 struct support will break, because bash introduced an automatic rehashing feature.

    bash used to allocate a fixed size hash table for every associative array, and it never changed for the life of the array. Now, if you add lots of items to the array, bash will dynamically rehash it with a more efficient bucket size.

    This causes two problems:

    1. The order of elements might change.
    2. Any saved pointers might now become invalid.

    This is a note to self to investigate solutions.

    There doesn't seem to be an easy flag we can set, I asked on the mailing list and Chet seemed willing to add one, but not before 5.1 is released. I think a solution might be to just make our own alternative to hash_create().... I dunno.

    opened by taviso 0
  • Can't build on macOS

    Can't build on macOS

    Hello, I'm having issues trying to build on macOS.

    I use the latest version of bash (not the outdated version macOS comes with, so it should support bash plugins)

    $ bash --version
    GNU bash, version 5.0.11(1)-release (x86_64-apple-darwin18.6.0)
    

    Running ./configure gives me

    configure: WARNING: elfutils is not available, struct support will not be available
    [...]
    checking whether bash symbols are exported... no
    configure: error: in `~/code/ctypes.sh':
    configure: error: unable to build a test extension
    See `config.log' for more details
    

    In config.log I see

    configure:13864: checking whether bash symbols are exported
    configure:13891: gcc -o conftest -g -O2  -I./include  -shared -fPIC conftest.c -lelf  >&5
    Undefined symbols for architecture x86_64:
      "_num_shell_builtins", referenced from:
          _conftest_function in conftest-efa94e.o
    ld: symbol(s) not found for architecture x86_64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    configure:13891: $? = 1
    configure: failed program was:
    | /* confdefs.h */
    [...]
    |     #include "builtins.h"
    | 
    |     int conftest_function(WORD_LIST *list)
    |     {
    |         exit(num_shell_builtins
    |                 ? EXIT_SUCCESS
    |                 : EXIT_FAILURE);
    |     }
    | 
    |     struct builtin conftest_struct = {
    |         .name       = "conftest",
    |         .function   = conftest_function,
    |         .flags      = BUILTIN_ENABLED,
    |         .long_doc   = NULL,
    |         .short_doc  = NULL,
    |         .handle     = NULL,
    |     };
    [...]
    configure:13911: result: no
    configure:13913: error: in `~/code/ctypes.sh':
    configure:13915: error: unable to build a test extension
    See `config.log' for more details
    

    I don't know if this has something to do with include/builtins.h since that what defines the extern of num_shell_builtins. The only other thing I can think of is not having struct support since I don't have elfutils installed (I don't think it builds for macOS), and the code it tries to compile has a struct. I might be barking up the wrong tree though; I don't know how to resolve this myself.

    Thank you for your time.

    opened by beaumartinez 16
  • Consider busybox support?

    Consider busybox support?

    I'd love to use ctypes.sh in my embedded platforms, but they all use Busybox and ash (Almquist Shell) due to size constraints. Have you considered supporting Busybox's ash?

    I think it would be great to be able to have the ctypes.sh capability on the embedded platform to marry shell scripting and the compiled code ... but without the bloat of full bash.

    Regardless of what you decide, I'll probably look into it downstream.

    opened by crazychenz 0
  • Dependencies for Struct support on Ubuntu 18.04

    Dependencies for Struct support on Ubuntu 18.04

    Here's what I needed to install before ctypes.sh would compile and pass all test in /tests folder:

    # mandatory to built ctypes.sh without Structs:
    sudo apt install autoconf libffi-dev libltdl-dev automake libtool
    
    

    above are mandatory for ./autogen.sh to build ./configure

    # mandatory to build ctypes.sh with Structs:
    sudo apt install libdw-dev libelf-dev
    
    

    The build script states that the failure is a lack of elfutils when actually it's a lack of libdwfl.h

    The error would be much more helpful if it reported what the actual failure was instead of reporting the name of a package which it mistakenly assumes will bring in what's missing.

    opened by w-barath 2
Releases(v1.2)
Owner
Tavis Ormandy
Tavis Ormandy
A tool for generating cross-language type declarations and interface bindings.

Djinni Djinni is a tool for generating cross-language type declarations and interface bindings. It's designed to connect C++ with either Java or Objec

Dropbox 2.8k Sep 27, 2022
Erlang interface to eBPF

ebpf Erlang eBPF library Overview ebpf facilitates basic interaction with the Linux eBPF system from Erlang.

null 48 Sep 16, 2022
A portable foreign-function interface library.

Status libffi-3.4 was released on TBD. Check the libffi web page for updates: URL:http://sourceware.org/libffi/. What is libffi? Compilers for high le

null 2.6k Oct 5, 2022
The DSiLanguagePatcher increases accessibility to foreign region DSi consoles by providing a mean to change the user interface language.

DSi Language Patcher The DSi Language patcher is a small tool, which runs on your DSi (homebrew execution required) and create a copy of your original

null 19 May 13, 2022
Bash math utilities

Bashmash Bash math utilities What is Bashmash? Bashmash is a set of math utilities for the Bash language. It simplifies common mathematical operations

Hubert Pastyrzak 11 Mar 11, 2021
Blog post on using a custom Bash builtin to parse INI config files

Writing a Bash Builtin in C to Parse INI Configs Why Not Just Parse INI Configs With Bash? Shell languages such as Bash excel at certain tasks, such a

Jesse Hathaway 15 Aug 12, 2022
CVE-2021-3493 Ubuntu OverlayFS Local Privesc (Interactive Bash Shell & Execute Command Entered)

CVE-2021-3493 Ubuntu OverlayFS Local Privesc Description "Ubuntu specific issue in the overlayfs file system in the Linux kernel where it did not prop

3ND 32 Aug 29, 2022
bash prompt

promptprint Dependencies C compiler Description My bash prompt was getting complex, so I decided to turn it into a tiny c program. It displays an elid

martin 2 Nov 25, 2021
hooking the execve syscall, to randomly sabotage typed bash commands.

Syscall hooks A small project of hooking the execve() syscall, to randomly sabotage typed bash commands. This project was tested on 5.11.0-38-generic.

ilevi 4 Aug 2, 2022
This program try to recreate bash --posix comportement in certain limite

minishell : petitcoquillage This program try to recreate bash --posix comportement in certain limite Execution : To execute this Program you have to b

null 61 Sep 15, 2022
This project simulates the functionality of the Unix Bash shell.

Minishell This project simulates the functionality of the unix Bash shell. Description The project aim was to recreate the functionality of the unix B

Andreas Gebert 0 Jan 6, 2022
Minishell - This project aims to recreate most of the bash shell.

minishell Contents Info How to use Clone repo and submodules Linux dependencies MAC dependencies Compile and execute Install Features Built-in command

izenynn 2 Jun 9, 2022
A purely bash web server, no socat, netcat, etc...

Bash-web-server A purely bash web server, no socat, netcat, etc... Requirement bash 5.1 loadable accept builtin (http://git.savannah.gnu.org/cgit/bash

null 615 Sep 25, 2022
File path converter for Windows & Git Bash

windows-git-bash-path-converter Motivation Made this because it was so mad to convert path between Windows and Git Bash How to use Windows file path t

Jooho Lee 3 Mar 15, 2022
📖 A collection of pure bash alternatives to external processes.

NEW: pure sh bible ( ?? A collection of pure POSIX sh alternatives to external processes). pure bash bible A collection of pure bash alternatives to e

dylan 31.9k Oct 1, 2022
Collection of DLL function export forwards for DLL export function proxying

dll-exports Collection of DLL function export forwards for DLL export function proxying. Typical usecase is for backdooring applications for persisten

Magnus Stubman 47 Sep 14, 2022
C-function for traversing files/directories effectively and calling a given function with each encountered file and a void-pointer as parameters

C-function for traversing files/directories effectively and calling a given function with each encountered file and a void-pointer as parameters

null 1 Jun 27, 2022
Legion Low Level Rendering Interface provides a graphics API agnostic rendering interface with minimal CPU overhead and low level access to verbose GPU operations.

Legion-LLRI Legion-LLRI, or “Legion Low Level Rendering Interface” is a rendering API that aims to provide a graphics API agnostic approach to graphic

Rythe Interactive 26 Aug 13, 2022
A testing micro framework for creating function test doubles

Fake Function Framework (fff) A Fake Function Framework for C Hello Fake World! Capturing Arguments Return Values Resetting a Fake Call History Defaul

Mike Long 515 Sep 29, 2022
Nameof operator for modern C++, simply obtain the name of a variable, type, function, macro, and enum

_ _ __ _____ | \ | | / _| / ____|_ _ | \| | __ _ _ __ ___ ___ ___ | |_ | | _| |

Daniil Goncharov 1.4k Sep 26, 2022