Watch for file changes and auto restart an application using fork checkpoints to continue the process (for quick live development)

Overview

Forkmon

Watch for file changes and auto restart an application using fork checkpoints to continue. Intended for quick live development. This works only on Linux systems.

Quick Usage

Small example of compiling forkmon and using with Lua to reload scripts on the middle:

git clone https://github.com/edubart/forkmon.git && cd forkmon
make
alias forkmon-lua='LD_PRELOAD=`pwd`/forkmon.so FORKMON_FILTER="%.lua$" lua'
forkmon-lua tests/example.lua

Now when running the above you should get something like:

[forkmon] watch '/home/bart/projects/forkmon/example.lua'
startup
[forkmon] watch '/home/bart/projects/forkmon/foo.lua'
foo!
finished

The application will keep running, waiting for any watched file change. Now try to edit to foo.lua and change the print to 'hello world!', you should get something like:

[forkmon] file '/home/bart/projects/forkmon/tests/foo.lua' changed, resuming from it..
hello world!
finished

Notice that only 'hello world!' is printed, but not 'startup', this means the application did not restart from beginning, but from the middle instead! Because a forked (cloned) application was waiting for changes in the foo.lua and resumed.

How it works

The Linux fork() function has the interesting property of cloning a child process when called, the child process memory remains the same as the parent process, however both can run independently, each one with its own state and memory. But this is not a heavy operation, the process memory is not duplicated, instead copy on write is used, thus this is a lightweight operation. This property of fork() allows to us to create checkpoints of the program that we can later use to resume a new process from the checkpoint without require the application to start from beginning, effectively rolling back state and memory in time. If we hook all fopen() calls we can make checkpoints every time a file is opened, then using inotify Linux API we can watch files for changes and once a file change is detected we can resume a new process from its checkpoint instead of restarting the whole application. This allows to gain a few startup time in live development scenarios.

Options

The tool can be configured using the following environment variables:

  • FORKMON_FILTER pattern that a watched file name should match, following Lua pattern rules, multiple filter patterns can be used when using the ; separator.
  • FORKMON_QUIET if set, the tool will be quiet and not print anything.
  • FORKMON_NO_COLORS if set, no colors will be used in terminal output.
  • FORKMON_RESTART_DELAY how many milliseconds to wait before restarting the application when a file has changed (default 20). Must be more than 0, so the OS can properly flush all file changes.
  • FORKMON_IGNORE_DELAY how many milliseconds to ignore new changes after a file has been changed (default 200). Must be more than 0, so when saving files in a batch does not trigger many restarts.

Motivation

I had this idea other day when thinking in ways to speedup the Nelua compiler, this tool can be used there to skip redundant compiler work. Because usually when you edit a source file the compiler needs to go parsing all sources again, even things before the source file change. This tool is a "hacky" way to allow the compiler to skip parsing and analyzing everything before a source file change. And if the sources are designed in such a way that a separate single source file requires all hardly ever changing files (similar to precompiled headers in C world), then the compilation can be much faster by skipping lots of parsing.

Although this was made for quick live development with Nelua on Linux, it can be used to speedup other console applications or compilers. And probably even servers that goes through a lot of loading and processing during startup, though the server startup need to be designed in such way that the checkpoints places does not have networking or multithreading going on.

Limitations

  • Only works well with application where all state to checkpoint is available in CPU memory, such as single threaded console applications. This is not the case for networking, graphical or multithreading applications, to make it work in such kind of applications extra work would be needed.
  • The application must use open, fopen or fopen64 to open files, as these are the only file opening functions hooked. Some applications uses the openat syscall and this case will be missed.
  • The application should not launch sub processes, thus this tool does not work well with GCC/Clang compilers as they launch itself when compiling, however it works with TCC compiler thus TCC could be even faster for live development!
  • File descriptor offset are shared between processes (a fork() behavior), and this can be problematic for applications that keep files open instead of caching them in memory.

Troubleshooting

In case you get the error [forkmon] failed to initialize inotify then is probably because too many inotify instances are active, then try to increase this limit with sudo sysctl -w user.max_inotify_instances=1024 or kill any zombie processes of the application (in edge cases can happen).

Implementation details

This has been implemented using the Nelua programming language, however a standalone C file is bundled in the repository, thus just a C compiler is needed to compile the project.

License

MIT License, see LICENSE

Issues
  • It doesn't work as expected with some editors

    It doesn't work as expected with some editors

    Just add the option to remove remove bin/linux64/forkmon.so as well.

    By the way, I cannot make it work...well, the alias work, but does not monitor :shrug:

    alias forkmon-nelua='LD_PRELOAD=/PortableApps/forkmon/bin/linux64/forkmon.so FORKMON_FILTER=".nelua$" nelua'

    opened by stefanos82 33
Owner
Eduardo Bart
Open source developer, creating Nelua programming language and other development tools.
Eduardo Bart
A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)

Catch2 v3 is being developed! You are on the devel branch, where the next major version, v3, of Catch2 is being developed. As it is a significant rewo

Catch Org 15.4k Aug 1, 2022
A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)

Catch2 v3 is being developed! You are on the devel branch, where the next major version, v3, of Catch2 is being developed. As it is a significant rewo

Catch Org 15.3k Jul 26, 2022
DotX64Dbg aims to provide a seamless way to write and test plugins for X64Dbg using .Net 5.0 and C#.

DotX64Dbg (EARLY ALPHA) Plugins and Scripting with C# for x64Dbg. Create Plugins for X64Dbg with ease DotX64Dbg aims to provide a seamless way to writ

ζeh Matt 7 Jan 21, 2022
The world's first free and open-source PlayStation 3 emulator/debugger, written in C++ for Windows and Linux.

The world's first free and open-source PlayStation 3 emulator/debugger, written in C++ for Windows and Linux.

null 11.5k Aug 4, 2022
CppUTest unit testing and mocking framework for C/C++

CppUTest CppUTest unit testing and mocking framework for C/C++ More information on the project page Slack channel: Join if link not expired Getting St

CppUTest 1.1k Jul 27, 2022
Googletest - Google Testing and Mocking Framework

GoogleTest OSS Builds Status Announcements Release 1.10.x Release 1.10.x is now available. Coming Soon Post 1.10.x googletest will follow Abseil Live

Google 27.1k Jul 31, 2022
A simple C++ 03/11/etc timer class for ~microsecond-precision cross-platform benchmarking. The implementation is as limited and as simple as possible to create the lowest amount of overhead.

plf_nanotimer A simple C++ 03/11/etc timer class for ~microsecond-precision cross-platform benchmarking. The implementation is as limited and simple a

Matt Bentley 89 Jul 15, 2022
🧪 single header unit testing framework for C and C++

?? utest.h A simple one header solution to unit testing for C/C++. Usage Just #include "utest.h" in your code! The current supported platforms are Lin

Neil Henning 499 Aug 5, 2022
Anti-Debug and Anti-Memory Dump for Android

AntiDebugandMemoryDump Anti-Debug and Anti-Memory Dump for Android Some known techniques for anti-debug and anti-memory dump have been used in this pr

Darvin 165 Aug 5, 2022
An efficient OpenFST-based tool for calculating WER and aligning two transcript sequences.

fstalign Overview Installation Dependencies Build Docker Quickstart WER Subcommand Align Subcommand Inputs Outputs Overview fstalign is a tool for cre

Rev 103 Jun 1, 2022
🍦IceCream-Cpp is a little (single header) library to help with the print debugging on C++11 and forward.

??IceCream-Cpp is a little (single header) library to help with the print debugging on C++11 and forward.

Renato Garcia 388 Aug 2, 2022
HyperDbg debugger is an open-source, hypervisor-assisted user-mode, and kernel-mode Windows debugger 🐞

HyperDbg debugger is an open-source, hypervisor-assisted user-mode, and kernel-mode Windows debugger with a focus on using modern hardware technologies. It is a debugger designed for analyzing, fuzzing and reversing. ??

HyperDbg 1.7k Jul 28, 2022
Palanteer is a set of high performance visual profiler, debugger, tests enabler for C++ and Python

Palanteer is a set of lean and efficient tools to improve the general software quality, for C++ and Python programs.

Damien Feneyrou 1.9k Aug 2, 2022
Hibizcus is a collection of tools - Font proofing and debugging tools

Hibizcus Font proofing and debugging tools. Written by: Muthu Nedumaran Hibizcus is a collection of tools written to proof and debug in-house develope

Muthu Nedumaran 18 Jun 21, 2022
Windows-only Remote Access Tool (RAT) with anti-debugging and anti-sandbox checks

RATwurst Windows-only Remote Access Tool (RAT) with anti-debugging and anti-sandbox checks. For educational purposes only. The reason behind this proj

AccidentalRebel 28 Jul 24, 2022
x64Dbg plugin that enables C# plugins with hot-loading support and scripting.

DotX64Dbg (EARLY ALPHA) Plugins and Scripting with C# for x64Dbg. Create Plugins for X64Dbg with ease DotX64Dbg aims to provide a seamless way to writ

x64dbg 83 Jul 23, 2022
A simple D3D11 Hook for x64 and x86 games. This project is ready to compile (x64 or x86).

D3D11Hook Features: Good Performance Simple, clean, GUI. Rendering using ImGui Clean code Easy to use with another project C++ 17 Xor String Ready to

null 15 Jul 26, 2022
Templight 2.0 - Template Instantiation Profiler and Debugger

Templight is a Clang-based tool to profile the time and memory consumption of template instantiations and to perform interactive debugging sessions to gain introspection into the template instantiation process.

Sven Mikael Persson 588 Jul 21, 2022
Parca-agent - eBPF based always-on profiler auto-discovering targets in Kubernetes and systemd, zero code changes or restarts needed!

Parca Agent Parca Agent is an always-on sampling profiler that uses eBPF to capture raw profiling data with very low overhead. It observes user-space

Parca 170 Aug 2, 2022
A modern day direct port of BOOM 2.02 for modern times. Aiming to tastefully continue the development of BOOM, in the style of TeamTNT.

ReBOOM ReBOOM is a continuation of the BOOM source port, version 2.02. what is it ReBOOM is a source port, directly ported from BOOM 2.02 with additio

Gibbon 12 Jul 27, 2022
T-Watch 2020 v1 compatible firmware providing WiFi and BLE testing tools (and also, a watch :D)

ESP-IDF template app This is a template application to be used with Espressif IoT Development Framework. Please check ESP-IDF docs for getting started

Damien Cauquil 39 Jul 31, 2022
Minimal Linux Live (MLL) is a tiny educational Linux distribution, which is designed to be built from scratch by using a collection of automated shell scripts. Minimal Linux Live offers a core environment with just the Linux kernel, GNU C library, and Busybox userland utilities.

Minimal Linux Live (MLL) is a tiny educational Linux distribution, which is designed to be built from scratch by using a collection of automated shell scripts. Minimal Linux Live offers a core environment with just the Linux kernel, GNU C library, and Busybox userland utilities.

John Davidson 1.3k Aug 2, 2022
C64 Watch is a customized T-Watch 2020 that was inspired by the Commodore 64 computer. It features a C64 theme and a built-in BASIC interpreter.

C64 Watch C64 Watch is a customized T-Watch 2020 that was inspired by the Commodore 64 computer. It features a C64 theme and a built-in BASIC interpre

Nick Bild 28 Jul 24, 2022
✔️The smallest header-only GUI library(4 KLOC) for all platforms

Welcome to GUI-lite The smallest header-only GUI library (4 KLOC) for all platforms. 中文 Lightweight ✂️ Small: 4,000+ lines of C++ code, zero dependenc

null 6.3k Jul 31, 2022
Multi Theft Auto (MTA) is a software project that adds network play functionality to Rockstar North's Grand Theft Auto game series,

Multi Theft Auto: San Andreas Multi Theft Auto (MTA) is a software project that adds network play functionality to Rockstar North's Grand Theft Auto g

null 2 Oct 22, 2021
Process Ghosting - a PE injection technique, similar to Process Doppelgänging, but using a delete-pending file instead of a transacted file

Process Ghosting This is my implementation of the technique presented by Gabriel Landau: https://www.elastic.co/blog/process-ghosting-a-new-executable

hasherezade 462 Jul 25, 2022
anthemtotheego 342 Jul 23, 2022
U++ is a C++ cross-platform rapid application development framework focused on programmer's productivity. It includes a set of libraries (GUI, SQL, Network etc.), and integrated development environment (TheIDE).

Ultimate++ Ultimate++ is a C++ cross-platform rapid application development framework focused on programmers productivity. It includes a set of librar

Ultimate++ 464 Jul 26, 2022