Node.js Workers, except on the same thread

Overview

synchronous-worker – Run Node.js APIs synchronously

Usage Example

const w = new SynchronousWorker();
const fetch = w.createRequire(__filename)('node-fetch');
const response = w.runLoopUntilPromiseResolved(fetch('http://example.org'));
const text = w.runLoopUntilPromiseResolved(response.text());
console.log(text);

API

new SynchronousWorker([options])

Create a new Node.js instance on the same thread. Valid options are:

  • sharedEventLoop: Use the same event loop as the outer Node.js instance. If this is passed, the .runLoop() and .runLoopUntilPromiseResolved() methods become unavailable. Defaults to false.
  • sharedMicrotaskQueue: Use the same microtask queue as the outer Node.js instance. This is used for resolving promises created in the inner context, including those implicitly generated by async/await. If this is passed, the .runLoopUntilPromiseResolved() method becomes unavailable. Defaults to false.

While this package will accept { sharedEventLoop: false, sharedMicrotaskQueue: true } as options, passing them does not typically make sense.

synchronousWorker.runLoop([mode])

Spin the event loop of the inner Node.js instance. mode can be either default, once or nowait. See the libuv documentation for uv_run() for details on these modes.

synchronousWorker.runLoopUntilPromiseResolved(promise)

Spin the event loop of the innsert Node.js instance until a specific Promise is resolved.

synchronousWorker.runInWorkerScope(fn)

Wrap fn and run it as if it were run on the event loop of the inner Node.js instance. In particular, this ensures that Promises created by the function itself are resolved correctly. You should generally use this to run any code inside the innert Node.js instance that performs asynchronous activity and that is not already running in an asynchronous context (you can compare this to the code that runs synchronously from the main file of a Node.js application).

synchronousWorker.loopAlive

This is a read-only boolean property indicating whether there are currently any items on the event loop of the inner Node.js instance.

synchronousWorker.stop()

Interrupt any execution of code inside the inner Node.js instance, i.e. return directly from a .runLoop(), .runLoopUntilPromiseResolved() or .runInWorkerScope() call. This will render the Node.js instance unusable and is generally comparable to running process.exit().

This method returns a Promise that will be resolved when all resources associated with this Node.js instance are released. This Promise resolves on the event loop of the outer Node.js instance.

synchronousWorker.createRequire(filename)

Create a require() function that can be used for loading code inside the inner Node.js instance. See module.createRequire() for details.

synchronousWorker.globalThis

Returns a reference to the global object of the inner Node.js instance.

synchronousWorker.process

Returns a reference to the process object of the inner Node.js instance.

FAQ

What does this module do?

Create a new Node.js instance, using the same thread and the same JS heap. You can create Node.js API objects, like network sockets, inside the new Node.js instance, and spin the underlying event loop manually.

Why would I use this package?

The most common use case is probably running asynchronous code synchronously, in situations where doing so cannot be avoided (even though one should try really hard to avoid it). Another popular npm package that does this is deasync, but deasync

  • solves this problem by starting the event loop while it is already running (which is explicitly not supported by libuv and may lead to crashes)
  • doesn’t allow specifying which resources or callbacks should be waited for, and instead allows everything inside the current thread to progress.

How can I avoid using this package?

If you do not need to directly interact with the objects inside the inner Node.js instance, a lot of the time Worker threads together with Atomics.wait() will give you what you need. For example, the node-fetch snippet from above could also be written as:

const {
  Worker, MessageChannel, receiveMessageOnPort
} = require('worker_threads');

const { port1, port2 } = new MessageChannel();
const notifyHandle = new Int32Array(new SharedArrayBuffer(4));

const w = new Worker(`
const {
  parentPort, workerData: { notifyHandle, port2 }
} = require('worker_threads');

(async () => {
  const fetch = require('node-fetch');
  const response = await fetch('http://example.org');
  const text = await response.text();
  port2.postMessage({ text });
  Atomics.store(notifyHandle, 0, 1);
  Atomics.notify(notifyHandle, 0);
})();`, {
  eval: true, workerData: { notifyHandle, port2 }, transferList: [ port2 ]
});

Atomics.wait(notifyHandle, 0, 0);
const { text } = receiveMessageOnPort(port1).message;
console.log(text);

That’s arguably a bit more complicated, but doesn’t require any native code and only uses APIs that are also available on lower Node.js versions.

Which Node.js versions are supported?

In order to work, synchronous-worker needs a recent Node.js version, because older versions are missing a few bugfixes or features. The following PRs are relevant for this (all of them are included in Node.js 15.5.0):

My async functions/Promises/… don’t work

If you run a SynchronousWorker with its own microtask queue (i.e. in default mode), code like this will not work as expected:

const w = new SynchronousWorker();
let promise;
w.runInWorkerScope(() => {
  promise = (async() => {
    return await w.createRequire(__filename)('node-fetch')(...);
  })();
});
w.runLoopUntilPromiseResolved(promise);

The reason for this is that async functions (and Promise .then() handlers) add their microtasks to the microtask queue for the Context in which the async function (or .then() callback) was defined, and not the Context in which the original Promise was created. Put in other words, it is possible for a Promise chain to be run on different microtask queues.

While I find this behavior counterintuitive, it is what the V8 engine does, and is not under the control of Node.js or this package.

What this means is that you will need to make sure that the functions are compiled in the Context in which they are supposed to be run; the two main ways to achieve that are to:

  • Put them in a separate file that is loaded through w.createRequire()
  • Use w.createRequire(__filename)('vm').runInThisContext() to manually compile the code for the function in the Context of the target Node.js instance.

For example:

const w = new SynchronousWorker();
const req = w.createRequire(__filename);
let promise;
w.runInWorkerScope(() => {
  promise = req('vm').runInThisContext(`(async(req) => {
    return await req('node-fetch')(...);
  })`)(req));
});
w.runLoopUntilPromiseResolved(promise);

I found a bug/crash while using this package. What do I do now?

You can file a bug report on Github. Please include a reproduction, the version of this package that you’re using, and the Node.js version that you’re using, and ideally also make sure that it’s a first-time report.

Issues
  • Running the example from the readme results in a

    Running the example from the readme results in a "symbol lookup error"

    When run on a Raspberry Pi 4, the example program from the readme results in a "symbol lookup error".

    Here is some information related to the setup:

    [email protected]:~/synchronous-worker $ node -v
    v15.5.0
    [email protected]:~/synchronous-worker $ npm -v
    7.3.0
    [email protected]:~/synchronous-worker $ gcc --version
    gcc (Raspbian 8.3.0-6+rpi1) 8.3.0
    Copyright (C) 2018 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    
    [email protected]:~/synchronous-worker $ uname -a
    Linux raspberrypi 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux
    [email protected]:~/synchronous-worker $ cat /etc/debian_version 
    10.4
    [email protected]:~/synchronous-worker $ 
    

    When installed, synchronous-worker installs successfully:

    [email protected]:~/synchronous-worker $ npm i synchronous-worker
    
    added 3 packages, and audited 4 packages in 5s
    
    found 0 vulnerabilities
    [email protected]:~/synchronous-worker $ 
    

    Running the example program from the readme results in a "symbol lookup error":

    [email protected]:~/synchronous-worker $ cat t.js 
    const SynchronousWorker = require('synchronous-worker');
    const w = new SynchronousWorker();
    const fetch = w.createRequire(__filename)('node-fetch');
    const response = w.runLoopUntilPromiseResolved(fetch('http://example.org'));
    const text = w.runLoopUntilPromiseResolved(response.text());
    console.log(text);
    
    [email protected]:~/synchronous-worker $ node t.js 
    node: symbol lookup error: /home/pi/synchronous-worker/node_modules/synchronous-worker/build/Release/synchronous_worker.node: undefined symbol: _ZN4node17CreateEnvironmentEPNS_11IsolateDataEN2v85LocalINS2_7ContextEEERKSt6vectorISsSaISsEESA_NS_16EnvironmentFlags5FlagsENS_8ThreadIdESt10unique_ptrINS_21InspectorParentHandleESt14default_deleteISF_EE
    [email protected]:~/synchronous-worker $ 
    

    I downloaded Node.js v15.5.0 (node-v15.5.0-linux-armv7l.tar.gz) here. I'm not sure if it's relevant, but the Raspberry Pi 4 is a 64-bit machine and node-v15.5.0-linux-armv7l.tar.gz is the 32-bit version of Node.js v15.5.0. Things were done this way as I'm using the 32-bit version of Raspberry Pi OS (previously called Raspbian).

    opened by fivdi 9
  • reject the promise if stop() throws

    reject the promise if stop() throws

    I have situation where calling stop() would cause an 'illegal operation' error being thrown by FreeEnvironment (I arrived there through some printfs).

    Funnily enough, I can only reproduce this on top of Fastify+tap. I've tried reasoning on why could be causing this but I was not able to recreate the same "perfect storm". If you'd like, y7ou can take a look at this branch: https://github.com/mcollina/fastify-isolate/tree/broken-isolate.

    Let me know if you have some way in which I could find out how to write a unit test for this.

    opened by mcollina 3
  • Node v18 support

    Node v18 support

    When adding support to my module to Node v18, I had the following error:

    node: symbol lookup error: /home/matteo/repositories/synchronous-worker/build/Release/synchronous_worker.node: undefined symbol: _ZN4node17CreateEnvironmentEPNS_11IsolateDataEN2v85LocalINS2_7ContextEEERKSt6vectorISsSaISsEESA_NS_16EnvironmentFlags5FlagsENS_8ThreadIdESt10unique_ptrINS_21InspectorParentHandleESt14default_deleteISF_EE
    

    Note I have this error only on Linux and not on Mac.

    opened by mcollina 2
  • Node 17.6.0 failed to compile

    Node 17.6.0 failed to compile

    npm i --save synchronous-worker fails on node 17.6.0 and npm 8.5.2

    npm ERR! code 1
    npm ERR! path /Users/yagiz/Developer/project/node_modules/synchronous-worker
    npm ERR! command failed
    npm ERR! command sh -c node-gyp rebuild
    npm ERR! CXX(target) Release/obj.target/synchronous_worker/src/binding.o
    npm ERR! gyp info it worked if it ends with ok
    npm ERR! gyp info using [email protected]
    npm ERR! gyp info using [email protected] | darwin | arm64
    npm ERR! gyp info find Python using Python version 3.9.10 found at "/opt/homebrew/opt/[email protected]/bin/python3.9"
    npm ERR! gyp info spawn /opt/homebrew/opt/[email protected]/bin/python3.9
    npm ERR! gyp info spawn args [
    npm ERR! gyp info spawn args   '/Users/yagiz/.npm-global/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
    npm ERR! gyp info spawn args   'binding.gyp',
    npm ERR! gyp info spawn args   '-f',
    npm ERR! gyp info spawn args   'make',
    npm ERR! gyp info spawn args   '-I',
    npm ERR! gyp info spawn args   '/Users/yagiz/Developer/project/node_modules/synchronous-worker/build/config.gypi',
    npm ERR! gyp info spawn args   '-I',
    npm ERR! gyp info spawn args   '/Users/yagiz/.npm-global/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
    npm ERR! gyp info spawn args   '-I',
    npm ERR! gyp info spawn args   '/Users/yagiz/Library/Caches/node-gyp/17.6.0/include/node/common.gypi',
    npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
    npm ERR! gyp info spawn args   '-Dvisibility=default',
    npm ERR! gyp info spawn args   '-Dnode_root_dir=/Users/yagiz/Library/Caches/node-gyp/17.6.0',
    npm ERR! gyp info spawn args   '-Dnode_gyp_dir=/Users/yagiz/.npm-global/lib/node_modules/npm/node_modules/node-gyp',
    npm ERR! gyp info spawn args   '-Dnode_lib_file=/Users/yagiz/Library/Caches/node-gyp/17.6.0/<(target_arch)/node.lib',
    npm ERR! gyp info spawn args   '-Dmodule_root_dir=/Users/yagiz/Developer/project/node_modules/synchronous-worker',
    npm ERR! gyp info spawn args   '-Dnode_engine=v8',
    npm ERR! gyp info spawn args   '--depth=.',
    npm ERR! gyp info spawn args   '--no-parallel',
    npm ERR! gyp info spawn args   '--generator-output',
    npm ERR! gyp info spawn args   'build',
    npm ERR! gyp info spawn args   '-Goutput_dir=.'
    npm ERR! gyp info spawn args ]
    npm ERR! gyp info spawn make
    npm ERR! gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
    npm ERR! ../src/binding.cc:5:9: warning: '_GLIBCXX_USE_CXX11_ABI' macro redefined [-Wmacro-redefined]
    npm ERR! #define _GLIBCXX_USE_CXX11_ABI 0
    npm ERR!         ^
    npm ERR! <command line>:7:9: note: previous definition is here
    npm ERR! #define _GLIBCXX_USE_CXX11_ABI 1
    npm ERR!         ^
    npm ERR! ../src/binding.cc:231:28: error: invalid argument type 'v8::Maybe<bool>' to unary expression
    npm ERR!   if (context.IsEmpty() || !InitializeContext(context)) {
    npm ERR!                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~
    npm ERR! 1 warning and 1 error generated.
    npm ERR! make: *** [Release/obj.target/synchronous_worker/src/binding.o] Error 1
    npm ERR! gyp ERR! build error
    npm ERR! gyp ERR! stack Error: `make` failed with exit code: 2
    npm ERR! gyp ERR! stack     at ChildProcess.onExit (/Users/yagiz/.npm-global/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
    npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:527:28)
    npm ERR! gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
    npm ERR! gyp ERR! System Darwin 21.3.0
    npm ERR! gyp ERR! command "/opt/homebrew/Cellar/node/17.6.0/bin/node" "/Users/yagiz/.npm-global/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
    npm ERR! gyp ERR! cwd /Users/yagiz/Developer/project/node_modules/synchronous-worker
    npm ERR! gyp ERR! node -v v17.6.0
    npm ERR! gyp ERR! node-gyp -v v8.4.1
    npm ERR! gyp ERR! not ok
    
    npm ERR! A complete log of this run can be found in:
    npm ERR!     /Users/yagiz/.npm/_logs/2022-02-27T18_36_41_050Z-debug-0.log
    
    opened by anonrig 0
Owner
Anna Henningsen
she/her
Anna Henningsen
High Quality DeNoise 3D is an AviSynth port of the MPlayer filter of the same name

High Quality DeNoise 3D is an AviSynth port of the MPlayer filter of the same name. It performs a 3-way low-pass filter, which can completely remove high-frequency noise while minimizing blending artifacts.

null 12 Jun 7, 2022
Flutter app that syncs clipboards between devices in the same local network.

clipboard_sync A flutter app that syncs clipboards between devices in the same LAN using Interprocess communication (Sockets). What it does ? group of

Pushpavel 5 Oct 29, 2021
This project helps a person park their car in their garage in the same place every time.

garage-parking-sensor Description This project is developed to help a person park their car in their garage in the same place every time. Normally peo

Calvin Pereira 2 Sep 13, 2021
StarkScript - or the Stark programming language - is a compiled C-based programming language that aims to offer the same usability as that of JavaScript's and TypeScript's

StarkScript StarkScript - or the Stark programming language - is a compiled C-based programming language that aims to offer the same usability as that

EnderCommunity 5 May 10, 2022
MozJPEG improves JPEG compression efficiency achieving higher visual quality and smaller file sizes at the same time

Mozilla JPEG Encoder Project MozJPEG improves JPEG compression efficiency achieving higher visual quality and smaller file sizes at the same time. It

Mozilla 4.9k Jun 24, 2022
This repo contains BOTH c++ and BP examples to acheive the same logic, but in each frameworks specific ways

ApparatusCppMoveRandomly Hey there! This repo contains BOTH c++ and BP examples to acheive the same logic, but in each frameworks specific ways. I int

null 2 Jan 24, 2022
It creates a random word by mixing two English common words into a single one, each one with the first character in capital letter. It also allow you to scroll down infinitely without repeating the same word twice.

startup_namer A new Flutter project. Getting Started This project is a starting point for a Flutter application. A few resources to get you started if

Samuel Cobas 2 Feb 3, 2022
Yeah, so basically this program will give you the same information of a neofetch with the difference that is written in C.

neofetch-c Basically this program will give you the same information of a neofetch with the difference that is written in C. Build Compile Before runn

Francesco P. 2 Mar 27, 2022
A C implementation of Alien: Fate of the Nostromo, a 2021 board game of the same name

ALIEN: Fate of The Nostromo This is a terminal-based clone of the "ALIEN: Fate of The Nostromo" board game, implemented in C. Installation Debug Insta

Charles Averill 15 Jun 10, 2022
SmartShunt ve.direct to ESPHOME node

VictronSmartShunt-ESPHOME SmartShunt ve.direct to ESPHOME node A configured uart component is required. Example: victron: uart_id: the_uart sensor:

null 17 May 9, 2022
LLVM bindings for Node.js/JavaScript/TypeScript

llvm-bindings LLVM bindings for Node.js/JavaScript/TypeScript Supported OS macOS Ubuntu Windows Supported LLVM methods listed in the TypeScript defini

ApsarasX 229 Jun 27, 2022
Integrate the ZENO node system into Blender for creating robust physics animations!

ZenoBlend Integrate the ZENO node system into Blender for creating robust physics animations! End-user Installation Goto Release page, and click Asset

Zenus Technology 31 Jun 10, 2022
Arduino M-BUS Master node for Arduino MKR M-BUS Shield

Arduino M-BUS Master node for Arduino MKR M-BUS Shield This software will read out a M-BUS device connected to an Arduino MKR board equipped with our

null 4 Dec 11, 2021
hotcaKey is the global shortcut (aka hotkey) module for node.js and electron.

?? hotcaKey is the global shortcut (aka hotkey) module for node.js and electron. hotcakey is now actively under deploment, so api may have

daylilyfield 6 Jun 20, 2022
A C++ Node.js module that helps gathering informations on segmentation fault

node-segfault-handler A C++ Node.js module that helps gathering informations on segmentation fault Supported Platforms Linux Linux Alpine Windows MacO

Shiranuit 9 Jun 17, 2022
Node running standalone on PC, with interface - self-containing all dependencies

GMD Node Windows Application It is the GMD Node App for Windows packaged in a simple "one-click" installer containing all necessary dependencies. We a

Geoma COOP 3 Jan 4, 2022
A rosbag2 recorder node that backs up split files to an external location during recording

System Data Recorder (SDR) A lifecycle node and executable for recording topic data to a rosbag2 bag, while simultaneously copying the split bag files

Open Robotics 4 Mar 9, 2022
A composable container for Adaptive ROS 2 Node computations. Select between FPGA, CPU or GPU at run-time.

adaptive_component A composable stateless container for Adaptive ROS 2 Node computations. Select between FPGA, CPU or GPU at run-time. Nodes using har

ROS 2 Hardware Acceleration Working Group 6 Apr 25, 2022
Seam is a pin-based node editor for OpenFrameworks that makes prototyping visual systems easier and faster.

seam Seam is a pin-based node editor for openFrameworks built using: openFrameworks Dear ImGui the node editor extension for ImGui It is heavily WIP,

Austin Clifton 2 Jan 2, 2022