Were there any major changes to garbage collection?

Take this with a grain of salt since usually the bugs I present is some weird thing I messed up in my side, but this was a strange one.

In my app I have certain indices in the stash reserved for some object pools. The pools get allocated at the beginning as arrays, and they are left there. I simply grab the heapptr and add/remove objects to that array. I haven't had any issues with it ever, suddenly today I got an assert triggered when I pushed the stashed pool's heapptr:

/* One particular problem case is where an object has been
		 * queued for finalization but the finalizer hasn't been
		 * executed.
		 */
		duk_heaphdr *curr;
		for (curr = thr->heap->finalize_list;
		     curr != NULL;
		     curr = DUK_HEAPHDR_GET_NEXT(thr->heap, curr)) {
			DUK_ASSERT(curr != (duk_heaphdr *) ptr);
		}

Again, I might have corrupted something somewhere, who knows, just looking for some info in case there were some changes that might have caused this? This portion of my code (with the pool) hasn't been touched in a very long time...

The heapptr which is an array inside an array inside the heap stash is only ever grabbed from the stash index once at the beginning, the pushed into the stack only 3 places in the code. The stash is not messed with again other than in initializing those arrays at the beginning.

If it helps, the objects that are stored in that pool (the heapptr that caused the assert) are revived when being finalized. That is, the pool is full of preallocated Ecmascirpt objects that I grab from the pool return them when I need one, then nullify that object's index. When the object get's finalized I keep it alive by storing it back in the pool.

Add support for safe debugger heap walking:

• Add a primitive to inspect an object and its properties and flags without side effects. This allows the debug client to enumerate properties, look up the prototype chain, walk referenced objects, etc.
• Add a mechanism for some kind of heap locking to avoid garbage collection while walking the heap.

Tasks:

• [x] Add a new debugger command for inspecting an object
• [x] Add new debugger commands to "lock" the heap for walking (see discussion in #358) => For now, uses automatic locking during paused state; future work is to add an explicit GC during paused state if that is necessary for some reason
• [x] Add sizeof(void *) to BasicInfo
• [x] Add optional heapptr target to GetBytecode?
• [ ] Debugger protocol documentation
• [ ] Wiki page for debugger heap walking with more verbose discussion and example command sequences - https://github.com/svaarala/duktape-wiki/pull/81
• [ ] Some minimal implementation in the web UI
• [ ] Document garbage collection behavior during paused state into debugger documentation; refzero cases go through mark-and-sweep now
• [ ] Read through debugger.rst and update design notes and future work re: inspect
• [ ] Releases

Low level issues to resolve:

• [x] What to do about virtual properties - i.e. properties logically part of the object and visible to Ecmascript code, but not concretely part of the object's internal property table? For comparison, enumeration code needs to "fake" these properties. => For now, virtual properties are not returned in inspection; future work.
• [ ] Make the InspectHeapObject command optional because of the relatively high footprint.
• [ ] Should property keys be object references to heap strings? If yes, what to do about artificial keys? If no, how would user code be able to dump all heap strings? Should there be a LookupString command to find an interned string?

Consider overlaps with existing commands:

• [ ] InspectHeapObject and DumpHeap: Could DumpHeap simply provide a list of heap pointers which the debug client would then inspect using InspectHeapObject so that there'd be no formatting overlap?
• [ ] InspectHeapObject and GetBytecode: GetBytecode could be replaced by simply getting a reference to the currently executing function and then adding the bytecode, constants, and inner functions into the InspectHeapObject result. There's a small issue in representing structured values there; temporary objects may need to be created to hold the constants, etc.

When the debugger is attached and an error is thrown with no active catch site, either a runtime error thrown by Duktape or directly by Ecmascript code, the debug client is notified and execution is paused at the error site to allow the developer to view program state. This is useful, as this information will be lost once the stack is unwound.

Implementation checklist:

• [x] Implement the basic logic for debugger error intercept.
• [x] Add a Throw notification to alert the client to the error.
• [x] Ensure yield/resume awareness to avoid spurious pauses.
• [x] Fix C stack overflow when an error is thrown at max recursion (double error pileup).
• [x] Fix crashes when intercepting errors originating from the compiler (fixed by not intercepting them).
• [x] Fix an assert failure when execution is stopped inside a Duktape/C call.
• [x] Finalize changes and ensure consistency with Duktape codebase.

With #265 bytecode dispatch uses a cached thr->curr_pc pointer which is copied back into the topmost activation act->curr_pc whenever activation states may change or an up-to-date activation state is needed (e.g. for line numbers in the debugger).

This can probably be improved a bit further:

• Use a local variable in the bytecode executor for curr_pc. Make that variable volatile so that it is always stored back into the stack frame after a dispatch.
• Store a pointer to the stack frame location which holds curr_pc (i.e. a duk_instr_t ** with &curr_pc as its value) in thr->ptr_curr_pc.
• Wherever thr->curr_pc was copied back into act->curr_pc after #265, copy the value from *thr->ptr_curr_pc instead.

This is faster because thr is sometimes spilled but the stack frame pointer is not.

Preliminary testing suggests this is roughly 5% faster than #265 (to be verified with more careful testing).

Changes:

• [x] Main source changes to implement the change
• [x] Perhaps simplify management of thr->ptr_curr_pc: bytecode executor can establish it, and restore it before exiting (it always has a setjmp catchpoint so it can do this reliably)
• [x] Check volatile declarations
• [x] Verify performance difference
• [x] Update doc/execution.rst
• [x] Testcases (apitest, qecmatest, test262)
• [x] Test debugger manually

This is a prototype for an implementation approach where Symbols are duk_hstrings with a specific byte prefix which cannot occur for valid (extended) UTF-8 strings. The existing internal string concept is essentially taken a step further to support ES6 Symbols. There's no new internal type for Symbols as such, and the C API sees Symbols as just strings.

• [x] Code change for symbol support
• [x] Fix compiler warnings
• [x] Fix Reflect.keys()
• [x] Testcase support
• [x] Fix perftest coercion issues
• [x] Check key coercion for duk_def_prop() and other property calls
• [x] Wiki pull for symbols howto (should cover the C API aspects especially) - https://github.com/svaarala/duktape-wiki/pull/152
• [x] 2.x migration notes
• [x] Releases entry
• [x] Read through pull comments and add cleanup issues to #1120

Fixes #597. Possibly related to debugger read callback re-entry issue described in #591.

Tasks:

• [x] Fix debugger re-entry when dbg_processing was set to zero during detach1()
• [x] Fix Duktape never doing detach2() if detach1() occurred outside of debugger message loop
• [x] Fix write_cb NULLing for duk_debug_write_byte()
• [x] Any way to repeat @harold-b's issue with repeated read calls not resetting callbacks to NULL? => Could not reproduce.
• [x] Add fix backport notes for earlier releases where appropriate
• [x] Releases

This adds application-defined message support to the debug protocol. When Duktape receives a request message beginning with the code for AppRequest (currently 0x22), it pushes all the remaining dvalues in the message to the valstack and calls heap->dbg_request_cb. That callback can push its own values to the stack, and these are sent back to the client as a REP, or optionally return -1 to send the value on the stack top as an ERR response.

It is also possible for the target to send application-defined notifys (AppNotify, 0x07) to the client by calling duk_debugger_notify().

This branch is merge-ready.

A module loading framework and require() implementation based on Node.js. Semantics match Node.js wherever possible and most features of the Node module system are provided.

https://nodejs.org/api/modules.html

Features:

• Application provides callbacks to perform module resolution and loading
• Module caching based on resolved module ID
• Tolerates circular require() chains
• Supports module.exports, module.loaded, and module.require
• Supports require.cache
• Supports __filename (but not __dirname)

Checklist:

• [x] Expose duk__push_require_function() to calling code? => duk_module_node_init() registers a global require, so not needed
• [x] Manual tests to ensure correctness of all features such as module.loaded, require.cache, etc.
• [x] Full documentation, in particular for callbacks
• [x] More/better sanity checks/guards around callback invocations? => callbacks are now Duktape/C functions
• [x] Rename extra to module-node
• [x] Match Node.js module ID semantics (module.id === module.filename)

Implements #961.

Add TextEncoder and TextDecoder built-ins as described by the WHATWG Encoding specification. For this pull, only UTF-8/CESU-8 support is implemented. UTF-16 and all legacy encodings are left as future work.

Specification: https://encoding.spec.whatwg.org/#api

Checklist:

• [x] Implement the WHATWG Encoding API: TextEncoder and TextDecoder
• [x] Config option - DUK_USE_ENCODING_BUILTINS
• [x] Support Encoding API with bufferobjects disabled? Bindings could then work with plain buffers instead of Uint8Arrays (out of spec).
• [x] Node.js Buffer .toString() can share code with TextDecoder? => Save for later
• [x] Test cases for both encode and decode
• [x] Website note (Encoding API listed in features)
• [x] RELEASES entry

try { throw new Error("A pig ate everyone at 8:12"); }
finally { print("*munch*"); }

The finally runs, but leads to "internal error in bytecode executor longjmp handler". What's odd is that I can't always reproduce it. Depending on the surrounding lines, it sometimes works correctly.

Change duk_hobject_run_finalizer() so that:

• If DUK_HEAPHDR_FLAG_FINALIZED is already set, do nothing.
• Set DUK_HEAPHDR_FLAG_FINALIZED just before running finalizer.

This should guarantee the semantics described in #109 (= at most one finalizer call per rescue cycle) because:

• The only place where a finalizer is called is duk_hobject_run_finalizer().
• The FINALIZED flag is only cleared by an explicit rescue, detected by either mark-and-sweep or refcounting.

Other changes:

• Heap destruction finalizer behavior changes: allow finalizers to create new finalizable objects whose finalizers are also executed, sanity limit.
• Explicit Proxy check, don't call finalizer on a Proxy object.

Tasks:

• [x] Review changes
• [x] Try to come up with a repro case for finalizer re-run, most obvious cases are already fixed
• [x] Test case for runaway finalizer which spawns 1 or 2 new objects per finalization
• [x] Test case for new finalizer argument
• [x] Ensure "mark-and-sweep running" is checked even when mark-and-sweep not enabled
• [x] Website changes for new guarantee
• [x] Internal doc changes for new guarantee
• [x] Add second parameter to finalizer indicating if rescue is possible or not (true: heap being destroyed, can't rescue; false: normal rescue possible)
• [x] Finalizer documentation changes for new 2nd argument
• [x] Run heap destruction finalizers in a loop to allow any new finalizable objects created by previous finalizers to also be finalized
• [x] Add a sanity limit to the finalization loop
• [x] Releases
• [x] Assert run with mark-and-sweep + refcounting
• [x] Assert run with mark-and-sweep only
• [x] Assert run with refcounting only

Fixes #109. See #108.

Hi! We met an ASAN UAF report while testing the APIs, the POC is:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "duktape.h"

duk_context* r;
int main(void)
{
  r = duk_create_heap_default();
  duk_push_buffer(r, 0x400000, 0x0);
  void* p1 = duk_to_pointer(r, 0);
  printf("ptr:%p\n", p1);
  printf("value:%x\n", *(int*)p1);
  return 0;
}

And the ASAN report is:

ptr:0x7ffff43fe800
=================================================================
==524610==ERROR: AddressSanitizer: heap-use-after-free on address 0x7ffff43fe800 at pc 0x0000004c4419 bp 0x7fffffffe620 sp 0x7fffffffe618
READ of size 4 at 0x7ffff43fe800 thread T0
    #0 0x4c4418 in main (~/a.out+0x4c4418)
    #1 0x7ffff7c45082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
    #2 0x41c44d in _start (~/a.out+0x41c44d)

0x7ffff43fe800 is located 0 bytes inside of 4194336-byte region [0x7ffff43fe800,0x7ffff47fe820)
freed by thread T0 here:
    #0 0x49490d in free (~/a.out+0x49490d)
    #1 0x4edd37 in duk_replace ~/duktape-2.7.0/src-input/duk_api_stack.c:1272:2
    #2 0x4edd37 in duk_to_pointer ~/duktape-2.7.0/src-input/duk_api_stack.c:3625:2

previously allocated by thread T0 here:
    #0 0x494b8d in malloc (~/a.out+0x494b8d)
    #1 0x55dd58 in duk__heap_mem_alloc_slowpath ~/duktape-2.7.0/src-input/duk_heap_memory.c:58:9

SUMMARY: AddressSanitizer: heap-use-after-free (~/a.out+0x4c4418) in main

Could you help us to verify this issue? Thank you!

used for the plugins system

https://openrct2.io/

https://github.com/OpenRCT2/OpenRCT2

many plugins here https://openrct2plugins.org/

I'm referring to this list https://wiki.duktape.org/projectsusingduktape

I noticed on the website under the "Similar Engines" section that MuJS is listed as being licensed under Affero GPL.

But on the MuJS website the license is listed as ISC. The COPYING file in the repo confirms this.

Is there any place where I can read info about the planned changes for Duktape 3.0? Also, any idea of the kind of timeline for the new version to hit beta?

Hello there, when I try to use a simple for loop, I get "SyntaxError: parse error (line 2)" I can use while loops, but any for loops seems to cause parse errors.

Here is the really simple code that doesn't work: (Uses a custom print function bound via C)

const arr = [3, 5, 7];
for (let i of arr) {
   print(i); // logs 3, 5, 7
}

Here is the really simple code that does work:

const arr = [3, 5, 7];
const counter = arr.length;
const index = 0;
while (index < counter)
{
	print(arr[index])
	index++;
}