This PR Implements a corresponding part of the IC spec: https://github.com/dfinity/interface-spec/pull/6

It also splits up IC.Test.Agent and IC.Test.Spec to reduce the excessive memory consumption of compiling those modules.

This allows the test driver to withhold the response to a message, and control when they are released, in order to produce situations with outstanding call contexts.

In an ideal world (from our pov), we could instrument and control the system's scheduler this way, but we can't. So instead, we use some tricks. Ideally, the details of this trick are irrelevant to the users of this function (yay, abstraction), and if we find better tricks, we can swap them out easily. We'll see if that holds water.

One problem with this approach is that a test failure could mean that the system doesn't pass the test, but it could also mean that the system has a bug that prevents this trick from working, so take care.

The current trick is: Create a canister (the "stopper"). Make it its own controller. Tell the canister to stop itself. This call will now hang, because a canister cannot stop itself. We can release the call (producing a reject) by starting the canister again.

Some tests are added, and others are now using this mechanism.

This has uncovered bugs in ic-ref related to the handling of stopped.

Following up on our discussion regarding the multivalue target feature support, we have decided to update the experimental System Api and not rely on this feature anymore.

This MR updates the tests according to the Public Spec.

We would like to implement the new 64-bit stable memory APIs in the reference implementation so that we can also make a new release of the Interface spec and have end users start using the extended stable memory.

Looking into the code a bit I figured out that the main blocker right now is the Memory module exposed by Winter which supports 32-bit memories and thus cannot be used to support the 64 bit stable memory APIs.

My idea on how this can be introduced in the least disruptive way is to provide a 64-bit Memory in Winter alongside the 32-bit one. Then the WinterMemory wrapper module can use this new 64-bit memory for either the old APIs (checking of course that the size is not bigger than allowed and casting 32 bit integers to 64 bit where necessary) or the new APIs. I think this would also be in line with future extensions of Winter (to support 64-bit wasm memories I assume you'd need a 64 bit memory implementation).

If the above sounds like a reasonable plan on a high level, then the next question would be how to best achieve this in practice. Looking at the code, it seems that we could maybe make MemoryInst generic by the size of memory and save us from some ugly code duplication. This seems the most promising approach and one that should relatively minimize the amount of changes in general.

I'm curious to see what others think and open to other ideas.

The DFINITY SDK team has notified us that they rely on ic-hs in their tests. In order for them to make a release for x86_64-darwin they need a build of ic-us for that platform.

We recently removed support for x86_64-darwin (https://github.com/dfinity/ic-hs/pull/81) because we switched to building remotely on nixbuild.net which doesn't support that platform (yet). The reason we switched was the high-memory usage in building https://github.com/dfinity/ic-hs/pull/79 which required more memory (~ 18.2 GB) than what's available on GitHub runners (7 GB).

So in order to support x86_64-darwin again we need to:

• Enable x86_64-darwin builds again. For that platform we just build locally on GitHub runners.

• Fix the high-memory usage. Some things we can do/try:

  • Split up IC.Test.Agent and IC.Test.Spec since compiling those modules requires massive RAM.
  • Switch to a more recent GHC with the hope that it fixes the memory issue. GHC-9.2 in particular as made improvements there. See: https://well-typed.com/blog/2021/03/memory-return/ To switch we probably need to upgrade to nixpkgs-22.05 (which we're doing in https://github.com/dfinity/ic-hs/pull/76).
  • Ideally we gain an understanding of why GHC suddenly requires so much memory to build IC.Test.Agent. Space profiling GHC could provide an answer here.

Using the latest nixpkgs-unstable and GHC 8.10.4, I get the following error during compilation:

[49 of 57] Compiling IC.Serialise     ( src/IC/Serialise.hs, dist/build/IC/Serialise.o, dist/build/IC/Serialise.dyn_o )

src/IC/Serialise.hs:25:1: error:
    Could not find module ‘System.Random.Internal’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
25 | import System.Random.Internal (StdGen(..))
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Pinging @nomeata

GitHub Actions Runners only have 7GB of memory which is proving to be too small to built ic-hs. So instead of building nix derivations on a runner locally we configure the runner to build derivations remotely on nixbuild.net.

The above requires access to a private SSH key, which is used to authenticate with nixbuild.net. This key is not accessible on PRs originating from forks because that would allow leaking the key. For those PRs we don't configure nixbuild.net as a remote builder and build locally instead.

in #30 we tracked a PR against nixpkgs, which isn't very stable (e.g. if it gets force-pushed to, it may break this repository). The changes we are trying to include are still not in nixpkgs master, but they are closer already, on the haskell-upgrades branch. So track that, until it is merged into master (via https://github.com/NixOS/nixpkgs/pull/138596)

This change implements 64 bit stable memory API.

ic0.stable64_size : () -> (page_count : i64);                               // *
ic0.stable64_grow : (new_pages : i64) -> (old_page_count : i64);            // *
ic0.stable64_write : (offset : i64, src : i64, size : i64) -> ();           // *
ic0.stable64_read : (dst : i64, offset : i64, size : i64) -> ();            // *

This change introduces a new representation of stable memory: we use size + history of writes instead of a flat byte buffer. This results in a pretty straightforward code and allows us to handle even huge heaps gracefully, which is crucial for testing interaction between 32 and 64 bit APIs.

Fixes #28.

Mergify is a great tool that leaves merging to this bot, meaning less maual work for us.

One great feature is that it supports squash merges that take the PR description as the commit message. So instead of a commit message like https://github.com/dfinity/ic-hs/commit/fb558eb58aed501a6b46768baf0a50ece6bedeaa we’d get something nicely curated like the PR description of https://github.com/dfinity/ic-hs/pull/30. You can achieve the same manually (as explained here), and that’s how I have been merging PRs in this repo so far, but it is tedious work, so won’t always work.

There is more that we can do with mergify, e.g. auto-merge dependency bumps.

We already have mergify enabled for the sister repository dfinity/motoko, so I see no problem doing it here too. Someone just has to enable it in https://github.com/organizations/dfinity/settings/installations/12369523, and I will prepare a PR with the configuration.

@basvandijk, are you still doing this kind of infra work? Or who should I nudge?

This gives us:

• GHC-9.0.2

• Building now uses less than half the memory as before: from > 12 GB on GHC-8.10.7 to < 6 GB on GHC-9.0.2.

• All dependencies upgraded to their latest versions.

• A working haskell-language-server which allows IDEs like VS Code to analyse ic-hs. For VS Code use the Nix Environment Selector plugin to load shell.nix and use the Haskell extension to get:

We're noticing that on https://github.com/dfinity/ic-hs/pull/79 GHC requires massive amounts of memory to build certain modules (IC.Test.Agent being the worst offender requiring more than 13GB while GitHub Runners only have 7GB causing OOM kills).

What's worse is that multiple components in the cabal file require the same modules causing the same module to be build multiple times, often in parallel exacerbating the memory problem.

This fixes the duplicate compilation issue by letting the components in the cabal file only depend on the ic-hs library.

This might be deterimental to ghcid as mentioned in the cabal file. But the associated ticket has been closed so the problem is hopefully fixed in an upcoming GHC release.

drun triggers the heartbeat, but ic-ref-run appears not to, though ic-ref supports it. This means we can only test heartbeat functionality on drun for Motoko.

When building ic-ref-dist, I get the following error message:

~/ic-hs$ nix-build -A ic-ref-dist --max-jobs 1
building '/nix/store/2m7aj49cw81kwajibhva97fb1ix01f66-git-ls-files.drv'...
these derivations will be built:
  /nix/store/78d90xbxi22n83rwfcd3dnzr96gdgy54-ic-hs-0.0.1.drv
  /nix/store/q14jr2a65xq180cr2f2llxsasm77s4pc-ic-ref-dist.drv
building '/nix/store/78d90xbxi22n83rwfcd3dnzr96gdgy54-ic-hs-0.0.1.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/3x5z3qcpdpilx8wwqmpzi7xlzb4m89gy-ghc-8.10.7.
unpacking sources
unpacking source archive /nix/store/2slzdkdcjhcfgb5xwrxxsg7iqnmpm4vn-src
source root is src
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/private/tmp/nix-build-ic-hs-0.0.1.drv-0/setup-package.conf.d -j8 -threaded -rtsopts
[1 of 1] Compiling Main             ( /nix/store/4mdp8nhyfddh7bllbi7xszz7k9955n79-Setup.hs, /private/tmp/nix-build-ic-hs-0.0.1.drv-0/Main.o )
...
libSystem.B.dylib does not exist. Try again
Please specify now the directory where this library can be found (or write 'quit' to abort):
libSystem.B.dylib does not exist. Try again
Please specify now the directory where this library can be found (or write 'quit' to abort):
libSystem.B.dylib does not exist. Try again
Please specify now the directory where this library can be found (or write 'quit' to abort):
libSystem.B.dylib does not exist. Try again
Please specify now the directory where this library can be found (or write 'quit' to abort):
...

I'm on macOS Big Sur 11.6, intel chip.

The README.md states:

The ic-ref program starts a webserver at http://0.0.0.0:8001/ that implements the Internet Computer interface, and can be used with dfx --client http://0.0.0.0:8001/.

But dfx doesn't understand that command. I think at this point the correct way to do this is to use icx-proxy, right? We should update the docs to reflect that.