A haskell toolbox for the Internet Computer

Overview

The Internet Computer Haskell Toolkit

This repository contains a bunch of Internet-Computer related code written to support the following use cases:

ic-ref: a IC reference implementation

The ic-ref binary is a partial implementation of the external interface of the Internet Computer, as specified in the Interface Spec.

The goals of the reference implementation are

  • It evolves in lock-step with the Interface Spec. Versioned releases of the Interface Spec should come with a complete implementation of ic-ref.

  • Supplement the prose and pseudo-code in the Interface Spec for additional and concrete clarity.

  • Ideally, relevant code pieces of ic-ref are as easy to understand as carefully written pseudo-code.

  • Increase weight of and confidence in the Interface Spec, by demonstrating implementability.

  • Aid in the development of the Interface Spec by uncovering omissions, inconsistencies or unexpected complexity.

  • Allow testing of external clients (like dfx) directly against the reference implementation.

  • Aid in the production implementation of the Internet Computer, by allowing to probe the reference implementation to better understand intended behaviour and by comparing the behaviour of the two.

  • Performance is good enough to run small examples.

  • The primary focus is describing the happy path execution, and not necessarily the precise error reporting behaviour upon bad usage (e.g. bad canisters, resource exhaustion, module validation).

Additionally, we want ic-ref to be a useful tool for Canister developers to run their canisters locally. This adds additional goals:

  • The state of ic-ref can be persisted and picked up later.

  • Debugging/logging/trace features that aid understanding the behaviour and/or help debug canisters.

Should these goals eventually conflict with the goals for a reference implementation, e.g. becauese they impose complexity that is not easy to contain in auxillary modules, a project split might be considered.

There are also explicit non-goals to keep in mind:

  • ic-ref does not need to support canisters that are large or very long-running.

  • No persistence across different versions of ic-ref.

  • It is explicitly not desirable to share code between reference and production implementation, to achieve the multi-version cross-checking effect. Not using Rust for ic-ref helps with that.

  • No guaranteed protection against bad effects from malicious interactions.

  • No duplication of commonly available functionality. In particular, the assumption is that the production implementation will use a mature Wasm embedder that implements Wasm validation correctly, so ic-ref does not itself implement validation.

Furthermore there are some stretch goals that would be nice to have, but not if requires compromising the main goals.

  • The reference implementation describes one possible execution, but not all possible behaviours of the Interface Spec. If this can be changed (e.g. using non-deterministic modeling of computation) without compromising readability and normal execution, then this would be nice.

  • It could serve as a starting point for applying formal verification to this part of the system, e.g. by converting the (non-plumbing) modules to Coq using hs-to-coq, or by implementing them in a theorem prover and extracting Haskell code from it.

To achieve these goals, the following design decisions are made:

  • ic-ref is implemented in Haskell, to optimize for development speed, type-checking safety, and readablity of carefully selected portions of the code.

  • As far as possible, a module either

    • corresponds closely to the spec, and is written with readability as a high priority, avoiding language features that obscure meaning. The rough goal is “executable pseudo-code”. The use of advanced langauge features or non-idiomatic code that help readability are encouraged.

      Examples: IC.Ref, IC.Canister.Impl, IC.HTTP.RequestId

    • is a plumbing module that handles some technical aspect, and pave the way for the simplicity in the previously mentioned modules. It is expected that reading such modules may require high level of familiarity with Haskell.

      Examples: IC.Wasm.Imports, IC.HTTP.CBOR.

    This is an ongoing refinement process, striving for a probably unattainable ideal as the goal.

Usage

The ic-ref program starts a webserver at http://0.0.0.0:8001/ that implements the Internet Computer interface, and can be used with dfx --client http://0.0.0.0:8001/.

If you point your browser to http://0.0.0.0:8001/ you get the evolution of the IC state as JSON. Recommended to use Firefox, as it provides a nice UI for browsing JSON.

If the --state-file FILE argument is given, ic-ref will persist its state in this file. Note that if that file cannot be read (e.g. because it is from an incompatible version of ic-ref), starting ic-ref will fail.

ic-ref-test: An acceptance test suite

As the dual to the reference implementation, the ic-ref-test program is a specification compliance acceptance test that can be run against an Internet Computer instance (e.g. ic-ref, the replica) and runs a large number of functional tests against it.

Usage

Pass --endpoint http://localhost:8080/ to run against a specific node.

With the -p pattern flag you can select individual tests; those whose names contain the pattern. See https://github.com/feuerbach/tasty#patterns for advanced use of this flag.

When passing --rerun, the test suite will remember which tests have failed, and only run those that failed last tests (or all again, if none have failed last run).

ic-ref-run: A sandboxed scripted IC

The ic-ref-run tool provides a simplified mock environment for testing Canisters that does not require networking. It takes scripted input to indicate which canisters to install, and which messages to execute.

This is used, for example, in the test suite of the Motoko compiler.

ic-request-id: Calculate the representation-independent hash

The ic-request-id tool takes a CBOR-request (stdin or via a file) and calculates its request id.

ic-hs: The library

The modules of the above tools can be used for other purposes. In that sense, the whole project is one big Haskell library that can be used in quick experiments, in the Haskell REPL, or as a libary to build other tools (e.g. a test framework for canisters as in the case of the Internet Identity).

To use the Haskell REPL to interact with the internet computer, follow this pattern:

~/dfinity/ic-hs $ cabal repl ic-ref-test
…
Ok, 27 modules loaded.
*Main> :m + *IC.Test.Spec
*IC.Test.Spec *Main> :set -XOverloadedStrings
*IC.Test.Spec *Main> R r <- connect "http://localhost:34677/"
Fetching endpoint status from "http://localhost:34677"...
Spec version tested:  0.14.0
Spec version claimed: 0.14.0
*IC.Test.Spec *Main> r $ install noop
"\NUL\NUL\NUL\NUL\NUL\NUL\NUL\SOH\SOH\SOH"
*IC.Test.Spec *Main> cid1 <- r $ install noop
*IC.Test.Spec *Main> prettyBlob cid1
"00000000000000020101"
*IC.Test.Spec *Main> r $ call cid1 (setGlobal "Foo" >>> replyData "Hello")
"Hello"
*IC.Test.Spec *Main> r $ query cid1 (replyData getGlobal)
"Foo"

It’s necessary to wrap all lines with the r $ … for now; this sets the endpoint parameter.

Running

NOTE: The following assumes access to a nix cache that has built the artifacts already. There is no publicly available nix cache yet, so “fastest way” may be a bit of an euphemism.

This is the fastest way to run ic-ref or ic-ref-test is to use the following commands in this directory:

nix run -f . -c ic-ref
nix run -f . -c ic-ref-test

You can also pass arguments, e.g.

nix run -f . -c ic-ref-test --endpoint http://0.0.0.0:8080 -p 'WebAuthn'

Developing on ic-ref

Running nix-shell gives you an environment that allows you to build the project using cabal build. You can also run cabal run ic-ref etc. to run it directly from source.

One possible workflow is to run

ghcid -c 'cabal repl ic-ref' -T Main.main

which will run ic-ref and restart upon file changes. Similarly

For ic-ref-test, before running it, you make sure you have built the universal canister.

The symbolic link in test-data/universal_canister.wasm points to the build output produced by

cd universal_canister
nix-shell --command 'cargo build --target wasm32-unknown-unknown --release'

You can now run the test suite from the top-level directory with

cabal run ic-ref-test

The -p flag, i.e.

cabal run ic-ref-test -- -p upgrade

allows you can run tests selectively (i.e. only those whose name include “upgrade”).

Again, you can use ghcid to run the test suite upon file changes:

ghcid -c 'cabal repl ic-ref-test' -T Main.main

and you can flags with

ghcid -c 'cabal repl ic-ref-test' -T Main.main --setup ':set args --rerun -p "query call"'

Versioning

This repository tags versions based on the version of the Interface Spec they implement, e.g. 0.18.0. Should older major released require additional commits (bugfixes, or additional minor releases) that cannot be created on master, a release-0.18 branch would be created.

Updating Haskell Packages

When the .cabal file of a Haskell package is changed you need to make sure the corresponding nix files nix/generated/ are kept in sync with it. These are automatically generate, run

nix-shell nix/generate.nix

to update.

Don't worry if you forget to update the default.nix file, the CI job check-generated checks if these files are in sync and fails with a diff if they aren't.

Comments
  • System API for ECDSA signing

    System API for ECDSA signing

    This PR Implements a corresponding part of the IC spec: https://github.com/dfinity/interface-spec/pull/6

    It also splits up IC.Test.Agent and IC.Test.Spec to reduce the excessive memory consumption of compiling those modules.

    opened by marcin-dziadus 16
  • ic-ref-test: Test stopping state

    ic-ref-test: Test stopping state

    This allows the test driver to withhold the response to a message, and control when they are released, in order to produce situations with outstanding call contexts.

    In an ideal world (from our pov), we could instrument and control the system's scheduler this way, but we can't. So instead, we use some tricks. Ideally, the details of this trick are irrelevant to the users of this function (yay, abstraction), and if we find better tricks, we can swap them out easily. We'll see if that holds water.

    One problem with this approach is that a test failure could mean that the system doesn't pass the test, but it could also mean that the system has a bug that prevents this trick from working, so take care.

    The current trick is: Create a canister (the "stopper"). Make it its own controller. Tell the canister to stop itself. This call will now hang, because a canister cannot stop itself. We can release the call (producing a reject) by starting the canister again.

    Some tests are added, and others are now using this mechanism.

    This has uncovered bugs in ic-ref related to the handling of stopped.

    opened by nomeata 10
  • Update system API for 128 bit cycles tests

    Update system API for 128 bit cycles tests

    Following up on our discussion regarding the multivalue target feature support, we have decided to update the experimental System Api and not rely on this feature anymore.

    This MR updates the tests according to the Public Spec.

    opened by AlexandraZapuc 10
  • Remove duplicate compilation

    Remove duplicate compilation

    We're noticing that on https://github.com/dfinity/ic-hs/pull/79 GHC requires massive amounts of memory to build certain modules (IC.Test.Agent being the worst offender requiring more than 13GB while GitHub Runners only have 7GB causing OOM kills).

    What's worse is that multiple components in the cabal file require the same modules causing the same module to be build multiple times, often in parallel exacerbating the memory problem.

    This fixes the duplicate compilation issue by letting the components in the cabal file only depend on the ic-hs library.

    This might be deterimental to ghcid as mentioned in the cabal file. But the associated ticket has been closed so the problem is hopefully fixed in an upcoming GHC release.

    opened by basvandijk 9
  • run system tasks periodically

    run system tasks periodically

    Currently, system tasks (heartbeat and global timer) are only triggered upon receiving an HTTP request. This MR adds a new execution thread that periodically triggers these tasks every second.

    opened by mraszyk 8
  • Implement 64-bit stable memory APIs

    Implement 64-bit stable memory APIs

    We would like to implement the new 64-bit stable memory APIs in the reference implementation so that we can also make a new release of the Interface spec and have end users start using the extended stable memory.

    Looking into the code a bit I figured out that the main blocker right now is the Memory module exposed by Winter which supports 32-bit memories and thus cannot be used to support the 64 bit stable memory APIs.

    My idea on how this can be introduced in the least disruptive way is to provide a 64-bit Memory in Winter alongside the 32-bit one. Then the WinterMemory wrapper module can use this new 64-bit memory for either the old APIs (checking of course that the size is not bigger than allowed and casting 32 bit integers to 64 bit where necessary) or the new APIs. I think this would also be in line with future extensions of Winter (to support 64-bit wasm memories I assume you'd need a 64 bit memory implementation).

    If the above sounds like a reasonable plan on a high level, then the next question would be how to best achieve this in practice. Looking at the code, it seems that we could maybe make MemoryInst generic by the size of memory and save us from some ugly code duplication. This seems the most promising approach and one that should relatively minimize the amount of changes in general.

    I'm curious to see what others think and open to other ideas.

    enhancement 
    opened by dsarlis 8
  • Provide x86_64-darwin builds again since dfinity/sdk depends on it

    Provide x86_64-darwin builds again since dfinity/sdk depends on it

    The DFINITY SDK team has notified us that they rely on ic-hs in their tests. In order for them to make a release for x86_64-darwin they need a build of ic-us for that platform.

    We recently removed support for x86_64-darwin (https://github.com/dfinity/ic-hs/pull/81) because we switched to building remotely on nixbuild.net which doesn't support that platform (yet). The reason we switched was the high-memory usage in building https://github.com/dfinity/ic-hs/pull/79 which required more memory (~ 18.2 GB) than what's available on GitHub runners (7 GB).

    So in order to support x86_64-darwin again we need to:

    • Enable x86_64-darwin builds again. For that platform we just build locally on GitHub runners.

    • Fix the high-memory usage. Some things we can do/try:

      • Split up IC.Test.Agent and IC.Test.Spec since compiling those modules requires massive RAM.
      • Switch to a more recent GHC with the hope that it fixes the memory issue. GHC-9.2 in particular as made improvements there. See: https://well-typed.com/blog/2021/03/memory-return/ To switch we probably need to upgrade to nixpkgs-22.05 (which we're doing in https://github.com/dfinity/ic-hs/pull/76).
      • Ideally we gain an understanding of why GHC suddenly requires so much memory to build IC.Test.Agent. Space profiling GHC could provide an answer here.
    opened by basvandijk 7
  • Cannot nix-build on BigSur

    Cannot nix-build on BigSur

    When building ic-ref-dist, I get the following error message:

    ~/ic-hs$ nix-build -A ic-ref-dist --max-jobs 1
    building '/nix/store/2m7aj49cw81kwajibhva97fb1ix01f66-git-ls-files.drv'...
    these derivations will be built:
      /nix/store/78d90xbxi22n83rwfcd3dnzr96gdgy54-ic-hs-0.0.1.drv
      /nix/store/q14jr2a65xq180cr2f2llxsasm77s4pc-ic-ref-dist.drv
    building '/nix/store/78d90xbxi22n83rwfcd3dnzr96gdgy54-ic-hs-0.0.1.drv'...
    setupCompilerEnvironmentPhase
    Build with /nix/store/3x5z3qcpdpilx8wwqmpzi7xlzb4m89gy-ghc-8.10.7.
    unpacking sources
    unpacking source archive /nix/store/2slzdkdcjhcfgb5xwrxxsg7iqnmpm4vn-src
    source root is src
    patching sources
    compileBuildDriverPhase
    setupCompileFlags: -package-db=/private/tmp/nix-build-ic-hs-0.0.1.drv-0/setup-package.conf.d -j8 -threaded -rtsopts
    [1 of 1] Compiling Main             ( /nix/store/4mdp8nhyfddh7bllbi7xszz7k9955n79-Setup.hs, /private/tmp/nix-build-ic-hs-0.0.1.drv-0/Main.o )
    ...
    libSystem.B.dylib does not exist. Try again
    Please specify now the directory where this library can be found (or write 'quit' to abort):
    libSystem.B.dylib does not exist. Try again
    Please specify now the directory where this library can be found (or write 'quit' to abort):
    libSystem.B.dylib does not exist. Try again
    Please specify now the directory where this library can be found (or write 'quit' to abort):
    libSystem.B.dylib does not exist. Try again
    Please specify now the directory where this library can be found (or write 'quit' to abort):
    ...
    

    I'm on macOS Big Sur 11.6, intel chip.

    opened by nmattia 6
  • Cannot build Serialise.hs with GHC 8.10.4

    Cannot build Serialise.hs with GHC 8.10.4

    Using the latest nixpkgs-unstable and GHC 8.10.4, I get the following error during compilation:

    [49 of 57] Compiling IC.Serialise     ( src/IC/Serialise.hs, dist/build/IC/Serialise.o, dist/build/IC/Serialise.dyn_o )
    
    src/IC/Serialise.hs:25:1: error:
        Could not find module ‘System.Random.Internal’
        Use -v (or `:set -v` in ghci) to see a list of the files searched for.
       |
    25 | import System.Random.Internal (StdGen(..))
       | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    

    Pinging @nomeata

    opened by jwiegley 6
  • canister http requests test

    canister http requests test

    Update canister http_request functionality in ic-ref and ic-ref-test:

    • replace a local test server by httpbin
    • include command line options for specifying httpbin hostname and polling timeout
    • do not always reset reject replies from management canister to have reject code 5
    • model and test http_request fees
    • update http_request in ic.did (add field max_response_bytes and variants post and head to method)
    • handle https requests in IC/Ref/IO using TLS
    • update some reject and error codes in ic-ref to match replica
    opened by mraszyk 4
  • Use nixbuild.net as a remote nix builder on CI

    Use nixbuild.net as a remote nix builder on CI

    GitHub Actions Runners only have 7GB of memory which is proving to be too small to built ic-hs. So instead of building nix derivations on a runner locally we configure the runner to build derivations remotely on nixbuild.net.

    The above requires access to a private SSH key, which is used to authenticate with nixbuild.net. This key is not accessible on PRs originating from forks because that would allow leaking the key. For those PRs we don't configure nixbuild.net as a remote builder and build locally instead.

    opened by basvandijk 3
  • Documentation for running with dfx is out-of-date

    Documentation for running with dfx is out-of-date

    The README.md states:

    The ic-ref program starts a webserver at http://0.0.0.0:8001/ that implements the Internet Computer interface, and can be used with dfx --client http://0.0.0.0:8001/.

    But dfx doesn't understand that command. I think at this point the correct way to do this is to use icx-proxy, right? We should update the docs to reflect that.

    documentation 
    opened by kritzcreek 1
  • Use a single canister for many tests

    Use a single canister for many tests

    so that we save may canister installations. Oddly, does not speed things up; it seems that the tests that cannot share the canister are the bottleneck.

    opened by nomeata 0
Owner
DFINITY
The Internet Computer aims to reinvent the internet as a computer to host secure software and a new breed of open internet services.
DFINITY
Introduction to Computer Systems (II), Spring 2021.

Introduction to Computer Systems (II) Spring 2021, Fudan University.

null 29 Dec 13, 2022
哈尔滨工业大学(深圳)计算机专业课程攻略 | Guidance for courses in Department of Computer Science, Harbin Institute of Technology (Shenzhen)

哈工大(深圳)计算机专业课程攻略 受浙江大学相关项目和清华大学相关项目启发,创立了本项目。 若要下载单个文件夹,复制该文件夹的网址,粘贴入DownGit中,选择Download即可。 感谢对本项目贡献的同学:第二届院学生会学术部全体成员、hewei2001、chh13502、xyfJASON、ail

null 847 Dec 31, 2022
COMPUTER ENGINEERING - SEM 3-8 College Assignments and PPTs

Contributors ✨ Thanks goes to these wonderful people (emoji key): Subham Agrawal ?? ✅ Pritesh Kumar Tripathi ?? ✅ Sakshi Jain ?? ✅ Krithikha Bala ?? ✅

Subham Agrawal 29 Aug 3, 2022
University of Bergamo - C++ Dev Course (Computer Engineering) 2021/2022

University of Bergamo - 21012 Dev Course C++ 2021/2022 This git repository contains the source code in C++ language to support the lessons of the C++

Mauro Pelucchi 11 Dec 1, 2022
A project for Computer Programming (2) Course // C language

Fun-programming-project A project for Course : Computer Programming (2) “ CS181 ” presented to Dr. Abeer Alhujaylan Team work : Feda mousa leen mohamm

Feda Mousa 2 Feb 4, 2022
Internet Identity, a blockchain authentication system for the Internet Computer

Internet Identity Service See ./docs/internet-identity-spec.adoc for a details specification and technical documentation.

DFINITY 238 Dec 25, 2022
✔️The smallest header-only GUI library(4 KLOC) for all platforms

Welcome to GUI-lite The smallest header-only GUI library (4 KLOC) for all platforms. 中文 Lightweight ✂️ Small: 4,000+ lines of C++ code, zero dependenc

null 6.6k Jan 8, 2023
A Haskell library for fast decoding of JSON documents using the simdjson C++ library

hermes A Haskell interface over the simdjson C++ library for decoding JSON documents. Hermes, messenger of the gods, was the maternal great-grandfathe

Josh Miller 36 Dec 5, 2022
A c++ toolbox of locality-sensitive hashing (LSH), provides several popular LSH algorithms, also support python and matlab.

LSHBOX-0.9 A C++ Toolbox of Locality-Sensitive Hashing for Large Scale Image Retrieval, Also Support Python and MATLAB. Change Log 2015.07.04 A new LS

null 269 Nov 13, 2022
Toolbox that makes homebrewing the PS Vita easier

VitaDeploy Toolbox that makes homebrewing the Playstation Vita/TV easier Features file manager (VitaShell) sd2vita mount/format firmware updater/downg

null 164 Jan 2, 2023
A generic and robust calibration toolbox for multi-camera systems

MC-Calib Toolbox described in the paper "MultiCamCalib: A Generic Calibration Toolbox for Multi-Camera Systems". Installation Requirements: Ceres, Boo

null 204 Jan 5, 2023
Zero-Knowledge Proof Toolbox

Zkrypt是一个开源的C语言零知识证明算法库,旨在向用户提供简洁、高效的非交互式零知识证明协议接口,用户可以通过调用接口实现完整的零知识证明协议的流程,包括公共参数设置、证明生成和验证等步骤。 本项目由北京大学关志的密码学研究组开发维护。 特性 支持多种零知识证明协议(包括Groth16, Plo

Zhi Guan 16 Dec 14, 2022
Algorithmic Toolbox

Algorithmic Toolbox This online course covers basic algorithmic techniques and ideas for computational problems arising frequently in practical applic

Nourhan Elsherif 2 Jan 3, 2022
Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

Apache Arrow Powering In-Memory Analytics Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enabl

The Apache Software Foundation 10.8k Dec 29, 2022
BayesOpt: A toolbox for bayesian optimization, experimental design and stochastic bandits.

BayesOpt: A Bayesian optimization library BayesOpt is an efficient implementation of the Bayesian optimization methodology for nonlinear optimization,

Ruben Martinez-Cantin 341 Nov 29, 2022
Digital Signal Processing Library and Audio Toolbox for the Modern Synthesist.

Digital Signal Processing Library and Audio Toolbox for the Modern Synthesist. Attention This library is still under development! Read the docs and ch

everdrone 81 Nov 25, 2022
A long-read analysis toolbox for cancer genomics

Lorax: A long-read analysis toolbox for cancer genomics In cancer genomics, long-read de novo assembly approaches may not be applicable because of tum

Tobias Rausch 11 Dec 15, 2022
C-based/Cached/Core Computer Vision Library, A Modern Computer Vision Library

Build Status Travis CI VM: Linux x64: Raspberry Pi 3: Jetson TX2: Backstory I set to build ccv with a minimalism inspiration. That was back in 2010, o

Liu Liu 6.9k Jan 6, 2023
C-based/Cached/Core Computer Vision Library, A Modern Computer Vision Library

Build Status Travis CI VM: Linux x64: Raspberry Pi 3: Jetson TX2: Backstory I set to build ccv with a minimalism inspiration. That was back in 2010, o

Liu Liu 6.9k Jan 6, 2023
🌼 Homework of Computer Systems: A Programmer's Perspective (3rd Edition) and Autolab solutions of CMU 15-513: Intro to Computer Systems

Exercisebook of Computer Systems: A Programmer's Perspective, 3/E (CS:APP3e) CS:APP3e is written by Randal E. Bryant and David R. O'Hallaron, Carnegie

halfrost 31 Nov 20, 2022