A haskell toolbox for the Internet Computer

Related tags

ic-hs
Overview

The Internet Computer Haskell Toolkit

This repository contains a bunch of Internet-Computer related code written to support the following use cases:

ic-ref: a IC reference implementation

The ic-ref binary is a partial implementation of the external interface of the Internet Computer, as specified in the Interface Spec.

The goals of the reference implementation are

  • It evolves in lock-step with the Interface Spec. Versioned releases of the Interface Spec should come with a complete implementation of ic-ref.

  • Supplement the prose and pseudo-code in the Interface Spec for additional and concrete clarity.

  • Ideally, relevant code pieces of ic-ref are as easy to understand as carefully written pseudo-code.

  • Increase weight of and confidence in the Interface Spec, by demonstrating implementability.

  • Aid in the development of the Interface Spec by uncovering omissions, inconsistencies or unexpected complexity.

  • Allow testing of external clients (like dfx) directly against the reference implementation.

  • Aid in the production implementation of the Internet Computer, by allowing to probe the reference implementation to better understand intended behaviour and by comparing the behaviour of the two.

  • Performance is good enough to run small examples.

  • The primary focus is describing the happy path execution, and not necessarily the precise error reporting behaviour upon bad usage (e.g. bad canisters, resource exhaustion, module validation).

Additionally, we want ic-ref to be a useful tool for Canister developers to run their canisters locally. This adds additional goals:

  • The state of ic-ref can be persisted and picked up later.

  • Debugging/logging/trace features that aid understanding the behaviour and/or help debug canisters.

Should these goals eventually conflict with the goals for a reference implementation, e.g. becauese they impose complexity that is not easy to contain in auxillary modules, a project split might be considered.

There are also explicit non-goals to keep in mind:

  • ic-ref does not need to support canisters that are large or very long-running.

  • No persistence across different versions of ic-ref.

  • It is explicitly not desirable to share code between reference and production implementation, to achieve the multi-version cross-checking effect. Not using Rust for ic-ref helps with that.

  • No guaranteed protection against bad effects from malicious interactions.

  • No duplication of commonly available functionality. In particular, the assumption is that the production implementation will use a mature Wasm embedder that implements Wasm validation correctly, so ic-ref does not itself implement validation.

Furthermore there are some stretch goals that would be nice to have, but not if requires compromising the main goals.

  • The reference implementation describes one possible execution, but not all possible behaviours of the Interface Spec. If this can be changed (e.g. using non-deterministic modeling of computation) without compromising readability and normal execution, then this would be nice.

  • It could serve as a starting point for applying formal verification to this part of the system, e.g. by converting the (non-plumbing) modules to Coq using hs-to-coq, or by implementing them in a theorem prover and extracting Haskell code from it.

To achieve these goals, the following design decisions are made:

  • ic-ref is implemented in Haskell, to optimize for development speed, type-checking safety, and readablity of carefully selected portions of the code.

  • As far as possible, a module either

    • corresponds closely to the spec, and is written with readability as a high priority, avoiding language features that obscure meaning. The rough goal is “executable pseudo-code”. The use of advanced langauge features or non-idiomatic code that help readability are encouraged.

      Examples: IC.Ref, IC.Canister.Impl, IC.HTTP.RequestId

    • is a plumbing module that handles some technical aspect, and pave the way for the simplicity in the previously mentioned modules. It is expected that reading such modules may require high level of familiarity with Haskell.

      Examples: IC.Wasm.Imports, IC.HTTP.CBOR.

    This is an ongoing refinement process, striving for a probably unattainable ideal as the goal.

Usage

The ic-ref program starts a webserver at http://0.0.0.0:8001/ that implements the Internet Computer interface, and can be used with dfx --client http://0.0.0.0:8001/.

If you point your browser to http://0.0.0.0:8001/ you get the evolution of the IC state as JSON. Recommended to use Firefox, as it provides a nice UI for browsing JSON.

If the --state-file FILE argument is given, ic-ref will persist its state in this file. Note that if that file cannot be read (e.g. because it is from an incompatible version of ic-ref), starting ic-ref will fail.

ic-ref-test: An acceptance test suite

As the dual to the reference implementation, the ic-ref-test program is a specification compliance acceptance test that can be run against an Internet Computer instance (e.g. ic-ref, the replica) and runs a large number of functional tests against it.

Usage

Pass --endpoint http://localhost:8080/ to run against a specific node.

With the -p pattern flag you can select individual tests; those whose names contain the pattern. See https://github.com/feuerbach/tasty#patterns for advanced use of this flag.

When passing --rerun, the test suite will remember which tests have failed, and only run those that failed last tests (or all again, if none have failed last run).

ic-ref-run: A sandboxed scripted IC

The ic-ref-run tool provides a simplified mock environment for testing Canisters that does not require networking. It takes scripted input to indicate which canisters to install, and which messages to execute.

This is used, for example, in the test suite of the Motoko compiler.

ic-request-id: Calculate the representation-independent hash

The ic-request-id tool takes a CBOR-request (stdin or via a file) and calculates its request id.

ic-hs: The library

The modules of the above tools can be used for other purposes. In that sense, the whole project is one big Haskell library that can be used in quick experiments, in the Haskell REPL, or as a libary to build other tools (e.g. a test framework for canisters as in the case of the Internet Identity).

To use the Haskell REPL to interact with the internet computer, follow this pattern:

~/dfinity/ic-hs $ cabal repl ic-ref-test
…
Ok, 27 modules loaded.
*Main> :m + *IC.Test.Spec
*IC.Test.Spec *Main> :set -XOverloadedStrings
*IC.Test.Spec *Main> R r <- connect "http://localhost:34677/"
Fetching endpoint status from "http://localhost:34677"...
Spec version tested:  0.14.0
Spec version claimed: 0.14.0
*IC.Test.Spec *Main> r $ install noop
"\NUL\NUL\NUL\NUL\NUL\NUL\NUL\SOH\SOH\SOH"
*IC.Test.Spec *Main> cid1 <- r $ install noop
*IC.Test.Spec *Main> prettyBlob cid1
"00000000000000020101"
*IC.Test.Spec *Main> r $ call cid1 (setGlobal "Foo" >>> replyData "Hello")
"Hello"
*IC.Test.Spec *Main> r $ query cid1 (replyData getGlobal)
"Foo"

It’s necessary to wrap all lines with the r $ … for now; this sets the endpoint parameter.

Running

NOTE: The following assumes access to a nix cache that has built the artifacts already. There is no publicly available nix cache yet, so “fastest way” may be a bit of an euphemism.

This is the fastest way to run ic-ref or ic-ref-test is to use the following commands in this directory:

nix run -f . -c ic-ref
nix run -f . -c ic-ref-test

You can also pass arguments, e.g.

nix run -f . -c ic-ref-test --endpoint http://0.0.0.0:8080 -p 'WebAuthn'

Developing on ic-ref

Running nix-shell gives you an environment that allows you to build the project using cabal build. You can also run cabal run ic-ref etc. to run it directly from source.

One possible workflow is to run

ghcid -c 'cabal repl ic-ref' -T Main.main

which will run ic-ref and restart upon file changes. Similarly

For ic-ref-test, before running it, you make sure you have built the universal canister.

The symbolic link in test-data/universal_canister.wasm points to the build output produced by

cd universal_canister
nix-shell --command 'cargo build --target wasm32-unknown-unknown --release'

You can now run the test suite from the top-level directory with

cabal run ic-ref-test

The -p flag, i.e.

cabal run ic-ref-test -- -p upgrade

allows you can run tests selectively (i.e. only those whose name include “upgrade”).

Again, you can use ghcid to run the test suite upon file changes:

ghcid -c 'cabal repl ic-ref-test' -T Main.main

and you can flags with

ghcid -c 'cabal repl ic-ref-test' -T Main.main --setup ':set args --rerun -p "query call"'

Versioning

This repository tags versions based on the version of the Interface Spec they implement, e.g. 0.18.0. Should older major released require additional commits (bugfixes, or additional minor releases) that cannot be created on master, a release-0.18 branch would be created.

Updating Haskell Packages

When the .cabal file of a Haskell package is changed you need to make sure the corresponding nix files nix/generated/ are kept in sync with it. These are automatically generate, run

nix-shell nix/generate.nix

to update.

Don't worry if you forget to update the default.nix file, the CI job check-generated checks if these files are in sync and fails with a diff if they aren't.

Issues
Owner
DFINITY
The Internet Computer aims to reinvent the internet as a computer to host secure software and a new breed of open internet services.
DFINITY
Ideas, thoughts, and notes on a typeclass/interface based polymorphism pattern for standard C

Polymorphism through Typeclasses / Interface / Traits Ideas, thoughts, and notes on an action based polymorphism pattern for good ol' C. Originally us

Chase 13 Jun 26, 2021
This repository is a summary of the basic knowledge of recruiting job seekers and beginners in the direction of C/C++ technology, including language, program library, data structure, algorithm, system, network, link loading library, interview experience, recruitment, recommendation, etc.

?? C/C++ 技术面试基础知识总结,包括语言、程序库、数据结构、算法、系统、网络、链接装载库等知识及面试经验、招聘、内推等信息。This repository is a summary of the basic knowledge of recruiting job seekers and beginners in the direction of C/C++ technology, including language, program library, data structure, algorithm, system, network, link loading library, interview experience, recruitment, recommendation, etc.

huihut 19.6k Jul 26, 2021
Presentation Materials from CppCon 2015

CppCon 2015 Presentation Materials https://github.com/CppCon/CppCon2015 is the canonical location for presentations and code from CppCon 2015. For Spe

CppCon 1.7k Jul 22, 2021
Renew and Refill Bob Cassette for a fraction of the cost!

Bob Cassette Rewinder: Hacking Detergent DRM for 98.7% Cost Saving Get Bob Rewinder | Official Discord | User Manual How I defeated Bob Dishwasher's d

null 852 Jul 26, 2021
IoT for Beginners - 12 Weeks, 24 Lessons, IoT for All!

Azure Cloud Advocates at Microsoft are pleased to offer a 12-week, 24-lesson curriculum all about IoT basics. Each lesson includes pre- and post-lesson quizzes, written instructions to complete the lesson, a solution, an assignment and more. Our project-based pedagogy allows you to learn while building, a proven way for new skills to 'stick'.

Microsoft 6.1k Jul 28, 2021
A curated list of project-based tutorials in C

A list of tutorials that work towards the making of small to large projects in C.

R 7.1k Jul 26, 2021
The presentations of C++Now 2019.

C++Now 2019 Presentations Slide Deck Guidelines C++Now 2019 IMPORTANT: Talks that do not have slides or video will have a red question mark ( ❓ ) befo

null 77 Jun 14, 2021
🌼 Homework of Computer Systems: A Programmer's Perspective (3rd Edition) and Autolab solutions of CMU 15-513: Intro to Computer Systems

Exercisebook of Computer Systems: A Programmer's Perspective, 3/E (CS:APP3e) CS:APP3e is written by Randal E. Bryant and David R. O'Hallaron, Carnegie

halfrost 21 Jul 21, 2021
A comprehensive catalog of modern and classic books on C++ programming language

A comprehensive catalog of modern and classic books on C++ programming language

Yurii Cherkasov 159 Jul 21, 2021
Slides and other materials from CppCon 2018

CppCon 2018 Presentation Materials https://github.com/CppCon/CppCon2018 is the canonical location for presentations and code from CppCon 2018. For Spe

CppCon 1.3k Jul 27, 2021
Presentation Material for C++Now 2017

C++Now 2017 Presentations Slide Deck Guidelines for Speakers Tuesday, May 16 Paepcke Morning (9:00-10:30) Rust: Hack Without Fear!, Niko Matsakis (vid

null 424 Jul 22, 2021
C++Now 2013 PresentationsC++Now 2013 Presentations - Presentation materials presented at C++Now 2013.

C++Now 2013 Presentations Note: some of the presentations exceed GitHub's blob size limit of about 10MB. These presentations have been split into mult

null 191 Mar 12, 2021
Introduction to Computer Systems (II), Spring 2021.

Introduction to Computer Systems (II) Spring 2021, Fudan University.

null 12 Jul 16, 2021
Simulation code for the specific PDP-10 serial number 32 at the Stanford A. I. Lab in 1974 as a solo processor with all the I/O devices simulated as on the PDP-10. Omit the co-processor PDP-6 sn16.

KA10 sn32 Synopsis This repository contains software and documentation for running the unique PDP-10 KA serial number 32 that was at Stanford in July

Saildart Archive 5 May 14, 2021