Large scale embeddings on a single machine.

Related tags

Cryptography marius
Overview

Marius

Marius is a system under active development for training embeddings for large-scale graphs on a single machine.

Training on large scale graphs requires a large amount of data movement to get embedding parameters from storage to the computational device. Marius is designed to mitigate/reduce data movement overheads using:

  • Pipelined training and IO
  • Partition caching and locality-aware data orderings

Details on how Marius works can be found in the documentation, or in our OSDI 21' Paper.

Requirements

(Other versions may work, but are untested)

  • Ubuntu 18.04 or MacOS 10.15
  • CUDA 10.1 or 10.2 (If using GPU training)
  • CuDNN 7 (If using GPU training)
  • 1.7 >= pytorch
  • python >= 3.6
  • pip >= 21
  • GCC >= 9 (On Linux) or Clang 12.0 (On MacOS)
  • cmake >= 3.12
  • make >= 3.8

Installation from source with Pip

  1. Install latest version of PyTorch for your CUDA version:

    Linux:

    • CUDA 10.1: python3 -m pip install torch==1.7.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html
    • CUDA 10.2: python3 -m pip install torch==1.7.1
    • CPU Only: python3 -m pip install torch==1.7.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

    MacOS:

    • CPU Only: python3 -m pip install torch==1.7.1
  2. Clone the repository git clone https://github.com/marius-team/marius.git

  3. Build and install Marius cd marius; python3 -m pip install .

Full script (without torch install)

git clone https://github.com/marius-team/marius.git
cd marius
python3 -m pip install .

Installation from source with CMake

  1. Clone the repository git clone https://github.com/marius-team/marius.git

  2. Install dependencies cd marius; python3 -m pip install -r requirements.txt

  3. Create build directory mkdir build; cd build

  4. Run cmake in the build directory cmake ../ (CPU-only build) or cmake ../ -DUSE_CUDA=1 (GPU build)

  5. Make the marius executable. make marius_train -j

Full script (without torch install)

git clone https://github.com/marius-team/marius.git
cd marius
python3 -m pip install -r requirements.txt
mkdir build
cd build
cmake ../ -DUSE_CUDA=1
make -j

Training a graph

Training embeddings on a graph requires three steps.

  1. Define a configuration file. This example will use the config already defined in examples/training/configs/fb15k_gpu.ini

    See docs/configuration.rst for full details on the configuration options.

  2. Preprocess the dataset marius_preprocess fb15k output_dir/

    The first argument of marius/tools/preprocess.py defines the dataset we wish to download and preprocess, in this case fb15k. The second argument tells the preprocessor where to put the preprocessed dataset.

  3. Run the training executable with the config file. marius_train examples/training/configs/fb15k_gpu.ini

The output of the first epoch should be similar to the following.

[info] [03/18/21 01:33:18.778] Metadata initialized
[info] [03/18/21 01:33:18.778] Training set initialized
[info] [03/18/21 01:33:18.779] Evaluation set initialized
[info] [03/18/21 01:33:18.779] Preprocessing Complete: 2.605s
[info] [03/18/21 01:33:18.791] ################ Starting training epoch 1 ################
[info] [03/18/21 01:33:18.836] Total Edges Processed: 40000, Percent Complete: 0.082
[info] [03/18/21 01:33:18.862] Total Edges Processed: 80000, Percent Complete: 0.163
[info] [03/18/21 01:33:18.892] Total Edges Processed: 120000, Percent Complete: 0.245
[info] [03/18/21 01:33:18.918] Total Edges Processed: 160000, Percent Complete: 0.327
[info] [03/18/21 01:33:18.944] Total Edges Processed: 200000, Percent Complete: 0.408
[info] [03/18/21 01:33:18.970] Total Edges Processed: 240000, Percent Complete: 0.490
[info] [03/18/21 01:33:18.996] Total Edges Processed: 280000, Percent Complete: 0.571
[info] [03/18/21 01:33:19.021] Total Edges Processed: 320000, Percent Complete: 0.653
[info] [03/18/21 01:33:19.046] Total Edges Processed: 360000, Percent Complete: 0.735
[info] [03/18/21 01:33:19.071] Total Edges Processed: 400000, Percent Complete: 0.816
[info] [03/18/21 01:33:19.096] Total Edges Processed: 440000, Percent Complete: 0.898
[info] [03/18/21 01:33:19.122] Total Edges Processed: 480000, Percent Complete: 0.980
[info] [03/18/21 01:33:19.130] ################ Finished training epoch 1 ################
[info] [03/18/21 01:33:19.130] Epoch Runtime (Before shuffle/sync): 339ms
[info] [03/18/21 01:33:19.130] Edges per Second (Before shuffle/sync): 1425197.8
[info] [03/18/21 01:33:19.130] Edges Shuffled
[info] [03/18/21 01:33:19.130] Epoch Runtime (Including shuffle/sync): 339ms
[info] [03/18/21 01:33:19.130] Edges per Second (Including shuffle/sync): 1425197.8
[info] [03/18/21 01:33:19.148] Starting evaluating
[info] [03/18/21 01:33:19.254] Pipeline flush complete
[info] [03/18/21 01:33:19.271] Num Eval Edges: 50000
[info] [03/18/21 01:33:19.271] Num Eval Batches: 50
[info] [03/18/21 01:33:19.271] Auc: 0.973, Avg Ranks: 24.477, MRR: 0.491, [email protected]: 0.357, [email protected]: 0.651, [email protected]: 0.733, [email protected]: 0.806, [email protected]: 0.895, [email protected]: 0.943

To train using CPUs only, use the examples/training/configs/fb15k_cpu.ini configuration file instead.

Using the Python API

Sample Code

Below is a sample python script which trains a single epoch of embeddings on fb15k.

import marius as m
from marius.tools import preprocess

def fb15k_example():

    preprocess.fb15k(output_dir="output_dir/")
    
    config_path = "examples/training/configs/fb15k_cpu.ini"
    config = m.parseConfig(config_path)

    train_set, eval_set = m.initializeDatasets(config)

    model = m.initializeModel(config.model.encoder_model, config.model.decoder_model)

    trainer = m.SynchronousTrainer(train_set, model)
    evaluator = m.SynchronousEvaluator(eval_set, model)

    trainer.train(1)
    evaluator.evaluate(True)


if __name__ == "__main__":
    fb15k_example()

Marius in Docker

Marius can be deployed within a docker container. Here is a sample ubuntu dockerfile (located at examples/docker/dockerfile) which contains the necessary dependencies preinstalled for GPU training.

Building and running the container

Build an image with the name marius and the tag example:
docker build -t marius:example -f examples/docker/dockerfile examples/docker

Create and start a new container instance named gaius with:
docker run --name gaius -itd marius:example

Run docker ps to verify the container is running

Start a bash session inside the container:
docker exec -it gaius bash

Sample Dockerfile

See examples/docker/dockerfile

FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
RUN apt update

RUN apt install -y g++ \ 
         make \
         wget \
         unzip \
         vim \
         git \
         python3-pip

# install gcc-9
RUN apt install -y software-properties-common
RUN add-apt-repository -y ppa:ubuntu-toolchain-r/test
RUN apt update
RUN apt install -y gcc-9 g++-9
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 9
RUN update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-9 9

# install cmake 3.20
RUN wget https://github.com/Kitware/CMake/releases/download/v3.20.0/cmake-3.20.0-linux-x86_64.sh
RUN mkdir /opt/cmake
RUN sh cmake-3.20.0-linux-x86_64.sh --skip-license --prefix=/opt/cmake/
RUN ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake

# install pytorch
RUN python3 -m pip install torch==1.7.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html

Citing Marius

@inproceedings {MariusOSDI2021,
    author = {Jason Mohoney and Roger Waleffe and Yiheng Xu and Theodoros Rekatsinas and Shivaram Venkataraman},
    title = {Marius: Learning Massive Graph Embeddings on a Single Machine},
    booktitle = {15th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 21)},
    year = {2021},
    publisher = {{USENIX} Association},
}
Comments
  • ERROR during the pip installation process

    ERROR during the pip installation process

    I was trying to install pip and

    ERROR: Could not find a version that satisfies the requirement torch (from marius==0.0.2) (from versions: none)
    ERROR: No matching distribution found for torch (from marius==0.0.2)
    

    keeps popping up.

    Tried to pip install torchvision==0.1.8 in command line and it showed Successfully installed torch-1.11.0 torchvision-0.1.8. Then, when I tried to pip3 install . again, the same error appears. I am wondering how to solve this to proceed. Thank you.

    question 
    opened by lwwlwwl 17
  • 'test_edges.bin' and 'validation_edges.bin' are not created when preprocess ogbn_products.

    'test_edges.bin' and 'validation_edges.bin' are not created when preprocess ogbn_products.

    Hi, I want to run marius with ogbn-products dataset.

    I executed the following command: marius_preprocess --dataset ogbn_products --output_dir datasets/ogbn_products

    There was no problem running it, but only 'train_edges.bin' was created in ogbn_products/edges. There is no 'test_edges.bin' and 'validation_edges.bin'. How could I get them??

    Thanks a lot.

    question 
    opened by qhtjrmin 13
  • Training Wikidata embedding

    Training Wikidata embedding

    I'm trying to create embeddings for Wikidata, using this conf file [general] device=CPU num_train=611058458 num_nodes=91580024 num_valid=612283 num_test=612283 experiment_name=wikidata num_relations=1390 ...

    However, I am getting the error:

    ValueError: cannot create std::vector larger than max_size()

    Looking for any workaround, thanks

    question 
    opened by dlauc 13
  • marius_preprocess triggers program aborted

    marius_preprocess triggers program aborted

    Describe the bug run marius_preprocess or import preprocess would trigger the following error.

    free(): invalid pointer
    Aborted
    

    To Reproduce Steps to reproduce the behavior:

    1. Run the given example 'marius_preprocess output_dir/ --dataset fb15k' OR
    2. 'from marius.tools import preprocess' in Python

    Environment gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04) Python 3.8.5

    bug 
    opened by VeritasYin 7
  • CUDA error: device-side assert triggered when trying to execute example scripts

    CUDA error: device-side assert triggered when trying to execute example scripts

    Describe the bug I successfully installed the program and it passed test/cpp/end_to_end, then when I tried to execute examples/training/scripts/fb15k_gpu.sh (and also some other configs with GPU enabled), it triggered a nll_loss_backward_reduce_cuda_kernel_2d assertion failure.

    To Reproduce Steps to reproduce the behavior:

    1. I execute bash examples/training/scripts/fb15k_gpu.sh
    2. marius_preprocess step is able to be executed without any problems
    3. When marius_train proceeds to backward for the first batch of the first epoch, the following error occurs:
    [email protected]:~/marius$ bash examples/training/scripts/fb15k_gpu.sh 
    fb15k
    Downloading fb15k.tgz to output_dir/fb15k.tgz
    Extracting
    Extraction completed
    Detected delimiter: ~   ~
    Reading in output_dir/freebase_mtr100_mte100-train.txt   1/3
    Reading in output_dir/freebase_mtr100_mte100-valid.txt   2/3
    Reading in output_dir/freebase_mtr100_mte100-test.txt   3/3
    Number of instance per file:[483142, 50000, 59071]
    Number of nodes: 14951
    Number of edges: 592213
    Number of relations: 1345
    Delimiter: ~    ~
    ['/home/nfp/.local/bin/marius_train', 'examples/training/configs/fb15k_gpu.ini']
    [info] [10/28/21 22:12:59.865] Start preprocessing
    [debug] [10/28/21 22:12:59.866] Initializing Model
    [debug] [10/28/21 22:12:59.866] Empty Encoder
    [debug] [10/28/21 22:12:59.866] DistMult Decoder
    [debug] [10/28/21 22:12:59.867] data/ directory already exists
    [debug] [10/28/21 22:12:59.867] data/marius/ directory already exists
    [debug] [10/28/21 22:12:59.867] data/marius/embeddings/ directory already exists
    [debug] [10/28/21 22:12:59.867] data/marius/relations/ directory already exists
    [debug] [10/28/21 22:12:59.867] data/marius/edges/ directory already exists
    [debug] [10/28/21 22:12:59.867] data/marius/edges/train/ directory already exists
    [debug] [10/28/21 22:12:59.867] data/marius/edges/evaluation/ directory already exists
    [debug] [10/28/21 22:12:59.867] data/marius/edges/test/ directory already exists
    [debug] [10/28/21 22:12:59.880] Edges: DeviceMemory storage initialized
    [debug] [10/28/21 22:12:59.894] Edges shuffled
    [debug] [10/28/21 22:12:59.894] Edge storage initialized. Train: 483142, Valid: 50000, Test: 59071
    [debug] [10/28/21 22:13:00.004] Node embeddings: DeviceMemory storage initialized
    [debug] [10/28/21 22:13:00.004] Node embeddings state: DeviceMemory storage initialized
    [debug] [10/28/21 22:13:00.004] Node embeddings initialized: 14951
    [debug] [10/28/21 22:13:00.014] Relation embeddings: DeviceMemory storage initialized
    [debug] [10/28/21 22:13:00.014] Relation embeddings state: DeviceMemory storage initialized
    [debug] [10/28/21 22:13:00.014] Relation embeddings initialized: 1345
    [debug] [10/28/21 22:13:00.014] Getting batches from edge list
    [info] [10/28/21 22:13:00.014] Training set initialized
    [debug] [10/28/21 22:13:00.014] Getting batches from edge list
    [debug] [10/28/21 22:13:00.014] Batches initialized
    [info] [10/28/21 22:13:00.015] Evaluation set initialized
    [info] [10/28/21 22:13:00.015] Preprocessing Complete: 0.149s
    [debug] [10/28/21 22:13:00.032] Loaded training set
    [info] [10/28/21 22:13:00.032] ################ Starting training epoch 1 ################
    [trace] [10/28/21 22:13:00.032] Starting Batch. ID 0, Starting Index 0, Batch Size 10000 
    [trace] [10/28/21 22:13:00.034] Batch: 0 Accumulated 11109 unique embeddings
    [trace] [10/28/21 22:13:00.034] Batch: 0 Accumulated 640 unique relations
    [trace] [10/28/21 22:13:00.034] Batch: 0 Indices sent to device
    [trace] [10/28/21 22:13:00.034] Batch: 0 Node Embeddings read
    [trace] [10/28/21 22:13:00.034] Batch: 0 Node State read
    [trace] [10/28/21 22:13:00.034] Batch: 0 Relation Embeddings read
    [trace] [10/28/21 22:13:00.034] Batch: 0 Relation State read
    [trace] [10/28/21 22:13:00.035] Batch: 0 prepared for compute
    [debug] [10/28/21 22:13:00.040] Loss: 124804.266, Regularization loss: 0.012812799
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [3,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [4,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [5,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [8,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [9,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [10,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [11,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [12,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [14,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [15,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [16,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [17,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [18,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [19,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [20,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [21,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [22,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [23,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [24,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [25,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [26,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [27,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [28,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [29,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [30,0,0] Assertion `t >= 0 && t < n_classes` failed.
    /pytorch/aten/src/ATen/native/cuda/Loss.cu:455: nll_loss_backward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [31,0,0] Assertion `t >= 0 && t < n_classes` failed.
    Traceback (most recent call last):
      File "/home/nfp/.local/bin/marius_train", line 8, in <module>
        sys.exit(main())
      File "/home/nfp/.local/lib/python3.6/site-packages/marius/console_scripts/marius_train.py", line 8, in main
        m.marius_train(len(sys.argv), sys.argv)
    RuntimeError: CUDA error: device-side assert triggered
    CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
    For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
    Exception raised from launch_unrolled_kernel at /pytorch/aten/src/ATen/native/cuda/CUDALoops.cuh:132 (most recent call first):
    frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f95645bcd62 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libc10.so)
    frame #1: void at::native::gpu_kernel_impl<at::native::BinaryFunctor<float, float, float, at::native::AddFunctor<float> > >(at::TensorIteratorBase&, at::native::BinaryFunctor<float, float, float, at::native::AddFunctor<float> > const&) + 0xb37 (0x7f95665b2f27 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda_cu.so)
    frame #2: void at::native::gpu_kernel<at::native::BinaryFunctor<float, float, float, at::native::AddFunctor<float> > >(at::TensorIteratorBase&, at::native::BinaryFunctor<float, float, float, at::native::AddFunctor<float> > const&) + 0x113 (0x7f95665bf333 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda_cu.so)
    frame #3: void at::native::opmath_gpu_kernel_with_scalars<float, float, float, at::native::AddFunctor<float> >(at::TensorIteratorBase&, at::native::AddFunctor<float> const&) + 0xa9 (0x7f95665bf4c9 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda_cu.so)
    frame #4: <unknown function> + 0xe5d953 (0x7f9566592953 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda_cu.so)
    frame #5: at::native::add_kernel_cuda(at::TensorIteratorBase&, c10::Scalar const&) + 0x15 (0x7f95665930a5 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda_cu.so)
    frame #6: <unknown function> + 0xe5e0cf (0x7f95665930cf in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda_cu.so)
    frame #7: at::native::structured_sub_out::impl(at::Tensor const&, at::Tensor const&, c10::Scalar const&, at::Tensor const&) + 0x40 (0x7f95a9f1ef00 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #8: <unknown function> + 0x25e52ab (0x7f9567d1a2ab in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda_cu.so)
    frame #9: <unknown function> + 0x25e5372 (0x7f9567d1a372 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda_cu.so)
    frame #10: at::_ops::sub_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0xb9 (0x7f95aa55d3f9 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #11: <unknown function> + 0x34be046 (0x7f95ac03c046 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #12: <unknown function> + 0x34be655 (0x7f95ac03c655 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #13: at::_ops::sub_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x13f (0x7f95aa5b5b2f in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #14: <unknown function> + 0x3f299b0 (0x7f95acaa79b0 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #15: torch::autograd::generated::LogsumexpBackward0::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x1dc (0x7f95abd1447c in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #16: <unknown function> + 0x3896817 (0x7f95ac414817 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #17: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x145b (0x7f95ac40fa7b in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #18: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x57a (0x7f95ac4107aa in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #19: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x89 (0x7f95ac4081c9 in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #20: <unknown function> + 0xc71f (0x7f962b3ad71f in /home/nfp/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #21: <unknown function> + 0x76db (0x7f962d01f6db in /lib/x86_64-linux-gnu/libpthread.so.0)
    frame #22: clone + 0x3f (0x7f962d35871f in /lib/x86_64-linux-gnu/libc.so.6)
    

    Expected behavior The program works well for CPU configs:

    [email protected]:~/marius$ bash examples/training/scripts/fb15k_cpu.sh 
    fb15k
    Downloading fb15k.tgz to output_dir/fb15k.tgz
    Extracting
    Extraction completed
    Detected delimiter: ~   ~
    Reading in output_dir/freebase_mtr100_mte100-train.txt   1/3
    Reading in output_dir/freebase_mtr100_mte100-valid.txt   2/3
    Reading in output_dir/freebase_mtr100_mte100-test.txt   3/3
    Number of instance per file:[483142, 50000, 59071]
    Number of nodes: 14951
    Number of edges: 592213
    Number of relations: 1345
    Delimiter: ~    ~
    ['/home/nfp/.local/bin/marius_train', 'examples/training/configs/fb15k_cpu.ini']
    [info] [10/28/21 22:19:07.259] Start preprocessing
    [info] [10/28/21 22:19:08.397] Training set initialized
    [info] [10/28/21 22:19:08.397] Evaluation set initialized
    [info] [10/28/21 22:19:08.397] Preprocessing Complete: 1.137s
    [info] [10/28/21 22:19:08.410] ################ Starting training epoch 1 ################
    [info] [10/28/21 22:19:08.904] Total Edges Processed: 50000, Percent Complete: 0.099
    [info] [10/28/21 22:19:09.252] Total Edges Processed: 95000, Percent Complete: 0.198
    [info] [10/28/21 22:19:09.700] Total Edges Processed: 152000, Percent Complete: 0.298
    [info] [10/28/21 22:19:09.998] Total Edges Processed: 190000, Percent Complete: 0.397
    [info] [10/28/21 22:19:10.418] Total Edges Processed: 237000, Percent Complete: 0.496
    [info] [10/28/21 22:19:10.809] Total Edges Processed: 286000, Percent Complete: 0.595
    [info] [10/28/21 22:19:11.211] Total Edges Processed: 336000, Percent Complete: 0.694
    [info] [10/28/21 22:19:11.567] Total Edges Processed: 383000, Percent Complete: 0.793
    [info] [10/28/21 22:19:11.958] Total Edges Processed: 432000, Percent Complete: 0.893
    [info] [10/28/21 22:19:12.320] Total Edges Processed: 478000, Percent Complete: 0.992
    [info] [10/28/21 22:19:12.357] ################ Finished training epoch 1 ################
    [info] [10/28/21 22:19:12.357] Epoch Runtime (Before shuffle/sync): 3946ms
    [info] [10/28/21 22:19:12.357] Edges per Second (Before shuffle/sync): 122438.414
    [info] [10/28/21 22:19:12.358] Pipeline flush complete
    [info] [10/28/21 22:19:12.374] Edges Shuffled
    [info] [10/28/21 22:19:12.374] Epoch Runtime (Including shuffle/sync): 3963ms
    [info] [10/28/21 22:19:12.374] Edges per Second (Including shuffle/sync): 121913.195
    [info] [10/28/21 22:19:12.389] Starting evaluating
    [info] [10/28/21 22:19:12.709] Pipeline flush complete
    [info] [10/28/21 22:19:15.909] Num Eval Edges: 50000
    [info] [10/28/21 22:19:15.909] Num Eval Batches: 50
    [info] [10/28/21 22:19:15.909] Auc: 0.941, Avg Ranks: 40.139, MRR: 0.336, [email protected]: 0.212, [email protected]: 0.476, [email protected]: 0.600, [email protected]: 0.707, [email protected]: 0.827, [email protected]: 0.895
    [info] [10/28/21 22:19:15.920] Evaluation complete: 3531ms
    [info] [10/28/21 22:19:15.931] ################ Starting training epoch 2 ################
    [info] [10/28/21 22:19:16.361] Total Edges Processed: 46000, Percent Complete: 0.099
    [info] [10/28/21 22:19:16.900] Total Edges Processed: 97000, Percent Complete: 0.198
    [info] [10/28/21 22:19:17.424] Total Edges Processed: 156000, Percent Complete: 0.298
    [info] [10/28/21 22:19:17.697] Total Edges Processed: 189000, Percent Complete: 0.397
    [info] [10/28/21 22:19:18.078] Total Edges Processed: 238000, Percent Complete: 0.496
    [info] [10/28/21 22:19:18.466] Total Edges Processed: 288000, Percent Complete: 0.595
    [info] [10/28/21 22:19:18.825] Total Edges Processed: 336000, Percent Complete: 0.694
    [info] [10/28/21 22:19:19.160] Total Edges Processed: 381000, Percent Complete: 0.793
    [info] [10/28/21 22:19:19.584] Total Edges Processed: 436000, Percent Complete: 0.893
    [info] [10/28/21 22:19:19.909] Total Edges Processed: 481000, Percent Complete: 0.992
    [info] [10/28/21 22:19:19.928] ################ Finished training epoch 2 ################
    [info] [10/28/21 22:19:19.928] Epoch Runtime (Before shuffle/sync): 3997ms
    [info] [10/28/21 22:19:19.928] Edges per Second (Before shuffle/sync): 120876.16
    [info] [10/28/21 22:19:19.929] Pipeline flush complete
    [info] [10/28/21 22:19:19.947] Edges Shuffled
    [info] [10/28/21 22:19:19.948] Epoch Runtime (Including shuffle/sync): 4016ms
    [info] [10/28/21 22:19:19.948] Edges per Second (Including shuffle/sync): 120304.29
    [info] [10/28/21 22:19:19.961] Starting evaluating
    [info] [10/28/21 22:19:20.246] Pipeline flush complete
    [info] [10/28/21 22:19:20.255] Num Eval Edges: 50000
    [info] [10/28/21 22:19:20.255] Num Eval Batches: 50
    [info] [10/28/21 22:19:20.255] Auc: 0.972, Avg Ranks: 21.458, MRR: 0.431, [email protected]: 0.294, [email protected]: 0.595, [email protected]: 0.719, [email protected]: 0.812, [email protected]: 0.906, [email protected]: 0.949
    [info] [10/28/21 22:19:20.271] Evaluation complete: 309ms
    [info] [10/28/21 22:19:20.282] ################ Starting training epoch 3 ################
    [info] [10/28/21 22:19:20.694] Total Edges Processed: 47000, Percent Complete: 0.099
    [info] [10/28/21 22:19:21.042] Total Edges Processed: 95000, Percent Complete: 0.198
    [info] [10/28/21 22:19:21.425] Total Edges Processed: 143000, Percent Complete: 0.298
    [info] [10/28/21 22:19:21.872] Total Edges Processed: 203000, Percent Complete: 0.397
    ^C[info] [10/28/21 22:19:22.195] Total Edges Processed: 244000, Percent Complete: 0.496
    [info] [10/28/21 22:19:22.561] Total Edges Processed: 288000, Percent Complete: 0.595
    [info] [10/28/21 22:19:22.971] Total Edges Processed: 342000, Percent Complete: 0.694
    [info] [10/28/21 22:19:23.266] Total Edges Processed: 380000, Percent Complete: 0.793
    [info] [10/28/21 22:19:23.747] Total Edges Processed: 438000, Percent Complete: 0.893
    [info] [10/28/21 22:19:24.101] Total Edges Processed: 479142, Percent Complete: 0.992
    ...
    

    Environment I tried on 2 machines and got the same error. Platform: linux (Ubuntu 18.04 LTS) Python version: 3.6.9 Pytorch version: 1.10.0+cu102; 1.10.0+cu113

    bug 
    opened by IronySuzumiya 5
  • README example not working

    README example not working

    Describe the bug

    Traceback (most recent call last):
      File "/Users/cthoyt/dev/marius/test.py", line 20, in <module>
        fb15k_example()
      File "/Users/cthoyt/dev/marius/test.py", line 8, in fb15k_example
        train_set, eval_set = m.initializeDatasets(config)
    RuntimeError: filesystem error: in copy_file: No such file or directory [training_data/marius/edges/train/edges.bin] [output_dir/train_edges.pt]
    

    To Reproduce

    I took the example from the README verbatim besides fixing the config path

    import marius as m
    
    def fb15k_example():
        config_path = "/Users/cthoyt/dev/marius/examples/training/configs/kinships_cpu.ini"
        config = m.parseConfig(config_path)
    
        train_set, eval_set = m.initializeDatasets(config)
    
        model = m.initializeModel(config.model.encoder_model, config.model.decoder_model)
    
        trainer = m.SynchronousTrainer(train_set, model)
        evaluator = m.SynchronousEvaluator(eval_set, model)
    
        trainer.train(1)
        evaluator.evaluate(True)
    
    
    if __name__ == "__main__":
        fb15k_example()
    
    

    Expected behavior A clear and concise description of what you expected to happen.

    Environment Mac os 11.2.3 big sur, python 3.9.2, pip installed from latest code on marius

    bug 
    opened by cthoyt 5
  • Change argument variable output_directory to data_directory

    Change argument variable output_directory to data_directory

    What is the documentation lacking? Please describe. The variable output_directory in the input arguments is overloaded. The directory is used to include both the input and output data for a data set. Imprecise naming leads to wrong use of the system.

    Describe the improvement you'd like Rename the variable to data_directory instead of output_directory

    bug documentation enhancement 
    opened by thodrek 4
  • Could I run C++ code for Marius?

    Could I run C++ code for Marius?

    Hi, I built execution files (marius_train and marius_eval), using CMakeLists.txt. However, when I run this execution file to execute as in the example of github, error occurs.

    $ ./marius_train examples/configuration/fb15k_237.yaml

    Result: Aborted (core dumped)

    Is the execution files created through CMake not working at the moment? Or is the input that should be entered differently from when running the marius python??

    Thanks

    question 
    opened by qhtjrmin 3
  • Marius++ code/example request

    Marius++ code/example request

    What is the documentation lacking? Please describe. A code example accompanying the Marius++ paper

    Describe the improvement you'd like A code example accompanying the Marius++ paper

    Additional context Thank you for releasing this amazing repo! Have you released the code/examples to accompany the Marius++ paper - it'd be great to be run Marius++ code to better understand the system. Thank you

    documentation 
    opened by 99snowleopards 3
  • RuntimeError: Expected all tensors to be on the same device

    RuntimeError: Expected all tensors to be on the same device

    Hi,

    I was trying to run marius on ogbn_products dataset on on gcp vm with following: CPU: 16 x Intel Haswell Memory: 60 GB Storage: 1 x SSD 100 GB GPU: 1 x NVIDIA Tesla P100 OS: Ubuntu 20.04

    I installed and ran marius using docker following this.

    OOM is triggered if all edges, features and embeddings are placed in DEVICE_MEMORY. Then, I tried to use mixed CPU-GPU following this sample file (where I think there may be some inconsistencies between the sample code and the explanation below for mixed CPU-GPU section). However, this triggers a RuntimeError where the error message is

    [email protected]:/marius# marius_train examples/configuration/ogbn_products.yaml
    [2022-07-05 15:01:54.234] [info] [marius.cpp:43] Start initialization
    [07/05/22 15:02:03.393] Initialization Complete: 9.158s
    [07/05/22 15:02:18.924] ################ Starting training epoch 1 ################
    Traceback (most recent call last):
      File "/usr/local/bin/marius_train", line 11, in <module>
        load_entry_point('marius==0.0.2', 'console_scripts', 'marius_train')()
      File "/usr/local/lib/python3.6/dist-packages/marius/console_scripts/marius_train.py", line 18, in main
        m.manager.marius_train(config)
    RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking arugment for argument index in method wrapper_index_select)
    

    The configuration file is as following:

    model:
      learning_task: NODE_CLASSIFICATION
      encoder:
        train_neighbor_sampling:
          - type: ALL
          - type: ALL
          - type: ALL
        layers:
          - - type: FEATURE
              output_dim: 100
              bias: true
          - - type: GNN
              options:
                type: GRAPH_SAGE
                aggregator: MEAN
              input_dim: 100
              output_dim: 100
              bias: true
          - - type: GNN
              options:
                type: GRAPH_SAGE
                aggregator: MEAN
              input_dim: 100
              output_dim: 100
              bias: true
          - - type: GNN
              options:
                type: GRAPH_SAGE
                aggregator: MEAN
              input_dim: 100
              output_dim: 47
              bias: true
      decoder:
        type: NODE
      loss:
        type: CROSS_ENTROPY
        options:
          reduction: SUM
      dense_optimizer:
        type: ADAM
        options:
          learning_rate: 0.01
    storage:
      device_type: cuda
      dataset:
        dataset_dir: datasets/ogbn_products/
      edges:
        type: HOST_MEMORY
        options:
          dtype: int
      features:
        type: HOST_MEMORY
        options:
          dtype: float
      embeddings:
        type: HOST_MEMORY
        options:
          dtype: float
    training:
      batch_size: 100
      num_epochs: 2
      pipeline:
        sync: true
    evaluation:
      batch_size: 100
      pipeline:
        sync: true
    
    

    I tried different combinations of setting the type of edges, features and embeddings, but all of them gave the same RuntimeError. Screen Shot 2022-07-05 at 11 20 55

    I am wondering how to solve this error to get marius run on ognb_products using gpu. Thank you.

    question 
    opened by lwwlwwl 3
  • Cannot import marius

    Cannot import marius

    Hello, I installed marius on my local server, using the command 'pip3 install .'. I successfully installed it, but I cannot import marius.

    If I try to import marius or run 'marius_preprocess', then I get the error below:

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/home/.conda/envs/marius/lib/python3.7/site-packages/marius/__init__.py", line 13, in <module>
        from . import _config as config
    ImportError: /home/.conda/envs/marius/lib/python3.7/site-packages/marius/libmarius.so: undefined symbol: _ZNSt12experimental10filesystem2v16statusERKNS1_4pathE
    

    Could I fix it and run marius??

    The following is my setting:

    cmake:
      version: 3.13.2
    cpu_info:
      num_cpus: 64
      total_memory: 754GB
    cuda:
      version: '10.2'
    gpu_info:
      - memory: 24GB
        name: NVIDIA TITAN RTX
    marius:
      bindings_installed: false
      install_path: N/A
      version: N/A
    openmp:
      version: '201511'
    operating_system:
      platform: Linux-4.15.0-162-generic-x86_64-with-debian-buster-sid
    pybind:
      PYBIND11_BUILD_ABI: _cxxabi1011
      PYBIND11_COMPILER_TYPE: _gcc
      PYBIND11_STDLIB: _libstdcpp
    python:
      compiler: GCC 7.5.0
      deps:
        breathe_version: 4.33.1
        numpy_version: 1.21.6
        omegaconf_version: 2.2.1
        pandas_version: 1.3.5
        pip_version: 21.2.2
        pyspark_version: 3.2.1
        pytest_version: 7.1.2
        sphinx_rtd_theme_version: 1.0.0
        torch_version: 1.8.1+cu102
        tox_version: 3.25.0
      version: 3.7.13
    pytorch:
      install_path: /home/.conda/envs/marius/lib/python3.7/site-packages/torch
      version: 1.8.1+cu102
    

    Thanks

    question 
    opened by qhtjrmin 3
  • CVE-2007-4559 Patch

    CVE-2007-4559 Patch

    Patching CVE-2007-4559

    Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

    If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

    opened by TrellixVulnTeam 0
  • Pyspark preprocessor outputs to s3

    Pyspark preprocessor outputs to s3

    the preprocessor now writes processed edge and node data to s3, but the data is split into many files. need to combine them.

    the following errors when the files are small,

    s3_obj.merge(output_filename, files_list)
    

    throws the error EntityTooSmall.

    once we have a single file, we can look into converting that to binary. Alternatively, we can define a custom writer that outputs in binary format without the intermediate csv files.

    opened by basavaraj29 0
  • Spark preprocessor optimization

    Spark preprocessor optimization

    • removing id assignment for edges
    • using zipwithindex instead of repartition(1) and windowing
    • parititonBy([src_bucket, dst_bucket])

    todo:

    • custom binary writer to eliminate intermediate csv
    opened by basavaraj29 3
  • About mini-batch training and edge bucket

    About mini-batch training and edge bucket

    Whether each bucket will perform mini-batch training? On your paper, whether each bucket performs 4(bound) mini-batch training? Is my understanding correct? Thanks a lot!

    question 
    opened by YijianLiu 1
  • Integrated s3 client lib and its dependencies

    Integrated s3 client lib and its dependencies

    the following libraries need to be installed in the base image

    apt install zlib1g-dev
    apt install zlib1g
    apt-get install libssl-dev
    apt install curl
    apt-get install libcurl4-openssl-dev
    

    creds are read from ~/.aws/credentials. format

    [default]
    aws_access_key_id=<...>
    aws_secret_access_key=<...>
    

    test code, will check in an example if the changes are good to go with.

    #include <aws/s3/S3Client.h>
    #include <aws/core/Aws.h>
    #include <aws/s3/model/ListObjectsRequest.h>
    #include <aws/s3/model/Object.h>
    
    Aws::SDKOptions options;
    Aws::InitAPI(options);
    Aws::Client::ClientConfiguration client_config;
    Aws::S3::S3Client s3_client(client_config);
    Aws::S3::Model::ListObjectsRequest request;
    
    const Aws::String bucket_name = "fb15k237";
    request.WithBucket(bucket_name);
    auto outcome = s3_client.ListObjects(request);
    
    if (!outcome.IsSuccess()) {
        std::cerr << "Error: ListObjects: " <<
                  outcome.GetError().GetMessage() << std::endl;
    }
    else {
        Aws::Vector<Aws::S3::Model::Object> objects =
                outcome.GetResult().GetContents();
    
        for (Aws::S3::Model::Object &object: objects) {
            std::cout << object.GetKey() << std::endl;
        }
    }
    
    opened by basavaraj29 0
  • Spark preprocessor now works with s3

    Spark preprocessor now works with s3

    expects env variables S3_BUCKET, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

    marius_preprocess --edges s3a://fb15k237/train.txt s3a://fb15k237/valid.txt s3a://fb15k237/test.txt --output_directory /home/data/datasets/fb15k_237/ --spark
    

    writes preprocessed output to the local directory as well as s3, can delete the files from local, but keeping them for now.

    opened by basavaraj29 0
Releases(v0.0.1)
  • v0.0.1(Sep 26, 2022)

    This release contains the initial artifact for the paper MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networks to be published at EuroSys 2023. The artifact contains the necessary code to reproduce experiments reported in the paper.

    Source code(tar.gz)
    Source code(zip)
Owner
Marius
Graph Learning at Scale
Marius
This is the code of our paper An Efficient Training Approach for Very Large Scale Face Recognition or F²C for simplicity.

Fast Face Classification (F²C) This is the code of our paper An Efficient Training Approach for Very Large Scale Face Recognition or F²C for simplicit

null 33 Jun 27, 2021
FG-Net: Fast Large-Scale LiDAR Point Clouds Understanding Network Leveraging Correlated Feature Mining and Geometric-Aware Modelling

FG-Net: Fast Large-Scale LiDAR Point Clouds Understanding Network Leveraging Correlated Feature Mining and Geometric-Aware Modelling Comparisons of Running Time of Our Method with SOTA methods RandLA and KPConv:

Kangcheng LIU 77 Dec 2, 2022
Square Root Bundle Adjustment for Large-Scale Reconstruction

Square Root Bundle Adjustment for Large-Scale Reconstruction

Nikolaus Demmel 202 Nov 24, 2022
A header-only C++ library for large scale eigenvalue problems

NOTE: Spectra 1.0.0 is released, with a lot of API-breaking changes. Please see the migration guide for a smooth transition to the new version. NOTE:

Yixuan Qiu 600 Nov 26, 2022
GalaxyEngine is a MySQL branch originated from Alibaba Group, especially supports large-scale distributed database system.

GalaxyEngine is a MySQL branch originated from Alibaba Group, especially supports large-scale distributed database system.

null 276 Nov 29, 2022
[NeurIPS 2021 Spotlight] Learning to Delegate for Large-scale Vehicle Routing

Learning to Delegate for Large-scale Vehicle Routing This directory contains the code, data, and model for our NeurIPS 2021 Spotlight paper Learning t

null 44 Nov 23, 2022
Codebase for "SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems"

Codebase for "SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems"

Beidi Chen 1k Nov 27, 2022
Hierarchical Engine for Large-scale Infrastructure Co-Simulation (HELICS)

A multi-language, cross-platform library that enables different simulators to easily exchange data and stay synchronized in time. Scalable from two si

GMLC-TDC 83 Oct 14, 2022
A large scale non-linear optimization library

Ceres Solver Ceres Solver is an open source C++ library for modeling and solving large, complicated optimization problems. It is a feature rich, matur

null 2.9k Dec 2, 2022
An Open-Source Analytical Placer for Large Scale Heterogeneous FPGAs using Deep-Learning Toolkit

DREAMPlaceFPGA An Open-Source Analytical Placer for Large Scale Heterogeneous FPGAs using Deep-Learning Toolkit. This work leverages the open-source A

Rachel Selina Rajarathnam 23 Sep 15, 2022
Learning embeddings for classification, retrieval and ranking.

StarSpace StarSpace is a general-purpose neural model for efficient learning of entity embeddings for solving a wide variety of problems: Learning wor

Facebook Research 3.8k Nov 29, 2022
An optimized version of the KGloVe code (global embeddings for Knowledge Graphs)

Optimized Implementation of the KGloVe algorithm This repository contains an optimized implementation of KGloVe, which was proposed in Cochez M., Rist

Michael Cochez 2 Feb 25, 2022
Code for ICLR'22 "Inductive Relation Prediction Using Analogy Subgraph Embeddings"

ANalogy subGraph Embedding Learning (GraphANGEL) Code for ICLR'22 "Inductive Relation Prediction Using Analogy Subgraph Embeddings" This includes pyto

Jinjiarui 27 Nov 16, 2022
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

This is the Vowpal Wabbit fast online learning code. Why Vowpal Wabbit? Vowpal Wabbit is a machine learning system which pushes the frontier of machin

Vowpal Wabbit 8.1k Nov 27, 2022
A modern dynamically typed programming language that gets compiled to bytecode and is run in a virtual machine called SVM (Strawbry Virtual Machine).

Strawbry A bytecode programming language. Here is what I want Strawbry to look like: var a = 1 var b = 2 var c = a + b print(c) func sqrt(x) { re

PlebusSupremus1234 6 Jan 5, 2022
Professor Terence Parr has taught us how to create a virtual machine Now it is time to pwn virtual machine

My First real world CTF Simple Virtual Machine Challenge description Professor Terence Parr has taught us how to create a virtual machine Now it is ti

null 1 Feb 17, 2022
Microsoft 2.5k Nov 28, 2022
This the contains the test examples and validator tool for the ISPD2021 Wafer-Scale Physics Modeling contest.

This readme documents information regarding the validator/scorer which will be used for the 2021 ISPD Contest problem: Wafer-Scale Physics Modelling

Cerebras 16 Aug 22, 2022
You Only Look Twice: Rapid Multi-Scale Object Detection In Satellite Imagery

YOLT You Only Look Twice: Rapid Multi-Scale Object Detection In Satellite Imagery As of 24 October 2018 YOLT has been superceded by SIMRDWN YOLT is an

CosmiQ Works 619 Nov 20, 2022
An Aspiring Drop-In Replacement for Pandas at Scale

Legate Pandas Legate Pandas is a distributed and accelerated drop-in replacement of Pandas. Legate Pandas enables high-performance, scalable execution

Legate 68 Oct 3, 2022