BladeBit - Fast Chia (XCH) RAM-only k32-only Plotter

Related tags

Data Structures bladebit
Overview

BladeBit Chia Plotter

Release Builds

A fast RAM-only, k32-only, Chia plotter.

Requirements

416 GiB of RAM are required to run it, plus a few more megabytes for stack space and small allocations.

64-bit is supported only, for obvious reasons.

Prerequisites

Only Linux & Windows are supported.

Linux

Install the following packages:

# CentOS or Amazon Linux
sudo yum group install -y "Development Tools"
sudo yum install -y cmake gmp-devel numactl-devel

# Ubuntu or Debian-based
sudo apt install -y build-essential cmake libgmp-dev libnuma-dev

Windows

Must have Visual Studio 2019 or its build tools installed.

Building

# Clone the repo & its submodules
git clone https://github.com/Chia-Network/bladebit.git && cd bladebit

# Create a build directory for cmake and cd into it
mkdir build
cd build

# Generate config files & build
cmake ..
cmake --build . --target bladebit --config Release

The resulting binary will be found under the build/ directory. On Windows it will be under build/Release/.

Usage

Run bladebit with the -h for complete usage and command line options:

# Linux
build/bladebit -h

# Windows
build/Release/bladebit.exe -h

License

Licensed under the Apache 2.0 license. See LICENSE.

Other Details

Disk I/O

Writes to disk only occur to the final plot file, and it is done sequentially, un-buffered, with direct I/O. This means that writes will be block-aligned. If you've gotten faster writes elsewhere in your drive than you will get with this, it is likely that it is using buffered writes, therefore it "finishes" before it actually finishes writing to disk. The kernel will handle the I/O in the background from cache (you can confirm this with tools such as iotop). The final writes here ought to pretty much saturate your sequential writes. Writes begin happening in the background at Phase 3 and will continue to do so, depending on the disk I/O throughput, through the next plot, if it did not finish beforehand. At some point in Phase 1 of the next plot, it might stall if it still has not finished writing to disk and a buffer it requires is still being written to disk. On the system I tested, there was no interruption when using an NVMe drive.

Pool Plots

Pool plots are fully supported and tested against the chia-blockchain implementation. The community has also verified that pool plots are working properly and winning proofs with them.

NUMA systems

Memory is bound on interleaved mode for NUMA systems which currently gives the best performance on systems with several nodes. This is the default behavior on NUMA systems, it can be disabled with with the -m or --no-numa switch.

Huge TLBs

This is not supported yet. Some folks have reported some gains when using huge page sizes. Although this was something I wanted to test, I focused first instead on things that did not necessarily depended on system config. But I'd like to add support for it in the future (trivial from the development point of view, I have just not configured the test system with huge page sizes).

Other Observations

This implementation is highly memory-bound so optimizing your system towards fast memory access is essential. CPUs with large caches will benefit as well.

Copyright 2021 Harold Brenes, Chia Network Inc

Comments
  • Still crashing at

    Still crashing at "Sorting F7 and writing C tables" despite Fix 251

    Forked master and then manually applied Fix 251 (since it still seems to only be in develop branch). Still crashing at "Sorting F7 and writing C tables" when running on 512 buckets. It can actually still happen with 256 buckets too - for me, on a run of 15 plots, it crashed in the same spot on the 11th plot. So not frequent, but not that rare either.

    This is on Windows 10, 7950x processor which has 32 threads available, I only used 24, 100G cache. Getting 20.7 minute plots with 256 buckets. Phase 1 Tables 1-7 do seem to go a bit faster with 512 buckets (256 - 476 seconds, 512 - 459.5 seconds) but, then, crash.

    This is the command I used to test (only difference between the two test runs for those times was the -b 512):

    .\bladebit.exe -n 1 -t 24 -p blahblahblah -f blahblahblah diskplot -a --cache 100G -b 512 -t1 C:\bladebit\bladebit1\ C:\bladebit\FinishedPlots\ >> C:\bladebit\bladebit.log

    opened by Qwinn1 0
  • diskplot crashes when using ZFS backed temp storage path

    diskplot crashes when using ZFS backed temp storage path

    As best I can tell, bladebit has some kind of bug where it's unable to use temp storage formatted in ZFS. My tests run as follows:

    ./bladebit -v -t 10 -f <farmer_key> -c <pubkey> diskplot --cache 200GB -b 64 -t1 /tank/bladebit-zfs-temp/ /mnt/bladebit-final/

    This command uses a ZFS formatted (specifically, raidz2) backed storage for the temp path and fails with the following crash dump.

    ./bladebit(_ZN11MTJobRunnerI9AnonMTJobLj256EE13RunJobWrapperEPS0_+0x41)[0x55931605bb91]
./bladebit(_ZN10ThreadPool17FixedThreadRunnerEPv+0x52)[0x55931604da52]                                ./bladebit(_ZN6Thread17ThreadStarterUnixEPS_+0x76)[0x55931605e396]
/lib/x86_64-linux-gnu/libc.so.6(+0x94b43)[0x7f5f144d0b43]
/lib/x86_64-linux-gnu/libc.so.6(+0x126a00)[0x7f5f14562a00]

    Running the same command, however, with the temp path set to an XFS partition on top of a ZFS volume, while slow, does succeed:

    ./bladebit -v -t 10 -f <farmer_key> -c <pubkey> diskplot --cache 200GB -b 64 -t1 /mnt/bladebit-xfs-tmp/ /mnt/bladebit-final/

    System information:

    OS: Ubuntu 22.04 Bladebit Version: 2.0.1-dev

    opened by tedwardd 0
  • showing help with

    showing help with "bladebit help ramplot" - unknown command - BUG

    When I type "./bladebit help ramplot" I get the following:

    [root@BladBit2-1 build]# [root@BladBit2-1 build]# ./bladebit help ramplot

    Fatal Error:
    Unknown command 'ramplot'. [root@BladBit2-1 build]# [root@BladBit2-1 build]#

    all other commands work like "./bladebit help diskplot" and others.

    opened by gibblinhimer 0
  • Can Bladebit ramplot add tmpfs support ?

    Can Bladebit ramplot add tmpfs support ?

    My computer have 768GB Ram,I create a 110GB tpmfs dir and mount to /mnt/ramdisk, but the Bladebit will print ERROR when I use the /mnt/ramdisk as the output dir.

    my plotting cmd:

    ./bladebit -f xxx -c xxx -n 100 /mnt/ramdisk

    this is the error log:

    ERROR: Failed to open plot output file at /mnt/ramdisk/plot-k32-xxxxxxxxxxx.plot ERROR: PLOT xxxxxxxxxxx failed... Trying next plot.

    opened by liyifeng 0
  • Crashed! on second plot at Table 3

    Crashed! on second plot at Table 3

    OS: Clear Linux 37640 CPU: Intel 12700T RAM: 48GB DDR4 SSD: 1024GB Kioxia M.2 PCIe Gen 4

    Command run ./bladebit -f xxxxxx-c xxxxxx -t 19 -n 12 diskplot -t1 /home/admin/temp -b 64 --cache 30G /run/media/admin/hdd/chia/farm

    crash.log './bladebit(_Z12CrashHandleri+0xaa)[0x55b8df83718a] /usr/lib64/glibc-hwcaps/x86-64-v3/libc.so.6(+0x3ac50)[0x55b8df21bc50] ./bladebit(_ZN15DiskBufferQueue15CmdWriteBucketsERKNS_7CommandEm+0xe4)[0x55b8df844684] ./bladebit(_ZN15DiskBufferQueue11CommandMainEv+0x48f)[0x55b8df845d3f] ./bladebit(+0x112e3f)[0x55b8df845e3f] ./bladebit(ZN6Thread17ThreadStarterUnixEPS+0x6d)[0x55b8df8381bd] /usr/lib64/glibc-hwcaps/x86-64-v3/libc.so.6(+0x90a6d)[0x55b8df271a6d] /usr/lib64/glibc-hwcaps/x86-64-v3/libc.so.6(__clone+0x40)[0x55b8df2fd800]`

    opened by gamerkonks 0
Releases(v2.0.1)
Owner
Chia Network
The green cryptocurrency with Chialisp
Chia Network
GitHub
Graphs the progress of block height in your Chia Full Node.

Chia Height Graph Monitor for Chia Block Height Introduction The chiaheightgraph tool will graph Chia Block Height changes in a linux terminal. Use a

Bram Stolk 9 Sep 14, 2021
A family of header-only, very fast and memory-friendly hashmap and btree containers.

The Parallel Hashmap Overview This repository aims to provide a set of excellent hash map implementations, as well as a btree alternative to std::map

Gregory Popovitch 1.7k Jan 9, 2023
A fast hash map/hash table (whatever you want to call it) for the C programming language.

C HashMap A fast hash map/hash table (whatever you want to call it) for the C programming language. It can associate a key with a pointer or integer v

Mashpoe 74 Dec 27, 2022
Simple C++ code to benchmark fast division algorithms

fast_division Simple C++ code to benchmark fast division algorithms relying on constant divisors. The code is a companion to the paper Integer Divisio

Daniel Lemire 39 Dec 27, 2022
This project Orchid-Fst implements a fast text string dictionary search data structure: Finite state transducer (short for FST) in c++ language.This FST C++ open source project has much significant advantages.

Orchid-Fst 1. Project Overview This project Orchid-Fst implements a fast text string dictionary search data structure: Finite state transducer , which

Bin Ding 10 Oct 18, 2022
Another neofetch-like utility but this time it's fast.

SystemFetch Another neofetch-like utility but this time it's fast. Example Speed Here is a table of the time it took to execute all of these programs,

YSU 10 Jul 22, 2021
This is a curve topology verification tool based on Fast Linking Numbers for Topology Verification of Loopy Structures.

Fast Linking Numbers This tool, called verifycurves, takes input models that consist of closed-loop curves, and outputs a topology certificate as a .t

Ante Qu 23 Dec 24, 2022
merge two sorted lists fast

Py Merge Merge sorted list faster than using list.sort or heapq.merge. import merge # create some sorted lists a = list(range(-100, 1700)) b = list(r

Earthly 10 Nov 21, 2021
A fast multi-producer, multi-consumer lock-free concurrent queue for C++11

moodycamel::ConcurrentQueue An industrial-strength lock-free queue for C++. Note: If all you need is a single-producer, single-consumer queue, I have

Cameron 7.4k Jan 2, 2023
C++ implementation of a fast hash map and hash set using hopscotch hashing

A C++ implementation of a fast hash map and hash set using hopscotch hashing The hopscotch-map library is a C++ implementation of a fast hash map and

Thibaut Goetghebuer-Planchon 578 Dec 23, 2022
🏅State-of-the-art learned data structure that enables fast lookup, predecessor, range searches and updates in arrays of billions of items using orders of magnitude less space than traditional indexes

The Piecewise Geometric Model index (PGM-index) is a data structure that enables fast lookup, predecessor, range searches and updates in arrays of bil

Giorgio Vinciguerra 650 Dec 31, 2022
Fast & memory efficient hashtable based on robin hood hashing for C++11/14/17/20

➵ robin_hood unordered map & set robin_hood::unordered_map and robin_hood::unordered_set is a platform independent replacement for std::unordered_map

Martin Leitner-Ankerl 1.3k Dec 31, 2022
C++ implementation of a fast hash map and hash set using robin hood hashing

A C++ implementation of a fast hash map and hash set using robin hood hashing The robin-map library is a C++ implementation of a fast hash map and has

Thibaut Goetghebuer-Planchon 878 Jan 2, 2023
A fast Python Common substrings of multiple strings library with C++ implementation

A fast Python Common substrings of multiple strings library with C++ implementation Having a bunch of strings, can I print some substrings which appea

Đào Nguyên Dương 7 Aug 21, 2022
An unordered C++ data container providing fast iteration/insertion/erasure while maintaining pointer/iterator validity to non-erased elements regardless of insertions/erasures. Provides higher-performance than std:: library containers for high-modification scenarios with unordered data.

plf::colony An unordered data container providing fast iteration/insertion/erasure while maintaining pointer/iterator/reference validity to non-erased

Matt Bentley 344 Dec 19, 2022
Typesafe, Generic & Extremely fast Dictionary in C 🚀

CDict.h Typesafe, Generic, and Extremely Fast Dictionary in C ?? Key Features Extremely fast non-cryptographic hash algorithm XXHash Complete Typesafe

Robus Gauli 18 Oct 5, 2022
Simple and fast configuration file library (written in C99)

Features Configuration file reading Supported operating systems Ubuntu MacOS Windows Build requirements C99 compiler CMake 3.10+ Cloning git clone htt

Nikita Fediuchin 3 May 26, 2022
Very Fast Non-Cryptographic Hash Function

KOMIHASH - Very Fast Hash Function Introduction The komihash() function available in the komihash.h file implements a very fast 64-bit hash function,

Aleksey Vaneev 94 Dec 12, 2022
A fast and efficient non-iterating hashmap library

libhash: a fast and efficient non-iterating hashmap library Libhash is a fast and efficient non-iterating hashmap library Usage Usage is easy and simp

Oğuzhan Eroğlu 6 Aug 19, 2022