chia-plotter (pipelined multi-threaded)

Overview

chia-plotter (pipelined multi-threaded)

This is a new implementation of a chia plotter which is designed as a processing pipeline, similar to how GPUs work, only the "cores" are normal software CPU threads.

As a result this plotter is able to fully max out any storage device's bandwidth, simply by increasing the number of "cores", ie. threads.

Sponsored by Flexpool.io - Check them out if you're looking for a secure and scalable Chia pool.

Usage

Join the Discord for support: https://discord.gg/pQwkebKnPB

For 
   
     and 
    
      see output of `chia keys show`.
To plot for pools, specify 
     
       address via -c instead of 
      
       , see `chia plotnft show`.

       
         needs about 220 GiB space, it will handle about 25% of all writes. (Examples: './', '/mnt/tmp/') 
        
          needs about 110 GiB space and ideally is a RAM drive, it will handle about 75% of all writes. Combined (tmpdir + tmpdir2) peak disk usage is less than 256 GiB. In case of 
         
           != 1, you may press Ctrl-C for graceful termination after current plot is finished, or double press Ctrl-C to terminate immediately. Usage: chia_plot [OPTION...] -k, --size arg K size (default = 32, k <= 32) -x, --port arg Network port (default = 8444, chives = 9699) -n, --count arg Number of plots to create (default = 1, -1 = infinite) -r, --threads arg Number of threads (default = 4) -u, --buckets arg Number of buckets (default = 256) -v, --buckets3 arg Number of buckets for phase 3+4 (default = buckets) -t, --tmpdir arg Temporary directory, needs ~220 GiB (default = $PWD) -2, --tmpdir2 arg Temporary directory 2, needs ~110 GiB [RAM] (default = 
          
           ) -d, --finaldir arg Final directory (default = 
           
            ) -w, --waitforcopy Wait for copy to start next plot -p, --poolkey arg Pool Public Key (48 bytes) -c, --contract arg Pool Contract Address (62 chars) -f, --farmerkey arg Farmer Public Key (48 bytes) -G, --tmptoggle Alternate tmpdir/tmpdir2 (default = false) -D, --directout Create plot directly in finaldir (default = false) -K, --rmulti2 arg Thread multiplier for P2 (default = 1) --help Print help 
           
          
         
        
       
      
     
    
   

Make sure to crank up if you have plenty of cores, the default is 4. Depending on the phase more threads will be launched, the setting is just a multiplier.

RAM usage depends on and . With the new default of 256 buckets it's about 0.5 GB per thread at most.

-G option will alternate the temp dirs used while plotting to give each one, tmpdir and tmpdir2, equal usage. The first plot creation will use tmpdir and tmpdir2 as expected. The next run, if -n equals 2 or more, will swap the order to tmpdir2 and tmpdir. The next run swaps again to tmpdir and tmpdir2. This will occur until the number of plots created is reached or until stopped.

RAM disk setup on Linux

sudo mount -t tmpfs -o size=110G tmpfs /mnt/ram/

Note: 128 GiB System RAM minimum required for RAM disk.

How to Support

XCH: xch1w5c2vv5ak08pczeph7tp5xmkl5762pdf3pyjkg9z4ks4ed55j3psgay0zh

XFX: xfx1succfn2z3uwmq50ukztjanrvs9kw294mqn4lv22rk6tzmcu7e2xsyxyaa5

XCC: xcc16j65y35fs8u289nq6krcyehsmp5eqd4we493rxf36pg7eymcqrqqltsrat

ETH-ERC20: 0x97057cdf529867838d2a1f7f23ba62456764e0cd

LTC: MNUnszsX2srv5EJpu9YYHAXb19MqUpuBjD

BTC: 15GSE5ymStxXMvJ58hyosEVm4FXFxUyJZg

Results

On a dual Xeon® E5-2650v2@2.60GHz R720 with 256GB RAM and a 3x800GB SATA SSD RAID0, using a 110G tmpfs for :

Click to expand
Number of Threads: 16
Number of Buckets: 2^8 (256)
Working Directory:   /mnt/tmp3/chia/tmp/ 
Working Directory 2: /mnt/tmp3/chia/tmp/ram/
[P1] Table 1 took 17.2488 sec
[P1] Table 2 took 145.011 sec, found 4294911201 matches
[P1] Table 3 took 170.86 sec, found 4294940789 matches
[P1] Table 4 took 203.713 sec, found 4294874801 matches
[P1] Table 5 took 201.346 sec, found 4294830453 matches
[P1] Table 6 took 195.928 sec, found 4294681297 matches
[P1] Table 7 took 158.053 sec, found 4294486972 matches
Phase 1 took 1092.2 sec
[P2] max_table_size = 4294967296
[P2] Table 7 scan took 15.5542 sec
[P2] Table 7 rewrite took 37.7806 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 46.7014 sec
[P2] Table 6 rewrite took 65.7315 sec, dropped 581295425 entries (13.5352 %)
[P2] Table 5 scan took 45.4663 sec
[P2] Table 5 rewrite took 61.9683 sec, dropped 761999997 entries (17.7423 %)
[P2] Table 4 scan took 44.8217 sec
[P2] Table 4 rewrite took 61.36 sec, dropped 828847725 entries (19.2985 %)
[P2] Table 3 scan took 44.9121 sec
[P2] Table 3 rewrite took 61.5872 sec, dropped 855110820 entries (19.9097 %)
[P2] Table 2 scan took 43.641 sec
[P2] Table 2 rewrite took 59.6939 sec, dropped 865543167 entries (20.1528 %)
Phase 2 took 620.488 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 73.1018 sec, wrote 3429368034 right entries
[P3-2] Table 2 took 42.3999 sec, wrote 3429368034 left entries, 3429368034 final
[P3-1] Table 3 took 68.9318 sec, wrote 3439829969 right entries
[P3-2] Table 3 took 43.8179 sec, wrote 3439829969 left entries, 3439829969 final
[P3-1] Table 4 took 71.3236 sec, wrote 3466027076 right entries
[P3-2] Table 4 took 46.2887 sec, wrote 3466027076 left entries, 3466027076 final
[P3-1] Table 5 took 70.6369 sec, wrote 3532830456 right entries
[P3-2] Table 5 took 45.5857 sec, wrote 3532830456 left entries, 3532830456 final
[P3-1] Table 6 took 75.8534 sec, wrote 3713385872 right entries
[P3-2] Table 6 took 48.8266 sec, wrote 3713385872 left entries, 3713385872 final
[P3-1] Table 7 took 83.2586 sec, wrote 4294486972 right entries
[P3-2] Table 7 took 56.3803 sec, wrote 4294486972 left entries, 4294486972 final
Phase 3 took 733.323 sec, wrote 21875928379 entries to final plot
[P4] Starting to write C1 and C3 tables  
[P4] Finished writing C1 and C3 tables   
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 84.6697 sec, final plot size is 108828428322 bytes
Total plot creation time was 2530.76 sec 

How to Verify

To make sure the plots are valid you can use the ProofOfSpace tool from chiapos:

git clone https://github.com/Chia-Network/chiapos.git
cd chiapos && mkdir build && cd build && cmake .. && make -j8
./ProofOfSpace check -f plot-k32-???.plot [num_iterations]

How to update to latest version

cd chia-plotter
git checkout master
git pull
git submodule update --init
./make_devel.sh

Future Plans

I do have some history with GPU mining, back in 2014 I was the first to open source a XPM GPU miner, which was about 40x more efficient than the CPU miner. See my other repos.

As such, it's only a matter of time until I add OpenCL support to speed up the plotter even more, keeping most of the load off the CPUs.

Dependencies

  • cmake (>=3.14)
  • libsodium-dev

Install

Windows

Binaries built by stotiks can be found here: https://github.com/stotiks/chia-plotter/releases

Arch Linux

First, install dependencies from pacman:

sudo pacman -S cmake libsodium gmp gcc11

Then, clone and compile the project:

# Checkout the source and install
git clone https://github.com/madMAx43v3r/chia-plotter.git
cd chia-plotter

git submodule update --init
./make_devel.sh
./build/chia_plot --help
CentOS 7
git clone https://github.com/madMAx43v3r/chia-plotter.git
cd chia-plotter

git submodule update --init
sudo yum install epel-release -y
sudo yum install cmake3 libsodium libsodium-static -y
ln /usr/bin/cmake3 /usr/bin/cmake
# Install a package with repository for your system:
# On CentOS, install package centos-release-scl available in CentOS repository:
sudo yum install centos-release-scl -y
# Install the collection:
sudo yum install devtoolset-7 -y
# Start using software collections:
scl enable devtoolset-7 bash
./make_devel.sh
./build/chia_plot --help
Clear Linux
sudo swupd update
sudo swupd bundle-add c-basic devpkg-libsodium git wget

echo PATH=$PATH:/usr/local/bin/ # for statically compiled cmake if not already in your PATH

# Install libsodium
cd /tmp
wget https://download.libsodium.org/libsodium/releases/LATEST.tar.gz
tar -xvf LATEST.tar.gz
cd libsodium-stable
./configure
make && make check
sudo make install
# Checkout the source and install
cd ~/
git clone https://github.com/madMAx43v3r/chia-plotter.git 
cd ~/chia-plotter
git submodule update --init
./make_devel.sh
./build/chia_plot --help
Ubuntu 20.04
sudo apt install -y libsodium-dev cmake g++ git build-essential
# Checkout the source and install
git clone https://github.com/madMAx43v3r/chia-plotter.git 
cd chia-plotter

git submodule update --init
./make_devel.sh
./build/chia_plot --help

The binaries will end up in build/, you can copy them elsewhere freely (on the same machine, or similar OS).

Debian 10 ("buster")

Make sure to add buster-backports to your sources.list otherwise the installation will fail because an older cmake version. See the debian backport documentation for reference.

# Install cmake 3.16 from buster-backports
sudo apt install -t buster-backports cmake
sudo apt install -y libsodium-dev g++ git
# Checkout the source and install
git clone https://github.com/madMAx43v3r/chia-plotter.git 
cd chia-plotter

git submodule update --init
./make_devel.sh
./build/chia_plot --help

The binaries will end up in build/, you can copy them elsewhere freely (on the same machine, or similar OS).

macOS

First you need to install the Brew package manager and Xcode OR Xcode Command Line Tools.

# Download Xcode Command Line Tools (skip if you already have Xcode)
xcode-select --install

# Now download chia-plotter's dependencies
brew install libsodium cmake git autoconf automake libtool wget
brew link cmake
git clone https://github.com/madMAx43v3r/chia-plotter.git 
cd chia-plotter
git submodule update --init
./make_devel.sh
./build/chia_plot --help

Linking libsodium should be performed automatically, but in case of failure please follow these instructions:

# If you downloaded Xcode run these:
sudo ln -s /usr/local/include/sodium.h /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/
sudo ln -s /usr/local/include/sodium /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/

# If you downloaded CommandLineTools run these:
sudo ln -s /usr/local/include/sodium.h /Library/Developer/CommandLineTools/usr/include
sudo ln -s /usr/local/include/sodium /Library/Developer/CommandLineTools/usr/include

Confirm which directory you have on YOUR Mac before applying following commands

# For x86_64 Macs
wget https://raw.githubusercontent.com/facebookincubator/fizz/master/build/fbcode_builder/CMake/FindSodium.cmake -O /usr/local/opt/cmake/share/cmake/Modules/FindSodium.cmake

or

# For ARM64 (M1) Macs
wget https://raw.githubusercontent.com/facebookincubator/fizz/master/build/fbcode_builder/CMake/FindSodium.cmake -O /opt/homebrew/Cellar/cmake/*/share/cmake/Modules/FindSodium.cmake

If a maximum open file limit error occurs (as default OS setting is 256, which is too low for default bucket size of 256), run this before starting the plotter

ulimit -n 3000

This file limit change will only affect the current session.

Running in a Docker container

In some setups and scenarios, it could be useful to run your plotter inside a Docker container. This could be potentially useful while running chia-plotter in Windows.

To do so, install Docker on your computer and them run the following command:

docker run \
  -v <path-to-your-tmp-dir>:/mnt/harvester \
  -v <path-to-your-final-dir>:/mnt/farm \
  odelucca/chia-plotter \
    -t /mnt/harvester/ \
    -d /mnt/farm/ \
    -p <pool-key> \
    -f <farm-key> \
    -r <number-of-CPU-cores>

💡 You can provide any of the plotter arguments after the image name (odelucca/chia-plotter)

In a Linux benchmark, we were able to find that running in Docker is only 5% slower than running in native OS.

For Windows users, you should check if your Docker configuration has any RAM or CPU limits. Since Docker runs inside HyperV, that could potentially constrain your hardware usage. In any case, you can set the RAM limits with the -m flag (after the docker run command).

Regarding multithread in Docker

While running in Windows, you may need to proper configure your Docker to allow multi CPUs. You can do so by following this article

In a nutshell, you could also pass the --cpus flag to your docker run command in order to achieve the same result.

So, for example, the following command...

docker run \
  -v <path-to-your-tmp-dir>:/mnt/harvester \
  -v <path-to-your-final-dir>:/mnt/farm \
  -m 8G \
  --cpus 8 \
  odelucca/chia-plotter \
    -t /mnt/harvester/ \
    -d /mnt/farm/ \
    -p <pool-key> \
    -f <farm-key> \
    -r 8

...would run your plotter with 8 CPU cores and 8GB of RAM.

Building a Docker container

Make sure your submodules are up-to-date by running git submodule update --init, then simply build with docker build .


Known Issues

  • Needs at least cmake 3.14 (because of bls-signatures)

How to install latest cmake on Ubuntu 18.04

Issues
  • Feature: Docker Image

    Feature: Docker Image

    This PR adds a Docker Image to improve portability. We can use that image for both Windows and Linux machines.

    How it works

    You can test a preview generated image with the following command on your machine:

    docker run \
      -v <path-to-your-tmp-dir>:/mnt/harvester \
      -v <path-to-your-final-dir>:/mnt/farm \
      odelucca/chia-plotter \
        -t /mnt/harvester/ \
        -d /mnt/farm/ \
        -p <pool-key> \
        -f <farm-key>
    

    Of course, you can provide whatever arguments you want after the image name.

    Also, you can make an alias, like:

    alias chia-plotter="docker run \
      -v <path-to-your-tmp-dir>:/mnt/harvester \
      -v <path-to-your-final-dir>:/mnt/farm \
      odelucca/chia-plotter"
    

    And call it with:

    chia-plotter <args>
    

    I've run a few tests, and it seems that it has a small performance cost. I was not able to define the exact percentage of the performance decrease yet, but still I think that is a pretty good alternative for running the plotter in Windows machines.

    Future improvements

    I see two important future improvements, they are:

    1. We need to host this image in a proper Docker repository
    2. We could create a simple script (maybe inside the plotter itself?) where it could identify the remaining farm disk storage and automatically work to fill it up. By doing so, we could simply launch new containers for each harvester memory / rig
    opened by delucca 218
  • Dual Xeon on Windows version doesn't work (kills process)

    Dual Xeon on Windows version doesn't work (kills process)

    So I'm using dual Xeon (8 cores, 16 threads each) server with 64G of ram and running Windows 2019 server. I start it with the following command (alpha version 0.2):

    .\chia_plot.exe -p "key" -f "key" -n 1 -r 24 -u 7 -t E:\directory\ -d D:\directory\

    What I get as a result is:

    Final Directory: D:\directory\
    Number of Plots: 1
    Process ID: 7128
    Number of Threads: 24
    Number of Buckets: 2^7 (128)
    

    And after that process is killed and does not continue.

    I tried with lower number of threads (went even down to 4).

    Was looking for help and got same configuration from a friend with same end result. Process killed and he's also not able to find a way to make it work.

    Any ideas?

    opened by Hedonay 109
  • Not compiling on Apple M1 system

    Not compiling on Apple M1 system

    Getting some compiling errors, is this info helpful:

    -- Configured operating system: MACOSX
    -- Configured GMP: /opt/homebrew/lib/libgmp.a
    -- Compiler flags:  -Wall   -O3 -funroll-loops -fomit-frame-pointer -pthread
    -- Linker flags: -L/Applications/Chia-arm/maxplotter/chia-plotter/build/_deps/relic-src/src/low/gmp/  -static
    -- Configured /Applications/Chia-arm/maxplotter/chia-plotter/build/_deps/relic-src/include/relic_conf.h.in
    SODIUM_FOUND in src/CMakeLists.txt
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /Applications/Chia-arm/maxplotter/chia-plotter/build
    [  0%] Building CXX object CMakeFiles/test_disk_sort.dir/test/test_disk_sort.cpp.o
    [  0%] Built target arith_objs
    [  2%] Built target blstmp
    [  4%] Built target blake3
    clang: error: unsupported option '-fopenmp'
    [  6%] Built target fse
    make[2]: *** [CMakeFiles/test_disk_sort.dir/test/test_disk_sort.cpp.o] Error 1
    make[1]: *** [CMakeFiles/test_disk_sort.dir/all] Error 2
    make[1]: *** Waiting for unfinished jobs....
    Scanning dependencies of target chia_plotter
    [  6%] Building C object CMakeFiles/chia_plotter.dir/lib/chacha8.c.o
    [  7%] Linking C static library libchia_plotter.a
    [ 92%] Built target relic_s
    [ 92%] Built target chia_plotter
    make: *** [all] Error 2
    
    
    opened by AndyRPH 48
  • GitHub Actions Docker build & push

    GitHub Actions Docker build & push

    This automatically builds a Docker image every time you commit, and pushes it to Docker Hub with the commit hash as the tag and sets it to 'master'. Every time you create a GitHub release, it will tag that with its git tag and sets the 'latest' tag to it. Also, pull requests will trigger a build but will not push. (for obvious reasons.) See it in action here.

    You need to add some GitHub Secrets for it to work properly. Go to repo Settings, Secrets, and add DOCKERHUB_USERNAME and DOCKERHUB_TOKEN. After that, it will work properly.

    opened by Randomblock1 26
  • make: *** No targets specified and no makefile found.  Stop.

    make: *** No targets specified and no makefile found. Stop.

    I have an issue,it seens no makefile OS:ubuntu18.04 cmake version:3.20.3 ,this is CLI print:

    • The CXX compiler identification is unknown CMake Error at CMakeLists.txt:3 (project): No CMAKE_CXX_COMPILER could be found.

      Tell CMake where to find the compiler by setting either the environment variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path to the compiler, or to the compiler name if it is in the PATH.

    -- Configuring incomplete, errors occurred! See also "/chia-plotter/build/CMakeFiles/CMakeOutput.log". See also "/chia-plotter/build/CMakeFiles/CMakeError.log". make: *** No targets specified and no makefile found. Stop. [email protected]:/chia-plotter#

    cmake error log:

    Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed. Compiler: CMAKE_CXX_COMPILER-NOTFOUND Build flags: Id flags: --target=arm-arm-none-eabi;-mcpu=cortex-m3

    The output was: No such file or directory

    Can anyone solved it?

    regards!

    opened by haruhituki 22
  • Different bucket counts per phase

    Different bucket counts per phase

    From user TER on discord:

    Spec = AMD 5950x, 64GB RAM 3200mhz, 2TB Nvme SSD

    Setting 1: -r 16, 128 buckets Finish in 2116 seconds

    Setting 2: -r 32, 128 buckets Finish in 2162 seconds

    Setting 3: -r 20, 512 buckets Finish in 2462 seconds But interestingly, Phase 1 down from 1000~ seconds to 654 seconds. Phase 3 went from 650 seconds to 1576 seconds. So total time still the same.

    Setting 4 : -r 20, 128 buckets Finish in 2113 seconds

    I was wondering, what I did wrong in my Setting 3, as the Phase 1 improved by 40%, but Phase 3 dropped by 60%, which end up with the same total time.

    opened by madMAx43v3r 21
  • you may need a cast phase3.hpp

    you may need a cast phase3.hpp

    /Volumes/null/chia-plotter/include/chia/phase3.hpp:48:24: error: no matching function for call to 'min' const auto count = std::min(L_position - L_offset, L_buffer.size()); ^~~~~~~~ /usr/local/opt/llvm/bin/../include/c++/v1/algorithm:2594:1: note: candidate template ignored: deduced conflicting types for parameter '_Tp' ('unsigned long long' vs. 'unsigned long') min(const _Tp& __a, const _Tp& __b)

    correction seems to be: const auto count = std::min<unsigned long>(L_position - L_offset, L_buffer.size());

    opened by Hardcore-fs 21
  • Question: Does parallel plotting makes sense in this plotter?

    Question: Does parallel plotting makes sense in this plotter?

    Hi !

    First and foremost, I've run a few tests yesterday with your plotter. The speed is impressive! My rig takes ~6h to plot using Chia default plotter, and with your library, it took only 1h10min (6x faster).

    Anyway, during my tests yesterday I tried two scenarios:

    • Serial plotting
    • Parallel plotting

    While serial plotting, it took 1h10min, but while parallel plotting (with proper stagger minutes) it took 3h30min (running 3/4 plots at the same time)

    So, my question is, since we assume this plotter optimizes the resource usage. Does it make sense to try to parallel plot? In my simple test scenario, it seems that while parallel plotting the plot time increased at the same rate. So, IMHO, it seems that we optimize the plotting so much that parallel plotting doesn't make sense.

    I would like to know your opinion regarding this.

    opened by delucca 20
  • Default final destination should also toggle when -G option is used

    Default final destination should also toggle when -G option is used

    First, thank you for this fantastic plotter! Increased my time even over the optimized pechy version of chiapos by almost 50%!

    Now, my issue/request: When you do chia-plotter --help, it says: "-d, --finaldir arg Final directory (default = tmpdir)", but if you don't use the -d option, leaving it to default to tmpdir, and use the -G toggle to alternate tmpdir and tmpdir2 with each run, the final destination does not stay consistent with the currently running tmpdir, instead it stays as whatever the original tmpdir was.

    The reason this matters is, I'm running tmpdir on one NVME drive, and tmpdir2 on a separate NVME drive for better throughput. I toggle the tmpdirs so that the wear on the NVMEs is more even. But when the final copy takes place, I want the destination to be on the same device that the plot was produced (the current tmpdir), so that no actual copy needs to take place, it's just a rename. Instead, what I'm getting is that the first, third, fifth runs are kept on the same drive with an instant rename, while the second, fourth, sixth runs are forced to copy from what was my tmpdir2 device on the odd numbered runs to my odd-tmpdir device, adding a couple of minutes and unnecessary 100gb write to my every other plot. I have cron jobs set up to properly archive from both tmpdir and tmpdir2 when plots are found there, and everything would work beautifully if plots created when tmpdir2 alternated to become the tmpdir would stay where they are for the final copy, but the way it is, the completed plots are always forced to wind up in the original tmpdir.

    Thanks for any help!

    opened by Qwinn1 19
  • Ran 27ebfcb and it finnished in 5289s

    Ran 27ebfcb and it finnished in 5289s

    Just wanted let you know! Nice work! The server was rather busy at the time too so I guess I can get around 10% better time. Plotted on raid5 with 6 x 800gb enterprise sas ssd's. got nvme too but didn't run on them now.

    I'm not sure how to put in the pool key etc yet but I suppose that's coming :) In testnet its apparently possible to plot for pools now, I'm going to replot at soon as it get released and hopefully with your code instead.

    I was able to stack the original plotter and get one every 1h, I think I should be able to stack this 2 maybe 3 times :)

    (27ebfcbf0a47ee3f57ad48811e50d64db72676db)

    Phase 1 took 2176.96 sec
    real    36m17.068s
    user    344m11.493s
    sys     27m20.482s
    
    Phase 2 took 1610.95 sec
    real    26m51.067s
    user    105m8.773s
    sys     8m52.344s
    
    Phase 3 took 1371.42 sec, wrote 21877156374 entries to final plot
    real    22m51.629s
    user    137m53.501s
    sys     20m29.217s
    
    Phase 4 took 129.688 sec, final plot size is 108835549588 bytes
    real    2m9.796s
    user    9m27.843s
    sys     0m37.823s
    
    opened by Damme 17
  • Ubuntu 20.04 crash without error

    Ubuntu 20.04 crash without error

    Hello.

    I have a problem with the Mad Max plotter. I've been testing it for 6 days and it always fails. Crashes without registration error and in any phase. Also I tested with many combinations of parameters. I have the lastest commit from github.

    I have 3 PCs for plots. Specs: Ubuntu 20.04 32 Gb mem Intel I5 10400 (6c/12Th) Tmp drive: 2x1 TB NVMe WD 750 black (RAID 0) Dst drive: HDD 10 TB Seagate

    Logs

    PC 1: [P4] Finished writing C2 table Phase 4 took 63.1196 sec, final plot size is 108820004972 bytes Total plot creation time was 2761.72 sec (46.0287 min) Started copy to /media/dtanel/hdd1plot/plot-k32-2021-06-15-17-42-ce5581aa1b3d7cefbff21e297d515f6f3f66529abd288df0abf2d4>Crafting plot 5 out of 6 Process ID: 27037 Number of Threads: 10 Number of Buckets: 2^7 (128) Pool Public Key: 928a308d319b9144c66258e504afc5513e0af5da995b4a805fe1a2dca092e37d162d4402e95ac9cf06701d1dfc2654a6 Farmer Public Key: 8f81d7a136514c4e2445573387623c3bf13c7610d963e39815ecb0bda98c5ffc95e627ab3e642f155d0f85c4ca12e1f8 Working Directory: /mnt/ssdtemp/ Working Directory 2: /mnt/ssdtemp/ Plot Name: plot-k32-2021-06-15-18-28-9cfc1805abf1ef8eb3377fdc389c03912b0647f4e40e56645bd79f1ed2a7d552 [P1] Table 1 took 23.6587 sec [P1] Table 2 took 198.889 sec, found 4295039052 matches

    PC 2: Number of Plots: 2 Crafting plot 1 out of 2 Process ID: 3252846 Number of Threads: 12 Number of Buckets: 2^7 (128) Pool Public Key: 928a308d319b9144c66258e504afc5513e0af5da995b4a805fe1a2dca092e37d162d4402e95ac9cf06701d1dfc2654a6 Farmer Public Key: 8f81d7a136514c4e2445573387623c3bf13c7610d963e39815ecb0bda98c5ffc95e627ab3e642f155d0f85c4ca12e1f8 Working Directory: /mnt/ssdtemp/ Working Directory 2: /mnt/ssdtemp/ Plot Name: plot-k32-2021-06-15-18-00-7544723d5d6dbebafff23b862445a2768805cb1d6c8b33cc7f9f8f4db3497f07 [P1] Table 1 took 20.3757 sec [P1] Table 2 took 187.388 sec, found 4295043619 matches [P1] Table 3 took 216.747 sec, found 4295123903 matches [P1] Table 4 took 265.005 sec, found 4295096255 matches [P1] Table 5 took 260.14 sec, found 4295125307 matches [P1] Table 6 took 249.603 sec, found 4295090098 matches [P1] Table 7 took 195.129 sec, found 4294939112 matches Phase 1 took 1394.51 sec [P2] max_table_size = 4295125307 [P2] Table 7 scan took 15.5494 sec [P2] Table 7 rewrite took 32.9628 sec, dropped 0 entries (0 %) [P2] Table 6 scan took 39.874 sec [P2] Table 6 rewrite took 67.0465 sec, dropped 581374701 entries (13.5358 %) [P2] Table 5 scan took 38.4818 sec [P2] Table 5 rewrite took 63.9578 sec, dropped 762078716 entries (17.7429 %) [P2] Table 4 scan took 38.0322 sec [P2] Table 4 rewrite took 63.7838 sec, dropped 828922210 entries (19.2993 %) [P2] Table 3 scan took 37.7558 sec [P2] Table 3 rewrite took 62.6074 sec, dropped 855189544 entries (19.9107 %) [P2] Table 2 scan took 37.6382 sec [P2] Table 2 rewrite took 63.1996 sec, dropped 865585616 entries (20.1531 %) Phase 2 took 568.038 sec Wrote plot header with 268 bytes [P3-1] Table 2 took 51.7436 sec, wrote 3429458003 right entries [P3-2] Table 2 took 44.3415 sec, wrote 3429458003 left entries, 3429458003 final [P3-1] Table 3 took 67.7006 sec, wrote 3439934359 right entries [P3-2] Table 3 took 46.3265 sec, wrote 3439934359 left entries, 3439934359 final

    PC 3: Crafting plot 5 out of 6 Process ID: 61559 Number of Threads: 10 Number of Buckets: 2^8 (256) Pool Public Key: 928a308d319b9144c66258e504afc5513e0af5da995b4a805fe1a2dca092e37d162d4402e95ac9cf06701d1dfc2654a6 Farmer Public Key: 8f81d7a136514c4e2445573387623c3bf13c7610d963e39815ecb0bda98c5ffc95e627ab3e642f155d0f85c4ca12e1f8 Working Directory: /mnt/ssdtemp/ Working Directory 2: /mnt/ssdtemp/ Plot Name: plot-k32-2021-06-15-18-31-3c67e66c7b417e01e54c1cdbacc9e1ff4805e5171c68839ad2fb73949841f204 [P1] Table 1 took 24.6473 sec

    Can you help me? Thank you

    opened by diegotanel 16
  • the -K option

    the -K option

    the -K option is not always applied when running phase 2 in a rewrite cycle (even subphases). restarting the program sometimes helps. I can not clearly catch the cause of this bug. if the parameter works, then rewriting the tables takes about twice as long as scanning. if the parameter is ignored, then 6 times more. at the same time, in terms of processor load, I observe a significant decrease in employment.

    Below are examples of launch and logs:

    ===============================

    C:\Users\GMND\AppData\Local\chia-blockchain\app-1.5.0\resources\app.asar.unpacked\daemon\madmax\chia_plot.exe -k 32 -x 8444 -n -1 -r 8 -u 256 -v 256 -K 4 -t D:\TEMP\ -2 R:\ -d D:\CHIA_NFT_PLOTS\FARM_1\ -c XXX -f XXX

    ==============SLOW================= [P2] max_table_size = 4294967296 [P2] Table 7 scan took 10.4885 sec [P2] Table 7 rewrite took 85.6975 sec, dropped 0 entries (0 %) [P2] Table 6 scan took 24.8612 sec [P2] Table 6 rewrite took 163.096 sec, dropped 581566470 entries (13.5478 %) [P2] Table 5 scan took 24.2158 sec [P2] Table 5 rewrite took 155.927 sec, dropped 762292550 entries (17.7534 %) [P2] Table 4 scan took 24.3306 sec [P2] Table 4 rewrite took 153.72 sec, dropped 829137030 entries (19.3073 %) [P2] Table 3 scan took 23.8456 sec [P2] Table 3 rewrite took 153.344 sec, dropped 855283939 entries (19.9147 %) [P2] Table 2 scan took 24.0242 sec [P2] Table 2 rewrite took 152.228 sec, dropped 865677337 entries (20.156 %) Phase 2 took 998.292 sec

    =============SPEED================== [P2] max_table_size = 4295026397 [P2] Table 7 scan took 10.5786 sec [P2] Table 7 rewrite took 27.3296 sec, dropped 0 entries (0 %) [P2] Table 6 scan took 25.0182 sec [P2] Table 6 rewrite took 47.7223 sec, dropped 581323905 entries (13.5349 %) [P2] Table 5 scan took 24.3874 sec [P2] Table 5 rewrite took 44.4134 sec, dropped 761974286 entries (17.741 %) [P2] Table 4 scan took 23.8291 sec [P2] Table 4 rewrite took 43.9112 sec, dropped 828893369 entries (19.2989 %) [P2] Table 3 scan took 24.3002 sec [P2] Table 3 rewrite took 43.5669 sec, dropped 855091557 entries (19.9091 %) [P2] Table 2 scan took 23.9598 sec [P2] Table 2 rewrite took 45.9016 sec, dropped 865614536 entries (20.154 %) Phase 2 took 387.812 sec

    opened by GMND 0
  • add fixed (1/512) Filter index

    add fixed (1/512) Filter index

    Had no idea how to describe this in a proper way so here is the background: Although I know spindowns of harddisks is a controversial topic: I want to let my chia harddisks spin down. Af first I was thinking of making k39 plots, but that's not really feasible. Therefore I want to put (k33 or k34) plots with the same filter index on the same drive. By doing so the disks have a lot of time to spin down.

    So here is the real question: Is it possible to create plots with a constant filter index? So I can choose on beforehand?

    opened by kiss81 7
  • K33 and K34 all [P2] Table x rewrites are Single Core

    K33 and K34 all [P2] Table x rewrites are Single Core

    AMD 5950x 128GB 3600 2x1.6TB PM1735 Clear Linux kernel 5.12 I have problem with k33 and k34 plots. As i observed the main difference starts from [P2] Table 6 rewrite If it goes SLOW than, as per Htop, all [P2] Table x rewrites are Single Core 100% and other cores 10-20% [P2] max_table_size = 8591960036 [P2] Table 7 scan took 37.1078 sec [P2] Table 7 rewrite took 62.7559 sec, dropped 0 entries (0 %) [P2] Table 6 scan took 43.7843 sec [P2] Table 6 rewrite took 346.423 sec, dropped 1162409851 entries (13.5306 %) [P2] Table 5 scan took 42.7258 sec [P2] Table 5 rewrite took 317.952 sec, dropped 1523678697 entries (17.737 %) [P2] Table 4 scan took 43.0958 sec [P2] Table 4 rewrite took 315.809 sec, dropped 1657525624 entries (19.2956 %) [P2] Table 3 scan took 41.9618 sec [P2] Table 3 rewrite took 320.874 sec, dropped 1710002810 entries (19.9067 %) [P2] Table 2 scan took 41.8325 sec [P2] Table 2 rewrite took 320.281 sec, dropped 1731051201 entries (20.1521 %) Phase 2 took 1951.04 sec If it goes FAST than all cores are 70-90% used [P2] max_table_size = 8591485063 [P2] Table 7 scan took 27.2468 sec [P2] Table 7 rewrite took 56.8793 sec, dropped 0 entries (0 %) [P2] Table 6 scan took 45.1812 sec [P2] Table 6 rewrite took 90.4128 sec, dropped 1162401340 entries (13.5309 %) [P2] Table 5 scan took 43.7353 sec [P2] Table 5 rewrite took 86.8796 sec, dropped 1523755776 entries (17.7379 %) [P2] Table 4 scan took 42.9656 sec [P2] Table 4 rewrite took 85.4762 sec, dropped 1657598359 entries (19.2965 %) [P2] Table 3 scan took 42.7092 sec [P2] Table 3 rewrite took 84.9902 sec, dropped 1710037887 entries (19.9072 %) [P2] Table 2 scan took 42.5686 sec [P2] Table 2 rewrite took 84.6988 sec, dropped 1731088017 entries (20.1523 %) Phase 2 took 750.125 sec per final i have difference Total plot creation time was 4306.52 sec (71.7754 min) and Total plot creation time was 2983.01 sec (49.7169 min) Hope it helps to find the problem. Best regards

    opened by Grosik1577 2
  • CentOS 9 Stream compile problem

    CentOS 9 Stream compile problem

    When issuing the command, "./make_devel.sh" I get the following error:

    
    -- The C compiler identification is unknown
    -- The CXX compiler identification is unknown
    -- The ASM compiler identification is unknown
    -- Didn't find assembler
    CMake Error at CMakeLists.txt:12 (project):
      No CMAKE_C_COMPILER could be found.
    
      Tell CMake where to find the compiler by setting either the environment
      variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
      the compiler, or to the compiler name if it is in the PATH.
    
    opened by brianwfreeman 0
  • error: call to non-‘constexpr’ function ‘long int sysconf(int)’

    error: call to non-‘constexpr’ function ‘long int sysconf(int)’

    Cannot compile code on Fedora 35, v33 was just fine.

    gcc --version gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9) Copyright (C) 2021 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

    cmake --version cmake version 3.22.2 CMake suite maintained and supported by Kitware (kitware.com/cmake).

    make --version GNU Make 4.3 Built for x86_64-redhat-linux-gnu Copyright (C) 1988-2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

    git log commit ecec17d25cd547fa4bb64b2eb7455b831c8a2882 (HEAD -> master, origin/master, origin/HEAD) Merge: efa5b2a 70bb67f Author: Max [email protected] Date: Fri Feb 11 19:27:08 2022 +0100

    Merge pull request #988 from risner/stage-option
    
    Add -s stagedir option
    

    [100%] Linking CXX executable runbench In file included from /usr/include/signal.h:328, from /opt/madMAx43v3r/chia-plotter/lib/bls-signatures/src/../contrib/catch/catch.hpp:8034, from /opt/madMAx43v3r/chia-plotter/lib/bls-signatures/src/test.cpp:20: /opt/madMAx43v3r/chia-plotter/lib/bls-signatures/src/../contrib/catch/catch.hpp:10822:58: error: call to non-‘constexpr’ function ‘long int sysconf(int)’ 10822 | static constexpr std::size_t sigStackSize = 32768 >= MINSIGSTKSZ ? 32768 : MINSIGSTKSZ; | ^~~~~~~~~~~ In file included from /usr/include/bits/sigstksz.h:24, from /usr/include/signal.h:328, from /opt/madMAx43v3r/chia-plotter/lib/bls-signatures/src/../contrib/catch/catch.hpp:8034, from /opt/madMAx43v3r/chia-plotter/lib/bls-signatures/src/test.cpp:20: /usr/include/unistd.h:640:17: note: ‘long int sysconf(int)’ declared here 640 | extern long int sysconf (int __name) __THROW; | ^~~~~~~ compilation terminated due to -fmax-errors=1. make[2]: *** [lib/bls-signatures/src/CMakeFiles/runtest.dir/build.make:76: lib/bls-signatures/src/CMakeFiles/runtest.dir/test.cpp.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:895: lib/bls-signatures/src/CMakeFiles/runtest.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... [100%] Built target runbench [100%] Linking CXX executable UnitTests_bech32 [100%] Linking CXX executable test_phase_3 [100%] Linking CXX executable test_phase_1 [100%] Built target test_phase_3 [100%] Built target test_phase_1 [100%] Built target UnitTests_bech32 [100%] Linking CXX executable chia_plot_k34 [100%] Linking CXX executable chia_plot [100%] Built target chia_plot_k34 [100%] Built target chia_plot make: *** [Makefile:136: all] Error 2

    compile.log

    opened by xorinox 1
A Pipelined RISC-V RV32I Core in Verilog

Toast-RV32i Toast is a RISC-V soft core written in Verilog that implements a subset of the RV32I ISA version 2.2. Documentation: https://toast-core.re

George Yu 25 Jul 20, 2022
A cross platform shader language with multi-threaded offline compilation or platform shader source code generation

A cross platform shader language with multi-threaded offline compilation or platform shader source code generation. Output json reflection info and c++ header with your shaders structs, fx-like techniques and compile time branch evaluation via (uber-shader) "permutations".

Alex Dixon 275 Jul 18, 2022
V-I Diagram Plotter using SerialPort in C#

V-I Diagram Plotter using SerialPort in C# This application was used for a research on a memristive device which was published in this paper in 2016:

Meysam Parvizi 2 Feb 10, 2022
A demonstration of various different techniques for implementing 'threaded code,' a technique used in Forth and in virtual machines like the JVM.

Threaded code is a technique used in the implementation of virtual machines (VMs). It avoids the overhead of calling subroutines repeatedly by 'thread

null 24 Jul 24, 2022
PlotFS is a fuse filesystem for efficient storage of Chia plot files.

PlotFS PlotFS is a fuse filesystem for efficient storage of Chia plot files. PlotFS is not a traditional filesystem. It is mounted read only for farmi

Matthew Szatmary 46 Jul 27, 2022
Multi-dimensional dynamically distorted staggered multi-bandpass LV2 plugin

B.Angr A multi-dimensional dynamicly distorted staggered multi-bandpass LV2 plugin, for extreme soundmangling. Based on Airwindows XRegion. Key featur

null 18 Apr 17, 2022
Unicorn is a lightweight, multi-platform, multi-architecture CPU emulator framework, based on QEMU.

Unicorn Engine Unicorn is a lightweight, multi-platform, multi-architecture CPU emulator framework, based on QEMU. Unicorn offers some unparalleled fe

lazymio 1 Nov 7, 2021
multi-sdr-gps-sim generates a IQ data stream on-the-fly to simulate a GPS L1 baseband signal using a SDR platform like HackRF or ADLAM-Pluto.

multi-sdr-gps-sim generates a GPS L1 baseband signal IQ data stream, which is then transmitted by a software-defined radio (SDR) platform. Supported at the moment are HackRF, ADLAM-Pluto and binary IQ file output. The software interacts with the user through a curses based text user interface (TUI) in terminal.

null 57 Jul 23, 2022
C/C++ language server supporting multi-million line code base, powered by libclang. Emacs, Vim, VSCode, and others with language server protocol support. Cross references, completion, diagnostics, semantic highlighting and more

Archived cquery is no longer under development. clangd and ccls are both good replacements. cquery cquery is a highly-scalable, low-latency language s

Jacob Dufault 2.3k Jul 27, 2022
Robust multi-prompt delimited control and effect handlers in C/C++

libmprompt Note: The library is under development and not yet complete. This library should not be used in production code. Latest release: v0.2, 2021

Koka Language and Related Tools 85 Jul 27, 2022
FastPath_MP: An FPGA-based multi-path architecture for direct access from FPGA to NVMe SSD

FastPath_MP Description This repository stores the source code of FastPath_MP, an FPGA-based multi-path architecture for direct access from FPGA to NV

Beehive lab 19 Jun 15, 2022
the ares multi-system console emulation suite

ares ares is a multi-system emulator that began development on October 14th, 2004. It is a descendent of higan and bsnes, and focuses on accuracy and

higan 315 Aug 3, 2022
CERBERUS 2080â„¢, the amazing multi-processor 8-bit microcomputer

CERBERUS 2080â„¢ CERBERUS 2080â„¢, the amazing multi-processor 8-bit microcomputer: a fully open-source project available for anyone to peruse, build, mod

The Byte Attic 66 Jun 22, 2022
Multi-GNSS Precise Point Postioning with Ambiguity Resolution

This is demo for multi-GNSS precise point positioning with ambiguity resolution (PPP-AR), which is based on RTKLIB and RTKLIB_demo5. FEATURES ppp-ar w

Chen Chao 6 Jun 28, 2022
A Multi-sensor Fusion Odometry via Smoothing and Mapping.

LVIO-SAM A multi-sensor fusion odometry, LVIO-SAM, which fuses LiDAR, stereo camera and inertial measurement unit (IMU) via smoothing and mapping. The

Xinliang Zhong 139 Aug 5, 2022
Multi Tool for BFBB

Bubble Buddy A Multi Tool for BFBB ( Currently only a save editor but HIPHOP tools are planned :) ) How to use Download bubblebuddy.exe here. Use the

Will 4 Jul 25, 2022
Online multi-agent trajectory planner using linear safe corridor (LSC)

lsc_planner This package presents an efficient multi-agent trajectory planning algorithm which generates safe trajectories in obstacle-dense environme

Jungwon Park 24 Aug 1, 2022
Multi watch-face solution for Watchy

watchy999 A Multi watch-face solution for Watchy What started out as a way to have more than one watch face on the Watchy by SQFMI, has turned into so

null 27 Jun 13, 2022
A simple C++ library with multi language interfaces (Java, NodeJS, Python...)

cpp-to-x A simple C++ library with multi language interfaces (Java, NodeJS, Python...) Why This is just a learning experiment to see how you can write

James Roberts 1 Nov 25, 2021