🎩 Command-line itch.io helper

Overview

butler

Build Status codecov Go Report Card MIT licensed

butler is the itch.io command-line tools - all by itself.

It is used by:

  • Content creators on itch.io to push builds quickly & reliably
  • the itch app for some network, filesystem and patching operations

Documentation

Documentation for butler is available as a Gitbook:

Questions about butler are welcome on its Issue tracker, or, if the matter is private, itch.io support.

Integrations

The following projects integrate butler as part of their workflow:

  • itchy-electron lets you package your Electron games for itch.io and upload them there
  • gradle-butler-plugin is a Gradle plugin for automatically installing, updating, and running butler as part of your build.

Authors

butler was mostly written by Amos Wenger, but wouldn't have been possible without the work of many before him.

Amos would like to thank in particular Leaf Corcoran, JesΓΊs Higueras and TomΓ‘Ε‘ Duda.

License

butler is released under the MIT License. See the LICENSE file for details.

Additional licenses

While butler built from source is fully MIT-licensed, some components it can use at runtime (if present) have other licenses:

  • The 7-zip decompression engine (the github.com/itchio/boar/szextractor package) opens dynamic libraries for libc7zip, and 7-zip, which have components licensed under the LGPL 2.1 license and the MPL 2.0 license, along with specific terms for the RAR extraction code.
Issues
  • Fix wharf's resumable upload client behavior + collect error traces in case of service degradation

    Fix wharf's resumable upload client behavior + collect error traces in case of service degradation

    I'm getting the following error while trying to push new version of a file.

    $ butler push linux quyse/flaw:model-editor-linux
    > For channel `model-editor-linux`: last build is 3183, downloading its signature
    
    > Pushing 8.0 MB (1 files, 0 dirs, 0 symlinks)
    *errors.errorString itch.io API error: uploaded file does not exist in storage (build/4513/13440-signature-default: 404)
    /usr/local/go/src/runtime/asm_amd64.s:1998 (0x490431)
    

    What does it possibly mean? I've recently upgraded docker container which is pushing builds during CI, but I don't think I've changed anything about butler.

    infrastructure 
    opened by quyse 91
  • Upload arbitrary files?

    Upload arbitrary files?

    Is there a way to upload an arbitrary file (like an .apk or a .tar.bz2) to a channel using butler? Even if it doesn't get the nice speed boost from the compression, having such a feature would be nice, as it would allow tools (such as Ren'Py) to use butler for all upload needs.

    opened by renpytom 18
  • Detect .zip and .tar.gz by file extension, add --single-file option

    Detect .zip and .tar.gz by file extension, add --single-file option

    This is my proposed solution for #51

    It seems silly to force people to create directories with only one file in them if they just want to upload one file.

    At the same time, I don't feel comfortable adding metadata such as "this should actually end up being a compressed zip file with a custom file extension" β€” wharf works on folders/file hierarchies.

    It looks like diffing .jars (also zip-based) doesn't yield too bad results? http://myrealitycoding.github.io/2016/05/31/deploy-to-itch.html β€” if someone has subsequent builds of a .love game, I'd be curious to try diffing them.

    opened by fasterthanlime 14
  • How about signature checks after

    How about signature checks after "butler cp" is done?

    My current game has a "role-your-own" type of game update download system (also with patches based on the rsync algorithm), and while this works, I really want to phase it out and simply not support it anymore. So I'm just browsing around the code a little, trying to see if I can wrap my own launcher around butler. Ultimately I want to have my game up in the itch.io client, but also in my own launcher just for this particular game, basically wrapping some of the download/patch functionality itch.io client would.

    That said, from my experience with thousands of players having downloaded my games over multiple iterations of the update system, I'm curious why butler doesn't perform any integrity checks post-download? I actually tested this by running "butler cp --resume", cancel it halfway through, purposely mess up the file, then continue downloading. As silly as this may seem, this is an actual problem that occurs a lot!

    I think butler would benefit from having a sort of signature check post-download (like during verify/heal), and then, when applicable, try to re-download the parts that got corrupted. Especially in my case where the game comes closed to 700MB in zipped form, having a corrupted zip on your hands and having to redownload the whole thing and hope for the best is problematic for a lot of people.

    While this is probably easily added to butler itself, the itch.io API would obviously need to be updated to support this.

    opened by LennardF1989 13
  • Set a display name in butler push commands

    Set a display name in butler push commands

    One thing I'd like to be able to do on butler commands is change the display name of updated builds (the ones that display here):

    image

    It would be nice to be able to do something like this in my upload batch files:

    butler push ../builds/windows64/ terrycavanagh/dicey-dungeons:windows64 --userversion %VERSIONNUM% --displayname "Dicey Dungeons v%VERSIONNUM% Windows"

    (Again, apologies if this is already possible and I just missed it, am new to Butler and couldn't find it in the documentation!)

    opened by TerryCavanagh 12
  • Update external file downloads through Butler

    Update external file downloads through Butler

    In addition to my two regular downloads, I have a source download on my game page that points to current release version's tag on Bitbucket (e.g. https://bitbucket.org/philomory/ld38/get/0.2.0.zip). It would be nice to be able to use butler to update that external file link to point to the correct version when that changes.

    opened by philomory 8
  • "invalid memory address or nil pointer dereference"

    I've been encountering this issue lately on Windows 10. Has been working for a year without it. This is the output I get in verbose mode:

    butler.exe diff \\server\SomeFolder \\server\SomeOtherFolder \\server\patch.pwr -v
    βˆ™ Hashing \\server\SomeFolder
    β–β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œ   0.00%                                                  panic: runtime error: invalid memory address or nil pointer dereference
    [signal 0xc0000005 code=0x0 addr=0x48 pc=0x880a00]
    
    goroutine 1 [running]:
    github.com/itchio/wharf/pwr.ComputeSignatureToWriter(0x121d800, 0xc00002a118, 0x0, 0x121dc00, 0xc000128100, 0xc000737260, 0xc00012c120, 0x0, 0x0)
            C:/Users/amos/go/pkg/mod/github.com/itchio/[email protected]/pwr/sign.go:54 +0x120
    github.com/itchio/wharf/pwr.ComputeSignature(0x121d800, 0xc00002a118, 0x0, 0x121dc00, 0xc000128100, 0xc000737260, 0x0, 0x11efc40, 0x8, 0x20, ...)
            C:/Users/amos/go/pkg/mod/github.com/itchio/[email protected]/pwr/sign.go:30 +0xcb
    github.com/itchio/butler/cmd/diff.Do(0xc00029eec0, 0x0, 0x0)
            C:/Multi-Runner/builds/58469115/0/itchio/butler/cmd/diff/diff.go:134 +0x18ba
    github.com/itchio/butler/cmd/diff.do(0xc000170900)
            C:/Multi-Runner/builds/58469115/0/itchio/butler/cmd/diff/diff.go:61 +0xdf
    main.doMain(0xc00003a090, 0x5, 0x7)
            C:/Multi-Runner/builds/58469115/0/itchio/butler/main.go:264 +0xb66
    main.main()
            C:/Multi-Runner/builds/58469115/0/itchio/butler/main.go:136 +0x69
    
    • Tried reinstalling Butler (v15.17.0), same error.
    • It works on other content though that's sitting on \some-other-server.
    • I checked and the machine does have read and write access to the relevant places in \server.
    • Tried running the same patch from several machines on the same network, same problem everywhere. All machines have sufficient ram and disk space.
    opened by ghost 7
  • Missing prereqs

    Missing prereqs

    I was trying to set net-4.7.1 on the itch manifest and was getting Unknown prerequisite listed: net-4.7. After digging in the source code of butler, I found this URL https://broth.itch.ovh/itch-redists/info/LATEST/unpacked here https://github.com/itchio/butler/blob/544e1b8609f8dab386cc67cd06bf999e47029b28/cmd/validate/validate.go#L261

    The JSON returned when querying that URL contains net-4.5.2, net-4.6 and net-4.6.2, but not net-4.7 or net-4.7.1.

    Is this correct? Is there currently a way to set net-4.7.1 on the manifest? In case if not, would it be possible to update the list to get it available?

    opened by srodrigo 7
  • Stuck at 50% when uploading the first build of a package

    Stuck at 50% when uploading the first build of a package

    OS: Windows 10, Version 1809 (64-bit) Butler version v14.13.0, built on Sep 27 2018 @ 18:57:19, ref 4dbdb0829e70b9987193b13c8772204ec06cdfc9

    When I try to upload the first build of a package (example: linux-cli), butler starts uploading, then swiftly goes to 50% (all the while, showing no network activity) and gets stuck there.

    butler -v output
    βˆ™ For channel `linux-cli`: pushing first build
    Created signature build file: &{ID:416240 UploadURL:http://commondatastorage.googleapis.com/itchio/build/110516/[email protected].gserviceaccount.com&Expires=1539291742&Signature=Lmxq34UHpVjWwoKPr%2FEPTFvD4bBtJ8p8pQMnUyJgRVqcJCJV5WHPRzVEqqe3SQTP0QvVS7yppvkAu4O5wovAoIt61FCAEgUGeynsIsDBRA%2Bqy0p0i4wzvZ9mAmnhjcqg35eBPaKf%2FJ80mbM7Ru%2FfGji1E%2FGC%2FhGDUS4Q5oymg7LDbBBzEEaM0xLcI3XjDzVKxPGWbsNS%2B6pb1ZkHiGl0rn%2F%2FmQYYdNQisl%2FPwwYk8SewT3TxIlGKkS9DdyBmDP0PI1NnLbuxH7z0GRP6W13Km9WM%2FGBTpfjQ1VqfKaYtnxbeU%2BE35B4NwIP2U6sufhS3RzO%2BZVsVbgjNcOhfBBLJmA== UploadParams:map[] UploadHeaders:map[Content-Disposition:attachment Origin:* x-goog-resumable:start]}
    Created patch build file: &{ID:416241 UploadURL:http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ== UploadParams:map[] UploadHeaders:map[Content-Disposition:attachment Origin:* x-goog-resumable:start]}
    Started resumable upload session http://commondatastorage.googleapis.com/itchio/build/110516/[email protected].gserviceaccount.com&Expires=1539291742&Signature=Lmxq34UHpVjWwoKPr%2FEPTFvD4bBtJ8p8pQMnUyJgRVqcJCJV5WHPRzVEqqe3SQTP0QvVS7yppvkAu4O5wovAoIt61FCAEgUGeynsIsDBRA%2Bqy0p0i4wzvZ9mAmnhjcqg35eBPaKf%2FJ80mbM7Ru%2FfGji1E%2FGC%2FhGDUS4Q5oymg7LDbBBzEEaM0xLcI3XjDzVKxPGWbsNS%2B6pb1ZkHiGl0rn%2F%2FmQYYdNQisl%2FPwwYk8SewT3TxIlGKkS9DdyBmDP0PI1NnLbuxH7z0GRP6W13Km9WM%2FGBTpfjQ1VqfKaYtnxbeU%2BE35B4NwIP2U6sufhS3RzO%2BZVsVbgjNcOhfBBLJmA==&upload_id=AEnB2UqmuT6n03y9yzGmv2_DCsZrWeStrGBZO-PcwQoebBo_wP9tR1tbX9qQLX93T2LpzRblUpwxuuupTdIJP2VnEDLLnPzpug
    Started resumable upload session http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ
    Launching patch & signature channels
    Waiting for source container
    βˆ™ Pushing 70.61 MiB (185 files, 1 dirs, 0 symlinks)
    Building diff context
    ▐░                  β–Œ   0.00%         - network idle, 67.57 MiB left           [ru-0] Uploading 4 chunks
    ▐░░                 β–Œ   0.00%         - network idle, 67.75 MiB left           [cu-0] β†’ Uploading 0-1048575 (more to come)
    β–β–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘      β–Œ   2.18%   9m48s - network idle, 44.87 MiB left           [cu-0] ← 308 Resume Incomplete (in 12.0913113s)
    [cu-0] βœ“ Commit succeeded (4 blocks stored @ 84.69 KiB / s)
    [ru-0] Uploading 64 chunks
    [cu-0] β†’ Uploading 1048576-17825791 (more to come)
    β–β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œ  55.70%     13s - network idle, 13.52 MiB left          [ru-1] Uploading last 1 chunks
    [cu-1] β†’ Uploading 0-42307 (final slice)
    β–β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œ  55.64%     25s - network idle, 13.55 MiB left          [cu-1] ← 200 OK (in 14.7477616s)
    [cu-1] βœ“ 41.32 KiB upload complete!
    β–β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œ  55.64%   1m22s - network idle, 13.55 MiB left          [cu-0] while uploading 1048576-17825791: 
    Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    network error: Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    Sleeping 0 seconds then retrying
    [cu-0] β†’ Uploading 1048576-17825791 (more to come)
    β–β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œ  55.64%   2m34s - network idle, 13.55 MiB left          [cu-0] while uploading 1048576-17825791: 
    Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    network error: Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    Sleeping 1 seconds then retrying
    [cu-0] β†’ Uploading 1048576-17825791 (more to come)
    β–β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œ  55.64%   3m49s - network idle, 13.55 MiB left          [cu-0] while uploading 1048576-17825791: 
    Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    network error: Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    Sleeping 4 seconds then retrying
    [cu-0] β†’ Uploading 1048576-17825791 (more to come)
    β–β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œ  55.64%    5m6s - network idle, 13.55 MiB left          [cu-0] while uploading 1048576-17825791: 
    Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    network error: Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    Sleeping 9 seconds then retrying
    [cu-0] β†’ Uploading 1048576-17825791 (more to come)
    β–β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œ  55.64%   6m27s - network idle, 13.55 MiB left          [cu-0] while uploading 1048576-17825791: 
    Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    network error: Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    Sleeping 16 seconds then retrying
    [cu-0] β†’ Uploading 1048576-17825791 (more to come)
    β–β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œ  55.64%   7m54s - network idle, 13.55 MiB left          [cu-0] while uploading 1048576-17825791: 
    Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    network error: Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    Sleeping 25 seconds then retrying
    [cu-0] β†’ Uploading 1048576-17825791 (more to come)
    β–β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–Œ  55.64%   9m26s - network idle, 13.55 MiB left          [cu-0] while uploading 1048576-17825791: 
    Put http://commondatastorage.googleapis.com/itchio/build/110516/4[email protected]viceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    network error: Put http://commondatastorage.googleapis.com/itchio/build/110516/[email protected]rviceaccount.com&Expires=1539291742&Signature=Z2GC3QOfN7DA91yNinln8hl4iw6dGvQfvu7h23Qae5XpswekdBMka%2BsiAB3me4PKqXbWC826SRHKLSmAWaTuLXUwZ%2FEcZhKoYaX8fONBPP%2FGTfLFKwEX8%2FqtuH3cTyv4CrLjvVcJ0yQYVKXs3YNK1Z%2BWm2Vn1W%2Bm5gfC3b7OLU%2BXY7dpqHoI38N4N04Dm1g8R2rqDPm3sfJOed2Ka7KgMg93hPSmEc48UkC2Kj2jrb9ybc6ZH3mXv6ldjicNLzJzHv8hDZwdtHIk2oKKZiCufMOYMVujionmUVx%2Brro1W%2BbVa7VVcBItbNnJ155k3Gg232uRWCkK5%2B1xjqXBDqotUQ==&upload_id=AEnB2UqvFg717hiN_KikGmGfUDZ_QoLst0B6fOAuG-JdwznkpcrZm8WGQ8F2eO1_d4lkcwbTtozljONC_lq1V1Ci-opwZvDXSQ: EOF
    Sleeping 36 seconds then retrying
    (This is where I stopped the upload)
    
    opened by FirehawkV21 7
  • Beeps? in this day and age?

    Beeps? in this day and age?

    Here's a test executable that, for me, on Windows 10:

    • beeps once in mintty (msys2 shell)
    • beeps twice in cmd.exe

    The third beep is never heard - this is what butler uses as a prefix for tasks.

    Attached is beeptest.exe, signed with our certificate. I'm curious how it behaves on Windows 7!

    beeptest.zip

    opened by fasterthanlime 7
  • Backend problem: GCS occasionally closes connection immediately?

    Backend problem: GCS occasionally closes connection immediately?

    While processing a build:

    Get (snipped): EOF
    git.itch.ovh/itchio/wharfd/vendor/github.com/itchio/wharf/pools/cachepool.(*CachePool).Preload
    	/home/amos/goprod/src/git.itch.ovh/itchio/wharfd/vendor/github.com/itchio/wharf/pools/cachepool/cachepool.go:73
    git.itch.ovh/itchio/wharfd/jobs.DoApplyTask.func2
    	/home/amos/goprod/src/git.itch.ovh/itchio/wharfd/jobs/task_apply.go:197
    runtime.goexit
    	/usr/local/go/src/runtime/asm_amd64.s:2361
    Could not preload file 796 (res/music/FM1_nodrums.wav)
    
    opened by fasterthanlime 6
Releases(v15.21.0)
  • v15.21.0(May 12, 2021)

    • Bump up default context deadline to 15 seconds to help with slower connections not allowing builds to be pushed
    • Add flag "--context-timeout" for setting the timeout to any value
    • Butler is now signed with itch corp. certificate on macOS and Windows
    Source code(tar.gz)
    Source code(zip)
  • v1.0.1(Oct 22, 2016)

  • v1.0.0(Oct 22, 2016)

    v1.0.0 introduces many new concepts in butler, @elisee asked me what was new so, here it goes:

    Manifests (intro)

    One big feature is dividing containers in blocks. Up till then we had "archives" (containers compressed as a .zip file), "signatures" (MD5 hashes of <=16KB blocks), and "patches" (list of instructions).

    This release introduces a new file format, "manifests", which contains SHAKE256 32-bit hashes of 4MB blocks. They're supported by the "ls" and "file" command, and most of the code writing or reading it is available in the blockpool package of wharf.

    There are no new user-facing commands in butler allowing the creation of manifests, this will come in later releases, more details in the later sections.

    Healing

    Another big feature is healing: given a signature and a heal source, the verify command can now redownload parts of files that were corrupted. It'll also restore missing directories and missing/corrupted symlinks.

    Note that when --heal is used, the exit code behavior of verify is modified: it'll now return 0 if it healed everything successfully, whereas without --heal it would've returned 1.

    Plans: The apply command will also get a --heal parameter, so that if patch application fails for a reason or another, it'll be able to heal the result, instead of forcing the butler user to fall back to a full re-download.

    itchfs

    For healing to be interesting, the heal source must be remote β€” and yet verify's --heal parameter accepts a local path.

    However, butler now accepts, for all path arguments, itchfs:///XXX urls. All valid URLs can be devised from the sources.go file of go-itchio.

    Note that the API key is passed as a query parameter.

    • Example URL: itchfs:///upload/3/download?api_key=XXX

    (Yes, that's three slashes - the third is the root, much like file:///etc/hosts)

    This means, when an itchfs is passed to the --heal parameter of the verify command, butler will only download the parts of the remote .zip file it needs to heal the local container. Due to the structure of the .zip format, this is relatively efficient:

    • First request establishes the size of the file
    • Second request to seek to the end and read the Central directory
    • Then, reading from any files is just an additional request. If files are read in-order, the same request is re-used throughout.

    The itch.io-specific parts are in go-itchio, the io.ReaderAt interface for remote files can be found in httpkit.

    Note that the remote server must support the Range HTTP header and respond with a 206 partial content, otherwise an error is thrown.

    itchfs is implemented in a pluggable way, which means other vendors are welcome to come up with their own URL schemes and build a version of butler that supports them (for example, for building their own launcher/updater system).

    Resumable unzip

    The unzip command now accepts a --resume-file parameter. With itchfs URLs, it can be used to download and extract a file at the same time (and resume at the beginning of the last file it was working on, using the resume file's info).

    BPS / ETA

    All progress-emitting commands now output eta (estimated time remaining, in seconds) in json output mode. They also now emit progress (floating number between 0.0 and 1.0) in addition to percent (floating number between 0.0 and 100.0, kept for backwards compatibility for now).

    A few commands (like unzip, cp) are also getting bps (bytes per second processed). This is useful to show download/processing speed graphs:

    screen shot 2016-10-11 at 19 58 37

    Manifests (plans)

    The big idea for manifests is for them to be usable as a heal source for verify, apply, and as an argument to a new command unsplit. Since manifests are only a list of hashes, access control is left as an exercise to the user β€” for itch.io, it'll be implemented in the form of download sessions tied to a certain set of blocks, expiring, and started with a valid itch.io API key, which allows for monitoring of abuse and prevents Dropshipping blocks.

    Source code(tar.gz)
    Source code(zip)
  • v0.1.4(Oct 23, 2015)

  • v0.1.3(Oct 23, 2015)

  • v0.1.2(Oct 23, 2015)

  • v0.1.1(Oct 23, 2015)

  • v0.1.0(Oct 23, 2015)

Owner
itch.io
Open indie game marketplace and DIY game jam host
itch.io
A simple to use, composable, command line parser for C++ 11 and beyond

Clara v1.1.5 !! This repository is unmaintained. Go here for a fork that is somewhat maintained. !! A simple to use, composable, command line parser f

Catch Org 652 Jul 27, 2022
A library for interactive command line interfaces in modern C++

cli A cross-platform header only C++14 library for interactive command line interfaces (Cisco style) Features Header only Cross-platform (linux and wi

Daniele Pallastrelli 826 Aug 3, 2022
CLI11 is a command line parser for C++11 and beyond that provides a rich feature set with a simple and intuitive interface.

CLI11: Command line parser for C++11 What's new β€’ Documentation β€’ API Reference CLI11 is a command line parser for C++11 and beyond that provides a ri

null 2.2k Aug 5, 2022
Lightweight C++ command line option parser

Release versions Note that master is generally a work in progress, and you probably want to use a tagged release version. Version 3 breaking changes I

null 3.2k Aug 8, 2022
A simple to use, composable, command line parser for C++ 11 and beyond

Lyra A simple to use, composing, header only, command line arguments parser for C++ 11 and beyond. Obtain License Standards Stats Tests License Distri

Build Frameworks Group 350 Aug 4, 2022
A single header C++ library for parsing command line arguments and options with minimal amount of code

Quick Arg Parser Tired of unwieldy tools like getopt or argp? Quick Arg Parser is a single header C++ library for parsing command line arguments

null 45 Jul 14, 2022
CLIp is a clipboard emulator for a command line interface written in 100% standard C only. Pipe to it to copy, pipe from it to paste.

CLIp v2 About CLIp is a powerful yet easy to use and minimal clipboard manager for a command line environment, with no dependencies or bloat. Usage Sy

A.P. Jo. 12 Sep 18, 2021
pbr2gltf2 is a command line tool for converting PBR images to a glTF 2.0 material.

pbr2gltf2 is a command line tool for converting PBR images to a glTF 2.0 material. The tool is detecting depending on the filename, which PBR information is stored. It swizzles the images and does reassign the channels to a glTF 2.0 image. The tool stores the images plus a minimal, valid glTF 2.0 file containing the required material, textures and images.

UX3D GmbH 23 Jul 31, 2022
null 76 Apr 18, 2022
A command-line tool to display colorful distro information.

sjfetch A command-line tool to display colorful distro information.

Fikret Musk 6 Apr 6, 2022
LwSHELL is lightweight, platform independent, command line shell for embedded systems.

LwSHELL is lightweight, platform independent, command line shell for embedded systems. It targets communication with embedded systems from remote terminal to quickly send commands and the retrieve data from the device.

Tilen Majerle 64 Jul 21, 2022
Toybox: all-in-one Linux command line.

Toybox: all-in-one Linux command line.

Rob Landley 1.7k Jul 29, 2022
Simple command line tool that processes image files using the FidelityFX Super Resolution (FSR) or Contrast Adaptive Sharpening (CAS) shader systems.

Simple command line tool that processes image files using the FidelityFX Super Resolution (FSR) or Contrast Adaptive Sharpening (CAS) shader systems.

GPUOpen Effects 176 Jul 29, 2022
Command-line flag parsing in C

flag.h Inspired by Go's flag module: https://pkg.go.dev/flag WARNING! The design of the library is not finished and may be a subject to change. Quick

Tsoding 37 Jul 15, 2022
A command line tool with no external dependencies to print information about an X server instance.

xinfo A command line tool with no external dependencies to print information about an X server instance. Building and running To build the code in thi

Jean-Michel Gorius 6 Jan 13, 2022
Windows command line program for Spleeter, written in pure C, no need of Python.

SpleeterMsvcExe is a Windows command line program for Spleeter, which can be used directly. It is written in pure C language, using ffmpeg to read and write audio files, and using Tensorflow C API to make use of Spleeter models. No need to install Python environment, and it does not contain anything related to Python.

Wudi 169 Aug 4, 2022
easy to use, powerful & expressive command line argument parsing for modern C++ / single header / usage & doc generation

clipp - command line interfaces for modern C++ Easy to use, powerful and expressive command line argument handling for C++11/14/17 contained in a sing

AndrΓ© MΓΌller 903 Aug 8, 2022
Parse command line arguments by defining a struct

Parse command line arguments by defining a struct Quick Start #include <structopt/app.hpp> struct Options { // positional argument // e.g., .

Pranav 404 Jul 28, 2022
Rizin - UNIX-like reverse engineering framework and command-line toolset.

Rizin - UNIX-like reverse engineering framework and command-line toolset.

Rizin Organization 1.6k Aug 6, 2022