Pipe Viewer - monitor the progress of data through a pipe

From https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=507682 in 2014 - Anthony DeRobertis [email protected]

Package: pv Version: 1.1.4-1 Severity: wishlist File: /usr/bin/pv

It appears that pv always averages the transfer rate over the display interval. It'd be nice if that could be controlled seperately. Right now, if you have very bursty data (for example, lzcat receiving compressed data from the network, writing uncompressed to disk), you have to use a very long update interval, on the order of a minute, to get a reasonable rate estimate. It'd be nice to keep the 1s update interval, but take the average over a longer period.

It'd be awesome if pv recognized bursty data and adjusted the averaging itself, but that's way more than I expect.

-- System Information: Debian Release: lenny/sid APT prefers testing APT policy: (500, 'testing'), (100, 'unstable') Architecture: i386 (i686)

Kernel: Linux 2.6.26-1-686 (SMP w/2 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash

Versions of packages pv depends on: ii libc6 2.7-16 GNU C Library: Shared libraries

pv recommends no packages.

Versions of packages pv suggests: ii doc-base 0.8.16 utilities to manage online documen

-- no debconf information

Fixes a build failure like this:

x86_64-pc-linux-gnu-gcc -O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches -I./src/include -Isrc/include -DHAVE_CONFIG_H -DLOCALEDIR=\"/usr/share/locale\" -c -o src/pv/number.o src/pv/number.c
x86_64-pc-linux-gnu-gcc -O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches -I./src/include -Isrc/include -DHAVE_CONFIG_H -DLOCALEDIR=\"/usr/share/locale\" -c -o src/pv/signal.o src/pv/signal.c
x86_64-pc-linux-gnu-gcc -O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches -I./src/include -Isrc/include -DHAVE_CONFIG_H -DLOCALEDIR=\"/usr/share/locale\" -c -o src/pv/state.o src/pv/state.c
src/pv/number.c: In function ‘pv_getnum_ull’:
src/pv/number.c:32:13: error: ‘NULL’ undeclared (first use in this function)
   32 |         if (NULL == str)
      |             ^~~~
src/pv/number.c:9:1: note: ‘NULL’ is defined in header ‘<stddef.h>’; did you forget to ‘#include <stddef.h>’?
    8 | #include "pv.h"
  +++ |+#include <stddef.h>
    9 |
src/pv/number.c:32:13: note: each undeclared identifier is reported only once for each function it appears in
   32 |         if (NULL == str)
      |             ^~~~
src/pv/number.c: In function ‘pv_getnum_d’:
src/pv/number.c:122:13: error: ‘NULL’ undeclared (first use in this function)
  122 |         if (NULL == str)
      |             ^~~~
src/pv/number.c:122:13: note: ‘NULL’ is defined in header ‘<stddef.h>’; did you forget to ‘#include <stddef.h>’?
make: *** [Makefile:362: src/pv/number.o] Error 1
make: *** Waiting for unfinished jobs....
src/pv/loop.c: In function ‘pv_main_loop’:

Thanks!

Hello, it looks like the current implementation of pv always display the total xfered (-b/%b) bytes in human friendly form, sometimes this is inconvenient when debugging a shell script with non-interactive pv usage (e.g. -qSs)

https://wiki.archlinux.org/title/Dd#pv_(optional_use_zsh_for_read/write_with_offset)

I tried with dd and curl as potential alternatives but also not ideal:

• the GNU implementation of dd seemingly doesn't print a N bytes copied line if of= is closed earlier than if= (common in SIGPIPE errs), I think this design is due to POSIX's write() does not report partial successfully written bytes on errors.
• curl -qgsfw "%{stderr}%{size_download}" file:///dev/urandom | pv -qSs 512400 >/dev/null where curl seems does not always print actually written bytes in case of SIGPIPE (similar reason as dd below), hence wont useful for this setup.

pv is the only program in my knowledge that utilize splice() for streaming data in given length, could you please consider to implement a option that print these values in exact number of bytes to ease diagnosing of such kinds of shell scripts?

Intended to address Issue #44 to allow pv -s @filename like curl(1).

Also fixes compilation on latest OSX on Apple Silicon where stat64() and friends are linkable but not compilable.

I would like to be able to use pv to monitor a background transfer and periodically check the current progress. At a glance, a combination of pv -q & and pv -R $! seems like it should do the job but fails on two accounts:

1. pv -q & immediately stops, as it wants access to the terminal (even with -q)
2. pv -R ... affects the behaviour of the first pv in its terminal, rather than showing the progress in the current terminal

(1) is demonstrable -- just running pv -q </dev/zero >/dev/null & ends up with a stopped job that only works when forgrounded with fg (2) is demonstrable with help from screen or tmux -- running pv -q </dev/zero >/dev/null (no &) in one window and then running pv -R $(pidof -s pv) in a second does not display anything in the second window; while running pv -R $(pidof -s pv) -p starts writing output in the first window.

pv -d $(pidof pv) does not appear to help, I assume because the first pv already has the files open, so never "sees" a new open "event". Likewise pv -d "$(pidof pv):0" doesn't work, because the first pv has opened devices, not files (I am really hoping to monitor transfer between non-files, specifically stdout-to-fifo). I've also tried dd ... &; pv -d $(pidof dd), which suffers the same.

I suppose both pv -R and pv -d would also be limited to only having state information about the transfer since the second pv started, and would not have access to the time/byte-count/avg-rate of the running first pv.

As such, my enhancement request is twofold:

1. ability to background a pv without it stopping for terminal access
2. a specific switch, e.g. -M <pid>, to monitor (or mirror) the existing pv process indicated by <pid>, e.g by instructing the first process to IPC its transfer history to the second, and for the second to output to the terminal.
   • if a fifo-driven --redir-to / --redir-from semantic makes more sense than some other IPC, that would be fine

Note that I am not just hoping to solve the issue of backgrounding and checking in the same session -- I would hope/expect that implementing (1) would just allow CTRL+Z and judicious use of fg/bg for that purpose. I am also looking to have pv invoked somewhere else (e.g. by cron) and be able to attach to it for occasional progress updates by an operator. It just so happens that I need a long-running-writer | pv > /tmp/fifo & behaviour, so that something else can read the fifo.

(If any of this exists in the releases after 1.6.0, please advise/accept my apologies for the noise!)

I have an application that starts a background pv process to generate a progress bar for a complex pipeline. This used to work as I expected in pv 1.6.6. It no longer works in pv 1.6.20. pv runs, but the progress bar is not displayed.

In the new routine pv_in_foreground(void), the code does not check for an error return from tcgetpgrp(STDERR_FILENO).

our_process_group = getpgrp(); 
tty_process_group = tcgetpgrp(STDERR_FILENO);                                                                                                             
if (our_process_group == tty_process_group)                                                                                                               
    return true; 

Changing the code to the following does what I want:

our_process_group = getpgrp();
tty_process_group = tcgetpgrp(STDERR_FILENO);   
if (tty_process_group == -1)
    return true;
if (our_process_group == tty_process_group)                                                                                                               
    return true; 

Alternatively, a command line flag to force pv_in_foreground() to always return true would be helpful.

Hello,

pv is awesome and I'm using it for diagnostics and other purposes. Thanks for making it!

There's one idea I've been thinking about. What about the ability to run commands at certain percentages? Perferably with formatting support. This would useful for long-running transfers that you can't/don't want to observe, but still want to be able to get pv's stats.

e.g. something like pv --command="notify-send 'pv' '%p'" --command-interval 25 foo > bar would show a notification for every 25 percent to completion. Or --command="echo -ne '\a'" --command-interval 10 would ring the terminal bell every 10 percent.

Cheers! :)

pv seems to have a race condition that’s triggered intermittently. Take the following script:

# File name: bug-repro.sh

#!/bin/bash

progress() {
  # Perform some relevant calculations that may take a few millis,
  # such as calculating total size of files passed to this function.
  declare -r num_blocks="$(du -Ac "$@" | tail -1 | cut -f1)"
  declare -r bytes_per_block=512
  declare -r size_in_bytes="$(( num_blocks * bytes_per_block ))"

  # For purposes of reproducing this bug: If the above calculations
  # weren't slow enough to introduce the race condition, let's simulate
  # some more operations before calling `pv`.
  for i in {1..100}; do
    :
  done

  # Now call `pv` using the `--wait` switch
  pv --wait -s "${size_in_bytes}"
}

tar -cf - "$@" | progress "$@" | gpg -c > /dev/null

If I run it via $ ./bug-repro.sh file1 file2 file3, I expect that pv will wait for some bytes before displaying its progress bar, since --wait was used.

Instead, I see this before I start entering a password for GPG:

Enter passphrase: 64.0KiB 0:00:01 [62.2KiB/s] [===>                             64.0KiB 0:00:04 [0.00 B/s] [===>                               ] 13% ETA 0:00:25

Interestingly, if I hard-code the -s size, pv waits as expected:

# This behaves as expected
tar -cf - "$@" | pv --wait -s 12345 | gpg -c > /dev/null

Note, it may be necessary to have the following GPG configuration to successfully repro this:

$ head ~/.gnupg/{gpg,gpg-agent}.conf
==> /Users/danielyli/.gnupg/gpg.conf <==
use-agent
pinentry-mode loopback

==> /Users/danielyli/.gnupg/gpg-agent.conf <==
allow-loopback-pinentry

remote-cksum test failed on Fedora 37/rawhide ppc64le. Tests are good on other architectures.

+ make test
sh ./autoconf/scripts/run-test.sh ./pv .
000 - cat:             OK
001 - interval:        OK
002 - rate:            OK
003 - progress:        OK
004 - timer:           OK
005a - eta:            OK
005b - fineta:         OK
006 - ratecount:       OK
007 - bytes:           OK
008 - numeric:         OK
009 - quiet:           OK
010 - pipe:            OK
011 - cksum:           OK
012 - averagerate:     OK
013 - 1mboundary:      OK
014 - 1mboundary2:     OK
015 - cksumpipe:       OK
016 - numeric-timer:   OK
017 - numeric-bytes:   OK
018 - remote-format:   OK
019 - remote-cksum:    pv: 2411631: message not received
FAILED
020 - stop-at-size:    OK
make: *** [Makefile:277: test] Error 1

full log: build.log