WDT Warp speed Data Transfer

Last update: Dec 31, 2022

Related tags

Overview

`WDT` Warp speed Data Transfer

Design philosophy/Overview

Goal: Lowest possible total transfer time - to be only hardware limited (disc or network bandwidth not latency) and as efficient as possible (low CPU/memory/resources utilization)

We keep dependencies minimal in order to maximize portability and ensure a small binary size. As a bonus, this also minimizes compile time.

We aren't using exceptions for performance reasons and because using exceptions would make it harder to reason about the control flow of the library. We also believe the WDT library is easier to integrate as a result. Our philosophy is to write moderately structured and encapsulated C code as opposed to using every feature of C++.

We try to minimize the number of system calls, which is one of the reasons we are using blocking thread IOs. We can maximize system throughput because at any given point some threads are reading while others are writing, and data is buffered on both paths - keeping each subsystem busy while minimizing kernel to userspace switches.

Terminology

WDT uses "Mbytes" everywhere in its output as 1024*1024 bytes = 1048576 bytes (technically this should be the new mebibyte (MiB) standard but it felt Mbytes is be more in line with what other tools are using, clearer, easier to read and matching what a traditional "megabyte" used to mean in historical memory units where the address lines are binary and thus power of two and not of ten)

Example

While WDT is primarily a library, we also have a small command line tool which we use for tests and which is useful by itself. Here is a quick example:

Receiver side: (starts the server indicating destination directory)

[ldemailly@devbig074]$ wdt -directory /data/users/ldemailly/transfer1

Sender side: (discover and sends all files in a directory tree to destination)

[root@dev443]$ wdt -directory /usr/bin -destination devbig074.prn2

[=================================================] 100% 588.8 Mbytes/s
I0720 21:48:08.446014 3245296 Sender.cpp:314] Total sender time = 2.68699
seconds (0.00640992 dirTime). Transfer summary : Transfer status = OK. Number
of files transferred = 1887. Data Mbytes = 1582.08. Header Kbytes = 62.083
(0.00383215% overhead). Total bytes = 1658999858. Wasted bytes due to
failure = 0 (0% overhead). Total sender throughput = 588.816 Mbytes/sec
(590.224 Mbytes/sec pure transf rate)

Note that in this simple example with lots of small files (/usr/bin from a linux distribution), but not much data (~1.5Gbyte), the maximum speed isn't as good as it would with more data (as there is still a TCP ramp up time even though it's faster because of parallelization) like when we use it in our production use cases.

Performance/Results

In an internal use at Facebook to transfer RocksDB snapshot between hosts we are able to transfer data at a throttled 600 Mbytes/sec even across long distance, high latency links (e.g. Sweden to Oregon). That's 3x the speed of the previous highly optimized HTTP-based solution and with less strain on the system. When not throttling, we are able to easily saturate a 40 Gbit/s NIC and get near theoretical link speed (above 4 Gbytes/sec).

We have so far optimized WDT for servers with fast IOs - in particular flash card or in-memory read/writes. If you use disks throughput won't be as good, but we do plan on optimizing for disks as well in the future.

Dependencies

CMake for building WDT - See build/BUILD.md

gflags (google flags library) but only for the command line, the library doesn't depend on that

gtest (google testing) but only for tests

glog (google logging library) - use W*LOG macros so everything logged by WDT is always prefixed by "wdt>" which helps when embedded in another service

Parts of Facebook's Folly open source library (as set in the CMakefile) Mostly conv, threadlocal and checksum support.

For encryption, the crypto lib part of openssl-1.x

You can build and embed wdt as a library with as little as a C++11 compiler and glog - and you could macro away glog or replace it by printing to stderr if needed.

Code layout

Directories

top level Main WDT classes and Wdt command line source, CMakeLists.txt
util/ Utilities used for implementing the main objects
test/ Tests files and scripts
build/ Build related scripts and files and utils
fbonly/ Stuff specific to facebook/ (not in open source version)
bench/ Benchmark generation tools

Main files

CMakeLists.txt, .travis.yml, build/BUILD.md,travis_linux.sh,travis_osx.sh Build definition file - use CMake to generate a Makefile or a project file for your favorite IDE - details in build/BUILD.md
wdtCmdline.cpp

Main program which allows to have a server or client process to exercise the library (for end 2 end test as well as a standalone utility)

wcp.sh

A script to use wdt like scp for single big files - pending splitting support inside wdt proper the script does the splitting for you. install as "wcp".

WdtOptions.{h|cpp}

To specify the behavior of wdt. If wdt is used as a library, then the caller get the mutable object of options and set different options accordingly. When wdt is run in a standalone mode, behavior is changed through gflags in wdtCmdLine.cpp

WdtThread.{h|cpp} Common functionality and settings between SenderThread and ReceiverThread. Both of these kind of threads inherit from this base class.
WdtBase.{h|cpp}

Common functionality and settings between Sender and Receiver

WdtResourceController.{h|cpp}

Optional factory for Sender/Receiver with limit on number being created.

Producing/Sending

ByteSource.h

Interface for a data element to be sent/transferred

FileByteSource.{h|cpp}

Implementation/concrete subclass of ByteSource for a file identified as a relative path from a root dir. The identifier (path) sent remotely is the relative path

SourceQueue.h

Interface for producing next ByteSource to be sent

DirectorySourceQueue.{h|cpp}

Concrete implementation of SourceQueue producing all the files in a given directory, sorted by decreasing size (as they are discovered, you can start pulling from the queue even before all the files are found, it will return the current largest file)

ThreadTransferHistory.{h|cpp}

Every thread maintains a transfer history so that when a connection breaks it can talk to the receiver to find out up to where in the history has been sent. This class encapsulates all the logic for that bookkeeping

SenderThread.{h|cpp}

Implements the functionality of one sender thread, which binds to a certain port and sends files over.

Sender.{h|cpp}

Spawns multiple SenderThread threads and sends the data across to receiver

Consuming / Receiving

FileCreator.{h|cpp}

Creates file and directories necessary for said file (mkdir -p like)

ReceiverThread.{h|cpp}

Implements the functionality of the receiver threads, responsible for listening on a port and receiving files over the network.

Receiver.{h|cpp}

Parent receiver class that spawns multiple ReceiverThread threads and receives data from a remote host

Low level building blocks

ServerSocket.{h|.cpp}

Encapsulate a server socket listening on a port and giving a file descriptor to be used to communicate with the client

ClientSocket.{h|cpp}

Client socket wrapper - connection to a server port -> fd

Protocol.{h|cpp}

Decodes/Encodes meta information needed to interpret the data stream: the id (file path) and size (byte length of the data)

SocketUtils.{h|cpp}

Common socket related utilities (both client/server, sender/receiver side use)

Throttler.{h|cpp}

Throttling code

ErrorCodes.h

Header file for error codes

Reporting.{h|cpp}

Class representing transfer stats and reports

Future development/extensibility

The current implementation works well and has high efficiency. It is also extensible by implementing different byte sources both in and out. But inserting processing units isn't as easy.

For that we plan on restructuring the code to use a Zero copy stream/buffer pipeline: To maintain efficiency, the best overall total transfer time and time to first byte we can see WDT's internal architecture as chainable units

[Disk/flash/Storage IO] -> [Compression] -> [Protocol handling] -> [Encryption] -> [Network IO]

And the reverse chain on the receiving/writing end The trick is the data is variable length input and some units can change length and we need to process things by blocks Constraints/Design:

No locking / contention when possible
(Hard) Limits on memory used
Minimal number of copies/moving memory around
Still works the same for simple read file fd -> control -> write socked fd current basic implementation

Possible Solution(?) API:

Double linked list of Units
read/pull from left (pull() ?)
push to the right (push() ?)
end of stream from left
propagate last bytes to right

Can still be fully synchronous / blocking, works thanks to eof handling (synchronous gives us lock free/single thread - internally a unit is free to use parallelization like the compression stage is likely to want/need)

Another thing we touched on is processing chunks out of order - by changing header to be ( fileid, offset, size ) instead of ( filename, size ) and assuming everything is following in 1 continuous block (will also help the use case of small number of large files/chunks) : mmap'in the target/destination file The issue then is who creates it in what order - similar to the directory creation problem - we could use a meta info channel to avoid locking/contention but that requires synchronization

We want things to work with even up to 1 second latency without incurring a 1 second delay before we send the first payload byte

Submitting diffs/making changes

See CONTRIBUTING.md

Please run the tests

CTEST_OUTPUT_ON_FAILURE=1 make test

And ideally also the manual tests (integration/porting upcoming)

wdt_e2e_test.sh wdt_download_resumption_test.sh wdt_network_test.sh wdt_max_send_test.sh

(facebook only:) Make sure to do the following, before "arc diff":

 (cd wdt ; ./build/clangformat.sh )
 # if you changed the minor version of the protocol (in CMakeLists.txt)
 # run (cd wdt ; ./build/version_update.tcl ) to sync with fbcode's WdtConfig.h

 fbconfig  --clang --sanitize=address -r  wdt

 fbmake runtests --run-disabled --extended-tests
 # Optionally: opt build
 fbmake runtests_opt
 fbmake opt
 # Sender max speed test
 wdt/test/wdt_max_send_test.sh
 # Check buck build
 buck build wdt/...
 # Debug a specific test with full output even on success:
 buck test wdt:xxx -- --run-disabled --extended-tests --print-passing-details\
   --print-long-results

and check the output of the last step to make sure one of the 3 runs is still above 20,000 Mbytes/sec (you may need to make sure you /dev/shm is mostly empty to get the best memory throughput, as well as not having a ton of random processes running during the test)

Also :

Update this file
Make sure your diff has a task
Put (relevant) log output of sender/receiver in the diff test plan or comment
Depending on the changes
- Perf: wdt/wdt_e2e_test.sh has a mix of ~ > 700 files, > 8 Gbytes/sec
- do run remote network tests (wdt/wdt_remote_test.sh)
- do run profiler and check profile results (wdt/fbonly/wdt_prof.sh) 80k small files at > 1.6 Gbyte/sec

Comments

File permission transfer
Implementation and test for keeping file permission during transfer v0.1

Design decision:

Get permission from the same st_mode that is used to get file size, and store permission in each metadata entry of DirectorySourceQueue, in order to avoid calling stat another time.

Default permission is 512 (01000). When perm in WdtFileInfo is default, library behavior is not changed.

Test:

Encode and decode in Protocol

Print src and dst file permissions to file and compare in E2E

TODO:

Shall we consider dir permission?

What if blockDetails.allocationStatus == TO_BE_DELETED but receiver's file permission is owner only?

What if sender side allows only owner's read?

permission changed – only chmod the file. Don’t trans again

CLA Signed
opened by dliang36 19
Always fails to transfer the whole files

I get the following log: root@DQ5606-C12-2:/home/wdt_test# wdt --directory /home/wdt_test | ssh cn.wdthost.com wdt --directory /data/dev/wdt_test - I1207 02:50:52.842162 21692 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1207 02:50:52.842989 21692 EncryptionUtils.cpp:83] wdt> Openssl library initialized I1207 02:50:52.843101 21692 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1207 02:50:52.843622 21692 WdtResourceController.cpp:29] wdt> Updated max number of senders for root controller to 0 I1207 02:50:52.843740 21692 WdtResourceController.cpp:22] wdt> Updated max number of receivers for root controller to 0 I1207 02:50:52.843777 21692 Throttler.cpp:73] wdt> No average rate specified I1207 02:50:52.843806 21692 Throttler.cpp:80] wdt> No peak rate specified I1207 02:50:52.843847 21692 Wdt.cpp:24] wdt> One time initialization of WDT for wdt I1207 02:50:52.843894 21692 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1207 02:50:52.843991 21692 Receiver.cpp:46] wdt> WDT Receiver 1.27.1612021 p 27 I1207 02:50:52.844125 21692 WdtBase.cpp:82] wdt> Setting transfer id 540054088 I1207 02:50:52.844192 21692 WdtBase.cpp:94] wdt> using wdt protocol version 27 I1207 02:50:52.844244 21692 FileCreator.cpp:319] wdt> dir already exists / I1207 02:50:52.844298 21692 FileCreator.cpp:319] wdt> dir already exists /home/ I1207 02:50:52.844334 21692 FileCreator.cpp:319] wdt> dir already exists /home/wdt_test/ I1207 02:50:52.844372 21692 Receiver.cpp:170] wdt> aes128gcm encryption is enabled for this transfer I1207 02:50:52.844410 21692 Receiver.cpp:173] wdt> Receiver generating encryption key for type aes128gcm I1207 02:50:52.844635 21692 EncryptionUtils.cpp:115] wdt> New encryption params 0x7ffc1d2991b0 2:...11920925460756788275... I1207 02:50:52.845219 21692 ReceiverThread.cpp:987] wdt> Thread[0, port: 47908] Listening on port 47908 I1207 02:50:52.845518 21692 ReceiverThread.cpp:987] wdt> Thread[1, port: 49590] Listening on port 49590 I1207 02:50:52.845783 21692 ReceiverThread.cpp:987] wdt> Thread[2, port: 45411] Listening on port 45411 I1207 02:50:52.846050 21692 ReceiverThread.cpp:987] wdt> Thread[3, port: 35384] Listening on port 35384 I1207 02:50:52.846314 21692 ReceiverThread.cpp:987] wdt> Thread[4, port: 58549] Listening on port 58549 I1207 02:50:52.846580 21692 ReceiverThread.cpp:987] wdt> Thread[5, port: 49513] Listening on port 49513 I1207 02:50:52.846845 21692 ReceiverThread.cpp:987] wdt> Thread[6, port: 54616] Listening on port 54616 I1207 02:50:52.847108 21692 ReceiverThread.cpp:987] wdt> Thread[7, port: 47603] Listening on port 47603 I1207 02:50:52.847178 21692 Receiver.cpp:212] wdt> Registered 8 successful sockets I1207 02:50:52.847273 21692 wdtCmdLine.cpp:321] wdt> Starting receiver with connection url wdt://DQ5606-C12-2?dir=%2fhome%2fwdt_test&enc=2:...11920925460756788275...&id=540054088&ports=47908,49590,45411,35384,58549,49513,54616,47603&recpv=27 I1207 02:50:52.847404 21692 Receiver.cpp:455] wdt> Starting (receiving) server on ports [ 47908 49590 45411 35384 58549 49513 54616 47603 ] Target dir : /home/wdt_test I1207 02:50:52.847502 21692 Throttler.cpp:73] wdt> No average rate specified I1207 02:50:52.847534 21692 Throttler.cpp:80] wdt> No peak rate specified I1207 02:50:52.847564 21692 WdtBase.cpp:183] wdt> Enabling throttling avgRate: -1 Mbytes/sec, peakRate: 0 Mbytes/sec, bucketLimit: 0 Mbytes, throttlerLogTimeMillis: 0 I1207 02:50:52.847981 21702 Receiver.cpp:396] wdt> Progress reporter updating every 200 ms I1207 02:50:53.048244 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:53.248508 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:53.448701 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:53.648888 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:53.849079 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:54.049265 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:54.249464 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:54.449650 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:54.649834 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:54.850025 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:55.050211 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:55.250371 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:55.450556 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:55.650743 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:55.850953 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:56.051161 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:44.761514 13436 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1207 18:50:44.762029 13436 EncryptionUtils.cpp:83] wdt> Openssl library initialized I1207 18:50:44.762055 13436 WdtFlags.cpp:63] wdt> Running WDT 1.27.1612021 p 27 I1207 18:50:44.762279 13436 WdtResourceController.cpp:29] wdt> Updated max number of senders for root controller to 0 I1207 18:50:44.762300 13436 WdtResourceController.cpp:22] wdt> Updated max number of receivers for root controller to 0 I1207 18:50:44.762308 13436 Throttler.cpp:73] wdt> No average rate specified I1207 18:50:44.762310 13436 Throttler.cpp:80] wdt> No peak rate specified I1207 18:50:44.762320 13436 Wdt.cpp:24] wdt> One time initialization of WDT for wdt I1207 18:50:44.762336 13436 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1207 18:50:44.762446 13436 wdtCmdLine.cpp:282] wdt> Parsed url as wdt://DQ5606-C12-2?dir=%2fdata%2fdev%2fwdt_test&enc=2:...11920925460756788275...&id=540054088&ports=47908,49590,45411,35384,58549,49513,54616,47603&recpv=27 I1207 18:50:44.762462 13436 wdtCmdLine.cpp:365] wdt> Making Sender with encryption set = 1 I1207 18:50:44.762475 13436 WdtResourceController.cpp:357] wdt> First time (default) is seen, creating. I1207 18:50:44.762483 13436 WdtResourceController.cpp:29] wdt> Updated max number of senders for to 0 I1207 18:50:44.762488 13436 WdtResourceController.cpp:22] wdt> Updated max number of receivers for to 1 I1207 18:50:44.762498 13436 Throttler.cpp:98] wdt> Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0 I1207 02:50:56.251382 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:50:56.269986 21699 Receiver.cpp:93] wdt> Starting new transfer, peerIp 120.132.7.182 , transfer id 540054088 I1207 02:50:56.270177 21699 WdtSocket.cpp:67] wdt> Successfully read encryption settings 49513 aes128gcm I1207 02:50:56.274632 21694 WdtSocket.cpp:67] wdt> Successfully read encryption settings 47908 aes128gcm I1207 18:50:44.762518 13436 Sender.cpp:47] wdt> WDT Sender 1.27.1612021 p 27 I1207 18:50:44.762532 13436 WdtResourceController.cpp:379] wdt> Successfully added a sender for identifier DQ5606-C12-2 I1207 18:50:44.762543 13436 WdtBase.cpp:94] wdt> using wdt protocol version 27 I1207 18:50:44.762549 13436 Sender.cpp:84] wdt> Encryption is enabled for this transfer I1207 18:50:44.762572 13436 DirectorySourceQueue.cpp:127] wdt> Root dir now /data/dev/wdt_test/ I1207 18:50:44.762593 13436 Sender.cpp:311] wdt> Client (sending) to DQ5606-C12-2, Using ports [ 47908 49590 45411 35384 58549 49513 54616 47603 ] I1207 18:50:44.762599 13436 Sender.cpp:325] wdt> Skipping throttler setup. External throttler set.Throttler details : avgRate: -1 Mbytes/sec, peakRate: 0 Mbytes/sec, bucketLimit: 0 Mbytes, throttlerLogTimeMillis: 0 I1207 18:50:44.763731 13439 Sender.cpp:41] wdt> Starting a new transfer 540054088 to DQ5606-C12-2 I1207 18:50:44.763741 13437 DirectorySourceQueue.cpp:240] wdt> Exploring root dir /data/dev/wdt_test/ include_pattern : exclude_pattern : prune_dir_pattern : I1207 18:50:44.764354 13437 DirectorySourceQueue.cpp:389] wdt> Number of files explored: 1 opened 0 with direct 0 errors false I1207 18:50:44.764436 13446 Sender.cpp:418] wdt> Progress reporter tracking every 200 ms I1207 18:50:44.964593 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 18:50:44.973379 13443 SenderThread.cpp:85] wdt> Thread[5, port: 49513] Connection took 1 attempt(s) and 0.209555 seconds. port 49513 I1207 18:50:44.976073 13438 SenderThread.cpp:85] wdt> Thread[0, port: 47908] Connection took 1 attempt(s) and 0.212263 seconds. port 47908 I1207 18:50:44.984274 13444 SenderThread.cpp:85] wdt> Thread[6, port: 54616] Connection took 1 attempt(s) and 0.220449 seconds. port 54616 I1207 18:50:44.986655 13439 SenderThread.cpp:85] wdt> Thread[1, port: 49590] Connection took 1 attempt(s) and 0.222851 seconds. port 49590 I1207 02:50:56.283303 21695 WdtSocket.cpp:67] wdt> Successfully read encryption settings 49590 aes128gcm I1207 18:50:44.988518 13442 SenderThread.cpp:85] wdt> Thread[4, port: 58549] Connection took 1 attempt(s) and 0.224649 seconds. port 58549 I1207 02:50:56.287132 21698 WdtSocket.cpp:67] wdt> Successfully read encryption settings 58549 aes128gcm I1207 02:50:56.291884 21700 WdtSocket.cpp:67] wdt> Successfully read encryption settings 54616 aes128gcm I1207 18:50:45.000504 13440 SenderThread.cpp:85] wdt> Thread[2, port: 45411] Connection took 1 attempt(s) and 0.236658 seconds. port 45411 I1207 02:50:56.311111 21696 WdtSocket.cpp:67] wdt> Successfully read encryption settings 45411 aes128gcm I1207 02:50:56.354485 21701 ReceiverThread.cpp:195] wdt> Thread[7, port: 47603] entered ACCEPT_WITH_TIMEOUT state I1207 02:50:56.354607 21697 ReceiverThread.cpp:195] wdt> Thread[3, port: 35384] entered ACCEPT_WITH_TIMEOUT state I1207 02:50:56.451607 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.00102392 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:45.164758 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:56.651818 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000487037 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:45.364905 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:56.852028 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000319506 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:45.565044 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:57.052247 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000237731 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:45.765179 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:57.252461 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000189284 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:45.965308 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:57.452673 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000157241 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:46.165446 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:57.652895 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000134475 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:46.365588 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:57.853124 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000117467 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:46.565747 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:58.053350 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000104279 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:46.765897 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:58.253563 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 9.37532e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:46.966025 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:58.453788 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 8.51572e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:47.166177 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:58.654011 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 7.80053e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:47.366334 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:58.854233 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 7.19616e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:47.566481 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:59.054456 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 6.67871e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:47.766621 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:59.254686 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 6.23066e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:47.966785 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:59.454839 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 5.8391e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:48.166957 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:59.655055 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 5.49373e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:48.367097 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:50:59.855269 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 5.18694e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:48.567258 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:00.055480 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.91261e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:48.767405 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:00.255697 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.66583e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:48.967564 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:00.455852 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.44272e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:49.167713 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:00.656059 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.23993e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:49.367856 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:00.856274 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.05484e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:49.567999 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:01.056495 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.88522e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:49.768143 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:01.256732 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.72922e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:49.968300 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:01.456986 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.58525e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:50.168464 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:01.657230 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.45198e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:50.368628 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:01.857601 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.3282e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:50.568780 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:02.057842 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.21306e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:50.765522 13441 WdtSocket.cpp:282] wdt> socket io timed out after 5000 ms, retries 8754349 fd 6 doneBytes 0 I1207 18:50:50.765550 13445 WdtSocket.cpp:282] wdt> socket io timed out after 5000 ms, retries 7521722 fd 8 doneBytes 0 E1207 18:50:50.765683 13441 WdtSocket.cpp:324] wdt> Unable to read tag at end of stream got -1 needed 16 0x00000000000000000000000000000000 E1207 18:50:50.765702 13445 WdtSocket.cpp:324] wdt> Unable to read tag at end of stream got -1 needed 16 0x00000000000000000000000000000000 E1207 02:51:02.064534 21697 WdtSocket.cpp:29] wdt> Failed to read encryption settings 0 35384 E1207 02:51:02.064535 21701 WdtSocket.cpp:29] wdt> Failed to read encryption settings 0 47603 E1207 02:51:02.064581 21697 ReceiverThread.cpp:44] Read error on 35384 after 0: Resource temporarily unavailable [11] E1207 02:51:02.064587 21701 ReceiverThread.cpp:44] Read error on 47603 after 0: Resource temporarily unavailable [11] E1207 02:51:02.064631 21697 ReceiverThread.cpp:265] wdt> Thread[3, port: 35384] socket read failure 256 -1 E1207 02:51:02.064643 21701 ReceiverThread.cpp:265] wdt> Thread[7, port: 47603] socket read failure 256 -1 I1207 02:51:02.064666 21701 ReceiverThread.cpp:195] wdt> Thread[7, port: 47603] entered ACCEPT_WITH_TIMEOUT state I1207 02:51:02.064647 21697 ReceiverThread.cpp:195] wdt> Thread[3, port: 35384] entered ACCEPT_WITH_TIMEOUT state I1207 18:50:50.768913 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) I1207 02:51:02.258024 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.10564e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 18:50:50.969048 13446 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete) *** Error in `wdt': free(): invalid pointer: 0x00007fd614000930 *** I1207 02:51:02.333204 21697 ReceiverThread.cpp:237] wdt> Thread[3, port: 35384] entered SEND_LOCAL_CHECKPOINT state checkpoint-port: 35384 num-blocks: 0 seq-id: 0 block-offset: 0 received-bytes: 0 I1207 02:51:02.333727 21701 ReceiverThread.cpp:237] wdt> Thread[7, port: 47603] entered SEND_LOCAL_CHECKPOINT state checkpoint-port: 47603 num-blocks: 0 seq-id: 0 block-offset: 0 received-bytes: 0 I1207 02:51:02.458220 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.00516e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) E1207 02:51:02.570698 21697 WdtSocket.cpp:256] non-retryable error encountered during socket io 31 0 1: Connection reset by peer [104] E1207 02:51:02.570750 21697 WdtSocket.cpp:29] wdt> Failed to read encryption settings -1 35384 E1207 02:51:02.570780 21697 ReceiverThread.cpp:44] Read error on 35384 after 0: Connection reset by peer [104] E1207 02:51:02.570806 21697 ReceiverThread.cpp:265] wdt> Thread[3, port: 35384] socket read failure 256 -1 I1207 02:51:02.570835 21697 ReceiverThread.cpp:195] wdt> Thread[3, port: 35384] entered ACCEPT_WITH_TIMEOUT state I1207 02:51:02.570884 21697 EncryptionUtils.cpp:366] wdt> Encryption finish tag = 0x5ca08bb044c63b00118e0fde4f84db1b E1207 02:51:02.571048 21701 WdtSocket.cpp:256] non-retryable error encountered during socket io 32 0 1: Connection reset by peer [104] E1207 02:51:02.571104 21701 WdtSocket.cpp:29] wdt> Failed to read encryption settings -1 47603 E1207 02:51:02.571120 21701 ReceiverThread.cpp:44] Read error on 47603 after 0: Connection reset by peer [104] E1207 02:51:02.571135 21701 ReceiverThread.cpp:265] wdt> Thread[7, port: 47603] socket read failure 256 -1 I1207 02:51:02.571148 21701 ReceiverThread.cpp:195] wdt> Thread[7, port: 47603] entered ACCEPT_WITH_TIMEOUT state I1207 02:51:02.571177 21701 EncryptionUtils.cpp:366] wdt> Encryption finish tag = 0xea11379ebf080cedf46543d84682b1fc I1207 02:51:02.658404 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.91099e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) W1207 02:51:02.667558 21700 ReceiverThread.cpp:75] wdt> Eof on 27 E1207 02:51:02.667672 21700 ReceiverThread.cpp:537] wdt> Thread[6, port: 54616] could not read entire content for ffmpeg.tar.bz2 port 54616 I1207 02:51:02.667717 21700 ReceiverThread.cpp:195] wdt> Thread[6, port: 54616] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:02.667752 21700 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode W1207 02:51:02.719172 21695 ReceiverThread.cpp:75] wdt> Eof on 23 E1207 02:51:02.719282 21695 ReceiverThread.cpp:537] wdt> Thread[1, port: 49590] could not read entire content for ffmpeg.tar.bz2 port 49590 I1207 02:51:02.719326 21695 ReceiverThread.cpp:195] wdt> Thread[1, port: 49590] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:02.719358 21695 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode E1207 02:51:02.757189 21699 ReceiverThread.cpp:572] wdt> Thread[5, port: 49513] socket read failure 256 0 I1207 02:51:02.757382 21699 ReceiverThread.cpp:195] wdt> Thread[5, port: 49513] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:02.757423 21699 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode W1207 02:51:02.780213 21698 ReceiverThread.cpp:75] wdt> Eof on 25 E1207 02:51:02.780335 21698 ReceiverThread.cpp:537] wdt> Thread[4, port: 58549] could not read entire content for ffmpeg.tar.bz2 port 58549 I1207 02:51:02.780380 21698 ReceiverThread.cpp:195] wdt> Thread[4, port: 58549] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:02.780412 21698 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode I1207 02:51:02.858681 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.82251e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) W1207 02:51:02.864729 21694 ReceiverThread.cpp:75] wdt> Eof on 21 E1207 02:51:02.864847 21694 ReceiverThread.cpp:537] wdt> Thread[0, port: 47908] could not read entire content for ffmpeg.tar.bz2 port 47908 I1207 02:51:02.864890 21694 ReceiverThread.cpp:195] wdt> Thread[0, port: 47908] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:02.864923 21694 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode I1207 02:51:03.058962 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.73924e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:03.259237 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.66075e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:03.459511 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.58663e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:03.659790 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.51653e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:03.860057 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.45013e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:04.060333 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.38714e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:04.260581 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.32732e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) W1207 02:51:04.331140 21696 ReceiverThread.cpp:75] wdt> Eof on 29 E1207 02:51:04.331253 21696 ReceiverThread.cpp:537] wdt> Thread[2, port: 45411] could not read entire content for ffmpeg.tar.bz2 port 45411 I1207 02:51:04.331310 21696 ReceiverThread.cpp:195] wdt> Thread[2, port: 45411] entered ACCEPT_WITH_TIMEOUT state E1207 02:51:04.331352 21696 EncryptionUtils.cpp:473] wdt> Need tag for gcm mode I1207 02:51:04.460783 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.27043e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) E1207 02:51:04.571967 21697 ReceiverThread.cpp:218] wdt> Thread[3, port: 35384] accept() failed with error CONN_ERROR timeout 2000 I1207 02:51:04.572010 21697 ReceiverThread.cpp:866] wdt> Thread[3, port: 35384] entered FINISH_WITH_ERROR state I1207 02:51:04.572070 21697 Receiver.cpp:30] wdt> Adding global checkpoint 35384 0 0 E1207 02:51:04.572098 21701 ReceiverThread.cpp:218] wdt> Thread[7, port: 47603] accept() failed with error CONN_ERROR timeout 2000 I1207 02:51:04.572106 21697 ReceiverThread.cpp:964] wdt> Thread[3, port: 35384] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0585938 (100% overhead). Total bytes = 60. Wasted bytes due to failure = 0 (100% overhead). Encryption type = aes128gcm. I1207 02:51:04.572131 21701 ReceiverThread.cpp:866] wdt> Thread[7, port: 47603] entered FINISH_WITH_ERROR state I1207 02:51:04.572213 21701 Receiver.cpp:30] wdt> Adding global checkpoint 47603 0 0 I1207 02:51:04.572237 21701 ReceiverThread.cpp:964] wdt> Thread[7, port: 47603] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0585938 (100% overhead). Total bytes = 60. Wasted bytes due to failure = 0 (100% overhead). Encryption type = aes128gcm. I1207 02:51:04.661051 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.21624e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:04.861266 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.1646e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:05.061517 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.11529e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:05.261729 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.06819e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:05.461952 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.02314e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:05.662168 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.98001e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:05.862378 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.93869e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:06.062590 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.89905e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:06.262804 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.861e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:06.463014 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.82445e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:06.663224 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.7893e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:06.863433 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.75549e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:07.063647 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.72292e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:07.263859 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.69155e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:07.464063 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.66129e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:07.664271 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.6321e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:07.864480 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.60392e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:08.064704 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.57669e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:08.264933 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.55037e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:08.465144 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.52492e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:08.665357 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.50029e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) E1207 02:51:08.668330 21700 ReceiverThread.cpp:218] wdt> Thread[6, port: 54616] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:08.668367 21700 ReceiverThread.cpp:866] wdt> Thread[6, port: 54616] entered FINISH_WITH_ERROR state I1207 02:51:08.668419 21700 Receiver.cpp:30] wdt> Adding global checkpoint 54616 0 0 I1207 02:51:08.668440 21700 ReceiverThread.cpp:964] wdt> Thread[6, port: 54616] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 7222578. Wasted bytes due to failure = 7222545 (100% overhead). Encryption type = aes128gcm. E1207 02:51:08.719959 21695 ReceiverThread.cpp:218] wdt> Thread[1, port: 49590] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:08.719993 21695 ReceiverThread.cpp:866] wdt> Thread[1, port: 49590] entered FINISH_WITH_ERROR state I1207 02:51:08.720039 21695 Receiver.cpp:30] wdt> Adding global checkpoint 49590 0 0 I1207 02:51:08.720057 21695 ReceiverThread.cpp:964] wdt> Thread[1, port: 49590] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 8882978. Wasted bytes due to failure = 8882945 (100% overhead). Encryption type = aes128gcm. E1207 02:51:08.758036 21699 ReceiverThread.cpp:218] wdt> Thread[5, port: 49513] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:08.758093 21699 ReceiverThread.cpp:866] wdt> Thread[5, port: 49513] entered FINISH_WITH_ERROR state I1207 02:51:08.758157 21699 Receiver.cpp:30] wdt> Adding global checkpoint 49513 0 0 I1207 02:51:08.758193 21699 ReceiverThread.cpp:964] wdt> Thread[5, port: 49513] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0292969 (100% overhead). Total bytes = 16777246. Wasted bytes due to failure = 16777216 (100% overhead). Encryption type = aes128gcm. E1207 02:51:08.781049 21698 ReceiverThread.cpp:218] wdt> Thread[4, port: 58549] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:08.781083 21698 ReceiverThread.cpp:866] wdt> Thread[4, port: 58549] entered FINISH_WITH_ERROR state I1207 02:51:08.781129 21698 Receiver.cpp:30] wdt> Adding global checkpoint 58549 0 0 I1207 02:51:08.781147 21698 ReceiverThread.cpp:964] wdt> Thread[4, port: 58549] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 10327465. Wasted bytes due to failure = 10327432 (100% overhead). Encryption type = aes128gcm. I1207 02:51:08.865571 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.47644e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) E1207 02:51:08.865777 21694 ReceiverThread.cpp:218] wdt> Thread[0, port: 47908] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:08.865810 21694 ReceiverThread.cpp:866] wdt> Thread[0, port: 47908] entered FINISH_WITH_ERROR state I1207 02:51:08.865855 21694 Receiver.cpp:30] wdt> Adding global checkpoint 47908 0 0 I1207 02:51:08.865874 21694 ReceiverThread.cpp:964] wdt> Thread[0, port: 47908] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 1342578. Wasted bytes due to failure = 1342545 (100% overhead). Encryption type = aes128gcm. I1207 02:51:09.065784 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.45334e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:09.266005 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.43095e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:09.466218 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.40924e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:09.666429 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.38818e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:09.866642 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.36774e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:10.066854 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.34789e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) I1207 02:51:10.267066 21702 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.32861e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete) E1207 02:51:10.331954 21696 ReceiverThread.cpp:218] wdt> Thread[2, port: 45411] accept() failed with error CONN_ERROR timeout 6000 I1207 02:51:10.331990 21696 ReceiverThread.cpp:866] wdt> Thread[2, port: 45411] entered FINISH_WITH_ERROR state I1207 02:51:10.332042 21696 Receiver.cpp:30] wdt> Adding global checkpoint 45411 0 0 I1207 02:51:10.332068 21696 Receiver.cpp:107] wdt> Ending the transfer 540054088 I1207 02:51:10.332085 21696 ReceiverThread.cpp:964] wdt> Thread[2, port: 45411] Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 2661065. Wasted bytes due to failure = 2661032 (100% overhead). Encryption type = aes128gcm. E1207 02:51:10.332516 21692 Reporting.cpp:200] wdt> Did not receive all the blocks sent by the sender -1 0 I1207 02:51:10.332576 21692 Reporting.cpp:339] wdt> wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.32242e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (incomplete) W1207 02:51:10.332618 21692 Receiver.cpp:331] wdt> WDT receiver's transfer has been finished I1207 02:51:10.332646 21692 Receiver.cpp:332] wdt> Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.307617 (100% overhead). Total bytes = 47214030. Wasted bytes due to failure = 47213715 (100% overhead). Encryption type = none. Previously sent bytes : 0. W1207 02:51:10.332726 21692 Receiver.cpp:300] wdt> Threads have already been joined. Returning the transfer report E1207 02:51:10.332757 21692 Reporting.cpp:200] wdt> Did not receive all the blocks sent by the sender -1 0 E1207 02:51:10.333192 21692 wdtCmdLine.cpp:374] wdt> Returning with code 3 CONN_ERROR I1207 02:51:10.333254 21692 WdtResourceController.cpp:294] wdt> Shutting down the controller (0 senders 0 receivers)

opened by nxtreaming 18
wdt with 3 Mac computers with macOS Sierra 10.12.5
Here is my configuration: 3 Mac computers with macOS Sierra 10.12.5 Using mac AirPort for router.

iMac - wired LAN to AirPort- IP:10.0.1.9

MacBookPro - WiFi to AirPort- IP:10.0.1.10

Mac-mini - WiFi to AirPort- IP:10.0.1.7

Executing commands from 1): ssh-copy-id [email protected] - OK ssh-copy-id [email protected] - OK So there is no need for ssh password

This is OK:copy from MacBook wdt --encryption_type=none -directory /Users/MyName/Desktop/Test | ssh [email protected] wdt --encryption_type=none -directory /Users/MyName/Documents/books -

I can see the files in Desktop/Test

This is NOT OK:copy from mac-mini wdt --encryption_type=none -directory /Users/MyName/Desktop/Test | ssh [email protected] wdt --encryption_type=none -directory /Users/MyName/Documents/books -

I cannot see the files in Desktop/Test

Following: https://github.com/facebook/wdt/blob/master/build/BUILD.md CTEST_OUTPUT_ON_FAILURE=1 make test on

iMac

MacBookPro all tests passed, on 3) mac-mini CTEST_OUTPUT_ON_FAILURE=1 make test is returning the following: (stopped because it took very long) make_test.txt

wdt --encryption_type=none -directory /Users/MyName/Desktop/Test | ssh [email protected] wdt --encryption_type=none -directory /Users/MyName/Documents/books - log: wdt.txt
opened by constantinevassil 17
Fail to transfer data.

I fail to transfer any data between two host( one is located in USA West, another is in China). the wdt always reports: Receiver and sender id mismatch xxxxx yyyyyy

The following is the log of wdt receiver:

root@i-19uz5u6n:/main_disk/test_data# wdt -directory /main_disk/test_data/ I0905 19:08:55.293244 6624 WdtFlags.cpp:64] Running WDT 1.16.1508311 p 16 I0905 19:08:55.293895 6624 Receiver.cpp:126] WDT Receiver 1.16.1508311 p 16 I0905 19:08:55.294070 6624 WdtBase.cpp:304] Generated a transfer id 708577588 I0905 19:08:55.294128 6624 WdtBase.cpp:272] using wdt protocol version 16 I0905 19:08:55.294659 6624 Receiver.cpp:160] Registered 8 sockets I0905 19:08:55.294714 6624 Receiver.cpp:172] Transfer id 708577588 I0905 19:08:55.294776 6624 wdtCmdLine.cpp:168] Starting receiver with connection url wdt://i-19uz5u6n?ports=22356,22357,22358,22359,22360,22361,22362,22363&protocol=16&id=708577588 I0905 19:08:55.294911 6624 wdtCmdLine.cpp:83] Setting up abort 0 seconds. I0905 19:08:55.294942 6624 Receiver.cpp:419] Starting (receiving) server on ports [ 22356 22357 22358 22359 22360 22361 22362 22363 ] Target dir : /main_disk/test_data/ I0905 19:08:55.294976 6624 FileCreator.cpp:222] dir already exists / I0905 19:08:55.294998 6624 FileCreator.cpp:222] dir already exists /main_disk/ I0905 19:08:55.295013 6624 FileCreator.cpp:222] dir already exists /main_disk/test_data/ I0905 19:08:55.295054 6624 WdtBase.cpp:292] Throttling not enabled I0905 19:08:55.298210 6633 Receiver.cpp:376] Progress reporter updating every 20 ms I0905 19:09:18.116607 6628 Receiver.cpp:523] New transfer started 1 E0905 19:09:18.116955 6628 Receiver.cpp:815] Receiver and sender id mismatch 736683926 708577588 I0905 19:09:18.116976 6628 Receiver.cpp:1255] Thread[3, port: 22359] entered SEND_ABORT_CMD state I0905 19:09:18.117035 6628 Receiver.cpp:1313] Thread[3, port: 22359] entered WAIT_FOR_FINISH_WITH_THREAD_ERROR state I0905 19:09:18.117059 6628 Receiver.cpp:110] Adding global checkpoint 22359 0 0 E0905 19:09:18.126565 6626 Receiver.cpp:815] Receiver and sender id mismatch 736683926 708577588 I0905 19:09:18.126641 6626 Receiver.cpp:1255] Thread[1, port: 22357] entered SEND_ABORT_CMD state I0905 19:09:18.126684 6626 Receiver.cpp:1313] Thread[1, port: 22357] entered WAIT_FOR_FINISH_WITH_THREAD_ERROR state I0905 19:09:18.126704 6626 Receiver.cpp:110] Adding global checkpoint 22357 0 0 E0905 19:09:18.129009 6629 Receiver.cpp:815] Receiver and sender id mismatch 736683926 708577588 I0905 19:09:18.129056 6629 Receiver.cpp:1255] Thread[4, port: 22360] entered SEND_ABORT_CMD state I0905 19:09:18.129114 6629 Receiver.cpp:1313] Thread[4, port: 22360] entered WAIT_FOR_FINISH_WITH_THREAD_ERROR state I0905 19:09:18.129143 6629 Receiver.cpp:110] Adding global checkpoint 22360 0 0 E0905 19:09:18.131338 6632 Receiver.cpp:815] Receiver and sender id mismatch 736683926 708577588 I0905 19:09:18.131372 6632 Receiver.cpp:1255] Thread[7, port: 22363] entered SEND_ABORT_CMD state I0905 19:09:18.131412 6632 Receiver.cpp:1313] Thread[7, port: 22363] entered WAIT_FOR_FINISH_WITH_THREAD_ERROR state I0905 19:09:18.131431 6632 Receiver.cpp:110] Adding global checkpoint 22363 0 0 E0905 19:09:18.134858 6631 Receiver.cpp:815] Receiver and sender id mismatch 736683926 708577588 I0905 19:09:18.134884 6631 Receiver.cpp:1255] Thread[6, port: 22362] entered SEND_ABORT_CMD state I0905 19:09:18.134917 6631 Receiver.cpp:1313] Thread[6, port: 22362] entered WAIT_FOR_FINISH_WITH_THREAD_ERROR state I0905 19:09:18.134933 6631 Receiver.cpp:110] Adding global checkpoint 22362 0 0 I0905 19:09:18.136365 6630 Receiver.cpp:628] Thread[5, port: 22361] entered ACCEPT_WITH_TIMEOUT state I0905 19:09:18.136404 6627 Receiver.cpp:628] Thread[2, port: 22358] entered ACCEPT_WITH_TIMEOUT state E0905 19:09:18.143194 6630 Receiver.cpp:815] Receiver and sender id mismatch 736683926 708577588 I0905 19:09:18.143245 6630 Receiver.cpp:1255] Thread[5, port: 22361] entered SEND_ABORT_CMD state I0905 19:09:18.143299 6630 Receiver.cpp:1313] Thread[5, port: 22361] entered WAIT_FOR_FINISH_WITH_THREAD_ERROR state I0905 19:09:18.143329 6630 Receiver.cpp:110] Adding global checkpoint 22361 0 0 E0905 19:09:18.385826 6627 Receiver.cpp:815] Receiver and sender id mismatch 736683926 708577588 I0905 19:09:18.385893 6627 Receiver.cpp:1255] Thread[2, port: 22358] entered SEND_ABORT_CMD state I0905 19:09:18.385964 6627 Receiver.cpp:1313] Thread[2, port: 22358] entered WAIT_FOR_FINISH_WITH_THREAD_ERROR state I0905 19:09:18.385990 6627 Receiver.cpp:110] Adding global checkpoint 22358 0 0 E0905 19:09:18.792495 6625 Receiver.cpp:815] Receiver and sender id mismatch 736683926 708577588 I0905 19:09:18.792554 6625 Receiver.cpp:1255] Thread[0, port: 22356] entered SEND_ABORT_CMD state I0905 19:09:18.792611 6625 Receiver.cpp:1313] Thread[0, port: 22356] entered WAIT_FOR_FINISH_WITH_THREAD_ERROR state I0905 19:09:18.792634 6625 Receiver.cpp:110] Adding global checkpoint 22356 0 0 I0905 19:09:18.792659 6625 Receiver.cpp:475] Received done for all threads. Transfer session 1 finished W0905 19:09:18.793324 6631 Receiver.cpp:1425] Last thread finished. Duration of the transfer 0.676722 W0905 19:09:18.793570 6624 Receiver.cpp:277] WDT receiver's transfer has been finished I0905 19:09:18.793588 6624 Receiver.cpp:278] Transfer status (local) = ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header kBytes = 0.109375 (100% overhead). Total bytes = 112. Wasted bytes due to failure = 0 (100% overhead). root@i-19uz5u6n:/main_disk/test_data#

The following is the log of wdt server(sender) root@localhost:/relay/dev# wdt -directory /relay/dev/test_data/ -destination rtmp.imbatv.cn I0905 11:09:18.096493 576 WdtFlags.cpp:64] Running WDT 1.16.1508311 p 16 I0905 11:09:18.097306 576 Sender.cpp:155] WDT Sender 1.16.1508311 p 16 I0905 11:09:18.097429 576 DirectorySourceQueue.cpp:116] Root dir now /relay/dev/test_data/ I0905 11:09:18.097575 576 WdtBase.cpp:304] Generated a transfer id 736683926 I0905 11:09:18.097633 576 WdtBase.cpp:272] using wdt protocol version 16 I0905 11:09:18.097725 576 wdtCmdLine.cpp:212] Starting sender with details wdt://rtmp.imbatv.cn?ports=22356,22357,22358,22359,22360,22361,22362,22363&protocol=16&dir=/relay/dev/test_data/&id=736683926 I0905 11:09:18.097775 576 wdtCmdLine.cpp:83] Setting up abort 0 seconds. I0905 11:09:18.097811 576 Sender.cpp:376] Client (sending) to rtmp.imbatv.cn, Using ports [ 22356 22357 22358 22359 22360 22361 22362 22363 ] I0905 11:09:18.097975 576 WdtBase.cpp:292] Throttling not enabled I0905 11:09:18.098412 577 DirectorySourceQueue.cpp:214] Exploring root dir /relay/dev/test_data/ include_pattern : exclude_pattern : prune_dir_pattern : I0905 11:09:18.098726 577 DirectorySourceQueue.cpp:362] Number of files explored: 1, errors: false I0905 11:09:18.102386 587 Sender.cpp:1300] Progress reporter tracking every 20 ms [ ] 0% 0.0 0.0 Mbytes/s I0905 11:09:18.318181 582 Sender.cpp:507] Connection took 1 attempt(s) and 0.218243 seconds. port 22359 I0905 11:09:18.324580 586 Sender.cpp:507] Connection took 1 attempt(s) and 0.222995 seconds. port 22363 [ ] 0% 0.0 0.0 Mbytes/s I0905 11:09:18.324606 585 Sender.cpp:507] Connection took 1 attempt(s) and 0.2235 seconds. port 22362 I0905 11:09:18.325098 579 Sender.cpp:507] Connection took 1 attempt(s) and 0.226289 seconds. port 22356 I0905 11:09:18.325328 584 Sender.cpp:507] Connection took 1 attempt(s) and 0.224259 seconds. port 22361 I0905 11:09:18.326326 580 Sender.cpp:507] Connection took 1 attempt(s) and 0.227023 seconds. port 22357 I0905 11:09:18.326859 583 Sender.cpp:507] Connection took 1 attempt(s) and 0.226104 seconds. port 22360 I0905 11:09:18.332675 581 Sender.cpp:507] Connection took 1 attempt(s) and 0.233638 seconds. port 22358 [ ] 0% 0.0 0.0 Mbytes/s E0905 11:09:18.791946 586 SocketUtils.cpp:124] non-retryable error encountered during socket io 5 64197 2: Broken pipe [32] E0905 11:09:18.792007 586 SocketUtils.cpp:124] non-retryable error encountered during socket io 5 0 0: Broken pipe [32] E0905 11:09:18.792027 586 Sender.cpp:1214] Write error 64197 (262144). fd = 5. port = 22363: Broken pipe [32] I0905 11:09:18.792060 586 Sender.cpp:722] entered CHECK_FOR_ABORT state 7 I0905 11:09:18.792086 586 Sender.cpp:973] entered PROCESS_ABORT_CMD state 7 W0905 11:09:18.792109 586 Sender.cpp:996] Received abort on 7 remote protocol version 16 remote error code ID_MISMATCH file 100MB-singapore.bin checkpoint 0 W0905 11:09:18.792124 586 WdtBase.cpp:223] Setting the abort code 12 I0905 11:09:18.792139 586 Sender.cpp:1129] Port 22363 done. Transfer status (local) = ABORT, (remote) = ID_MISMATCH. Number of blocks transferred = 0. Data Mbytes = 0. Header kBytes = 0.0595703 (100% overhead). Total bytes = 64258. Wasted bytes due to failure = 64197 (100% overhead). Total throughput = 0 Mbytes/sec E0905 11:09:18.792309 583 SocketUtils.cpp:124] non-retryable error encountered during socket io 6 61277 2: Broken pipe [32] E0905 11:09:18.792340 583 Sender.cpp:1230] Transfer aborted during block transfer 22360 100MB-singapore.bin E0905 11:09:18.792361 583 Sender.cpp:1116] Transfer aborted 22360 ID_MISMATCH I0905 11:09:18.792377 583 Sender.cpp:1129] Port 22360 done. Transfer status (local) = ABORT, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header kBytes = 0.0585938 (100% overhead). Total bytes = 61337. Wasted bytes due to failure = 61277 (100% overhead). Total throughput = 0 Mbytes/sec E0905 11:09:18.792641 582 SocketUtils.cpp:124] non-retryable error encountered during socket io 4 61280 2: Broken pipe [32] E0905 11:09:18.792688 582 Sender.cpp:1230] Transfer aborted during block transfer 22359 100MB-singapore.bin E0905 11:09:18.792701 582 Sender.cpp:1116] Transfer aborted 22359 ID_MISMATCH I0905 11:09:18.792711 582 Sender.cpp:1129] Port 22359 done. Transfer status (local) = ABORT, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header kBytes = 0.0556641 (100% overhead). Total bytes = 61337. Wasted bytes due to failure = 61280 (100% overhead). Total throughput = 0 Mbytes/sec [ ] 0% 0.0 0.0 Mbytes/s E0905 11:09:18.810423 581 SocketUtils.cpp:143] transfer aborted during socket io 9 1 2 E0905 11:09:18.810451 581 Sender.cpp:1116] Transfer aborted 22358 ID_MISMATCH I0905 11:09:18.810461 581 Sender.cpp:1129] Port 22358 done. Transfer status (local) = ABORT, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header kBytes = 0.271484 (100% overhead). Total bytes = 278. Wasted bytes due to failure = 0 (100% overhead). Total throughput = 0 Mbytes/sec E0905 11:09:18.811182 585 SocketUtils.cpp:143] transfer aborted during socket io 8 42297 2 E0905 11:09:18.811213 585 Sender.cpp:1230] Transfer aborted during block transfer 22362 100MB-singapore.bin E0905 11:09:18.811236 585 Sender.cpp:1116] Transfer aborted 22362 ID_MISMATCH I0905 11:09:18.811250 585 Sender.cpp:1129] Port 22362 done. Transfer status (local) = ABORT, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header kBytes = 0.0585938 (100% overhead). Total bytes = 42357. Wasted bytes due to failure = 42297 (100% overhead). Total throughput = 0 Mbytes/sec E0905 11:09:18.814739 580 SocketUtils.cpp:124] non-retryable error encountered during socket io 3 61277 2: Broken pipe [32] E0905 11:09:18.814767 580 Sender.cpp:1230] Transfer aborted during block transfer 22357 100MB-singapore.bin E0905 11:09:18.814781 580 Sender.cpp:1116] Transfer aborted 22357 ID_MISMATCH I0905 11:09:18.814790 580 Sender.cpp:1129] Port 22357 done. Transfer status (local) = ABORT, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header kBytes = 0.0585938 (100% overhead). Total bytes = 61337. Wasted bytes due to failure = 61277 (100% overhead). Total throughput = 0 Mbytes/sec E0905 11:09:18.821446 584 SocketUtils.cpp:143] transfer aborted during socket io 10 42297 2 E0905 11:09:18.821491 584 Sender.cpp:1230] Transfer aborted during block transfer 22361 100MB-singapore.bin E0905 11:09:18.821512 584 Sender.cpp:1116] Transfer aborted 22361 ID_MISMATCH I0905 11:09:18.821527 584 Sender.cpp:1129] Port 22361 done. Transfer status (local) = ABORT, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header kBytes = 0.0585938 (100% overhead). Total bytes = 42357. Wasted bytes due to failure = 42297 (100% overhead). Total throughput = 0 Mbytes/sec [ ] 0% 0.0 0.0 Mbytes/s E0905 11:09:18.935071 579 SocketUtils.cpp:143] transfer aborted during socket io 7 52517 2 E0905 11:09:18.935142 579 Sender.cpp:1230] Transfer aborted during block transfer 22356 100MB-singapore.bin E0905 11:09:18.935178 579 Sender.cpp:1116] Transfer aborted 22356 ID_MISMATCH I0905 11:09:18.935196 579 Sender.cpp:1129] Port 22356 done. Transfer status (local) = ABORT, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header kBytes = 0.0585938 (100% overhead). Total bytes = 52577. Wasted bytes due to failure = 52517 (100% overhead). Total throughput = 0 Mbytes/sec I0905 11:09:18.935250 579 Sender.cpp:1098] Last thread finished 0.837375 I0905 11:09:18.935696 576 Reporting.cpp:114] Could not send all the bytes 104857600 0 [ ] 0% 0.0 0.0 Mbytes/s
I0905 11:09:18.935737 576 Sender.cpp:348] Total sender time = 0.83739 seconds (0.000807822 dirTime). Transfer summary : Transfer status (local) = ERROR, (remote) = ID_MISMATCH. Number of blocks transferred = 0. Data Mbytes = 0. Header kBytes = 0.679688 (100% overhead). Total bytes = 385838. Wasted bytes due to failure = 385142 (100% overhead). All files failed. Total sender throughput = 0 Mbytes/sec (0 Mbytes/sec pure transfer rate) root@localhost:/relay/dev#

opened by nxtreaming 17
Execute cmake pathtowdtsrcdir -DBUILD_TESTING=on ，，，error

-- Boost version: 1.64.0 -- Found the following Boost libraries: -- system -- filesystem -- Configuring done CMake Error in CMakeLists.txt: Target "option_type_test_short_flags" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "option_type_test_long_flags" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdt_gen_files" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdt4tests" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdtbenchtestslib" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdtbenchlib" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "protocol_test" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdtbin" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdt" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "folly4wdt" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "file_reader_test" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdt4tests_min" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdt_url_test" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdt_gen_stats" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdt_gen_test" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "test_encdeci64_func" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdt_fd_test" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "test_stats" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "histogram" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdt_min" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "resource_controller_test" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "wdt_misc_tests" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

CMake Error in CMakeLists.txt: Target "encryption_test" requires the language dialect "CXX14" (with compiler extensions), but CMake does not know the compile flags to use to enable it.

-- Generating done -- Build files have been written to: /usr/local/src/wdt

opened by shanwu888 13
IP Connection Fails when given an Ipv4 instead of a name as input and --ipv6 is true (default)

I get the following error when trying to connect to the receiver via IP address.

E0729 17:08:27.555832 26484736 ClientSocket.cpp:49] Failed getaddrinfo 192.168.2.2 , 22363 : 8 : nodename nor servname provided, or not known

I thought this was a DNS lookup issue, but that doesn't seem to be the case. Adding the address to my host file and using the specified hostname resulted in the same error.

Sender is OS X 10.9.5 Receiver is 10.10.2

Direct connection via ethernet.

opened by oblogic7 10

Build failure on macOs 10.12

Hey, I am trying to build wdt on osx 10.12, I followed the instructions for osx. I tried building with both Xcode & Unix Make Files I see the same error:

[ 51%] Performing configure step for 'gtest'
In file included from /Users/tckb/Documents/Work/wdt/SenderThread.cpp:9:
/Users/tckb/Documents/Work/wdt/../wdt/SenderThread.h:86:31: error: no member named 'to' in namespace 'folly'
    threadStats_.setId(folly::to<std::string>(threadIndex_));
                       ~~~~~~~^
/Users/tckb/Documents/Work/wdt/../wdt/SenderThread.h:86:45: error: expected '(' for function-style cast or type construction
    threadStats_.setId(folly::to<std::string>(threadIndex_));
                                 ~~~~~~~~~~~^
In file included from /Users/tckb/Documents/Work/wdt/Sender.cpp:11:
/Users/tckb/Documents/Work/wdt/../wdt/SenderThread.h:86:31: error: no member named 'to' in namespace 'folly'
    threadStats_.setId(folly::to<std::string>(threadIndex_));
                       ~~~~~~~^
/Users/tckb/Documents/Work/wdt/../wdt/SenderThread.h:86:45: error: expected '(' for function-style cast or type construction
    threadStats_.setId(folly::to<std::string>(threadIndex_));
                                 ~~~~~~~~~~~^

Am I missing something here?

opened by tckb 7

build failed, help me, please

gcc version

gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

cmake version

cmake version 3.10.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

error message

opened by foreverneverer 6
Option to compile wdt against libfolly rather than building libfolly4wdt subset

Having a subset of folly build with wdt makes packaging the library and utils difficult with package managers. Is it possible / supported to build against a full version of folly?

Relevant to https://github.com/Homebrew/homebrew-core/pull/39345

TITLE WAS: "Could folly (complete) be used rather than folly4wdt?"

opened by hjmallon 6
std not detected

In file included from /home/nasrinj/wdt/../wdt/util/CommonImpl.h:13:0, from /home/nasrinj/wdt/../wdt/util/FileCreator.h:13, from /home/nasrinj/wdt/util/FileCreator.cpp:9: /home/nasrinj/wdt/../wdt/Reporting.h: In constructor ‘facebook::wdt::TransferStats::TransferStats(bool)’: /home/nasrinj/wdt/../wdt/Reporting.h:110:16: error: ‘make_unique’ is not a member of ‘std’ mutex_ = std::make_uniquefolly::RWSpinLock();

I am not sure why I get this error while trying to make.

I followed all the steps from the BUILD.md file. My glog install had failed too with the following error :

/home/nasrinj/glog/src/googletest.h:93: undefined reference to google::FlagRegisterer::FlagRegisterer<std::string>(char const*, char const*, char const*, std::string*, std::string*)' /home/nasrinj/glog/src/googletest.h:94: undefined reference togoogle::FlagRegisterer::FlagRegistererstd::string(char const*, char const*, char const*, std::string*, std::string*)' /home/nasrinj/glog/src/googletest.h:96: undefined reference to google::FlagRegisterer::FlagRegisterer<bool>(char const*, char const*, char const*, bool*, bool*)' /home/nasrinj/glog/src/googletest.h:100: undefined reference togoogle::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, int*, int*)'

Are these errors related in someway?

opened by NasrinJaleel93 6
Waiting for PROCESS_WAIT_CMD state takes long time

A question here, I had a 120MB single file, and I used WDT to transfer it between two physical servers. This took about 30 seconds for sender and receiver finished. The puzzle is that I observed the whole file already transferred to the disk, but the sender side went into entered PROCESS_WAIT_CMD state for literally 25 seconds. What the thing holds one of thread takes so long to get out this state? This behavior sound defend the purpose of fast data transfer.

The log snippets: I1214 16:47:11.470233 23157 DirectorySourceQueue.cpp:201] wdt> Using list of file info. Number of files 1 I1214 16:47:11.504415 23158 SenderThread.cpp:85] wdt> Thread[0, port: 4430] Connection took 1 attempt(s) and 0.0341567 seconds. port 4430 I1214 16:47:11.505105 23209 SenderThread.cpp:85] wdt> Thread[1, port: 4431] Connection took 1 attempt(s) and 0.0326994 seconds. port 4431 I1214 16:47:11.505519 23216 SenderThread.cpp:85] wdt> Thread[3, port: 4433] Connection took 1 attempt(s) and 0.0329971 seconds. port 4433 I1214 16:47:11.505672 23220 SenderThread.cpp:85] wdt> Thread[4, port: 4434] Connection took 1 attempt(s) and 0.0330901 seconds. port 4434 I1214 16:47:11.507572 23211 SenderThread.cpp:85] wdt> Thread[2, port: 4432] Connection took 1 attempt(s) and 0.0351163 seconds. port 4432 I1214 16:47:11.507681 23224 SenderThread.cpp:85] wdt> Thread[5, port: 4435] Connection took 1 attempt(s) and 0.0350508 seconds. port 4435 I1214 16:47:11.507918 23229 SenderThread.cpp:85] wdt> Thread[6, port: 4436] Connection took 1 attempt(s) and 0.035202 seconds. port 4436 I1214 16:47:11.508317 23230 SenderThread.cpp:85] wdt> Thread[7, port: 4437] Connection took 1 attempt(s) and 0.035554 seconds. port 4437 I1214 16:47:14.219935 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:15.220541 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:16.220765 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:17.221560 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:18.221844 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:19.222250 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:20.222795 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:21.223304 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:22.223743 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:23.224333 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:24.225175 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:25.225862 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:26.226440 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:27.226626 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:28.227392 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:29.227818 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:30.228229 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:31.228562 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:32.229580 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:33.229594 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:34.230316 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:34.685650 23224 SenderThread.cpp:712] wdt> Thread[5, port: 4435] entered PROCESS_WAIT_CMD state I1214 16:47:35.230864 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:35.686082 23224 SenderThread.cpp:712] wdt> Thread[5, port: 4435] entered PROCESS_WAIT_CMD state I1214 16:47:36.231479 23230 SenderThread.cpp:712] wdt> Thread[7, port: 4437] entered PROCESS_WAIT_CMD state I1214 16:47:36.455055 23209 SenderThread.cpp:922] wdt> Thread[1, port: 4431] Port 4431 done. Transfer status = OK. Number of blocks transferred = 1. Data Mbytes = 16. Header Kbytes = 0.334961 (0.00204444% overhead). Total bytes = 16777559. Wasted bytes due to failure = 0 (0% overhead). Encryption type = none. Total throughput = 0.640447 Mbytes/sec:

opened by zxpan 6
build failed

[100%] Linking CXX executable _bin/wdt/wdt /usr/bin/ld: libwdt_min.so.1.32.1910230: undefined reference to folly::detail::annotate_rwlock_acquired_v' /usr/bin/ld: libwdt_min.so.1.32.1910230: undefined reference tofolly::detail::annotate_rwlock_released_v' collect2: error: ld returned 1 exit status make[2]: *** [CMakeFiles/wdtbin.dir/build.make:132：_bin/wdt/wdt] 错误 1 make[1]: *** [CMakeFiles/Makefile2:169：CMakeFiles/wdtbin.dir/all] 错误 2 make: *** [Makefile:136：all] 错误 2

opened by bolunfeng 0
OSX Catalina Build Error Workarounds

I had a string of puzzles I had to solve to get this running on my MAC (ubuntu 18 and 20, no problem). They took quite some time to figure out, so i added them to the troubleshooting section. If the info better belongs elsewhere, feel free to do with it as you see fit. I'm just hoping to save some people time I already spent :-)

Thanks for the package- it rocks.
CLA Signed

opened by iamh2o 3
Build instructions on building glog from source are no longer correct
https://github.com/google/glog no longer contains the 'configure' script, but instead provides instructions for cmake to build and install.

i.e.

cd glog ./configure # add --with-gflags=whereyouinstalledgflags # to avoid ERROR: unknown command line flag 'minloglevel' later make -j && sudo make install

is incorrect unless you're using an older version of glog. glog-0.3.5 still has the configure script and works without problem.
opened by CaptJak 0
Socket timeout issue

I ran my WDT between two dockers, both being in host network. I used the following ssh command :

sshpass -p password ssh -o StrictHostKeyChecking=no -p 52022 -o UserKnownHostsFile=/dev/null [email protected] wdt -directory / -transfer_id f115f8bca955db40c614b8b9d9efe45c -start_port 22372 -overwrite=true | wdt -directory / -destination 192.168.99.99 -transfer_id f115f8bca955db40c614b8b9d9efe45c -start_port 22372 -manifest /tmp/tmp7s6rlqa9.tmp -overwrite=true

There seems to be an issue with the sockets, It worked fine in test but for some reason, I cannot get it to work in production:

See logs here : wdt_error.txt

opened by HugoHumbert 0
Any plans to make new release?

I think that it would be good to flush currently committed changes and make new release :) Last release was in 2016 and since then has been added +170 commits.

opened by kloczek 3

fatal error: 'wdt/Wdt.h' file not found

Build fails (rev.3573f1c):

[1/35] /usr/bin/c++ -DSTANDALONE_APP -I/usr/local/include -I. -I/usr/ports/net/wdt/work/wdt-1.27.1612021-171-g3573f1c/.. -msse4.2 -mpclmul -O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -fno-strict-aliasing -fno-omit-frame-pointer -std=gnu++14 -MD -MT CMakeFiles/wdtbin.dir/Wdt.cpp.o -MF CMakeFiles/wdtbin.dir/Wdt.cpp.o.d -o CMakeFiles/wdtbin.dir/Wdt.cpp.o -c /usr/ports/net/wdt/work/wdt-1.27.1612021-171-g3573f1c/Wdt.cpp
FAILED: CMakeFiles/wdtbin.dir/Wdt.cpp.o 
/usr/bin/c++ -DSTANDALONE_APP -I/usr/local/include -I. -I/usr/ports/net/wdt/work/wdt-1.27.1612021-171-g3573f1c/.. -msse4.2 -mpclmul -O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -fno-strict-aliasing -fno-omit-frame-pointer -std=gnu++14 -MD -MT CMakeFiles/wdtbin.dir/Wdt.cpp.o -MF CMakeFiles/wdtbin.dir/Wdt.cpp.o.d -o CMakeFiles/wdtbin.dir/Wdt.cpp.o -c /usr/ports/net/wdt/work/wdt-1.27.1612021-171-g3573f1c/Wdt.cpp
/usr/ports/net/wdt/work/wdt-1.27.1612021-171-g3573f1c/Wdt.cpp:1:10: fatal error: 'wdt/Wdt.h' file not found
#include <wdt/Wdt.h>
         ^~~~~~~~~~~
1 error generated.

It seems to depend on itself being installed?

opened by yurivict 5

Releases(v1.27.1612021)

v1.27.1612021(Dec 5, 2016)
Long time since the previous release (I hope you enjoy github master / trunk which we keep stable!) with lots of changes:

Common "wdt> " prefix in all the logging (helps finding/filtering wdt specific logs when embedded in another service)

Progress report during file discovery

Throttler changes working with the resource controller and can be lowered in the middle of a transfer and improved fairness.

Multiple applications support within one process, Wdt options not forced to be a singleton, apply to a tree of sub object below the application wdt object

Allow receiver to specify its --hostname

New --fsync option to fsync after the last block of each file is written.

Incremental transfers (-enable_download_resumption) improved reporting

Wdt URI encoding/decoding is now binary safe (transferid can be any binary data)

Fixed disk mode (files transferred serially and not cut in blocks) >2Gb errors; integer overflow

Fewer dependencies. Uses std::make_uniq (need cxx14)

Receiver now drives Sender's download resumption through the URI

Benchmark generation (bench/) code

New more efficient Varint encoding and less likely to have memory corruption

Bug fixes (see commits) like listening sockets staying open after receiver timeout

Misc changes like: Release/build improvements, Code quality improvements like constructor cleanups (#16)

Enjoy!
Source code(tar.gz)
Source code(zip)
v1.26.1603040(Mar 5, 2016)
Option to use fadvise(POSIX_FADVISE_DONTNEED) to avoid taking too much cache/buffers on sender/read path

other fixes/improvements

Simpler/shorter tests

Source code(tar.gz)
Source code(zip)
v1.26.1602090(Feb 9, 2016)
All of the goodies in v1.25 plus:

Support for deletion during resumption through the delete_extra_files - if a subsequent snapshot has fewer files than were already sent before, delete those first. This is useful for use cases like rocksdb/myrocks where after compaction some files disappear

Dynamic ports (available ports) is now the default, you don't need to specify -start_port 0 anymore for the receiver side of wdt, if you want to revert to fixed ports, use -static_ports or set a non 0 start_port explicitly. This should make it easier for wdt transfers using pipe to connect the receiver and sender to always succeed even if another transfer is already ongoing on the same destination

Source code(tar.gz)
Source code(zip)
v1.25.1602051(Feb 8, 2016)
Long time since the previous release (but there is always github master!) - lot's of changes:

New features/major changes:

Much better command line interface to match the need of the encryption key transmission securely, "-" argument for the sender reads the connection url from stdin, which the receiver emits on stdout. So the 2 can be plugged through a pipe (securely using ssh - which will only be used for secure key exchange, everything else blazing fast and encrypted through WDT's own socket)

ssh dsthost wdt -directory destdir | ssh srchost wdt -directory srcdir - # Or more complex: (ssh localhost wdt -fork -start_port 0 -directory /tmp/dst1; ls *.h ) | wdt -manifest - -

Aes GCM crypto support (requires a recent openssl version) : is now the default - it does both encryption and validates the integrity in 1 pass. We also support this during error handling only resuming at the last block that was successfully validated.

Socket buffer size can be set explicitly (using -send_buffer_size and -receive_buffer_size)

Bug fixes:

improved TransferLogManager locking (fail fast when can't be acquired) and detection of deleted containing directory/log file

ResourceController double create, wdtSend can cancel previous / preexisting transfer

with opening files during discovery

Misc:

Consistent use of "Mbytes" throughout WDT for 1024*1024 = 1048576 bytes (aka Mebibyte (MiB)

Testing/path improvements

Removing direct use of WdtOptions::get across the code base - now the options can be different for different sender/receivers in the same process.

Source code(tar.gz)
Source code(zip)
v1.23.1511300(Dec 1, 2015)
Big release !

New features:

WDT now supports encryption (aes128)

The receiver is generating a one time 128bits secret, encoded in the URL. This secret needs to be transmitted securely (at facebook we use secure thrift for this but you can use ssh for instance). Note that we do not try to authenticate and only encrypt.

Other improvements

Also estimate the time it may take for a slow receiver to consume the enqueued bytes to avoid timing out waiting for the ack to the end of transfer

There is a new https://github.com/facebook/wdt/blob/master/Wdt.h simpler API for C++ (which is work in progress but shows the direction we are headed)

Bug fixes:

file descriptor (in FileInfo api) wasn't used when download resumption was also on, locking , error code summary, ...

Known issues:

Issue #102
Source code(tar.gz)
Source code(zip)
v1.22.1510270(Oct 28, 2015)

Important bug fixes (notification change in big refactor was incorrect. issue with long running mode,...)

Changed code layout, moved to <wdt/X.h> from "X.h"

Clang-3.7 warnings

Summarize the error codes for the sender (more specific than ERROR/OK, for instance ABORT is differentiated from CONN_ERROR)
Source code(tar.gz)
Source code(zip)
v1.22.1510210(Oct 21, 2015)
New Features

Refactored the code to split Sender and Receiver into clearly identifiable single threaded code and multi-threaded part (SenderThread / ReceiverThread).

Wdt now has a thread controller, with thread primitives such as Barrier and Funnel Executors which made the writing of certain features of sender and receiver easier like doing process version mismatch, or sending file chunks in download resumption.

wdt_max_send_test.sh has been updated and is more robust. This test can be used to benchmark wdt and detect regressions.

Bugs fixed

A sender thread could fail on a connection and have a file in its thread transfer history that has not been acknowledged, however it wouldn't stop other threads to finish their part of the transfer even on getting the checkpoint, since the last source would not return to queue. This would result in the final status of transfers being error.

Transfer log manager had a writer thread that would write empty buffer to the transfer log file.

gcc4.9 SIOPF fix for one our static strings in GFlags.

Source code(tar.gz)
Source code(zip)
v1.21.1510050(Oct 6, 2015)

Changed the retry logic to be based on lack of progress instead of being arbitrarily attributed to files retry counts

Fixed serious int overflow in lseek result handling - files between 2 and 4g (and 6 and 8g and 10-12...) ie whose size has bit 31 set and being set using block mode would fail

Removed the "e"xit command on receiver (dated from old times where the receiver would run forever by default)

Download resumption (TransferLogManager), thread history, checkpoints cleanup and important bug fixes
Source code(tar.gz)
Source code(zip)
v1.20.1509241(Sep 26, 2015)

Really fixed URL backward compatibility (use -url_backward_compatibility=true if you have a wdt v20 talking to v18 or earlier)

new -exit_on_bad_flags=false if you want to continue despite bad command line (useful for forward compatibility)

shorter urls (wdt://host:port?num_port=4) and correct escaping of ipv6 [addr]

wdt receiver will now not overwrite any file by default unless you add -overwrite or use download resumption

misc bug fixes and cleanup
Source code(tar.gz)
Source code(zip)
v1.19.1509200(Sep 21, 2015)
Changes since previous tag (Sept 16) :

Added fd to FileInfo. If fd, is provided, we use that fd and do not open the file. I am currently using pread always. I have to more perf test for pread. Max send is really slow in my new dev server, so need to test for perf in some other machine.

If connection url was provided, sender ignored file list. Fixed that and added a test for file info list

Added an option in e2e simple test to test o_direct mode

Simplifying o_direct code. We should never define macros named O_DIRECT.

Source code(tar.gz)
Source code(zip)
v1.19.1509160(Sep 17, 2015)

fixes #73 (major version negotiation issue with URI/URL. v19 command line will work with any previous version but v18 for instance would crash v17 etc...). If using the C++ api directly and checking the errorCode of TransferRequest you need to use v20 and url_backward_compatibility=true option.

reset the connection timeout back to 1s (for high latency links)
Source code(tar.gz)
Source code(zip)
v1.18.1509120(Sep 13, 2015)

Notable changes from previous version:

WDT can now resume interrupted transfer based solely on file size based an examining the directory tree (as opposed to previously using it's own transfer log)

If you are using wdt on single disk systems use that mode and disable blocks mode, if you are using flash continue to use the transfer log and the block mode which are most effectives when doing simultaneous IOs is not expensive (like flash, unlike single disk systems)
Source code(tar.gz)
Source code(zip)
v1.17.1509100(Sep 11, 2015)

Notable changes from previous version:

WDT now correctly binds to multiple sockets for hosts with both IPv4 and IPv6 addresses

Bug fixes and more/better tests
Source code(tar.gz)
Source code(zip)
v1.16.1508311(Sep 1, 2015)

First github release (should have started tagging earlier but let's start now)

No major known bugs

Our versioning scheme is vMAJOR.MINOR.BUILD where BUILD is YYMMDDP = date + build index starting at 0 (first build of the day) up to 9 (10th build of the day)
Source code(tar.gz)
Source code(zip)
v1.16.1508310(Sep 1, 2015)

shell script problem in new test fixed in next build v1.16.1508311
Source code(tar.gz)
Source code(zip)