libTorrent BitTorrent library

According to the unfixed rtorrent bug #386, there is another big problem with libtorrent announcing. If one single user is seeding a high number of torrents and restarts his rtorrent, it results in massive announce hammering.

First the scenario: The tracker sends interval and min interval to rtorrent. The interval will be ignored (see issue #386 rtorrent), but rtorrent should actually use min interval of course. Let's say the user has to restart his rtorrent with 5000 torrents. 5000 torrents announce, get the random interval and are supposed to use that interval for the next announce. But that's not working, because of how the interval to use is calculated by libtorrent. So all 5000 torrents will announce at the same time again instead of what they've been told by the tracker. That comes close to a DoS attack.

How to fix this: Here is a different version of tracker_next_timeout_promiscuous() from tracker_controller.cc which does respect what the tracker sent:

uint32_t tracker_next_timeout_promiscuous(Tracker *tracker) {
	if ((tracker->is_busy() && tracker->latest_event() != Tracker::EVENT_SCRAPE) || !tracker->is_usable())
		return ~uint32_t();

	int32_t interval, min_interval, use_interval, since_last;

	if (tracker->failed_counter())
		interval = 5 << std::min<int>(tracker->failed_counter() - 1, 6);
	else
		interval = tracker->normal_interval();

	min_interval = tracker->min_interval();

	if (min_interval < 300)
		min_interval = 300;

	if (interval < min_interval)
		use_interval = min_interval;
	else
		use_interval = interval;

	since_last = cachedTime.seconds() - (int32_t)tracker->activity_time_last();
	return std::max(use_interval - since_last, 0);
}

After that it works as it should. I hope, that will be fixed soon, because it's a big problem for trackers with users who seed thousands of torrents. Also most shared seedbox providers use rtorrent and the users can't apply changes to libtorrent.

Edit:// This bug only appears, if the min interval is set over 30 minutes. Below 30 minutes it works, but I don't know why any tracker should set the min interval below that value, because rtorrent ignores the default interval anyway.

libtorrent fails to compile with clang when clang is using libc++ as its C++ stdlib (for example, on Mac OS X 10.9 where that is the default). The failure comes from libtorrent using the TR1 prefixed versions of a few headers; libc++ includes only the C++11 versions of these, not the TR1 versions.

You can see the build logs here: https://gist.github.com/mistydemeo/7271614 I also notice that configure says:

checking for C++11 support... no

Which isn't correct. Seems to come from this in the configure test for C++11 support:

conftest.cpp:32:24: error: union member 'b1' has a non-trivial copy constructor
union test { Bar b1; };
^
/Library/Developer/CommandLineTools/usr/bin/../lib/c++/v1/unordered_map:733:5: note: because type 'std::__1::unordered_map<Foo *, int, std::__1::hash<Foo *>, std::__1::equal_to<Foo *>, std::__1::allocator<std::__1::pair<Foo *const, int> > >' has a user-provided copy constructor
unordered_map(const unordered_map& __u);
^
1 error generated.

$ make /Library/Developer/CommandLineTools/usr/bin/make all-recursive Making all in src Making all in torrent Making all in data /bin/sh ../../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../../../src/torrent/data -I../../.. -I../../../../src/torrent/data -I../../../../src/torrent/data/.. -I../../../../src/torrent/data/../.. -I../../../.. -I/usr/include -I/usr/include -D_THREAD_SAFE -g -O2 -g -DDEBUG -Wall -std=c++11 -fvisibility=hidden -MT block.lo -MD -MP -MF .deps/block.Tpo -c -o block.lo ../../../../src/torrent/data/block.cc libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../../../../src/torrent/data -I../../.. -I../../../../src/torrent/data -I../../../../src/torrent/data/.. -I../../../../src/torrent/data/../.. -I../../../.. -I/usr/include -I/usr/include -D_THREAD_SAFE -g -O2 -g -DDEBUG -Wall -std=c++11 -fvisibility=hidden -MT block.lo -MD -MP -MF .deps/block.Tpo -c ../../../../src/torrent/data/block.cc -fno-common -DPIC -o .libs/block.o In file included from ../../../../src/torrent/data/block.cc:44: In file included from ../../../../src/torrent/data/../../protocol/peer_connection_base.h:46: In file included from ../../../../src/torrent/data/../../protocol/encryption_info.h:40: ../../../../src/torrent/data/../../utils/rc4.h:48:10: fatal error: 'openssl/rc4.h' file not found #include <openssl/rc4.h> ^ 1 error generated. make[4]: *** [block.lo] Error 1 make[3]: *** [all-recursive] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2

Couple of our users reported that they can't compile rtorrent-ps-ch on newer distros (not just on a particular one), one of them, @colinhd8 even came up with an unexpected fix: unset the CXXFLAGS env var in our script. The only common thing amongst those distros is the gcc compiler version, which is v6.x .

Since I wasn't sure where the actual issue relies (in our automagic function, other part of our script, etc.) I've created a minimal build script on fresh install of Debian 9 (using gcc v6.3.0) to see whether rtorrent can be built or not.

Conclusion: if the CXXFLAGS env var is set (even when it's empty string) building "fails" at the end of compilation process only when rtorrent tries to link libtorrent.so :

../rak/priority_queue_default.h: In destructor ‘rak::priority_item::~priority_item()’:
../rak/priority_queue_default.h:56:95: warning: throw will always call terminate() [-Wterminate]
	   throw torrent::internal_error("priority_item::~priority_item() called on a queued item.");
																							   ^
../rak/priority_queue_default.h:56:95: note: in C++11 destructors default to noexcept
mv -f .deps/main.Tpo .deps/main.Po
/bin/bash ../libtool  --tag=CXX   --mode=link g++  -g -DDEBUG -Wall   -I/home/test/lib/rtorrent-0.9.7-1.5.1/include  -I/home/test/lib/rtorrent-0.9.7-1.5.1/include   -L/home/test/lib/rtorrent-0.9.7-1.5.1/lib -o rtorrent main.o libsub_root.a ui/libsub_ui.a core/libsub_core.a display/libsub_display.a input/libsub_input.a rpc/libsub_rpc.a utils/libsub_utils.a  -lncursesw -lcppunit -L/home/test/lib/rtorrent-0.9.7-1.5.1/lib -lcurl -ltorrent -lpthread -L/home/test/lib/rtorrent-0.9.7-1.5.1/lib   -lxmlrpc_server -lxmlrpc  -lxmlrpc_xmlparse -lxmlrpc_xmltok -lxmlrpc_util -lpthread 
libtool: link: g++ -g -DDEBUG -Wall -I/home/test/lib/rtorrent-0.9.7-1.5.1/include -I/home/test/lib/rtorrent-0.9.7-1.5.1/include -o rtorrent main.o  -L/home/test/lib/rtorrent-0.9.7-1.5.1/lib libsub_root.a ui/libsub_ui.a core/libsub_core.a display/libsub_display.a input/libsub_input.a rpc/libsub_rpc.a utils/libsub_utils.a -lncursesw -lcppunit /home/test/lib/rtorrent-0.9.7-1.5.1/lib/libcurl.so /home/test/lib/rtorrent-0.9.7-1.5.1/lib/libtorrent.so -lxmlrpc_server -lxmlrpc -lxmlrpc_xmlparse -lxmlrpc_xmltok -lxmlrpc_util -lpthread -Wl,-rpath -Wl,/home/test/lib/rtorrent-0.9.7-1.5.1/lib -Wl,-rpath -Wl,/home/test/lib/rtorrent-0.9.7-1.5.1/lib
/home/test/lib/rtorrent-0.9.7-1.5.1/lib/libtorrent.so: undefined reference to `torrent::PeerConnection<(torrent::Download::ConnectionType)1>::offer_chunk()'
/home/test/lib/rtorrent-0.9.7-1.5.1/lib/libtorrent.so: undefined reference to `torrent::PeerConnection<(torrent::Download::ConnectionType)0>::offer_chunk()'
collect2: error: ld returned 1 exit status

The same can be achieved earlier when rtorrent is configured with xmlrpc suppport: this time the checking macro fails at the biginning of building process of rtorrent: resulting the very same error message as above in config.log.

In summary:

• other env vars we use aren't affected by this only CXXFLAGS
• it's distro independent
• it only depends on the version of gcc: 4.x , 5.x work fine (I'm not sure about 7.x , like in newer Arch Linux)
• both master and feature-bind branches are affected
• strangly enough 0.9.6/0.13.6 is NOT affected by this

This is a really nasty bug, e.g. in Gentoo such env vars can be defined system wide in /etc/make.conf. And that means no additional flags can be set in this way. I quickly went through the configure.ac and all the m4 macros in scripts dir but I didn't see anything unusual. More than that almost all the marcos are identical in rtorrent and libtorrent projects: so I'm not sure at all that rtorrent isn't affected by this, since libtorrent builds fine seemingly until it's not linked!

Maybe it would be worth to modify Travis scripts to:

• include newer compilers: 4.x, 5.x, 6.x, 7x
• export these empty env vars at the beginning of the script

Example distros that are affected:

• Debian 9, gcc 6.3
• Fedora 24 x64 GCC:6.3.1 20161221 (Red Hat 6.3.1-1) (GCC)
• CentOs 7 x64 GCC:gcc version 6.2.1 20160916 (Red Hat 6.2.1-3) (GCC)
• Ubuntu Server 17.04, gcc 6.3.0
• ViodLinux gcc 6.3

libtorrent builds fine, but the tests don't build.

Log: Making check in src make[1]: Entering directory '/home/ingmar/Repositories/libtorrent/src' Making check in torrent make[2]: Entering directory '/home/ingmar/Repositories/libtorrent/src/torrent' Making check in data make[3]: Entering directory '/home/ingmar/Repositories/libtorrent/src/torrent/data' make[3]: Nothing to be done for 'check'. make[3]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/torrent/data' Making check in download make[3]: Entering directory '/home/ingmar/Repositories/libtorrent/src/torrent/download' make[3]: Nothing to be done for 'check'. make[3]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/torrent/download' Making check in peer make[3]: Entering directory '/home/ingmar/Repositories/libtorrent/src/torrent/peer' make[3]: Nothing to be done for 'check'. make[3]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/torrent/peer' Making check in utils make[3]: Entering directory '/home/ingmar/Repositories/libtorrent/src/torrent/utils' make[3]: Nothing to be done for 'check'. make[3]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/torrent/utils' make[3]: Entering directory '/home/ingmar/Repositories/libtorrent/src/torrent' make[3]: Nothing to be done for 'check-am'. make[3]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/torrent' make[2]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/torrent' Making check in data make[2]: Entering directory '/home/ingmar/Repositories/libtorrent/src/data' make[2]: Nothing to be done for 'check'. make[2]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/data' Making check in dht make[2]: Entering directory '/home/ingmar/Repositories/libtorrent/src/dht' make[2]: Nothing to be done for 'check'. make[2]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/dht' Making check in download make[2]: Entering directory '/home/ingmar/Repositories/libtorrent/src/download' make[2]: Nothing to be done for 'check'. make[2]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/download' Making check in net make[2]: Entering directory '/home/ingmar/Repositories/libtorrent/src/net' make[2]: Nothing to be done for 'check'. make[2]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/net' Making check in protocol make[2]: Entering directory '/home/ingmar/Repositories/libtorrent/src/protocol' make[2]: Nothing to be done for 'check'. make[2]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/protocol' Making check in tracker make[2]: Entering directory '/home/ingmar/Repositories/libtorrent/src/tracker' make[2]: Nothing to be done for 'check'. make[2]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/tracker' Making check in utils make[2]: Entering directory '/home/ingmar/Repositories/libtorrent/src/utils' make[2]: Nothing to be done for 'check'. make[2]: Leaving directory '/home/ingmar/Repositories/libtorrent/src/utils' make[2]: Entering directory '/home/ingmar/Repositories/libtorrent/src' make[2]: Nothing to be done for 'check-am'. make[2]: Leaving directory '/home/ingmar/Repositories/libtorrent/src' make[1]: Leaving directory '/home/ingmar/Repositories/libtorrent/src' Making check in test make[1]: Entering directory '/home/ingmar/Repositories/libtorrent/test' make LibTorrentTest make[2]: Entering directory '/home/ingmar/Repositories/libtorrent/test' /bin/sh ../libtool --tag=CXX --mode=link x86_64-pc-linux-gnu-g++ -g -O2 -DNDEBUG -fvisibility=hidden -pthread -L/usr/lib64 -lcppunit -ldl -ldl -o LibTorrentTest ../src/LibTorrentTest-thread_disk.o rak/LibTorrentTest-allocators_test.o rak/LibTorrentTest-ranges_test.o data/LibTorrentTest-chunk_list_test.o data/LibTorrentTest-hash_check_queue_test.o data/LibTorrentTest-hash_queue_test.o protocol/LibTorrentTest-test_request_list.o torrent/LibTorrentTest-http_test.o torrent/LibTorrentTest-object_test.o torrent/LibTorrentTest-object_test_utils.o torrent/LibTorrentTest-object_static_map_test.o torrent/LibTorrentTest-object_stream_test.o torrent/LibTorrentTest-tracker_controller_test.o torrent/LibTorrentTest-tracker_controller_features.o torrent/LibTorrentTest-tracker_controller_requesting.o torrent/LibTorrentTest-tracker_list_test.o torrent/LibTorrentTest-tracker_list_features_test.o torrent/LibTorrentTest-tracker_timeout_test.o torrent/utils/LibTorrentTest-log_test.o torrent/utils/LibTorrentTest-log_buffer_test.o torrent/utils/LibTorrentTest-net_test.o torrent/utils/LibTorrentTest-option_strings_test.o torrent/utils/LibTorrentTest-test_extents.o torrent/utils/LibTorrentTest-test_queue_buckets.o torrent/utils/LibTorrentTest-signal_bitfield_test.o torrent/utils/LibTorrentTest-thread_base_test.o tracker/LibTorrentTest-tracker_http_test.o LibTorrentTest-main.o ../src/libtorrent.la ../src/torrent/libsub_torrent.la ../src/torrent/data/libsub_torrentdata.la ../src/torrent/download/libsub_torrentdownload.la ../src/torrent/peer/libsub_torrentpeer.la ../src/data/libsub_data.la ../src/dht/libsub_dht.la ../src/download/libsub_download.la ../src/net/libsub_net.la ../src/protocol/libsub_protocol.la ../src/tracker/libsub_tracker.la ../src/utils/libsub_utils.la ../src/torrent/utils/libsub_torrentutils.la -lcrypto libtool: link: x86_64-pc-linux-gnu-g++ -g -O2 -DNDEBUG -fvisibility=hidden -pthread -o .libs/LibTorrentTest ../src/LibTorrentTest-thread_disk.o rak/LibTorrentTest-allocators_test.o rak/LibTorrentTest-ranges_test.o data/LibTorrentTest-chunk_list_test.o data/LibTorrentTest-hash_check_queue_test.o data/LibTorrentTest-hash_queue_test.o protocol/LibTorrentTest-test_request_list.o torrent/LibTorrentTest-http_test.o torrent/LibTorrentTest-object_test.o torrent/LibTorrentTest-object_test_utils.o torrent/LibTorrentTest-object_static_map_test.o torrent/LibTorrentTest-object_stream_test.o torrent/LibTorrentTest-tracker_controller_test.o torrent/LibTorrentTest-tracker_controller_features.o torrent/LibTorrentTest-tracker_controller_requesting.o torrent/LibTorrentTest-tracker_list_test.o torrent/LibTorrentTest-tracker_list_features_test.o torrent/LibTorrentTest-tracker_timeout_test.o torrent/utils/LibTorrentTest-log_test.o torrent/utils/LibTorrentTest-log_buffer_test.o torrent/utils/LibTorrentTest-net_test.o torrent/utils/LibTorrentTest-option_strings_test.o torrent/utils/LibTorrentTest-test_extents.o torrent/utils/LibTorrentTest-test_queue_buckets.o torrent/utils/LibTorrentTest-signal_bitfield_test.o torrent/utils/LibTorrentTest-thread_base_test.o tracker/LibTorrentTest-tracker_http_test.o LibTorrentTest-main.o -L/usr/lib64 /usr/lib64/libcppunit.so -ldl ../src/.libs/libtorrent.so ../src/torrent/.libs/libsub_torrent.a ../src/torrent/data/.libs/libsub_torrentdata.a ../src/torrent/download/.libs/libsub_torrentdownload.a ../src/torrent/peer/.libs/libsub_torrentpeer.a ../src/data/.libs/libsub_data.a ../src/dht/.libs/libsub_dht.a ../src/download/.libs/libsub_download.a ../src/net/.libs/libsub_net.a ../src/protocol/.libs/libsub_protocol.a ../src/tracker/.libs/libsub_tracker.a ../src/utils/.libs/libsub_utils.a ../src/torrent/utils/.libs/libsub_torrentutils.a /usr/lib64/../lib64/libstdc++.so -lm -lcrypto -pthread -Wl,-rpath -Wl,/usr/lib64/../lib64 torrent/utils/LibTorrentTest-log_test.o: In function utils_log_test::test_file_output()': /home/ingmar/Repositories/libtorrent/test/torrent/utils/log_test.cc:140: warning: the use oftmpnam' is dangerous, better use mkstemp' ../src/torrent/utils/.libs/libsub_torrentutils.a(log.o): In functiontorrent::log_gz_file_write(std::tr1::shared_ptrtorrent::log_gz_output&, char const_, unsigned long, int)': /home/ingmar/Repositories/libtorrent/src/torrent/utils/log.cc:377: undefined reference to gzwrite' /home/ingmar/Repositories/libtorrent/src/torrent/utils/log.cc:379: undefined reference togzwrite' /home/ingmar/Repositories/libtorrent/src/torrent/utils/log.cc:380: undefined reference to gzwrite' /home/ingmar/Repositories/libtorrent/src/torrent/utils/log.cc:383: undefined reference togzwrite' /home/ingmar/Repositories/libtorrent/src/torrent/utils/log.cc:388: undefined reference to gzwrite' ../src/torrent/utils/.libs/libsub_torrentutils.a(log.o):/home/ingmar/Repositories/libtorrent/src/torrent/utils/log.cc:386: more undefined references togzwrite' follow ../src/torrent/utils/.libs/libsub_torrentutils.a(log.o): In function log_gz_output': /home/ingmar/Repositories/libtorrent/src/torrent/utils/log.cc:76: undefined reference togzopen' ../src/torrent/utils/.libs/libsub_torrentutils.a(log.o): In function ~log_gz_output': /home/ingmar/Repositories/libtorrent/src/torrent/utils/log.cc:77: undefined reference togzclose' /home/ingmar/Repositories/libtorrent/src/torrent/utils/log.cc:77: undefined reference to gzclose' ../src/protocol/.libs/libsub_protocol.a(request_list.o): In functiontorrent::RequestList::delegate()': /home/ingmar/Repositories/libtorrent/src/protocol/request_list.cc:127: undefined reference to `torrent::Delegator::delegate(torrent::PeerChunks_, int)' collect2: error: ld returned 1 exit status Makefile:791: recipe for target 'LibTorrentTest' failed make[2]: *** [LibTorrentTest] Error 1 make[2]: Leaving directory '/home/ingmar/Repositories/libtorrent/test' Makefile:1504: recipe for target 'check-am' failed make[1]: *** [check-am] Error 2 make[1]: Leaving directory '/home/ingmar/Repositories/libtorrent/test' Makefile:486: recipe for target 'check-recursive' failed make: *** [check-recursive] Error 1

In rtorrent ce16621 some magic was added to src/thread_base.cc to migitate high load problems. A similar issue manifests on FreeBSD in the "new" libtorrent method thread_base::interrupt() in src/torrent/utils/thread_base.cc, added with commit aa9d73e4. The attached commit fixes this issue the same way as it was done in rtorrent.

Makefile:424: recipe for target 'diffie_hellman.lo' failed make[3]: *** [diffie_hellman.lo] Error 1 make[3]: *** Waiting for unfinished jobs.... mv -f .deps/instrumentation.Tpo .deps/instrumentation.Plo make[3]: Leaving directory '/tmp/libtorrent-0.13.6/src/utils' Makefile:573: recipe for target 'all-recursive' failed make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory '/tmp/libtorrent-0.13.6/src' Makefile:505: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/tmp/libtorrent-0.13.6' Makefile:414: recipe for target 'all' failed make: *** [all] Error 2

I'm getting this erro, i saw some reated issues and i need to patch, but how can i patch? Is there any step by step tutorial?

Thanks in advanced.

Im running debian 9.

First of all, thanks for adding ipv6 support! (#111) I just quickly tried it out, and I came across the following problems (both were existed with the previosly mentioned patches):

A. Can't connect to udp trackers, error message: Tracker: [Could not open UDP socket.] Used config: note, that network.bind_address is disabled to be able to use both ipv4 AND ipv6.

#network.bind_address.set = 192.168.1.6
schedule2 = ip_tick, 0, 1800, network.local_address.set=somehost.no-ip.org
network.port_range.set = 60367-60367
trackers.use_udp.set = true
dht.mode.set = auto
dht.port.set = 60378

Is the missing network.bind_address option the problem? tracker_udp.cc#L148

B. It breaks this setup: DHT not enabled. even when it is Error message: Scheduled command failed: dht_node: DHT not enabled.

Hello, the 0.13.5 release ( https://github.com/rakshasa/libtorrent/releases/tag/0.13.5 ) has the 0.13.4 version of "configure.ac" file. It causes rtorrent 0.9.5 to not configure, as it finds libtorrent 0.13.4 instead of 0.13.5. ^^

We're seeing requests from different peers using the same peer ID. This doesn't seem right, could you fix this? Note the different ports, indicating different peers. This causes tracker-side stats problems when multiple users share one IP address. How's the peer ID generated?

?info_hash=<>&peer_id=-lt0D20-9%27%96k%24%B0%DC%8F%14%CC2%F9&key=3c7e178b&ip=<>&compact=1&numwant=100&port=54946&uploaded=1221299856&downloaded=8642776720&left=0 ?info_hash=<>&peer_id=-lt0D20-9%27%96k%24%B0%DC%8F%14%CC2%F9&key=3c7e178b&ip=<>&compact=1&numwant=100&port=56056&uploaded=6619008928&downloaded=8642776720&left=0

Rather than current implentation of truncate/overwrite by default.

The only real difference from standard open_file is in the ofstream constructor. Added tests as well.

Makes using simple file names great again, and prevent rtorrent from overwriting all files on a restart. I also ran into a small issue when using logrotate.d and the copytruncate directive. After a log rotation had been performed, and when rtorrent began writing to file again, it wouuld be filled with null 0's up to the last position it had been written to. This would potentially solve that.

Currently, ipv6 is only supported with HTTP trackers ("peers6" key).

https://www.bittorrent.org/beps/bep_0015.html IPv6 announces have the same structure as v4 ones, including the used action number except that the stride size of <IP address, TCP port> pairs in the response is 18 bytes instead of 6.

As far as I understand, the current code incorrectly interprets these 18 bytes as 3 pairs of <IPv4 address, TCP port>.

@rakshasa: Can you look to create a new release?

Please look all PRs, all forks, all issues:

• https://github.com/rakshasa/libtorrent/network
• https://github.com/rakshasa/libtorrent/pulls
• https://github.com/rakshasa/libtorrent/issues

Thanks in advance.

Hi,

I was trying to uprev libtorrent on Solaris and found that gmake check fails to build unit tests. sig_t is missing on Solaris.

Attached patch workarounds that. Consider adopting it, please.

Thank you

Martin check.txt