uTP is designed to be a background network protocol that will yield to TCP traffic, making it much more network friendly. It also carries the side-benefit of being immune to TCP resets, which is often used by many ISPs for throttling.

After reading BEP 29, I'd like to say that the specification is incomplete. In particular, it doesn't say how we find out if a given peer supports µTP.

I'm also fairly fairly suspicious of the congestion control as described in the document; I suspect that the document is incomplete, and that the µTorrent people are doing more careful stuff. I strongly recommend that this specification should not be implemented unless and until the IETF declares it to be safe for use on the Internet.

Having looked over the spec and what uTorrent does I actually think it is complete with respect to negotiation, just unclear. BEP 29 defines the uTP protocol. uTorrent just tries use it by connecting to a UDP port on the client, and if the handshake fails it falls back to TCP. I suppose technically there might be situations where clients could have distinct port ranges for UDP and TCP, but from all appearances they don't worry about that.

In other words, there is no explicit communication that a peer supports uTP, you find out by trying to use uTP against it.

Since they're definitely not stupid people, they're probably doing something smarter than that, but I'm not in touch with any of the µTorrent developers. Charles, perhaps you could drop your pals a note?

The following patch (Linux-only) allows using TCP-LP, a less than best-effort sender-only modification of TCP, together with Transmission.

Just tried this - almost i'd say nothing at all in difference compared to deluge doing the same downloads with the same connections amount.. Hydri says that tcp-lp requires support on the both ends, maybe because of this its this useless..

...
Set congestion control algorithm: Operation now in progress
Set congestion control algorithm: Operation now in progress
Set congestion control algorithm: Operation now in progress
Set congestion control algorithm: Success
Set congestion control algorithm: Success
Set congestion control algorithm: Success
...
Set congestion control algorithm: Success
Set congestion control algorithm: Success
Set congestion control algorithm: Success
Set congestion control algorithm: Resource temporarily unavailable
Set congestion control algorithm: Resource temporarily unavailable
Set congestion control algorithm: Success
Set congestion control algorithm: Success
Set congestion control algorithm: Success
...

Do you mean µTP, or the patch to tweak the congestion control algorithm?

If the latter, then It Works For Me (TM). The failure report above was on a hacked non-standard unreleased kernel, and doesn't make sense to me -- it indicates that setsockopt returns -1 but doesn't set errno.

Furthermore, my patch doesn't do anything unless the user manually edits settings.json. Hence, it should be safe to deploy.

Unless you can see a reason why it could break anything, I suggest committing.

As for me, any attempts to do something with TCP flow control looks like a fruitless effort. Look, you can do whatever you want on local machine with it's scheduling, BUT that's where one problem occurs: what if there is router on the way? You can schedule packets on fast local 100MBps link to router as you want them to. But then packets hit router(s) which have to forward them to ISP. Router usually lacks 100MBps link to Internet and hence have to buffer all your data if they arrive fast and send them as actual connection rate permits (and it's much slower than 100MBps).

So:
1) Your scheduling efforts for TCP are useless in many configurations. Routers would use their own algos to (re)schedule your (buffered) data packets queued in their queues.
2) As the result, usually you have too much stuff in output buffers of router and awfully bad ping times since there is ton of TCP connections who can't accurately measure state of link and transmit direction getting jammed on DSL and similar slow-send links.

There is not much you can do with TCP about this issue. On other hand UDP-based protocol can provide low-level link status data (like round-trip times and packets loss rate) quite easily so it's possible to dynamically readjust speeds according to actual state of things (i.e. reduce speed when link becomes busy and round-trip times are getting high, more packets getting lost, etc). Also in theory it's possible to do STUN-like tricks to connect two peers behind NATs via UDP by playing certain small tricks (like STUN does).

The status is that nothing's been done on it. This is going to be a major undertaking. Unless my timeline is wrong, uTorrent's had paid programmers working on this feature for over a year and they still don't have all the details hammered out, and it's still not fully documented.

Transmission is a much smaller and unfunded project. In order for us to adopt uTP we will need at least two things:

A fully written spec. According to Switeck, the uTorrent developers have made some major changes since the BEP was written.

Another programmer who's well-versed with the ins and outs of UDP programming

One alternative would be to wait for libtorrent-rasterbar to implement uTP and use that as our reference implementation. Even if uTP has undocumented changes, libtorrent-rasterbar has a leg up on the competition because its author works for uTorrent and has either written parts of uTP or can at least read its source code, which is something that the authors of other engines like rTorrent, KTorrent, Transmission, and Vuze can't do.

I have trouble seeing how any of these are objections serious enough to prevent use. Also I believe each has been addressed before, but I'll reply to them anyway.

µTP (the framing mechanism) is broken beyond repair

I'm not sure what you mean by "broken beyond repair" since it actually works and can be improved upon. I'm going to guess, though, that you mean there is no MTU discovery. Our solution is a to guess at MTU based on address type, and use the most conservative packet sizes for each scenario. Since we just released the source to uTP, you can see the MTU calculation we do in utp_utils.cpp
Nothing, however, is stopping more explicit MTU discovery from being added to the implementations should there be a case this does not handle well.

2a. LEDBAT (the delay-based mechanism) might or might not be a good idea;

In our measurements at massive scale and the extensive study of the congestion controller at Internet2, the LEDBAT delay-based congestion controller does perform well. A key property is that in the worst case it reacts with the same behaviors as TCP on loss and timeout, so it will not "break the internet" should the delay be unrelated to congestion.

2b. at any rate, it belongs in TCP, not in a broken, proprietary framing mechanism.

It would be great if this were implemented as an option in TCP. LEDBAT is the first step towards this goal, not to mention each OS which has to implement it, and get it right. TCP-based BitTorrent? has a problem today, and a UDP-based transport solves it. I believe it is even stronger proof that the OSes should support it if a wide-scale application like BitTorrent? has adopted it.

3?. uTP uses small packets (sometimes, if misconfigured)

This complaint by basically one forum member is a non-issue for adopting uTP elsewhere. The dynamic packet size code is uTorrent specific feature, and is not required by or related to implementing uTP. Packets of the full size (up to the MTU) may be used at all times.

In any case, uTP is open source now. You may test it yourself for issues, and we can all contribute to making BitTorrent? clients interoperate without smashing the internet with congestion.

As I have extensively explained on the BitTorrent?.org forums, using an idiosyncratic, ad-hoc, mis-designed framing scheme causes a number of serious issues which can be avoided simply by layering LEDBAT over TCP. (Note that I am speaking about µTP here, and deliberately avoiding discussing LEDBAT, which I am not competent to judge.)

Since I have already spent a non-negligible amount of my copious free time writing up my criticism, I refuse to repeat it here. I suggest we move this discusion back where it belongs, namely to

There is at least one major issue with uTP exists at the moment: when network is congested, uTP resorts to very small packets. The problem is: what if congestion occurs not in user's router buffer but anywhere else, i.e. say, ISP network runs on full capacity, some router reached it's limits, or some remote channels were saturated? Then, uTP dumbly reduces packet size. The "only" problem is that this behavior makes things even worse and leads to some kind of collapse. This increases total PPS (packets per seconds) rate in channels, making routers jobs even harder. Even some decent ISP routers may fail to cope with such "gift" and user's modems and other routers are often just running out of CPU power when trying to process such PPS. Also there is more headers to send and still same amount of data to transfer. Hence more stuff to transfer in total. More headers, less useful data. Surely, overloaded channels really need to be overloaded even better. So, while uTP solves particular problem and nice idea, actual implementation creates new troubles: it actually helps networks to collapse under heavy loads when there is any "global" congestion occurs. So, some ISPs even have started filtering out uTP packets to reduce network loads. As for me, it have to be reimplemented without resorting to small packets.

ghazel wrote:
uTP uses small packets ...
This complaint by basically one forum member ...
...and it has been fixed for 2.0.3

I also believe that implementation issues should be set apart from the uTP core protocol/lib, and not hold up other clients from implementing it. Still, uTorrent is kind of feasibility demo for it, and I'm still eager to see a test with 2.0x demonstrating how effective it is in congestion control.

I'm glad to see positive corrective action on behalf of uTorrent devs (though delayed by ~6 months since being reported ... ). It surely looks much improved in 2.03 beta, and reduce both payload's overhead and it's related PPS. Though, connections related PPS can still ​be much improved.

Pulled, although I don't necessarily agree with "make a note" -- I'm not planning to do any packet size variation in Transmission (we've discussed this in detail with Arvid, and he's not really sure it's a good thing either).

I'm seeing wide instabilities in read bandwidth, which indicates that the bandwidth management code might need to be tweaked for µTP; it's more likely that since the bandwidth measurement performed with µTP is lower down the stack (there are no kernel buffers for µTP), it simply needs to be smoothed before being displayed to the user.

jch, what is the use case for incoming only? and, is it a use case that many users will encounter?

Outgoing entails making lots of probably-going-to-fail uTP attempts. Accepting only incoming uTP, not so much. Weak networking will choke-and-die from heavy uTP packet rates, more from number of connections than speed. ...though some to treat each UDP packet as though it were a separate session/connection!
uTorrent v1.8.x did incoming only by default to ease the appearance of uTP ...devs could test "against" uTorrent v1.8.x clients in the wild without almost the whole customer base running uTP. Unfortunately, that's probably not so much an option now...since uTorrent itself is "firing blind" with both TCP and uTP at ips gathered from the tracker and probably DHT.

jch, what is the use case for incoming only? and, is it a use case that many users will encounter?

Outgoing entails making lots of probably-going-to-fail uTP attempts. Accepting only incoming uTP, not so much. Weak networking will choke-and-die from heavy uTP packet rates, more from number of connections than speed. ...though some to treat each UDP packet as though it were a separate session/connection!
uTorrent v1.8.x did incoming only by default to ease the appearance of uTP ...devs could test "against" uTorrent v1.8.x clients in the wild without almost the whole customer base running uTP. Unfortunately, that's probably not so much an option now...since uTorrent itself is "firing blind" with both TCP and uTP at ips gathered from the tracker and probably DHT.

uTorrent (2.2x) can still control in/out direction for both TCP and uTP in bt.transp_disposition advanced settings. Practically it has not use now. Tho, in the past, it was useful for testing. Plus it makes it possible to be able to 'cooperate' with other uTP clients, while minimizing outgoing traffic, that can heart crappy routers...

Alternatively, is there a known downside to building libutp unconditionally? I doubt there are many environments building Transmission that don't have a C++ compiler built in. Making that a precondition would allow us to get rid of some ugly #ifdefs in libtransmission.

On the user-configuration level, utp-enabled should be default to 'false', at least for now.

Alternatively, is there a known downside to building libutp unconditionally? I doubt there are many environments building Transmission that don't have a C++ compiler built in. Making that a precondition would allow us to get rid of some ugly #ifdefs in libtransmission.

I'm a little worried about embedded builds. I know that OpenWRT includes a C++ cross-compiler, but how many other embedded systems are either including Transmission or are considering doing so?

The problem starts when I re-establish use of uTP, not on disabling it. I suppose other clients get confused about my type of connection? Not sure if it would recover with time but a pause all/resume all fixes the problem.

How are you measuring that? We shouldn't be connecting over µTP any faster than we do over TCP.

--jch

Please see attachment which shows periodic spikes to 6 every 10 seconds using r11889 and 80 connected peers, no DHT or LPD, and browser to post. Same conditions with r11970 produced a steady average of ~80 spiking to 200-300. Max of 500 was seen for several seconds at application start-up.

We're currently not handling the case of an outgoing connexion to a µTP peer that doesn't do encryption. I.e. if we're configured to attempt encrypted connexions, we do

(1) encrypted µTP; on failure

(2) encrypted TCP; on failure

(3) plaintext TCP.

(We're slightly smarter than that, actually, if (1) fails during the handshake, we skip to (3) directly.)

Now what that means is that we never attempt a plaintext µTP connexion -- if the peer fails encrypted µTP, we move to plaintext TCP straight away. That's good enough for me, but if anyone wants to try his hand at fixing that, the fun happens in handshake.c:gotError which interacts in marvelous ways with peer-io.c:tr_peerIoReconnect.

If I add a global UTPSocket counter to libtransmission, it keeps balancing out to 0 at the end.

Ah, sorry -- I didn't realise we're properly destroying all the peer-ios. I was under the impression we were just trusting the OS to shut down the sockets (which obviously doesn't happen in the case of µTP sockets).

So all's right -- it looks like the only remaining issue is the throughput controller.

I checked in some simple code for throughput control last night in r12063. It's still preliminary and will benefit from testing/tuning, but it's much closer to the desired speed levels than earlier builds.

just a couple of observations comparing the above two builds (i have both installed in Linux Debian):

1) with utp running the CPU is about twice that for the same torrent using the 2.22 release
2) also, the upload rate with utp is not as steady, i.e., has a tendency to swing up and down more when observing in real time.

overall utp-enabled torrents are working fine, but with the above 2 caveats

Oh, and one important thing that I'd like to be sure of: the CPU load goes back down to what it was in 2.20 when you disable µTP, right? That's really important if we want Transmission to remain useful on embedded hardware.

gunzip: there have been a lot of changes in trunk since the 2.2x branch was forked, so without more information it's hard to know whether the increased CPU load is a uTP issue or not. If you disable uTP and restart, does that have any effect on CPU load?

Also, would it be possible to pastebin a perf profile of where trunk is using your cycles?

Well that's a good improvement. The most expensive function has dropped down to #12 in the list.

The #1 and #2 functions are tr_isPeerIo() and tr_bandwidthClamp() now. Let's try inlining tr_isPeerIo() to see if that makes any difference. Also I see a way to avoid some number crunching in tr_bandwidthClamp().

gunzip, could you apply cpu-02.diff to r12168 or higher and report back? :)

I'm wondering what the load / profiling looks like with r12170 with uTP enabled and disabled now. Are enabled and disabled closer together in wrt CPU load? What is the load like compared to a few days ago before we started testing CPU cycles?

Hmm, not a big fan of inlining isPeerIO -- it bloats the code uselessly.

In the abstract I agree about inlining bloat.

The reason I let this one through is that r12170 didn't trigger gcc's warning that inlining function X will grow the code.

If there's a better test that should use instead of gcc's warning system, I would love to hear about it, because that's something I've wondered about too.

Wouldn't it be better to simply remove some of the asserts?

Yes, and I did remove some redundant ones in r12170. And, of course, assertions are compiled out of release builds anyway. IMO tr_isPeerIo() is not the most interesting line in that profile list.

I'm more surprised that bandwidthClamp() didn't drop more in cpu-04.diff. Of the libtransmission functions, bytesUsed() is the next highest on the list, and I don't see any way at all to tweak that.

Of the libutp functions, the most expensive function is UTPSocket::check_timeouts()... we could try increasing our periodic call to UTP_CheckTimeouts() from 50ms to some higher value. Is 50ms that much better than, say, 100ms? The comment in the code says alus "says 50ms works for them".

The behavior of running check_timeouts less often than every 50ms is Untested. I know at intervals of seconds it has Bugs. Could you provider more insight into which part of check_timeouts is taking all the time? It's a fairly simple function.

I'm wondering what the load / profiling looks like with r12170 with uTP enabled and disabled now. Are enabled and disabled closer together in wrt CPU load? What is the load like compared to a few days ago before we started testing CPU cycles?

since the total percent always adds to 100% in those profile reports, if you push something down something else has to go up?

in terms of the CPU load on my system there is no noticable difference now between utp enabled/disabled:

thats a 5-minute time period sampled every second using top command in batch mode, and it's the CPU of the daemon only. also, the tests were started shortly after the torrent was added and when I always see the highest CPU rates. when the torrent reaches seeding, the CPU drops considerably in both cases.

UTPSocket::check_timeouts() is much lower in these new profiles, for whatever reason, even though we're calling it just as frequently as before. Maybe these profiles are "noisy" and we should look at trends rather than individual data points? It's possible that check_timeouts() isn't as bad off as it looked earlier.

In both new reports, bandwidthClamp() is still the most expensive function, and probably could be banged on some more.

tr_isPeerIo() is in second place in both reports, but since it'll be compiled out of the official builds, I'm not worried about that.

Anyway, since we've worked things down to the point where using uTP doesn't cost any more than not using uTP, that solves the issue for this uTP ticket :)

Recall that µTP computes all timeouts in user-space; that's fine on systems that implement clock_gettime as a vsyscall, but I'd like to see a profile on a system that doesn't (e.g. Linux/MIPS, as in most cheap NAS systems).

I am working at the European Project P2P-Next [www.p2p-next.org]. Within this project a new protocol similar with uTP is designed and developed, named swift. We want to make a comparison test between classical BitTorrent? transfer over TCP, uTP transfer and swift transfer. We are planning to test TCP and uTP transfer with a Transmission client.

Our experiments could also be useful to you by testing transmission uTP facility and discovering bugs. We are going to use a cluster from University Politehnica of Bucharest along with a virtualization solution that will provide the possibility of running a big swarm of transmission clients simultaneously.

I installed two transmission 2.30b2 clients on two machines that are part of University Politehnica of Bucharest. Each one has an external IP address and are part of the same network. One of them is seeding a 64MiB file and the other is trying to download it using its torrent file. Both transmission clients are configured to use uTP (utp-enabled property from settings.json is set to true).

The problem is that the leecher cannot download from the seeder. I also tried to add into the swarm a KTorrent Linux client, configured to use only uTP. It manages to download a small amount of the file, but after a while it stops and the peers from its peer list disappear. Transmission clients also manage to transfer some percents of the file while KTorrent runs, but they also stop after a while. I also tried to use uTorrent from Windows, but the seeder is soon marked as snobbed because it is not able to deliver more than a few bytes from the file. Most of the time the download speed is 0, but from time to time it increses to about 1KiB/s.

If I deactivate uTP from the seeder by setting utp-enabled to false in settings.json, the transfer works perfectly at maximum speed.

What is the problem?

I would also like to know if utp-enabled property sets transmission to exclusively use uTP or it also can use TCP if it figures out that the client does not support uTP?

I installed two transmission 2.30b2 clients on two machines that are part of University Politehnica of Bucharest. Each one has an external IP address and are part of the same network. One of them is seeding a 64MiB file and the other is trying to download it using its torrent file. Both transmission clients are configured to use uTP (utp-enabled property from settings.json is set to true).

The problem is that the leecher cannot download from the seeder. I also tried to add into the swarm a KTorrent Linux client, configured to use only uTP. It manages to download a small amount of the file, but after a while it stops and the peers from its peer list disappear. Transmission clients also manage to transfer some percents of the file while KTorrent runs, but they also stop after a while. I also tried to use uTorrent from Windows, but the seeder is soon marked as snobbed because it is not able to deliver more than a few bytes from the file. Most of the time the download speed is 0, but from time to time it increses to about 1KiB/s.

If I deactivate uTP from the seeder by setting utp-enabled to false in settings.json, the transfer works perfectly at maximum speed.

jordan, I think 1024 bytes is to small value. Cause these 1024 bytes are sent every BANDWIDTH_PERIOD_MSEC, which is 500 ms by default. As a result we get too low upload speed. In practice I get about 20 mbit/s UTP upload speed with value of 1024000 bytes. But I guess it's not best solution. I suppose you need optimize this chunk of code regarding UTP behaviour.

Well, one, we're in freeze for 2.30 right now so unless the current code is broken for the average user, I'm hesitant to refactor a critical piece of code like this right now. 20 Mbit/s is not something that the average user has to worry about :)

Two, that 1024 value is used inside a loop, so right now we call UTP_Write() multiple times with small buffer sizes. After looking over UTP_Write(), it's not clear to me how that approach is much worse than what your proposal of using a larger buffer size in fewer calls.

The downside to the "big buffer" approach is that a single peer would get the lion's share of the client's upload bandwidth.

I corrected the bug marked in r12361 and then I tested again. Now the transfer from one seeder to one leecher works and also with two seeders.

When using transmission with uTP disabled in a swarm with two peers the TCP transfer occurs at an average of 4MiB/s between seeder and leecher. But when I restart the test with uTP enabled the average speed is about 500KiB/s which is very small. The two machines that host each peer are in the same network and do not have bandwidth limitation.

I also made a test with two transmission seeders using uTP placed on the same network and two other clients from a different network. I first tested with KTorrent in Ubuntu and then uTorrent in Windows. KTorrent could not download the whole file from the seeders. After a while it stops and all the peers disappear from its list passing into state "stalled". It may be an issue of KTorrent, which is the first Linux client to implement uTP as I know. On the other hand uTorrent has the best transfer. It reaches a download speed of 1.3MiB/s from each seeder. However, this speed is still smaller than the TCP one witch was about 4MiB/s with just one seeder.

Could I do something to optimize transfer speed? Or transmission with uTP is not yet optimized on this part?

calin.burloiu: Transmission's use of uTP is of course pretty new, so yes it's possible that there are ways to improve the way we use it. Have you tried the "big buffer" suggestion Vincent made in comment:111? I'd be happy to be proven wrong in exchange for faster peers... :)

Please note that we use a single UDP socket for all our µTP traffic -- so you only pay the send and receive buffers once.

No major objection to reducing the send buffer -- it's probably not used much. The receive buffer, on the other hand, is all we have in order to buffer µTP packets that arrive while we're context-switched, and on slow machines it's essential.

in past, especially in seeding state, transmission seemed to discriminate against those peers with utp (T flag) and i noticed a strong tendency to upload to non-utp peers in the swarm. this behavior was repeatable and persistent over many torrents so don't think it was by chance. i often disabled utp because it seemed to inhibit seeding performance.

now, after testing with 4 torrents, uploading appears much more "balanced", i.e., seeding is occurring with both utp and non-utp peers.