Posted
by
samzenpus
on Thursday May 29, 2014 @11:24PM
from the greased-lightning dept.

colinneagle (2544914) writes "What if you could transmit data without link layer flow control bogging down throughput with retransmission requests, and also optimize the size of the transmission for network efficiency and application latency constraints? In a Network World post, blogger Steve Patterson breaks down a recent breakthrough in stateless transmission using Random Linear Network Coding, or RLNC, which led to a joint venture between researchers at MIT, Caltech, and the University of Aalborg in Denmark called Code On Technologies.

The RLNC-encoded transmission improved video quality because packet loss in the RLNC case did not require the retransmission of lost packets. The RLNC-encoded video was downloaded five times faster than the native video stream time, and the RLNC-encoded video streamed fast enough to be rendered without interruption.

In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet."

Acknowledgment is conceptually a separate thing from retransmission requests. If the normal transmission stream is such that dropped packets can be detected, then the only return communication necessary is the occasional "please repeat packet X" Of course you probably still want *some* acknowledgment to avoid the scenario where someone watches only the first 5 seconds of video before shutting off their computer but the server keeps streaming video for the next hour to a now non-existent client, but it nee

How is R-S theoretically best, when it's been beaten by almost everything else in wireless FEC?

I saw this and was looking for the tech details. It looks like they are putting in FEC at the application layer. It's been done before. They are just hiding it better to not have to explain why 1/8th of the data is padding (or whatever ratio they are using).

And it may be different applications. R-S may be "perfect" for the type of coding it does, but it isn't purely FEC, so it's both perfect, and beaten by lots of codes for wireless FEC. Or it's perfect for "random" drops/noise, but noise/drops aren't random. So many others can beat it because "perfect" is a mathematical truth, but doesn't work for the real world.

>"A better coding for data error correction and redundancy than Reed-Solomon" - this is News for Nerds after all.

Well, yeah. There's lots of things better than Reed-Solomon for doing video encoding in a lossy environment. I worked on such a project at UCSD in 1998 or so with John Rogers and Paul Chu that could eat pretty impressive amounts of noise and still have a usable video signal without retransmissions. It's start getting pretty muddy looking when you really cranked up the losses, but unlike JPEG o

Yup, sounds a lot like par2, except this system works inline, in a streaming context to repair mangled blocks/packets, where par2 uses out-of-band data (the par2 files) to do the repairs after all the data is transmitted.

And like par2, it's going to require a healthy amount of processing from your CPU

The trends to higher-performance multicore processors and parallel operations everywhere in the network and on mobile devices lends itself to an encoding scheme utilizing linear algebra and matrix equations that might not have been possible in the past.

Notice they talk about multicore processors and not some hardware decoding embedded in the networking chip.From their published paper

Abstract-- Random Linear Network Coding (RLNC) providesa theoretically efficient method for coding. The drawbacks associ-ated with it are the complexity of the decoding and the overheadresulting from the encoding vector. Increasing the field size andgeneration size presents a fundamental trade-off between packet-based throughput and operational overhead. On the one hand,decreasing the probability of transmitting redundant packets isbeneficial for throughput and, consequently, reduces transmissionenergy. On the other hand, the decoding complexity and amountof header overhead increase with field size and generationlength, leading to higher energy consumption. Therefore, theoptimal trade-off is system and topology dependent, as it dependson the cost in energy of performing coding operations versustransmitting data. We show that moderate field sizes are thecorrect choice when trade-offs are considered. The results showthat sparse binary codes perform the best, unless the generationsize is very low.

Processing power is going to be an issue in mobile devices which have the most to gain from this innovation.

Processing power is going to be an issue in mobile devices which have the most to gain from this innovation.

I don't see this as a problem at all. This is the type of thing that screams for a dedicated processor, not a GP chip, and it's typical to see a 99% reduction in power consumption when you go this route.

It probably will be if it turns out to be useful, but I doubt processing power will be a significant factor in the idea's commercial success or failure. The desktop video card is the supercomputer of a decade ago, albeit without the memory, disk space and air-conditioning bill. My $150 card maxes out at a teraflop. There are demos on the net of 3D modelling (with sound) running on a cell phone using Nvidia's APX2500 chip, android + java. Developers can develop stuff on their desktop video card and drop it s

Indeed. Back in the day my company used custom hardware - 8x8 binary convolver - to do pattern recognition on images. It ran at the unbelievable rate of 12 billion pixel operations per second. The chips were originally from the cruise missile program, so we had to get a DoD license for every one we shipped. By the mid-1990s a reasonably fast Pentium could do the same work in software. And I'm pretty sure that some ray tracing code that ran on the Cray X-MP would now run faster on my phone.

If you named all the files the same, then the repairs are done in-band. All that's "new" here is calling all the packets *.PAR, rather than some RAR and some PAR. You need one extra block per block lost (though you can have multiple blocks per packet and other tricks to hide the actual redundancy).

There's no difference between the two (what's the difference between a 1 flipped to a 0, or a 1 that was lost and so interpreted as a 0?), and FEC is frequently (typically?) used to recover lost data. Interleaving is often used such that losing an entire transmitted packet results only in losing small parts of multiple actual packets, which FEC can then compensate for.

FEC is commonly used in streaming over IP applications to support lost packets, see for example the "above IP" FEC used in LTE multicast (eMBMS) or DVB-H, Raptor [wikipedia.org]. In those applications the L2 transport has its own error detection scheme, so IP packets will typically be either fully lost or ok (in other words, the probability of having a corrupt packet is very low).

Wow, were you aiming for humour? - Because it would be sad if you were serious

What people are saying is they have implemented FEC at the packet level. I don't know much about wireless but I do remember the maths lectures I attended 25yrs ago on error correction techniques. At the time I could perform the FEC algorithm by hand, I consider myself "mathematically inclined" and have been awarded bits of paper attesting to that inclination but to this day I still have just enough understanding of the mathemat

I would say FEC to is correct the current bit state, while this method RLNC is to correct a past or future packet.. I am totally suggesting is with fast enough processing power the protocol would assume every packet is missing and have recovered packets "ready to go" if one is missing...

Do all of these ultimately result in a corrected transmission? How about a transmission that degrades gracefully instead, so you don't have to re-transmit OR send redundant information that isn't even used unless packets are lost. Each packet would contain only a little low-frequency information , since most of the bits are high-frequency information that will not be missed as much. Maybe your smartphone could receive 4k video broadcasts and just discard 80% of the packets before even decoding them. I'm

This is yet another form of forward error correction. The claim is that it doesn't take any extra data to add forward error correction. That seems doubful, since it would violate Shannon's coding theorem.

This was tested against 3% random packet loss. If you have statistically well behaved packet loss due to noise, FEC works great. If you have bursty packet loss due to congestion, not so great.

The claim is that it doesn't take any extra data to add forward error correction. That seems doubful, since it would violate Shannon's coding theorem.

Yeah, that would be impossible. But I don't think they meant to claim that there is no extra data. When they say "The combined packet length is no longer than either of the two packets from which it is composed." I took that to mean that each packet is a fixed length. Not that they didn't add extra data to get to that packet length.

This was tested against 3% random packet loss. If you have statistically well behaved packet loss due to noise, FEC works great. If you have bursty packet loss due to congestion, not so great.

Yeah. It might be good for for wireless networks, where even a "5 bar" wireless network connection has a fairly consistent 1% packet loss. The QUIC [connectify.me] protocol is another atte

What if you could transmit data without link layer flow control bogging down throughput with retransmission requests

TFS makes it look like network coding can magically send data without any form of ACK & Retransmit. Network coding still requires feedback from a flow control protocol. You need to tweak the percentage of extra packets to send based on the number of useful / useless packets arriving, and you still need to control the overall rate of transmitted packets to avoid congestion. The goal is to make sure that every packet that arrives is useful for decoding the stream, regardless of which packets are lost. So

I'm thinking of Network Coding Meets TCP [google.com]. Though that doesn't give a great background. I've experimented with my own implementation, but had to shelve it due to lack of time. I'll try to quickly summarise the core idea;

You have packets [p1,.. pn] in your stream's transmission window.

Randomly pick coefficients [x1,..., xn] and calculate a new packet = x1*p1 + x2*p2 + ,.... (in a galois number field, so '+' is an XOR and '*' is a pre-computed lookup table). Sending the coefficients in the packet header.

TCP sends packet and then waits for ACK. However they eliminated the ACK part and just added extra parity in the packets. So, even when packet is lost, it can be reconstructed.

Of course there is a probability that multiple packets are lost so that it cannot be reconstructed. Of course, there is more redundant data in the stream instead of the small ACK packets. The packet loss characteristics of the channel probably makes this approach more efficient.

The ACK is crucial. Even if you pipeline your packets so you do not have to wait for each ACK to return, they are still vital to measuring network capacity. Too many people trying to send too much data will eventually overwhelm the transmit buffer at some congestion point, and loss will shoot past the ability of your error correction to compensate for.

And why do they use TCP if they are trying to avoid retransmissions due to lost/corrupt packets?

This seems to say that it's most trying to avoid link-layer retransmission, not transport-layer. So somehow I need to figure out all the links my transmission is traversing and disable link-layer retransmission on all of them?

What if you could transmit data without link layer flow control bogging down throughput with retransmission requests, and also optimize the size of the transmission for network efficiency and application latency constraints? Researchers from MIT, Caltech and the University of Aalborg claimed to have accomplished this...

it can "piggyback" on TC-IP but you have to use their "software" on either end

RLNC encoding can ride on top of the TCP-IP protocol, so implementatio

Many of the details fail under closer examination. It doesn't "use" TCP-IP. It could use a propritary IP stack that is TCP/IP compatible. So it'll use IP addresses and port numbers like a TCP packet would so that switches and routers in the middle wouldn't know or care what it's doing. If they put it on an IPX/SPX core it'd fail to route across the Internet. But it isn't TCP. It doesn't use a TCP compliant stack. It just looks like one to the outside world. It will not re-t

Like Anonymous Coward, yours were my thoughts exactly. Why would one use TCP to stream video? It's one of the tenants of networking that losing packets is preferable to increasing jitter in the video or audio feed. And off the top of my head, I'd say that there hasn't been a widely used connection-based layer 2 protocol since X.25. Hell, that's why Frame Relay and later ATM were invented - to let the transport layer handle error detection (and retransmission if required). Even Ethernet uses just a CRC for forward error correction - if the receiver can't fix errors, the frame is dropped. It's up to the upper layers do actually do anything about it. And let's not get started about a 3% random error distribution in a wireless link - everyone knows that fading causes a whole stream of consecutive packets to be lost, not just an even statistical distribution of them. Stephen Max Patterson at Network World just proved he isn't qualified to write for Network World... And just a nit for you, AK Marc - if someone says UDP is "running over TCP/IP", tell them to put down the router and step away from the rack. They just aren't qualified.

And just a nit for you, AK Marc - if someone says UDP is "running over TCP/IP", tell them to put down the router and step away from the rack. They just aren't qualified.

How about "UDP is a member of the TCP/IP suite of protocols"? Though most leave off the "suite of protocols" at the end, it's implied. I know lots of people smarter than you that would say "UDP is a subset of TCP/IP" or some other wording to that effect. Even Wikipedia [wikipedia.org] agrees with me. TCP/IP is shorthand for The Internet Protocol Suite, which includes UDP. So UDP "runs over" TCP/IP

I think the writer is confused. This sounds like a non-routed layer 2 error-correction protocol for error prone networks, like cell phone data and Wi-Fi. The only relation this has to the Internet is that it can carry TCP/IP traffic, just like Ethernet, Frame Relay, ATM, and any number of other layer 2 protocols can carry TCP/IP.

Yeah, they probably are using UDP in some video tests, but I think that is irrelevant. If this is a competing technology to Reed-Solomon encoding, then it is almost certainly a data-link layer protocol and is agnostic to network or session protocols.

From Wikipedia:

Reed–Solomon codes have since found important applications from deep-space communication to consumer electronics. They are prominently used in consumer electronics such as CDs, DVDs, Blu-ray Discs, in data transmission technologies such as DSL and WiMAX, in broadcast systems such as DVB and ATSC, and in computer applications such as RAID 6 systems.

I think the writer's mistake was in seeing everything through Internet-colored glasses. The article specifically says "link layer", not "session layer". TCP is a session layer technology. My best guess from the article is that this is not an e

But the "frame" doesn't exist end-to-end, but is re-created for each network segment. So their statements that it runs on the endpoints and doesn't require any changes in the middle indicates it is a higher layer protocol. Some of the layer 2 already include FEC. So why have double FEC? (it's bad). It's not double FEC at the same layer, but an application-layer FEC (based on their description). You can have them on different layers because they are correcting different things. L-2 FEC is correcting bit

And why do they use TCP if they are trying to avoid retransmissions due to lost/corrupt packets?

This seems to say that it's most trying to avoid link-layer retransmission, not transport-layer. So somehow I need to figure out all the links my transmission is traversing and disable link-layer retransmission on all of them?

I believe the issue is that you can't sell it to the cable companies and the DSL providers that implement PPPOE in order to track your surfing, to make sure you are buying your television programming from them, rather than file sharing, and they can intentionally make things like Tor not work. Not that PMTUD works on those things unless the modem proxies the ICMP messages, which are usually blocked by the cable companies, unless you explicitly ifconfig down to 1492 yourself, or enable active probing for bl

Engineers want to try it because it's a cool idea. Biz, focused on profit, doesn't. This is why AT&T rejected Kleinrock and others who came to them with ideas of the internet, The market delayed implementation out of short-sighted, blinkered concentration on profit as opposed to the General Welfare.

With net neutrality, the incentive for trying such ideas will be nonexistent.

As long as there is a significant bottleneck somewhere in the last mile or so (typically 4G/LTE or crowded WiFi) the incentive for efficient use of the available bandwidth will remain huge, regardless of politics.

The thing that might hamper development is if one company manages to get a monopoly on the creative content so that everyone have to use their service regardless of how shitty it performs.

In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet.

If that's "over-simplified" I am not sure I want to try to read the paper. That paragraph alone gave me a headache. Maybe it's the grapevine and the paper hurts less to read. Meh, tomorrow.

If that's "over-simplified" I am not sure I want to try to read the paper. That paragraph alone gave me a headache. Maybe it's the grapevine and the paper hurts less to read. Meh, tomorrow.

If it is like most such articles, they use 35 pages of polynomial math over a GF(2) field as a way to mathematically formalize what could be better explained with a one-page circuit diagram with a few XOR and shift registers and one paragraph of commentary, so yeah, wait for the boiled down version.

There's no single paper. There's a lot of papers that refine the technique and apply it to different problems. To even begin to understand it, you must have a good grasp of network coding as it's actually applied in real life. The butterfly network is but a starting point.

Nothing says the previous packet has the previous sequence number. Packets may arrive in any order, or not arrive at all (the point here). The stack makes it all work at the ends. This is all layman should need to understand, and as you have shown, all they can be expected to understand. It was so wored for a reason, and it it's lost on the layman, that's all right.

The word "previous" says the previous packet has the previous sequence number. I that reasonable people skilled in the art would generally assume previous meant sequentially previous, not time-of-receipt previous.

However I would say is that nothing says that the "immediately earlier" packet has the previous sequence number.

We could get pedantic and talk about sequentially previous and chronologically previous, further clarifying that chronologically previous is from the point of view of the receiver (since

Maybe they are trying to cover the case where they come out of order, then there's a drop?

But I agree in principle, the whole thing was poorly worded. Like it was written by a writer that doesn't understand it. That was dictated to by an engineer who was ordered to not be specific about it.

I remember working on a project where network coding was proposed for micro satellite cluster communications. If I remember correctly, network coding requires that all the nodes in the network have complete knowledge of the state of the network at any given transmission window. This requires transmission of the network state which used something like 7% overhead. The routing of a message from one end of the cluster to the other was difficult. I believe it might have been an np-complete problem. Have th

TFA doesn't seem to state what their assumptions were on how packets get lost and how many packet losses the algorithm can deal with, and what their distribution is. There are a lot of ways you can drop k packets out of n packets sent.

If you assume that every packet has a k/n chance of being lost, then being able to reconstruct a single missing packet could be incredibly useful. However cell phone packet losses tend to be incredibly bursty, i.e. they will have a very throughput for a while, but then all of a sudden(maybe you changed towers or went under a bridge etc) lose a whole ton of packets. Can this algorithm deal with bursty losses? I wish TFA was a bit more clear on that

(I haven't read this paper yet, but I've read other Network Coding data and experimented with the idea myself)

With TCP, when a packet is lost you have to retransmit it. You could simply duplicate all packets, like using RAID1 for HDD redundancy. But this obviously wastes bandwidth.

Network coding combines packets in a more intelligent way. More like RAID5 / RAID6 with continuous adaptation based on network conditions. Any packet that arrives may allow you to deduce information that you haven't seen before.

Reminds me a little of a random project I started back in college where I'd transmit a file in a bunch of packets where each contained the original file modulo a specified prime number. That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of th

That link shows their website, where they tell us that if the sender is about to send packet 83, and figures out it didn't get an acknowledgement for packet 22, it then has to resend packets 22 to 82. Which seems an entirely stupid thing to do, and an obvious improvement would be to resend only the packets that are actually lost.

Guess what: According to the Wikipedia article about TCP, that's what the "selective acknowledgment" (SACK) option defined in RFC 2018 does. So _poof_ goes the benefit of this sc

"That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of them they could use the extended euclidean algorithm to reconstruct the original file."

So if the receiver got 10K copies of the 1st packet and nothing else it could still reconstruct the file? Thats impressive. Which college was this, Hogworts?

"That way, if the file was split into 10,000 packets, then the transmitter could send out a constant stream of packets module different primes and as soon as the receiver got any 10,000 of them "

How can you split something into 10K packets then get *any* 10K of them? He's either saying he sends more than 10K or the receiver has to receive all 10K packets that were sent to reconstruct the file which I doubt is what he meant.

In over-simplified terms, each RLNC encoded packet sent is encoded using the immediately earlier sequenced packet and randomly generated coefficients, using a linear algebra function. The combined packet length is no longer than either of the two packets from which it is composed. When a packet is lost, the missing packet can be mathematically derived from a later-sequenced packet that includes earlier-sequenced packets and the coefficients used to encode the packet.

Uh... could you simplify it just a little more?

How does a "later-sequenced packet [...] include earlier-sequenced packets"?

They say "randomly" generated coefficients, but I'll bet you can use a psudo random number generator and pass in the same seed value to both the sender and reciever. Bam, now both sides have the same set of semi random coefficients to use when doing the fancy linear algebra.

I've been parsing this kind of press release for a long, long time now. I can pretty much tell what we're dealing with by how hard it is to state the advantage of a new approach in narrow and precise language.

That this blurb doesn't even disclose the error model class (error correction is undefined without doing so) suggests that the main advantage of this codec lies more in the targeting of a particular loss model than a general advance in mathematical power.

Any error correction method looks good when fed data that exactly corresponds to the loss model over which it directly optimises.

The innovators of this might have something valuable, but they are clearly trying to construe it as more than it is. This suggests that there are other, equally viable ways to skin this particular cat.

Why would you put that on top of a streaming protocol like TCP/IP that's got all the flow control and retry logic happening under the covers? Surely you'd run RLNC over UDP!

UDP is a subset of TCP/IP. They worded it horribly, but it's likely not using TCP for flow control. Note the large number of odd statements, "TCP-IP" rather than TCP/IP. And "the immediately earlier sequenced packet" rather than "the previously transmitted packet" or "the packet with the previous sequence number" or something else that isn't unclear. I doubt they had an engineer write it, and the tech writer didn't understand what they were writing about.

UDP is a subset of TCP/IP. They worded it horribly, but it's likely not using TCP for flow control. Note the large number of odd statements, "TCP-IP" rather than TCP/IP. And "the immediately earlier sequenced packet" rather than "the previously transmitted packet" or "the packet with the previous sequence number" or something else that isn't unclear. I doubt they had an engineer write it, and the tech writer didn't understand what they were writing about.

UDP is not a subset of TCP. It's just that no one says "UDP/IP". It's a socketless datagram and has its own protocol ID.

UDP is not a subset of TCP. It's just that no one says "UDP/IP". It's a socketless datagram and has its own protocol ID.

http://en.wikipedia.org/wiki/I... [wikipedia.org] TCP/IP is shorthand for The Internet Protocol Suite. UDP is a protocol in The Internet Protocol Suite. Therefore, you are 100% wrong, in your wording, UDP is a subset of TCP. You just chose to leave off the "/IP" because you know you are wrong.

I have the same question. From the article, "This is because RLNC can recreate any packet lost on the receiving side from a later sequenced packet." If this is truly the case, just send me the very last packet and I can recreate all the packets that were supposed to come before by working backwards.