Now that we have come so far already and developed new techniques I wanted to express my thoughts about why I think we are succeeding and how we can best proceed.

Why are we succeeding?

I think there are two main reasons why we are succeeding in what was deemed to be impossible. The first is that we really went outside-the-box with our approach. The second is that our techniques are such that we can collaborate in our efforts. Let me elaborate on both.

Outside-the-box thinking:

The "conventional" way of thinking when repairing a video is (1) to try to repair all predictable header information and (2) try to repair each decoding error one at the time. The thinking is: since you cannot decode something if you haven't decoded everything before it (from the point of a key frame) you can only fix the error where the decoder halts.

Keep in mind that everything in mpeg-4 compressed video is highly dependent on each other: if decoding of a block fails you lose all data in three dimensions! You lose every block to the right, every row of blocks to the bottom and every block of every frame after this frame in this right-lower region. This is horrible. Especially when the error occurs near the top of the key frame. And when you can't repair an error you're stuck. This is the situation we started in. On the blog of Benoît Joossen he describes what he did to extract parts of the two key frames and could not do much more than that, not without enormous effort.

This situation looks very similar to having an error in encrypted data where the encryption algorithm used the "Propagating cipher-block chaining"-mode. Meaning that if you have a single error in the data everything after that consists of seemingly random bits when you decrypt. However compression is very different from encryption: while the end-result of both encrypted and compressed data may look similar - they both look like random bits - the goal of encryption is to prevent decoding without a key while the goal of compression is to lower the amount of bits needed.

We can learn something very useful though from the area of encryption: if you want to break a cipher you ultimately want to recover it's "internal state". And if a cipher is weak this is actually possible. Since a compression algorithm can always be considered a weak cipher algorithm (it's not designed to be protecting it's "internal state") there might be a way to recover it's "internal state" after an error has occurred.

The mpeg-4 decoder algorithm has something which you could call an "internal state". It contains the current x and y of the macroblock, the bitposition of that block, the DC values from either the top or left block and some other values (like MV values). If you have these values at a certain x,y then you can decode a video from there.

What has happened is that after some time struggling with the problem and realizing the bad situation we were in, we had two major breakthroughs in this regard. One was that we told the decoder to ignore errors. This allowed us to see what the decoder would do after the error, if it could somehow self-recover somewhat. And this appeared to be the case: parts of it's "internal state" were apparently restored because it was clearly producing recognizable parts in the picture (albeit in the wrong color, brightness and x,y-position). The clock was first found this way.

The other breakthrough was that we managed to change parts of the "internal state": by setting the bitposition of a macroblock the best effort by Benoît Joossen was essentially recreated. But in our case there was no brute forcing needed (which is what Benoît Joossen used). We could simply set the bitposition in the "internal state" at the moment a certain macroblock was supposed to be decoded. Later the tool was released so that everybody could start trying mmb-commands.

Which brings us to the second reason why we are succeeding.

Collaboration becomes possible:

There is no substitute for the human eye! And brain!

It is simply incredible how much time and effort people are putting into this major task! After the mmb-command became usable people started to play around with it and were quickly producing nice I-frame results. This lead to more added features and more use of those features, leading to more progress. Within a few days we had a wiki, a repository, command line repair tools and a really cool online repair tool! The amount of constant updates on the wiki is also impressive. Some people seem absolutely dedicated making this video work. Almost every day I start reading this thread and the forum simply won't let me "like" fast enough (120 seconds delay). Thats how good it is.

All this has personally inspired me to do even more. It was really nice to see the first P-frames coming out. I'm so proud of everyone here . The latest video was also awesome. And the tweet from Elon made it all worth it for me.

Anyway. It turns out this problem was very fit to be crowd-sourced. NSF rocks!

How should we proceed?

Here are the three phases I mentioned earlier for reference:

Quote

What this whole problem boils down to is three phases (per frame):

1) Extracting: Correcting the predictable streamheaders and I/P-frame headers. This way we (or actually the decoder) can extract all the video data without missing packets.

2) Positioning: Finding all the macroblock-data (that is: finding all bitpositions where each mb-data starts) and then assigning the right macroblocks to the these starting bitpositions.

3) Repairing: Correcting left-over DC-values errors either by bit flipping or by changing the DC-values after decoding. This should implcitly also fix "bad inheritance": where a macro inherents its DC base value from the wrong macroblock (that is: left or top).

I think we are doing a very good job at 1 and 2 and we are also doing a bit of bitflipping (3). As I mentioned last time I've got some ideas how to make things even better. Here is a description of the tools I am thinking about:

Macroblock data finder:

In a lot of cases I see that there are still quite large swaths of blocks that are "turned off" (as in a -1 somewhere before it) because of some error that occurred earlier. It's not always easy to know where actual non-error data begins again. Especially the last row of the I-frames. And I think I have found out a way to find most missing data for macroblocks.

The algorithm goes like this. You iterate through every possible starting position (begin-bit to end-bit of the frame data) and record at which bitposition it begins and at which bitposition it ends (which is either an error or the end of the frame data) of each of these starting positions. You count the amount of bits and macroblocks are decoded for each of these starting positions until it hits an error (or the end of the frame data). So a really good part for I-frame 169 would start at bitposition 550: it decodes many (thousands!) of bits and many blocks from bit position 550 before it hits an error.

Now you sort all these bitsequences by the length they have (the total number of bits they decode from that bitposition). You take the first of that list because that is obviously the best sequence. Then you remove it from the list and also remove all sequences that either end or start within the bitrange of the "winning" sequence. They are all considered to be of less value. And you keep doing this: taking the first sequence, removing all sequences that lie in its coverage area. Until you have no sequences left.

I believe at the end you will have all the longest bitsequences that are available in the frame data. You "only" have to assign the right macroblocks to the starting bitpositions of these sequences in this set. And as said before I believe it would be helpful if we auto-generate all the likely starting positions for each frame and make these available as a static file to the online editor. This would allow the user to choose from good starting positions.

Brute force bit flipper inside the decoder:

Another idea is to do the bruteforce bitflipping inside the decoder. The main reason to do the brute forcing inside the code of the decoder is that it is likely to be much faster. Lets say you have four blocks of -1 (for example 15:12, 16:12, 17:12 and 18:12 in I-frame 150). To the left and right of it is are good macroblocks (14:12 at 45301 and 19:12 at 46628). You remove the four -1s so it will give an error again. Now you say to the decoder (in broteforce mode) try bitflipping 1-3 bits between the starting position of 15:12 (45472) and the starting position of 19:12 (46628) and each time you do this try to decode from 15:12 and see if you find 4 macroblocks and end at exactly at 19:12 (46628).

The advantage of this would be that there is a very clear end goal which is not likely to happen by chance. The decoder doesn't have to decode the entire frame over and over again (only 4 macroblocks decoding each time). And you could even reduce that to close to just one macroblock decoding each try. And if it found a candidate it would simply print it out.

I think that would be the fastest way of bitflipping. There may be smarter variations of it. It is however not trivial (at least not for me ) to "reset" an already decoded macroblock. @michaelni: do you know how to do this? Because that would be required.

DC value fixer:

Changing the DC values turns out to be very tricky. Not only can you get some unpredictable behaviour to the right and bottom of a block, it is also possible that everything gets messed up if a block-position is changed above and to the left. And it's also very tedious I think. Hard to get it perfect.

There may be a way to (semi-)automatically fix the DC values (here I am less certain though). We are assuming all marcoblock have been given a position or are marked as -1. So no error blocks. And we have done all the bitflipping we could imagine. So this is basically something you do at the very end: "post-processing" if you will.

This is roughly how it would go. You disable color output. So only luminance comes out. You set 0,0 to -1 and you set 42:0 it designated bitposition. To the right of 42:0 is the last block of the row. It is affected by DC values of 42:0. If you see a horizontal "line" on the 8th pixel from above (as in: consistently lighter or darker pixels between 8 pixels and 9 pixels from above) then you know that the DC values of your block is off and change it. Then you go do the same with 41:0. But now you can check a longer (more reliable) horizontal line.

There are still some problems with that idea though. I will have to experiment and think about it longer.

You did really great! Those headers were totally mangled, but after your fixes your result file is pretty clean from a TS point of view.

Just that one duplicate continuity counter - it was odd, the lead-in and tail-out of that section were quite clean but the numbers just didn't line up. Maybe that's a side effect of earlier alignment efforts.

Quote

I hope you don't mind but I've processed it a little more and fixed a couple of PCC discontinuities that were remaining. I've also removed the AF from any packets where either the AF is longer than the packet, or when the AF shows a wildly invalid PTS time. This is hopefully the correct thing to do, but please take a look and let me know what you think.

Why would I mind?

I've been putting some thought into parsing code to deal with the PES headers and the frame headers, but haven't had time to deal with it. Had a big deadline yesterday for a symposium presentation I'll be doing the second week of June and have a pile of essays to write for a course I'm taking - the work I did yesterday was to wind down after a 60-hour workweek.

I agree clearing the AF bit in packets where it's clear that its definitely not an adaptation field that follows is the right thing to do. As Shanuson points out there's a clear pattern to AFs which we can use to figure them out. There were sections where there were a really wild number of flipped bits, and figuring out whether or not an AF is valid wasn't going to be a "wind down" sort of thing for me to think about last night. There were one or two that showed up clearly in Wireshark that I fixed, but I'm glad you were able to take care of the rest of it.

I got the info I needed to extract the frames with ffmpeg, but I wound up with a completely blank iframe for some reason that I didn't have time to investigate, and I haven't downloaded MichaelNi's patched ffmpeg yet. It did deliver a full 20 frames, though.

"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Question: when we use -mmb X:offset:pattern, are we flipping bits before or after the entropy decoding stage? Is this simply equivalent to flipping bits in the input file? Or is it essentially just a low-level way of editing the MB directly?

From what I understand it is equivalent to flipping the bits in the input file. It happens well before the decoding of any blocks start.

Can you animate these frames? Legs are starting to open frames 85-90...

If you paste the attached MMB into the http://spacex2.slapbet.com/ frame 72 entire-segment box, and then hit the play button at the right end of the slider bar below the image, you'll see the beginning of the legs opening - this should not be used for ongoing work, as I simply nulled out some of the intervening pframes which were making it difficult to see the leg movement and combined all the work up to this point.

"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Another idea is to do the bruteforce bitflipping inside the decoder. The main reason to do the brute forcing inside the code of the decoder is that it is likely to be much faster. Lets say you have four blocks of -1 (for example 15:12, 16:12, 17:12 and 18:12 in I-frame 150). To the left and right of it is are good macroblocks (14:12 at 45301 and 19:12 at 46628). You remove the four -1s so it will give an error again. Now you say to the decoder (in broteforce mode) try bitflipping 1-3 bits between the starting position of 15:12 (45472) and the starting position of 19:12 (46628) and each time you do this try to decode from 15:12 and see if you find 4 macroblocks and end at exactly at 19:12 (46628).

The bash script that I have been using now spits out roughly 900 combinations per minute in, using one thread. This produces a rate sufficient for a 4-bit search over a 50-bit area in a matter four hours, and works pretty much exactly how you described.. The difference being that I am grepping the output of FFmpeg. A few hundred candidates were produced, of which 40 or so look good. My next step is to add generation of the following P-frame to cross-reference these 40 candidates.

The size of the search space follows "bit-space choose bit-size", which worked out to "50 choose 4" for the 4-bit run.

Moving this searching into FFmpeg is an obvious next step, I sincerely hope it's possible!

Actually, it occurs to me that the straight up "ps" approach to tracking workers may have problems with the above script, because the workers are constantly starting and stopping ffmpeg, which ps is looking for.

I am interested to see the results. I'm just so busy that my "recreation" basically involves this thread and a few other news sites at the moment. Maybe you have found something that will be very useful to my endeavors in the future, at least on one level of scale! Although I must admit, I do enjoy coding this type of thing.

Okay, I looked up some xargs stuff...The single flip should be pretty simple, something like this should do it:

In the mmb you can now use BITPOS as the parameter for example X:BITPOS:08. Unfortunately, I can't check whether it works and actually runs faster right now...Also, I'm having some trouble when it comes to multiple flips. The problem is, that you can only -I one parameter in xargs, so some bash trickery (like creating an array with the params in) will be necessary. Now if we get the bitflipping built into the decoder as arnezami suggested (which would be awesome!), that will be kinda pointless, but I least I now know how to move and rename a bunch of numbered files at once (using multiple processes even!)

I think there are two main reasons why we are succeeding in what was deemed to be impossible. The first is that we really went outside-the-box with our approach. The second is that our techniques are such that we can collaborate in our efforts.

I may be a noob here, but as an old-timer on IRC, Usenet, and forums, I've been over-the-top impressed with this project. Every forum with a critical mass of talent and altruistic members seems to adopt a project sooner or later that defines the spirit of the group. And it's often called the Epic Thread. And mention of it can go viral, and it's talked about at trade meetings, cocktail parties, and campfires for a generation.

I'm certain I missed a few here in the past, but this one seems to be the Thread of 2014 in my opinion. Carry on, and thanks!

OK, here's my fixed version of the part 15 TS file. As far as I can see there's a damaged P-frame starting in packet 108 (offset 0x4f50 bytes into the file), but it's damaged at the MPEG4 level and I'm only fixing TS-level things at the moment.

I've set its AF to 7 bytes so that the MPEG4 data will be aligned properly, but I've not put a PCR in it because it's very damaged (and as far as I can see, ffmpeg doesn't seem to mind the lack of a PCR).

It should be ready for MPEG4 experts to start playing with it - if anyone wants to correct the header of this packet, a new P-frame should hopefully emerge. Let us know how you go!

OK, here's my fixed version of the part 15 TS file. As far as I can see there's a damaged P-frame starting in packet 108 (offset 0x4f50 bytes into the file), but it's damaged at the MPEG4 level and I'm only fixing TS-level things at the moment.

I've set its AF to 7 bytes so that the MPEG4 data will be aligned properly, but I've not put a PCR in it because it's very damaged (and as far as I can see, ffmpeg doesn't seem to mind the lack of a PCR).

It should be ready for MPEG4 experts to start playing with it - if anyone wants to correct the header of this packet, a new P-frame should hopefully emerge. Let us know how you go!

34 bits flipped between them (not counting the CC and the last half byte D)Actually, i never checked if there is more header after the 00 00 01 b6 5 which we should repair. I will do that when the rest is done.

Cheers Shanuson

Edit: Ok I looked at your fixed part15 file. There were still many unfixed Ts-packets, most with the correct PID and CC but wrong flags set. The TS-Header of a data packet should really only look like 47 03 e8 1X, with 2 exceptions, the first and last of a frame which should be easily identified.

Edit: Ok I looked at your fixed part15 file. There were still many unfixed Ts-packets, most with the correct PID and CC but wrong flags set. The TS-Header of a data packet should really only look like 47 03 e8 1X, with 2 exceptions, the first and last of a frame which should be easily identified.

That's great, thank you for taking a look. Your changes all look good! I'd missed that packet with the huge AF at offset 0x52fc, good catch.

For the other "unfixed" packets I'd just left the erroneous Payload Unit Start Indicator bits in there. I agree with you that they should be removed - I had left them in there because (as far as I can see) they don't trip ffmpeg up, and if I'm in doubt I like to change as little as possible. I'm always worried about changing too much, it always feels nicer to make as small a change as I can. However, in future I can remove these too if you think it makes it better.

For the other "unfixed" packets I'd just left the erroneous Payload Unit Start Indicator bits in there. I agree with you that they should be removed - I had left them in there because (as far as I can see) they don't trip ffmpeg up, and if I'm in doubt I like to change as little as possible. I'm always worried about changing too much, it always feels nicer to make as small a change as I can.

It's not so much the size of the change, but the quality of the change that's important. There's a true and original state underneath all that corruption, and that's where we should aim as long as we can aim with confidence and precision.

"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

This is just a proof of concept. I deciphered a few of the timestamps on the images and interpolated the rest. Due to interlacing and propagating errors, even if we get good mbs in the pframes the timestamps are still going to be gibberish most of the time. So, I propose we just redraw them. Any concerns about this?

This is just a proof of concept. I deciphered a few of the timestamps on the images and interpolated the rest. Due to interlacing and propagating errors, even if we get good mbs in the pframes the timestamps are still going to be gibberish most of the time. So, I propose we just redraw them. Any concerns about this?

Seems like a very good idea, with a few constraints:- they need to be correct- they are only applied at time of generating the movie file, so not to mess up de frame reconstruction process. In the end it is much better to have them fixed instead of reapplied.

But if you are at it, is it easy for you to also add a side view impression of how the stage would look at a every frame?

I think it would really help the viewers better understand what they are looking at. Including frames that are too corrupted. Also it would trigger a discussion on what happens in every frame.

This is just a proof of concept. I deciphered a few of the timestamps on the images and interpolated the rest. Due to interlacing and propagating errors, even if we get good mbs in the pframes the timestamps are still going to be gibberish most of the time. So, I propose we just redraw them. Any concerns about this?

Another route to go would be to put the frame# in the upper corner (I remember one of the previous gifs posted did this).

Just a quick note before I have to go do other stuff. I'm working on brute-force bit flipping a bit. As I expected, making a good objective function is not so easy, looking at the sum of squared errors across block edges is a good first stab, but it needs more work for sure. The combinatorics is the second problem, there are just way too many combinations of bits to flip, and some sort of clever search algorithm needs to be devised. Also it would really help to know a bit more about the error correction method used by the telemetry stream.

I have one idea for speeding things up though, which I haven't tried yet but will. What if you concatenate n copies of the I-frame together into a single file, each with a different bit flipped, and then put that into FFmpeg. Would that yield n separate frames in a single call? If so, that would amortise the overhead I think.

Excellent work on the TS repair by the way, that will really help because it will allow us to take things one frame at a time. That has to be easier than solving the whole problem in one go.

This is just a proof of concept. I deciphered a few of the timestamps on the images and interpolated the rest. Due to interlacing and propagating errors, even if we get good mbs in the pframes the timestamps are still going to be gibberish most of the time. So, I propose we just redraw them. Any concerns about this?

Reminds me of the Doctor Who Restoration Team who fix old (1960s onwards) episodes. With the credits, they know what they're supposed to say; they know what font etc they were in, so they simply make crisp new sets and slot them in. They use another trick for the titles, most of which would have been identical for each episode. They make a compilation of the best bits from individual episodes and use that for all the episodes.

Seems like a very good idea, with a few constraints:- they need to be correct- they are only applied at time of generating the movie file, so not to mess up de frame reconstruction process. In the end it is much better to have them fixed instead of reapplied.

IMO this is an historical document. First, you should obtain the best version of the data that is present. Ideally, someone should detail all the individual steps to get from the original data to the present so that future historians can verify its correctness (hope you're all keeping notes! ). Once that has been done and archived, then you can start on these and similar techniques to improve the appearance of the video.

For the final work product (if there is such a thing) I suggest a video that repeats the sequence several times interleaved with some text screens that explain who / what / why at a high level.

1) Begin with a brief paragraph about the circumstances and the challenge.2) Full video first pass, show the original video before restoration.3) Another paragraph, introduce the NSF forum and crowdsourced effort.4) A series of before / after shots of individual frames to indicate progress with captions highlighting tools used, round-the-globe contributions, people contributing from their own skillsets....5) Full video, showing a "true" restoration (no enhancements)6) Full video, show a side-by-side presentation. in the top left corner of screen (20%), show a graphic animation of the landing process in the main frame bottom right (80%) show the restored video again. For the graphic animation, perhaps the YouTube video by Oberg N titled "F9R in KSP" is pretty close to what we would need. Perhaps this can be captioned with the interpretation. 7) A final pass, full screen of the team's video including enhancements such as a frame counter at the bottom right and wronkiew's nice clock in the bottom left. 8) End with credits thanking core people - slow scroll - and every username on the forum - fast scroll.

This is just a proof of concept. I deciphered a few of the timestamps on the images and interpolated the rest. Due to interlacing and propagating errors, even if we get good mbs in the pframes the timestamps are still going to be gibberish most of the time. So, I propose we just redraw them. Any concerns about this?

As just an interested onlooker, my opinion is just to correct as much as possible, but not to overwrite anything like timestamps.

Otherwise, someone somewhere is gonna do a "moon landings were fake" meme on this, highlighting the bits that don't match the rest of the stream. (Yes, yes - but you know that they will.)

This is just a proof of concept. I deciphered a few of the timestamps on the images and interpolated the rest. Due to interlacing and propagating errors, even if we get good mbs in the pframes the timestamps are still going to be gibberish most of the time. So, I propose we just redraw them. Any concerns about this?

I agree with a number of other posters (Jakusb, CR) that this is an historic document and should this be restored as carefully as possible, however, I also agree that this timestamp information is important. I would like to see any intensional additions (other than bitflips) to be included not as overlays but rather as additions in a larger screen such that the restored video is the "picture in picture" or as an overlay in an extended sequence such as mhenderson suggests here.

For the final work product (if there is such a thing) I suggest a video that repeats the sequence several times interleaved with some text screens that explain who / what / why at a high level.

1) Begin with a brief paragraph about the circumstances and the challenge.2) Full video first pass, show the original video before restoration.3) Another paragraph, introduce the NSF forum and crowdsourced effort.4) A series of before / after shots of individual frames to indicate progress with captions highlighting tools used, round-the-globe contributions, people contributing from their own skillsets....5) Full video, showing a "true" restoration (no enhancements)6) Full video, show a side-by-side presentation. in the top left corner of screen (20%), show a graphic animation of the landing process in the main frame bottom right (80%) show the restored video again. For the graphic animation, perhaps the YouTube video by Oberg N titled "F9R in KSP" is pretty close to what we would need. Perhaps this can be captioned with the interpretation. 7) A final pass, full screen of the team's video including enhancements such as a frame counter at the bottom right and wronkiew's nice clock in the bottom left. End with credits thanking core people - slow scroll - and every username on the forum - fast scroll.