The technology behind image and video compression is an amazing thing. It began as an effort to get more of our pixelated scanned photographs onto a single floppy disk and has now grown into a scientific discipline that blends visual perception with information theory. Throw in some compression algorithms with size / CPU time tradeoffs and a dash of obnoxious patent trolling and you have a field of study that can keep you busy for an entire career.

The early compression systems (like .gif) were just focused on ways to pack together redundant pixels. Kind of “hey, the next twenty pixels are all exactly the same, so instead of repeating the same bit of information 20 times, I’ll do it once with a note on how many time to repeat it.” It was actually a bit more complicated than that, but you get the idea.

This worked great when images were made from 16, 64, or 256 really distinct colors. If there are only 16 different colors in your image, there’s going to be a lot of repetition. However, as the number of colors grows, detecting duplicates and repeating patterns becomes useless. At some point a clever engineer noticed that while your average image of 16 million colors had almost no points where the same color pixel was used again and again, there were a lot of situations where the pixels were nearly the same.

Human beings sense changes in brightness much, much more easily than we detect changes in color. So, what if we just reduced the color diversity a bit? Don’t mess with the brightness, but just fiddle with the hue of pixels to reduce the number of distinct colors. This can be tricky. Check out this image:

This is our example image, reduced to just 16 colors. You could compress the hell out of this sucker. However, it kind of looks a bit horrible. In our 16 color palette, we made room for a lot of blues. The sky came out in decent shape, but the landscape was ruined. There just weren’t enough shades of orange to make it work. We could go the other way, and favor the orange, which would result in…

The ground came out pretty nice (for a 16 color image) but the sky is ruined. But! What if we didn’t use a one-size-fits-all color palette for the whole image? What if we examined the thing in sections, and just reduced the color locally?

What if we looked at section A, figured out how much color diversity was there, and made a palette just for that area. Then do the same for section B. What you end up with is a quad tree, which is something I describe in my Terrain Project. Start with a big section of image. Is there a lot of color diversity? If yes, then cut this section into four pieces and examine each quadrant. Keep dividing until you’ve got sections with sufficiently homogeneous color.

However, you will still end up with squares with a lot of color diversity. I’m sure some of you remember the early days of the internet, when people turned Jpeg compression way up to make their webpage images load faster on those old 14.4k modems. (Shudder.) The compression works great for those large areas of similar color, but it leaves ugly seams where two things of different hue and brightness come together.

Note those square areas where the sky touches the landscape. You end up with a single square that needs a lot of oranges and a lot of blues. It tries to split the difference, and you end up with a mess. Check out this up-close view of what it did to the rock arch in this image:

Shrinking images is a complex business. I’m glossing over a lot of details here. There are a lot of different ways to reduce color, or to try and preserve changes in brightness while allowing hues to drift. There are tradeoffs to the various methods, based on what is the most important about the image in question.

Now take everything we’ve just learned, and add it to the even more convoluted science of moving images. In most videos, a majority of the pixels are the same from one frame to the next.

When you have someone talking, there usually isn’t a lot of motion in the scene. Their mouth moves. Their head moves slightly. They blink. From one frame to the next, only a very small percent of the overall image needs to change. Sure, there are times when the camera is moving like crazy, and in those cases it gets harder to get much in the way of savings. But most of the time, 90% (a number I made up, but is reasonable) of the pixels will be the same between any two adjacent frames.

But even among those pixels that are changing, they’re not changing much. When Rutger Hauer moves his mouth, there are slight shifts in the lighting on his cheek, but it’s still skin-tone under a blue light.

A single 1920×1080 Blu-Ray image is comprised of 2,073,600 pixels. At full color, each pixel would take 3 bytes, which means a single uncompressed frame would be 6MB. That’s 144MB of data a second for video, or 8GB a minute. A dual-layer Blu-Ray holds 50GB of data. Which means that on the average Blu-Ray disc, by the time you were done with the FBI warnings, the main menu, ad the trailers, the disc would be full. Without compression, you would literally run out of space before you reached the opening credits.

So we’re getting a ridiculous amount of compression, is what I’m saying. We’re getting more than people might have thought was mathematically possible 20 years ago. Videos aren’t stored as a series of fixed images, but instead they begin with a reference image / keyframe, and then subsequent frames are built by keeping track of the slight differences from one instant to the next. These changes are cumulative, making pixels brighter or darker as characters move around the scene. Then the camera undergoes a significant change (the lighting changes, or the POV shifts) and we get a new keyframe, and the process begins again.

Now, you don’t want to end up with a situation where a fast-moving scene of flashing lights ends up taking a ton of space. Without some sort of safeguard, a really busy scene could bloat the individual frames. You might end up in a situation where the device is physically incapable of moving data off of the disc rapidly enough to keep the video going without stuttering. (I don’t actually know if this is possible now, but I know it was a concern in the past.) You don’t want some section of shaky-cam and flashing lights to blow your storage, throughput, or CPU budget. So, videos are throttled to only feed so much data a second. If the raw footage is too complex, the compression will reduce the image quality. Effectively, the movie will get blurrier. Since this happens mostly in sequences of flashing lights and rapid camera movement, it’s actually hard to tell.

Now, this means that some types of things will compress far better than others. For years I wondered why I couldn’t find a decent version of Groove is in the Heart by Dee Lite. The YouTube version is pixel soup. You can google around and find slightly better versions, but they’re all basically mush. Perhaps the label just hasn’t seen fit to put out the official video, but I kind of suspect that problem is the video itself. Made in 1990, it’s almost perfectly engineered to thwart modern compression techniques. Most of the video is people in multicolored clothing, dancing in front of a kaleidoscopic background. They are going to be very few reusable pixels from one frame to the next.

Which brings me, 1,200 words later, to the point of this article. As I said above, as you watch a movie it’s really just altering the on-screen images a little bit at a time. But what happens if some of those keyframes end up missing? What happens if your video is riddled with Swiss-cheese gaps, and you have sections where you’ve got all the changes, but not the base image being changed?

This is apparently possible for people downloading movies via bittorrent. The peer-to-peer nature of the thing means that you download sections of a video chaotically, based on what everyone else has, what you need, and which peers are available. Apparently video players will play these fragmented things. Which results in the following video. Note that according to the poster, no editing was performed on this. This is exactly how the incomplete video appeared:

It’s rather interesting when even complete video files skip, and end up skipping a new base image. A few frames here or there the changes and blur aren’t drastic, but often noticable. When it skips a base image you get some pretty interesting ghosting and colors out of the video for a while.

I couldn’t watch more than a min of the video because the music was painful.

Also, They will only play if you have the first several blocks, and no major sections missing, or it crashes at a major blank or doesn’t recognize the file as a video.

Yeah, I disbelieve. Click and hold on the volume slider and drag it all the way to the left – that should completely mute it.
Also: Ah, yes, ‘volume control, Volume Mixer, physical mute buttons on your computer, and turning the speakers off.’ We have dismissed those claims.

I mean, I actually liked the music, but my point is that there are several ways to view the video without it. It’s certainly not as if any of the original audio is present (i.e. there’s nothing to miss that’s intrinsic to the mechanics of the corrupted video stream should you mute it).

Maybe it’s a browser issue? I’m on IE (*sigh* work computer), and testing by cranking the computer’s volume to its max (with pretty darn good headphones), if I drop Vimeo’s volume slider all the way down… the video plays silently. So… yeah.

That looks less like BitTorrent, and exactly like what happened when my hard drive was on the fritz and would randomly take ~1000 milliseconds to deliver a request. At least using VLC, it’ll just play the bits you have downloaded perfectly, and just stop once it hits an undownloaded piece (Or so I’ve heard *cough*).

Either that’s the case, or they’re using some really small pieces on a very large file, where that person’s media player can skip over the frame or two that fits into each piece.

Quick way of ruining video files: toss files into a USB stick from Windows, into a Linux system. Delete them from the stick and replace with the copies on Linux. Take them back into Windows.

Note: I don’t know what filesystem you need in Linux for this to happen. I also don’t know if it works on videos in any format. Moving a file from Linux to Windows or Windows to Linux may be enough. Don’t know if moving through a network causes this, however.

That’s… unlikely. Generally speaking a file is just a stream of bits. Both Windows and Linux can quite reliably and accurately copy streams of bits. As someone who moves video back and forth between Linux and Windows with some regularity, I’ve certainly never seen this effect. That includes copying videos from USB sticks onto my Linux file server, then streaming the videos to my Windows desktop.

So why have you seen it? One possibility is that you have video files with alternate/multiple data streams. Copying multi-stream files between different operating systems is fraught with peril. Unsurprisingly, no major video format (or formats of anything intended to be cross-platform) uses multiple streams. Unless you’re doing something odd, I’d be surprised if you were encountering this.

Possibility two: Windows and Linux are handling the USB stick differently. Uncommon, but entirely possible. I have a USB drive kicking around that Mac OS X and Linux like, but Windows hates. Might be a glitch in the hardware that the platforms handle differently. If the problem is limited to one or two USB sticks, this would be my guess. Another possibility is that Linux doesn’t entirely understand the format on the USB stick. USB sticks tend to be formatted FAT32, which in practice means a 4GB limit on any single file. If modern Windows are using some extension to store larger files, but Linux doesn’t know about it, that could certainly caught problems reading the file.

Thinking back I might’ve moved files between partitions/hard drives from Linux. I have some memory of putting files on both FAT32 and NTFS systems, and only ones for the other went strange.

It should be noted that that was a few years back and I have some preternatural ability finding, and using, versions of Linux distros that act more wonky than they should. Even when using ones that are widely considered stable. If one of them has a broken piece handling EXT-FAT, or EXT-NTFS, transfers it can get unpredictable.

And let me tell you, I used an old (2002, I think) Linux distro once that hammered in pretty hard that you simply don’t move stuff from NTFS to EXT if it’s in any way sensitive (cares if it loses individual bits). So moving something that was TARred? Not a good idea.

Also thinking back, I have history of bad hard drives, USB sticks and motherboards. Right now I have stick that didn’t work on a friend’s computer without first formatting it, and then didn’t work on mine until I reformatted it. Both used Windows XP. That computer of mine has never been able to boot from any USB stick despite claiming the functionality in the BIOS, so I’m thinking it doesn’t handle USB ports quite right. Another stick did work on both computers without formatting.

Since it was several years back, I can’t remember if it was that one drive that started having bad sectors. If it was, then I’d say that’s the culprit. But the weird thing was, when I moved files around in Windows, no problem. When I moved them around in Linux, again no problem.

My memory is way too bad. I’m going to have to think about this some more since I think I had something similar happen last year as well, but it’s too late to check them now.

I hadn’t considered the possibility of NTFS getting involved. If Linux was reading or writing to NTFS somewhere in there, especially if it was a few years ago, there is no end of possible badness. I don’t know if things have improved; it doesn’t come up for me these days.

Bah, I didn’t know what your point was going to be, so I was getting ready to point out to people this is why videos can look funny when you skip a key frame, but it turned out that’s exactly were you were already headed :(

At first, I was actually getting strangely nauseous from watching it… and I never have a problem with shaky cam. It’s like I couldn’t focus on the important parts of the movie, and my brain just couldn’t handle it.

Then at some point, my brain took a step back and started doing a weird analysis, noting how the accidental repetition suddenly seemed to impose extra meaning and emphasis on it. And how the (almost) still shots then resulted in those weird morph sequences, which I then started trying to piece together. Same with the suddenly frozen chunks of scenes that hung around. I found myself always wanting to piece together why it was left on screen and how it related to the rest of it.

I suppose this could be viewed as a commentary on how trained I am by modern video media. I have certain expectations about how things like, zoom, morphs, etc. function and what that means to the story, characters, and narrative being portrayed. To have that all psuedo-editing thrown around chaotically pulled out that need to understand, even if there wasn’t really anything to understand.

As a guy who used to work in image and video compression professionally, I’m not 100% sure I buy that this had no digital effects applied. However, with the variety of different codecs out there I suppose a few of the more unlikely “effects” of trying to play a damaged video are remotely possible. (It’s really hard to prove a negative.)

Still, most codecs would puke at such an incomplete file. (That’s a technical term. ;-) )

In the end, I think I just comes down to heavy editing of digital gibberish to add a bit of a “theme” to the bits that were included.

If any of you were getting bored of Shamus’ technical talk near the end there, just be glad he didn’t get into wavelet compression. ;-D

I’ve gotta agree with HiEv here… there’s no way that wasn’t edited. I’ve used multiple players, multiple codecs, and multiple incomplete file over the years, and have never seen that sort of error occur. Particularly the repetitive skipping and replaying involved.

Most of them seemed fine to me – the looping is super weird, but I can see how it could happen. It looks like the codec is using a circular buffer for storing some intermediate data – decoded key-frames, or motion-vectors or something – and isn’t checking timestamps for validity before composing like most other codecs are. It’s the equivalent of when a game freezes and your sound starts looping – that’s your soundcard (probably OS more likely nowadays) just keeping on going through it’s circular buffer without the game updating it. All the other stuff is pretty classic application of motion-vectors (that trippy slime-trailing and the “painted-over-scene” effect) and other inter-frame deltas to a missing key-frame (incidentally key-frames are normally also written every few seconds – say 5 – to reduce error accumulation, which is why a partial video will often “snap back” partway through a scene)

The original isn’t a music video, it’s an TV show with ~45 minute episodes. The person who made that video just took bits of the incomplete video file from one episode, edited them together, and added their own soundtrack that wasn’t part of the original.

Oh great, I’m going to start noticing this stuff everywhere now aren’t I? And here I always prided myself as someone who never worried about the the number of colors or the blurriness of the pixels. (Edit: btw, did anyone else blow up the very first picture to see how much difference there actually was?)

So is this all done automatically with a single algorithm when compressing a movie into a DVD or Blu-Ray disk? or are you able flag say more “action-heavy” scenes in order to avoid losing quality?

I believe modern high-definition compression algorithms use quality based compression in addition to a hard bit-rate limit. This means that you tell the code the total size of the compressed video you want, and it does several passes to estimate the total size and adjust the overall quality to meet that goal. So if you have a few action heavy scenes they will show up fine, as long as there are enough “talking in the coffee shop” scenes too. If you get too crazy, the quality will go down (like with Groove is in the Heart) but most of the time you can borrow space from the calm scenes to pay for the crazy ones.

Basically that. You can tell it, “Don’t go over this amount of information in a single second,” which you do to make sure the DVD/BluRay player can still play it.

Then, you tell it what quality you want and have it do multiple passes (typically 2, I think). It uses the first pass to analyze the video and find which spots are easy and which ones are hard so it can decide how much information it can use in each section for the final pass in order to get the closest to the quality you want everywhere without ever going over the limit you’ve set for it.

Not unique to bittorrent obviously. These artifacts can occur when parts of a file is missing/damaged or in case of real-time streaming skipped.
So you got digital cable or satellite, streaming of video real-time (youtube and plenty other services real-time bittorrent streaming as well), games (pre-rendered cutscenes), downloaded video (checksums should normally catch
these during downloading but…) and then there are bugs or glitches, ranging from software bugs (driver/mediafilter conflicts, GFX card drivers, OS; config/user faults) to hardware glitches (RAM, CPU, disk, power fluctuations).

The same happens with music, images, and text documents. Ever noticed a webpage where the content sorta shows up but everything looks messed up and then you refresh the page and everything looks great, color, layout, styling. It’s possible that at first try the .css file was damaged or did not get through in one piece.

It is also not uncommon for hard-drives to deteriorate over time, which might cause the odd file to end up garbled in part. The worst however is RAM or CPU issues as those are involved during checksum calculations and might cause false positives.

Shamus, maybe you could do a post about checksums later some time. And point out how great they are (and aexamples of asll the playes they arte used, TCP/web, video, audio, games, HD’s etc.) but also the flaw they have (using less data to represent more data than the representation data can match, birthday paradox etc.)

I remember learning about this sort of stuff in one of my IT classes, though that focused mostly on streaming for the web.

Whenever I am watching a film on Netflix Instant, I crack a smile when it goes from a rather static scene to an action scene with a lot of motion, and suddenly it goes from pristine appearance to artifacts everywhere.

Though the absolute best: rain. A film with rain in it is going to get artifacts out the wazoo.

I have a good understand of how lossy still image and video compression work, but I’d never seen it explained quite this way. It’s very clear. If I ever find myself wanting to explain it to someone else, I’ll point them here.

Recently took a class on compression. That shit gets crazy. One day someone realized you can split the image into2 parts contrast and color(like a black and white image and then the colors) as long as you don;t mess with the contrast image you can cut the amount of color data down to 1/16th before people begin to really notice I can;t remember what the compression is called but at the full image vs 1/4th color data no one could reliably tell them apart at 1/16th color data about half the people could tell but it was only after intense studying.

This is what JPEG and most other related image/video compression schemes do (though it’s three components, not two). They convert the colors from the RGB colorspace (Red, Green, Blue) to the Y’CbCr colorspace (roughly speaking, Y’ = brightness, Cb = Blue/Yellow range, Cr = Red/Green range). You can lose a lot more information in the Cb and Cr components than the Y component and still look OK usually, so you can optionally compress them more than the Y.

Still, if you compress the Cb and Cr too much you get this color blurring that’s especially noticeable around sharp edges of objects in images.

Interestingly, people actually intentionally exploit these glitches in the video compression algorithms for interesting effects. It’s called ‘datamoshing’ and some of the results are incredibly trippy. One of my favourite examples is this video.

This tutorial on datamoshing has a great explanation of what is happening in the compression algorithms and how to exploit it.

So, out of curiosity, is this another case like camera film and vinyl records where -by the criteria of fidelity -analog technology is superior? Or does electromagnetic tape have similar problems and if you want the “cinema experience” you’re just going to have to get a reel-to-reel projector and set it up yourself?

Well the problem with analog data storage is the copying. You inevitable lose a bit of it with each copy, until you might hardly recognize the original. Digital data on the other hand can be copied exactly without loss of information from the source. That is reason enough for digital to have almost entirely sent analog media into oblivion.

P.S.: Funny, why did this post end up “awaiting moderation”? Is the word “oblivion” now a potential bad word?

It contains a minute quantity of Space Bass, which is almost makes it worthwhile. Other than that, I can imagine it would make for an interesting experience for those who partake in the activity Bootsy refers to in the end.

Unrelated to compression algorithms but similar in results, here’s video of Virtua Fighter after hardware damage. Tech glitchyness is really a beautiful aesthetic. Not enough games/films/animations take advantage of it. Fez does a bit with this, it’s one of the game’s better attributes.

I think the best example I’ve seen of this kind of thing is here. Whether or not the commentary is to your liking; trying to average out the inputs to work out approximately where the road is, while mentally taxing, could actually make an interesting game mechanic.

>> “For years I wondered why I couldn’t find a decent version of Groove is in the Heart by Dee Lite.”

I think the real question here is why you WANTED to find Groove is in the Heart. There is no “decent” version. Whenever I hear the part of that song where she’s wailing I wonder “why doesn’t someone put that poor cat out of its misery?”

I guess I am the only one, but all that lossy compression technology we have to me seems kind of like a time bomb. Most of it nowadays is heavily based on human perception and trying everything to remove all that a typical human would not notice anyway (this is, for example, the main feature of mp3 audio).

I think this will bite us back one day, in either or both of two exciting ways: humans improving themselves biologically through genetic engineering (How about eagle-like sharpness and insect-like speed? Even a seemingly crisp high-definition video would probably then look like a choppy, sluggy blurry mess), and aliens with vastly different eyes that might have a really hard time noticing the actual information in still images, video and audio amongst all the noise that humans don’t perceive, but them do.

You call this farfetched? Tell that to your grand-grand-grandchildren for whom media from today might be entirely incomprehensible to the point of calling our time the information dark ages. Heh.

(Now to think of it, this could make for an interesting novel, heh.)

On a serious note, I am highly impressed be the ways these compression technologies have gone forward, particularly the way some modern video codecs, as mentioned in Shamus’ post, store few full frames and mostly changes relative to the previous frame.

Concerning your first point, I think our future bio-engineered overlords will be fine. We already have TV’s that interpolate ‘missing frames’ when they play 24 fps content at 300 MHz. Sure, this has it’s own shortcomings, but by the time we’re doing bio-engineering I’m sure we’ve got those sorted out. Besides, we’ll all agree that LP records are inferior in audio quality by todays standards, but most people will agree that that crackling adds to the charm of the medium.

About the alien thing, I was reminded of this almost, but not entirely unrelated fragment from the show QI: Pigeons and the movies.

BTW, if you found the keyframe/delta -thing interesting, maybe here’s another fancy nugget of knowledge;
For the best quality per bitrate there’s a compression process known as two-pass encoding (or n-pass). The first time the encoder goes through the stream, it isn’t actually encoding anything, it’s just taking notes. Notes about where the ‘heavy action’ is, lots of pixel changes versus the quiet bits, where only tiny bits of the frame change.
On a subsequent pass, it then distributes the available bandwith, so that the high action parts have much more bandwith allocated than the relative quiet parts. That way the intense moments might be using something like 2000 kbit/sec, and the quiet scenes maybe 200 kbit/sec. That way you wouldn’t notice a difference in quality like you would if every second was encoded at the same bitrate.

“We already have TV’s that interpolate ‘missing frames’ when they play 24 fps content at 300 MHz. Sure, this has it’s own shortcomings, but by the time we’re doing bio-engineering I’m sure we’ve got those sorted out”

Wait, what?
Lossless compression works by throwing away redundancy, not by removing and putting back real information (which would be impossible). Which it why you can’t for example compress data to infinity; or why zipping JPEG files normally won’t give you any benefits.

CD is “flat within a fraction of a decibel from at least 20 Hz to 20 kHz at full output”,
LP is “with some cartridges, reasonably flat within a few decibels to 30 kHz, with gentle roll-off”.
LP “Frequency deviations of 5-10 dB or greater are not uncommon in the 20 kHz range for many records”
LP high frequencies are not full signal.
LP low frequencies are obscured by turntable rumble.
LP theoretical range is higher than ideal but no normal LP equipment can handle it.

Practical limit of analog/digital is 2Hz-48KHz samplingrate (1Hz-24KHz frequencyrange) and 20-24Bit, beyond this differences should be transparent. (during mixing/mastering higher bitrates and ssamplingrates is obviously desired, but that is mastering related and not playback and a whole different subject)

The defacto word on Vinyl vs CD is http://wiki.hydrogenaudio.org/index.php?title=Myths_%28Vinyl%29
Hydrogen audio is not (or at least should not
be) audiophiles, they are mostly audiofacters, if you make any statements on audio quality on the forums without backing it up with a ABX test they’ll virtually bitchslap you.
Most of the audio codec (and other audiotech) developers frequent the forums well.

Interesting fact.
My dynamic microphone has a SNR of about 85dB under ideal conditions in my work room (aka Room Sound). A Vinyl is unable to fully reproduce that as the SNR is 75dB, a CD audio on the other hand is able to reproduce it since it has a SNR of 96dB.

Please keep in mind that vinyl is analog and CD is digital. CDDA was created as a digital alternative (if it was a replacement it would have kept the sound characteristics as well (PS! There are DAC’s that let you color sound that way though!))
Vinyl runs into many physical limitations (mastering/production/equipment/needles/motor/misc playback issues/LP maintenance),
CDDA has artificial limitations. (CDDA could technically handle the bit/freq that any WAV stored on a data CD can handle)

But there is a reason why vinyl is still alive and cassette tape is not, good vinyl quality, good care and good equipment will give you the best that analog can give.
(Note! Majority of vinyl these days are [ADA/DDA] and not [AAA] like in the olden days, modern vinyl fidelity is far greater than old records, provided the modern mastering avoids the loudness war, both CD and vinyl sounds like crap then)

Basically its an electronic component that changes size depending on the current that passes trough it.Currently that means that it can be used to make memory that stores data physically,like a vinyl record,but can be easily erased and rewritten,like magnetic memory.

But,it also means that we can,in the future,increase the size of bits,so instead of just 0 and 1,computers will be able to work with bits that can have multiple values(like an analogue computer).Hence instead of of further miniaturizing computer components,we will be able to increase the amount of data a single component can process.

Well to my knowledge memristor research is more memory size and density, not making computers analog.

Besides, analog data storage is the most lossy of all, because you have no absolute control over the amount of loss and cannot, for example, compress with a loss of information and then keep this compressed version unchanged across copies.

Analog copies always automatically lose information. You can reduce the effect but entropy is merciless and precision is finite. And worst of it all: just because you lose information when copying analog media, that does not mean you save storage space. The “free” space is filled with noise instead.

“Well to my knowledge memristor research is more memory size and density, not making computers analog.”

Currently,yes,because its the most efficient way to integrate it with existing technology.The implication of the component,however,is the ability to make a chip that deals not just with on and off states,but with states in between.That,basically,is an analogue computer.

As for it being more lossy,I have no idea,Im not an expert in the field.But at least it seems to be more resistant to aging than current disks.Still,nothing we have can beat the eternity of a rock.We should switch to rock computers,and preserve our culture in the millenia to come.

No, what you are saying is that computers entirely made from memristors may be built to use more than two digit. Such a device might not use the bit (=binary digit) as its smallest unit, but something with more states in a single digit. For example, if it could represent sixteen states, it would have hexadecimal digits. But as you can literally see, there are still digits, everything is still digital.

Analog technology does not have the notion of digits. Instead it uses continuous states to, theoretically, more precisely represent a piece of information, but that always (as in, physically inevitable) comes at the price that copies of analog data will lose information and introduce small amounts of noise, that over generations of copies cause the information to deteriorate.

Digital data on the other hand has definite states out of a finite set of values. This places an upper limit on the precision of a value (for example, the precision of decimal, i.e. floating point numbers), but that limit becomes moot in most practical applications given enough digits used in a single value, while still retaining the property that copies are exact 1:1 duplicates indistinguishable from the original.

And in terms of the eternity of a rock. That is actually a good example here, too. Encode analog numbers into a rock, say by straight lines of varying length, their length representing the number, and see how precisely you can a) produce the lines and b) measure them later and c) produce new lines (copies) in a different rock. Compare that with digital information carved in rock, let’s just say a succession of dots and dashes that together form a single number out of ones and zeroes, and you could store stuff that could still be read and copied later without any kind of information loss (unless the rock itself gets damaged, but that is a problem in both cases).

Well,technically all the world is digital,only the digits are so small and numerous.But thats just splitting hairs.

Anyway yes,computers will,even with memristors,still be digital.However,that wont be due to technology,but because we still have to develop a way to easily program with that technology.Maybe in a few centuries we will develop a language that can operate on a really analogue computer.

Interesting, but Quantum bits are more interesting. As far as I can understand (though the wikipedia articles never called it this, so you heard it here from me first folks):

A quantum bit might as well be called a floating point bit,
as it has the potential to be -1.0 to +1.0 at it’s simpler two planes/vectors,
add another vector (third?) and I have no clue how to describe it, two dimensional floating point bit maybe?

This means that a floating point number could be stored in a quantum bit. That is a potential storage reduction and computational improvement at the magnitude of 2147483648:1 (or is it 4294967296:1 ?) Or is it… Power of Two “rule” kinda gets messed up with quantum bits so I really have no idea on that one.

FYI, computers used to be analog. The Digital vs. Analog war was pretty much settled before I was born. 8 bits, until quite recently, provided more gradient levels than an electronic screen could display. 16 bits provides more levels than the human eye can make out on an analog oscilloscope. Increasing bitwidths have provided finer and finer levels of quantization, to the point where, for example, a 64-bit Zbuffer provides unlimited depth resolution in 3D gaming for all functional intents and purposes.(hint: the RANGE of a 64-bit value is 20 digits long. If we ever go to 128 bits, the range will be nearly 40 digits long. A Googol can be over-represented with a scant 512 bits, 64 bytes, 16 32-bit Int values, or just 8 Long Integers. More bits are required to store a simple cube in OpenGL display format.) Digital also has the advantage of “perfect duplication,” which analog just can’t provide.

Computers aren’t going back to analog any time soon except for fun experimentation purposes. There’s just no real reason to.

Memristors are most interesting as an application for volatile memory. Being passive components, they have no functional difference between powered and unpowered states, meaning a computer using memristor-based system memory would require less than half the energy of a modern-day machine, and would require no power to maintain sleep mode indefinitely. For a similar technology, look up Ferromagnetic RAM.

Aliens “with vastly different eyes” will also have the same problem with losslessly compressed video (and analog storage). Everywhere where you quantize and/or range-limit the data the universe throws at you, they have lost.
From the frame rate to the sound and color spectrum, as well as all of our display technology; and even filming techniques, framing and artistic visual expressions, all are completely tuned to how our biology and psyche receives and processes data.

Even if we find aliens (for which there are arguments for and against; I don’t think we have enough information to make a strong case either way) that are similar enough to us so that we can meanigfully interacts (and given the vast number of possibilities, I’m not sold on that either), I have no delusion whatsoever that we can share any of our music, movies and books with them.
The same goes for live stage plays and musical performances. All the ideas and harmonies are completely subjective and totally dependant on our biology.

I agree. But I would compare this more with the situation modern movies vs old black-and-white silent films. Of course those lack two dimensions of information, color and sound, and may appear boring and cheap nowadays, but for the purpose of retrieving the information contained within, this is not a problem.

Those aliens might not find our movies or music enjoyable, they might recognize missing hues of color, pitches of sound, they might find the resolution laughable low or whatever, but what is there can be deciphered.

Lossy compression on the other hand loses information in a relatively arbitrary manner. Instead of, say, only containg sounds between a certain lowest and highest frequency, mp3 audio missing entire blocks of sound if they are low and the right before them was something loud. Our hypothetical alien might at this point consider the file damaged, and might wonder what it just missed that should have been in this microsecond.

On the other hand, a representation such strictly tailored to human perception could give those aliens the means to get a better understanding of human senses, because ideally everything contained in such a piece of data is within the human perception, no less and no more.

Reminds me of a funny incident that occured once: I rented Poseidon (the poseidon adventure remake), and in one scene it seamlessly jumped to another one 7 minutes later, it took us about 5 minutes to realise they skipped a plot detail (although they said they’d climb across some shaft, we assumed it happened off screen). Only after did we watch the timing on the DVD player did we realise, but there were no distorted pixels, sound flowed smoothly, etc.

Memresistors might increase the data density on a disk, BUT Shamus was talking about streaming media from the NET. In that case there are HARD limits to the bitrate some systems can achieve, and these limits are not tied to the memory storage (simply put there is never enough of bandwidth).

I know that I’m a little late to the chat party, but I found this relevant. A similar phenomenon to that MadMen video can be seen by those with a digital television receiving poor transmission from the broadcast source. Those with cable, satellite, & other dedicated means of signal reception won’t experience this, but those who rely on good (?) ol’ fashioned airwaves can behold this in all its pixelated glory.

Apparently, just like with the video above, a key frame will be missed after a bout of bad signal fidelity, & then the on-screen frames will freeze, scramble, & sometimes motion will be seen that is affecting an out-of-date frame. It looks like a the people on screen are wearing a screen-capture of themselves like an ill-fitting blanket for a second or two, before the next key-frame comes down the pike & everything catches up. It’s a bizarre effect, & I sometimes wonder if my toddler will grow up to tune this some of weirdness out, like I did with snowy static on the screen while I was trying to watch Captain N on Saturday mornings.

According to Wikipedia, Devin Townsend plays some form of heavy metal. This is the part where I sink to your level and sneer at YOUR music because it’s so different from mine. My subjective tastes are superior to yours because I like it more!

Isn’t this fun? Isn’t this a rewarding conversation? Isn’t it great being a couple of insufferable music snobs?

I didn’t mean it like that!
I’m not trying to be some kind of music snob, I was just joking around! Anyone has the right to enjoy any kind of music they want! People with different tastes of music mixing together is what keeps music itself fresh.
This is one of the things that annoy me about online communication; I intended the comment as a humorous jibe at the song (Somewhat like Amarsir and DGM have already done a few comments back), not some kind of elitist attack on your musical taste!
Looking at it in hindsight, it might’ve been a bit too much (With the whole cringe-inducing part….yeah, I’ll just shut up next time you mention music), but the last thing I want to do is come across as some kind of insufferable music snob.

Sure you could say Devin Townsend ‘plays some form of heavy metal’, but why do that when you alternatively say he ‘wrote a concept album about an extraterrestrial being searching for the ultimate cup of coffee’.

And if you think image and video compression is complicated – take a look at audio compression. The theories involved in audio compression are much, much more complicated and rooted in human perception – and typically result in lesser amounts of compression!

Good explanations, though. I worked on MPEG (video and audio) compression, so I’m familiar with its techniques. I didn’t know how .gif (which I thought was just raw) and other early techniques worked.

Are they?
As far as I know, both jpeg and audio compression is based on FFT (That’s fast fourier transform) or similar (Discreet Cosine Transformation in the jpeg case — which is FFT for integers).
The “human perception” bit applies to both audio and video compression.
Audio: Make an FFT and remove the weakest frequencies. The louder the main frequency, the more of the others you can remove. Repeat until the predefined bitrate is reached.
Jpeg: transform to brightness/hue/saturation space, reduce hue/saturation resolution, then cut into 8×8 blocks and make DCT, then remove the weakest frequencies.
jpeg2: Do the same, only using “wavelets” going in diferent direction.
MPEG2: a few jpegs, and in between stuff is reconstructed from motion information and difference pictures (there are two different ways for this, but let’s ignore that here). This gives us the really beautiful transitions in some scenes in the video. The jpeg frame is missing, but the diffference and movement information is still applied.

I think most of the psycho-stuff goes into video compression these days, and I can’t really say how it actually works. Is it just much more clever motion detection at the cost of more processing power? Anyway, there needs to be a solid understanding of what a viewer will notice and what not, in order to have a good priority list for what to leave out when compressing an already well-compressed DVD video to 1/10 its size, without the audience noticing…

The human mind is far more able to interpolate visual input than it is able to interpolate audio input. (This is why 2D animation/movies/TV/games/print magazines work, this is why 3D animation/movies/TV/games/magazines work (for most people at least))
This is why audio glitches stand out so much more than video glitches when you watch a movie for example.
I have found I’m quite able to “ignore” bad video as long as the audio is quite good.
If the audio is really bad then even the most amazing image quality can not save the experience from sucking.

“Audio: Make an FFT and remove the weakest frequencies. The louder the main frequency, the more of the others you can remove. Repeat until the predefined bitrate is reached.”

This is how I’ve seen it explained many, many times. And it is wrong.

You can’t ‘remove’ frequencies. If you do, you can’t rebuild your signal in the time domain. What you actually do is look for the perceptually inaudible frequency bins and reduce their bit depth, thus making them full of (imperceptible) noise. So you’re really just hiding quantization noise where it (hopefully) can’t be heard.

That sounds simple, but the whole ‘perceptually’ is extremely non-trivial. MPEG Layer-III involves extremely complicated (and proprietary) psychoacoustics for trying to determine what is and isn’t audible, and applying those decisions iteratively to the frequency coded signal. And then it uses Huffman coding on top of that! And it still has trouble with noise bursts in the higher frequency ranges.

The much simpler algorithms in video compression achieve much higher compression (and still have trouble with grass (high frequencies) and sky (color quantization)).

okay! I didn’t know about the quantization bit. Just looked at the wikipedia article … I think this’ll take a little more time.

Although I’m still pretty sure you _can_ completely remove the low-amplitude frequencies from a Fourier-transformed function, transform it back and get something that looks like the original, but with less detail.
As far as I understand, an mp3 looks sort of like a line from a jpeg image, except without those ugly joints at the the 8×8 pixel block borders.
Also: ogg vorbis is free (no Fraunhofer Patents required), and it’s more efficient than mp3, regarding quality/storage (not regarding cpu time, I think)

Anyway, I can understand audio compression to the point that it still make sense. but still I have no idea how or why video compression these days is so much better than a few years ago, how motion compensation works (decoding: ok, not horribly difficult to understand the basic principle: move this thing by x, then apply colour difference y), but encoding?), and all those other fancy things.

Maybe the theory behind video compression isn’t going as deep, but for all I know I’d still say that the algorithm itself is more complicated. Even if there’s a lot more different things you can do to a video (2D images, plus time, times three colour channels -> 6 dimensional data matrix!) to make it smaller, compared to a vector, which audio data is (or two vectors for stereo).

I actually worked on the central engine of an MPEG-2 video compressor, the ‘motion estimation’ block. As y’all know, MPEG video has different kinds of frames: I (basically just a JPEG), B (backward predicted), and P (forward and backward predicted).

To get the B and P frames you divide the picture into tiles (in some proprietary way, ours was hierarchically descending in size) and then perform searches of the previous and/or next frames to see which area of the picture the tile best matches (By subtracting! You’re looking for the smallest absolute difference.) The you store the differece and an offset vector.

It’s remarkably low tech, and very brute force. And it works quite well as long as the motion between frames is more or less linear, maybe panning in or out slowly. Big transitions or weird rotational motions screw it something awful.

Now, I know this was just a long set-up for something else, but I think you really should have posted the 16-colour-images as .png, not .jpg. Would also have proven the point of how well that compresses (extremely well!)

That just reminded me of Psychonauts as in the Milla Dance Party only the Milla music is better,reminds me of that one figment i know where is but no idea how to get and it´s at the highest point of Her Mind.

Having worked at a print shop, I have to say that people just don’t understand what “compression” or “scaling” are. They loved to bring in a 300×200 pixel (or so) jpeg and ask “can you blow this up so it is like, 2’x3′ on this poster we need made?” For non-graphics folks reference: a real-size, “perfect” quality (based on the original’s compression algorithm, of course) printout of that image, at our default of 300 dots-per-inch, would be 1 inch by 2/3 of an inch.

My favourite bit in that direction was someone trying to send me an image via e-mail, finding it too large, and then scaling it waaaaay down, so the BMP file would be below 1 MB :)

Of course the end result was completely useless, so I told him to please make a jpeg but with higher resolution.

… so he scaled the BMP up again and made a jpeg.

Oh, and a tiny nitpick: People really should say and write “ppi” if they mean pixels per inch, not printer dots (for photos, your laser/inkjet printer can take about 8×8 dots to resolve a pixel, and you won’t see any individual pixels — for line art it’s a different story, though)

Personal, unasked for, tip regarding these sort of news: Unless you’re willing to do the leg work yourself and find out what is actually happening, ignore them. Few news sites report on patents etc. in a truly informative way, so you don’t get any kind of a clue what is going on.

For instance I’m not willing to go through the trouble of finding out what is going on and what the patent actually covers and so on. I have no clue what that patent actually covers and the quote they have from the patent is useless in that regard. So I can’t tell if they’re patent trolling or what.

The quote in news like these are always from the portion that describes the big picture, so if someone patents a new type of a lens for sunglasses the quote could be something like “a system that describes a see-through material that affects light passing through it”. The technology could be incredible and the patent could be perfectly alright, but that quote makes it sound like they just patented anything that in some way lets light through, changing it at the same time.

And let me tell you, reading a patent file is harder than it should be. Either that or I’m an idiot.

Yeah, I’m not putting much thought into it myself, it’s just that the name rang a bell because I had read that post from Shamus recently, and I felt curious about what he knew on the subject and what was his opinion (if any).

The company doing the suing is “Worlds Inc.”, while Shamus’ old company is “Activeworlds Corp.” — not the same. Although what they do looks fairly equal, from glancing at both companies’ websites for a few seconds.

There are probably hundreds of different lossless compression schemes out there. LZW (Lempel-Ziv-Welch) compression is used by GIF images, and for a long time the patent on it caused many problems for developers wanting to use it (the patent has since expired). The patent issues led to the development of the far superior PNG (Portable Network Graphics) format, which uses the patent-free DEFLATE/zlib compression method in addition to some filtering techniques to help it get better compression. For black and white images, such as text, TIFF (Tagged Image File Format) uses G3 or G4 lossless compression (and RLE (Run Length Encoding) or LZW lossless compression for color).

Those are the most common still-image lossless compression technologies, and they mostly work more-or-less the same. They take the image a pixel-thin line at a time and try to compress repetitions and patterns along that line. This is easier in GIF since it limits you to 256 (8-bit) colors or less in the palette, and easier yet for G3 and G4 since they only work with black and white (1-bit) images.

PNG’s filtering technique changes each pixel into a difference of it and the pixel before it, and then applies a similar compression to your basic ZIP file. PNG supports up to 40-bit images (32-bit color plus an 8-bit transparency (alpha) layer), but its compression still works better at lower bit depths.

I’m afraid I’m not quite as well informed when it comes to lossless video though, even though I worked for a company that developed a (now defunct) lossless video codec.

Yeah, PNG’s compression algorithm has various compression settings, but on the highest compression setting it should almost always beat LZW (GIF’s compression algortithm) when using the same bit depth (the only difference between PNG compression settings is the compression time vs. end size trade-off, with the highest compression being the slowest). Most PNGs are 24-bit (8 bits per channel RGB), but you can make them just about any standard bit-depth. You’ll probably never see a 32-bit CMYK PNG with an 8-bit alpha channel, but PNG is flexible enough for that to be possible.

I highly recommend using PNG instead of GIF for lossless images, with the exception of animation. (I’m not a fan of most animated GIFs, but, meh.)

There is very little support for MNG (Multiple-image Network Graphics) and APNG (Animated Portable Network Graphics), the two animated PNG formats, in most browsers, so I can’t recommend them currently. If you had to pick one of those two, APNG has slightly wider support than MNG. And yeah, they use the same compression as PNG.

In short, the rule is that JPEG should be used for photographs or other complicated images, but for simple images or images with sharp edges, such as most ink drawings or text, PNG should be used. If you *must* use animation, then either GIF or Flash.

I’m never using APNG. The idea is clever, but not smart. Acting like you’re a standard PNG file and only getting caught if the handler knows about APNGs? Sounds good, but isn’t. Especially since knowing what I’m getting would be nice and knowing that something is supposed to be animated and that if it isn’t I know something is wrong.

To those who don’t know: APNG files are in the beginning identical to PNG files. Once there’s a call for an end to the PNG file there’s data for the APNG. Basically if you’re opening an APNG file with a software that can only show PNG files it will show the first frame (don’t know if you can show a frame that doesn’t exist in the animation itself) as if it was a PNG.

The idea was that it could be used on websites even with little support, because anything that had an animation would still show up as normal PNG files in a browser that didn’t support it. Therefore any interface was usable with them anyway.

Problem: If something can be conveyed in a still picture online, use a still picture. It loads faster and that’s pretty important if you want people to spend time in your website.

Also animation like these are rarely better in an interface, and if you’re using for something else, like entertainment, confusing the viewer by acting like they’re having no problems when they’re not in fact seeing what you want them to see is not necessarily a good idea.

Yes, APNG isn’t a good way to go about it, since it’s basically just a PNG with custom chunks added to the file, however it is more widely supported. My browser, Opera, for example, actually natively supports APNGs, however it doesn’t support MNGs without a plugin. According to Wikipedia, Konqueror is the only browser with full native support for MNGs, while both Firefox and Opera have full native support for APNGs.

I really wish MNG support was more widespread, because I’d really like to see GIFs done away with entirely. However, until there’s native IE support, it will never happen.

The reason why Firefox doesn’t have MNG support is really stupid, by the way. Back when they were trying to make Mozilla lighter they decided to drop MNG because “it wasn’t widely used”. People were offering to help with its maintenance, but were ignored.

Then years later they went “we’d like to get someone to manage MNG support for Firefox. Any takers?” Yeah, I’m certain everyone will jump at the change of being the first one who’s work is negated the next time those guys decide that Firefox is too heavy.

40 bits, really? I always thought the 32 bits were 3×8 for the RGB channels plus 8 bit for alpha.
In PNGs, you can drop the alpha channel if not needed, which saves space, and they can also use a 256-colour palette (or just 2 colours).
Other than compressing them along a line, I think there’s also a diagonal or vertical option, but I’m not really sure about that any more … ah, what am I talking, see for yourselves:http://en.wikipedia.org/wiki/Portable_Network_Graphics

About the “40-bit PNGs”, yeah, that’s what I was saying above. It’s 8 bits per channel for the four “color” channels (CMYK = Cyan, Magenta, Yellow, and Black; a colorspace normally used for printing) plus 8 more bits for the alpha (transparency) channel. And yes, the alpha channel is completely optional. I just mentioned this to show how flexible PNGs are (compared to GIF, which only supports up to 8-bit color images with a single transparency color), I wasn’t trying to imply that this is how most PNGs are normally set up. As I said above, most PNGs are 24-bit RGB instead.

The other compression method you’re talking about is the Adam7 compression, which is for creating interlaced PNGs. Interlaced PNGs are primarily useful for progressively displaying an image as it is downloaded on a slow connection. In other words, you can see the whole image quite quickly as it downloads, and the image gets sharper and sharper as the image proceeds to completion. With high speed Internet being widespread now, there is little need for interlaced PNGs anymore. JPEG also supports interlaced images for the same purpose, and for the same reason as above you’ll rarely see them either. (I almost mentioned Adam7 earlier, but decided not to because it’s pretty much irrelevant nowadays.)

This is why I always want to shoot the idiots who say “what’s the point of compression, what’s the point of saving data? We have unlimited bandwidth and storage space now.” IDIOTS!

Rough calculations using the numbers Shamus used in the blog post, a 50GB Blu-Ray could hold barely 2 hours of VHS-quality video. Kick it up to DVD, and you have enough space for one and a half episodes of your favorite cartoon.(32 minuts, 21 seconds)

And that’s without audio.

I’m also sick of having to download 10GB of content to play my Steam games. I spent more time installing Batman: Arkham Asylum than playing it. I’ve almost spent more time installing Portal 2 than playing it, and I played the single player story on that one THREE TIMES, and the multiplayer once.

I have for years cursed at people who have said stuff like “it’s only 1GB”. Even now that I have a faster internet connection, it’s not exactly instant and “only 1GB” stuff tends to stack on. Especially if it’s an update for an MMO. One update per month, 12 GB at the end of the year. Yeah, real “pocket change”.

People who say “what’s the point of compression” either don’t understand that the data they’re using is already compressed or want a simple solution to compression problems. That is “there’s no need for compression because couple of years from now we’ll run into magical fairies that will make the problem go away”.

In either case, like you say, they don’t understand how much space non-compressed data takes and that simply making everything “uncompressed” doesn’t mean that the quality of the file is higher.

But I’ve had arguments with a friend about “everything going wireless” and he didn’t listen to a word I said about lag, inefficiency and unreliability, so I don’t hold my breath that these people will understand either.

The first time I ever read a description of video compression was in “The Moon is a Harsh Mistress”, which was published in 1965. I’ve never been able to determine whether Heinlein was describing something totally new, or quoting existing theory.

[…] convey the sense of beauty I’m talking about. I’ll leave you with something I caught on Shamus Young’s blog a few months back that plays with a similar concept, but in the realm of digital video rather than […]

I do agree with all the ideas you have offered to your post.
They are really convincing and can definitely work.
Nonetheless, the posts are very short for beginners.
Could you please extend them a little from subsequent time?
Thank you for the post.

Ear deformities, especially in children can become a target for the less caring children resulting in the child being
ridiculed and teased. Redheads and folks with course hair will typically be higher shedders
than blondes or those with fine hair. Such devices as
KTP, ERBIUM, Pulse DYE and Alexandrite have short
length of laser light waves, and they are very effective for removing
unwanted hair on a light colored skin.

One Trackback

[…] convey the sense of beauty I’m talking about. I’ll leave you with something I caught on Shamus Young’s blog a few months back that plays with a similar concept, but in the realm of digital video rather than […]