Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

An anonymous reader writes "Linux kernel developers are currently evaluating the possibility of using QR codes to display kernel oops/panic messages. Right now a lot of text is dumped to the screen when a kernel oops occurs, most of which isn't easily archivable by normal Linux end-users. With QR codes as Linux oops messages, a smart-phone could capture the display and either report the error string or redirect them to an error page on Kernel.org. The idea of using QR codes within the Linux kernel is still being discussed by upstream developers."

Very unlikely.. the information in a QR code is probably just enough to say "I run kernel X (build Y) and it crashed with error code Z at instruction 12345 in module 123", if it was a kernel dump that's different but I have seen these without the QR codes and there's nothing sensitive there.

It might actually be more than that. Worst case, the screen in in 80x25 text mode (assuming a PC), which gives 2,000 binary bits, but if you start playing around with extended ASCII graphics characters you could probably encode a KB of data quite easily. Hardly a crash dump, but easily enough to get across the essentials.

You just have to reprogram the VGA font table with 2 wide by 4 high bitmaps (because you can fit 256 such glyphs into the standard vga font table), and you now have 16000 pixels to work with instead of 2000, a bitmap display with a 160x100 pixel resolution; VGA text mode is 640x400 pixels, and each virtual pixel is 4x4 screen pixels, the standard VGA font is 8x16 pixels.

BTW, you can't encode 2000 bits into a QR code with a 2000 bit bitmap, as it has parity and spatial clock recovery built in to the code.

``Hardly a crash dump, but easily enough to get across the essentials.''

Here's a crazy idea: instead of working on displaying cutesy graphics images that need to be decoded using a smart phone and a web site, what about actually generating a freakin' crash dump? Is there a technical reason that Linux is unable to do this? If crash dumps are really not possible, how about a plain 'ol text file in the root directory containing the reason for the crash/panic?

Just from my own experience the only kernel panic I've ever encountered was due to a failed SATA controller. But conceivably bugs in new beta file systems (you jumped on the brtfs bandwagon yet?), hardddisk failure, SATA failure, or failure in the PCI controller.

Mind you that's not a reason not to do it, just that there are times when it's not useful. The same machine with the dead SATA controller also had Windows on it which managed to bluescreen without creating a crash dump.

I have a couple of bad micro SD cards. Put one into a card reader on a Linux machine, try to read some of the files, and presto magnifico, kernel panic every time. Windows handles this corruption a lot more elegantly, incidentally, but the hardware is toast either way.

Kernel crashes occur when the kernel enters an inconsistent or invalid state from which it cannot recover.

When a user program fails, the kernel maintains consistency, can cleanly terminate the process, and can accurately report the cause of the failure if need be (illegal instruction, deadlock, access violation, etc...).

When a kernel fails the very systems that it relies on to report failures may very well be compromised by whatever caused the kernel to fail in the first place. As such, any kernel fault reporting needs to be incredibly robust and as independent of other kernel mechanisms as possible. Dumping text to a serial terminal is the preferred method because it's incredibly simple and relies on nothing else, meaning that barring a failure of the system memory it should always act as a reliable fallback.

Dumping kernel memory to a disk might fail if the state of the file system is compromised, if the storage controller is compromised, or if any number of intermediary systems are compromised by the inconsistent state of the kernel. Many operating systems do attempt to dump crash memory to the swap file / swap partition as this is less likely to cause data corruption than writing to a particular file in the file system.

You usually don't want to write to the filesystem in the event of a kernel panic. It could make things worse and corrupt it. Once you kernel panic you are basically screwed and can't rely on any services beyond really low level BIOS stuff to work. Poking some text to the screen buffer is about it.

Windows does core dumps using a specially reserved area of the boot drive and using low level boot driver calls. It can still fail but at least has a fairly low probability of damaging the filesystem further. I suppose Linux could maybe dump to the swap partition or something.

As AmiMoJo also noted, when you have a kernel panic all bets are off regarding which parts of the kernel are OK. If the behaviour of the disk driver or filesystem have been affected, it could damage your filesystem to try to write a kernel dump into a normal disk partition. It might work but it does seem a good idea to be properly paranoid. I didn't know that Windows uses a special reserved area of the boot drive - that does make sense as a solution!

There have been various systems for crash dumping under Linux, though. I think the de-facto solution (the one that was accepted by the kernel devs) ended up being kdump, which is based on kexec (kexec is "boot directly to a new kernel from an old kernel, without a reboot"). This allows full crash dumps with (hopefully) decent safety, so it is possible to do this if configured.

In kdump, you have a "spare" kernel loaded in some reserved memory and waiting to execute. When the primary kernel panics it will (if possible) begin executing the dump kernel, which is (hopefully) able to reinitialise the hardware and filesystem drivers, then write out the rest of memory to disk. I'm not sure how protected kdump's kernel is from whatever trashed the "main" kernel but there are things that would help - for instance, if they map its memory read only (or even keep it unmapped) so that somebody's buffer overflow can't just scribble on it during the crash.

Obviously, having a full kernel available to do the crashdump makes it easier to do other clever tricks, in principle - such as writing the dump out to a server on the network. That's not new, in that there used to be a kernel patch allowing a panicked kernel directly to write out a dump to network, it just seems easier to do it the kdump way, with a whole fresh kernel. Having a fully-working kernel, rather than one which is trying to restrict its behaviour, means you can rely on more kernel services - and probably just write your dumper code as a userspace program! Having just installed system-config-kdump on Fedora 20, I see that there's an option to dump to NFS, or to an SSH-able server - the latter would never be sanely doable from within the kernel but pretty easy from userspace.

Various distros do support kdump. I think it's often not enabled by default and does require a (comparatively small) amount of reserved RAM. So that's some motivation for basic QR code tracebacks. I suppose another reason is if they expect they can mostly decipher what happened from a traceback, without the dump being necessary - plus, with a bug report you can easily C&P a traceback.

This discussion has just inspired me to install the tools, so maybe I'll find out what it's like...

If crash dumps are really not possible, how about a plain 'ol text file in the root directory containing the reason for the crash/panic?

It is easily possible to do what you want. Trivially so... so why isn't it done? There is an answer to that question my friend and the answer is this:

A kernel will panic when it detects that the environment it is running in is not the same environment it thinks it is running in. What this means is that the kernel can no longer be certain of anything, up to and including whether or not it can write coherently to a file system. Rather than potentially trashing your file system, the kernel just prints to the s

We are encoding the full Oops, i.e. from the "cut here" to
the "end trace" marker. Classic won't ever go away, and we had already created a configuration option called CONFIG_QR_OOPS that can disable this at all.
In case your distro or you had compiled it in and you don't want to have QR codes on your screen, I just added a new kernel parameter currently called 'qr_oops', which can as well disable it.

Anything that couldn't is probably FAR TOO LONG to even start taking down any other way and almost certainly will cut through the screen buffer limit anyway (every kernel panic I've had - which is about a dozen I think - was like that).

Let's compare and contrast to, say, Windows. Bluescreen with minidump and error code that has 7 million potential causes.

At least with a QR code, for those totally undumpable errors, you stand half a chance of snapping it and providing several kiloybytes of useful information for someone to work from - that they know hasn't been transcribed wrongly. And can be taken from even a completely hung machine.

It's a good idea. Someone needs to make a patch for it. The biggest problem - as always - will be making sure you can get to the point that you can write to the video memory and do so with enough processing / storage to be able to write something useful into the QR code.

But that can be compressed as it doesn't NEED to be human-readable any more. So you can easily fit in a few Kb of useful data, I should think.

And as data density rises, so does the error correction but if the QR code reads (you have a device that reads them directly, why bother to snap a shot then process the image separately?) then it was a success. Hover and hold until you get the beep, on almost any smartphone made this decade.

QR codes use Reed-Solomon error correction, so you don't get missing or corrupt data (in that the QR reader knows if it reconstructed all the data correctly or not). Readers will typically only "read" the code if they manage to reconstruct the entire thing. The error correction helps compensate for poor image quality, and the fact that the image is monochrome makes things like exposure less critical. There are four levels of error correction, which allow for the reconstruction of 7%, 15%,25%, or 30% of codewords respectively.

QR codes can store up to a bit under 3KB of data (the largest size with the lowest error correction), but I couldn't get my phone to read any v40 QR codes (the largest ones), and v25 took some effort. The plan for QR codes of kernel oopses will probably fail for that reason, if nothing else (that they need v40 codes to store an entire oops, and few phones will read v40 codes).

I'm sure the folks discussing this are smarter than I am, but this is Slashdot, so I'll do some uninformed speculation anyway. Because it's fun.

The sample oops is 3134 bytes in plain ASCII. Plain old zip gives 1589 bytes, xz does a little better with 1492 (and only 1446 with lzma). I believe xz could do even better if the dictionary could be fixed and thus not embedded in the file. Doing a base64 encode on that gets to 1929 bytes.

So it looks to me (based on my sample of one...) like you could use version 27

This produces output too large to fit on most screens unless your console font is ridiculously small or your screen resolution is higher than most (my notebook has a 15" 1680x1050 panel). Also, it appears the console (on Ubuntu, anyway) doesn't handle UTF-8. Falling back to ANSI, it's even farther from being feasible.

Yes this causes a big output since my (and I suspect yours as well) terminal uses size 12 fonts, but in the crash situation we do it one screen pixel per QR pixel. There is a file called qr_code.png in that folder I gave in the thread, which is the actual result unscaled. It's 147x147 which can fit on every screen I know of.
How to handle textmode is still an ongoing question. We'll first get it working on the framebuffer then maybe we'll find a solution for textmode, if it's even possible.

What is the size limit of a QRcode? Doesn't increasing the density of information contained in the image lead to the possibility of corrupt/missing data due to poor camera quality, motion blur, over/under exposure, or something along those lines?

They contain quite a bit of error correction so either it won't decode or it will with >99.99% probability decode correctly. You can store almost 3kB in a 177x177 QR code, but for the same physical area it's a trade-off between information density and readability. The bigger you make the QR "pixels" the less camera resolution you need, but the less information you can store as well.

I worry about the long-term applicability of QR codes. In 10 years, are they still going to be convenient to read, or is some guy going to have to dig out an old smartphone to read the error output from his decade old system?

The QR code has really nothing to do with smartphones. You could take a picture of it with any camera and upload it. QR Codes are just a software library that evaluates based on white and black marks within the picture.

I strongly support backwards compatibility, but chances are you won't have anyone to send it to, if you're running a 10 year old kernel. No one will want to debug that, except maybe yourself, but then you'll probably have all the stuff you need to read it (and might even have the text output piping to somewhere else, too).

1) No, 2953 bytes is not enough for a "kernel dump". "Kernel dump" as a term/phrase doesn't even make any sense, come to think of it. Did you mean a stack trace? Register dump? Because "kernel dump" makes me think of "memory dump", i.e. dumping all contents of RAM to swap + rebooting system (which later notices the crash dump header in swap and hopefully extracts it).

2) If just a stack trace or register dump: 40-L may be too high a resolution to reliably work when using a mobile phone camera to take a p

I have a better idea: how about just keeping things how they are. People using mobile phones to take a photo of a stack trace + register dump mostly works reliably (barring wobbly hands).

^^ This.

Add a bit of OCR software and you have a system that can both be read by humans without the aid of special software and by computers to produce textual output with a bit of special software (you need a bit of special software anyway for QR codes, so you don't lose anything).

Anything's an improvement over:"My computer froze.""What happened?""It put some message on the screen.""What did it say?""Something about an error.""What error?""I dunno. It had some numbers and letters and stuff."

"My computer froze.""What happened?""It put some white and black crap on the screen.""What did it say?""How the fuck should I know? It was random white and black dots! Like a fucking Rorschach test!""It probably was a kernel panic. What was the error?""I dunno, because like I said, ALL IT HAD WAS SOME DOTS AND SHIT. Then it rebooted! So it's gone! FUCK!"

How is that an improvement? Yes it's a change, but it's not an improvement.

I think most people with smartphones recognize QR codes now, so there is at least a chance they will be able to take an image or use a decoder app on it. Well, next time anyway. And hopefully it will have enough info for you to know what happened.

Really? You think your end user who hasn't got the brains to take a screenshot of human readable text and send it to you and who probably has never even heard of QR codes is going to have the presence of mind and technical knowledge and ability to take a picture of the code and send it to you?

That has to be one of the dumbest things I've heard on slashdot...and that's REALLY saying something.

It's even more worrying that the Linux Kernel devs are giving this idea the time of day.

I think having the option to scan the QR code with a simple message to do so is one more way to get the info needed.Aiming a smartphone at the screen is easier than framing a screen with your phone's camera and hoping for a solid shot without a flash before it does something even stranger.

They're used on beer ads, chain pizza ads, breakfast cereal and at Disney parks.So yes, I think the average end user has a shot at this.

Anything's an improvement over:"My computer froze.""What happened?""It put some message on the screen.""What did it say?""Something about an error.""What error?""I dunno. It had some numbers and letters and stuff."

"Show me!""I already rebooted it."

Personally I would rather have a more sophisticated crash dump system, like other OSs, because whatever is going to fit in a QR code isn't going to help much unless you're looking up known issues in an enterprise Linux vendor's bug database. That's assuming they can cram a stack trace into QR codes, AAAAND you have a problem that leaves a predictable stack trace.

I don't remember the last time I had a Solaris system crash that didn't leave a dump (try not to giggle). It wo

This is why my alternate OS eschews absolute minimalism and includes mandatory "userspace" features in its design, so it can rely on them being present. I handle the whole (multi) boot process within the OS, so I can launch other OSs from within a running instance. Boot process integration was necessary for firmware segmented loading (optionally put part of the OS in firmware, see: Coreboot). Since the OS handles boot itself it can avoid immediately crapping all over memory at boot and instead upon soft-b

I think it's a great idea to make error reporting easier. I recently experienced an oops but didn't report it because there was no immediate way to do it. However relying on a framebuffer being present is a mistake, in my case it was on an embedded headless system, and framebuffers generally are available only on desktops which are far from being the majority of Linux usage.

Linux must be ubuntufied. We need to hide everything because it's way to complicated for the common user or his dog. We need more splash-screens to hide all the stuff that makes no sense anyway. Who want's to know if a module didn't get loaded? As a matter of fact, we should remove unnecessary logs (like message, dmesg, audit), because nobody gives a rats ass. Also: Why have a console? Or init-mode 3 ? People want the graphical stuff, let's get rid of all the ballast like command-line. Those few people still using ancient tools like 'make', 'vi' or (o my god) 'ifconfig' should go and find themselves something else to brag with. Linux MUST go mainstream.

That is somewhat a problem, I agree. Please make this QR code display friendly, such as "Your operating system kernel has crashed. For advanced users, a QR code containing additional information is provided below. By taking a photograph of it, you can help the developers to solve the problem. [QR code picture] Press any key to restart." Maybe include the Tux logo there to show that this is about Linux.

Wayland should improve the situation too, as we can settle with a proper graphics more earlier at the boot.

(Not true as I haven't communicated that idea to anyone - and an idea can't be stolen anyway).

There are a number of advantages of doing this, non-technical people are likely to be familiar with QR-codes, most people have access to digital cameras and resources to convert the QR code to a link and using this works as long as the screen can be written to. Storage system failure or network system failures wouldn't be a hinder to provide a thorough failure analysis.

It says, ``... most of which isn't easily archivable by normal Linux end-users. Abnormal Linux end-users easily archive the text. If you have to use QR codes... maybe you aren't the right kind of Linux end-user. Just saying.

Good idea, but I hope they keep all existing systems in place, and make it optional. Graphics drivers are massively complex, and are probably a significant source of oops. If displaying a QR code means that the kernel needs to interact more with the drivers, and (oh god i hope not) change the resolution to display a QR code, then I expect more fail. People can take photos of the crash messages in 80x25 character consoles anyway, so let's not destroy that.

Or just display a short number code. Displaying a QR code won't solve anything, it will just obfuscate the error and leave the user without any easily memorable reference. This sounds more to me like "let's do it because it's modern and hip" rather than it being actually useful.

Or just display a short number code. Displaying a QR code won't solve anything, it will just obfuscate the error and leave the user without any easily memorable reference. This sounds more to me like "let's do it because it's modern and hip" rather than it being actually useful.

The QR code can not only indicate the exact location of the error, but can take you to a website on the phone, with a url long enough to log many key points about the error.

Even if it logs very little, developers will get more input this way than they do now, because when your machine is crashed, you can't report anything and once it reboots, you have other priorities than digging in the last crash dump.

However, other than collecting statistics, it might not do any good. Even when you do submit a du

Is the point to give the users something useful? I assumed it was to give the devs something to debug. Asking the user to transcribe a kernel stack trace is not likely to happen, but a QR code that encodes a bit of text saying 'please email to crash@linux.org' followed by the stack trace would mean that you'd actually get something vaguely useful in some cases, rather than the user just shrugging and power cycling the box.

Can kernel panic info also be saved to a disk log instead of just showing it on the screen? It could also be logged to a USB drive or something connected to the network or in the extreme case, CMOS space in BIOS.

Whatever the method, the log information should be easily accessible on reboot without the use of cameras/cell phones etc.

There isn't, and that is quite common actually. However the QR code could encode more information and, with some nifty algorithms, can be automatically interpreted from a photograph to kernel crash information files.

That would be so tricky that it's probably not feasible for the attacker. Basically you would have to both have a vulnerability in the QR code processing code in the phone and, a compromised Linux kernel in your PC which injects malicious data into the dump.

Now they want to bury the kernel error messages in a QR code? That REALLY takes the cake.

This is not burying or hiding. It's there in full view if you have a decoder, and even better, you can send it to a mailing list without having to transcribe the sodding thing by hand. Bonus: no transcription errors!

Seems like a win to me especially as no one is ever planning on making it mandatory. It's a fantastic option if you don't have a convenient way to log a serial console.