Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

An anonymous reader writes "This year's
Usenix security symposium
includes a
paper
that implements a "cheat" utility, which allows any non-privileged user to
run his/her program, e.g., like so 'cheat 99% program'
thereby insuring that the programs would get 99% of the CPU
cycles, regardless of the presence of any other applications in the
system, and in some cases (like Linux), in a way that keeps the program
invisible from CPU monitoring tools (like 'top'). The utility exclusively
uses standard interfaces and can be trivially implemented by any
beginner non-privileged programmer. Recent efforts to improve the
support for multimedia applications make systems more susceptible to
the attack.
All prevalent operating systems but Mac OS X are vulnerable, though by
this kerneltrap story,
it appears that the new CFS Linux scheduler attempts to address the
problem that were raised by the paper."

it works by avoiding running during the exact moment of a clock tick (which would be the moment when CPU usage per-process is checked). to start running immediately after a clock tick is (apparently) easy, but to stop before the next tick is harder. the paper suggests using some kind of get_cycles assembly instruction to count how many CPU cycles there are per clock tick, and use that number to gauge when the next clock tick is going to occur by counting how many CPU cycles have elapsed.

This is a bit different. It's a way to convince the OS to give you more time slices than you'd normally be allocated. e.g. If you ran that program of yours twice at the same priority level, both instances should get ~50% of the CPU time. If one of the instances implemented this privilege boosting scheme however, it would get to hog all the CPU time while your other spinlocked program starved.

Yeah, it surprised me too when I ran it for the first time. it is easy enough to fix (can't remember right now, but i recall something about a "limits" file) but should hint that that default is set too high.

Besides the syntax comment the other poster said, it could've also been that the school implemented per-user process limits on the machine. Linux has had this capability for years and years; most people just don't bother setting it, but universities hosting machines for programming students pretty much have to set it for exactly this sort of thing, whether it be accidental or malicious.

This would render the machine unusable.. until about a year back, when something changed - which leads me to suspect that the kernel has had protection against this sort of thing for a while now.

I guess they just put on a nproc limit on each user. It's just a trivial security measure against simple fork bombs. Assuming your Linux system uses PAM (most modern distros do), take a look at/etc/security/limits.conf.

No, it isn't. It is one of the most misused uses of the apostrophe, but it is not in any way correct English. The plural of an acronym is formed by adding a lowercase "s". Don't believe me? Look it up.

After two years of undergraduate linguistics, though it was many years ago, I'd like to point out that if it is one of the most misused it is probably now correct (not that correct means much when talking about a natural language).

"Correct" makes sense when talking about latin because it is not a natural language - latin is a logic game; english, however, is used to satisfy the fashions and fads of any given week and changes at the demand of its users.

If you check the linux kernel mailing list for Vassili Karpov, you should find test cases that demonstrate this behavior and tools for monitoring actual CPU usage for a variety of platforms, though I notice no mention of any of that in the paper.

Publishing papers takes a lot of time, as anybody who ever done it would know... For example, the post you mention is from Feb 2007. By then, according to the usenix-security call for papers, the paper has already been submitted. Also, google-ing "cheat" around revealed this technical report: http://leibniz.cs.huji.ac.il/anon?View=1&num=1&pid %5B1%5D=870&abstract=1 [huji.ac.il] (seems the initial version of the paper) which is dated May 2006.

Now I have no excuse to avoid working on the dbase (Access and VBA, ugh)

I was an "Access Developer" for a while, even a consultant doing the same (yes, I would sell my soul for a dollar)
Now I have a *Real* job as part of a programming team working with a *mostly* real RDBMS.

you will be in my prayers, brother-in-arms.

It's worse than that, I'm a software development major in college, working as an intern consultant. I don't even know VB, but using Real Languages gives me enough to fly by the seat of my pants. The dbase is done by some guy who can no longer be found, writes spaghetti code and has a fondness for loops while doing lookups. It's been 'upgraded' from Access '95, to '97, to 2K, yet I'm not allowed to just drop the thing into SQL Server and use.NET to put a new front end on it. It is the bane of my existe

I seem to recall usenet discussions about this circa the time of !uucp!newsglop!.....
It seemed the Unix scheduler would let certain IO operations hog the CPU. And if you somehow installed your app as a IO driver or IO completion routine, then your app could hog the CPU.
Similarly since day one of Windows soundcards you could set your app to realtime_priority and everything else would suffer.
Not exactly smokin' hot off the press.

That's not what the paper talks about. The vulnerability is that the scheduler gathers statistics (used to make scheduling decisions) by checking who is running at every clock tick. By running only between clock ticks and never running at the time of a clock tick, your process can use a lot of CPU without the scheduler knowing.

***I seem to recall usenet discussions about this circa the time of !uucp!newsglop!..... Not exactly smokin' hot off the press.***

Not exactly. This is a technique that will, in prinicple, work with any scheduler that prioritizes tasks on the basis of time ticks previously used by the task. That turns out to be most of them. The technique does not require being an I/O driver, other special task, or having unusual user priviliges.

This year's Usenix security symposium includes a paper that implements a "cheat" utility, which allows any non-privileged user to run his/her program, e.g., like so 'cheat 99% program' thereby insuring that the programs would get 99% of the CPU cycles, regardless of the presence of any other applications in the system, and in some cases (like Linux), in a way that keeps the program invisible from CPU monitoring tools (like 'top').

Next up, a virus which senses bad grammar and punishes you by using 99% of

I've even gone as far as to compiling a minimal Linux distribution for one of my test machines so my CPU intensive application can squeek out every last drop of performance as possible. Beyond the normal renice -20

It works by sleeping at the right point in time. You really hack up the timeslices and decrease the overall efficiency (more context switches), so it's only good if you want to steal cycles where you are not really allowed to.

In my days (yes, I'm an old fart) - the schedulers had basic principles :

- Voluntary yielding led you to get accounted for the time you spent running.- You could stay in the interactive queue for only a certain amount of time. After some amount of time had passed (a few secs) you were either bumped to non-interactive if you were running (with longer time slices but lower priority) or removed off the scheduler list for good (if the time spent there was idling). They had a special 'idle but interactive' (not eligible for dispatching) queue for that.- Scheduling a new task restarted a new time slice

That particular scheduler even had a 3 queue system so that if you got accidentally bumped into the non-interactive queue or if your process was semi-interactive you had a better chance of gaining interactive status again. And they had a 'really' not interactive queue for those CPU hogging processes.

Of course this requires the hardware to have a precise timing feature (something with a granularity that is finer than the process interleaving time slice time and ideally in the magnitude of instruction execution). And this scheduler wasn't using time sampling and time quantums.. (but something more like the OSX timer on demand paradigm).

The cheat program hogs the CPU by using it when the host OS isn't looking. As a result, it avoids the scrutiny of the OS's scheduler and is actually given a priority boost by some schedulers because of its good behavior.

This is accomplished by sleeping for a fixed amount in between OS clock ticks. The timeline looks like this:

Hardware is set to generate a "tick" event every N milliseconds.

Tick event occurs, which is handled by the OS.

OS notes which process is current running on the CPU and bills it for this tick.

OS wakes up cheating process, which is currently sleeping, and allows it to run.

Cheating process runs for M (< N) milliseconds, then requests to go to sleep for 0 milliseconds. This causes the cheating process to sleep until just after the next tick.

Red Light Green Light.Your goal is to run as quickly as you can towards me. When I turn and face you and say "Red Light" you must stop moving and if I catch you moving I make you start over from the beginning. When I'm not looking and say "Green Light" you can move again.

In this case, the goal is to cover the greatest total distance instead of just reaching my position; so we could adapt it from running to eating: The "winner" is the one who eats the most and the losers end up hungry.

(reply to self after RTFA)What 'saved' the Mac OS was its different use of timing triggers. "All" other OS'es use one common steadily ticking clock as a dealer of time slots. This allows the cheat to "skip to the start of the line (queue)" every time it's had its turn.

OTOH, the Mac uses a stack of alarms set to specific points in the future, and polled in order as they occur. So the difference on Mac OS is that there's no skipping the queue, it's rather "there is no queue, we'll call you when it's your turn

> I know, [Mac OS X is not BSD] but in broad terms (especially in peoples' minds) it's still seems to be "the closest"

Yeah but neither Mac OS X nor FreeBSD runs in the mind at this time. The BSD subsystem in Mac OS X is an optional install, a Unix compatibility layer. The kernel is called xnu and although it is descended from Mach it is also descended from Mac OS and NeXT and also it is not a microkernel.

OS X has a hybrid kernel, XNU, a major part of it based on Mach, as well as incorporating sizable bits of FreeBSD, plus Apple bits that are not related to either one. It's not Mach, and it's not FreeBSD.

According to the paper, the reason Mac OS X is not vulnerable is that it uses one-shot timers scheduled for exactly when the next event needs to occur, rather than periodic "ticks" with a fixed interval between them. The "tickless idle" feature introduced in Linux 2.6.21 (currently only on x86, I believe) takes the same approach, and very possibly makes Linux immune too.

(Ironically, immediately after discussing OSX's ticklessness, the paper mentions that "the Linux 2.6.16 kernel source tree contains 8,997 occurrences of the tick frequency HZ macro, spanning 3,199 files", to illustrate how difficult it is to take a tick-based kernel and make it tickless. But those kernel hackers went and did it anyway.)

The tickless feature isn't yet implemented on all architectures that Linux supports, though. I think AMD64 support for it is supposed to come in 2.6.23, along with the new CFS scheduler.

They took too long to publish this. Linux 2.6.21 (released in April) added support for using one-shot timers instead of a periodic tick, so it avoids the problem like OS X does. In addition to resolving this issue, tickless is important for saving power (because the processor can stay in a low-power state for long enough to get substantial benefits compared to the power cost of starting and stopping) and for virtual hosting (where the combined load of the guest OS scheduler ticks is significant on a system with a large number of idle guests). As a side effect, while the accounting didn't change at that point, the pattern a task has to use to fool the accounting became impossible to guess.

The CFS additionally removes the interactivity boost in favor of giving interactive tasks no extra time but rather just quick access to their available time, which is what they really benefit from.

The crux of the problem is that the OS uses statistical sampling to account for CPU usage by user processes. Since the sampling occurs at regular intervals, it can be avoided by a cheating program. I can see two possible defenses against this:

Modify the sampling mechanism so that it occurs at irregular intervals. This makes it difficult (but probably not impossible) for the cheater to avoid the sampler. (Apparently, the Mac OS uses this technique, although not for security reasons.)

The second one is obviously the better one. I think this is basically what the CFS does. (the following is my understanding. It may be wrong) For processes it figures out what amount of time each process should have (based on the the number of processes. It tracks how much time each process is owed (in the case of 5 processes each deserves 1/5 of the total processor time). It subtracts the time used on each scheduler event (clock tick or voluntary yield.) Each clock tick the scheduler transfers control to t

The paper is quite long, so here's a summary (take this with a grain of salt, who wants accurate information should still RTFP):

Most OSes (Linux, Solaris, Windows but not Mac OS X) are tick-based. This means that the kernel is called from hardware periodically (this is the "HZ" value you set in the Linux kernel). Some of them (Linux) simply check which process is running at each tick and compute statistics based on that ("sample-based statistics"). This means that the process running when the tick happens is billed for the entire period of the tick.

Since ticks are typically "long" (typically 1-10 ms on Linux) more than one process may run during this period. In other words, using this approach leads to inaccuracies in the process billing. If all programs "play by the rules" this works quite well on average though.

Next thing: the classic schedulers typically maintain some sort of "priority" value for each process, which decreases whenever the process is running and increases when it's not. This means that a process runs for some time, its priority decreases, and then another process (which hasn't been running for some time) takes over.

You can exploit that by always sleeping when a tick happens and running only in-between ticks. This makes the kernel thinks that your process is never running and give it a high priority. So, when your process wakes up just after a tick happened, it will have a higher priority than most other processes and be given the CPU. If it goes to sleep again just before the next tick, its priority will not be decreased. Your process will (almost) always run when it wants to and the kernel will think that it's (almost) never running and keep its priority high. You win!

Another aspect is that modern kernels (at least Linux and Windows) distinguish between "interactive" (e.g. media players) and "non-interactive" processes. They do so by looking how many times a process goes to sleep voluntarily. An interactive program (such as a media player) will have many voluntary sleeps (e.g. inbetween displaying frames) while a non-interactive program (e.g. a compiler or some number crunching program) will likely never go to sleep voluntarily. The scheduler gives the interactive programs an additional priority boost.

Since the cheating programs go to sleep very often (at every tick) the kernel thinks they're "very interactive", which makes the situation worse.

Some of the analyzed OSes - even if tick-based - do not use sample-based statistics in the kernel but they do use sample-based statistics for scheduling decisions. So the kernel sees that a process is taking more CPU than it should but it will still keep on scheduling it.

Mac OS X is not affected because it has a tickless kernel (e.g. without periodic interrupts). Because of that sample-based statistics don't work and it has to use accurate statistics, which make it unaffected by the bug.

This bug can be exploited to (at least)

- get more CPU than you're supposed to- hinder other programs in their normal work- hide malicious programs (such as rootkits) which do work in the background

Here's a list with the OSes (this USED TO BE a nicely formatted table, but the darned Slashdot "lameness filter" forced me to remove much of the nice lines and the "ecode" tag collapses whitespace).

I guess that Mac OS X doesn't need a interactive/non-interactive distinction because of its different (tickless) approach. I assume that interactive applications can (implicitly or explicitly) can be recognized as such in a different way. Does anyone have more information on that?

Yield()ing just before timer tick is a neat trick to grab cycles, but what use are cycles? This might have been interesting on time-share machines 20 years ago. But now cycles are in gross surplus on most machines. And processes carefully controlled on loaded machines. Until this piggy can be remotely deployed, it isn't much of a hazard.

A very simple patch is to issue RDTSC instructions at process restart and blocking syscall to count the cycles actually used. That way the extensive tick-code doesn't need to be modified.

I certainly wouldn't block RDTSC or interfere with the piggy's use of it. Just use it to watch'em.

As for users on a cluster/shared host, that becomes an administrative issue. A thief can be caught (process watchdog on HD or ethernet interrupts) and booted. Probably a significant punishment to the thief, most of whom don't need cycles. Those who do are tied to software.

Chris Torek gave a presentation at UseNIX about how a constant quantum could result in a process having its CPU usage unaccounted.

His solution was to use a randomized quantum. Not unique per process, but randomized when the kernel starts running each process. That gave you a better accounting of the CPU time (statistics, doncha know:)), but also made this kind of attach much, much harder.

I'm somewhat disappointed that I did not see Chris and Steven's paper referenced in this one. (I believe that the t

I remember seeing this done on the VAX/VMS mainframe back in 1987. In that environment, it simply meant that you kept track of your timeslice and voluntarily gave it up before the scheduler took it away from you. That meant you got put at the top of the run queue, and unless someone else was doing the same thing, you were the next program to run. Voila... 99% CPU for you!

Of course, ordinary users were given a limited amount of CPU time (as well as connect time, disk space, etc), so for the ordinary student, this just meant they used it up in a day or two instead of having a whole month. But then again, for class accounts, they could usually beg for more.

Under unix variants, one could do the same by implementing cpu quotas at the user level. I've seen network packet quotas, and I'm sure someone out there has done cpu quotas along the same lines.

What?! I'm really not sure what's being said here. I understand the idea behind this, but the wording of the Slashdot piece is difficult to penetrate, even by Slashdot standards.

I hard a hard time reading it as well, but then I saw it (kind of like when you suddenly "see" the picture in a stereogram). Proper punctuation, whitespace, formatting, and font changes help a lot. It should have been:

.. allows any non-privileged user to run his/her program, like so:

cheat 99% program

thereby insuring...

where cheat is the name of the compiled utility that lets you "cheat", 99% is an argument to cheat, and program is the name of some other program that you want to run at 99% of the CPU. I.e., the command line syntax resembles renice.

I'm assuming that we're saying that this application can get 99% of the time-slices on an otherwise occupied system, starving other tasks for resources.

We already have that. They're called McAfee Automatic Update and Windows Automatic Update.

God dammit, I hate those things. I turn on my office computer in the morning, and just let it sit for ten minutes because it's otherwise useless. (I turned-off Windows Automatic Update, but McAfee was more than happy to fill its shoes in needless resource hogging.)

I don't know. I think retractions would screw with everything else. If you make a boneheaded statement (and I've done it more than once myself), it should stand. Otherwise, everyone who responds to correct your misstatement will look insane, and it'd be hard to metamod, because the comments wouldn't necessarily fit the context anymore, etc.

Why not leave the post but allow a "retracted" tickbox? Thus at least the owner of the comment can effectively say "I was wrong, boneheaded, whatever" without having to post another comment and wait two minutes to do it? and all that shows up it a one-liner under the comment:This comment has been retracted by its poster

That'd be fine, or even cool. It'd deflect the inevitable storm of 500 people saying, "Wrong n00b!" and not reading down far enough to see that you admitted it already, and let the whole discussion move on to more productive things.

Absolutely. In fact I think it should go half a step further. In the interest of civility, using this feature should hide the message from casual viewing. But a single click will still bring up the original so that you can't use slashdot to be a complete ass and then censor yourself after the damage is done:)

By "retraction", I meant that in the sense that newspapers use the term: the publication of a statement which redacts a previously published statement (e.g. my post in response to my initial post). The fact that Slashdot won't let me post a reply to my own post for a minute means that I sit there hitting "submit" on my one-line, "oops, I meant..." post for a minute. It's just annoying.The ability to edit one's comments would be nice, but I'd only want to see that kind of feature if you could actually review

That isn't likely to happen without a change in attitude due to both starting furthur behind and progressing more slowly. The current malware situation looks like bad SF and a morality tale of what happens when you allow really stupid things to happen (eg. letting arbitrary code embedded in images run - hopefully that person was dismissed from Microsoft).

I am advocating that people should know what their applications will do with different inputs. The above example was a mind bogglingly stupid mistake - viewing an image was enough to spread a virus! The added rant by the poster above blaming poor practice on somebody else and making assumptions about myself (why would I want people to use garbage just because it has a good licence?) I consider a fairly lame apology for somebody else's major mistake that completely ignored any sort of security. It allowed

***that others are starting to look after the *nix world for weaknesses? Once windows is equal or better than *nix in terms of security, then all the security and malware people will start looking at us.***

Of course not. It shows that OS research work is likely to be done on a Unix of some sort where the source code is available for anaylsis

TFA points out that Windows is just as vulnerable to these cheats as BSD, Linux and Solaris. The cheat works by releasing the CPU just before the end of a time tic

If you just want to DoS the box as a local user (which is all this lets you do, from a security standpoint), then there are much easier ways of doing this on OS X via the VM subsystem. So easy that I've managed to do it with my own code a couple of times purely by accident and had to power cycle the box to stop the process (the same code runs fine on FreeBSD, by the way, it just chews up a lot of memory).

I've noticed a similar thing, more that anything that causes heavy disk IO kills the system. More so when I've been experimenting with virtual machine implementations on the platform I've noticed heavy lag issues that I don't see on equivalent systems. Locking up a OSX box isn't too hard.

If you just want to DoS the box as a local user (which is all this lets you do, from a security standpoint), then there are much easier ways of doing this on OS X via the VM subsystem.

You're missing the point here. Because the CPU accounting is off it's possible to do a QoS attack on a box rather than a DoS, that's virtually impossible to detect as the end user. From his or her standpoint, the system will be sluggish, but because of the way the attack works various random processes will seem to be taking up all that extra slack so that most likely no one process will appear to be hogging the CPU.

There's also the possibility when combined with a worm or rootkit, as well as a bot net to

I guess I'm not hip, but what exactly is the difference between a QoS attack and a DoS attack? I mean severly degrading the quality of the service potentially up to the point of denying it *is* a DoS attack.

A DoS attack is an extreme form of QoS. If you perform a QoS attack on someone their performance is reduced, but the system is still usable, where as in a DoS the goal is to make the system totally unusable. In some ways a QoS is even more effective than a DoS because it's more subtle and causes more frustration. If for instance a website gets DoSed the owner is upset and will try to get someone to investigate and shutdown if possible whoever is DoSing them, and the users simply cannot connect to the serv

At least this paper should help dispel that old "Mac OS X is BSD with eye candy" meme. While reading it, it's hard not to realize that XNU (the OS X kernel) and the BSD kernel are completely different beasts. Figure 1 in particular drives the point home: it shows that with respect to the timing model used, you have OS X and RTOSs on one side, and FreeBSD, Linux, Windows etc. on the other.

I'll prove you wrong as soon as that stupid spinning beach ball of death lets me do something.

Why is that? You could kill -9 as many times as you like in a given day... Well unless of course one lawyerd being killed causes more lawyerd instances to spawn and then they all 'sue'. Ok yeah, that could suck.