Posted
by
Hemos
on Monday October 03, 2005 @09:37AM
from the protecting-ourselves dept.

An anonymous reader writes "There's an article by Jason Miller on innovation in Unix that talks about OpenBSD's new heap protection mechanism as a major boon for security. Sounds like OpenBSD is going to be the first to support this new security method."

Yep. After all, the goal of the OpenBSD project is not simply to be the most secure operating system ever. The goal is to provide the best security, along with a number of other goals, such as running Unix software, achieving good performance, and providing a good (according tho the stated goals even "the best") development platform.

Malloc is slow. Per studies, 20-30% of CPU time wasted on memory management.

Interesting paper - thanks for the link.

However I find the conclusions given a bit dubious. For instance the claim that allocations are "free" is somewhat dubious - garbage collecting languages put all of the work on the back-end, giving the illusion of a free front-end (whereas non-GC languages put the hard work on the front-end). Yet for every object you create on the heap that is more work the heap walker has to do each GC to detect orphaned objects - a non-trivial task. It then has to free all of those objects, and because of the "free" allocation it has to move all of the objects and rebase every object pointer in the application. It doesn't take a genius to realize that is a signficiant task in an application of a real-world size.

I think the proof is in the empirical proof - how many high performance memory intensive Java applications are there?

It doesn't take a genius to realize that is a signficiant task in an application of a real-world size.

And anyone who's run a JVM knows about the price of this task -- yes GC takes time.

However, as I understood the article, the author was making a point that the way most C programmers manage memory tends to make the task more time consuming than is necessary. Therefore relying on a known optimized implementation rather than reinventing the wheel every time may be preferred. After all, it is just the VM implementor that needs to understand how to optimize the memory management, not the application developers. So yes, where the time is spent is shifted but also the amount of total execution time spent on memory management can be reduced -- because the task is managed differently.

As for the specific details of this paper, they're basically discussing how to determine which objects can be safely allocated from the stack, instead of heap, and therefore can be discarded without the usual book keeping required from a heap GC.

how many high performance memory intensive Java applications are there

Java is so widely used on the server side and middleware, it cannot be difficult to come up with examples -- Tomcat, J2EE app servers, etc. eBay for instance advertizes very clearly on their front page to be powered by Sun's Java technology. There are individual Java systems that manage millions of transactions daily, and there must be thousands of systems out there that do this every day with Java.

Well the same issue exists with Copy-on-Write. COW implementations give an impression of faster copying, because they back off all the copying until the first damage to that information. Turns out that in some cases you don't need to make an actual fresh copy in every situation, and that sometimes, you just copy it to use it, don't damage it then just return. In those cases COW gains you time. Since you only copy when you must. You just have to deal with blocking during the first write to the copy. (Y

About 15 years ago I started using a technique to improve performance when doing lots and lots of very short term mallocs.

Essentially, I'd create a large ring buffer of malloced temporary buffers of some standard length. Any time a temporary buffer was needed, I'd grab the next one in the ring.

Before the buffer was provided to the function asking for it, the length would be checked. If the requested length was longer than the current length, the buffer would be freed and one of at least the proper length would be allocated. (I normally allocated by buffers in byte multiples of some fixed constant, usually 32.)

The idea was that by the time it was reused, what was already in the buffer was no longer needed. To achieve that, I'd estimate how many buffers might be needed in the worst case and then multiply that number by 10 for safety's sake.

My primary use of this was when doing enormous numbers of allocations of memory for formatting purposes. The function doing the formatting would request a buffer large enough to hold whatever it would need, write the formatted data into the buffer, and then return a pointer to the buffer. The calling function would simply use the buffer and never have to worry about freeing it.

The performance results were superb except in the very simplest cases where you allocated the buffers without ever using them.

I've never known anyone else who used this kind of approach although I've showed it to a large number of people.

And it can seriously increase the speed of a bit of code. I recently profiled some of my code and found that it was spending around 40% of its time in malloc and free. I created a very simple memory pool (no ring buffers, just a simple linked list with insertions and removals at the same end, nice and fast) for each frequently-allocated object type, and saw a huge speed increase.

Trying programming in Java sometime outside of a website applet, kid, and maybe you will learn something.

I'm quite willing to concede the argument that programmer time is dearer than processor time; but why has the the Real Time Specification for Java [java.net], for instance, only been able to achieve 100 microsecond [rtcmagazine.com] interrupt response times, when 2.0 microsecond [cotsjournalonline.com] response times aren't unheard of in other domains?

Java's RTspec is not designed around those applications. In the Real time world a lot of people are still using ASM because in plenty of applications programmer time is much cheaper than CPU time. If your paying for 10,000,000 CPU's then and 6 programmers then paying for 50c CPU's vs. 1$ CPU's is worth a lot.

I work with real time systems and 0.0001 seconds (100 microseconds) is plenty fast for most Human to Real time systems applications. Granted JAVA is not what you want for fine-tuning your Engines performance but it's plenty fast for most applications. What makes Java so useful is you get to avoid most of the really time consuming bugs. Compare a fully functional java based multithreaded HTTP server with the C / C++ equivalent and it's going to be 1/3rd as much code. And will operate at vary close to the same speeds. In other words it's designed around applications where programmer time is worth more than machine time. We already have C so Java was built around the 95% of applications that don't need inline ASM.

I have killed BSD UNIX with buggy C networking code which is the only thing I have been unable to duplicate with good Java code. You can do bit twiddling in Java, but it's faster in C. You can have hundreds of threads doing their own thing in either but it's much easer to do that in Java than C/C++. The secret is to know enough about how Java works so that you avoid things like creating new threads that eat up a lot of time. Once you understand how things work you can use things like Thread Pooling that are extremely efficient. Instead of complaining that concatenating Strings takes so long try learning about what other tools are out there like StringBuffer.

ISTR that MS tried doing immediate free and it broke some programs that depend on the memory still being sround after being freed, so they made it optional. Sounds like OpenBSD is playing hardball here.

I don't know if it was on purpose but your sig -- which mentions Intron [wikipedia.org] (aka Junk DNA) is very apt.OSes can put Intron between Exon (useful DNA -- useful stuff on the heap) todetect badly behaving apps!

In other words, so-called "Junk DNA" may actually have a use...HEAP PROTECTION;-)

In some cases introns serve a known purpose in indexing for the enzymes which work with DNA. Junk DNA is the stuff with no known purpose at all.

You're still correct. It is heap protection in an evolutionary way. Heap protection on computers seeks to safeguard the data as it is. Junk DNA accepts that things are going to get corrupted and seeks to make it statistically less likely for the important parts to get corrupted.

You're still correct. It is heap protection in an evolutionary way. Heap protection on computers seeks to safeguard the data as it is. Junk DNA accepts that things are going to get corrupted and seeks to make it statistically less likely for the important parts to get corrupted.

Just being a "dummy target" seems like an inefficient use for extra DNA. I would have expected something along the lines of ECC like reed-solomon coding to have evolved. Kinda shoots down the "intelligent design" theory too, unless

First, their system is working properly. Their OS is working, and their application is working. And then SP2 comes out, and they install it. And their application stops working.

Who are they likely to blame? Microsoft of the creator of their application? They'll look at what changed, and decide that Microsoft is to blame. And if this application is important, they'll even back out SP2 to

In the three cases we ran into when I was a consultant last year we placed the blame squarely on the shoulders of the vendor with the foobar product. They were coding in an unsafe manner to an undocumented API, it was 100% their fault. Now a naive home user with no real computer savy might blame MS, but when you look at the tradeoffs it seems like the security cleanup was good for the entire computer using public, even if a few people did have an app or two broken.

In the three cases we ran into when I was a consultant last year we placed the blame squarely on the shoulders of the vendor with the foobar product

Ok, but most Windows users don't have anything to do with consultants, and think that vendors are the guys who sell hotdogs (or the things you close to stop the wind from coming in, depending on where they live.)

Ultimately, think of Grandma. She fires up AOL, then her computer tells her that she needs SP2. She goes `ok' and many hours later, SP2 is don

None of the applications which were actually broken [microsoft.com] by XP SP2 are something granny would run. On the other hand some of the applications that were affect [microsoft.com] were, but those were generally minor problems.

You mean the Data Execute Protection [microsoft.com] from Microsoft? OpenBSD has had that for a long time already, only they named it w^x [neohapsis.com].

This new feature from OpenBSD is the use of guard pages and the immediate freeing of memory. In essence this means that both bad programming and exploit attempts are much more likely to result in a core dump then some unidentifiable and non reproducible corruption or a working exploit. Many people consider that a good thing because it will result in bugs being found in userland applications that would have otherwise stayed unnoticed. So even if you don't use OpenBSD yourself this is helping your system becomming more secure and better. And if you are running OpenBSD there is o need to worry too much about the stability of this feature, it was actually enabled shortly after the 3.7 release and has been in every snapshot on the way to 3.8.

And I have to agree with the author that the best thing is that we get all the goods without ever having to switch them on!

"You mean the Data Execute Protection from Microsoft? OpenBSD has had that for a long time already, only they named it w^x."

They also didn't need the per-page execute bit to do it. You need a fairly new machine to get the protection, but my 486 firewall has it. They also have stack protection, which is helpful because even if the heap and stack aren't executable you can overwrite return addresses or pointers to functions, and have them point to existing code that can be tricked into doing something malicious.

In fairness, I don't know that we can really blame Microsoft for that
one...

XP's memory protection works just fine (assuming the CPU supports NX
pages) - The concept takes almost no thought to implement. The problem,
however, arises from 99.999% of existing software not caring in the
least about the separation of code and data. Usually that doesn't
cause any problems, but when a "clever" (I put that in quotes as the
bad-idea-of-the-day, but

The NX flag is basically opt-in, you can request pages that aren't marked with it. In an OS that flags everthing as NX by default, you'd need to rebuild your application (although XP allows you do disable NX on a process by process basis). The point of NX isn't to prevent stuff like Scheme working, it's to prevent exploits or accidents.

There have existed functions in Windows and Linux for a long time that mark a page as executable. Even though Linux or Windows never really enforced this to be set when executing data, it was required by specification.

Now that they're enforcing the specification, people complain that it broke.

Hey, PearPC was written before they enforced the specification, and Sebastian Biallas had the brilliant notion to actually follow the spec, and mark things as executable. Thus, when SP2 came out, PearPC worked fine.

Usually things break when moving to a newer version because some area of the spec wasn't very heavily stressed, and people writing code that just works (not as in, it works, but as in barely works,) thusly never really bothered shooting towards the spec. Then when the spec is enforced, they get all upiddy claiming that it breaks their app. Their app was broken to begin with, the previous implementations that you were relying on just didn't care.

For instance, when libc 5 (I think, don't hawk me about versions, if you know the correct versions, then please correct me, but I'm working off of a poor memory of the version numbers) came out, it enforced against passing a NULL file pointer. Before hand some people had hacked their code such that if an open failed, and returned a NULL file pointer, they didn't care or print an error message. They just kept going, since it would just waste CPU cycles, as nothing would get outputted or read from the file. It was silently gracefully failing for them, and they used that.

Then libc 5 comes out, and they break this silent graceful failure, and started reporting errors, or crashing when you passed them a NULL file pointer. People yelled and bitched, because they broke their app. But remember, THEIR APPS WERE BROKEN IN THE FIRST PLACE.

That's why I don't like people griping about "blah blah upgrade broke my app". Unless you can state that your app was built to spec from the beginning, then that upgrade didn't break your app. It was broken to begin with. The new upgrade just showed you how it was broken.

Kudos to the OpenBSD folks for being at the cutting edge, in terms of implementations of these security features. Where they lead, surely others will follow and we'll be seeing this feature become commonplace. As their focus is security, its understandable that they lead more incentives in these areas than more mainstream Linux distributions.

I know it seems strange...but OpenBSD isn't a Linux distribution at all.

I know its hard to wrap head around. Its one of those things you just have to accept. In addition:-deep down, cows are not people too. So you can eat 'em, I guess.-neither are cats or dogs. So don't force them to wear clothing.-neither is information. So it doesn't care about being free or anything else.-"Windows" is somehow both an operating system and a Window manager. You're not supposed

You can certainly read it that way if you like. You wouldn't be conveying what that sentence did, though.As their focus is security, its understandable that they lead more incentives in these areas than more mainstream Linux distributions.

In this, the word "more" describes "Linux distributions." As it is a comparison, it also refers back to the subject of the phrase. The subject being "they," a pronoun which refers to "OpenBSD." This comparison also draws them into the same category because its comparin

Won't this crutch actually tempt people to write sloppy memory management because "the heap manager will catch it?"

Well, it'll catch it... and the application will immediately crash.

I don't see how that encourages writing `sloppy memory management'. Nobody wants an application that crashes. (Granted, it's correct for an application to immediately exit when it's in danger of doing damage to something, but to an end user, they don't want their application to crash.)

You mean it will encourage sloppy coding more than the current system?

"hey, it might do something strange every once in a while, but at least it keeps running and doesn't crash, so the code is 'good-enough'".

I think Theo is doing entirely the right thing by killing badly written apps rather than letting them do bad things to the system. It's much more likely to make people fix bad code than the current system.

When my application needs a chunk of memory, it sends a specially crafted HTTPS request to my bank, debits the account, sends a fax to the local computer shop who then sends a tech over to install the DIMMs.

When the application is finished with the memory, it sends a FAX to the local electronics recycling facility who sends out a tech to remove the DIMMs and melt them down into whatever.

Using this method of heap memory allocation (I call it "ACAlloc" for "Anonymous Coward Alloc" has been 100% effective and I have NEVER had a heap overflow exploit in any of my code.

Ok, we start out with 'protection', then we move to 'a heap' of protection, most assuredly to be followed by 'a whole heap' of protection. I can only see this spiral continuing until Bill Gates himself gets up on stage at CES in an Elvis suit promising 'a hunka- hunka- burnin protection'. *SHUDDER* Time to take a cold shower.

"And if we have to sacrifice a little usability on the way there, then so be it." Nice dream , meanwhile in the real world both users and most coders(if they dare to admit it) will NOT sacrfice usability or ease ofcoding for security measures that (in their minds) are nothing todo with their application. Unless that is they're forced to eitherby company policy or restrictions in the OS. If there are restrictionsin the OS then IT tech leads might start to ask "well, I can do this inOS A , why won't OS B let

What they restrict here, is extremely shitty programming. What's the point of allowing accessing memory areas out-of-bounds? Fewer crashes, because the system does not notice them? This is a very bad idea. If you don't know how to handle arrays and you don't know any other means to restrict yourself from doing something stupid, OpenBSD shows the best way to do it (as last instance).

The reason is very simple: There will always be some applications where the security of the system is a paramount point. Where it does have to do with the application. OpenBSD caters directly to those people, and those applications.

Now, you are right, this means OpenBSD is likely to never get as large a following as Linux or even FreeBSD, but they honestly don't care. They are making a system that fits their goals, and security is among the top goals.

This actually allows security to spread: Once these changes are on a 'major' system, applications start to be ported to work with them, which means the changes can be ported to other systems.

OpenBSD is a security testing ground. If it's features get in your way, you use a different system. This won't be the first time that advice will have been applicable.

Could this technology be implemented in the Microsoft Windows systems to be more secure than Linux?

It certainly could be implemented, but it would have the slight drawback that a huge proportion of apps would stop working.

It's going to break a lot of stuff on OpenBSD as well, but because of their audience they can get away with e.g telling you not to run Apache because it's insecure. Also, the OSS apps that break because they assume a specific memory management model will get fixed. No-one is going to be a

I don't know if GNU malloc uses mmap() or brk() for its allocation, but in both cases small memory chunk that the user allocates are taken from bigger, contiguous blocks of memory.Maybe that's primitive, but it ensures that usually small memory requests are fast, and don't have to much space overhead, either.

If the memory de/allocation patterns are really bad, though, you get fragmentation with the mentioned problems. It's a tradeoff. Note that OpenBSD didn't choose its way for reducing heap usage (maybe

I don't know if GNU malloc uses mmap() or brk() for its allocation, but in both cases small memory chunk that the user allocates are taken from bigger, contiguous blocks of memory.

It uses mmap() for big allocs (IIRC the threshold is 4 MB) and brk() for smaller ones.

There would be one solution, and that's using different arenas, or memory regions for allocation. For instance every window might have its own allocation region, so when you close the window/document, the memory BLOCK is freed.

Well, I've turned it on now. No noticeable performance hit, and no mysterious application failures. Just compiled a Visual C++ program and it ran fine. Same for G++ on CYGWIN.

Also, I have not been hacked anytime in the last ~5 minutes, whatever that's worth (but would I know?).

As an aside, I read the paper on the Microsoft DEP flaw a few months ago, and wasn't that impressed. It looks very hard to exploit. And since DEP is a added protection mechanism, the existence of a small, hard-to-expliot flaw

He explains that for over a decade efforts have been made to find and fix buffer overflows, and more recently bugs have been found in which software is reading before the start of a buffer, or beyond the end of the buffer.

The solution that the kerneltrap.org refers to against buffer overflows is to:

unmap memory as soon as it is freed, so as that to cause a SIGSEV when illegaly accessed.

have some free 'guard' space between allocated blocks.

My opinion is that #1 will slow software down, although it will make it indeed more secure. #2 will make it more difficult to exploit buffer overflows, since the space between two allocated heap blocks will be random (and thus the attacker may not know where to overwrite data).

Unless I haven't understood well, these solutions will not offer any real solution to the buffer overflow problem. For example, stack-based exploits can still be used for attacks. The solution shown does not mention usage of the NX bit (which is i86 specific). It is a purely software solution that can be applied to all BSD-supported architectures.

Since all the problems relating to buffers (overflow and underflow) that have costed billions of dollars to the IT industly is the result of using C, doesn't anyone think that it is time to stop using C? there are C-compatible languages that allow bit manipulation but don't allow buffer overflows; e.g. Cyclone [wikipedia.org].

C is actually the most secure language currently, AFAICT. Languages with higher level intrinsics (C++, Java, Basic, Mono, Objective-C, etc) have a more complex implementation that may allow different exploit vectors; while languages with real-time interpretation or runtime code generation (Java or Mono with JIT) will wind up disabling things like strong memory protection policies (strict Data/Code separation -- code is data when generated at runtime) and may not in their own code create a backup buffer overflow protection.

In the event of a screw-up on the part of the JIT or runtime programmer for any language, every program is instantly vulnerable, and all of this generic proactive security stuff is disabled because this "secure language" doesn't work in an "inherantly secure" environment, only a much weakened one. C's runtime is rather basic (and it's still huge), as is its language; people still screw that up once in a while, but rarely.

While these "shiney new secure languages" may boast "immunity to buffer overflows," their runtimes are still designed around other concepts that may leave holes. Look at this memory allocator and think about a bug in the allocator that smashes up its own memory before it gets everything set up; because the new protections aren't yet set in place, it'd be totally vulnerable at that point (no practical exploit of course). A bug that forgets to add guard pages (generates 0 guard pages every time) might occur too in one update. Now add to that something like Java or Mono-- interpreted or not, you're running on a whole -platform- instead of just a runtime. C++ instruments loads of back-end object orientation.

So in short, C is a very basic language that has easily quantifiable attack vectors, and thus the system can be targeted around these for security. Several such enhancements exist, see PaX, GrSecurity, W^X, security in heap allocators, SELinux, Exec Shield, ProPolice. Higher level languages like C++ implement back-end instrumentation that ramps up complexity and may open new, unprotected attack vectors that are harder to quantify. Very high end languages on their own platform, like Java and Mono, not only implement massive complexity, but rely on a back-end that may lose its security due to bugs. Platform languages may also be interpreted or runtime generated, in which case they may require certain protections like PaX' strict NX policy to vanish; in some cases these models (as an implementation flaw) also don't work well with strict mandatory access control policies under systems like SELinux.

Face it. C is the best language all around for speed, security, portability, and maintainability. Assembly only brings excessive speed at the cost of all else; and higher level languages sacrifice both speed and real security (despite their handwaving claims of built-in security) at varying degrees for portability, speed of coding, and maintainability. Even script languages working inside a real tightly secured system would more easily fall victim to cross-site scripting, the injection of script into the interpretation line; under such a system, any similar attack is impossible in a C program.

On a side note, I'd love to see a RAD for C. Think Visual Basic 6.0, but open source, using C/GTK+. Glade comes close. . . .

The solution shown does not mention usage of the NX bit (which is i86 specific).

Actually, x86 is one of the last into town with "NX bit" functionality. POWER (and PPC I guess), PA-RISC, Sparc, Alpha, etc. on the big-iron has had this feature as a standard part of their architecture (along with the OSes that run on them) for bloody ages now... on those CPUs, even Linux has had hardware support since before whenever x86 got NX support.

Languages like Lisp, Haskell, Scheme and ML allow you to avoid buffer and heap overflows (assuming the language implementation is correct).

It is therefore, in my opinion, less optimal (from a security perspective) to use something like "C" for a complicated app like sendmail, web server or secure shell daemon (sshd) than it is to use a language like "C".

ALL buffer and heap overflows in individual programs are the fault of bad programming, not bad programming languages. If a programmer is not competent enough to test their code, and then blame the programming language when there is a problem, should not be coding. It is simply irresponsible.

You forgot to add Java to your list of programming languages which goes out of its way to assist in preventing and avoiding both buffer and heap overflows/errors (i.e. bounds checking and similiar technology).

Although you can blame the programmer for writing a faulty application, I don't think that is relevant.You are a taxi driver. Your job is to deliver people from point A to point B, safely and quickly.

You can choose from two cars.

Car A is such that if you make an error when driving, it will do something bad -- perhaps even blow up, killing the passengers. Furthermore, a driver can accidentally drive the car into an obstacle, ruining everything.

ALL buffer and heap overflows in individual programs are the fault of bad programming, not bad programming languages.

I agree, and I'd go one step further to say that we're starting to get into a situation where programmers are learning and using languages (such as Java) that don't allow this particular kind of sloppy coding. The problem is that many of these programmers aren't even aware of the concept of a buffer overflow, let alone how to actively detect or prevent it.

All languages allow you to avoid buffer and heap overflows if the language implementation is correct. Buffer and heap overflows are programming errors, not problems with a language. Some languages just have better facilities to force well-behaved code. You can add Java and C# to your list there by the way.

Languages like Lisp, Haskell, Scheme and ML allow you to avoid buffer and heap overflows (assuming the language implementation is correct).It is therefore, in my opinion, less optimal (from a security perspective) to use something like "C" for a complicated app like sendmail, web server or secure shell daemon (sshd) than it is to use a language like "C".

The problem is for one reason (or randomness) or another, no mainstream daemons or operating systems or client applications are written in Lisp, Haskell, Sc

I don't understand how you'll do a language like ML or lisp without automatic storage management.The problem has to do with returning closures with bound variables -- the nice thing about having a GC is that they will get freed, eventually.

If you do manual memory management, will you manage this yourself (for closures)? Or will you ban closures? If you are banning closures, you've lost one of the key benefits of using Lisp or ML --- you just got rid of lambda (in its most general form), and you'll just have

I'm surprised that modern heaps still put writable data segments adjacent to executable code segments. Self-modifying code is rare enough that all code should be in read-only (except by privileged processes, like the kernel which sets it up and tears it down) segments, except when a process has privilege to write to its own code segment. Then its code segment should be a data segment, with other security features applied (that are too expensive to apply to every code segment). Generally then, segments are either writeable or executable. Data segments could still get overwritten, which could put unsafe values in unexpected variables (like "write to that file" instead of this file). But at least those insecure operations are limited to the operations programmed into the code, not arbitrary new code inserted into executables by buffer overflows.

After decades of these problems, and known techniques already applied in Computer Science, its surprising that we're only now seeing these techniques deployed in popular OS'es like OpenBSD. Hopefully the open nature of OpenBSD and other OSS OS'es will see them tested for winning strategies quickly, and widely adopted.

Your example is one of the few cases of self-modifying code (or permission to do so). Old programming languages used to declare some "variables" as "constants". I don't know why we discarded that mode.

I wonder of all these creative heap protection mechanisms (GCC 4.1, NX, etc) will largely remove the need for managed code (ie Java and C#). If you can write in C++ and not worry about double-free()s that pretty neat. Maybe even a "disruptive technology".

Not be using free in the first place. It's an error to code C++ as if it were C—use new and delete

Not be doing explicit memory management. Use RAII (Resource Acquisition Is Initialization) practices. Not using RAII in C++ is an error. (Beyond resource handling nightmares, you aren't being exception safe without RAII.)

You don't "rearrange" the MMU, you just check/clear the dirty bit in the page table. This bit is set by the hardware regardless so setting it is zero overhead. Clearing these bits is still only 0.006% the time of scanning all of memory. So unless you write to 32k unique pages per GC you get a huge benefit from this.You could probably also use the MMU to reduce pauses in the gc... you determine what objects are unreachable in the background and using the dirty bit you can tell which pages may have referen

Electric Fence explicitly allocates dead pages at the end (or configurably, the beginning) of buffers. It can also protect the memory immediately as it is freed. I think it was first published in 1987.

It may be a legitimate invention - it is cited as prior art in an ATT patent. This is also the first known example of a prior Open Source publication causing a patent filer to cite. ATT also removed a claim from the patent that my work invalidated. Just search for "Perens" in the U.S. patent database to find the patent.

We don't run it on production programs because of its overhead. To do this sort of protection exhaustively, it requires minimum two pages of the address space per allocation: one dead page between allocations and one page allocated as actual memory. This is a high overhead of page table entries, translation lookaside buffers, and completely destroys locality-of-reference in your application. Thus, expect slower programs and more disk-rattling as applications page heavily. If you are to allocate and release memory through mmap, you get a high system call overhead too, and probably a TLB flush with every allocation and free.

Yes, it makes it more difficult to inject a virus. Removing page-execute permission does most of that at a much lower cost - it will prevent injection of executable code but not interpreter scripts.

I don't think the BSD allocator will reveal more software bugs unless the programmers have not tested with Electric Fence.

I don't think the BSD allocator will reveal more software bugs unless the programmers have not tested with Electric Fence.

I think that's exactly the problem though. There are users out there installing applications they found somewhere, by somebody who may or may not have bothered to use a good debugger. This will prevent those unknown bad apps from fouling up the system.

I'm still on OpenBSD 3.7. I haven't tried the 3.8 builds, but I'm hoping the overhead from this won't be too bad.

I don't think the BSD allocator will reveal more software bugs unless the programmers have not tested with Electric Fence.

The OpenBSD allocator already has revealed a number of software bugs (in X11, in ports, they lurk everywhere). Some of the bugs found were years old. Thats the point of the testing process in the OpenBSD release cycle.

I think you're missing the point behind the integration of these technologies into OpenBSD. The idea is that they are always on, with a performance hit that is acceptable that your day-to-day programs can be protected, and most crucially, used under them. Not sitting in a debug environment getting limited regressions and unit tests that the particular programmer felt like writing (and if I want that, I run it under Valgrind, which has a near-miraculous tendency to find lurking bugs).

And considering them in isolation is also dangerous. When you combine the address randomization, W^X, heap protection, propolice canaries and local variable re-ordering, you're left with a system that has accepted a reasonable performance hit in return for a large amount of protection against badly written code. Sprinkle in regular audits, timely releases every 6 months to keep our users up-to-date on stable systems, and a 'grep culture' to hunt down related bugs in the source tree when a bug does strike.

As others have pointed out, other "hero projects" have stuck bits and bobs into their respective distributions. But how many have had the discipline to follow through, maintain and integrate their patches, test the fallout and release a complete OS with thousands of third-party packages year after year... probably only Microsoft, but the first thing they do at the sign of incompatibility is to turn the protection off. Oh well:)

The guards pages are only practical for larger allocations due to hardware limitations, as you say. Strings are protected by different means such as Propolice; in order to minimise the overhead of Propolice, it "detects" strings (as opposed to byte buffers) and specifically protects them with canaries to try and find overflows that would smash the stack (the local variable re-arrangement tries to put these buffers as close to the canary as it can at compile time). The string detection is a heuristic as gc

" The more hurdles that one has to jump through for good security, the less likely people will go through the trouble. OpenBSD allows even the most inexperienced users to take advantage of these technologies without any effort. "

I'd be interested to know what the performance impact of this is - exactly what counts as "too much of a slowdown".

Application heap allocation has "traditionally" been fairly inexpensive unless the heap has to be grown (update a couple of free block pointers/sizes) and the cost of growing the heap (which requires extending the virtual address space and therefore fiddling with page tables which would on a typical CPU require a mode change) is mitigated by allocating more virtual address space than is immediately needed.

If free space is always unmapped then each block allocation will require an alteration to the page tables, as will each unallocation. Not to mention that could cause the page-translation hardware to operate sub-optimally since the range of addresses comprising the working set will constantly change.

If most allocations are significantly less than a page size, then the performance impact may be minimal since whole pages will rarely become free, but if allocations typically exceed a page size, that would no longer be true. If the result is that some applications simply implement their own heap management to avoid the overhead, then you've simply increased the number of places that bugs can occur.

If this is the case, it's great that such a feature should go into an OS by default. I personally love anything that gives me confidence in the implementation of any applications I write, especially as this type of technique makes debugging much easier.

Just like...Windows 95 was more secure than Windows 3.1Windows 98 was more secure than Windows 95Windows NT was much more secure than Windows 98Windows 2000 was the mother of all securityWindows XP, this time we got it rightWindows XP SP2 this time we really really got it right, promise, cross my heart and hope to dieWindows 2003, most secure server OS ever built!Windows VISTA, even better than the worlds best system ever built, this time ill put my mother up here on this 100 fot pole and if im wrong may she fall down into that pit of crocodiles!

Until i have a real life experience of good Windows security i will tend to think back and remember all the former promises that have gone down the drain. Today you expect Microsoft to promise things that arent really true.

Then again, while I'm not defending Microsoft, here's my take on their 'security':Yes, plenty, and maybe even most of their promises about being a generally secure system are complete and utter rubbish. However, I'm willing to bet that each of their OSes are more secure than the last one. The problem is that they still leave plenty of holes open when they do things like (to point out the landmark example) weld the web browser to the kernel. I know that most people crack windows because it's easy, but while

The MS thing is just support for no-execute: the bit that says that this page is only code and not data and you shouldn't try to run anything in here. Everyone has supported this for ages.

This is more. It looks like they are adding extra 'tripwire' pages to the heap, so if an attacker manages to write to part of the heap they shouldn't, there's a good chance they'll hit a tripwire and be detected.

Sure, you can find demonstrations of things that fail in C which won't fail in Common Lisp. But Common Lisp is not exactly known for the human readability of its syntax, and human readability is the key to reducing algorithmic bugs.

Do you also advocate four times the memory usage and double the speed? Do you blame the language or the speaker when they can't formulate a proper sentence construct? How about we just teach the programmers better instead of bitching about the tool they use.

If half the C coders out there knew the differences between stack, heap, and namespaces, we would not even be debating this issue. Don't blame the coders, blame the universities.

Who is "not giving back?"Feel free to download all of their source code before complaining.If you don't like it, don't use it, even though I'm sure you too benefited from some security fixes that originated with OpenBSD.

But they don't care, they're not trying to be FreeBSD or a Linux distribution, they're trying to be OpenBSD and a part of that is not letting people's perception of optimum performance get in the way of doing what is right by them.

You gotta remember, the project doesn't do it for outsiders, what they do is for themselves. They want security and are willing to pay performance and ease of use to get it, it's like a mantra for them, never take the path of least resistance.

If this looses like 5 or 10 percent of it's performance on my machines I won't mind, it's another layer of protection and I like having it and am fine with the cost, faster hardware isn't that expensive. If something I run crashes, I will report to the people that wrote it, telling them that I found a problem that was found by OpenBSD's malloc, maybe they'll even devote an old test box to checking for bugs on it.

If OpenBSD was trying to be a Linux distribution then we'd not have most of the good stuff that makes OpenBSD unique.