Notes on Programming & Mathhttps://progmathnotes.wordpress.com
Various notes on Programming, Mathematics, and similar stuffMon, 19 Mar 2018 12:12:36 +0000enhourly1http://wordpress.com/https://s2.wp.com/i/buttonw-com.pngNotes on Programming & Mathhttps://progmathnotes.wordpress.com
PSA: How to safely check buffer sizes in Chttps://progmathnotes.wordpress.com/2014/04/23/psa-c-safe-buffer-check/
https://progmathnotes.wordpress.com/2014/04/23/psa-c-safe-buffer-check/#commentsWed, 23 Apr 2014 17:50:52 +0000http://progmathnotes.wordpress.com/?p=71A common task in C is checking whether some data fits in a buffer. Unfortunately, doing it can be quite tricky, as http://lwn.net/Articles/278137/ shows. Going by the standard, the problem is that in C creating a pointer beyond the end of an array is undefined behaviour – which means the compiler can do whatever it wants – including sending your customers’ credit card numbers to Russia (actually that’s quite a probable outcome, because, as the link shows, the compiler may introduce a security bug, which will be exploited and result in the credit card numbers being sent).

First, I’ll like to say that C does allow you to create a pointer to exactly-the-end-of-an-array (say, if A is an array of length 5, then you can create a pointer to (the non-existent) A[5], as long as you don’t dereference it).

The code in red tests for overflow by first creating a pointer overflowing a buffer, which overflows a buffer and begets undefined behaviour, and then tries to check the length, but the check comes too late – the program’s behaviour is already undefined, and the compiler, knowing so, may remove the check.
The code in orange is slightly less wrong – the check is actually well-defined, because unsigned integer addition is defined as addition modulo 256^sizeof(size_t) – however, it checks the wrong thing, because if an attacker puts, say, len=(size_t)-sizeof(unsigned int), then len+sizeof(unsigned int) = -4+4 = 0, and the check will succeed and copy a huge amount of data off the end of the buffer, causing problems. I repeat, the red and orange codes are evil – use the blue or turquoise code unless you are introducing a backdoor.

Note that in these examples, read_uint is assumed to be a macro/function that reads sizeof(unsigned int)=4 bytes from a char* as an unsigned integer (this isn’t the same as *(int*)packet, because of alignment and endianness issues) – there does not seem to be any standard here. Assuming data is big-endian (as in most network protocols), it can be defined as:

]]>https://progmathnotes.wordpress.com/2014/04/23/psa-c-safe-buffer-check/feed/1arielbydOn Exceptionshttps://progmathnotes.wordpress.com/2014/03/25/on-exceptions/
https://progmathnotes.wordpress.com/2014/03/25/on-exceptions/#respondTue, 25 Mar 2014 19:53:55 +0000http://progmathnotes.wordpress.com/?p=52Another programming subject that tends to confuse people is Exceptions. It appears that each programming language (including *each* machine code architecture) and each operating system has its own subtly-incompatible version.

On the surface, the idea appears quite simple: sometimes exceptional situations occur, and the program must handle them. However, when you dig deeper, you find several similar but distinct ways of thinking about it. I’ll try to detail the ones I heard about. Note that people occasionally design one implementation but actually implement another!

The oldest way of thinking about exceptions is as return codes – like in C. They have the advantages of being standard within a language and containing more information than a typical error code, as well as not overloading the return output. This method looks quite fine on the surface, but it loses the explicitness of error codes, causing many a subtle bug.

The second way I found is the rather operational way of exceptions being a “longjmp to a stored handler”. This kind of thinking is generally fine when programming is side-effect-free but can cause quite a bit of a mess otherwise, especially if garbage-collection is missing.

The essential problem in these methods is that exceptions add many paths to a program that are rarely tested. This is especially pronounced in out-of-memory exceptions, which in many programming styles can essentially occur every second line, but in practice almost never happen.

Note that in many of these cases the program should just exit. It was noticed that all languages already have a similar “mechanism” to this – by the halting problem functions can always use an infinite amount of time before returning (observe that a common mechanism of dealing with OOM – paging/swapping – can, in particularly bad situations, cause functions to use a practically infinite amount of time and you can’t do much about it). In some ways this behaviour is not that different from a hard-reset of the computer (e.g. the power being pulled) – which may very well be a (user-caused) consequence of it happening to a critical program.

Papers typically call that behaviour divergence for rather obscure reasons, which is the name I’ll use in the rest of this post. I’ll like to note that C11 actually prohibits it, in clause 1.10/24:

The implementation may assume that any thread will eventually do one of the following:

— terminate,

— make a call to a library I/O function,

— access or modify a volatile object, or

— perform a synchronization operation or an atomic operation

However, several programs have decided to use a divergence-like mechanism to handle (some kinds of) exceptions – a good example is most GUI toolkits, which like to abort on memory allocation failure. This has the advantage of essentially not having error paths to test.

However, this method has the big disadvantage of leaking the entire program on an exception. Often, this isn’t an issue: unlike what many CS instructors believe, the OS keeps track of a program’s resources and frees them when the program exits (in any way). But sometimes one wants only an operation to diverge – consider the case of a web browser with 30 tabs open, when the user enters a page with a badly-designed script that uses many gigabytes of memory – one (of course) won’t like to leak the entire tab, but one won’t like to close the entire browser either.

For these kinds of cases, one wants a cancellation method. This is quite similar to divergence, except it releases the resources used by the program. As I said earlier, typical operating systems already provide it if the program is separated into processes, and a similar mechanism, separating a program into disjoint heaps, is often used to have the advantage of this method without these issues.

However, canceling a tangled program is generally a tangled issue that I will leave for a later post.

CHANGES 2017/01/27 – fixed formatting

Advertisements

]]>https://progmathnotes.wordpress.com/2014/03/25/on-exceptions/feed/0arielbydOn Pseudorandomnesshttps://progmathnotes.wordpress.com/2014/03/24/on-pseudorandomness/
https://progmathnotes.wordpress.com/2014/03/24/on-pseudorandomness/#respondMon, 24 Mar 2014 15:57:47 +0000http://progmathnotes.wordpress.com/?p=22It has been said many times that the easiest way to break a crypto-system is by breaking its random-number generator. This seems to have a basis in practice – the PS3[1] and Debian[2] are only 2 of the most famous serious flaws caused by one. An important cause for this is that the PRNG (for obvious reasons) aren’t really visible on the wire or in other places. Another important cause is that PRNGs are used to perform several different but related tasks in a confusing way that isn’t really documented anywhere.

I can’t do anything to help the former problem, but I’ll try to explain the latter.

I will use PRNGs to refer to both the cryptographically-secure ones and the non-cryptographically-secure ones together (rather then calling the cryptographically-secure ones CSPRNGs). This is because “cryptographic-security” is only one of the axes that make a PRNG useful (or secure) for a purpose.

Almost always, one prefers programs to be deterministic (if only because that makes debugging much easier). To help it, CPUs are typically written to behave in a highly deterministic manner (except when one uses shared-memory parallelism, but that’s a problem for another post). However, sometimes, this property can be sometimes a problem. A few cases when this occurs are:

1) Some algorithm works properly in the average case, but displays bad behaviour in the worst case, which is triggered either by being similar-in-structure to the program (as in quicksort on a sorted list) or by input intentionally crafted by an adversary (as in Hash-Table DoS[3]). Note that in some cases, typically when a program does some kind of Monte-Carlo simulation (i.e. estimates a distribution by sampling, e.g. fuzzing), it can return wrong results rather than just taking a long time to finish.

2) Often, one needs to generate unique identifiers for some purpose and does not care if the length is overly long. Cryptographic systems often require this to avoid replay attacks (another confusing thing that I am going to write a post about) and keymat-reuse issues (sometimes called many-time-pads).

3) One needs to generate a secret key that cannot be predicted by an adversary, perhaps even for a long amount of time – decades, in a few cases. (e.g. all crypto stuff – short-term authentication cookies to a browser game, long-term keys protecting Top Secret documents, and everything between them).

4) Stream Ciphers – if a completely pseudorandom stream is xorred with a secret, the secret can be recovered from the result only by these knowing the stream. This provides a fast, securely-implementable form of encryption.

There are probably more uses, but these are the most common.

It seems that there are four properties of PRNGs that are used in the application:

1) Weak Unpredictability – Adversaries can’t predict the PRNG’s output. However, the set of outputs may still be relatively small so a brute-force attack may be possible. This is quite similar to strong unpredictability detailed below.

2) Strong Unpredictability – Adversaries can’t predict the PRNG’s output or iterate over all possibilities. For practical reasons (e.g. backups), this is often divided into short-term and long-term unpredictability, where long-term secrets shouldn’t be stored in the clear on non-specially-protected devices. An annoying problem with this property is that a backdoor can make a PRNG predictable to these in the know while being random to everybody else – making checking for it impossible – and that is the classic failure mode of this property.

3) Uniqueness – The results are distinct in different runs. This is distinct from unpredictability – generating a single stream with a PRNG from a secret seed but not saving the current position when the system reboots, from example, can destroy Uniqueness while preserving Unpredictability. The classic failure mode is a guard asking the same challenge and accepting the same response every day – allowing eavesdroppers to simply repeat it.

4) Whiteness – The stream does not have visible internal correlations. This is the property typically checked by “randomness tests” (e.g. Diehard) and often confused with Pseudorandomness. This can be further divided into Classical Whiteness – the absence of obvious patterns, and Cryptographic Whiteness – which requires the absence of all computably-discoverable patterns, which is important in e.g. stream ciphers. The classic failure mode is a subtle internal bias – slowly leaking a secret over multiple ciphertexts.

These properties can be combined in various ways – a stream, for example, can be Unpredictable and Unique, both not Unpredictably Unique, for example when a professor generates an exam by randomly selecting a few questions from a relatively small and constant set. Concatenating streams creates a stream that has the union of the properties of the substreams, but not together.

Preventing worst-cases typically requires a small amount of (Classical) Whiteness that in adversarial scenarios is Weakly Unpredictable (of course, if one leaks the randomness via the algorithm results, then ze needs not to reuse it). Note that CBC nonce problems[4] basically require this.

Hash Functions typically take a stream, combine its properties and add Whiteness – so H(secret||ctr++) has all 3 properties together – the secret gives Unpredictability, the counter gives Uniqueness, and the hash combines them and gives Whiteness. However, hash functions don’t generate Unpredictability or Uniqueness – H(current-time) is quite predictable and H(secret) isn’t unique.

Hardware RNGs often provide Unique Unpredictability, but often not Whiteness.

As a part of the language, the only mainstream language that supports this feature is C++, although several novel languages, like Haskell and Rust, support it too. This is claimed to account for much of its speed advantage over managed languages, such as Java, in abstracted code.

Because of this, I was disappointed when it turned out that I have to write my supposed-to-be-fast alpha-beta implementation in C for my software project in a generic manner, which would seemingly require a virtual call every node and probably an allocation (because different boards have different sizes).

I thought about this problem and I think I found a solution. Before presenting it, I would like to describe a few quite-common C preprocessor tricks.

Trick 1 – foreach in C:

This is a quite old trick that can be used to implement foreach in C – the oldest use I found is in BSD’s queue.h

I think the best way to introduce it is by example – which iterates over a list-style linked list.