Here we want the data ordered in descending (largest first) sequence, so we have(*) to use code block to alter the default ordering.

The { $b cmp $a } part of that tells sort that we wish to perform the ordering ourselves instead of using the default as we did in 1 above (which is equivalent to { $a cmp $b }). This says that for each pair of values in the input, Perl passes them to us (as $a and $b), and we perform a comparison and return

-1 if $a is less than $b

1 if $a is greater than $b

0 if $a equals $b

With this information, sort can correctly do what it needs to do to reorder the input appropriately.

By using the built-in cmp operator that is specifically designed for comparing strings and producing these values, we achieve the result we are after.

Supplying the arguments $a and $b to the function in reverse order has the effect of reversing the sort.

However, if we use cmp on numeric data, we do not get the effect we want:

The difference is that <=> compares the (binary) numerical value of its arguments, instead of the bytewise string comparisons of the ASCII representation that cmp does.

my @sortedNumbers = sort{ $a <=> $b } @numbers;

And for an ascending numerical sort, we just pass the parameters to <=> in the 'right' order.

How sort actually works internally doesn't really matter. What does matter is that being built-in to the language, it understands Perl's internals, is always available and about as fast as you are likely to achieve.

So, sorting data is easy, when you want to sort according to the entire value of each element of the array or list. It gets a little more complex when you want to sort the elements by a sub field of each element.

For example, say you have a set of ids (A123 B421 C987 etc.) and you want to order them by just the numeric part, then you need to extract that part before supplying it to the appropriate comparison function:

This works fine, but each element will be passed to the comparison block multiple times, and that means we are extracting the sort field from each element multiple times, which can slow things down. For more complex cases, (eg. data consisting of dates), this could be significant. The purposes of the "advanced sorting" mechanisms is to reduce this slowdown by performing the extraction of the significant data from each input element once only.

The ST

One mechanism, and possibly the simplest to understand, is merlyn's ST (Schwartzian Transform). This works by pre-processing the elements to extract the fields, and packages them into anonymous arrays. The list of anonymous arrays are then sorted according to the extracted field. And finally, the original elements are recovered from the list of sorted anonymous arrays and stored into the sorted array.

This is identical to the previous example, but more compact. It can also be formatted as a single line, though that will usually mean that it extends beyond the boundary at which PM does its dumb wrapping. It can also be laid out in several different ways to that I've shown, but I find that this makes the three significant steps of the algorithm clearly delineated, by indenting the contents of the code blocks just as you would for any other block of code. Eg. The body of if and while statements etc. I like this consistency.

The efficiency of the ST comes from only performing the extractions of the field data once. Indexing the anonymous arrays is very efficient and so the sorting and reassembly doesn't costs much extra. It also extends to sorting multiple fields in an obvious way:

Here we've ordered the data according to the numeric field as before, but this time we've used the alpha field to 'tie-break' values when the numeric values are the same. It's easy to see how this can be further extended to handle as many fields of any type we need. This simple, obvious extension mechanism is the great strength of the ST.

The caveats

The downside of the ST is that for large datasets, the creation and destruction of all the small anonymous arrays can itself prove to be fairly expensive of time and memory.

It's also the case that calling back into Perl code via the sort comparison block imposes a significant time cost. (Don't worry about this for the simple cases of sort{ $a <=> } ... and sort{ $b cmp $a } shown above, as Perl has optimisations to recognise these simple cases, and it doesn't actually do a callback at all).

But for more complex sorting, these callbacks are expensive, and if you can avoid them, it will save some considerable time.

The GRT

This is where the (Guttman Rosler Transform) comes in. The basic idea here is that instead of building an anonymous array in the pre-processing stage, we prepend the field data to the front of the string; sort the strings; then strip the bit we stuck on the front back off.

Going back to the simpler example of a field sort above, we can construct the GRT as follows:

A cheat

This is actually something of a cheat of a GRT. It only works because in my carefully constructed example data, the numeric fields all have the same number of digits, with smaller values having leading zeros. Whilst this works fine, and makes the algorithm easy to follow (try adding print statements inside the code blocks to follow what is going on), real life data is rarely so accommodating. Whilst you could use sprintf to add leading zeros and so make your ascii encoded numeric fields sort correctly using the string comparisons (cmp), it turns out that there is a better (faster) way.

That way is to use pack to convert the numeric fields into binary values that will sort correctly using a string comparison function. It is convenient that binary encode integers (NOTE:In 'network' format only. That is 'N'&'n' *NOT* 'V'&'v') will sort correctly using a string comparison function. As will floating point data in IEEE formats (the format used internally by Perl on most but not all systems).

That may not look so very different from the previous version, but the big advantage is that it will handle any integer values correctly, whereas with the previous version, you need to know how big the largest numeric field is, so that you can add the correct number of leading zeroes.

It also trivially extends to handle the two fields example shown above for the ST:

The pattern

Hopefully, the pattern is becoming obvious. Use pack with appropriate templates 'N', 'n', 'd' etc. to prepend the sort field data to the strings in the order or precedence. Then use unpack and the same template as you used for the sort fields, but wrapped in 'x[...]' to strip the prefix off and recover the original elements.

I have never made this into a module but normally just copy it into the script

Your post was a great addition to the sorting tutorial. So it pains me to point out Sort::Naturally

-xdg

Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

Hi
Liked the tutorial. One additional thing that you might want to reference at this point

How sort actually works internally doesn't really matter. That it is about as fast as you are likely to get; that it is built-in to language so always available; and that it understands Perl internals; does.

is that the internals of the Perl sort changed from quicksort to a mergesort between 5.6 and 5.8.
Although the main change is probably to defend against worst-case O(N**2) behaviour, the user can be bitten by this "feature"
From CPAN Sort.pm

A stable sort means that for records that compare equal, the original input ordering is preserved. Mergesort is stable, quicksort is not.

This means if you use 5.6, and your comparator returns 0 for 2 items that are in fact different, the input order and the output order of those items may change. This behaviour burned me recently; others may want to know about it.

GRT requires that you stringify (serialize) the values of the list you want to sort. That's not always possible. Eg. if it's a list of objects ...

And even if it's just a list of references to some data structures, by serialization and deserialization you end up with copies of the data structures. Which may matter a lot if you keep other references (in)to the structures.

Okay shmem. That first sentence was always a mess, and correcting the grammer whilst leaving the structure intact didn't help much. Is this any better?

How sort actually works internally doesn't really matter. What does matter is that being built-in to the language, it understands Perl's internals, is always available and about as fast as you are likely to achieve.

s/the the/the/ should be s/the the/to the/. That I haven't noticed this before, having now read and re-read this dozens of times, does not surprise me. That many other pairs of eyes have looked right past this, does. Welcome to my world.

Corrections made. Thanks.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

How sort actually works internally doesn't really matter. What does matter is that being built-in to the language, it understands Perl's internals, is always available and about as fast as you are likely to achieve.

Nice. Now I've got it - the "..., does." was refering to the "doesn't matter" from the previous sentence.
Anyways - it's way clearer now.

That I haven't noticed this before, having now read and re-read this dozens of times, does not surprise me. That many other pairs of eyes have looked right past this, does. Welcome to my world.

clap, clap, thank you - having stuttered as a child I just stumble over repeated words. And, good you showed me the "...,does." construct again ;-)