For Method -> "Congruential" the page notes that this generates low-quality randomness. What is known about the others? I came across a paper today (available here and here) showing that Method -> "Rule30CA" has an extremely small effective size, suggesting that it is probably unsuitable for simulation work. I was a little concerned on reading this, since there was nothing in the help that suggested it should not be used.

Which of the others are known to be good? In particular, is the default Method -> "ExtendedCA" known to pass such tests as BigCrush or Dieharder? I know of results for the Mersenne Twister, of course (does well on all but the linear complexity tests).

Reading a bit through the paper szabolcs refers to tells me that CA30 certainly isn't of low quality. It passes all 23 tests in the Diehard program. However, there seem to be better rules.
–
Sjoerd C. de VriesMar 19 '12 at 23:46

To learn about issues with the legacy Random numbers check MathGroup. Search for "Andrzej Kozlowski Random" and I think you will see those issues explained and discussed.
–
Ted ErsekMar 19 '12 at 23:58

3 Answers
3

Well, regarding your question about ExtendedCA, the doc does not include a single reference to academic literature, so you have to take their word for it:

The cellular automaton used by "ExtendedCA" produces an extremely high level of randomness. It is so high that even using every single cell in output will give a stream of bits that passes many randomness tests, in spite of the obvious correlation between one cell and five previous ones.

I had, in the past, looked at published literature on people testing this PRNG, but could never find any reference to it being heavily tested (and reported), so I decided against using it for my own work.

I finally found some time to investigate this. I think it warrants a detailed response. In places I will repeat what others have pointed out, but I wanted something that ties together the various threads as best I can discern them.

I'm not certain what is meant by the Rule 30 RNG having "an extremely small effective size". Possibly it refers to taking bits only from the middle column? That is in fact what it does, or at least did the last time I looked at the implementation code. That is relatively less speed efficient than other RNGs but does give pseudorandom sequences of high quality. More on that in a bit.

I see the claim: "ExtendedCA: What is this? Apparently hasn't been tested." The documentation states

The cellular automaton used by "ExtendedCA" produces an extremely high level of randomness. It is so high that even using every single cell in output will give a stream of bits that passes many randomness tests, in spite of the obvious correlation between one cell and five previous ones.

I will give a bit more detail on this matter. In house testing has shown ExtendedCA passing all Diehard and BigCrush tests. One test has a p-value around .993, with all others in the range .01-.99. Moreover some correlation testing has been done that is outside of the tests in TestU01 (the Crush suite).

I have not yet managed to locate a copy of that first article. I did find some explanation of it in a more recent paper by Lacharme, Martin, and Sole (see refs below). I gather the issue is strictly of cryptographic usage, where (un)predictability is more important than (pseudo)randomness. They indicate that it may be possible to reconstruct part of the initial configuration given the stream of middle column bits. While this is out of my area, I will concede that this might be problematic for that type of use. That said, let me also remark that the initial configuration part reconstructed is from the left half set of columns. If one has a look at the Rule 30 output from a very specific one bit initial cell, as seen in the "Structure and properties" section in Rule 30's Wikipedia article, one will observe considerably more regularity on the left side than the right. This leads me to suspect that knowing the leftmost initial bits will not be of general help in full reconstruction of the middle column. But again, I'm no expert on this.

Pseudorandomness is an entirely different matter. As I mentioned above, Rule 30 actually tests quite well in this regard. So let me take up the matters under discussion in Sipper & Tomassini (1996). It is important to understand what exactly is their claim. They tested using not just the center column, but all columns, of the various RNGs. Used in this way, Rule 30 is indeed quite bad. That's why it does not get used in this way. I note that the authors point out that this is the actual use (first paragraph of section 2). They also state quite clearly (section 4):

The relatively low results obtained by the rule 30 CA may be due to the fact that we considered N random sequences generated in parallel, rather than the single one considered by Wolfram.

The point, I think, was not that they were claiming Rule 30 generator is bad when properly used, but rather that they were able to find ones that operate more efficiently in terms of how many bits can be used per iteration.

As mentioned in another response and comments thereto, they then (section 5) state (emphasis mine):

It seems safe to say at this point that our co-evolved generators are at least as good as the best available CA randomizers.

Here is the point. They tested against the suite by Knuth which, at the time of that writing, was "best practice". The Marsaglia Diehard test suite showed up also in 1996 so it may be no surprise they were not aware of it. The l'Ecuyer Crush tests were not around for several more years. Later analysis, as indicated in the paper by Seredynski, Bouvry, and Zumaya (linked to by a comment to this query, see below for another link), indicates that a related evolved CA-based generator, by Tomassini and Perrenoud (see below for ref), does not fare terribly well on Diehard. While I do not know if this is also the case for the Sipper and Tomassini RNG, I will speculate that the Sipper/Tomassini generator would be no better, and perhaps worse, since (going by dates of publication) it was very likely developed a few years earlier. The upshot is that the Rule 30 and ExtendedCA RNGs are, by the standards of current testing, quite sound. Some of the others that showed up in the earlier literature apparently fall short.

Some other pseudorandom remarks.

(From comments)

[T]he ExtendedCA generator is just a simplified version of CA30 so is likely to suffer from the same problems."

It may have similar problems but it is not a simplified version of Rule 30. It uses a neighborhood of five noncontiguous cells (this is indicated in the first paragraph describing it from the documention I referenced above).

Reading about the evolution methods used to construct nonuniform CA-based RNGs leads me to think they should not of necessity work well. That is to say, they might, but one has to be lucky. Here is my reasoning. First, as best I can tell, they select based on an entropy condition that will be satisfied even better by a low discrepancy sequence generator. Such sequences have excellent qualities e.g. for quasi-Monte-Carlo integration. But they are quite far from random. Second is that the winners of each "round" (in the sense of evolutionary methods) are heavily dependent on what I'll call "the kindness of neighbors". So there is no compelling reason to believe that a local rule that seemingly works well will continue to do so if its neighbors change. This is a general problem with evolutionary methods that have such linkage between genes or their equivalent units. Such methods do in practice still often give good results, due to the effect of co-evolution amongst neighbors. But still...

(From comments):

Now remember that Rule 30 was introduced for the purpose of crypto, so this is a reasonable avenue to check.

I cannot myself give an accurate accounting of the historical origins of Rule 30 applications. For what it is worth, here is what Stephen Wolfram writes in his 2002 NKS tome.

"I originally studied rule 30 in the context of basic science, but I soon realized that it could serve as the basis for practical random sequence generation and cryptography, and I analyzed this extensively in 1985. (Most but not all of the results from my original paper are included in this book, together with various new results.) In 1985 and soon thereafter a number of people (notably Richard and Carl Feynman) tried to cryptanalyze rule 30, but without success. From the beginning, computations of spacetime entropies for rule 30 (see page 960) gave indications that for strong cryptography one should not sample all cells in a column, and in 1991 Willi Meier and Othmar Staffelbach described essentially the explicit cryptanalysis approach shown on page 601. Rule 30 has been widely used for random sequence generation, but for a variety of reasons I have not in the past much emphasized its applications in cryptography."

I hope this is of some use for understanding the qualities of the CA-based RNGs in Mathematica.

I don't have much time right now to write a longer post (and anyway, this is not really an area in which I would trust myself as an "expert") but I think it is better not to use the words "high quality" and "low quality" too loosely. Essentially, when dealing with quasi-random number generators there are at least two senses in which these terms are used, and they are not equivalent (at least, it is not clear to me that they are equivalent).
The first sense is statistical. Basically, a RNG is of "high quality" in this sense, if random sequences generated by it pass a variety of tests for "randomness". These tests use only statistical methods, which do not involve knowing anything about how the sequence was generated.
I believe that in this sense all the generators used by Mathematica 8 are of "high quality". In particular, I have found the Rule 30 generator very suitable for simulation in finance, and I think it has a record of success in passing a large number of statistical tests, that many other generators fail (e.g. Marsaglia's "Diehard" test suite). In Mathematica before version 6 the Rule 30 algorithm was used to generate random integers but a Linear Congruential generator (Marsaglia-Zaman) was used to generate random reals. The Linear Congruential generator actually failed one of Marsaglia's tests and instances of poor quality randomness started to appear in MathGroup posts at about the same time. Unfortunately before version 6 of Mathematica it was impossible to modify random number generation without incurring a huge performance penalty.
This is why in version 6 the entire approach to random number generation was changed. Not only are the generators of high statistical quality, but it is quite easy to define your own generator (if you have one) which will work comparably to built-in ones.

There is also another meaning of "high quality" about which I know much less, which involves cryptography. Basically this is not about the statistical quality of randomness but about guessing or discovering the key that generates these pseudo random sequences. I may be wrong, but I think, in the case of cellular generators you actually use the knowledge that the generator is a cellular automaton. It then turns out that the Rule 30 generator is not as secure as used to be thought, in other words, one can discover the key too easily for comfort. But it is not clear to me that this has any relevance to the usefulness of this generator for simulation.

I have to repeat again, this is not really my subject so some of the things I have written may be wrong. I am a mathematician (originally a "pure" one) who about 10 years ago became interested in using probability theory in mathematical finance. This is why I got interested in this strange subject. The reason why I call it "strange" is because very little seems actually proved in the strict mathematical sense in it. Of course, there are some random generators, such as the linear congruential ones, about which everything that is known is fully "proved" but as far as I know, the cellular automata ones are not of this kind. Most or perhaps all of the knowledge, including these questions of "quality" is "experimental", which means that, like most experimental knowledge, it can change radically at any moment. However, I have not yet heard of anyone having discovered any problems with simulation performed with Mathematica 6 and later. Of course, if you come across them you should write to the MathGroup. As I mentioned above, you can also define your own generators. I don't have the time right know to explain how it is done, but a full explanation is included in the Mathematica documentation.

I'm only talking about the first sense, non-cryptographic quality. I very much disagree that Rule30CA is high-quality in this sense. But my real interest is in the default method's quality. I haven't seen an analysis of it so I don't know what to believe.
–
CharlesMar 20 '12 at 13:02

Well, this is not a matter of "opinions" I think, but of evidence. If you do have evidence, please post it to the MathGroup. I have used Rule 30 CA a lot for computing option prices by Monte Carlo methods and in cases where analytic formulas are known I have always found quite fast convergence to the correct value. I have also never heard of anyone getting any bad answers. As for the default method: its advantage over Rule 30 is speed. I think Rule 30 has been better tested and I have seen people switching to Rule 30 because it's high quality is more seen as better documented.
–
Andrzej KozlowskiMar 20 '12 at 13:11

The first source you have posted is concerned only with cryptographic quality. The second source (Sipper and Tomassini) keeps referring to the high quality of Wolfram's Rule 30 generator, e.g. "The first work examining the application of CAs to random number generation is that of Wolfram, in which Rule 30 is extensively studied, demonstrating its ability to produce highly random temporal bit sequences". The entire point of the paper is to demonstrate the existence of non-uniform CA based generators which are at least as good as CA 30. Note "at least as good" not "as bad".
–
Andrzej KozlowskiMar 20 '12 at 14:53

Sipper & Tomassini say that their method is "markedly improved in comparison to the rule 30 CA" (and somewhat better than rules 90 and 150), noting Rule 30's poor performance on chi-square and serial correlation. On Table 1 they give the performance for Rule 30; the number of tests is very small but I would sooner describe the results as "abysmal" than "high quality". 50% success rate for chi-square under procedure (a)?
–
CharlesMar 20 '12 at 16:50

Mathematica is a registered trademark of Wolfram Research, Inc. While the mark is used herein with the limited permission of Wolfram Research, Stack Exchange and this site disclaim all affiliation therewith.