Does anyone know of an algorithm that will visit each element of a collection, lets say an array, once only but in a pseudo random manner?

My array has potentially many many elements (i.e. 2^28) so storing which elements that have been visited nor generating a random index sequence are viable options.

I am hoping that there is an algorithm, given a random seed will and the current loop count will give the next element index with the constraint that index returned is unique and thus will achieve coverage over all elements when the loop count == the number of elements in the array.

Well, this method isn't completely random, but it might work for what you need.

1) Ensure that your list's length is a multiple of X (Some value), padding it if you need to.2) Randomly select an index point in your array. This is your start index.3) Randomly select an iteration direct (boolean up/down) and a move direction (boolean up/down).3) Iterate through your list from your start index, jumping X elements in your iteration direction each time, wrapping around start/end when you get there.4) When you get your start index, move once in your move direction unless you've moved X times, then go back to 3.

Inspired by your solution, i have come up with a solution that seems to work for odd numbered elements...

1. pick a random positive integer R12. create a step interval I by multiplying the next whole integer after dividing the number of elements by 2 and then multiplying that by R1 and then add 1. i.e. R1 = ((int) (Math.max(((double)array.length)/2))*R1 + 13. pick a random index as the start point R24. set current index to R2. i.e. index = R25. the next index will be: index = (index +1 + I ) % array.length;6. repeat step 5 until all elements visited.

If you really don't want to make a copy of the collection, you can add a layer of indirection.

Create an int[collection.size()] filled with the values: {0,1,2,...,collection.size()-1}. Then shuffle that. Then iterate over your int[] and use the values (indices) to lookup elements in the original collection.

Making a copy of the original collection and shuffling that is a lot easier though.

Hi, appreciate more people! Σ ♥ = ¾Learn how to award medals... and work your way up the social rankings!

I am hoping that there is an algorithm, given a random seed will and the current loop count will give the next element index with the constraint that index returned is unique and thus will achieve coverage over all elements when the loop count == the number of elements in the array.

Impossible. It would require a RNG with perfect distribution equal to its period, which would mean it would be the opposite of random, as each output would depend on all previous ones.

Quickest solution I can think of off the top of my head is walk down the array once and swap each item with a randomly selected one (this may in fact be the algorithm for shuffle, but you can write it by hand for arrays)

Inspired by your solution, i have come up with a solution that seems to work for odd numbered elements...

1. pick a random positive integer R12. create a step interval I by multiplying the next whole integer after dividing the number of elements by 2 and then multiplying that by R1 and then add 1. i.e. R1 = ((int) (Math.max(((double)array.length)/2))*R1 + 13. pick a random index as the start point R24. set current index to R2. i.e. index = R25. the next index will be: index = (index +1 + I ) % array.length;6. repeat step 5 until all elements visited.

now to see whether i can find one for even numbers.

Even looking at this method, I don't believe that it would work. By that, besides the fact that at least one of your method calls listed is incorrect (Math.max(double value)?) Anyway, about 10% of the time when I try this algorithm (If I have it written correctly, hah!) I end up with it repeating the same element each time.

What do you really what to do here? This sounds likes you have a broken problem statement to me.

To answer you question as posed...use a quasi-random number generator. The simplest would be to rearrange the bits of the index in fixed pattern. This is easiest when the size in insured to be power-of-two...otherwise it gets trickier.

To answer you question as posed...use a quasi-random number generator. The simplest would be to rearrange the bits of the index in fixed pattern. This is easiest when the size in insured to be power-of-two...otherwise it gets trickier.

I was thinking among similar lines: just XOR-ing your POT index. But there will be noticable patterns.

Hi, appreciate more people! Σ ♥ = ¾Learn how to award medals... and work your way up the social rankings!

Yes...if you need need more random than that..you have to shuffle bits around.

Along the same lines of your suggestion...assuming that the 'in-order' version has no meaning, you could do it two passes. First walk the data in-order. Second pass shuffles elements within a bucket size. Every other set of passes, you offset the bucket boundaries by 1/2 the bucket size, so elements can eventually move anywhere in the array.

(EDIT: shuffling in buckets is to take advantage of caching BTW)

EDIT 2: For large arrays, buckets makes much more sense...collapse the two passes into one, working inside a bucket, and for slightly added randomness the starting point could be chosen by a pseudo-RNG. But this is all speculation since we don't know the real problem statement.

wow, thanks for all the interest. I will have to take time to digest it all

I shall elaborate why I need to perform random iteration

I have a fixed length input bit stream over which i run an algorithm to find the smallest unique symbol length S (in bits) that when the stream is split in to S bit symbols no symbol is repeated. This S value is used later on in other algorithms.

However these algorithms need S to be 64 bits or less. Normally the bit stream is a compressed data stream so S is < 64 bit but it can occur that uncompressed streams, or streams with uncompressed portions are input. In that case S is likely to be > 64. So to help reduce the impact of such streams I want to pre-process the input stream to reduce the apparent large string of repeated bits.

This pre-processing is envisioned to be a simple XOR for each S bit blocks of the input stream with a random iteration of an imaginary array of sequential long integers. Imaginary as the index into the array is the same as the value in the array. This process is then reversed at a later stage.

The sequential imaginary array is to be used so that the original input stream bit probabilities are not affected by the pre-processing.

My hunch seems correct, if the interval is a prime number then a modified version of the algorithm I stated seems to work for all the testing I have performed. I will still look at the other alternatives suggested to see which gives the best speed / memory usage / randomness compromise.

Well if randomness is really important, then you can use a linear congruent generator..the trick is that you have to compute the constants each time the length changes, which boils down to performing an extended GCD. Generating the sequence of values would require a modulo...but walking random memory should hide the cost of the divide.

The period of an RNG (which includes a LCG) is the interval for which it doesn't repeat its entire sequence of pseudorandom numbers. There's no way it keeps from repeating the same individual number in its codomain, because that would be the complete opposite of random. It'd be like a coin toss that always alternated heads and tails.

Yeesh, just walk the list and swap randomly. If you need to keep the original list, swap the indexes. Or do what the database kids do and sort by random() (but that'll hurt doing it with 260+ million elements unless you go mapreduce)

My hunch seems correct, if the interval is a prime number then a modified version of the algorithm I stated seems to work for all the testing I have performed. I will still look at the other alternatives suggested to see which gives the best speed / memory usage / randomness compromise.

The interval and length of array have to be relatively prime.

So if the interval is prime, then it also has to not be a factor of the array length. An easy way would be to make the array length a power of two then use any prime number >= 3 as the interval.

Don't worry, I will be performing some benchmarking to see whether the iterate and swap is acceptable and that applies to all suggestions

Perhaps a little more clarification can help understand the problem... In my pervious post I defined S as symbol size in bits. Since the bit steam split into S sized chunks. The count of chunks is going to be less than 2^S. the preprocessing array (imaginary or not) will still need to be of 2^S length even if most elements are not used. So having that array is a bit wasteful and potentially slower due to random memory access than a mor computationally expensive algorithm that generates a non repeating random sequence of indecies into an imaginary array.

From this simple benchmark, for an array size of 2^20, the PIS method seems to be about 5x faster than the Bucket method when utilising the random indecies sequence only as i will ultimately be doing for my situation.

@UprightPath: can you give a little more info about the constructor parameters of your implementation? I would love to add it to the benchmark but am unsure what values to put in for the "range" and "rangeStart" parameters.

collection: The List in question, instead of writing it as a purely array based, I decided to use lists. Anyway, it'd be fairly easy to change it out.rangeStart: The index at which you'd like to start the random iteration.range: The number of elements, from rangeStart to rangeStart + range, you'd like to iterate through.step: This is just some number that is a factor of range.

/** Set the length of the sequence. Must be a power-of-two. */publicvoidsetLength(intlength) {mask = length-1;reset(); }

/** Create a new permutation of the current length. */publicvoidreset() {// this is all overkillm = (rng.nextInt()<<3)|5; // anything such that (m & 7)==5 a = (rng.nextInt()<<1)|1; // any odd valuev = (rng.nextInt()); // basically sets the offset that zero will appear }

@UprightPath: thanks for the definitions Unfortunately I think I might have stuffed up integrating your code (i get infinite recursion) ... Can you have a look over my extensions to see what i did incorrectly? My main changes was always use the collection size /2 as the step. in cases where the collection size is not even, i assume that the collection size is +1 and perform a check to see whether the next "random" index is >= the collection size, if so then ignore it and get the next "random" index. The range is always the size of the Collection (or +1 if odd). The code is included in the above link.

I looked at what you had and solved the issues that you were having (My original code was written with the intent to be able to pick a subset of a list to perform the function on, rather than always performing it on the whole collection, fixing that...)

java-gaming.org is not responsible for the content posted by its members, including references to external websites,
and other references that may or may not have a relation with our primarily
gaming and game production oriented community.
inquiries and complaints can be sent via email to the info‑account of the
company managing the website of java‑gaming.org