NAME

List::Gen - provides functions for generating lists

VERSION

version 0.974

SYNOPSIS

this module provides higher order functions, list comprehensions, generators, iterators, and other utility functions for working with lists. walk lists with any step size you want, create lazy ranges and arrays with a map like syntax that generate values on demand. there are several other hopefully useful functions, and all functions from List::Util are available.

FUNCTIONS

mapn {CODE} NUM LIST

this function works like the builtin map but takes NUM sized steps over the list, rather than one element at a time. inside the CODE block, the current slice is in @_ and $_ is set to $_[0] . slice elements are aliases to the original list. if mapn is called in void context, the CODE block will be executed in void context for efficiency.

zip takes a list of array references and generators. it interleaves the elements of the passed in sequences to create a new list. zip continues until the end of the shortest sequence. LIST can be any combination of array references and generators.

interleaves the passed in lists to create a new list. zipmax continues until the end of the longest list, undef is returned for missing elements of shorter lists. LIST can be any combination of array references and generators.

note that this method of constructing an array ref from a list is roughly 40% faster than [ LIST ], but with the caveat and feature that elements are aliases to the original list

&\(LIST)

a synonym for cap , the symbols &\(...) will perform the same action. it could be read as taking the subroutine style reference of a list. like all symbol variables, once imported, &\ is global across all packages.

generators

in this document, a generator is an object similar to an array that generates its elements on demand. generators can be used as iterators in perl's list control structures such as for/foreach and while . generators, like programmers, are lazy. unless they have to, they will not calculate or store anything. this laziness allows infinite generators to be created. you can choose to explicitly cache a generator, and several generators have implicit caches for efficiency.

there are source generators, which can be numeric ranges, arrays, or iterative subroutines. these can then be modified by wrapping each element with a subroutine, filtering elements, or combining generators with other generators. all of this behavior is lazy, only resolving generator elements at the latest possible time.

all generator functions return a blessed and overloaded reference to a tied array. this may sound a bit magical, but it just means that you can access the generator in a variety of ways, all which remain lazy.

there is also a user space subroutine named &last that is installed into the calling namespace during the execution of the loop. calling it without arguments has the same function as the builtin last . calling it with an argument will still end the looping construct, but will also cause the loop to return the argument. the done ... exception also works the same way as &last(...)

$gen->range # range(0, $gen->size - 1)
$gen->keys # same as $gen->range, but a list in list context
$gen->values # same as $gen, but a list in list context
$gen->kv # zip($gen->range, $gen)
$gen->pairs # same as ->kv, but each pair is a tuple (array ref)

the zip and the cross methods all use the comma operator ( ',' ) by default to join their arguments. if the first argument to any of these methods is code or a code like string, that will be used to join the arguments. more detail in the overloaded operators section below

$gen->leaves # returns a coderef iterator that will perform a depth first
# traversal of the edge nodes in a tree of nested generators.
# a full run of the iterator will ->reset all of the internal
# generators

the methods duplicate and extend the tied functionality and are necessary when working with indices outside of perl's array limit (0 .. 2**31 - 1) or when fetching a list return value (perl clamps the return to a scalar with the array syntax). in all cases, they are also faster than the tied interface.

the methods that do not have a useful return value, such as ->say, return the same generator they were called with. this lets you easily insert these methods at any point in a method chain for debugging.

if the first argument to a ->zip or ->cross method is not an array or generator, it is assumed to be a subroutine and the corresponding ->(zip|cross)with method is called:

$gen1->zipwith('+', $gen2) ~~ $gen1->zip('+', $gen2);

hyper operators:

not quite as elegant as perl6's hyper operators, but the same idea. these are similar to zipwith but with more control over the length of the returned generator. all of perl's non-mutating binary operators are available to use as strings, or you can use a subroutine.

in the last example, a bare string is the final element, and precedence rules keep everything working. however, if you want to use a non generator as the first element, a few parens are needed to force the evaluation properly:

my $y = 'y' <<('.'>> <1..>);
$y->say(5); # 'y1 y2 y3 y4 y5'

otherwise 'y' << '.' will run first without overloading, which will be an error. since that is a bit awkward, where you can specify an operator string, you can prefix R or r to indicate that the arguments to the operator should be reversed.

my $y = <1..> <<'R.'>> 'y';
$y->say(5); # 'y1 y2 y3 y4 y5'

just like in perl6, hyper operators are recursively defined for multi dimensional generators.

mutable generators

mutable generators (those returned from mutable, filter, While, Until, and iterate_multi) are generators with variable length. in addition to all normal methods, mutable generators have the following methods:

$gen->when_done(sub {...}) # schedule a method to be called when the
# generator is exhausted
# when_done can be called multiple times to
# schedule multiple end actions
$gen->apply; # causes the generator to evaluate all of its elements in
# order to find out its true size. it is a bad idea to call
# ->apply on an infinite generator

due to the way perl processes list operations, when perl sees an expression like:

print "@$gen\n"; # or
print join ' ' => @$gen;

it calls the internal FETCHSIZE method only once, before it starts getting elements from the array. this is fine for immutable generators. however, since mutable generators do not know their true size, perl will think the array is bigger than it really is, and will most likely run off the end of the list, returning many undefined elements, or throwing an exception.

the solution to this is to call $gen->apply first, or to use the $gen->all method with mutable generators instead of @$gen , since the ->all method understands how to deal with arrays that can change size while being read.

perl's for/foreach loop is a bit smarter, so just like immutable generators, the mutable ones can be dereferenced as the loop argument with no problem:

... foreach @$mutable_generator; # works fine

stream generators

the generators filter, scan, and iterate (all of its flavors) have internal caches that allow random access within the generator. some algorithms only need monotonically increasing access to the generator (all access via repeated calls to $gen->next for example), and the cache could become a performance/memory problem.

the *_stream family of generators do not maintain an internal cache, and are subsequently unable to fulfill requests for indicies lower than or equal to the last accessed index. they will however be faster and use less memory than their non-stream counterparts when monotonically increasing access is all that an algorithm needs.

stream generators can be thought of as traditional subroutine iterators that also have generator methods. it is up to you to ensure that all operations and methods follow the monotonically increasing index rule. you can determine the current position of the stream iterator with the $gen->index method.

all threads are local to a particular generator, they are not shared. if the passed in generator was cached (at the top level) that cache is shared and used automatically. this includes most generators with implicit caches. threads_slice and threads_all can be called without starting the threads explicitly. in that case, they will start with default values.

the threaded methods only work in perl versions 5.10.1 to 5.12.x, patches to support other versions are welcome.

to specify an infinite range, you can pass range an infinite value ( 9**9**9 works well), or the glob ** , or the string '*'

range(1, 9**9**9) ~~ range(1, **) ~~ range(1, '*') ~~ <1..*> ~~ <1..>

ranges only store their endpoints, and ranges of all sizes take up the same amount of memory.

gen {CODE} GENERATOR

gen {CODE} ARRAYREF

gen {CODE} SIZE

gen {CODE} [START STOP [STEP]]

gen {CODE} GLOBSTRING

gen is the equivalent of map for generators. it returns a generator that will apply the CODE block to its source when accessed. gen takes a generator, array ref, glob-string, or suitable arguments for range as its source. with no arguments, gen uses the range 0 .. infinity .

gen provides the functionality of the identical ->gen(...) and ->map(...) methods.

note that while effort has gone into making generators as fast as possible there is overhead involved with lazy generation. simply replacing all calls to map with gen will almost certainly slow down your code. use these functions in situations where the time / memory required to completely generate the list is unacceptable.

gen and other similarly argumented functions in this package can also accept a string suitable for the <glob> syntax:

makegen converts an array to a generator. this is normally not needed as most generator functions will call it automatically if passed an array reference

makegen considers the length of ARRAY to be immutable. changing the length of an array after passing it to makegen (or to gen and like argumented subroutines) will result in undefined behavior. this is done for performance reasons. if you need a length mutable array, use the array function. changing the value of a cell in the array is fine, and will be picked up by a generator (of course if the generator uses a cache, the value won't change after being cached).

you can assign to the generator returned by makegen , provided the assignment does not lengthen the array.

list converts a list to a generator. it is a thin wrapper around makegen that simply passes its @_ to makegen . that means the values in the returned generator are aliases to list's arguments.

list(2, 5, 8, 11)->map('*2')->say; # '4 10 16 22'

is the same as writing:

(gen {$_*2} cap 2, 5, 8, 11)->say;

in the above example, list can be used in place of cap and has exactly the same functionality:

(gen {$_*2} list 2, 5, 8, 11)->say;

array [ARRAY]

array is similar to makegen except the array is considered a mutable data source. because of this, certain optimizations are not possible, and the generator returned will be a bit slower than the one created by makegen in most conditions (increasing as generator functions are stacked).

it is ok to modify ARRAY after creating the generator. it is also possible to use normal array modification functions such as push , pop , shift , unshift , and splice on the generator. all changes will translate back to the source array.

you can think of array as converting an array to an array reference that is also a generator.

an infinite generator that returns SCALAR for every position. it is equivalent to gen {SCALAR} but a little faster.

iterate {CODE} [LIMIT|GENERATOR]

iterate returns a generator that is created iteratively. iterate implicitly caches its values, this allows random access normally not possible with an iterative algorithm. LIMIT is an optional number of times to iterate. normally, inside the CODE block, $_ is set to the current iteration number. if passed a generator instead of a limit, $_ will be set to sequential values from that generator.

generators produced by iterate have an extra method, ->from(LIST). the method must be called before values are accessed from the generator. the passed LIST will be the first values returned by the generator. the method also changes the behavior of $_ inside the block. $_ will contain the previous value generated by the iterator. this allows iterate to behave the same way as the like named haskell function.

iterate_stream is a version of iterate that does not cache the generated values. because of this, access to the returned generator must be monotonically increasing (such as repeated calls to $gen->next).

iterate_multi {CODE} [LIMIT]

the same as iterate, except CODE can return a list of any size. inside CODE, $_ is set to the position in the returned generator where the block's returned list will be placed.

the returned generator from iterate_multi can be modified with push, pop, shift, unshift, and splice like a normal array. it is up to you to ensure that the iterative algorithm will still work after modifying the array.

the ->from(...) method can be called on the returned generator. see iterate for the rules and effects of this.

iterate_multi_stream {CODE} [LIMIT]

iterate_multi_stream is a version of iterate_multi that does not cache the generated values. because of this, access to the returned generator must be monotonically increasing (such as repeated calls to $gen->next).

keyword modification of a stream iterator (with push, shift, ...) is not supported.

gather {CODE} [LIMIT]

gather returns a generator that is created iteratively. rather than returning a value, you call take($return_value) within the CODE block. note that since perl5 does not have continuations, take(...) does not pause execution of the block. rather, it stores the return value, the block finishes, and then the generator returns the stored value.

you can not import the take(...) function from this module. take(...) will be installed automatically into your namespace during the execution of the CODE block. because of this, you must always call take(...) with parenthesis. take returns its argument unchanged.

gather implicitly caches its values, this allows random access normally not possible with an iterative algorithm. the algorithm in iterate is a bit cleaner here, but gather is slower than iterate , so benchmark if speed is a concern

a non-cached version gather_stream is also available, see iterate_stream

gather_multi {CODE} [LIMIT]

the same as gather except you can take(...) multiple times, and each can take a list. gather_multi_stream is also available.

stream {CODE}

in the CODE block, calls to functions or methods with stream versions will be replaced by those versions. this applies also to functions that are called internally by List::Gen (such as in the glob syntax). stream returns what CODE returns.

placing code inside a stream block is exactly the same as placing local $List::Gen::STREAM = 1; at the top of a block.

glob STRING

<list comprehension>

by default, this module overrides perl's default glob function. this is because the glob function provides the behavior of the angle bracket delimited <*.ext> operator, which is a nice place for inserting list comprehensions into perl's syntax. the override causes glob() and the <*.ext> operator to have a few special cases overridden, but any case that is not overridden will be passed to perl's internal glob function (my @files = <*.txt>; works as normal).

since the angle brackets (< and >) are used as delimiters of the glob string, they both must be escaped with \ if used in the <...> construct.

<1..10 if \< 5>->say; # 1 2 3 4

due to <...> being a qq{} string, in the code areas if you need to write $_ write it without the sigil as _

<1 .. 10 if _**2 \> 40>->say; # 7 8 9 10

it can be escaped \$_ as well.

neither of these issues apply to calling glob directly with a single quoted string:

glob('1..10 if $_ < 5')->say; # 1 2 3 4

List::Gen ...

the subroutine Gen in the package List:: is a dwimmy function that produces a generator from a variety of sources. since List::Gen is a fully qualified name, it is available from all packages without the need to import it.

if given only one argument, the following table describes what is done:

slice will perform some optimizations if it detects that RANGE_GEN is sufficiently simple (something like range $x, $y, 1 ). also, stacked simple slices will collapse into a single slice, which turns repeated tailing of a generator into a relatively efficient operation.

test attaches a code block to a generator. it takes arguments suitable for the gen function. accessing an element of the returned generator will call the code block first with the element in $_ , and if it returns true, the element is returned, otherwise an empty list (undef in scalar context) is returned.

when accessing a slice of a tested generator, if you use the ->(x .. y) syntax, the the empty lists will collapse and you may receive a shorter slice. an array dereference slice will always be the size you ask for, and will have undef in each failed slot

the $gen->nxt method is a version of $gen->next that continues to call ->next until a call returns a value, or the generator is exhausted. this makes the ->nxt method the easiest way to iterate over only the passing values of a tested generator.

cache {CODE}

cache GENERATOR

cache list => ...

cache will return a cached version of the generators returned by functions in this package. when passed a code reference, cache returns a memoized code ref (arguments joined with $; ). when in 'list' mode, the source is in list context, otherwise scalar context is used.

flip is reverse for generators. the ->apply method is called on GENERATOR . $gen->flip and $gen->reverse do the same thing.

flip gen {$_**2} 0, 10 ~~ gen {$_**2} 10, 0, -1

expand GENERATOR

expand SCALE GENERATOR

expand scales a generator with elements that return equal sized lists. it can be passed a list length, or will automatically determine it from the length of the list returned by the first element of the generator. expand implicitly caches its returned generator.

you can even use the ->code method to tersely define a factorial function:

*factorial = <[..*] 1, 1..>->code;
say factorial(5); # 120

a stream version scan_stream is also available.

overlay GENERATOR PAIRS

overlay allows you to replace the values of specific generator cells. to set the values, either pass the overlay constructor a list of pairs in the form index => value, ..., or assign values to the returned generator using normal array ref syntax

recursive defines a subroutine named self(...) or NAME(...) during generator execution. when called with no arguments it returns the generator. when called with one or more numeric arguments, it fetches those indices from the generator. when called with a generator, it returns a lazy slice from the source generator. since the subroutine created by recursive is installed at runtime, you must call the subroutine with parenthesis.

of course the fibonacci sequence is better written with the glob syntax as <0, 1, *+*...> which is compiled into something similar to the example with iterate above.

mutable generators

filter {CODE} [ARGS_FOR_GEN]

filter is a lazy version of grep which attaches a code block to a generator. it returns a generator that will test elements with the code block on demand. filter processes its argument list the same way gen does.

filter provides the functionality of the identical ->filter(...) and ->grep(...) methods.

normal generators, such as those produced by range or gen , have a fixed length, and that is used to allow random access within the range. however, there is no way to know how many elements will pass a filter. because of this, random access within the filter is not always O(1) . filter will attempt to be as lazy as possible, but to access the 10th element of a filter, the first 9 passing elements must be found first. depending on the coderef and the source, the filter may need to process significantly more elements from its source than just 10.

in addition, since filters don't know their true size, entire filter arrays do not expand to the correct number of elements in list context. to correct this, call the ->apply method which will test the filter on all of its source elements. after that, the filter will return a properly sized array. calling ->apply on an infinite (or very large) range wouldn't be a good idea. if you are using ->apply frequently, you should probably just be using grep . you can call ->apply on any stack of generator functions, it will start from the deepest filter and move up.

the method ->all will first call ->apply on itself and then return the complete list

note: filter now reads one element past the last element accessed, this allows filters to behave properly when dereferenced in a foreach loop (without having to call ->apply). if you prefer the old behavior, set $List::Gen::LOOKAHEAD = 0 or use filter_ ...

filter_stream {CODE} ...

as filter runs, it builds up a cache of the elements that pass the filter. this enables efficient random access in the returned generator. sometimes this caching behavior causes certain algorithms to use too much memory. filter_stream is a version of filter that does not maintain a cache.

normally, access to *_stream iterators must be monotonically increasing since their source can only produce values in one direction. filtering is a reversible algorithm, and subsequently filter streams are able to rewind themselves to any previous index. however, unlike filter , the filter_stream generator must test previously tested elements to rewind. things probably wont end well if the test code is non-deterministic or if the source values are changing.

when used as a method, it can be spelled $gen->filter_stream(...) or $gen->grep_stream(...)

While {CODE} GENERATOR

Until {CODE} GENERATOR

While / ->while(...) returns a new generator that will end when its passed in subroutine returns false. the until pair ends when the subroutine returns true.

if $List::Gen::LOOKAHEAD is true (the default), each reads one element past its requested element, and saves this value only until the next call for efficiency, no other values are saved. each supports random access, but is optimized for sequential access.

these functions have all of the caveats of filter , should be considered experimental, and may change in future versions. the generator returned should only be dereferenced in a foreach loop, otherwise, just like a filter perl will expand it to the wrong size.

the generator will return undef the first time an access is made and the check code indicates it is past the end.

the generator will throw an error if accessed beyond its dynamically found limit subsequent times.

mutable takes a single fixed size (immutable) generator, such as those produced by gen and converts it into a variable size (mutable) generator, such as those returned by filter .

as with filter, it is important to not use full array dereferencing ( @$gen ) with mutable generators, since perl will expand the generator to the wrong size. to access all of the elements, use the $gen->all method, or call $gen->apply before @$gen . using a slice @$gen[5 .. 10] is always ok, and does not require calling ->apply.

mutable generators respond to the List::Gen::Done exception, which can be produced with either done , done_if , or done_unless . when the exception is caught, it causes the generator to set its size, and it also triggers any ->when_done actions.

generators returned from mutable have a ->set_size(int) method that will set the generator's size and then trigger any ->when_done(sub{...}) methods.

done [LAST_RETURN_VALUE]

throws an exception that will be caught by a mutable generator indicating that the generator should set its size. if a value is passed to done, that will be the final value returned by the generator, otherwise, the final value will be the value returned on the previous call.

done_if COND VALUE

done_unless COND VALUE

these are convenience functions for throwing done exceptions. if the condition does not indicate done then the function returns VALUE

strict {CODE}

in the CODE block, calls to functions or methods are subject to the following localizations:

local $List::Gen::LOOKAHEAD = 0;

the functions filter , While and their various forms normally stay an element ahead of the last requested element so that an array dereference in a foreach loop ends properly. this localization disables this behavior, which might be needed for certain algorithms. it is therefore important to never write code like: for(@$strict_filtered){...} , instead write $strict_filtered->do(sub{...}) which is faster as well. the following code illustrates the difference in behavior:

in the dwim $gen->(...) code deref syntax, if $DWIM_CODE_STRINGS has been set to a true value, bare strings that look like code will be interpreted as code and passed to gen (string refs to filter ). since this behavior is fun for golf, but potentially error prone, it is off by default. strict turns it back off if it had been turned on.

strict returns what CODE returns. strict may have additional restrictions added to it in the future.

combining generators

sequence LIST

string generators, arrays, and scalars together.

sequence provides the functionality of the overloaded + operator on generators:

zipgen is a lazy version of zip . it takes any combination of generators and array refs and returns a generator. it is called automatically when zip is used in scalar context.

zipgen can be spelled genzip

unzip LIST

unzip is the opposite of zip src1, src2 . unzip returns 2 generators, the first returning src1, the second, src2. if LIST is a single element, and is a generator, that generator will be unzipped.

unzipn NUMBER LIST

unzipn is the n-dimentional precursor of unzip . assuming a zipped list produced by zip with n elements, unzip n list returns n lists corresponding to the lists originally passed to zip . if LIST is a single element, and is a generator, that generator will be unzipped. if only passed 1 argument, unzipn will return a curried version of itself:

this function works like the builtin map but consumes a list in pairs, rather than one element at a time. inside the CODE block, the variables $a and $b are aliased to the elements of the list. if mapab is called in void context, the CODE block will be executed in void context for efficiency. if mapab is passed an uneven length list, in the final iteration, $b will be undef

remove removes and returns elements from its source when CODE returns true. in the code block, if the source is an array, $_ is aliased to its elements. if the source is a hash, $_ is aliased to its keys (and a list of the removed key => value pairs are returned).

many of the functions in this package utilize closure objects to avoid the speed penalty of dereferencing fields in their object during each access. curse is similar to bless for these objects and while a blessing makes a reference into a member of an existing package, a curse conjures a new package to do the reference's bidding

Closure::Object is functionally equivalent to the following normal perl object, but with faster method calls since there are no hash lookups or other dereferences (around 40-50% faster for short getter/setter type methods)

the trade off is in creation time / memory, since any good curse requires drawing at least a few pentagrams in the blood of an innocent package.

the returned object is blessed into the conjured package, which inherits from the provided PACKAGE . always use $obj->isa(...) rather than ref $obj eq ... due to this. the conjured package name matches /${PACKAGE}::_\d+/

see List::Gen::Perl6 for a source filter that adds perl6's meta operators to use with generators, rather than the default overloaded operators

CAVEATS

version 0.90 added glob to the default export list (which gives you syntactic ranges <1 .. 10> and list comprehensions.). version 0.90 also adds many new features and bug-fixes, as usual, if anything is broken, please send in a bug report. the ending conditions of zip and zipgen have changed, see the documentation above. test has been removed from the default export list. setting $List::Gen::LIST true to enable list context generators is no longer supported and will now throw an error. list has been added to the default export list. genzip has been renamed zipgen

version 0.70 comes with a bunch of new features, if anything is broken, please let me know. see filter for a minor behavior change

versions 0.50 and 0.60 break some of the syntax from previous versions, for the better.

code generation

a number of the syntactic shortcuts that List::Gen provides will construct and then evaluate code behind the scenes. Normally this is transparent, but if you are trying to debug a problem, hidden code is never a good thing. You can lexically enable the printing of evaled code with:

a given code string is only evaluated once and is then cached, so you will not see any additional output when using the same code strings in multiple places. in some cases (like the iterate example above) the code is closing over external variables ( @pre and $fetch ) so you will not be able to see everything, but $SAY_EVAL should be a helpful debugging aid.

any time that code evaluation fails, an immediate fatal error is thrown. the value of $SAY_EVAL does not matter in that case.

captures of compile time constructed lists

the cap function and its twin operator &\ are faster than the [...] construct because they do not copy their arguments. this is why the elements of the captures remain aliased to their arguments. this is normally fine, but it has an interesting effect with compile time constructed constant lists:

the first example shows the expected speed increase due to not copying the values into a new empty array reference. the second example is much faster at runtime than the [...] syntax, but this speed is deceptive. the reason is that the list being passed in as an argument is generated by the compiler before runtime begins. so all perl has to do is place the values on the stack, and call the function.

normally this is fine, but there is one catch to be aware of, and that is that a capture of a compile time constant list in a loop or subroutine (or any structure that can execute the same segment of code repeatedly) will always return a reference to an array of the same elements.

AUTHOR

Eric Strom, <asg at cpan.org>

BUGS

overloading has gotten fairly complicated and is probably in need of a rewrite. if any edge cases do not work, please send in a bug report.

both threaded methods ($gen->threads_slice(...)) and function composition with overloaded operators (made with List::Gen::Lazy::fn {...}) do not work properly in versions of perl before 5.10. patches welcome

As a valued partner and proud supporter of MetaCPAN, StickerYou is
happy to offer a 10% discount on all Custom Stickers,
Business Labels, Roll Labels,
Vinyl Lettering or Custom Decals. StickerYou.com
is your one-stop shop to make your business stick.
Use code METACPAN10 at checkout to apply your discount.