README

NAME
Template::Benchmark - Pluggable benchmarker to cross-compare template
systems.
VERSION
version 1.09_02
SYNOPSIS
use Template::Benchmark;
my $bench = Template::Benchmark->new(
duration => 5,
template_repeats => 1,
array_loop => 1,
shared_memory_cache => 0,
);
my $result = $bench->benchmark();
if( $result->{ result } eq 'SUCCESS' )
{
...
}
DESCRIPTION
Template::Benchmark provides a pluggable framework for cross-comparing
performance of various template engines across a range of supported
features for each, grouped by caching methodology.
If that's a bit of a mouthful... have you ever wanted to find out the
relative performance of template modules that support expression parsing
when running with a shared memory cache? Do you even know which ones
*allow* you to do that? This module lets you find that sort of thing
out.
As of current writing, there are plugins to let you compare the
performance and features of 21 different perl template engines in a
total of 75 different configurations.
If you're just after results, then you should probably start with the
benchmark_template_engines script first, it provides a commandline UI
onto Template::Benchmark and gives you human-readable reports as a reply
rather than a raw hashref, it also supports JSON output if you want to
dump the report somewhere in a machine-readable format.
If you have no template engines already installed, or you want to
benchmark everything supported, I suggest you also look at the
Task::Template::Benchmark distribution which installs all the optional
requirements for Template::Benchmark.
IMPORTANT CONCEPTS AND TERMINOLOGY
Template Engines
Template::Benchmark is built around a plugin structure using
Module::Pluggable, it will look under "Template::Benchmark::Engines::*"
for *template engine* plugins.
Each of these plugins provides an interface to a different *template
engine* such as Template::Toolkit, HTML::Template, Template::Sandbox and
so on.
Cache Types
*Cache types* determine the source of the template and the caching
mechanic applied, currently there are the following *cache types*:
*uncached_string*, *uncached_disk*, *disk_cache*, *shared_memory_cache*,
*memory_cache* and *instance_reuse*.
For a full list, and for an explanation of what they represent, consult
the "Cache Types" in Template::Benchmark::Engine documentation.
Template Features
*Template features* are a list of features supported by the various
*template engines*, not all are implemented by all *engines* although
there's a core set of *features* supported by all *engines*.
*Features* can be things like *literal_text*, *records_loop*,
*scalar_variable*, *variable_expression* and so forth.
For a full list, and for an explanation of what they represent, consult
the "Template Features" in Template::Benchmark::Engine documentation.
Benchmark Functions
Each *template engine* plugin provides the means to produce a *benchmark
function* for each *cache type*.
The *benchmark function* is an anonymous sub that is expected to be
passed the template, and two hashrefs of template variables, and is
expected to return the output of the processed template.
These are the functions that will be benchmarked, and generally consist
(depending on the *template engine*) of a call to the template
constructor and template processing functions.
Each plugin can return several *benchmark functions* for a given *cache
type*, so each is given a tag to use as a name and a description for
display, this allows plugins like
Template::Benchmark::Engines::TemplateToolkit to contain benchmarks for
Template::Toolkit, Template::Toolkit running with Template::Stash::XS,
and various other options.
Each of these will run as an independent benchmark even though they're
provided by the same plugin.
Supported or Unsupported?
Throughout this document are references to whether a *template feature*
or *cache type* is supported or unsupported in the *template engine*.
But what constitutes "unsupported"?
It doesn't neccessarily mean that it's *impossible* to perform that task
with the given *template engine*, but generally if it requires some
significant chunk of DIY code or boilerplate or subclassing by the
developer using the *template engine*, it should be considered to be
*unsupported* by the *template engine* itself.
This of course is a subjective judgement, but a general rule of thumb is
that if you can tell the *template engine* to do it, it's supported; and
if the *template engine* allows *you* to do it, it's *unsupported*, even
though it's *possible*.
HOW Template::Benchmark WORKS
Construction
When a new Template::Benchmark object is constructed, it attempts to
load all *template engine* plugins it finds.
It then asks each plugin for a snippet of template to implement each
*template feature* requested. If a plugin provides no snippet then it is
assumed that that *feature* is unsupported by that *engine*.
Each snippet is then combined into a benchmark template for that
specific *template engine* and written to a temporary directory, at the
same time a cache directory is set up for that *engine*. These temporary
directories are cleaned up in the "DESTROY()" of the benchmark instance,
usually when you let it go out of scope.
Finally, each *engine* is asked to provide a list of *benchmark
functions* for each *cache type* along with a name and description
explaining what the *benchmark function* is doing.
At this point the Template::Benchmark constructor exits, and you're
ready to run the benchmarks.
Running the benchmarks
When the calling program is ready to run the benchmarks it calls
"$bench->benchmark()" and then twiddles its thumbs, probably for a long
time.
While this twiddling is going on, Template::Benchmark is busy running
each of the *benchmark functions* a single time.
The outputs of this initial run are compared and if there are any
mismatches then the "$bench->benchmark()" function exits early with a
result structure indicating the errors as compared to a reference copy
produced by the reference plugin engine.
An important side-effect of this initial run is that the cache for each
*benchmark function* becomes populated, so that the cached *cache types*
truly reflect only cached performance and not the cost of an initial
cache miss.
If all the outputs match then the *benchmark functions* for each *cache
type* are handed off to the Benchmark module for benchmarking.
The results of the benchmarks are bundled together and placed into the
results structure that is returned from "$bench->benchmark()".
OPTIONS
New <Template:Benchmark> objects can be created with the constructor
"Template::Benchmark->new( %options )", using any (or none) of the
options below.
uncached_string => *0* | *1* (default 1)
uncached_disk => *0* | *1* (default 1)
disk_cache => *0* | *1* (default 1)
shared_memory_cache => *0* | *1* (default 1)
memory_cache => *0* | *1* (default 1)
instance_reuse => *0* | *1* (default 1)
Each of these options determines which *cache types* are enabled (if
set to a true value) or disabled (if set to a false value). At least
one of them must be set to a true value for any benchmarks to be
run.
literal_text => *0* | *1* | *$n* (default 1)
scalar_variable => *0* | *1* | *$n* (default 1)
hash_variable_value => *0* | *1* | *$n* (default 0)
array_variable_value => *0* | *1* | *$n* (default 0)
deep_data_structure_value => *0* | *1* | *$n* (default 0)
array_loop_value => *0* | *1* | *$n* (default 0)
hash_loop_value => *0* | *1* | *$n* (default 0)
records_loop_value => *0* | *1* | *$n* (default 1)
array_loop_template => *0* | *1* | *$n* (default 0)
hash_loop_template => *0* | *1* | *$n* (default 0)
records_loop_template => *0* | *1* | *$n* (default 1)
constant_if_literal => *0* | *1* | *$n* (default 0)
variable_if_literal => *0* | *1* | *$n* (default 1)
constant_if_else_literal => *0* | *1* | *$n* (default 0)
variable_if_else_literal => *0* | *1* | *$n* (default 1)
constant_if_template => *0* | *1* | *$n* (default 0)
variable_if_template => *0* | *1* | *$n* (default 1)
constant_if_else_template => *0* | *1* | *$n* (default 0)
variable_if_else_template => *0* | *1* | *$n* (default 1)
constant_expression => *0* | *1* | *$n* (default 0)
variable_expression => *0* | *1* | *$n* (default 0)
complex_variable_expression => *0* | *1* | *$n* (default 0)
constant_function => *0* | *1* | *$n* (default 0)
variable_function => *0* | *1* | *$n* (default 0)
Each of these options sets the corresponding *template feature* on
or off. At least one of these must be true for any benchmarks to
run.
If any option is set to a positive integer greater than 1, it will
be used as a repeats value for that specific feature, indicating
that the feature should appear in the benchmark template that number
of times.
This stacks with any "template_repeats" constructor option, for
example if you supply "scalar_variable => 2" and "template_repeats
=> 30", you will have 60 "scalar_variable" sections in the benchmark
template.
Support for per-feature repeats values was added in 1.04.
features_from => *$plugin* (default none)
If set, then the list of *template features* will be drawn from
those supported by the given plugin.
If this option is set, any *template features* you supply as options
will be ignored and overwritten, and the plugin named will also be
benchmarked even if you attempted to exclude it with the
"skip_plugin" or "only_plugin" options.
You can supply multiple plugins to "features_from", either by
passing several "features_from" attributes, or by passing an
arrayref of plugin names, or a hashref of plugin names with
true/false values to toggle them on or off. The set of features
chosen will then be the largest common subset of features supported
by those engines.
# This sets the features to benchmark to all those supported by
# Template::Benchmark::Engines::TemplateSandbox
$bench = Template::Benchmark->new(
features_from => 'TemplateSandbox',
);
# This sets the features to benchmark to all those supported by BOTH
# Template::Benchmark::Engines::MojoTemplate and
# Template::Benchmark::Engines::HTMLTemplateCompiled
$bench = Template::Benchmark->new(
features_from => 'MojoTemplate',
features_from => 'HTMLTemplateCompiled',
);
# This sets the features to benchmark to all those supported by BOTH
# Template::Benchmark::Engines::MojoTemplate and
# Template::Benchmark::Engines::HTMLTemplateCompiled
$bench = Template::Benchmark->new(
features_from => {
MojoTemplate => 1,
HTMLTemplateCompiled => 1,
TemplateSandbox => 0,
},
);
Support for multiple "features_from" values was added in 1.09.
cache_types_from => *$plugin* (default none)
If set, then the list of *cache types* will be drawn from those
supported by the given plugin.
If this option is set, any *cache types* you supply as options will
be ignored and overwritten, and the plugin named will also be
benchmarked even if you attempted to exclude it with the
"skip_plugin" or "only_plugin" options.
You can supply multiple plugins to "cache_types_from", either by
passing several "cache_types_from" attributes, or by passing an
arrayref of plugin names, or a hashref of plugin names with
true/false values to toggle them on or off. The set of cache types
chosen will then be the superset of cache types supported by those
engines.
# This sets the cache types to benchmark to all those supported by
# Template::Benchmark::Engines::TemplateSandbox
$bench = Template::Benchmark->new(
cache_types_from => 'TemplateSandbox',
);
# This sets the cache types to benchmark to all those supported by EITHER of
# Template::Benchmark::Engines::MojoTemplate and
# Template::Benchmark::Engines::HTMLTemplateCompiled
$bench = Template::Benchmark->new(
cache_types_from => 'MojoTemplate',
cache_types_from => 'HTMLTemplateCompiled',
);
# This sets the features to benchmark to all those supported by EITHER of
# Template::Benchmark::Engines::MojoTemplate and
# Template::Benchmark::Engines::HTMLTemplateCompiled
$bench = Template::Benchmark->new(
cache_types_from => {
MojoTemplate => 1,
HTMLTemplateCompiled => 1,
TemplateSandbox => 0,
},
);
Support for multiple "cache_types_from" values was added in 1.09.
template_repeats => *$number* (default 30)
After the template is constructed from the various feature snippets
it gets repeated a number of times to make it longer, this option
controls how many times the basic template gets repeated to form the
final template.
The default of 30 is chosen to provide some form of approximation of
the workload in a "normal" web page. Given that "how long is a web
page?" has much the same answer as "how long is a piece of string?"
you will probably want to tweak the number of repeats to suit your
own needs.
duration => *$seconds* (default 10)
This option determines how many CPU seconds should be spent running
each *benchmark function*, this is passed along to Benchmark as a
negative duration, so read the Benchmark documentation if you want
the gory details.
The larger the number the less statistical variance you'll get, the
less likely you are to have temporary blips of the test machine's
I/O or CPU skewing the results, the downside is that your benchmarks
will take corresspondingly longer to run.
The default of 10 seconds seems to give pretty consistent results
for me within +/-1% on a very lightly loaded linux machine.
dataset => *$dataset_name* (default 'original')
dataset => { hash1 => { *variables* }, hash2 => { *variables* } }
Sets the dataset of *template variables* to use for the benchmark
function, either as the name of one of the presupplied datasets or
as a hashref to a custom dataset.
Currently the only presupplied dataset is 'original'.
If supplying a custom dataset the hashref should consist of only two
keys, named 'hash1' and 'hash2' the values of which will be passed
as the two hashes of *template variables* to each *benchmark
function*.
Please see the section "CUSTOM DATASETS" for more details on
supplying your own datasets.
The "dataset" option was added in 1.04.
style => *$string* (default 'none')
This option is passed straight through as the "style" argument to
Benchmark. By default it is 'none' so that no output is printed by
Benchmark, this also means that you can't see any results until all
the benchmarks are done. If you set it to 'auto' then you'll see the
benchmark results as they happen, but Template::Benchmark will have
no control over the generated output.
Might be handy for debugging or if you're impatient and don't want
pretty reports.
See the Benchmark documentation for valid values for this setting.
keep_tmp_dirs => *0* | *1* (default 0)
If set to a true value then the temporary directories created for
template files and caches will not be deleted when the
Template::Benchmark instance is destroyed. Instead, at the point
when they would have been deleted, their location will be printed.
This allows you to inspect the directory contents to see the
generated templates and caches and so forth.
Because the location is printed, and at an unpredictable time, it
may mess up your program output, so this option is probably only
useful while debugging.
skip_output_compare => *0* | *1* (default 0)
If enabled, this option will skip the sanity check that compares the
output of the different template engines to ensure that they're
producing the same output.
This is useful as a workaround if the sanity check is producing a
mismatch error that you deem to be unimportant.
It is strongly recommended that you never use this option without
first manually checking the mismatched outputs to be certain that
they are in fact unimportant.
If you find yourself needing to use this option as a workaround,
please file a bug report at
<http://rt.cpan.org/NoAuth/Bugs.html?Dist=Template-Benchmark>.
The "skip_output_compare" option was added in 1.05.
only_plugin => *$plugin* (default none)
skip_plugin => *$plugin* (default none)
If either of these two options are set they are used as a
'whitelist' and 'blacklist' of what *template engine* plugins to
use.
Each can be supplied multiple times to build the whitelist or
blacklist, and expect the leaf module name, or you can supply an
arrayref of names, or a hashref of names with true/false values to
toggle them on or off.
# This runs only Template::Benchmark::Engines::TemplateSandbox
$bench = Template::Benchmark->new(
only_plugin => 'TemplateSandbox',
);
# This skips Template::Benchmark::Engines::MojoTemplate and
# Template::Benchmark::Engines::HTMLTemplateCompiled
$bench = Template::Benchmark->new(
skip_plugin => 'MojoTemplate',
skip_plugin => 'HTMLTemplateCompiled',
);
# This runs only Template::Benchmark::Engines::MojoTemplate and
# Template::Benchmark::Engines::HTMLTemplateCompiled
$bench = Template::Benchmark->new(
only_plugin => {
MojoTemplate => 1,
HTMLTemplateCompiled => 1,
TemplateSandbox => 0,
},
);
PUBLIC METHODS
*$benchmark* = Template::Benchmark->new( *%options* )
This is the constructor for Template::Benchmark, it will return a
newly constructed benchmark object, or throw an exception explaining
why it couldn't.
The options you can pass in are covered in the "OPTIONS" section
above.
*$result* = $benchmark->benchmark()
Run the benchmarks as set up by the constructor. You can run
"$benchmark->benchmark()" multiple times if you wish to reuse the
same benchmark options.
The structure of the $result hashref is covered in "BENCHMARK
RESULTS" below.
*%defaults* = Template::Benchmark->default_options()
Returns a hash of the valid options to the constructor and their
default values. This can be used to keep external programs
up-to-date with what options are available in case new ones are
added or the defaults are changed. This is what
benchmark_template_engines does in fact.
*@cache_types* = Template::Benchmark->valid_cache_types()
Returns a list of the valid *cache types*. This can be used to keep
external programs up-to-date with what *cache types* are available
in case new ones are added. benchmark_template_engines does just
that.
*@features* = Template::Benchmark->valid_features()
Returns a list of the valid *template features*. This can be used to
keep external programs up-to-date with what *template features* are
available in case new ones are added. This is how
benchmark_template_engines gets at this info too.
$errors = $benchmark->engine_errors()
Returns a hashref of *engine* plugin to an arrayref of error
messages encountered while trying to enable to given plugin for a
benchmark.
This may be errors in loading the module or a list of *template
features* the *engine* didn't support.
$benchmark->engine_error( *$engine*, *$error_message* )
Pushes *$error_message* onto the list of error messages for the
engine plugin *$engine*.
*$number* = $benchmark->number_of_benchmarks()
Returns a count of how many *benchmark functions* will be run.
*$seconds* = $benchmark->estimate_benchmark_duration()
Return an estimate, in seconds, of how long it will take to run all
the benchmarks.
This estimate currently isn't a very good one, it's basically the
duration multiplied by the number of *benchmark functions*, and
doesn't count factors like the overhead of running the benchmarks,
or the fact that the duration is a minimum duration, or the initial
run of the *benchmark functions* to build the cache and compare
outputs.
It still gives a good lower-bound for how long the benchmark will
run, and maybe I'll improve it in future releases.
*@engines* = $benchmark->engines()
Returns a list of all *template engine plugins* that were
successfully loaded.
Note that this does not mean that all those *template engines*
support all requested *template features*, it merely means there
wasn't a problem loading their module.
*@features* = $benchmark->features()
Returns a list of all *template features* that were enabled during
construction of the Template::Benchmark object.
BENCHMARK RESULTS
The "$benchmark->benchmark()" method returns a results hashref, this
section documents the structure of that hashref.
Firstly, all results returned have a "result" key indicating the type of
result, this defines the format of the rest of the hashref and whether
the benchmark run was a success or why it failed.
"SUCCESS"
This indicates that the benchmark run completed successfully, there
will be the following additional information:
{
result => 'SUCCESS',
start_time => 1265738228,
title => 'Template Benchmark @Tue Feb 9 17:57:08 2010',
descriptions =>
{
'HT' =>
'HTML::Template (2.9)',
'TS_CF' =>
'Template::Sandbox (1.02) with Cache::CacheFactory (1.09) caching',
},
reference =>
{
type => 'uncached_string',
tag => 'TS',
output => template output,
},
benchmarks =>
[
{
type => 'uncached_string',
timings => Benchmark::timethese() results,
comparison => Benchmark::cmpthese() results,
},
{
type => 'memory_cache',
timings => Benchmark::timethese() results,
comparison => Benchmark::cmpthese() results,
},
...
],
}
"NO BENCHMARKS TO RUN"
{
result => 'NO BENCHMARKS TO RUN',
}
"MISMATCHED TEMPLATE OUTPUT"
{
result => 'MISMATCHED TEMPLATE OUTPUT',
reference =>
{
type => 'uncached_string',
tag => 'TS',
output => template output,
},
failures =>
[
{
type => 'disk_cache',
tag => 'TT',
output => template output,
},
...
],
}
WRITING YOUR OWN TEMPLATE ENGINE PLUGINS
All *template engine* plugins reside in the
"Template::Benchmark::Engines" namespace and inherit the
Template::Benchmark::Engine class.
See the Template::Benchmark::Engine documentation for details on writing
your own plugins.
CUSTOM DATASETS
Starting with version 1.04, Template::Benchmark has allowed you to
supply your own data to use as *template variables* within the
benchmarks.
This is done by supplying a hashref to the "dataset" constructor option,
with two keys, 'hash1' and 'hash2' which in-turn have hashref values
which are to be used as the hashrefs of *template variables* supplied to
the *benchmark functions*:
$bench = Template::Benchmark->new(
dataset => {
hash1 => {
scalar_variable => 'I is a scalar, yarr!',
hash_variable => {
'hash_value_key' =>
'I spy with my little eye, something beginning with H.',
},
array_variable => [ qw/I have an imagination honest/ ],
this => { is => { a => { very => { deep => { hash => {
structure => "My god, it be full of hashes.",
} } } } } },
template_if_true => 'True dat',
template_if_false => 'Nay, Mister Wilks',
},
hash2 => {
array_loop =>
[ qw/five four three two one coming ready or not/ ],
hash_loop => {
aaa => 'first',
bbb => 'second',
ccc => 'third',
ddd => 'fourth',
eee => 'fifth',
},
records_loop => [
{ name => 'Joe Bloggs', age => 16, },
{ name => 'Fred Bloggs', age => 23, },
{ name => 'Nigel Bloggs', age => 43, },
{ name => 'Tarquin Bloggs', age => 143, },
{ name => 'Geoffrey Bloggs', age => 13, },
],
variable_if => 1,
variable_if_else => 0,
variable_expression_a => 20,
variable_expression_b => 10,
variable_function_arg => 'Hi there',
},
},
);
There are no constraints on what *template variable* belongs in "hash1"
and "hash2", just so long as it occurs once and only once in both.
"scalar_variable"
This value gets interpolated into the template as part of the
"scalar_variable" feature, and can be any string value.
"hash_variable"
Used by the "hash_variable_value" feature, this value must be a
hashref with key 'hash_value_key' pointing at a string value to be
interpolated into the template.
"array_variable"
Used by the "array_variable_value" feature, this value must be an
arrayref consisting of at least 3 elements, the third of which will
be interpolated into the template.
"this"
The base of a deep data-structure for use in the
"deep_data_structure_value" feature, this should be a nested series
of hashrefs keyed in turn by 'this', 'is', 'a', 'very', 'deep',
'hash', 'structure', with the final 'structure' key referring to a
string value to interpolate into the template.
"array_loop"
Used within the "array_loop_value" and "array_loop_template"
features, this value should be an arrayref of any number of strings,
all of which will be looped across and included in the template
output.
"hash_loop"
Used within the "hash_loop_value" and "hash_loop_template" features,
this value should be a hashref of any number of string keys mapping
to string values, all of which will be looped across and included in
the template output.
"records_loop"
Used within the "records_loop_value" and "records_loop_template"
features, this value should be an arrayref of any number of hashrefs
"records", each consisting of a 'name' key with a string value, and
an 'age' key with integer value. Each record will be iterated across
and written to the template output.
"variable_if"
"variable_if_else"
These two keys are used by the "variable_if_*" and
"variable_if_else_*" features and should be set to a true or false
integer which will determine which branch of the if-construct will
be taken.
"template_if_true"
"template_if_false"
These two keys should have distinct string values that will be
interpolated into the template as the content in the *_if_template
and *_if_else_template features, depending on which branch is taken.
"variable_expression_a"
"variable_expression_b"
These two values are used in the "variable_expression" and
"complex_variable_expression" features, and should be integer
values.
For most consistent results they should probably be chosen with care
to keep the two features to integer mathematics to avoid rounding or
floating-point inconsistencies between different template engines,
the default values of 20 and 10 respectively do this.
See the Template::Benchmark::Engine documentation for further
details.
"variable_function_arg"
Used by the "variable_function" feature, this value should be a
string of at least 6 characters length, the 4th through 6th of which
will be used in the template output.
UNDERSTANDING THE RESULTS
This section aims to give you a few pointers when analyzing the results
of a benchmark run, some points are obvious, some less so, and most need
to be applied with some degree of intelligence to know when they're
applicable or not.
Hopefully they'll prove useful.
If you're wondering what all the numbers mean, the documentation for
Benchmark will probably be more helpful.
memory_cache vs instance_reuse
Comparing the *memory_cache* and *instance_reuse* times for an
*engine* should generally give you some idea of the overhead of the
caching system used by the *engine* - if the times are close then
they're using a good caching system, if the times are wildly
divergent then you might want to implement your own cache instead.
uncached_string vs instance_reuse or memory_cache
Comparing the *uncached_string* vs the *instance_reuse* or the
*memory_cache* (*instance_reuse* is better if you can) times for an
*engine* should give you an indication of how costly the parse and
compile phase for a *template engine* is.
uncached_string or uncached_disk represents a cache miss
The *uncached_string* or *uncached_disk* benchmark represents a
cache miss, so comparing it to the cache system you intend to use
will give you an idea of how much you'll hurt whenever a cache miss
occurs.
If you know how likely a cache miss is to happen, you can combine
the results of the two benchmarks proportionally to get a better
estimate of performance, and maybe compare that between different
engines.
Estimating cache misses is a tricky art though, and can be mitigated
by a number of measures, or complicated by miss stampedes and so
forth, so don't put too much weight on it either.
Increasing repeats emphasises template performance
Increasing the length of the template by increasing the
"template_repeats" option *usually* places emphasis on the ability
of the *template engine* to process the template vs the overhead of
reading the template, fetching it from the cache, placing the
variables into the template namespace and so forth.
For the most part those overheads are fixed cost regardless of
length of the template (fetching from disk or cache will have a,
usually small, linear component), whereas actually executing the
template will have a linear cost based on the repeats.
This means that for small values of repeats you're spending
proportionally more time on overheads, and for large values of
repeats you're spending more time on running the template.
If a *template engine* has higher-than-average overheads, it will be
favoured in the results (ie, it will rank higher than otherwise) if
you run with a high "template_repeats" value, and will be hurt in
the results if you run with a low "template_repeats" value.
Inverting that conclusion, if an *engine* moves up in the results
when you run with long repeats, or moves down in the results if you
run with short repeats, it follows that the *engine* probably has
high overheads in I/O, instantiation, variable import or somewhere.
deep_data_structure_value and complex_variable_expression are stress
tests
Both the "deep_data_structure_value" and
"complex_variable_expression" *template features* are designed to be
*stress test* versions of a more basic feature.
By comparing "deep_data_structure_value" vs "hash_variable_value"
you should be able to glean an indication of how well the *template
engine* performs at navigating its way through its variable stash
(to borrow Template::Toolkit terminology).
If an *engine* gains ranks moving from "hash_variable_value" to
"deep_data_structure_value" then you know it has a
more-efficient-than-average implementation of its stash, and if it
loses ranks then you know it has a less-efficient-than-average
implementation.
Similarly, by comparing "complex_variable_expression" and
"variable_expression" you can draw conclusions about the *template
engine's* expression execution speed.
constant vs variable features
Several *template features* have "constant" and "variable" versions,
these indicate a version that is designed to be easily optimizable
(the "constant" one) and a version that cannot be optimized (the
"variable" one).
By comparing timings for the two versions, you can get a feel for
whether (and how much) constant-folding optimization is done by a
*template engine*.
Whether this is of interest to you depends entirely on how you
construct and design your templates, but generally speaking, the
larger and more modular your template structure is, the more likely
you are to have bits of constant values "inherited" from parent
templates (or config files) that could be optimized in this manner.
This is one of those cases where only you can judge whether it is
applicable to your situation or not, Template::Benchmark merely
provides the information so you can make that judgement.
duration only effects accuracy
The benchmarks are carefully designed so that any one-off costs from
setting up the benchmark are not included in the benchmark results
itself.
This means that there *should* be no change in the results from
increasing or decreasing the benchmark duration, except to reduce
the size of the error resulting from background load on the machine.
If a *template engine* gets consistently better (or worse) results
as duration is changed, while other *template engines* are unchanged
(give or take statistical error), it indicates that something is
wrong with either the *template engine*, the plugin or something
else - either way the results of the benchmark should be regarded as
suspect until the cause has been isolated.
KNOWN ISSUES AND BUGS
Test suite is non-existent
The current test-suite is laughable and basically only tests
documentation coverage.
Once I figure out what to test and how to do it, this should change,
but at the moment I'm drawing a blank.
Results structure too terse
The results structure could probably do with more information such
as what options were set and what version of benchmark/plugins were
used.
This would be helpful for anything wishing to archive benchmark
results, since it may (will!) influence how comparable results are.
Engine meta-data not retrievable if template engine isn't installed
The *template engine* plugins are designed in a way that requires
their respective *template engine* to be installed for the plugin to
compile successfully: this allows the benchmarked code in the plugin
to be free of cruft testing for module availability, which in turn
gives it a more accurate benchmark.
The downside of this approach is that none of the meta-information
like syntaxes, engine descriptions, syntax type, and "pure-perl"ness
is available unless the *template engine* is available.
This is potentially sucky, but on the other hand it's probably an
indication that the meta-information ought to belong elsewhere,
probably in a different distribution entirely as it's not
specifically to do with benchmarking.
SEE ALSO
Task::Template::Benchmark
BUGS
Please report any bugs or feature requests to "bug-template-benchmark at
rt.cpan.org", or through the web interface at
<http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Template-Benchmark>. I
will be notified, and then you'll automatically be notified of progress
on your bug as I make changes.
SUPPORT
You can find documentation for this module with the perldoc command.
perldoc Template::Benchmark
You can also look for information at:
* RT: CPAN's request tracker
<http://rt.cpan.org/NoAuth/Bugs.html?Dist=Template-Benchmark>
* AnnoCPAN: Annotated CPAN documentation
<http://annocpan.org/dist/Template-Benchmark>
* CPAN Ratings
<http://cpanratings.perl.org/d/Template-Benchmark>
* Search CPAN
<http://search.cpan.org/dist/Template-Benchmark/>
ACKNOWLEDGEMENTS
Thanks to Paul Seamons for creating the the bench_various_templaters.pl
script distributed with Template::Alloy, which was the ultimate
inspiration for this module.
Thanks also for contributions by Goro Fuji and Alex Efros.
AUTHOR
Sam Graham <libtemplate-benchmark-perl BLAHBLAH illusori.co.uk>
COPYRIGHT AND LICENSE
This software is copyright (c) 2010-2011 by Sam Graham
<libtemplate-benchmark-perl BLAHBLAH illusori.co.uk>.
This is free software; you can redistribute it and/or modify it under
the same terms as the Perl 5 programming language system itself.