Sam's Blog

Template Benchmark 1.04: custom datasets and feature repeats

Date: Wednesday, 7 July 2010, 17:46.

Categories: perl, ironman, template-benchmark, benchmarking.

I've just uploaded Template::Benchmark v1.03_02, the first release candidate for v1.04.

This adds two of the features that have been on my long-term goals for this project: custom datasets, and per-feature repeats.

The first lets you supply your own data-structure to use within the benchmark templates (or choose from a presupplied list).

The second lets you choose on a feature-by-feature basis, how often that feature should be used in the generated benchmark template, rather than only choosing how often the entire template would be repeated.

Taken together, these two features allow you far greater control of the generated benchmark, allowing you to tailor it to fit your individual needs with greater precision, read on for more details.

Custom Datasets

The basic behaviour of Template::Benchmark is to generate a functionally-equivalent template for each template engine, and then plug in the same template variables to each and measure the results.

Until now you had a fair bit of control over what went into the generated template, by toggling features on and off, but you had no control over the data that the template was run over.

Custom datasets give you that control.

Let's take a hypothetical example: you've got a template setup that involves substituting large blobs of text into the middle of the template as a template variable.

Now the existing scalar_variable feature lets you benchmark substituting a simple template variable and sounds like exactly what you want, but there's a catch: the string that gets substituted is only 30-40 characters long - you want a big blob of 5000 characters, and the performance profile of that may not be the same. (It might be, it might not... but who knows if you can't measure it?)

This is where a custom dataset comes in, you replace the scalar_variable value with your 5000 of example text and now you're benchmarking what you wanted to benchmark.

How about records_loop_template? It only loops across 5 "records", and maybe you want to emulate the load of a search results page with 30, 50 or 100 records?

No problem, just create a custom dataset with a variable records_loop that contains however many records you want.

Now there are some constraints, you still can't alter the logic of the generated templates themselves, so you still have to supply data-structures in the format that they want and expect, but you can change the values in those data-structures to suit your situation more closely.

If you don't want to go to all the effort of defining the whole dataset, you can choose one of the presupplied ones by name. Currently there's only the original dataset, named 'original', but I intend to extend the selection somewhat. (And probably allow a plugin mechanism to let people write and distribute their own.)

Per-feature repeats

One of the things I've been less satisified about with Template::Benchmark is that "all features are equal": you can only toggle a feature on or off, it's either there or it isn't and if it is, it counts as much as every other feature.

For some features this isn't a problem, no two template uses are the same, and so having a benchmark template that has one if statement, one loop and one variable substitution seems as good a way of doing it as any other: it may not perfectly model anyone's template load, but it doesn't seem any worse than any other choice.

It starts to fall apart with some of the more esoteric features though, I mean it's useful to be able to benchmark constant-foldable constructs, but if you toggle on all those features, you're producing a benchmark that's assuming that those sorts of structures crop up every bit as frequently as simple template variable substitution.

That stretches the bounds of credibility somewhat, and it distorts the usefulness of the results.

Take for example GenericTemplateEngine who has no constant-folding and executes template expressions in 5 milliseconds, and OptimizedTemplateEngine who has constant-folding and executes expressions in 7 milliseconds unfolded, or 1 millisecond for folded.

If you enabled both features you'd end up with GenericTemplateEngine taking 10 millseconds for the entire template, and OptimizedTemplateEngine taking 8 milliseconds for the entire template, and you'd cheer OptimizeTemplateEngine on as the victor by 20%.

That might even be an accurate result if you do have a 50-50 split of constant expressions vs variable, but that seems a little unusual.

As it happens, I don't actually need to know what the ratio is, because instead I've now given you the tools to set it yourself.

With the per-feature repeats addition, you can say how often you want any particular feature to crop up in the generated template. (Note that this is independent of, and stacks with the existing template_repeats option.)

So, if you want to model a page that has 10 variable substitutions, 1 loop and 3 if statements, you just push those values into the repeats for each feature, and you'll get a benchmark template generated that does just that.

If you think one of those if statements might be constant-foldable, you can say instead that you have 2 variable_if_template features and 1 constant_if_template features, and so forth.


Taken together, custom datasets and per-feature repeats allow you to produce an even more flexible benchmark with Template::Benchmark, letting you simulate your needs more accurately to produce the results that are relevent to you and your needs.

The first release candidate, 1.03_02 is available on CPAN (Template::Benchmark), please take a look if these new features sound interesting to you.

The stable release of 1.04 will follow in a week or so, after the smoke-testers have had chance to file their reports. (Assuming nothing breaks, but what could possibly go wrong? *BZZZZT*)

Browse Sam's Blog Subscribe to Sam's Blog

By day of July: 01, 07, 15, 24.

By month of 2010: March, April, May, June, July, August, September, November.

By year: 2010, 2011, 2012, 2013.

Or by: category or series.


blog comments powered by Disqus
© 2009-2013 Sam Graham, unless otherwise noted. All rights reserved.