XSLTMark

Overview

XSLTMark is the first XSLT processor performance benchmarking application. It gauges the capabilities of XSLT processing engines by testing them on a common platform with a variety of stylesheets and inputs that sample the gamut of possible applications. XSLTMark is a completely open benchmark with all the source code freely available since its first release in October 2000.

Contents


Release Notes for Version 2.1.0

Release Notes for Version 2.0

(Release Notes for even older versions are available here.)

Understanding The Benchmark

The XSLTMark test cases have been designed to challenge processors with a variety of tasks and input conditions in order to provide a well-rounded benchmark and to facilitate analysis of which processors perform best for various applications. XSLTMark measures performance in four major categories:

The size and composition of the source document is also an important factor in determining the peformance of a transform. Larger source documents are always slower to process, but for some, the performance of the engines scales much better with increasing input size. Also, certain qualities of the input tree can have a profound effect, such as the breadth and depth of the tree, the number of text nodes, etc. The test bed includes input documents of a variety of lengths and compositions.

The Test Cases

The following table assesses each of the default test cases for its relevance to the particular processing and input categories:

Test Case Input Size Input Description Stylesheet Notes
alphabetize medium 100-row database table select, control Sorts the input tree according to element name.
attsets small sales report general Tests node-copying using named attribute sets.
avts medium 100-row database table select Tests attribute-value template expansion.
axis small   select Tests XPath selection along the different axes.
backwards small   control Reverses order of elements in input document.
bottles small initial size parameter function, control Generates "99 bottles of beer on the wall" song.
breadth small broad and shallow tree select, control Performs a search for a unique element in a large tree.
brutal small   select, function, control Executes many functions, sorts, etc.
chart small sales report select, control Generates an HTML chart of some sales data.
creation medium 100-row database table general Tests xsl:element and xsl:attribute.
current small   select Tests complex XPath node selection.
dbonerow large 10000-row database table select, control Selects a single row from a very large table.
dbtail medium 100-row database table select Prints a table by traversing the following-sibling axis.
decoy small 100-row database table match Same template as patterns, with some decoy templates thrown in.
depth small narrow and deep tree select, control Performs a search for a unique element in a large tree.
encrypt medium 100-row database table function Performs a Rot-13 operation on all element names and text nodes
functions medium 100-row database table function Tests a variety of number and string functions.
game small baseball game stats select, functions, control Produces a HTML table of the data.
html small   select, control Literal result element as stylesheet example from XSLT spec.
identity large 1000-row database table control The identity transform.
inventory small Inventory data select, control Produces a HTML table of the data.
metric small Data in metric notation function Converts metric units to English units.
number small   function Tests format-number() function.
oddtemplate small   match, select Tests a variety of complex match patterns.
patterns medium 100-row database table match Stylesheet contains extremely simple templates with tough patterns.
prettyprint medium 100-row database table control, function Formats the input input legal HTML.
priority small   select, control Pops the first element off a priority Queue and returns the queue.
products small Product data select, control Produces an HTML table from the data.
queens small initial size parameter function, control Solves the "8 Queens" problem. (Stylesheet by Oren Ben-Kiki, used with permission.)
reverser small The Gettysburg Address function, control Stylesheet copies input with text-node strings reversed.
stringsort large 1000-row database table control Performs a sort based on string keys.
summarize small "Queens" stylesheet function Reports information about an XSL stylesheet.
total small sales report select, function Reports on sales data.
tower small initial size parameter control, function Solves the Towers of Hanoi problem.
trend small Numerical data select, functions Computes trends in the input data.
union small   match, select Performs complex pattern matching.
xpath small   match Performs complex pattern matching.
xslbench1 small test1.xml from XSLBench general This test case is "test1.xsl" from Kevin Jones' XSLBench test suite, used with permission.
xslbench2 large A Midsummer Night's Dream match, select This test case is "test2.xsl" from Kevin Jones' XSLBench test suite, used with permission.
xslbench3 large A Midsummer Night's Dream select, function This test case is "test3.xsl" from Kevin Jones' XSLBench test suite, used with permission.

Interpreting the Results

XSLTMark generates a table of statistics, as in this sample output:


XSLTMark 2.0 (Feb 28 2001) running on Linux 2.2.5-15 x86
Java Runtime Environment is IBM Corporation 1.3.0
Current time is 12:00:31 EST, 26 Mar 2001

Results for driver XTDriver

                          prep  exec                                       
Test Component     iter   ms    ms     KB In     KB Out    KB/s      status

alphabetize         10    387   1616   196       156       109.31    OK
attsets             100   34    1116   121       255       169.02    OK
avts                100   8     2794   1966      1156      558.87    OK
axis                100   15    434    35        60        110.71    OK
backwards           100   7     1963   260       261       133.03    CHK OUTPUT
bottles             100   13    1597   4         1130      355.31    OK
breadth             100   10    1304   415       200       236.46    OK
brutal              100   14    996    130       394       263.36    OK
chart               100   14    919    121       359       261.57    OK
creation            50    26    2805   983       1520      446.32    OK
current             100   56    452    18        29        54.01     OK
dbonerow            10    16    23255  19621     8         422.07    OK
dbtail              100   8     3824   1966      503       322.95    OK

...

Aggr. Results       2780  1239  107881 44218     21904     306.46    
Arithmetic Mean           33.49                            279.15    
Geometric Mean                                             198.35    
Harmonic Mean                                              97.29  

Key:

iter The number of iterations conducted of this test.
prep ms The preparation time in milliseconds for this test's stylesheet.
exec ms The execution time in milliseconds for this test. This time includes time spent writing the output to a file, but not the time spent parsing the input .xml file or opening and closing the output file.
KB In Total kilobytes of source data processed (equals size of input document * iterations)
KB Out Total kilobytes of data output (equals size of output * iterations)
KB/s Throughput, calculated as (KB in + KB out) / (2 * time).
Result If output verification is available on your platform, this column contains the result of a simple compliance check: "OK" indicates success, "CHK OUTPUT" indicates a likely compliance violation, and "ERROR" indicates a failed test. "NO REF" indicates that no reference exists for the test case.
total-results This row contains the aggregated results for all tests whose result was "OK" or "NO REFERENCE". Note that the aggregated KB/s number is computed as (total KB in + total KB out) / (2 * total time), and not by averaging the KB/s numbers of each test case.

Benchmark Results for Popular Processors

DataPower Technology has tested several popular XSLT processors with the XSLTMark benchmark. The results can be viewed here.

How to Download the Software:

The latest version of XSLTMark is available at http://www.datapower.com/XSLTMark/download.

License

XSLTMark is Copyright © 2000, 2001 DataPower Technology, Inc.

This software has been released according to the terms stated in the LICENSE file found in the distribution.

Installation

  1. Download the XSLTMark package.

  2. Unpack the tar.gz or .zip distribution on your system.

    UNIX:
    tar xzf xsltmark.tar.gz

    Windows:
    Use WinZip or another package to expand the .zip file.

  3. Configure the runtime environment for XSLTMark.

    UNIX:
    Edit the runxsltmark shell script to configure the CLASSPATH variable and java runtime environment for your system.

Getting the Third-Party Packages

XSLTMark requires one or more of these processor packages, depending on which processor(s) you wish to benchmark.

Running

The procedure for running XSLTMark will vary according to your particular Java runtime environment.

UNIX:

We have provided a shell script to run XSLTMark from the command line:

./runxsltmark [parameters]

Synopsis:

XSLTMark [options] testdriver

Options:

-c file - Load test configuration from the specified file. This option may appear several times. If no configuration file is specified, XSLTMark uses the testcases/default.conf. See below for documentation for configuration files.

-n num - Repeat the benchmark num times for each test case (default is 1).

-t - Run only one iteration of each test case, ignoring the number of iterations each is configured for. This is useful for running quick conformance tests.

Drivers:

SaxonDriver - Driver for Saxon by Michael Kay.

XalanJDriver - Driver for the Apache Project's Xalan-Java processor.

XTDriver - Driver for James Clark's XT.

XSLTCDriver - Driver for Sun's XSLTC.

OracleXSLTDriver - Driver for Oracle's Java XSLT Processor.

ExternalDriver command - provides an interface to test drivers for processors that don't run on the Java platform. The command argument is the name of the driver executable to be invoked by ExternalDriver. The following drivers are included (make sure the directory:

msxsltdriver - driver for Microsoft's MSXML.

sablotdriver - driver for the Sablotron processor published by Ginger Alliance.

txdriver - driver for the Mozilla Project's Transformiix processor.

xalandriver - driver for the Apache Project's Xalan-C++ processor.

gnomexsltdriver - driver for LibXSLT.

4ssdriver - driver for Four Thought's 4-Suite (Requires python to execute).

Drivers for other processors are in the works. If you don't see a test driver for your favorite processor here, please write one for us :-) Send new driver code submissions to xsltmark@datapower.com.

Benchmark Configuration

XSLTMark uses a configuration file with a similar format to Windows' .INI files. The file specifies a list of test cases to execute with appropriate parameters. Each test case is declared in a block such as this one:

[testcasename]
input=input.xml
stylesheet=stylesheet.xsl
output=output.out
reference=input.ref
iterations=50
skipdriver=SomeDriver

The required fields input, stylesheet, and output define the input and output parameters of the test cases. The reference field specifies the expected output of the transform, as normalized by the included dgnorm utility (UNIX only). The iterations parameter defines the number of times the test case should be executed, and defaults to 100 if unspecified. The skipdriver directive indicates that the test should not be run for the specified processor. Comment lines begin with a '#' character.

Building

The source files have been included in the src subdirectory.

UNIX:

To build the XSLTMark Java classes, edit the Makefile according to your system configuration (CLASSPATH settings, java compiler, etc.), and issue a make command.

To build xalandriver or txdriver (the interfaces XSLTMark uses to control Xalan-C++ and TransforMiiX respectively) first build the Xalan or TransforMiiX binaries as directed by the package's own documentation. Then edit the XSLTMark Makefile to ensure the path settings are correct and issue a make xalandriver or make txdriver command.

Building dgnorm is slightly more involved. It uses a modified version of the EXPAT v1.95.1 library by James Clark, available from http://www.jclark.com/xml. Once you have downloaded the original library, you need to apply a patch so that dgnorm will build and work properly. The patch is included with XSLTMark in the file expat_dgnorm_patch. Once you have downloaded the EXPAT library, simply go to that directory and type patch -p 1 < XSLTMARKSRC/expat_dgnorm_patch where XSLTMARKSRC is the directory containing your XSLTMark source code. Your newly modified EXPAT library will still be backwards compatible, so you don't need to make a new copy.

Once you patched EXPAT, you need to modify the Makefile and set the EXPAT variable to the name of directory that contains the (newly modified) EXPAT libraries and include files. Finally, type make dgnorm.

Extending

XSLTMark defines a simple API for writing drivers for XSLT Processors.

Java Drivers:

All XSLTMark test drivers are extensions of the abstract class XSLTDriver, implementing a new driver is a matter of extending a few basic functions:

void init (String[] options)
This function is called at the beginning of the benchmark to allow the driver to load and initialize itself.

boolean chdir (String dir)
This function changes the driver's working directory to dir. It returns false upon success.

boolean loadStylesheet (String filename)
This function is called to instruct the driver to load the stylesheet specified by filename. It returns false upon success.

boolean loadInput (String filename)
This function is called to instruct the driver to load the source document specified by filename. It returns false upon success.

boolean runTransform (String filename, int iterations)
This function is called to instruct the driver to apply the current stylesheet to the current source document a number of times equal to iterations. The driver is responsible for updating the mTotalMillis variable according to how much time was spent executing the transform. The output document should be stored into the file specified by filename. It returns false upon success.

boolean shutdown ()
This function is called to instruct the driver to terminate itself.

The XTDriver source code (XTDriver.java) is a good example driver implementation.

External Drivers:

The ExternalDriver class benchmarks a non-java driver by executing a binary driver and communicating with it through a simple text interface via the process' stdin and stdout streams. The communication protocol contains the following directives, each of which is terminated by a newline (\n):

chdir dir
Instructs the driver to change its working directory to dir. The driver must respond with the string "OK" upon success.

stylesheet filename
Instructs the driver to load the stylesheet specified by filename. The driver must respond with "OK" upon success.

input filename
Instructs the driver to load the source document specified by filename. The driver must respond with "OK" upon success.

transform filename iterations
Instructs the driver to transform its current source document with its current stylesheet, and store the result in filename. The transform is to be executed a number of times equal to iterations. The driver must respond with a string of the form
"OK wallclock: ######.## ms; cpuclock: ######.##"
upon success.

terminate
Instructs the driver to exit. The driver must respond with "OK" and then exit upon receiving this command.

If you are intending to write a C++ driver, then most of the work has done for you in the abstract base class xmExternalDriver. See one of the provided C++ source code files for an example. If your driver extends this base class, then implementing an External Driver is as simple as defining the following abstract methods, each of which is called in response to one of the above commands. The actual timing, as well as all communication with XSLTMark program, is handled within the base class.

int loadStylesheet (char * filename);
Causes the driver to load a stylesheet contained in the given file and prepare it for execution. Returns 0 on success.

int setInputDocument (char * filename);
Causes the driver to load the input .XML file and prepare it to be transformed. Returns 0 on success.

int openOutput (char * outputFilename);
Causes the driver to open the output file. Returns 0 on success.

int runTransform ();
Executes the previously loaded stylesheet on the previously specified input document. The output should go to the file specified in the openOutput() call. Returns 0 on success.

int closeOutput ();
Closes the output file that was opened via openOutput(). Returns 0 on success.

int terminate();
Causes the driver to terminate. Returns 0 on success.

For example C++ code, please see the included C++ drivers. The FourThought driver (4ssdriver) provides an example of an ExternalDriver written without the aid of the xmExternalDriver base class.

Future Development

Features planned for future releases of XSLTMark include:

Contact xsltmark@datapower.com