XSLTMark is the first XSLT processor performance benchmarking application. It gauges the capabilities of XSLT processing engines by testing them on a common platform with a variety of stylesheets and inputs that sample the gamut of possible applications. XSLTMark is a completely open benchmark with all the source code freely available since its first release in October 2000.
(Release Notes for even older versions are available here.)
The XSLTMark test cases have been designed to challenge processors with a variety of tasks and input conditions in order to provide a well-rounded benchmark and to facilitate analysis of which processors perform best for various applications. XSLTMark measures performance in four major categories:
The size and composition of the source document is also an important factor in determining the peformance of a transform. Larger source documents are always slower to process, but for some, the performance of the engines scales much better with increasing input size. Also, certain qualities of the input tree can have a profound effect, such as the breadth and depth of the tree, the number of text nodes, etc. The test bed includes input documents of a variety of lengths and compositions.
The following table assesses each of the default test cases for its relevance to the
particular processing and input categories:
| Test Case | Input Size | Input Description | Stylesheet | Notes |
| alphabetize | medium | 100-row database table | select, control | Sorts the input tree according to element name. |
| attsets | small | sales report | general | Tests node-copying using named attribute sets. |
| avts | medium | 100-row database table | select | Tests attribute-value template expansion. |
| axis | small | select | Tests XPath selection along the different axes. | |
| backwards | small | control | Reverses order of elements in input document. | |
| bottles | small | initial size parameter | function, control | Generates "99 bottles of beer on the wall" song. |
| breadth | small | broad and shallow tree | select, control | Performs a search for a unique element in a large tree. |
| brutal | small | select, function, control | Executes many functions, sorts, etc. | |
| chart | small | sales report | select, control | Generates an HTML chart of some sales data. |
| creation | medium | 100-row database table | general | Tests xsl:element and xsl:attribute. |
| current | small | select | Tests complex XPath node selection. | |
| dbonerow | large | 10000-row database table | select, control | Selects a single row from a very large table. |
| dbtail | medium | 100-row database table | select | Prints a table by traversing the following-sibling axis. |
| decoy | small | 100-row database table | match | Same template as patterns, with some decoy templates thrown in. |
| depth | small | narrow and deep tree | select, control | Performs a search for a unique element in a large tree. |
| encrypt | medium | 100-row database table | function | Performs a Rot-13 operation on all element names and text nodes |
| functions | medium | 100-row database table | function | Tests a variety of number and string functions. |
| game | small | baseball game stats | select, functions, control | Produces a HTML table of the data. |
| html | small | select, control | Literal result element as stylesheet example from XSLT spec. | |
| identity | large | 1000-row database table | control | The identity transform. |
| inventory | small | Inventory data | select, control | Produces a HTML table of the data. |
| metric | small | Data in metric notation | function | Converts metric units to English units. |
| number | small | function | Tests format-number() function. | |
| oddtemplate | small | match, select | Tests a variety of complex match patterns. | |
| patterns | medium | 100-row database table | match | Stylesheet contains extremely simple templates with tough patterns. |
| prettyprint | medium | 100-row database table | control, function | Formats the input input legal HTML. |
| priority | small | select, control | Pops the first element off a priority Queue and returns the queue. | |
| products | small | Product data | select, control | Produces an HTML table from the data. |
| queens | small | initial size parameter | function, control | Solves the "8 Queens" problem. (Stylesheet by Oren Ben-Kiki, used with permission.) |
| reverser | small | The Gettysburg Address | function, control | Stylesheet copies input with text-node strings reversed. |
| stringsort | large | 1000-row database table | control | Performs a sort based on string keys. |
| summarize | small | "Queens" stylesheet | function | Reports information about an XSL stylesheet. |
| total | small | sales report | select, function | Reports on sales data. |
| tower | small | initial size parameter | control, function | Solves the Towers of Hanoi problem. |
| trend | small | Numerical data | select, functions | Computes trends in the input data. |
| union | small | match, select | Performs complex pattern matching. | |
| xpath | small | match | Performs complex pattern matching. | |
| xslbench1 | small | test1.xml from XSLBench | general | This test case is "test1.xsl" from Kevin Jones' XSLBench test suite, used with permission. |
| xslbench2 | large | A Midsummer Night's Dream | match, select | This test case is "test2.xsl" from Kevin Jones' XSLBench test suite, used with permission. |
| xslbench3 | large | A Midsummer Night's Dream | select, function | This test case is "test3.xsl" from Kevin Jones' XSLBench test suite, used with permission. |
XSLTMark generates a table of statistics, as in this sample output:
XSLTMark 2.0 (Feb 28 2001) running on Linux 2.2.5-15 x86 Java Runtime Environment is IBM Corporation 1.3.0 Current time is 12:00:31 EST, 26 Mar 2001 Results for driver XTDriver prep exec Test Component iter ms ms KB In KB Out KB/s status alphabetize 10 387 1616 196 156 109.31 OK attsets 100 34 1116 121 255 169.02 OK avts 100 8 2794 1966 1156 558.87 OK axis 100 15 434 35 60 110.71 OK backwards 100 7 1963 260 261 133.03 CHK OUTPUT bottles 100 13 1597 4 1130 355.31 OK breadth 100 10 1304 415 200 236.46 OK brutal 100 14 996 130 394 263.36 OK chart 100 14 919 121 359 261.57 OK creation 50 26 2805 983 1520 446.32 OK current 100 56 452 18 29 54.01 OK dbonerow 10 16 23255 19621 8 422.07 OK dbtail 100 8 3824 1966 503 322.95 OK ... Aggr. Results 2780 1239 107881 44218 21904 306.46 Arithmetic Mean 33.49 279.15 Geometric Mean 198.35 Harmonic Mean 97.29
iter |
The number of iterations conducted of this test. |
prep ms |
The preparation time in milliseconds for this test's stylesheet. |
exec ms |
The execution time in milliseconds for this test. This time includes time spent writing the output to a file, but not the time spent parsing the input .xml file or opening and closing the output file. |
KB In |
Total kilobytes of source data processed (equals size of input document * iterations) |
KB Out |
Total kilobytes of data output (equals size of output * iterations) |
KB/s |
Throughput, calculated as (KB in + KB out) / (2 * time). |
Result |
If output verification is available on your platform, this column contains the
result of a simple compliance check: "OK" indicates success, "CHK OUTPUT"
indicates a likely compliance violation, and "ERROR" indicates a failed test. "NO REF"
indicates that no reference exists for the test case. |
total-results |
This row contains the aggregated results for all tests whose result
was "OK" or "NO REFERENCE". Note that the aggregated KB/s number is computed as (total KB in +
total KB out) / (2 * total time), and not by averaging the KB/s numbers of each test case. |
DataPower Technology has tested several popular XSLT processors with the XSLTMark benchmark. The results can be viewed here.
The latest version of XSLTMark is available at http://www.datapower.com/XSLTMark/download.
XSLTMark is Copyright © 2000, 2001 DataPower Technology, Inc.
This software has been released according to the terms stated in the
LICENSE file found in the distribution.
tar xzf xsltmark.tar.gz runxsltmark shell script to configure
the CLASSPATH variable and java runtime environment for your
system. XSLTMark requires one or more of these processor packages, depending on which processor(s) you wish to benchmark.
saxon.jar is available from the
Saxon homepage.
The procedure for running XSLTMark will vary according to your particular Java runtime environment.
We have provided a shell script to run XSLTMark from the command line:
./runxsltmark [parameters] XSLTMark [options] testdriver -c file - Load test configuration from the specified
file. This option may appear several times. If no configuration file is
specified, XSLTMark uses the testcases/default.conf. See below for
documentation for configuration files.
-n num - Repeat the benchmark num times for each test
case (default is 1).
-t - Run only one iteration of each test case, ignoring the number
of iterations each is configured for. This is useful for running quick conformance
tests.
SaxonDriver - Driver for Saxon by Michael Kay.
XalanJDriver - Driver for the Apache Project's Xalan-Java processor.
XTDriver - Driver for James Clark's XT.
XSLTCDriver - Driver for Sun's
XSLTC.
OracleXSLTDriver - Driver for Oracle's
Java XSLT Processor.
ExternalDriver command - provides an interface to test drivers for processors
that don't run on the Java platform. The command argument is the name of the
driver executable to be invoked by ExternalDriver. The following drivers are
included (make sure the directory:
msxsltdriver - driver for Microsoft's
MSXML.
sablotdriver - driver for the Sablotron processor published by Ginger Alliance.
txdriver - driver for the Mozilla Project's Transformiix
processor.
xalandriver - driver for the Apache Project's Xalan-C++ processor.
gnomexsltdriver - driver for
LibXSLT.
4ssdriver - driver for Four Thought's
4-Suite (Requires
python to execute).
Drivers for other processors are in the works. If you don't see a test driver for your favorite processor here, please write one for us :-) Send new driver code submissions to xsltmark@datapower.com.
XSLTMark uses a configuration file with a similar format to Windows' .INI files. The file specifies a list of test cases to execute with appropriate parameters. Each test case is declared in a block such as this one:
[testcasename]
input=input.xml
stylesheet=stylesheet.xsl
output=output.out
reference=input.ref
iterations=50
skipdriver=SomeDriver
The required fields input, stylesheet, and
output define the input and output parameters of the test cases.
The reference field specifies the expected output of the transform,
as normalized by the included dgnorm utility (UNIX only). The
iterations parameter defines the number of times the
test case should be executed, and defaults to 100 if unspecified. The
skipdriver directive indicates that the test should not be
run for the specified processor. Comment lines begin with a '#'
character.
The source files have been included in the src subdirectory.
To build the XSLTMark Java classes, edit the Makefile according to your
system configuration (CLASSPATH settings, java compiler, etc.), and issue a
make command.
To build xalandriver or txdriver (the interfaces
XSLTMark uses to control Xalan-C++ and TransforMiiX respectively) first build
the Xalan or TransforMiiX binaries as directed by the package's own
documentation. Then edit the XSLTMark Makefile to ensure the path settings are
correct and issue a make xalandriver or make txdriver
command.
Building dgnorm is slightly more involved. It uses a
modified version of the EXPAT v1.95.1 library by James
Clark, available from
http://www.jclark.com/xml. Once you have downloaded the original
library, you need to apply a patch so that dgnorm will
build and work properly. The patch is included with XSLTMark in the
file expat_dgnorm_patch. Once you have downloaded the
EXPAT library, simply go to that directory and type
patch -p 1 < XSLTMARKSRC/expat_dgnorm_patch where
XSLTMARKSRC is the directory containing your XSLTMark
source code. Your newly modified EXPAT library will still
be backwards compatible, so you don't need to make a new copy.
Once you patched EXPAT, you need to modify the Makefile and set the
EXPAT variable to the name of directory that contains the
(newly modified) EXPAT libraries and include files. Finally, type
make dgnorm.
XSLTMark defines a simple API for writing drivers for XSLT Processors.
All XSLTMark test drivers are extensions of the abstract class XSLTDriver, implementing a new driver is a matter of extending a few basic functions:
void init (String[] options)
This function is called at the beginning of the benchmark to allow the driver to load and initialize itself.boolean chdir (String dir)
This function changes the driver's working directory todir. It returns false upon success.boolean loadStylesheet (String filename)
This function is called to instruct the driver to load the stylesheet specified byfilename. It returns false upon success.boolean loadInput (String filename)
This function is called to instruct the driver to load the source document specified byfilename. It returns false upon success.boolean runTransform (String filename, int iterations)
This function is called to instruct the driver to apply the current stylesheet to the current source document a number of times equal toiterations. The driver is responsible for updating themTotalMillisvariable according to how much time was spent executing the transform. The output document should be stored into the file specified byfilename. It returns false upon success.boolean shutdown ()
This function is called to instruct the driver to terminate itself.
The XTDriver source code
(XTDriver.java) is a good example driver
implementation.
The ExternalDriver
class benchmarks a non-java driver by executing a binary driver and
communicating with it through a simple text interface via the
process' stdin and stdout streams. The communication protocol
contains the following directives, each of which is terminated by a
newline (\n):
chdir dir
Instructs the driver to change its working directory todir. The driver must respond with the string "OK" upon success.stylesheet filename
Instructs the driver to load the stylesheet specified byfilename. The driver must respond with "OK" upon success.input filename
Instructs the driver to load the source document specified byfilename. The driver must respond with "OK" upon success.transform filename iterations
Instructs the driver to transform its current source document with its current stylesheet, and store the result infilename. The transform is to be executed a number of times equal toiterations. The driver must respond with a string of the form
"OK wallclock: ######.## ms; cpuclock: ######.##"
upon success.terminate
Instructs the driver to exit. The driver must respond with "OK" and then exit upon receiving this command.
If you are intending to write a C++ driver, then most of the work
has done for you in the abstract base class
xmExternalDriver. See one of the provided C++ source
code files for an example. If your driver extends this base class,
then implementing an External Driver is as simple as defining the
following abstract methods, each of which is called in response to
one of the above commands. The actual timing, as well as all
communication with XSLTMark program, is handled within the base
class.
int loadStylesheet (char * filename);
Causes the driver to load a stylesheet contained in the given file and prepare it for execution. Returns 0 on success.int setInputDocument (char * filename);
Causes the driver to load the input .XML file and prepare it to be transformed. Returns 0 on success.int openOutput (char * outputFilename);
Causes the driver to open the output file. Returns 0 on success.int runTransform ();
Executes the previously loaded stylesheet on the previously specified input document. The output should go to the file specified in theopenOutput()call. Returns 0 on success.int closeOutput ();
Closes the output file that was opened viaopenOutput(). Returns 0 on success.int terminate();
Causes the driver to terminate. Returns 0 on success.
For example C++ code, please see the included C++ drivers. The
FourThought driver (4ssdriver) provides an example of an
ExternalDriver written without the aid of the
xmExternalDriver base class.
Features planned for future releases of XSLTMark include:
Contact xsltmark@datapower.com