JMH

Your JMH benchmarks can generate a number of JSON files storing benchmark results, but it is difficult to produce a good-looking graph from them. Carte-JMH was written to produce graphs and tables that can be displayed on a website (see the CSS4J Benchmark results).

Running the JMH report application

To run the benchmark report app on JMH-produced JSON files, you have to prepare an XML configuration file and execute:

1. ./gradlew build uberjar
2. java -jar carte-jmh/build/libs/carte-jmh-<version>-all.jar --config=<path-to-config-file> *.json

The generic elements of the XML configuration file are described in the Configuration page, but Carte-JMH adds its own configurations that are better explained with a real example.

Learning by example

https://github.com/css4j/carte/blob/master/carte-jmh/examples/sac-benchmark.xml

As described in the Configuration page, the root is a <reportdef> element with a <name> child.

<name>SAC benchmarks</name>

Then, look at the following lines that declare a benchmark based on SACBenchmark.java:

    <benchmark id="sac-226kb">
        <class>io.sf.carte.mark.css.SACBenchmark</class>
        <caption>SAC benchmark</caption>
        <subcaption>Medium-large file (226 kB)</subcaption>
    </benchmark>

You can add as many <benchmark> elements as you like.

One should provide a description for the X axis:

    <x-axis-title>SAC Implementation</x-axis-title>

And you probably want to assign more readable descriptions (together with specific colours) to the methods in the benchmarks. For which you will use regular expressions in <method> elements:

    <methods>
        <method regex=".+SACParseStyleSheet$" color="#c04d4d">Css4j</method>
        <method regex=".+Batik$" color="#61c061">Batik CSS</method>
        <method regex=".+SSParser$" color="#4d4dc0">SS CSSParser</method>
    </methods>

Now, give a description to the units:

    <units>
        <unit-title unit-id="ops/s">Throughput (ops/s)</unit-title>
        <unit-title unit-id="ops/ms">Throughput (ops/ms)</unit-title>
        <unit-title unit-id="ops/us">Throughput (ops/μs)</unit-title>
        <unit-title unit-id="ops/ns">Throughput (ops/ns)</unit-title>
        <unit-title unit-id="s/op">Average time per operation (s)</unit-title>
        <unit-title unit-id="ms/op">Average time per operation (ms)</unit-title>
        <unit-title unit-id="us/op">Average time per operation (μs)</unit-title>
        <unit-title unit-id="ns/op">Average time per operation (ns)</unit-title>
    </units>

The document stores

The <storage> element declares the different data/graph stores (see the Configuration page for more details):

    <storage>
        <store id="fileStore" classname="io.sf.carte.report.FileStore">
            <directory>${java.io.tmpdir}/benchmark</directory>
        </store>
        <store id="documentStore" classname="io.sf.carte.report.DocumentStore">
            <pathname>${user.home}/www/css4j.github.io/sac-mark.html</pathname>
            <fallback baseuri="benchmark" format="png">
                ${user.home}/www/css4j.github.io/benchmark
            </fallback>
        </store>
    </storage>

• The element with a fileStore id tells that the directory ~/www/css4j.github.io/benchmark would be a FileStore containing the fallback images.

• In the element with documentStore id of that file, it is configured that the file ~/www/css4j.github.io/sac-mark.html is a DocumentStore where the SVG graphs will be put.

Test run

To produce the SAC tables and graphs on your own, all that you have to do is to download your copy of the sac-mark.html file and modify the paths in sac-benchmark.xml as necessary.

Then you could prepare a small script called sac-benchmark-charts.sh and put it in carte-jmh/examples:

#!/usr/bin/bash
CARTE_VERSION=0.1.0-SNAPSHOT
java -jar ../build/libs/carte-jmh-${CARTE_VERSION}-all.jar --config=sac-benchmark.xml "$@"

Now open a shell and execute it (again, in carte-jmh/examples which contains all the JSON files):

cd carte-jmh/examples
./sac-benchmark-charts.sh sac*.json

And you will reproduce the SAC JMH charts.

The examples directory contains a similar benchmark-charts.sh script that produces the DOM benchmarks instead.