HotSpot vs GraalVM CE & EE

With version 19.3 (released 2019-11-19) GraalVM supports Java 11. This benchmark compares the Community Edition (CE) with the Enterprise Edition (EE) and compares them both to a the usual HotSpot version of Java 11.

The JDK:s used are (as reported by JMH):

  • JDK 11.0.5, Java HotSpot(TM) 64-Bit Server VM, 11.0.5+10-LTS
  • JDK 11.0.5, OpenJDK 64-Bit GraalVM CE 19.3.0, 11.0.5+10-jvmci-19.3-b05-LTS
  • JDK 11.0.5, Java HotSpot(TM) 64-Bit Server VM, 11.0.5+10-jvmci-19.3-b05-LTS

(Odd that GraalVM EE is reported as “Java HotSpot(TM) 64-Bit Server VM”…)

The JDK:s are evaluated by executing 3 different linear algebra operations implemented in 3 different code libraries. This is done for a number of different matrix sizes ranging from 100×100 to 1000×1000. The exact same benchmark was recently done with AdoptJDK comparing the HotSpot and OpenJ9 VM:s (Java 13).

AdoptOpenJDK HotSpot vs OpenJ9

Operations

  • Matrix Multiplication
  • Solve Equation Systems
  • Singular Value Decomposition

Libraries

  • Apache Commons Math (ACM) v3.6.1 
  • Efficient Java Matrix Library (EJML) v0.38 
  • ojAlgo v48.0.0-SNAPSHOT 

Results

The charts show the number of operations per minute on the y-axis and matrix size (square) on the x-axis.

Matrix Multiplication

Solve Equation System

Singular Value Decomposition

Interpretation

First thing to realise is that any clearly visible line separation in the charts represents a significant speed difference, and there is (more than) a x10 speed difference between the slowest and fastest library-JDK combinations. Also note that differences between JDK:s (using the same library) are of the same magnitude as differences between libraries (using the same JDK). Choosing the right JDK is as important as picking the right code library – when doing this kind of math – and picking the right combination makes a huge difference.

The results here are entirely inline with what we’ve seen before. Today there are alternatives to getting “the” JDK from Oracle, and there are significant differences between some of those alternatives.

Here GraalVM EE performs very well – particularly on matrix multiplication. We’ve previously seen similar results with Open9 and Zing in other benchmarks, but they don’t consistently perform this well. It’s primarily the EE that performs well, and that’s a commercial offering from Oracle. At this point the CE is far from on par with the EE.

For now sticking with the proven HotSpot Server JVM seems to be the best alternative.

Benchmark Details

The tests were executed on a Google Cloud Platform Compute Engine: n1-highmem-4 (4 vCPUs Intel Skylake Xeon, 26 GB memory) running CentOS.

The benchmark code is here: https://github.com/optimatika/ojAlgo-linear-algebra-benchmark

The raw results (*.log and *.csv files) can be found here: https://github.com/optimatika/ojAlgo-linear-algebra-benchmark/tree/develop/results/2019/11/graal-v11