Some simple bits and pieces about Java EE architectures and Web solutions.

Tuesday, December 27, 2011

JAXB, SAX, DOM Performance

This post investigates the performance of unmarshalling an XML document to Java objects using a number of different approaches. The XML document is very simple. It contains a collection of Person entities.

There is a corresponding Person Java object for the Person entity in the XML
...

and a PersonList object to represent a collection of Persons.

The approaches investigated were:

Various flavours of JAXB

SAX

DOM

In all cases, the objective was to get the entities in the XML document to the corresponding Java objects. The JAXB annotations on the Person and PersonList POJOS are used in the JAXB tests. The same classes can be used in SAX and DOM tests (the annotations will just be ignored). Initially the reference
implementations for JAXB, SAX and DOM were used. The Woodstox STAX parsing was then used. This would have been called in some of the JAXB unmarshalling tests.

Test 1 illustrates how simple the progamming model for JAXB is. It is very easy to go from an XML file to Java objects. There is no need to get involved with the nitty gritty details of marshalling and parsing.

Test 2 - Using JAXB to unmarshall a Streamsource

Test 2 is similar Test 1, except this time a Streamsource object wraps around a File object. The Streamsource object gives a hint to the JAXB implementation to stream the file.

Test 3 - Using JAXB to unmarshall a StAX XMLStreamReader

Again similar to Test 1, except this time an XMLStreamReader instance wraps a FileReader instance which is unmarshalled by JAXB.

Test 4 - Just use DOM
This test uses no JAXB and instead just uses the JAXP DOM approach. This means straight away more code is required than any JAXB approach.Test 5 - Just use SAX
Test 5 uses no JAXB and uses SAX to parse the XML document. The SAX approach involves more code and more complexity than any JAXB approach. The Developer has to get involved with the parsing of the document.

The tests were run 5 times for 3 files which contain a collection of Person entities. The first first file contained 100 Person entities and was 5K in size. The second contained 10,000 entities and was 500K in size and the third contained 250,000 Person entities and was 15 Meg in size. In no cases was any XSD used, or any validations performed. The results are given in result tables where the times for the different runs are comma separated.

TEST RESULTS
The tests were first run using JDK 1.6.26, 32 bit and the reference implementation for SAX, DOM and JAXB shipped with JDK was used.

Unmarshall Type

100 Persons time (ms)

10K Persons time (ms)

250K Persons time (ms)

JAXB (Default)

48,13, 5,4,4

78, 52, 47,50,50

1522, 1457, 1353, 1308,1317

JAXB(Streamsource)

11, 6, 3,3,2

44, 44, 48,45,43

1191, 1364, 1144, 1142, 1136

JAXB (StAX)

18, 2,1,1,1

111, 136, 89,91,92

2693, 3058, 2495, 2472, 2481

DOM

16, 2, 2,2,2

89,50, 55,53,50

1992, 2198, 1845, 1776, 1773

SAX

4, 2, 1,1,1

29, 34, 23,26,26

704, 669, 605, 589,591

JDK 1.6.26 Test comments

The first time unmarshalling happens is usually the longest.

The memory usage for the JAXB and SAX is similar. It is about 2 Meg for the file with 10,000 persons and 36 - 38 Meg file with 250,000. DOM Memory usage is far higher. For the 10,000 persons file it is 6 Meg, for the 250,000 person file it is greater than 130 Meg.

The performance times for pure SAX are better. Particularly, for very large files.

The exact same tests were run again, using the same JDK (1.6.26) but this time the Woodstox implementation of StAX parsing was used.

Unmarshall Type

100 Persons time (ms)

10K Persons time (ms)

250K Persons time (ms)

JAXB (Default)

168,3,5,8,3

294, 43, 46, 43, 42

2055, 1354, 1328, 1319, 1319

JAXB(Streamsource)

11, 3,3,3,4

43,42,47,44,42

1147, 1149, 1176, 1173, 1159

JAXB (StAX)

30,0,1,1,0

67,37,40,37,37

1301, 1236, 1223, 1336, 1297

DOM

103,1,1,1,2

136,52,49,49,50

1882, 1883, 1821, 1835, 1822

SAX

4, 2, 2,1,1

31,25,25,38,25

613, 609, 607, 595, 613

JDK 1.6.26 + Woodstox test comments

Again, the first time unmarshalling happens is usually proportionally longer.

Again, memory usage for SAX and JAXB is very similar. Both are far better
than DOM. The results are very similar to Test 1.

The JAXB (StAX) approach time has improved considerably. This is due to the
Woodstox implementation of StAX parsing being used.

The performance times for pure SAX are still the best. Particularly
for large files.

The the exact same tests were run again, but this time I used JDK 1.7.02 and the Woodstox implementation of StAX parsing.

Unmarshall Type

100 Persons time (ms)

10,000 Persons time (ms)

250,000 Persons time (ms)

JAXB (Default)

165,5, 3, 3,5

611,23, 24, 46, 28

578, 539, 511, 511, 519

JAXB(Streamsource)

13,4, 3, 4, 3

43,24, 21, 26, 22

678, 520, 509, 504, 627

JAXB (StAX)

21,1,0, 0, 0

300,69, 20, 16, 16

637, 487, 422, 435, 458

DOM

22,2,2,2,2

420,25, 24, 23, 24

1304, 807, 867, 747, 1189

SAX

7,2,2,1,1

169,15, 15, 19, 14

366, 364, 363, 360, 358

JDK 7 + Woodstox test comments:

The performance times for JDK 7 overall are much better. There are some anomolies - the first time the 100 persons and the 10,000 person file is parsed.

The memory usage is slightly higher. For SAX and JAXB it is 2 - 4 Meg for the 10,000 persons file and 45 - 49 Meg for the 250,000 persons file. For DOM it is higher again. 5 - 7.5 Meg for the 10,000 person file and 136 - 143 Meg for the 250,000 persons file.

Note: W.R.T. all tests

No memory analysis was done for the 100 persons file. The memory usage was just too small and so it would have pointless information.

The first time to initialise a JAXB context can take up to 0.5 seconds. This was not included in the test results as it only took this time the very first time. After that the JVM initialises context very quickly (consistly < 5ms). If you notice this behaviour with whatever JAXB implementation you are using, consider initialising at start up.

These tests are a very simple XML file. In reality there would be more object types and more complex XML. However, these tests should still provide a guidance.

Conclusions:

The peformance times for pure SAX are slightly better than JAXB but only for very large files. Unless you are using very large files the performance differences are not worth worrying about. The progamming model advantages of JAXB win out over the complexitiy of the SAX programming model. Don't forget JAXB also provides random accses like DOM does. SAX does not provide this.

Performance times look a lot better with Woodstox, if JAXB / StAX is being used.

Thanks for sharing the test results, every result set I've seen (including my own) has always shown SAX to destroy the competition. I think your analysis of the test results is flawed though.

First off, the JDK is caching things by evidence of the high results on the first run, then drastically reduced times on every test after that. Maybe should have created a different data file for each cycle of the test so that it wouldn't be able to cache. For example, first test would process a 250K persons file, next test cycle process a 250K animals file, then a 250K cars file, then a 250K train file, etc. Would ensure the JDK doesn't skew your test results, but still allows you to see the difference in the JDK load/preparation times of the first cycle vs the following cycles.

Second, you're not taking full advantage of SAX because you're storing every Person object in a List. The entire point to using SAX is so that you do not store the entire contents of the XML into memory, so you're getting a lot of benefits from SAX already, but the memory footprint of SAX should be near zero if you do it right, which should reduce the times even further for SAX while not requiring any memory to do the work. If you needed 1 MB of memory to process the 100 persons file, then you should still only need 1 MB of memory to process the 250,000 persons file.

Lastly, I think your statement is wrong when you say times for SAX are only "slightly" better and only for large files, so you shouldn't use SAX unless working with those large files. Even with the test results you show, SAX is at least 4x faster (or more) than every other type of parsing, and that is on the small files. The larger files SAX is a minimum of 2x faster. That sure seems like its more than "slightly" better. If I can be guaranteed a 2x to 4x improvement in speed by writing a bit more logic into my code and require no memory footprint to do it, I'd do that any day unless I'm absolutely sure I will not be processing more than a few records.

1. The time isn't always much higher the first time. In some cases there is very little difference. For example, JAXB(Streamsource), JAXB (StAX) using JDK 1.6.26 using 10K Persons.But yes I agree - in general - the JDK looks like it is doing someting special the first time. I'd argue that's a find in this analysis not a flaw. The other tests scenarios you mention sound interesting. There's quite a lot of possible test permutations here - impossible to cover everything. What is most important - I'm sure you agree - is that you cover the scenarios that are most relevant for your architecture. I will try to cover more permutations in future posts.

2. Yes the SAX test is storing persons in a list. But that was intentional. As stated in the blog post the objective was "to get the entities in the XML document to the corresponding Java objects". That's a different test scenario to when you are only interested in some of the entities in the XML. In this test, we are interested in *all* entities. That means they have to be stored somewhere. It might have being more complete if the SAX handler had a getter but that won't change any of the times.

3. By "slight improvement" regarding SAX what I meant was the absolute difference rather than the relative difference. If you add log4J to a method which is just a simple accessor you will probably double the execution time of it. This would just be a simple boolean check to see if log4J was enabled but still it takes time. If you remove the log4J you may cut down the execution time of that method by half but in the overall scheme of things you are probably not getting any significant improvement. This is because both the check to see if log4J and the actual returning of a variable take very little time.

Except for cases of unmarshalling very large files, in most cases, the differences between JAXB and SAX were around 20 milliseconds. This absolute difference would be even less on a production server. Remember the tests were just on my laptop. What I am suggesting is that 20 milliseconds isn't going to get noticed, unless you are doing a very time critical application.

One has to remember what your architecture is trying to do. It's extremly unlikely that it is just parsing XML for the sake of it. Usually something else is going to happen, for example data is about to be stored in a database or sent to the Web Tier. Whatever it is, if there is a performance problem with the use case it would be more beneficial to consider everything. Checking your database indexs or zipping your javascript may yield a more noticeable benefit.

My argument is that in most cases the progamming model of JAXB will be beat the performance of SAX. However, if your requirements is for saving milli-seconds than its SAX.

About Me

Alex Staveley is a software professional passionate about software
engineering and technical architecture. He blogs about architectural
approaches, Java topics, web solutions and various technical bits and
pieces.