转: C++ vs. Java 1.6 - A fair Benchmark

C++ vs. Java 1.6 - A fair Benchmark <!---->

For one of my small private projects, I wanted to know if I could implement the program in Java instead of C++. The only critical feature for this program was speed, so I needed to know how fast Java is today using the latest VM, compared to C++ using a modern optimizing compiler.
There are lots of tests and benchmarks available on the web, but usually I don't trust any statistics I did not fake myself. So I did my own benchmark with 3 simple real-world test cases and one additional academic test. The tests I found on the web have been mostly usesless, especially if you are programming games. These are the ones I used:

• Sorting a huge array (5 mio doubles) using Heapsort, testing usual, random memory access quite well.
• Calling multiple nested loops (4), each one 280 times, computing a number in the inner loop based on the loop counters.
• Copying an array (8 MB of doubles) into another one, multiple times (100 times).
• Calculating 44 fibonacci numbers, testing recursive calls. Totally useless test for real world programs, but because every benchmark seems to use it, it was interesting to include it as well.

I tried to be very fair because I really needed realistic results to base my decision on. You can get the source code of my tests here to verify or review my tests here: benchmark.zip. If you like, extend it or use it for whatever you like.

The results
To make it short: As I expected C++ outperforms Java in every single test. But interestingly Java is getting closer now, here are my average results:

	Java	C++
Fibonacci	10767 ms	7468 ms
Heapsort	5454 ms	4344 ms
NestedLoops	8022 ms	3453 ms
CopyData	1204 ms	1125 ms

Details and Findings
Knowing a bit about the C++ and Java optimizers, I was also able to revert the results completely. For example although using equivalent Java and C++ code, I was able to make the C++ version take 16 seconds for the nested loops-test while Java only needed 1 second by adding some virtual method calls which I knew Java could optimize. I also was able to make the C++ version run in only 78 ms while the equivalent Java version needed even 10 seconds by only using static loops - the C++ optimizer then would not even loop anything and just add the values together. This is partially what some of the wrong benchmarks out there are doing - even if not intentional. If you take a look at the sourcecode of my benchmark, you'll also see that I only measured the actual calculations, I was not interested in the start up times at all. This would have been very unfair because java needs some more time to start up, and it doesn't matter at all if an end user program (especially game) needs some extra seconds in the beginning. That's also the reason I used the client and not the server java VM: No end user is using the server VM at all. Another detail was interesting: Java was running out of heap memory very fast. I even had to reduce the sizes of the arrays in the tests, Java wasn't even able to allocate two arrays of several million doubles. Of course this wasn't a problem on the C++ side at all.

My conclusion: Java is not as fast as C++ although asserted by several people. Simply as that. But with today's incredible hardware, nothing will prevent you from writing simple games with it, unless you are not trying to create a fast Software Rasterizer or similar.

18 comments, already:

interesting stuff. java has gotten quite fast as it seems. at least in some areas…
bakkdoor (link) - 29 01 07 - 22:13

Many thanks for that.
Funnily enough I really can use that right now :)
I wouldn’t have thought using virtual mehtods is such a slow-downer… still I think the Java programs are still slow in comparement to C++. And I hate to live without default parameters or (at a smaller scale) without operator overloading.
I just grew up with that :)
Gauntlet - 29 01 07 - 22:21

Well nice figures. Thanks for that!

But I have to add that though being very accurate and more fair than other tests I saw already – this is only half the truth again. I would admit that it is correct to say java is slower than c++ when it comes to primitve datatypes and optimizing nested loops but there are real objects also where the numbers may change somewhat. In fact it will depend upon the tasks.

By the way: the server vm is used often on servers’s – that is what it is optimzed for. The client VM is faster for running on machines using swing ui. Would be interesting to do some benchmerks to see what happens if the server vm is to be tested with jirr – the java binding of irrlicht.
To make it short: the server vm is ahead of the client vm by some frames (1-3%) in the tech demo (running java 6) when using default values – though it is possible to tune the client vm to reach even higher numbers.
I just was asthonished to see a decrease within the fibonacci test duration by roughly 20% by using the server vm whereas the nested loops were signifcant slower.

Also please note that java is limiting its maximum memory usage to 128MB by default: -Xmx256m (or any higher number) will increase that … There is also a switch for the initial heapsize: -Xms. http://performance.netbeans.org/howto/jv.. lists some more paramters though with every new java version there seem to be more flags – please ask sun for any hidden ones

PS: Do not blame java for being slow on the client when it is the programmer’s fault. The swing ui is somewhat critical to running cpu intensive tasks without launching such a task in a separate thread. That will stop rendering of the gui until the main task returns. And guess what several coders are hacking in …
java protagonist - 29 01 07 - 23:55

I think that Java can be closer to C++ in big real world examples, but of course it is difficult to have more than an impression about that, as almost no big C++ app has been translated to Java. May be it is because it is very easier for an average programmer to optimize a Java program than a C++ one (in C++, you have also to take care of memory leaks, pointers, etc… this can be a lot of work).

The only two examples that I know of that have been implemented both in C++ (or C) and Java are http://www.bytonic.de/html/jake2.html (Jake2) ,a
Hervé ( <script type="text/javascript"> <!----> </script> email) (link) - 29 01 07 - 23:59

(sorry, some mistyping).
The only two examples that I know of that have been implemented both in C++ (or C) and Java are http://www.bytonic.de/html/jake2.html (Jake2), a clone of Quake2 in Java, and http://cmusphinx.sourceforge.net/sphinx4.. (Sphynx), a speech recognizer first coded in C, then in Java. Jake2 performance in Java 5.0 are identical to the original C versions (in some configurations), and Sphynx4 (in Java) performs better than the C version.

But of course, to be fair, I have to say that Jake2 still binds to a low-level C OpenGL wrapper, and the algorithms in the Java version have surely been improved for performance…
Hervé ( <script type="text/javascript"> <!----> </script> email) (link) - 30 01 07 - 00:05

@Hervé: C++ programmers who really have to care about memory leaks and memory management in general have completely messed up (and I’m not talking about using goehm-gc et al).
ak ( <script type="text/javascript"> <!----> </script> email) (link) - 30 01 07 - 00:23

next up, c++ vs. C#, the ultimate benchmark throwdown!!
that’s one i wanna see
buhatkj - 30 01 07 - 00:50

Yes me too Niko I wonder how C# would perform against C++ and Java… After all its only fair that someone did this…
leo ( <script type="text/javascript"> <!----> </script> email) (link) - 30 01 07 - 07:51

And it would be interesting to see memory consumption too :P
Raedwulf ( <script type="text/javascript"> <!----> </script> email) - 30 01 07 - 09:35

I’ve never heard of anyone claiming Java is faster than C++ – that would be pretty much impossible since there is still a layer of interpretation and garbage collection going on, no matter how well optimised the VM is.

Most of the time when I worked extensively in (server-side) Java I had to defend that Java was fast enough for the types if application that should be written in it. Most people judging it did it with either very old VMs or on application tests which were totally unsuited for the sort of application you would typically build in Java. It’s like saying scripting languages are slower than C++ when running tight loops – well, duh. Too many developers look at the minutae without seeing the big picture – that it’s worth trading a known amount of performance in certain cases for benefits in other areas.
Steve ( <script type="text/javascript"> <!----> </script> email) (link) - 30 01 07 - 12:10

here is one claiming java is faster: http://kano.net/javabench/index
erik - 30 01 07 - 12:36

This prooved Java is really fast… for an interpreted language. Of course it can not be nearly as fast as C++. But a C++ vs C# would be cool, yes :)
jasper - 30 01 07 - 12:59

It would be interesting to hear about compiler version and compilation flags. Moreover you should Java grant some startup time, so better inline several calls to the test function and average over the inner call times. Also the latest gccs still optimize the loops away with many optimization settings.
hybrid - 30 01 07 - 14:34

I used the VisualStudio 2005 compiler with the standard release mode settings. Hm, another test with different compilers/settings would be interesting as well, right :)
But I don’t think it’s completely impossible that java programs are faster than c++: Java has the advantage that it can inline virtual functions during runtime and also has the GC, which can compact the heap resulting in better caching where C++ is forced to work with a fragmented heap. Lots of applications can profit from this resulting in possible faster speed than C++.
niko - 30 01 07 - 18:13

Java Vs C# which one is better for making a 3D game.
leo ( <script type="text/javascript"> <!----> </script> email) (link) - 31 01 07 - 16:40

ok..c++ out performs java…i would like to know how fast c# is,’cos i am working on a project with c#..
tunde ( <script type="text/javascript"> <!----> </script> email) (link) - 02 02 07 - 12:48

You never mentioned which C++ compiler you’re using. That could make a big difference. Using g++ 3.4.4 on Windows under Cygwin with -O3 option, I get these times:

fib: 17341
heap: 4770
nest: 4628
copy: 1932

Here is the code to use the non-MS-specific gettimeofday() call:
struct timeval tv1;
struct timeval tv2;
cout
Andy Tripp ( <script type="text/javascript"> <!----> </script> email) - 07 02 07 - 03:57

I see many people are interested in c# results… so was I. Here are my results with java (1.6), c# (vs2005sp1, release), c++ (vs2005sp1, release) on Athlon 64 3200+ Windows XP machine:
java: fib: 9531, heap: 3640, loops: 12688, copy: 1500
c# : fib: 12671, heap: 3765, loops: 9406, copy: 1453
c++ : fib: 8750, heap: 3594, loops: 6328, copy: 1500
c# port is pretty straightforward from java… One more thing. None of this languages is interpereted… c# and java use just-in-time compiler, while c++ uses “classic” one. All results are surprisingly close to each other and all languages should now be usable for most purposes. (Someone could try compiling java source to native (GCJ)?)
bd ( <script type="text/javascript"> <!----> </script> email) - 12 02 07 - 17:34