More Numbers (OpenDS and a hint at ApacheDS)

Some more numbers have come in. Howard continued over the weekend and into this morning. The last post was about benchmarking OpenLDAP 2.3.34 (OL) against Fedora Directory Server 1.0.4 (FDS). This post gives a glimpse of OpenLDAP 2.3.34 against OpenDS 0.1-34 (ODS) on the same hardware and software as FDS.

Authentication Rate performance of OpenDS was much closer to FDS than we expected; OpenLDAP was 3.7 - 3.9 times faster. OpenDS actually outperformed FDS at 2.5M entries (by a little). Load times and Search rate (see earlier post) were much worse (loading 247% - 265% the time for OL and 3.3 - 7.7 times slower search rate).

Oh yeah, Howard struggled to put ApacheDS's Release 1.0.1 through its paces. Let's just say that we agreed that enough was enough after a clean run at 250K. Load times - OpenLDAP 45 seconds, ApacheDS 3,051 seconds. Search rate: 1,689 entries/sec. Authentication Rate: 632 auths/sec.

First, OpenDS is to be congratulated on achieving almost as good authentication rate performance with their current code level as Fedora (descendant of iPlanet and Netscape's efforts) gets. We consider FDS performance-equivalent with Red Hat Directory Server (RHDS), the commercially supported Red Hat directory product and this bodes well for the OpenDS team.

The less said about the ApacheDS numbers, the better. The most important metric, Authentication Rate, was 4% of OpenLDAP and 14% of FDS. We wish the project well but it's evident from many of Howard's experiences with running this benchmark that there's lots of work to do on the code. For example, the load time was obtained using OpenLDAP's ldapadd tool, because using ApacheDS's bundled import tool would have taken pretty much forever. (It was only able to load 83,000 entries in 62 minutes, and the load rate was asymptotically approaching zero entries per second.) Remember, whatever else they do, computers are supposed to be fast, faster than humans. At these speeds, we could've carved the entry data by hand onto stone tablets faster than ApacheDS could load them.

Here are the graphs for the OpenDS versus OpenLDAP comparison. The OpenLDAP numbers are the same as in the (updated) FDS benchmark posting. Click on the images for a larger view.

 
Load Time

Note that smaller is better

 

 
Search Rate

 

 
Authentication Rate

 

... Marty and Howard