ebay Architecure notes

From: http://netmesh.info/jernst/Comments/sdforum-ebay-architecture.html

This week, I attended a very interesting presentation by Dan Pritchett and Randy Shoup, both senior technologists at eBay, on eBay's architecture. Some of it was as I would have expected, other things were, shall we say, counter-intuitive. Here is a random collection of notes, with some special exclamation marks:

• 212 million registered users, 1 billion photos
• 1 billion page views a day, 105 million listings, 2 petabytes of data, 3 billion API calls a month
• something like a factor of 35 in page views, e-mails sent, bandwidth from June 1999 to Q3/2006.
• 99.94% availability, measured as "all parts of site functional to everybody" vs. at least one part of a site not functional to some users somewhere
• 15,000 application servers, all J2EE. About 100 groups of functionality aka "apps". Notion of a "pool": "all the machines that deal with selling"... Well over 200 databases.
• Everything is planned with the question "what if load increases by 10x". Scaling only horizontal, not vertical: many parallel boxes.
• leverages MSXML framework for presentation layer (even in Java)
• Oracle databases, WebSphere Java (still 1.3.1)
• split databases by primary access path, modulo on a key
• every database has at least 3 on-line databases. Distributed over 8 data centers
• some database copies run 15 min behind, 4 hours behind
• no stored procedures. some very simple triggers.
• move cpu-intensive work moved out of the database layer to applications applications layer: referential integrity, joins, sorting done in the application layer! Reasoning: app servers are cheap, databases are the bottleneck.
• no client-side transactions. no distributed transactions
• J2EE: use servlets, JDBC, connection pools (with rewrite). Not much else.
• no state information in application tier. transient state maintained in cookie or scratch database
• app servers do not talk to each other -- strict layering of architecture
• Search, in 2002: 9 hours to update the index running on largest Sun box available -- not keeping up
• Average item on site changes its search data 5 times before it is sold (e.g. price), so real-time search results are extremely important.
• "Voyager": real-time feeder infrastructure built by eBay.. Uses reliable multicast from primary database to search nodes, in-memory search index, horizontal segmentation, N slices, load-balances over M instances, cache queries