Scaling an early stage startup

Scaling an early stage startup
by Mark Maunder <[e-mail]>

Why does performance and scaling quickly matter?
• Slow performance could cost you 20% of your revenue according to Google. 
• Any reduction in hosting costs goes directly to your bottom line as profit or can accelerate growth. 
• In a viral business, slow performance can damage your viral growth.

My first missteps
• Misconfiguration. Web server and DB configured to grab too much RAM.
• As traffic builds, the server swaps and slows down drastically. 
• Easy to fix – just a quick config change on web server and/or DB.

Traffic at this stage
• 2 Widgets per second • 10 HTTP requests per second. 
• 1 Widget = 1 Pageview • We serve as many pages as our users do, combined.

Keepalive – Good for clients, bad for servers.
• As http requests increased to 10 per second, I ran out of server threads to handle connections. 
• Keepalive was on and Keepalive Timeout was set to 300. 
• Turned Keepalive off.

Traffic at this stage
• 4 Widgets per second 
• 20 HTTP requests per second

Cache as much DB data as possible
• I used Perl’s Cache::FileCache to cache either DB data or rendered HTML on disk. 
• MemCacheD, developed for LiveJournal, caches across servers.
• YMMV – How dynamic is your data?

MySQL not fast enough
• High number of writes & deletes on a large single table caused severe slowness. 
• Writes blow away the query cache. 
• MySQL doesn’t support a large number of small tables (over 10,000). 
• MySQL is memory hungry if you want to cache large indexes. 
• I maxed out at about 200 concurrent read/write queries per second with over 1 million 
   records (and that’s not large enough).

Perl’s Tie::File to the early rescue
• Tie::File is a very simple flat-file API. 
• Lots of files/tables. 
• Faster – 500 to 1000 concurrent read/writes per second. 
• Prepending requires reading and rewriting the whole file.

BerkeleyDB is very very fast!

• I’m also experimenting with BerkeleyDB for some small intensive tasks. 
• Data From Oracle who owns BDB: Just over 90,000 transactional writes per second. 
• Over 1 Million non-transactional writes per second in memory. 
• Oracle’s machine: Linux on an AMD Athlon™ 64 processor 3200+ 
   at 1GHz system with 1GB of RAM. 7200RPM Drive with 8MB cache RAM.
   Source: http://www.oracle.com/technology/products/berkeley-db/pdf/berkeley-db-perf.pdf

Traffic at this stage
• 7 Widgets per second 
• 35 HTTP requests per second

Created a separate image and CSS server
• Enabled Keepalive on the Image server to be nice to clients. 
• Static content requires very little memory per thread/process. 
• Kept Keepalive off on the App server to reduce memory.
• Added benefit of higher browser concurrency with 2 hostnames.
   Source: http://www.die.net/musings/page_load_time/

Now using Home Grown Fixed Length Records
• A lot like ISAM or MyISAM 
• Fixed length records mean we seek directly to the data. No more file slurping. 
• Sequential records mean sequential reads which are fast.
• Still using file level locking. 
• Benchmarked at 20,000+ concurrent reads/writes/deletes.

Traffic at this stage
• 12 Widgets per second 
• 50 to 60 HTTP requests per second 
• Load average spiking to 12 or more about 3 times per day for unknown reason.

Blocking Content Thieves
• Content thieves were aggressively crawling our site on pages that are CPU intensive. 
• Robots.txt is irrelevant. 
• Reverse DNS lookup with ‘dig –x’
 • Firewall the &^%$@’s with ‘iptables’

Moved to httpd.prefork
• Httpd.worker consumes more memory than prefork because worker doesn’t share memory. 
• Tuning the number of Perl interpreters vs number of threads didn’t improve things. 
• Prefork with no keepalive on the app server uses less RAM and works well – for Mod_Perl.

The amazing Linux Filesystem Cache
• Linux uses spare memory to cache files on disk. 
• Lots of spare memory == Much faster I/O. 
• Prefork freed lots of memory. 1.3 Gigs out of 2 Gigs is used as cache. 
• I’ve noticed a roughly 20% performance increase since using it.

Tools
• httperf for benchmarking your server • Websitepulse.com for perf monitoring.

Summary
• Make content as static as possible. • Cache as much of your dynamic content as possible. 
• Separate serving app requests and serving static content. 
• Don’t underestimate the speed of lightweight file access API’s . 
• Only serve real users and search engines you care about.