More from Velocity 2009:

Jeremy Bingham,

Before the flood

  • fell over a lot
  • took a lot of admin time
  • slow load times

MySQL issues

  • legacy tables from nearly-decade-old Slash with bad primary keys
  • many MySQL 3.23isms—didn’t use any newer features
  • had to stop believing “things are there for a reason” (e.g. fulltext index: 9GB data, 17GB index)
  • keep all the old URLs working

IA Caucus first big night


  • Started with apache 1.3 as proxy, disk caching with push to disk
  • Brought site to its knees as everything updated cache in sync
  • switched to lighttpd using Vua, mod_magnet
  • switched to mod_mcpage, cache pages only


  • 10 2x Xeons 2GB, 2 2x Opterons 8GB, image/memcached/combined search/SMTP server
  • All independent, update separately
  • updated to 6 4x Xeons 8GB, 2 8x Xeons 16GB, RAID 10 with well-tuned xfs for DB

Traffic more than doubled at election peak over normal monthly, almost 3x

People liked to talk about Sarah Palin … a lot. How nice that she provided things to talk about.

Changes were in place by April 2008/Pennsylvania primary

  • perf was good
  • flash electoral map was on 100Mb switch
  • webhead loads ~0.5-1.8
  • Ads and map added some slowdowns

blog comments powered by Disqus


23 June 2009