geek!daily

LiveBlog: The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search

Eric Schurman (Microsoft/bing), Jake Brutlag (Google)

Experiments

  • Server delays (MS and Google)
  • Page weight variance
  • Progressive rendering

They have platforms for experimentation which allow fractional experiments

  • Divide users into small buckets
  • use good methodology (control group, experimental group(s))
  • Way better than usability tests

Server Delays

  • Goal [missed all of this due to an IM. Lesson learned]

Results

  • No statistically significant change @ ~50ms delay
  • Observable and fairly linear impact on delays 200/500/1000/2000ms.
  • Time to first click took ~2x delay—theory: user has opportunity to get distracted

Google Search Delay Experiment

  • Varied type of delay, magnitude, and duration (number of weeks) per user group
  • Pre-header delay: pause server processing upon receipt of req
  • Post-header delay: pause after sending on header, but before sending results
  • Post-ads delay: (ads are structurally first in page, can render before search result) put ads in separate http chunk, delay between ads and search results

Results:

  • Measure average daily searches per user
  • 50ms pre-header delays show no significant impact
  • 100ms pre-head, 200ms post-heads, 400ms post-head, 200ms post-ads (and others) showed linear progression in decreased avg daily searches
  • Also saw increase in internally monitored “abandonment rate”
  • Active users are more sensitive
  • drop-off continued to trend down linearly beyond 4 weeks; effect becomes more pronounced over time, and additive—200ms and 400ms groups diverge more strongly
  • Stopped injecting delays at week 7; recovery was significant immediately, but not fully realized at week 12—there was still a drop in activity for these groups

Page weight experiments

  • injected incompressible comments into various places of page
  • varied size of comments from 5% of page to 500% (most of larger loads were below the fold)
  • small payloads weren’t worrisome (tho stat’ly significant)
  • perf suffered slightly, but was US only experiment; global exp planned, will likely show significantly larger drop in perf
  • Click metrics were hurt more than query metrics

Progressive rendering experiment

  • Goal: determine impace sending visual header before results
  • Build page in phases, send using HTTP 1.1 chunked transfer encoding
  • Results: Large improvement due to parallelization. Time to first click was ~9% faster, more likely to refine query, more clicks, more likely to page thru results

HCI may state that 100-200ms isn’t perceptible; it still has effect.

Getting something to your user quickly is more important than when they receive their last byte

Experimentation platforms make all this research and hard numbers possible.

blog comments powered by Disqus

Published

23 June 2009

Categories