geek!daily

Live Blog: Comet & Other Browser Stuff

It's a big component of what we call "real-time"

This is the kind of stuff that puts up a banner on a Twitter search ("# New Tweets Found") and makes things feel real time.

List of what's being done to move notifications from server to browser:
* Ajax polling: Anybody using the JS set-timeout function, then refetch. Used by GMail, Campfire, et al
* Long polling (typically Comet): Used by FriendFeed, keeps an http connection open as long as possible (keep alive), reopen when timed-out. Keep a local thread pool.
* Flash sockets: Same as long polling, but from Flash. Don't have the same-origin policy. No limitaions on number of connections. Does gzipping.
* Reverse HTTP: hosting a little webserver inside a JS connection. Long pushing?
* Silverlight ("MS's version of Flash")
* HTML5 websockets: part of spec, doesn't exist and not implemented. Expected to be like flash sockets.

(things discarded as too old/obscure/painful/absurd):
* Java applets (David Weekly is about to get himself kicked ;)
* Big ugly JS kludges (iframe tricks, etc)
* What's that Opera thing ... unite? ubiquity? Unite. Every browser is a webserver. Sounds like reverse HTTP.
* Using XMPP format, arbitrary JSON structure, Atom.

Libraries:
* Orbited (?) tunnels TCP thru HTTP, treats each end like a socket. JS front-end, Python backend. Often paired with Twisted. Specifically meant to be Comet.
* stropheJS: javascript, can use flash sockets
* Tornado: python
* Cometd: Java, made by Dojo to work with dojo
* Dojo: javascript, can use flash sockets
* APE project (?)


What formats are people using to send data:
* JSON
* XMPP

Apps and their libs/methodologies:
* Meebo: ??
* Google Wave/GTalk: GWT RPC, long-polling, their own JS
* FB Chat: ??
* FriendFeed: Tornado server with long-polling, their own JS
* Superfeedr: uses BOSH, stropheJS. BOSH is kinda long polling, bidirectional (two cs open all time). Very similar to comet, more friendly than strict proxies
* Collecta: uses BOSH
* Twingly: orbited
* PBWorks: long polling on network dashboard to see updates come in live; wrote their own libraries, also use stropheJS
* StatusNet - identi.ca and ??, orbited and cometd (you can do either)

Flex and Air apps? The most interesting stuff is what used to be called Flash Media Server (been renamed). They've got their own P2P protocol. All sorts of funky stuff you can do. OpenSource version of FMS is Red Five.

In what situations are these libraries breaking down?
* Transparent/Opaque proxies that give repeats/dupes/hangs. Keep buffer on server and check for these. Sometimes have to close connection to flush thru proxies.
* "Everyone focuses on the newest hottest event-based framework, but the hardest part is that HTTP wasn't designed for long-polling."

Have to hold open request/response pairs. Connection setup and teardown is expensive

Guy at UK telecom has only ~3K IP connections available in London area. Comet is going to force an upgrade of their hardware, which will be expensive. Real world constraints will always present.

Real-time at UI can be distracting at best, horribly annoying at worst. Charts work great, but text moving too quickly becomes hard. UX is challenging. Would be nice to add items while autoscrolling relative to the focussed item

When you want to add a pause button to your site, that's FAIL. But the hover-over-conversation to pause semantic is pretty good.

Go to jschat.com to see a bad resize.

FB pioneered notifications really well in UX. You get a "toaster popup" that doesn't disturb your screen and it fades after a few moments plus a bar at the bottom to persist the aggregated notifications/count. Really nice.

Growl is also mentioned as a good model. Is there interest in a notification aggregator with contexts in the browser? Meebo wants to go that way as a notification aggregator. Adobe wants to do this on your desktop. BrowserPlus also hits Growl. Prowl == Pushed Growl.

If everyone started using XMPP, it would increase the message load ("three bazillion individual messages"). No one is bundling XMPP updates; no reason you couldn't. ActivityStreams could also serve.

Why are people using Atom/JSON for this? They're so extensible, but when you extend them so far they're just as verbose as XML. "But we just don't like XML."

Would be nice if you could select things to pull out of the stream and hold onto.

Seems to be consensus that chat belongs at the bottom of the browser, append at end. Everything with a permalink (blogs, tweets, etc) gets added to the top (?). Seems to be related to height issues.

Infinite scroll vs. "More" button

Titlebar flash (Gmail and GTalk)

Some sites make a tiny flash widget to play a sound

Haptic: twitter dmesg to cellphone buzz

Notifications can become another form of info overload/noise

(We flailed at making a 1D/2D chart to represent frequency and value ... FAIL)

Phonetop apps are like desktop apps. Leah feels strongly that they're headed down-and-to-the-right and browser-ish apps are taking supremacy due to interoperability.

Apple's surprise: phonetop apps took off. (?) vs. John Gruber: "The most used app on my iPhone is Safari."

You can find the whiteboard pics in my Flickr stream.

Live Blog: Web Aggregation, What Works and What Doesn't

[note: I originally scribbled this on paper thinking I could hand it off immediately, preventing the obligation of typing, posting, etc. Turns out I don't get off that lightly, so here's the spew in electrons.]

Scraping isn't a scalable model.

There are biz issues around aggregating data: many businesses don't want you to get their data, though many are becoming more open.

Doing aggregation right:
* minimize latency
* maximize engagement

When latency is high, it causes confusion and takes you out of real-time

Doing conditional gets can be somewhat useful.

Plaxo had to shard their crawlers, which lands you in the shared state/sync problem of any stateful system you want to scale horizontally.

Gnip integration has been good:
* Offload the long-running processes
* Gnip offers alerting or "fat ping" (ping includes update data)

Plaxo likes using the alert to escalate the priority of the crawler which fetches the rich data related to the update. This approach allows you to use a consistent model for content ingestion vs. get info from fat ping, then augment later.

Smarr: "Brad Fitzpatrick said, 'Make polling a special case of push.'" He attributed this to someone but I missed the attribution.

(Don't try to keep up with Joseph Smarr on paper. He's thinks too many cogent thoughts too quickly to preserve legibility)

Plaxo uses TripIt's RSS feed as alerting, grabs item ID, then uses their APIs to fetch rich data.

There's a move to homogenize the info from sites, which may not be a good idea. It suppresses the distinctive look and feel/experience of the publishing site. Allowing for these differences means more labor spent on making one-off shims, which increases maintenance. Still, right choice in order to provide value to the user.

Activity streams seek to provide more rich data in a somewhat normalized, extensible format.

Many/most sites aren't yet perfectly architected for real-time's push, ping, etc.

PubSubHubBub and Activity Streams are externally represented data shards

Plaxo's Pulse started with known architecture issues (in order to ship) and hit the wall sooner than expected. Threw hardware/software optimizations at the problem to move the wall far enough to give time for rearchitecture, sharding, and working out how to propagate changes throughout the system properly.

None of the NoSQL alternatives are quite ready for prime-time. Smarr: "It should be something that's just a primitive."

Conversation platforms are slightly different sorts of aggregation platforms. There are UI diffs (e.g. pause the stream when indicating interest). Handling the transition from slightly-latent/passive real-time to synchronous real-time/active not yet well-developed (think: when a comment inspires a conversation)

90-99% of the value of the real-time web is realized in not-real-time [unreal-time? ;] This is a big deal for discovery. Twitter and FB make this harder by obscuring history.

Ideal scalability/performance would be an index per user. This would be grossly inefficient due to the number of duplicate entries.

No one has nailed reader-controlled aggregation (Show me Joe's tweets and blogs but not his photos) quite yet.

Smarr: "If we're all kinda [sharing], we're all making each other smarter"

The firehose of info is a hard model to scale to. Ben Metcalfe proposes the garden hose -- a firehose filtered at the source according to your interests, which helps aggregators by allowing them to request the superset of all filters from a given publisher.

We really want to push contexts to the publishers and let them determine which content fits that context. Context shifts over time: Joe doesn't normally read my tweets (and why would he?) but when we're at a conference together, he's much more interested (thus the popularity of hashtags). This is a geographic and purpose-driven context (the conference) as well as Joe's context on me (Jim knows where the good bars are).

Folks like Twitter are so overloaded with info that they might not recognize non-immediate contexts that are interesting to me.

There's also the risk of exposing users to the amount of correlatable public data they have. Many don't want you to apply a transitive closure to identify them in all spaces even though doing so allows you to present a much more convenient UX around what they want you to aggregate.

Someone likened the real-time aggregation problem to a bar conversation: you get snippets here and there and follow your own thread of interestingness.

Three fundamental themes:
* How to specify contexts to data provider/publisher
* How to control access to private data (and carry ACLs with that data)
* How to do all this efficiently

Plaxo implemented polling-back-off (poll infrequently updated sources less frequently). Turns out this is a bad idea, as it introduces latency which makes it feel broken.

There's also the issue of aggregating conversation about web objecs (like blog posts) and how not to divert the conversation from the publisher's site. However, sometimes you want a private discussion of a public object (cf. LinkedIn company groups discussing an article)

Q: What's the state of open standards around this?
A: PubSubHubBub and Activity Streams are very exciting. OAuth as access delegation. There's still a lot of ground to cover.

Mozilla's TraceMonkey Avoids Trampling

Mozilla's TraceMonkey JIT Javascript compiler climbed down from the trees last week only to be somewhat lost in the thundering herd of Google's announcement of the Chrome browser and its accompanying V8 Javascript Virtual Machine.

Based on Andreas Gal's 2006 HotPathVM paper [PDF, 10pp] as well as his doctoral dissertation on bytecode compilation in VMs [zipped PDF, 138pp], TraceMonkey is offering speedups to rival those of the V8 JsVM. This is very worthy of note as both sides are open source and so can borrow learning and code from each other. Competition is good; open competition is even gooder. ;]

I'm not educated enough to offer in-depth commentary, so here's a quick collection of the announcements and discussions from those who are:

• Mike Shaver's original TraceMonkey announcement
• Brendan Eich's announcement, complete with benchmarks and metrics galore
• Andreas Gal's discussion of the underpinnings
• David Anderson's in-depth discussion of what trace-based compiling is
• InfoQ's interview with John Resig, jQuery creator and Mozilla's Javascript Evangelist

I'm starting to wonder if Chrome's accidental release wasn't as much a mailroom mistake as it was a tactic to not be an also-ran.

Google Chrome: The Next Big Thing?

A friend at work recommended I have a look at Google's explanation of Google Chrome as illustrated by Scott McCloud [via Google Blogoscoped where they were kind enough to scan the comic they received] and I couldn't wait. You see, Scott McCloud is the author of two of my favorite books, "Understanding Comics" and "Reinventing Comics" ... both must-reads if you're interested in comic history, the vocabulary of comics, or even just in thinking about how we communicate and interact with things on pages.

The Google Chrome strip [via Google Books] is an engaging 38 pages which lightly explains what they're trying to accomplish and how they've gone about it so far. Amusingly, it seems the comic went out a bit earlier than planned, but Google is taking things in stride, with grace and good humor, showing the rest of us how it should be done.

Some of the more interesting choices:

Webkit-based: Google Chrome is based on the Webkit browser engine, which is also used by Safari, Android, and a number of other mobile device browsers. Google's developers liked it because it's fast, memory-efficient, embeddable, easy to adopt, and generally keeps things simple.

The V8 Javascript Virtual Machine: written by "the V8 team in Denmark" specifically to address the shortcomings of previous JsVMs in a number of ways, including compilation to machine language and real garbage collection. I'm pretty excited about this, particularly that they've made V8 browser-independent.

One process per tab: They've implemented this as a protection from crashy apps as well as a way to clearly reveal what's chewing up resources in your browser. This concept also seems to be foundational to their anonytabs and anti-popup measures; it's definitely the basis of their sandbox security measures.

The New Tab Experience: When you open a new tab in Google Chrome, they assume you want to go somewhere rather than see your default page (or a blank page, which I've used for years to avoid unnecessary page loads). A nice UX touch that shows the depth to which they're thinking about this.

Gears Inside: It's no surprise that the Google Chrome browser builds in Google Gears, which has been aiming to bring the Web 2.0 experience even when offline for over a year now. I couldn't find any estimates of how widely installed Gears has become in the past fourteen months, but I'd guess it's low at present despite MySpace and Windows Mobile having jumped on board in the past six months.

There's lots of other good bits in the strip, like the anti-phishing measures page (which makes an analogy between malware and dog poop which couldn't have been made nearly so excellently in any other medium, imho) and a late appearance by Chris DiBona to sum up the open-sourcedness of it all.

I have some minor concerns about the extra resource consumption and possible need for interprocess communication (IPC) in their model. However, this is probably because my last significant experience with IPC dates back almost a decade to SGIs running IRIX. It was my experience then that IPC is hard to implement cleanly as well as the most frequent source of race conditions. It's likely times have changed since then. I hope.

The biggest disappointment? When the beta is officially released tomorrow, it'll be Windows only for now. The official blog post notes that they're working hard on versions for MacOS and Linux, too (and I suspect they would have liked those to be available on launch day), so when the Google Chrome project page lights up tomorrow, it won't be for the rest of us quite yet.

But I'll be eagerly watching, just the same. You don't see a new browser born every day.

Update: Paul Thurrott's WinSuperSite has screenshots purported to be Google Chrome [via Blogoscoped]

Data Portability In A Nutshell: You Own Your Data

I've already gotten pings (don't you people have other things to do? ;) about this morning's post on the LinkedIn blog, mostly around a more detailed explanation of the data portability goals with some privacy overtones, so I'll give a terse summary here:

Some useful jumping-off points:

• Data Portability "Action Packs" (aka summaries, executive and other)
• DPW Design Goals and Principles
• Work To Be Done/How To Contribute
• (Incredibly Rough Working Draft) Technical Blueprint

I'm only just getting caught up with what's gone before on this effort, so I don't have much more to offer than to say openness is good as long as it clearly respects privacy. It's that last bit about privacy which I think is getting lost in the thundering herd of press coverage, but that's what motivated me to get involved. It's my firm, personal belief that portability must account for privacy; you own your profile and your connection to me, but you don't get my profile and personal data in the bargain (unless I offer it).

While I don't agree with Mr. Howlett's title assertion, I absolutely adore the UK Data Protection Act (hey, Congress, you might have read this prior to the DMCA ...) and its intent and I'd expect it can and will be fully embraced in this effort; in that regard, I think Danny Ayers has some ideas heading down the right path, and in fact Robert Scoble had thoughts along these lines just after the debacle that shined a spotlight on data portability in the first place.

Context matters. Context always matters.

Sparky: The Alexa Toolbar for Firefox

Invaluable tool for quick info about any website you visit.

Link: Alexa - Alexa Toolbar.

"Beta" is the Web 2.0 "In Construction Guy"

I'm sitting in this year's Community Next Viral Marketing Conference where Jonathan Abrams (of Socializr) just noted that his web app "is gamma ... which is just like beta, but sillier." He's right; the beta badge has become the "in construction" of Web 2.0.

The Next Big Thing

If you've ever talked with Shel Israel (co-author of Naked Conversations), you've likely watched him strew the conversation with priceless observations pithily dressed so they'll cling to your grey matter like Paris Hilton somehow clings to her celebrity. I don't know that he's read "Made To Stick," but he's certainly tuned in to its message.

The one he dropped on the table today which made my brain explode, quoted as closely as I can manage:

"Computers lack common sense. [...] Integrating common sense with computing power [is the next big thing]."

This immediately rings true. The rise of social media, of crowdsourcing, of online communities are all based on finding ways to integrate common sense into the system. It's the thing people continually bring to the party that no amount of engineering will ever conquer (with apologies to Jeff Hawkins, many of whose aims I share).

All the better that PublicSquare fully embraces this; it's how I know we're on the right track.

A Change of Pace

One or both of you who read this blog may already know this, but today was my first full-time day with Cucina Media, we the makers of Public Square.

Today I:

• Fed my kids breakfast
• Saw them off to school (Beth took them)
• Talked with my partners about what the week holds in store
• Talked with our development partners about what we need to do in the next few weeks
• Met with a couple of prospective customers
• Talked at length with Christina about the focus of our next week, month, and quarter
• Chatted with Rob, a long-time friend and colleague who's just returned to the US after ~9 months in Asia
• Picked up Sam after pre-school
• Went grocery shopping with Sam in tow
• Cooked dinner for the family
• Played Elmo Uno with Beth and Sam
• Posted to my blog

It was an intense business day, with five meetings squeezed into six hours. We discussed everything from what needs to be done in the next week to the different opportunities out there for us and what it would mean -- to our company and to our values -- to take this or that one. Values. Objectives. What we want to do and why. Good stuff like that.

It was a very happy day. =]

Ajaxy Homepages

A nifty find via TechCrunch: NetVibes and ProtoPage, your personalizable ajaxy homepage. Toss in nearly any kind of content and neatly arrange or slop it around (respectively). Look for this on Google's personalized pages sometime later this evening or early tomorrow. ;]