geek!daily

Live Blog: Comet & Other Browser Stuff

It's a big component of what we call "real-time"

This is the kind of stuff that puts up a banner on a Twitter search ("# New Tweets Found") and makes things feel real time.

List of what's being done to move notifications from server to browser:
* Ajax polling: Anybody using the JS set-timeout function, then refetch. Used by GMail, Campfire, et al
* Long polling (typically Comet): Used by FriendFeed, keeps an http connection open as long as possible (keep alive), reopen when timed-out. Keep a local thread pool.
* Flash sockets: Same as long polling, but from Flash. Don't have the same-origin policy. No limitaions on number of connections. Does gzipping.
* Reverse HTTP: hosting a little webserver inside a JS connection. Long pushing?
* Silverlight ("MS's version of Flash")
* HTML5 websockets: part of spec, doesn't exist and not implemented. Expected to be like flash sockets.

(things discarded as too old/obscure/painful/absurd):
* Java applets (David Weekly is about to get himself kicked ;)
* Big ugly JS kludges (iframe tricks, etc)
* What's that Opera thing ... unite? ubiquity? Unite. Every browser is a webserver. Sounds like reverse HTTP.
* Using XMPP format, arbitrary JSON structure, Atom.

Libraries:
* Orbited (?) tunnels TCP thru HTTP, treats each end like a socket. JS front-end, Python backend. Often paired with Twisted. Specifically meant to be Comet.
* stropheJS: javascript, can use flash sockets
* Tornado: python
* Cometd: Java, made by Dojo to work with dojo
* Dojo: javascript, can use flash sockets
* APE project (?)


What formats are people using to send data:
* JSON
* XMPP

Apps and their libs/methodologies:
* Meebo: ??
* Google Wave/GTalk: GWT RPC, long-polling, their own JS
* FB Chat: ??
* FriendFeed: Tornado server with long-polling, their own JS
* Superfeedr: uses BOSH, stropheJS. BOSH is kinda long polling, bidirectional (two cs open all time). Very similar to comet, more friendly than strict proxies
* Collecta: uses BOSH
* Twingly: orbited
* PBWorks: long polling on network dashboard to see updates come in live; wrote their own libraries, also use stropheJS
* StatusNet - identi.ca and ??, orbited and cometd (you can do either)

Flex and Air apps? The most interesting stuff is what used to be called Flash Media Server (been renamed). They've got their own P2P protocol. All sorts of funky stuff you can do. OpenSource version of FMS is Red Five.

In what situations are these libraries breaking down?
* Transparent/Opaque proxies that give repeats/dupes/hangs. Keep buffer on server and check for these. Sometimes have to close connection to flush thru proxies.
* "Everyone focuses on the newest hottest event-based framework, but the hardest part is that HTTP wasn't designed for long-polling."

Have to hold open request/response pairs. Connection setup and teardown is expensive

Guy at UK telecom has only ~3K IP connections available in London area. Comet is going to force an upgrade of their hardware, which will be expensive. Real world constraints will always present.

Real-time at UI can be distracting at best, horribly annoying at worst. Charts work great, but text moving too quickly becomes hard. UX is challenging. Would be nice to add items while autoscrolling relative to the focussed item

When you want to add a pause button to your site, that's FAIL. But the hover-over-conversation to pause semantic is pretty good.

Go to jschat.com to see a bad resize.

FB pioneered notifications really well in UX. You get a "toaster popup" that doesn't disturb your screen and it fades after a few moments plus a bar at the bottom to persist the aggregated notifications/count. Really nice.

Growl is also mentioned as a good model. Is there interest in a notification aggregator with contexts in the browser? Meebo wants to go that way as a notification aggregator. Adobe wants to do this on your desktop. BrowserPlus also hits Growl. Prowl == Pushed Growl.

If everyone started using XMPP, it would increase the message load ("three bazillion individual messages"). No one is bundling XMPP updates; no reason you couldn't. ActivityStreams could also serve.

Why are people using Atom/JSON for this? They're so extensible, but when you extend them so far they're just as verbose as XML. "But we just don't like XML."

Would be nice if you could select things to pull out of the stream and hold onto.

Seems to be consensus that chat belongs at the bottom of the browser, append at end. Everything with a permalink (blogs, tweets, etc) gets added to the top (?). Seems to be related to height issues.

Infinite scroll vs. "More" button

Titlebar flash (Gmail and GTalk)

Some sites make a tiny flash widget to play a sound

Haptic: twitter dmesg to cellphone buzz

Notifications can become another form of info overload/noise

(We flailed at making a 1D/2D chart to represent frequency and value ... FAIL)

Phonetop apps are like desktop apps. Leah feels strongly that they're headed down-and-to-the-right and browser-ish apps are taking supremacy due to interoperability.

Apple's surprise: phonetop apps took off. (?) vs. John Gruber: "The most used app on my iPhone is Safari."

You can find the whiteboard pics in my Flickr stream.

Radio Open Source: David Weinberger

Sent to me by Christina, and leapt upon because (a) I really enjoyed his Library of Congress lecture, and (b) I'm feeling guilty for low blog entry count. From the summary of the April 26, 2007 edition of "Open Source with Christopher Lydon":

It’s hard to summarize his theory of everything in one sentence, but this is pretty close: “To get as good at browsing as we are at finding — and to take full advantage of the digital opportunity — we have to get rid of the idea that there’s a best way of organizing the world.”

I'm excited to see the buzz around David's latest book, "Everything is Miscellaneous," as I think he's very right. I'm also coming to the conclusion that taxonomies are merely folksonomies with a contextual hierarchy and which context might be shared. But I still have a lot to learn.

Link: Open Source » Blog Archive » Weinberger’s Miscellany.

Hey Apple, Let Me iWander!

[note: this is a repost from a long-neglected blog I started last summer while at SIGGRAPH]

I'm currently doing a work-from-hotel morning since there's nothing at the conference that lured me away this morning, listening to my rather eclectic ~4K song playlist on my iPod on shuffle. This leads to some interesting, occasionally jarring, boundaries between songs; it also got me thinking more about something I've wanted for a while.

If you're like me, you have moods. Sometimes my moods do an energetic sprint through hours of Pete Tong's Essential Mixes. Other times, they dance delightful undulations down the sandy beaches of Angelique Kidjo's Black Ivory Soul. Occasionally they strut, oozing an attitude of down-home Texas hip as only Stevie Ray Vaughan or Lyle Lovett can manage.

(We'll ignore the immature, pissed-off moods that stomp into Evanescence, shoot you the finger, and slam the door in your face. They just want attention. Don't give in.)

For me, the analogy is apt: music makes a path for me and, when I find I'm on the right one for the moment, I want to follow it until I come to a crossroad where another path seems more interesting.

I want to iWander.

If you think about it, it should be easy. Every song starts with some implicit metadata attached: title, artist, album, year, genre -- though the last is often argued over, especially out on the electronica/dance/techno/alternative fringe. Apple adds some more: your rating, grouping, playlists, last played, artwork, etc. I want to be able to add more, defining relationships between artists, songs, albums, etc. that exist only for me. I want to be able to change paths -- say, from playlist to album, then perhaps genre, etc. -- to follow my mood.

An example might go something like this:

• Texas Flood, Stevie Ray Vaughan, Greatest Hits
  Man, I loved living in Austin while he was there. More of him.
• Pride and Joy, Stevie Ray Vaughan, MTV Unplugged, Vol 1
  I remember when he did this at Austin City Limits. That reminds me of the time we saw Indigo Girls there ...
• Romeo and Juliet, Indigo Girls, Rites of Passage
  That was a good night; seeing them, Nancy Griffith, and Mary Chapin Carpenter on the stage was phenomenal
• This Shirt, Mary Chapin Carpenter, Hitchhiker Exampler
  Wow. For a cheap sampler of music from a mediocre cable show, this is amazing. Let's hear it all again.
  [...]
• Make It Easier, Indigo Girls, Hitchhiker Exampler
  Oh, yeah, they have a tune on this one, too. I must be in a mood.
  [...]
• Talkin' at the Texaco, James McMurtry, Hitchhiker Exampler
  He knows small-town like nobody's business, that's sure. Didn't David Garza do something fun like this?
• Big Stick, Twang-Twang-Shock-a-Boom, Me So Twangy
  Bummer they couldn't get along ... they were awesome.

...and so forth. Stevie Ray Vaughan could have just as easily branched to Robert Cray or Eric Clapton (great guitarists all and SRV died in a plane crash while on tour with both of them).

There's even a commercial angle: it's not legal for me to share my music with others, but give me a way to share my metadata and I can find other people who have similar but diverging tastes and find new music I'll probably like ... and, as a bonus, I can judge that likelihood by evaluating the nature of the association between the two ("Austin City Limits? That's a hick show ... ewww." or "David Garza was in a band? Wow!") and follow the paths that seem interesting through the iTunes Music Store and on down the sidewalk.

Hey, Apple, you listening?

Cleaning up old mail

Just a random thought as I start posting again: I want an easy way to clean up thousands of old emails quickly. Sure, I could leave them around, but frankly it just bugs me to know that I'm maintaining a heap of digital compost some of which I will never, not in a million years, find useful again.

The surest way would be to quickly tag any email I read and choose not to delete with some indicators about why I didn't just ditch it immediately. Evolution, the Linux Outlook-a-like, gives me a little of this via its Followup feature; Thunderbird didn't have a plugin to help with this when last I looked.

Still, there should be some helpful methods, such as a list of all the old mail's senders along with a count of how many they sent and expandable to see the titles. Let me easily select all or some of any sender's emails for deletion (for UI, think packages/subpackages in an installer). Others might be grading the importance of the old email according to length of thread, number of times I replied in the thread (indicating I found it important) and so forth.

And yes, I've got a lot of these to go through. =\

Leaftag: Tag files and stuff in Linux

Drew turned me on to Leaftag this morning. Looks wicked awesome for doing TagEverything (I need a a better name for that).

Tagging for the Linux desktop

Leaftag is a library and set of utilities for tagging files on the Linux desktop. It's a convenient way of organizing files in a non-hierarchical manner. Local files, websites, or anything with a URI can be tagged with one or more names and easily referenced by anything that supports Leaftag.

Libraries as Playlists

I love reading Dave Weinberger's blog; his excitement at the ways technology can augment the world mirrors my own. I love this post for the reduction of where Web 2.0 and ubiquitous metadata are heading us: "When all our works are digitized, a local library will be nothing but a playlist."

Think about that ... mash up Google Books with a video iPod and BAM -- it's a library in your pocket. Toss in Audible.com and it's even better.

I'm with Dave ... I can't wait!