geek!daily

Live Blog: Comet & Other Browser Stuff

It's a big component of what we call "real-time"

This is the kind of stuff that puts up a banner on a Twitter search ("# New Tweets Found") and makes things feel real time.

List of what's being done to move notifications from server to browser:
* Ajax polling: Anybody using the JS set-timeout function, then refetch. Used by GMail, Campfire, et al
* Long polling (typically Comet): Used by FriendFeed, keeps an http connection open as long as possible (keep alive), reopen when timed-out. Keep a local thread pool.
* Flash sockets: Same as long polling, but from Flash. Don't have the same-origin policy. No limitaions on number of connections. Does gzipping.
* Reverse HTTP: hosting a little webserver inside a JS connection. Long pushing?
* Silverlight ("MS's version of Flash")
* HTML5 websockets: part of spec, doesn't exist and not implemented. Expected to be like flash sockets.

(things discarded as too old/obscure/painful/absurd):
* Java applets (David Weekly is about to get himself kicked ;)
* Big ugly JS kludges (iframe tricks, etc)
* What's that Opera thing ... unite? ubiquity? Unite. Every browser is a webserver. Sounds like reverse HTTP.
* Using XMPP format, arbitrary JSON structure, Atom.

Libraries:
* Orbited (?) tunnels TCP thru HTTP, treats each end like a socket. JS front-end, Python backend. Often paired with Twisted. Specifically meant to be Comet.
* stropheJS: javascript, can use flash sockets
* Tornado: python
* Cometd: Java, made by Dojo to work with dojo
* Dojo: javascript, can use flash sockets
* APE project (?)


What formats are people using to send data:
* JSON
* XMPP

Apps and their libs/methodologies:
* Meebo: ??
* Google Wave/GTalk: GWT RPC, long-polling, their own JS
* FB Chat: ??
* FriendFeed: Tornado server with long-polling, their own JS
* Superfeedr: uses BOSH, stropheJS. BOSH is kinda long polling, bidirectional (two cs open all time). Very similar to comet, more friendly than strict proxies
* Collecta: uses BOSH
* Twingly: orbited
* PBWorks: long polling on network dashboard to see updates come in live; wrote their own libraries, also use stropheJS
* StatusNet - identi.ca and ??, orbited and cometd (you can do either)

Flex and Air apps? The most interesting stuff is what used to be called Flash Media Server (been renamed). They've got their own P2P protocol. All sorts of funky stuff you can do. OpenSource version of FMS is Red Five.

In what situations are these libraries breaking down?
* Transparent/Opaque proxies that give repeats/dupes/hangs. Keep buffer on server and check for these. Sometimes have to close connection to flush thru proxies.
* "Everyone focuses on the newest hottest event-based framework, but the hardest part is that HTTP wasn't designed for long-polling."

Have to hold open request/response pairs. Connection setup and teardown is expensive

Guy at UK telecom has only ~3K IP connections available in London area. Comet is going to force an upgrade of their hardware, which will be expensive. Real world constraints will always present.

Real-time at UI can be distracting at best, horribly annoying at worst. Charts work great, but text moving too quickly becomes hard. UX is challenging. Would be nice to add items while autoscrolling relative to the focussed item

When you want to add a pause button to your site, that's FAIL. But the hover-over-conversation to pause semantic is pretty good.

Go to jschat.com to see a bad resize.

FB pioneered notifications really well in UX. You get a "toaster popup" that doesn't disturb your screen and it fades after a few moments plus a bar at the bottom to persist the aggregated notifications/count. Really nice.

Growl is also mentioned as a good model. Is there interest in a notification aggregator with contexts in the browser? Meebo wants to go that way as a notification aggregator. Adobe wants to do this on your desktop. BrowserPlus also hits Growl. Prowl == Pushed Growl.

If everyone started using XMPP, it would increase the message load ("three bazillion individual messages"). No one is bundling XMPP updates; no reason you couldn't. ActivityStreams could also serve.

Why are people using Atom/JSON for this? They're so extensible, but when you extend them so far they're just as verbose as XML. "But we just don't like XML."

Would be nice if you could select things to pull out of the stream and hold onto.

Seems to be consensus that chat belongs at the bottom of the browser, append at end. Everything with a permalink (blogs, tweets, etc) gets added to the top (?). Seems to be related to height issues.

Infinite scroll vs. "More" button

Titlebar flash (Gmail and GTalk)

Some sites make a tiny flash widget to play a sound

Haptic: twitter dmesg to cellphone buzz

Notifications can become another form of info overload/noise

(We flailed at making a 1D/2D chart to represent frequency and value ... FAIL)

Phonetop apps are like desktop apps. Leah feels strongly that they're headed down-and-to-the-right and browser-ish apps are taking supremacy due to interoperability.

Apple's surprise: phonetop apps took off. (?) vs. John Gruber: "The most used app on my iPhone is Safari."

You can find the whiteboard pics in my Flickr stream.

Live Blog: Web Aggregation, What Works and What Doesn't

[note: I originally scribbled this on paper thinking I could hand it off immediately, preventing the obligation of typing, posting, etc. Turns out I don't get off that lightly, so here's the spew in electrons.]

Scraping isn't a scalable model.

There are biz issues around aggregating data: many businesses don't want you to get their data, though many are becoming more open.

Doing aggregation right:
* minimize latency
* maximize engagement

When latency is high, it causes confusion and takes you out of real-time

Doing conditional gets can be somewhat useful.

Plaxo had to shard their crawlers, which lands you in the shared state/sync problem of any stateful system you want to scale horizontally.

Gnip integration has been good:
* Offload the long-running processes
* Gnip offers alerting or "fat ping" (ping includes update data)

Plaxo likes using the alert to escalate the priority of the crawler which fetches the rich data related to the update. This approach allows you to use a consistent model for content ingestion vs. get info from fat ping, then augment later.

Smarr: "Brad Fitzpatrick said, 'Make polling a special case of push.'" He attributed this to someone but I missed the attribution.

(Don't try to keep up with Joseph Smarr on paper. He's thinks too many cogent thoughts too quickly to preserve legibility)

Plaxo uses TripIt's RSS feed as alerting, grabs item ID, then uses their APIs to fetch rich data.

There's a move to homogenize the info from sites, which may not be a good idea. It suppresses the distinctive look and feel/experience of the publishing site. Allowing for these differences means more labor spent on making one-off shims, which increases maintenance. Still, right choice in order to provide value to the user.

Activity streams seek to provide more rich data in a somewhat normalized, extensible format.

Many/most sites aren't yet perfectly architected for real-time's push, ping, etc.

PubSubHubBub and Activity Streams are externally represented data shards

Plaxo's Pulse started with known architecture issues (in order to ship) and hit the wall sooner than expected. Threw hardware/software optimizations at the problem to move the wall far enough to give time for rearchitecture, sharding, and working out how to propagate changes throughout the system properly.

None of the NoSQL alternatives are quite ready for prime-time. Smarr: "It should be something that's just a primitive."

Conversation platforms are slightly different sorts of aggregation platforms. There are UI diffs (e.g. pause the stream when indicating interest). Handling the transition from slightly-latent/passive real-time to synchronous real-time/active not yet well-developed (think: when a comment inspires a conversation)

90-99% of the value of the real-time web is realized in not-real-time [unreal-time? ;] This is a big deal for discovery. Twitter and FB make this harder by obscuring history.

Ideal scalability/performance would be an index per user. This would be grossly inefficient due to the number of duplicate entries.

No one has nailed reader-controlled aggregation (Show me Joe's tweets and blogs but not his photos) quite yet.

Smarr: "If we're all kinda [sharing], we're all making each other smarter"

The firehose of info is a hard model to scale to. Ben Metcalfe proposes the garden hose -- a firehose filtered at the source according to your interests, which helps aggregators by allowing them to request the superset of all filters from a given publisher.

We really want to push contexts to the publishers and let them determine which content fits that context. Context shifts over time: Joe doesn't normally read my tweets (and why would he?) but when we're at a conference together, he's much more interested (thus the popularity of hashtags). This is a geographic and purpose-driven context (the conference) as well as Joe's context on me (Jim knows where the good bars are).

Folks like Twitter are so overloaded with info that they might not recognize non-immediate contexts that are interesting to me.

There's also the risk of exposing users to the amount of correlatable public data they have. Many don't want you to apply a transitive closure to identify them in all spaces even though doing so allows you to present a much more convenient UX around what they want you to aggregate.

Someone likened the real-time aggregation problem to a bar conversation: you get snippets here and there and follow your own thread of interestingness.

Three fundamental themes:
* How to specify contexts to data provider/publisher
* How to control access to private data (and carry ACLs with that data)
* How to do all this efficiently

Plaxo implemented polling-back-off (poll infrequently updated sources less frequently). Turns out this is a bad idea, as it introduces latency which makes it feel broken.

There's also the issue of aggregating conversation about web objecs (like blog posts) and how not to divert the conversation from the publisher's site. However, sometimes you want a private discussion of a public object (cf. LinkedIn company groups discussing an article)

Q: What's the state of open standards around this?
A: PubSubHubBub and Activity Streams are very exciting. OAuth as access delegation. There's still a lot of ground to cover.

Reports of the Business Card's Death Are Greatly Exaggerated, or why bzCards != biz cards

Caught the article "rmbr launches mobile app to get rid of business cards" on VentureBeat which covers the announcement rmbrME, the new electronic connection tool from rmbr. (note: the rmbr.com site, which launched as a photo organizing site, currently redirects to rmbrme.com) The general idea is to trade vCards over email, IM, SMS, etc. and get rid of those pesky piles of pasteboard. If the ease of use isn't incentive enough, they're mixing in the funware idea via contests and leaderboards.

You may have guessed by now, but I think they've got the beginnings of a good idea and a poor implementation. I'm particularly amused by rmbr founder Gabe Zichermann's assertion that the business card's time has come and gone. Here's why I think he's wrong:

1. Business Cards Are Infinitely Customizable: I can quickly extend or personalize the information provided by my business card*. I can also correct bad information on the spot*, e.g. my title's changed, or I have a new phone number. And the only device required is any writing instrument.
2. Business Cards Don't Require Information Exchange: I don't have to ask you to give me contact info in order to give you my contact info. This preserves a level of anonymity which should not be undervalued. I wouldn't give contact info to everyone I've accepted a business card from; I've later chosen to contact some of those people.
3. Business Cards Are Trivial To Distribute: I can hand you a business card in just a few seconds, less if I'm already handing them out. I can place a stack of business cards in a tray on a counter to be taken by those interested without having to make any contact at all. I can drop them in the fishbowl of an excellent bakery and cafe in Wilson, NC in hopes of winning some goodies for the next time I'm there. I can hand it to someone who doesn't have a device. These examples all demonstrate the business card's continued practical utility.
4. Business Cards Are Static: They provide a clearly limited, time-sensitive set of information about me. If I change jobs, companies, phone numbers, email, or what-have-you, your data isn't current anymore. Effectively, the data ages and, in doing so, provides me some additional privacy with regards to those people with whom I've not formed a more permanent relationship than the original business card exchange (which, if I have, I likely want to provide a more dynamic link to my contact info). This is also true of a vCard exchange, but unclear with regards to bzCards. In my opinion, the lack of an analog to this aging process is a flaw in social networks which is becoming evident; it's currently all or nothing.

That said, I applaud the attempt toward a more dynamic contacts list and easier connections. There are some pieces still missing that, imho, are being overlooked by folks headed in that direction, but they'll come soon enough. And I'm utterly underwhelmed by the idea of competitions to send out the most business cards. It's quality, folks, not quantity.

* The business card with hand-written phone number comes from a 1997 Mother Jones article. Ironically, that lovely corrected business card image is from the "Why Use It?" page of MyDetails.biz, another digital business card replacement.

 

LinkedIn Mobile is Live!

It's been a busy few months since I joined LinkedIn, but some of that work is finally out there for the world to see: LinkedIn Mobile is now in public beta. You can get more details from the LinkedIn Blog.

Today's going to be an exciting day. =]

Data Portability In A Nutshell: You Own Your Data

I've already gotten pings (don't you people have other things to do? ;) about this morning's post on the LinkedIn blog, mostly around a more detailed explanation of the data portability goals with some privacy overtones, so I'll give a terse summary here:

Some useful jumping-off points:

• Data Portability "Action Packs" (aka summaries, executive and other)
• DPW Design Goals and Principles
• Work To Be Done/How To Contribute
• (Incredibly Rough Working Draft) Technical Blueprint

I'm only just getting caught up with what's gone before on this effort, so I don't have much more to offer than to say openness is good as long as it clearly respects privacy. It's that last bit about privacy which I think is getting lost in the thundering herd of press coverage, but that's what motivated me to get involved. It's my firm, personal belief that portability must account for privacy; you own your profile and your connection to me, but you don't get my profile and personal data in the bargain (unless I offer it).

While I don't agree with Mr. Howlett's title assertion, I absolutely adore the UK Data Protection Act (hey, Congress, you might have read this prior to the DMCA ...) and its intent and I'd expect it can and will be fully embraced in this effort; in that regard, I think Danny Ayers has some ideas heading down the right path, and in fact Robert Scoble had thoughts along these lines just after the debacle that shined a spotlight on data portability in the first place.

Context matters. Context always matters.

Me @ LinkedIn.com

Today is my first day as the manager of light engineering development at LinkedIn and I'm incredibly excited about it. Why's that, you might ask? Because LED is the group that's bringing agile development and Ruby on Rails to LinkedIn.

Who says Rails isn't ready for the enterprise? ;p

Live Blog: Graphing Social Patterns, Day 3, Opening the Social Graph

I took a session off to catch up on some other bits. Back in now.

OpenFaced panel: Opening Up the Social Graph - Tantek Celik (moderator), David Recordon SixApart, Chamath Palihapitiya Facebook, Joseph Smarr Plaxo, Ted Grubb Satisfaction Unlimited

Dave McClure "threw down the gauntlet" to Google and Yahoo: if you want to participate in open standards, show up.

[I've followed Joseph a bit as he's been talking about Pulse, a social network aggregator. While I don't know Ted, I know several other folks at Satisfaction via Christina. </disclaimer>]

Joseph led with a Pulse demo. Maybe that's what everyone will do, but I'm not diggin' it. Damn that boy talks fast.

David Recordin talks a bit about his role as kinda the gluemeister at 6A, standing among and between all the 6A products and looking at smart choices for them all. Also coming to this problem from the vox.com perspective. He also lent the laptop folks are demoing from.

Ted shows how Satisfaction is importing user profiles via microformats. MFs are supported all over the place. [me: we should add them to PS] [CW agrees]. Showed how to get the MF profile data from flickr. It also autosyncs, inheriting changes. [me: this lends an interesting model for connecting disparate graphs and makes the connections valuable]

Chamath got pinged on where he thinks open social graph fits ... and almost bit, but stepped back to present FBs take on social graphs. He asserts they believe the world has one social graph. FB has to make sure they make their version of social graph as open as possible (viz they had APIs for social graph data 6 months before FB platform). They intend to get more open over time. Urges to remember that they had to lay a foundation of privacy and respect before people would be comfortable sharing things like name, location, phone, AIM, etc. Scale of this effort is going to be in years, not months.

6A: David Recordon
FB: Chamath Palihapitiya
PX: Joseph Smarr
SU: Ted Grubb

6A: consider any obfuscated object (hashed email) same as the underlying object — if they don't want to share email, don't use hashed email as an identifier.

FB: If you can combine a good privacy model with a graph that remains relevant as it grows, you'll allow sites to remain relevant ... unless users decide to choose a different site. In the end, we're at their mercy. [me: as it should be]

PX: Everybody does better when users are in control of their information. http://opensocialweb.org/ User Bill of Rights [me: shades of Ira Glass's commentary about Ten Commandments] It's not a tech problem; it's more a social, political, privacy problem which is (hopefully) a better problem to have.

MOD: What are the problems for the OSG?

SU: Keeping it as simple and managable for the user
6A: Defining the problem is still the problem. We need to take a step back to see what consumers want. Dave Winer pointed out that they don't know what OSG means ... we need a consistent model, offline and online, which breaks it down in their terms (I have friends, etc.). Embody social conventions of the offline world: if you have the email address of someone in my contacts, you can have their phone number (you'd email me and ask, I'd say yes, etc.)

MOD polls the room for who's members of multiple nets, decreasing hands as he raises the number of nets.

MOD: How much does this matter?

[missed the answer. was short and felt unsubstantial]

PX: For the people I know on FB (a subset of all) I can get some information. Just like in the real world people want to opt in or out

SU: We want the conversations going inside Satisfaction, but we also want people [companies?] to be able to take those conversations to their own websites. We couldn't do that as a FB app.

FB: Info expires because we're custodians of the information. We don't own it.

6A: 24 hour timeout of data you get via the API for more than 24 hours. [me: be sure to segment and age that data]

FB: If they're engaged and using you regularly you'll always have that fresh info. [me: this sort of introduces aging memory to digital info]

Q: What are you all thinking about OpenID?
A: Nearly all do or are getting it running. SU is using MF data but not publishing it?

Q: What about the "Peace and Love" or "Platform Wars" Tim O'Reilly spoke of this morning?
6A: Ask again in a year.
FB: We intend to keep providing what it takes to empower developers to engage and please users.

Q: What does "open" mean?
PX: You're open when you free your data.
SU: Allowing users to own their data.
6A: Allow users to freely manipulate and remove their info.
FB: Enable users to decide how they interact with people.

Q: FB's API limits access to some data. That's at odds with being open.
FB: Our API is evolutionary. We want to keep our users data protected and iterate carefully.
6A: We're not yet sharing info via microformats as we don't think we've yet set the stage for users to understand that the info could be regarded as their identity and how it could be shared/used.

Q: Mark Zuckerberg said FB would deliver openID and microformats when users want it. Who wants it?
(lots of raised hands and pics)

Q: FB wants one social graph ... don't you have to open it up?
FB: We think there's one in the world, we're trying to model it accurately and efficiently.

Q: Bebo and others will likely release APIs which resemble FB. What do you think will be the open app model?
FB: Clearspring (?) already exists and embodies the model of build once publish many, which gets the details out of the way and allows companies to focus on engaging users.
PX: Less true of stitching the OSG across many sites. It's not a theoretical argument at this point.

MOD: What's next in terms of supporting OSG?
PX: Looking to help you syndicate your friends list across many sites.
SU: Implementing OpenID and syncing hCard
6A: Build in tools to allow our users to control who they interact with via their content.
FB: We want as much info to flow across the graph and touch everyone in the world.

[Interesting thought: mark all information with a value of how personal it is to you (think: slider with 3-5 presets, a la "more of this/less of this")]

Chamath was, I think, the most thoughtful and well-spoken. Joseph obviously has a billion ideas screaming around in his head waiting to get out. No slight to David or Ted implied or intended. And now I'm out, heading to see Joel Spolsky in Emeryville.

Live Blog: Graphing Social Patterns, Day 3, Facebook App Design: Elements of Style

Time for Jia Shen. He's a good speaker with dense content delivered well. This should be fun.

Jia Shen of RockYou!

RockYou does many popular FB apps as well as widgets all around; more than 10% penetration in all leading social networks.

FB apps are growing ~7x faster than widgets elsewhere.

Concept and Design

Think mathematically

Gauge target audience: get specific. how big is it? [me: he touches on TAM/SAM/SOM without knowing it]

Model which viral channels you'll be leaning into: invites? notifications?

Your app name is a viral channel: they add you, it's in their minifeed, newsfeed, and on left nav.

Why and how will it spread? How will people share it and share within it? Design your callouts from there.

Invites: for each install:

• User invites some people
• Some pct open invite
• Some pct who open invite install it
• Some pct who install will uninstall

...must be bigger than 1, more like 2. Over time, this will make growth.

Validate the use case -- typical users will use it?
Test call to action -- will they click?
Validate viral model -- any broken links?

Dev process should by hyper-agile with tons of instrumentation. Measure everything: invites, clickthrus, demos, minifeeds, etc.

Prepare for A/B testing. You're releasing fast, so you want to be able to change things around radically and quickly.

Site stats (google analytics and quantcast again)

Launch!

Phase it out there: make sure it's totally working before you make your splash (1-5K users will give good infor for tuning) then go big.

Promote on ad networks for guranteed performance and hitting your target audience. Tune the ad as you see results.

Ad nets on FB: RockYou (of course), SocialMedia, et al

Tuning

Validate the viral model: identify the total multipliers and look for weaknesses

Find new channels: how are users usint it? did you miss something? Integrate with other apps (viz. WereWolves, Vampires, Slayers, etc)

Tune your underlying channels: make sure your notifications aren't spammy, that you're targeting well, etc.

Monetizing it

Maintain comfortable growth and keep tuning, seeking more engagement. Encourage people to interact in your apps.

Engagement creates value for you to capture.

Some rehash of what I've heard from him before (no surprise) but some new stuff, and interesting to see his talk evolve.

Live Blog: Graphing Social Patterns, Day 3, FB Apps with ASP.NET, Silverlight, and Popfly

I'll be really thin here, splitting attention between session and some code. Also, it's more demo based, so this is largely my comments and inferences rather than Dan's talk.

Aiming to enable non-developers with Popfly.

Users are the stars. The "best" users should be treated like rockstars; get to know them and cater to them.

Embrace self-expression. Be personal.

Entertain. Evidently people want to earn points to buy virtual poop (example from a session I missed yesterday)

For devs: Free, client-side file hosting, works with open source AJAX. You can fully control the code.

The general interface is a node-based graph where you connect outputs to inputs; for my Anim/VFX friends, it's like Shake but with a white background and nodes look like spinny cubes. Ooo. ;p

You connect blocks and can see the results interactively. Good iterative development workflow.

It does dynamic resizing, so making it very small or very big is straightforward.

Building a block for it is pretty straihgtforward; you can clone others as a start, or start from scratch. It's just javascript underneath.

Also makes it easy for users to skin or mash things. Great engagement.

He shows you can use scriptaculous and, off-hand, says "it's an open-source library used on Apple.com" ... almost like a "see, it's not part of our empire, really"

All built on free frameworks (silverlight, et al) so cross-platform, cross-browser compatible.

Live Blog: Graphing Social Patterns, Day 3, FB Platform, Apps, Metrics

We're now 50 minutes behind schedule. Woot! Also, Dave McClure "dated himself" by mentioning back when he was programming VB was the rage. Made me chuckle.

Platform, Apps, and Metrics
Dave Morin, Facebook

FB is a tech company.

[I stepped out for a moment]

Concentrated pretty heavily in 18-24, but outside of that bias it's fairly in line with average demographics.

More than 50% users return daily. They like this; they want to be a utility. 60B pages per day, ~6th most trafficed site. They watch this but try not to pay too much attention to it.

FB doesn't want to own the social graph, just to map it as efficiently as possible.

They try to focus on allowing people to represent their identity online well.

Then, they try to make it really easy for people to communicate, both 1-to-1 and 1-to-many. It strengthens relationships.

They've grown ~3%/week. This has made the social graph stronger. The 43-millionth user gets the same utility from the social graph as the first [me: I'd argue more value, but he's being modest]

FB is the largest photo sharing site on the net, because it ties into the social graph.

FB events was done in a night. And it's more than 3x bigger than Evite.

FB Platform: Deep integration, Mass distribution, and New opportunities

What's it mean to be open when based on user data? [he left this hanging]

They want anyone to be able to build an app as good/bad as theirs.

Three integration points: Profiles, Full Pages (canvas pages), home page

[he walks thru the elements of profiles, etc. stuff you can get better explained in the wiki]

Best practices: (looking at photos)

context is king.

Canvas page: it's really important to represent the social context around your application. Show relevant, up-to-date content through the lens of your friends.

Content page: If a user adds a comment, they've bought in and are more likely to upload content, etc. Make sure you engage users as fractionally as they care to; make the threshold very low.

People aren't using the minifeed enough. Every time a user acts in some user/group's direction, put it on the minifeed.

Media in the feed updates (e.g. pix) drive engagement.

FB's opportunity composed of growth, engagement, and monetization. The platform makes growth and engagement as straightforward as possible.

fbFund was launched to help seed the community with grants without taking equity to help folks "get out from under the man"

How's it going: over 5K apps, over 100 new apps a day, over 90K developers, 80% of users have already adopted one or more apps.

He made good time; we're now only 25 minutes behind schedule.