geek!daily

Generate the regex for a TLD hostname from Perl

This was a quick, fun exercise to remind me that I can still write Perl. It fetches the list of TLDs from IANA, does a quick bit of munging, then renders a regex which should match any valid FQDN:

#!/usr/bin/env perl
use strict;
use warnings;

use LWP::Simple;

my $fqdn_regex;

if (my $content = get('http://data.iana.org/TLD/tlds-alpha-by-domain.txt')) {
  $fqdn_regex = '(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+(?:';
  $fqdn_regex .= join('|', grep (!/^(#|xn)/i, (split /\n/, lc($content))));
  $fqdn_regex .= ')';
}

my $regex = $fqdn_regex . '(?:\s|\/|$)';
print "$regex\n";

Several caveats:

• It doesn't match IPv4 dotted quad nor IPv6 ::-notation
• It intentionally ignores Internationalized Domain Name in Applications (IDNA) domains
• It borrows from my favorite reference for this, regular-expressions.info's page on email address regexes.

Maybe I'll extend it for completeness and/or rewrite it in Ruby someday. Until then, it'll always be ~/bin/tld_regex for me.

Generating with Ruby OFF Rails

I'm spending a fair amount of my time working on non-Rails Ruby projects lately and I've been missing the fine code generation toys in Rails. So tonight, for the fun of it and because I was hankering to play with cucumber, I hacked up the minimal set of bits that gets you script/generate in your non-Rails project.

This is how we do it:

> mkdir my_non_rails_proj
> cd my_non_rails_proj
> mkdir script config
> touch config/environment.rb
> cat > config/boot.rb
RAILS_ROOT = "#{File.dirname(__FILE__)}/.." unless defined?(RAILS_ROOT)
RAILS_ENV = nil
require 'rubygems'
require 'initializer'

Rails.configuration = Rails::Configuration.new
^D
> cp -p /path/to/gems/rails-2.3.2/bin/generate script/.
> ./script/generate
Usage: ./script/generate generator [options] [args]
Rails Info:
 -v, --version     Show the Rails version number and quit.
 -h, --help      Show this help message and quit.
[...]
> ./script/generate cucumber
  create features/step_definitions
  create features/step_definitions/webrat_steps.rb
  create features/support
  create features/support/env.rb
  create features/support/paths.rb
  create lib/tasks
  create lib/tasks/cucumber.rake
  create script/cucumber
> touch Rakefile
> rake features
[... large lack of FAIL ...]
>

Voila! Not guaranteed to work for all Rails generators, but it bootstrapped rspec and cucumber just fine, which was the point of my exercise. Someday I'll make a generator:bootstrap rake task or gem something if I need to do it again.

Re-finding my rubyligion ...

Life's been busy. Good, but busy. My gig at LinkedIn, two hours of commuting each day, plus our four person plus dog family certainly keeps me awash in things I should be doing. Thing is, ever since I stepped away from managing LinkedIn's rubyists, I haven't had much to keep me practicing my Ruby skills or tied to the Ruby community. That's made me a bit sad.

So, for the last few months, I've been finding spare-time, non-urgent Ruby projects on github to play with, forking them when I'm ready to make contributions. The one I'm currently twiddling most is a fork of Jon Maddox's IMDB library. He got the very basics in to deal with movies; I'm expanding it to handle TV series and episodes as well. In the process, I've added testing and some heavy refactoring, mostly to arrange the libraries in a more idiomatic fashion but also to swap out Hpricot for Nokogiri (sorry, why, but Nokogiri has better support for complex XPath and XSLT fun that simplifies some of the code dramatically).

I'm also digging into the guts of the Confluence4R and Jira4R libraries (and I am gratefully indebted to Ben Walding for setting up a github mirror of the Jira4R main git repo at Codehaus); for them, I'm mostly adding spec code to demonstrate what I think is true as I'm learning. Jira4R is driving me a bit nuts -- I'm having a love/hate relationship with its wsdl2ruby underpinnings. Slowly I learn.

But, then, that's the point, isn't it?

Cuzillion: 'cuz there's a zillion pages to check

Once in a while I get lucky. Most recently, this means that I was asked to write a blog post for LinkedIn about the tech talk Steve Souders gave for us last week: Even Faster Web Sites. Unfortunately, in the interests of brevity, I couldn't dive on something incidental to the talk which fascinated me: Cuzillion.

Cuzillion is one of those headslappers Steve seems to produce on a regular basis; it's a tool to model web pages and experiment with load order and methodology in order to better understand performance bottlenecks and find your way to a solution. He uses it to illustrate examples in his talks of late (as he did in the slides for his talk at LinkedIn), which highlights another feature: the ability to share a precise example among the team seeking a solution. It fascinates me because it's so straightforward and so obviously useful, yet you can twiddle it to great depth, willfully violating all the well-known best practices, and see clearly why that's not a good idea. It's like checkers: you get the basics instantly, and yet there's so much more there.

After playing with it for a while, it occurred to me that I very much want to say, "Cuzillion, go model the way this web page loads so I can play with it." When Steve announced Cuzillion, he mentioned that he intends to set up a Google Code repo for Cuzillion; when he gets around to that, it might be fun to spend a little time adding that feature.

Check it.

Old & Rusty to Shiny & New, Part 1

I'm feeling particularly masochistic today, so I'm going to tackle a project I've been putting off for a while: moving a Rails 1.2.6 app in our subversion repository into github, then updating it to Rails 2.1.

The first part, getting from svn to github, should be relatively painless according to github's "Importing From SVN" guide. It is, as long as you have already created your github repo (I hadn't) and already have git-svn installed (I didn't). So here's the expanded tutorial

1. Create your github repo

2. Be sure you have git-svn installed. I didn't, so here's what I had to do:

sudo port uninstall git-core
sudo port install git-core +svn

I had a bit of trouble with the serf port breaking during build time:

--->  Building serf with target all
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_www_serf/work/serf-0.2.0" && make all " returned error 2
Command output: /opt/local/share/apr-1/build/libtool --silent --mode=compile /usr/bin/gcc-4.0 -O2 -I/opt/local/include -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK -no-cpp-precomp -I. -I/opt/local/include/apr-1 -I/opt/local/include/apr-1  -c -o buckets/aggregate_buckets.lo buckets/aggregate_buckets.c && touch buckets/aggregate_buckets.lo
libtool: compile: unable to infer tagged configuration
libtool: compile: specify a tag with `--tag'
make: *** [buckets/aggregate_buckets.lo] Error 1

Error: The following dependencies failed to build: p5-svn-simple subversion-perlbindings serf p5-term-readkey
Error: Status 1 encountered during processing.

... so I got all medieval on it, building it manually, then finishing the port install:

> cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_www_serf/work/serf-0.2.0"
> head config.status
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by configure, which was
generated by GNU Autoconf 2.61.  Invocation command line was

  $ ./configure --prefix=/opt/local --with-apr=/opt/local --with-apr-util=/opt/local

## --------- ##
## Platform. ##
> sudo ./configure --prefix=/opt/local --with-apr=/opt/local --with-apr-util=/opt/local
[...generally successful configuration noises...]
> sudo make all
> [...generally successful build noises...]
> cd -
> sudo port install serf
[...generally successful install noises...]
> sudo port install git-core +svn
[...more generally successful install noises...]

3. Create an authors file

I made mine as authors.txt; it looks kinda like:

jim = Jim Meyer <jim@...>
joe = Joe DiMaggio<joe@...>

4. Run the commands from the github guide. My pseudo-values included below:

$ git-svn --authors-file authors.txt clone http://events-svn.example.com/ events_tmp
$ cd events_tmp
$ git remote add origin git@github.com:my_github_username/events.git
$ git push origin master

...é voila! It's done! It even (quite cleverly) recognized other authors and properly linked them. Very nice!

Using mysqldump with subqueries

I needed to dump some very specific portions of a large MySQL database; some of the data were second-degree relations (aka a Rails has-many-through). mysqldump won't allow a join in the --where clause, but it will allow a subquery, so I ended up with something that looked a bit like this:

mysqldump --lock-all-tables --where 'id in (select user_id from memberships where site_id = 123)' my_database users

Handy, that. Thanks to Austriaco who was doing this in a similar context.

OLPC: You Know, For Kids ... Theirs *and* Yours.

I just tripped over this video:

... which leads to the OLPC Laptop Giving website. The gist of it is that for $400 you can get an XO laptop for your kids ... and give one to kids who'll use it to change the world.

I think Sam's going to have a very happy holiday.

Live Blog: Graphing Social Patterns, Day 1, Geek Track, Late Afternoon

Despite being individually wonderful, fresh-popped popcorn and hot coffee don't go together at all. Bye bye, popcorn.

Jia Shen and Raymond (?) from RockYou! talking about launching and distributing your FB app. RockYou are the folks who brought Werewolves, Vampires, and SuperWall, as well as GlitterText.

Widgets for MySpace, Bebo, Friendster, are all like clothing -- self-expressive, no user interaction.

Facebook supports this, but also offers user engagement. It's like a MMORPG vs. the single-player games of the others.

Networks with engagement gives you ~7x more distribution.

(He showed quantcast rockyou.com web statistics)

RockYou is helping monetize now by offering an ad network on FB, both for use by developers to monetize their apps as well as seeking good matches between their apps and advertisers (e.g. Zombies & Resident Evil movie)

New application setup page walkthru:

Add FB Dev app, create a new app, *make sure it's checked on to allow folks to add it*. You can limit that to developers only further down the config.

Picking a good name: expressive, short, and catchy.

Callback URL is your server.

Interaction is: FB App page -> your callback URL -> returns text (FBML) -> FB puts it on the page

You can do iframe within FBML, so there's little advantage to doing iframe instead of FBML.

Developers of your app will be able to see debug messages (as comments in rendered HTML).

Post-Add URL: where you go when someone adds the app. Default is profile page; this is a great opportunity to move them into your app's design flow -- help them take the next step

Post-Remove URL: Isn't called in user space, but at backend.

Adding left vs. right: right side's wider, left will likely place you higher.

*BE SURE TO SET DEVELOPER MODE* They have a dev app with 300K users because they forgot. You don't want to deal with this.

Side Nav URL: set this! users like to nav to the app from there; if you leave it empty, you don't show up.

Private installation is good for dev mode -- don't announce you're adding your test/dev apps.

Make sure to have a banner/splash/screenshot for the app when you're ready to submit it. It shows on the profile page.

insert much configurational fun as we all make dirs on the rockyou dev server. If you want to see what my "app" is up to, it's here.

Each request sends the following pieces in the session:

• fb_sig_session_key: session key (hopefully infinite)
• fb_sig_user: uid
• fb_sig_added: boolean 1/0, is it added
• fb_sig_time: current time, float sec since epoch
• fb_sig_friends: comma sep'd list of uids

The wiki is a treasure trove. Keep it open all the time.

Tracking

You need to know you're tuning your app in the right direction. Here's some good stats to track:

• Infinite Sessions
• Page Views
• Installs/Uninstalls
• Invites Sent/Opened

If you have infinite session key, you can look at their info for as long as you're installed. Allows you to regen their profile regularly, etc.

Use Google Analytics, Quantcast to capture stats.

Not sure if FBJS allows Google Urchin yet.

Google Analytics doesn't manage to capture good stats (gets bad total uniques/page views) sometimes if you're doing "fancy things" where Quantcast is very straightforward, brute-force and rock-solid reliable.

Tracking via post-add URL: how are you getting installed? day-to-day growth is important for sensing how your user community is reacting. Set "next" parameter (&next=...) and send to something that reacts dynamically to that param.

Tracking via post-remove URL:  gets called at uninstall; just be sure you're being pinged by FB and catch the user id. You don't have to ditch their info, but note that they've uninstalled.

Debugging

You must have a dev app and a production app.

FB Dev blog explains how to make test accounts which won't be removed.

Use cases: They're looking at your app but haven't added; they've added; they're looking at someone's profile and have/don't have your app, .... Be sure you consider them all.

Look at HTML for debug info; it shows you everything FB set and sent in a comment.

Firebug: use the console to see the AJAX calls to be sure that the request went out right and got a good response.

Test on all browsers, including IE 6 & 7. Use Fiddler (Windows only HTTP debugging proxy) to see all the calls in IE.

Be sure your browser isn't caching content you expect to be updated.

MS Script Debugger for debugging JS. Allows you to break into JS and step thru.

File logging, email notifications in your app: you'll see invisible errors and track production issues.

Invites

FB gives multi-selector for picking your friends.

fb:request_form tag gives you an easy

After the user clicks the action in invites (either to do it or skip it) you can take care of them (preview, thank, etc). FB tells you who's been sent an invite (friend_ids)

Profile FBML is cached at FB, as are images. You can use API to invalidate cache data.

FB:Refs are also cached.

 

FB carries the bulk of the load. Usually scaling is a DB issue on your side.

Amazon EC2 and S3 are very useful for doing big FB apps.

And then we ran out of time. Jia and Raymond ran a tight session. It kept moving, they actively made sure everyone was moving along well, and I actually got an app running. They kept it really simple, from Hello World to some basic essential concepts, no DB, nothing unnecessary. An ideally structured hands-on workshop.

Live Blog: Graphing Social Patterns, Day 1, Geek Track, Post-Lunch

Ahhh, there's nothing like a conference box lunch. Thankfully so.

The example app Tyler wrote to share: Why Are You Awesome? (and source and db schema ql) (will also be posted at Tyler's blog)

Store session keys in db and you can use them as unique user ID for stats, being sure the user's got the latest update of your app, etc.

All the internal FB API calls are built on top of FQL, so writing your app in FQL where possible will give better performance. Ari: "The goal of FQL is not to hide any of the complexity of what we're going to have to do [...] Just to be clear: these queries aren't just run on a DB server; they're processed first."

FQL supports nested selects; they're resolved as individual ops, innermost to outermost.

Template stuff in a separate file as just FBML. (fbml.inc.php)

Interesting files:

• index.php: the main landing for users
• logic.inc.php:
• fbml.inc.php: template stuff including mock AJAX

Mock AJAX can be done from a link with three properties:
<a clickrewriteid="<div_to_rewrite>" clickrewriteurl="<query_url>" cickrewriteform="<form_from_dom_to_send_data_from>" ...>FOO!</a>

Don't call require_login for mock AJAX calls. Duh.

Mock AJAX returns are quite literally pushed into that div, errors included. Catch exceptions aggressively.

Security-wise, don't access the fb_sig_{user,profile} . You can use Firebug to check how secure your calls are ... if you could use Firebug to change the internals of the call and post as someone else, etc, that'd be bad, now, wouldn't it? (Tyler did this once to Superwall a bunch of people as Mark Zuckerberg, inviting them to go shopping for NorthFace jackets. Go figger.)

Make a legit FB test account each for your dev, staging, and live apps. Make your dev, staging apps visible only to you or only to a network of folks who're helping you. You really don't want your dev server buried by viral adoption.

Tyler pushes with rsync and --exclude fu to keep things configurationally close.

Integrating with your existing app:

• Add a column to track their FB id with your user id
• Notice cookies (FB, yours, etc) and let that influence your app's behavior

Profile Action Links (the links beneath profile pix) is more a function of the viewer than the person whose profile is being viewed.

Remember that co-developers can remove you as a developer. Choose wisely. You're limited to 5 other than you.

All in all, was less of a "Let's make an app!" and more "Here's a tour of an app." That's fine; I wasn't looking forward to reliving PHP4. Next: Launching and distribution

Live Blog: Graphing Social Patterns, Day 1, Geek Track, Pre-Lunch

I'm at Graphing Social Patterns for the next three days; it's a conference revolving around the Facebook platform. I've watched Giles live-blog and he always does it so well (and I always appreciate it when he does) so here goes:

Presented by R. Tyler Ballance of Slide, Ari Steinberg of Facebook

Useful URLs:

API console and FBML console, which returns to you exactly what your app will see: http://developers.facebook.com/tools.php

They just added bugzilla: http://bugs.developers.facebook.com/
Developer FB App: http://www.facebook.com/developers/
Developer docs and wiki.developers.facebook.com

Subscribe to the platform status feed to keep up with what's working/broken at the moment: http://www.facebook.com/feeds/api_messages.php

IRC: #facebook @ irc.freenode.net

Use JSON. It's much more compact compared to XML (~3x in some common cases), keeps bandwidth costs low and interactive response high.

FB is mostly PHP.

Pros & Cons of FB app langs:

PHP:
Pro: super easy to write, officially supported, wide variety of hosting
Con: Only does XML with FB; PHP makes it easy to write crap

Python: (PyFacebook API)
Pro: supports JSON or XML; integrates with Django easily
Con: hosting folks don't often host mod_python; uses simplejson by default (cjson is 70-80% faster); Python has a steeper learning curve

Ruby: (rFacebook API/lib)
Pro: RoR scales well; RoR encourages clean, readable code; RoR encourages testing; "Very Web 2.0"
Con: Ruby has an even higher learning curve; rFacebook isn't as RESTful as it could be; "really easy to write bad code in Ruby", "rFacebook doesn't quite conform to Ruby coding standards"

.NET: (Mono)
Pro: A variety of existing UserControls; usable from C# or VB.NET; tightly integrated with Visual Studio; supports LINQ (new feature in .NET 3/IIS); *very* well documented (70pp of very detailed docs)
Con: IIS or mod_mono/Apache2 (latter not fully supported); some aspects require .NET 3.5; very "enterprisey" and complicated (separates biz logic from display layer, etc); tightly bound to XML.

Other libs: ColdFusion, Perl, Java, C++

FB Platform is moving so quickly be sure that your chosen client lib can keep up.

Ari: if you know what you're doing, you can crank out your own FB API lib in The Language Of Your Choice in ~1-2 hrs.

Example app: "Why are you awesome?" written in PHP4, FBML-based, with only one request for info from FB.

The user needs to be called to action -- they need to add info, do stuff, manage something, or something else in order to be engaged. Find intriguing social contexts that engage folks.

Work hard to stay within FB's UI look and feel to increase user comfort. Compare Friends is an excellent example of going far, far away from it.

Make sure to add notifications to increase awareness of your app. Make sure they're not spammy, or people will mark them as spam and you'll be restricted to being seen only by your app's users.

Brevity is the soul of wit.

Use Mock AJAX to provide some cool interactivity.

There's a limit of 40 notifications per day, 10 user-to-user invites per day. Ari: "Ideally, we'd like to have the limits not exist [and everyone to behave sensibly] ... we just have the limits there to keep things from getting out of control."

FBML-based apps:

• Don't have to fetch tons of info (reduces your load). Free gifts is installed ~5M users, 500-900K daily users, and one server.
• Fewer FB API calls == less exposure to bandwidth latency, etc.
• Loads about as fast as the rest of FB, very responsive feeling.
• FBBYB: FB bugs become your bugs (and users don't make any distinction)
• FB has had issues with unicode
• Keep FB's bugzilla handy all-the-time

iFrame/HTML-based apps:

• You've got nearly absolute control (some browsers may not allow iFrames to set cookies, so do a page reload)
• Easier to port existing web apps
• Only constrained to 696 pixel width and terms of service (which you really need to read, particularly about what you can do with data or persistence).
• You have to call the FB API (lots), which can increase load on servers
• You don't have access to FBML
• You can totally hose the user experience (viz. scrolling iframes, bad self-written controls)

General Tips:

• Pick what you're most comfy with -- what you can ship fastest
• Subscribe to dev feeds; monitor FB's changes often.
• Use JSON.
• You can get an infinite session key using require install (vs. require login).
• require add will force the user to add the app before interacting, but they may not opt in just to test things

Let there be lunch and app-hacking afters.