Wow and Flutter

Tonight I needed to find out if there were any rooms on Lingr that hadn't had a room image created for them. This should never happen, but some exception emails that were being generated indicated that it was happening.

In about 10 seconds, I whipped this up in the console:

Room.find(:all, :select => :id).collect(&:id).inject([]) {|r, i| 
    r << i if !Image.find_by_imageable_type_and_imageable_id('Room', i)
    r
}

Awesome

Lately we've been noticing slowness on Lingr's database. Some queries were taking a ridiculously long time to complete, even some COMMITs were taking half a minute or more. The slowdown seemed to be across the board, rather than isolated into a particular table or a particular query.

After much investigation, MySQL upgrades, etc, Kenn finally noticed something weird in the slow query logs.

We had many queries in there that referenced time ranges, like this:

SELECT * from events WHERE created_at >= '2008-01-21 00:00:00' ...

but occasionally, one would show up that looked just a little bit different, like this:

SELECT * from events WHERE created_at >= '2008-01-21 00:00:00 PST' ...

With the help of MySQL's EXPLAIN statement, Kenn quickly discovered that while the first type of query would use indexes, and scan only, say, 100,000 rows, the second type would scan all 20,000,000+ rows. Further examination of the output of EXPLAIN revealed that the second query issued 4 warnings. SHOW WARNINGS revealed the following:

Incorrect datetime value: '2008-01-21 00:00:00 PST' for column 'created_at' at row 1

YIKES! We misformatted a timestamp in a query, and as a result, MySQL was scanning the entire 20M+ row table every time.

I tracked this query down in our code to something like this:

UserMessage.find_all_by_room_id @room.id, :conditions => "created_at >= '#{@time.to_s}'..."

As it turns out, time.to_s renders the timezone by default. Rails has added a nice format specifier, so you can say time.to_s(:db), and we were doing this everywhere else, but had forgotten the (:db) in this one place.

MySQL was barfing on the timezone specifier, and silently falling back to scanning the entire table. To add insult to injury, this query occurred in the rendering of an RSS feed, so it was used not infrequently.

Lesson learned- be very careful in how you format queries involving datetime fields. Forget to use the :db format specifier, and you could suffer terrible consequences.

For the extra curious, I did manage to find a MySQL ticket that seemed somewhat related.

I had a great time today at Day Two of the Internet Identity Workshop. I attended most of the OpenID-related sessions, covering OpenID Authentication 2.0, OpenID Attribute Exchange, and OpenID Assertion Quality Extension.

Apart from listening to the really smart and personable people presenting, what I really enjoyed most was the Unconference structure. This is my first Unconference and I am really loving the amount of interactivity and idea exchange that comes along with it.

What I took away from today's sessions, at a very high level, was an increased conviction that OpenID really has it right. While extensions like Attribute Exchange and Assertion Quality are important and will make OpenID even more attractive to end-users, OpenID is first and foremost about authentication. And the great news is, OpenID authentication works today- this isn't a spec with a real-soon-now implementation- the implementations are out there now.

I've explained the user benefits of OpenID authentication to several of my non-technical friends and they all got it immediately. That's a great sign that, given a critical mass of sites that can use OpenID for authentication, the end users are ready and willing to move to these new user-centric digital identities.

I had a great discussion with Scott Kveton of JanRain today, talking about how sites with existing user databases can adopt OpenID and allow federation between existing accounts and OpenID identifiers. Based on that discussion, I hope to add OpenID support to Lingr very soon.

Tomorrow, I'm looking forward to Dick Hardt's talk on OpenID roadmap, as well as a talk by Joseph Smarr on integration of OpenID into sites with existing user databases.

Protocol Battle Royale (hold the cheese)

Today I attended Day One of the Internet Identity Workshop, 2006b. Today's half-day agenda consisted of introductory presentations by some of the key players in the space- Microsoft, Eclipse Foundation, Sun/Liberty Alliance, OpenID, etc.

Before I give my opinionated review, let's be clear as to the perspective I'm coming from- I'm a website operator who would like to support open identity standards, so that my users have to trust me less (that lowers their barrier to using the site), and so that I have to code less (that lowers my barrier to providing the site).

I couldn't care less about the technical details of the protocol. In the past, I would have cared a lot about the technical details, but that was before I quaffed the getting real juice. Now, I just want what's easiest for me, and what's most attractive to my users. I have adopted laziness as a key working strategy, and it's working out splendidly so far :-)

So here I am, the skeptical pragmatist, listening to presentations, trying to distill it all into a practical approach to what seem like, at points, competing solutions. On one side I see the SAML/Liberty Alliance gang with their proposals, and on the other side there is the OpenID/Sxip gang with theirs.

And then it really struck me- something looks awfully familiar here. Substitute J2EE for SAML, and Rails for OpenID, and the comparison looks pretty much the same. Whereas Liberty and SAML seem to be trying for a comprehensive approach which can serve almost any situation, OpenID and Sxip are instead opting for the simplest solution that does what users and developers actually want now. And you needn't look too closely to see where my sympathies lie in that type of battle.

OpenID has some outstanding issues, for sure, but things seem like they are really moving forward. For example, from what little I heard about it today, the new Assertion Quality Extension seems like a big step in the right direction.

The momentum seems to be clearly with OpenID now- I'm eager to learn more over the next two days!

I will be attending the Internet Identity Workshop next week in Mountain View. If any readers are attending, please let me know and let's meet up!

I've been thinking a lot about OpenID lately- I've even prototyped a Rails site that uses OpenID for authentication (it was quite simple using JanRain's ruby-openid gem). I'm excited to attend this workshop and learn more about what i think is an important emerging technology!

Later today, we'll be rolling out an update to Lingr that includes, among other things, support for MicroID. MicroID is a way to claim ownership of your assets on the web- in particular, with Lingr, every chatroom now includes a MicroID meta tag that declares the room to belong to the email address of the Lingr user who created it.

Lingr's use of MicroID continues our push to support emerging metadata standards, as in our existing support for the rel-nofollow and rel-tag microformats.

Softies on Rails, a Rails blog evangelizing to .NET developers, has posted an interview they did with us last week. If you're interested in some insight into how Lingr was born, check it out!

Someone pointed me to this article, written some months ago by Erik Bruchez. There aren't any comments on the post, and googling for it turns up very limited citations, so, apparently, Mr. Bruchez isn't well read. Which is good, because he's missed the point entirely about building international-capable web applications with Rails.

To be fair, the article is about Ruby, not Rails, but, these days, when you talk about Web 2.0, as he does in the article, you can hardly separate the two. So, while Mr. Bruchez is almost entirely technically correct in his criticisms of Ruby's lack of Unicode support, the criticisms that he makes have very little bearing on most Web applications built around Rails.

The point he makes is that, since Ruby natively regards strings as simply streams of bytes, you can't do things like split a string into words using the native String methods. This is completely true. He also makes the point that Ruby doesn't natively support i18n/l10n. This is also completely true.

The problem is, both of these points are completely moot for most web applications.

Rails is specifically designed for web applications that use a relational database for storage. Now, most relational databases have supported Unicode for some time now. And the fact is, for operations like searching and collation, database-driven web applications typically use the database for these operations- they don't do them in-core using whatever language they are written in. So, the fact that I couldn't (easily) search for "日本語" in a Ruby String object is really not a problem- the fact is, I would definitely use the database to do that for me. The same goes for collation- I wouldn't load hundreds of database objects into memory, then try to sort them using Ruby- instead, I would craft an appropriate ORDER BY statement. And I can tell you from experience that MySQL does a fine job of searching and collation in Unicode.

Over at Lingr, we've built a web application that supports any character supported by Unicode. You'll see lots of international text in our tag clouds, and in our chatrooms. you can use our search function to find rooms about "日本語" quite nicely, thank you. All this built on a language that has no native Unicode support- imagine that!

So, while Mr. Bruchez's criticisms may be accurate, they are not necessarily relevant.

Let me get this right out front– I love Rails

My days were mired in a endless succession of javac compile cycles, huge bloated IDEs, xml config file madness, build hell, etc. Then I found Rails, and I’ve never looked back.

I love everything about Rails- the overriding philosophy of convention over configuration, the extraordinary thought and care that has gone into the API, and the seemingly endless joy of discovering new aspects of the API or even the Ruby language itself. Beyond all the technical arguments that might make Ruby and Rails the right choice for a given web application, the most important point of all is that Rails makes happy programmers. DHH has said it elloquently in his presentations, and in the Optimize for Happiness section of Getting Real-

A happy programmer is a productive programmer

This is absolutely, 100% true. Programmers are, by nature, self-motivated and driven by what inspires them. If you put them on a new framework with as much ellegance and depth as Rails, their productivity will go through the roof.

I have yet to meet anyone who tried Rails and didn’t get the bug almost immediately. On the contrary- most of the sad stories you hear about Rails involve technologists who find Rails and fall in love with it, but then can’t convince management to take a risk on it. Well, I’ve got the perfect solution for that- become management.

So there you have it- I’m a complete Rails nut. So don’t expect any blog entries about the latest C++ or Java developments- I’m sticking to what I love.