Preaching User-Driven Design

My cofounders and I started a company in July 2007 very deliberately before we’d decided what product to build.  We had a problem that we wanted to solve -- that it was too hard to make and carry out every day decisions using the web.  But we were not committed to a particular solution. In fact, our CTO had compelled us to put off choosing an idea until 2008.  Having the right set of cofounders, with whom I had a deep history and highly complimentary skills, was the most important factor in our success.  But that commitment to not pick an idea for five months, no matter how alluring those early products were, was ultimately the second most important factor and represents a criticial principle for other startups to follow.

If you are going to be doing a startup for an average of four years if you are unsuccessful and eight years if you are successful, it makes sense to devote six months to deciding what to do in the first place.  If you go with whatever comes to you first, unless you are very lucky, you will likely waste far more than half a year before you can pivot to success.  More often, you’ll run out of money and morale and end up in the dead pool. 

When you don’t have any way of knowing whether one idea is way better or way worse than the ideas that will come later, it helps to burn some early concepts to establish a baseline.  But it’s incredibly frustrating to be an entrepreneur and come up with one idea after the next only to have people not use them.  It’s also very discomforting to not even have a name for your startup.  So, when October 2007 came around we adopted the catchy but temporary name “Launching September”, committing ourselves to coming out with something by a year later (ultimately the month an end-to-end Aardvark went live). 

But it’s better to spend a month on an idea than a year.  It’s also essential that you restrict yourself to the kind of products that you could reasonably prototype (i.e., no critical low-latency component and not highly visual and not for enterprise).  At Launching September (later to become Aardvark), we would fake functionality with people on the backend and see if our early users -- many who didn’t know they were using a faked prototype -- kept coming back.   When users don’t click through on an invite, they wouldn’t want your value proposition even if you’d built it. 

Aardvark was the sixth idea that we tried out.  But, we didn’t just stop at being user-driven in selecting an idea.  After we committed to the Vark, we spent nine months doing wizard of Oz testing.  In the last months, before we switched over to an automated system, we had eight people pretending to be the system (though real users answered questions).  In every interaction, a person would be classifying queries, routing questions, and managing conversations.  By the time we turned Aardvark on, we’d had tens of thousands of sessions from which to learn how the system should be built.  That end result, informed by actual usage, was materially different from where we would have ended up had we gone off into a garage with the concept and emerged with an implementation three quarters of a year later. 

Even once we had an end-to-end system up and running and didn’t have to involve humans on the back-end, we continued to test the vast majority of features manually before they were engineered.  We built a tool so that a team member could put users through alternate flows and experiences.  After each test, our designers would email anyone who had experienced the experiment.  Over the lifetime of the company, we consistently enjoyed more than a 50% response rate on those emails.  Moreover, the quality and quantity of feedback improved as users became less likely to be friends or family of people working at the company.

At the end of the day, being user driven is a tax.  I’d estimate that we moved about half as quickly as if we’d just gone with our gut consistently.  In return, we dramatically reduced the chance that we would make wildly wrong bets and have to double back, abandoning large periods of work.  Ultimately, investors gave money as much for our process as for our team and concept.  Being truly user-driven also created a much better environment for our engineers, who got to work on features that had been vetted prior to beginning development.  Finally, our process let us avoid the tantalizing product concept ratholes that could have used up years of our collective energy and time to no avail.
6 responses
Max, that is some amazing commitment to the user-driven process you guys showed, and clearly it paid off.

I think your "tax" analogy is pretty interesting. While the "technical implementation tax" -servers, licenses, sw dev methodologies- has dropped through the floor in the last few years, the success of a product still is and always will be tied to real people and the market. I Dave McClure makes a similar point (which sometimes gets him booed by developers ;)

Saw your panel at SXSW - really impressed with your approach to user focused search. It's kinda the holy grail for brands to be able to help people find answers from existing customers/fans.

I've often wondered why Google, doesn't use Facebook Connect to help personalize search based on your social context.

Hey, I just hopped over to your site via StumbleUpon. Not somthing I would normally read, but I liked your thoughts none the less. Thanks for making something worth reading.
hosting companies in india || hosting a website
Thanks for sharing this interesting blog. Really this blog post is very useful for me and informative.
2 visitors upvoted this post.