Product Velocity

Rockets, Quizzes, and Killing Your Darlings: A Founder’s Guide to Product Velocity

Products we build as entrepreneurs have real value for people. But just like the stock market is a weighing machine and a voting machine, so too are our products.

Sexy growth hacks, one-time deals, and press mentions are all ways for the market to vote. But over time, products are a weighing machine. Folks will only continue to pay if you provide value. And product velocity is essential to providing product value.

If our startup were a rocket ship, the value that we provide is the distance we've traveled. Each product update moves the rocket (hopefully) in the forward direction, increasing your distance, increasing the value that you provide to users, and making your product more competitive.

Product velocity is how fast your rocket is going. The faster you make product velocity, the more value you provide users, the more users on the margin you'll be able to attract, and the more word of mouth from users that already love your product.

For StartPlaying there are natural outgrowths of the product that require engineering, design and product efforts to solve.

For example, the product can be hacked to play games in Spanish, French or German, (and I've seen them on the site). But it's inconvenient and hard to find. There are problems upon problems with playing for international players.

Product velocity, when geared towards these problems, can slingshot our rocket forward, dramatically increasing the product value, broadening the markets and creating products much more useful to many more people.

We spent years tinkering and overhauling the way we do product. And after four years of business and making over $50 million for our GMs, we finally got it right. Let me show you what we learned through the lens of a product we called Starter Set.

Starter Set began with an intuition, a gut feeling that we weren't serving the needs of new players.

This feeling came from two separate places. Devon, my cofounder, felt like new users would be best served with a different form factor - a quiz. A quiz allows us to ask players questions and based on those questions show them more personalized results.

I also felt when new users come to the site, they don't care too much about the specifics, and they might not even know what specifics to ask. They're really just looking to play a simple game of Dungeons & Dragons with a great Game Master (GM).

Based off our intuitions, we decided to do user research. Notice that we didn't just immediately build Starter Set. Instead, we used our intuition as a springboard and used research to validate and finetune it.

Our research was backed by The Mom Test. We tried not to assume. We didn’t cajole or wheedle users into answering questions the way we wanted. We got a real sense of what they would do and think.

We started off with a white-glove user interview. We placed an ad on Facebook targeting people that had heard about Dungeons & Dragons, and wanted to play in the next week (aka our target user group). We then hopped on a phone call and got to know them. We asked: Why they wanted to play? Who they were? And what their primary reservations were? Finally, we asked them for a credit card so that we could put them into a game.

We feigned charging them at the end of the process. And this was crucial. We had them put their money where their mouth is. And what’s amazing, people actually pulled their cards out. They wanted to play.

Now, it’s a little bit different when you're talking to a human being, you're more likely to acquiesce. But 90% of players were willing to pay to play.

Through this research, we learned the biggest reservations for players getting into a game. They wanted a game that could fit their schedule. They were nervous about playing with other experienced gamers. And they were unwilling to pay too much money.

Given all this information, we decided to do a secondary study. If this was a smaller product launch, we wouldn't. But because of the large upfront cost of building and facilitating a new product, we wanted to be sure.

We built out a front-end version of the quiz that addressed players' reservations. We showed them games with a variety of schedules, we reminded them throughout they would be playing with other new players, and we were upfront with the price of the game.

We recruited a whole batch of new players. And instead of walking them through, the players took our quiz themselves. And again, at the last step, they had to put in their credit card.

Around 60% of players took the full quiz and paid. The team was ecstatic.

This seemed like a great product innovation. Our site, which was geared towards gamers, could now serve a potentially larger audience. And we decided to build out this feature.

We did honest research. We hadn't put words into the mouths of our players. Instead, we diligently watched and assessed what they would do in a real-life situation. Then we built a product.

In building this product, we balanced two key tradeoffs. First, we wanted to build a minimum viable product (MVP). That means we wanted to build it as fast as possible. Because despite all our research, we were still unsure if Starter Set would attract players.

Building an MVP doesn’t mean building a crappy product. People won’t interact with a crappy product. They won’t buy. They'd think it’s fake.

So we had to balance between high standards and high velocity. And emblematic of the MO for important product launches, we focused on the happy path through the product.

We considered that 90% of our users would take the quiz, and 10% would somehow find their way into some edge case.

So we optimized for the 90%, the happy path.

We pushed the team for higher and higher standards on the happy path. The initial product was good enough, but not great. The team could do better. So we pushed for that extra 5%, and built something incredibly high-quality but only for the happy path.

And that's what an MVP is. It's not 14 or 15 features, poorly done. It's 1 or 2, built great.

I want to be specific about two product areas and the differing quality we put in.

We had an initial landing page that was the first thing all users saw. And we could have spent half a day on it, but instead we did seven revisions with custom graphics, custom typography, and custom code.

But compare that to our game monitoring system.

Players wouldn't see this. So we built a quick and sloppy admin dashboard on the backend. There was no automatic monitoring. It was manual, and it was tedious, but we didn't want to overbuild.

Our happy path was incredibly high quality, and we demanded the team give their best. Just like a great coach demands their team to push beyond their limits. But for products not visible to the consumer, we were okay with them being slapdash.

We had one last to-do before launch: measure, measure, measure.

Our key performance indicator (KPI) was purchases (or number of new players that joined StartPlaying because of Starter Set). So we only needed to measure purchases, right? Well, if we were lucky, yes, but otherwise we’d need more.

Consider two cases. First, we are blown away by our KPI. That's great! Problem solved, and we can just figure out how to scale up Starter Set. The second case, our KPI doesn’t move. We have no idea what went wrong. We are shit out of luck.

Believe me, we had messed this up before on other products. We had simply measured the KPI, and when we didn’t see positive changes, we had no idea why.

We thought Starter Set would fail for one of two reasons: either the quiz was not the correct form factor or we couldn't reach our market.

So we instrumented the whole quiz to understand what users were clicking and where they were dropping off. And then, and only then, after we had thoroughly tested and measured, did we launch.

When we launched, we launched with a bang. We looped in our partners and our GMs. We announced to players. We did PR. And we spent more on ads than any other point in the company's history.

Purchases started to pour in. The team was excited. But the product still wasn't doing well enough.

The point of Starter Set was to address a whole new type of player. If this user group was as big as we thought, we should have seen a huge spike in purchases. We didn’t.

Our KPI didn't move, but fortunately, we had measured each step of the process.

We emailed users that dropped off and asked them why. We also looked at the steps with the largest drop-off. With both quantitative and qualitative assessments made, we rapidly launched a new version of the quiz within a week.

We did a four-way A/B test. Each team member had their own intuitions. Some believed the shortest quiz would work, others that certain steps were unnecessary, and some thought even more steps would help the conversion rate. Because we were spending on ads, and we had instrumented so well, we should get results quickly.

And we did. We increased the conversion rate by a whopping 119% in two weeks.

If we had just, “launched the MVP” and not iterated, we would have missed a goldmine. This is why it's very important to do a full product rep.

Even if you do great research, your initial product won’t be exactly what users want. You'll need to iterate. You'll need to experiment. And you will likely improve the product to something far better than what was launched.

After the full product rep, we got Starter Set to a local maximum. Future research and A/B tests didn't find areas of large improvement, and we couldn’t optimize enough for just ads to drive adoption. The only way for Starter Set to work was to integrate with the full product.

But we were thoughtful here. We didn't rip out the bones of our site to integrate. Instead, we pursued our happy path approach, making sure the way 90% of users would enter Starter Set was clean and polished, even if the backend was a bit gnarly.

And Starter Set, a product we sunk months of time into, ultimately didn't improve our purchases.

Here's the part that I'm most proud of. Many teams would have stopped there. "Well, Starter Set's not harming anything, so may as well keep it." But we knew the value of focus. Focusing on a few core products that really work versus spreading ourselves too thin. So instead, we took the time to deprecate the product.

We spent months on this launch. Blood, sweat, tears and some great work were all put into making an awesome product.

But the product didn't work.

No matter how much research you do, if you are building on the edge, if you are building products for startups, some percent will not pan out.

The most courageous thing that you can do is admit that they didn't, cut your losses, and keep your team agile.

The work we did on Starter Set was some of the best work at StartPlaying and illustrative of high product velocity.

When building Starter Set, we focused on building Starter Set. All the creative minds at StartPlaying can often wander into other great ideas in different domains, but instead we focused each and every one on how we could move the needle here.

That allowed us to go further and faster with the same product velocity.

And there was only one person that could set that focus, the CEO.

Such a critical part of the CEO's role is to allocate resources, to make sure the right people are working on the right things. And it's so easy to get distracted, to spread yourself too thin, or to jump from project to project.

But so much of what worked with Starter Set only worked because of focus.

While the CEO needs to set focus, everyone else needs to participate. Research ideas, direction and execution can and should be spread out amongst the team. By owning one part of the project, by contributing to it, by putting at least one stone into the building of a pyramid, that project becomes yours. You're part of it. You're aligned to it. And your unique insights and wisdom can shine through.

The CEO needs to be curious, ask people questions and accumulate information. This does not mean you need to do what everyone else says. In fact, that would be a terrible idea. Instead, you need to gather as much disparate and diverse information from your team to make what you think is a right decision.

And here's the beauty, just by you making a decision, your company will be more successful than driving forward on whim, fancy or worst of all, democracy!

So focus the team, distribute the effort, collect and synthesize the results, and you decide on which problems we will solve.

Once you have problems, the solution experts on the team, designers and engineers, can begin to work. Product, engineering and design will all have perspectives on how best to build the product. Once again, it is your job to collect and synthesize them. Once again, better to make a wrong decision than no decision at all.

As CEO, you also need to hold your team to high standards. You need to be their coach.

I've had mediocre coaches before. They don't care. They'll pat your back. They'll smile. They'll say, "Good job," after you finish a race. The great coach that will push you beyond your limits, that will make you go from that easy 80/20 quality bar all the way up to plus ultra.

But being a great coach is hard because you constantly have to tell people no.

When you focus people on one idea, you end up saying no to all others. And when you push for higher quality standards, you say, "No, this isn't good enough. You can do better."

Exercising this ability to say no again and again and again becomes one of the most important skills you have as CEO. And, it wears on you.

I don't think it's necessarily human to say no all the time. There's some part of our makeup that believes in reciprocity. It believes that 50% of the time, you should be saying yes. But what you end up doing as CEO is saying no 95% of the time.

Finally, and arguably most importantly, you need to hold yourself accountable.

After a product launch, you could ignore the data and pat yourself on the back. Yes, you did it. You won. You built something new. Let's move on, let's keep doing the next thing.

That's the easy path.

The hard path is after you launch, you measure, and you see whether your hypothesis was right or wrong. Every product launch has a reason. Every reason has a hypothesis that will change some measurable outcome, and each hypothesis is either valid or invalid.

This is how you hold yourself accountable. This is how you be a great CEO. And this is how you impact product velocity. It's not a pithy statement. It's discipline times wisdom integrated over time.