Aprizi has been in open beta for six weeks now. These last six weeks have been an intense blur, a fire hose of information, and going open beta was the best possible thing we could have done. We are making some fairly big near-term changes because of this process. Here are 8 thoughts on the latest phase of our customer development process (see parts 3, 2, and 1).

For those new to the blog, Aprizi is building a “Pandora for shopping”. We are creating an entertaining, personalized experience that helps people discover cool new products and stores, especially those located slightly deeper into the “long tail” of ecommerce. While many things are currently being changed, you can play with the first version of our “minimum” beta at aprizi.com.

1. Interact with Customers However You Can
At this point, 400 people have used the beta and that has been a solid sample to accomplish our “learning” goals. We have talked with beta users over email and phone, and have physically watched over their shoulder as they have used the site. We wanted to answer some core questions: Do people care about what we are creating? What aspects of our MVP (and vision) are getting the strongest reaction? What will get us to product-market fit?

For the first several weeks, I tried to email every single person who signed up for the site. I asked them to get on the phone with me, or at least answer some questions over email. When it comes to learning from a user, in-person is best, then phone, then email, but any interaction is better than none. As Steve Krug says, nothing beats in-person observation for pointing out UX flaws. The number of people willing to talk to you is an important signal.

Talking to customers is addictive, and I found it to be a great stress reliever. An excited, happy user is the best cure for stress about your startup, but frankly any customer feedback at all feels good. Substance over theory, baby. Do not be afraid to get on the phone with customers — you might be surprised at how pleased people are to talk to a company that actually gives a damn about its customers!

2. Stay Flexible
You might find that you need to give yourself a daily quota, i.e. “I will email X new customers every morning”, but keep on evaluating your time involvement. At first, I reached out to every single new user. Then time demands forced me to scale back to just the new female sign-ups, since that is our primary demographic. This week I am pretty much putting customer development on hold. The reason: we have made our decisions on what comes next for the business, and now we need to roll up our sleeves and get there as quickly as possible.
3. Don’t Optimize for the Wrong Things (or “let some stuff suck”)During a beta process like this, you are painfully aware of the limitations of your crude product. While we worked hard to fix immediate technical and UX issues, we tried not to lose ourselves in optimization. Making the MVP really solid was not our goal; finding product-market fit was our goal. If a problem was getting in the way of our learning, we fixed it, but many things were left on the backlog. This is easier said than done, because as you talk to more users, you start hearing the same problems over and over again. Between your pride and a natural instinct to make customers happy, you will feel a massive urge to FIX it. One of the great things about having a co-founder is you can keep each other’s well-intentioned, but poorly-prioritized impulses in check.
4. Ruthless PrioritizationAs a small startup, your eyes will always be bigger than your stomach. This is actually a good thing, because it forces you to be absolutely ruthless in focus and prioritization. At Aprizi, we have a massive list of ideas and improvements. We have made a conscious decision to back-burner development and marketing effort tied to user growth, while we focus on product-market fit.

This can require some forbearance in your discussions with outsiders. I cannot tell you how many people have said “you should add [insert ‘share links’, FB Connect, Twitter Auth, etc]“. All in good time. We have to get the core product right first. Anything else is putting the cart before the horse.

“Prioritization for Pirates”: One interesting exercise is to categorize your backlog of development work using Dave McClure’s AARRR “metrics for pirates”. For each piece of work, label it one of the following: Acquisition, Activation, Retention, Referral or Revenue. Think through which of those AARRR issues are most critical to your business at your current stage. Make sure you include non-customer-facing work like infrastructure needs. I’ll note that analytics / instrumentation dev work does not fit neatly into this exercise, but it is obviously something that cannot be ignored. Arguably this exercise was overkill for our stage, but it still helped me think through our business in an interesting way.

5. Independence & JudgementYour job is not to abdicate vision in exchange for market feedback, but rather to synthesize that feedback with your vision. You internalize, synthesize, hypothesize and test, all in the name of charting a better course for your business. In some cases, you might find yourself disagreeing with users. Be careful, but don’t be afraid to stick by your guns.

For example, I’ve had people saying extremely vehemently, “if you are going to show me a recommendation, you *must* show me price,” or “your default really should be a traditional grid view, not this carousel.” In both cases, I disagree. I have a vision of a shopping entertainment experience after watching and talking to a whole bunch of users. I believe that I’m right.You might notice that I have made no mention of split testing. We do not have the time or resources to do split testing. Some might consider this heresy, but I believe split testing will be really important for the *next* phase of this company, not this current phase. We are still more in the realm of qualitative feedback and judgment calls, not quantitative optimization.

6. Exercise Patience
I think patience is a big part of the lean startup methodology. While time feels like the enemy when you are bootstrapping, don’t leap ahead with your first, immediate conclusions. It takes time to internalize all the feedback you are getting from the market, think through possible paths, chase a number of new hypotheses, and lock down firm decisions for how the business needs to evolve.

While our long-term vision of a “personal shopper” has remained remarkably consistent since Aprizi’s founding, this open beta has brought major shifts to our near-term priorities, thinking, and even our core business model. In weeks 1 and 2, we had guesses. By weeks 4 and 5, we were feeling confident on our directional and design changes.

7. Continually Re-Evaluate (except when you shouldn’t)
At the start of most weeks, I try to take a blank slate approach, and think, “if I were starting today from scratch, what would I do?” At the end of every week, I try to write up a summary of the week to our two F&F investors. Without getting too deep into the weeds, I try to cover the major accomplishments and learnings of the week, immediate challenges, and the questions we are trying to answer for the business. I find it to be a great mental exercise.

Product development work always takes longer than you want, so take advantage of this reality and keep on re-evaluating your conclusions. However, there are times when you need to put everything else on hold, hunker down, and do nothing but product execution. That is where we are for the next several weeks.

8. Talk to Anybody & Everybody
I have not just spoken to beta users during this process. I recommend talking to other entrepreneurs — they are a phenomenal sounding board for ideas. While we won’t look for funding for a few months, I continue to have the occasional meeting with investors because we learn so much from their questions. I should note, since this can be a topic of debate among entrepreneurs, that I have really enjoyed meeting with many of the younger VCs in the New York area — some great feedback. I also continue to have informal meetings with potential distribution partners, because those future needs might affect our current design decisions. I learn something from absolutely every conversation.
Conclusion
I haven’t spent much time talking about our specific decisions (more on that another time), but I can say that I am 200% glad we opened up the beta when we did. The last 6 weeks have improved focus and clarity. We have made some important changes to product design and even our monetization plans. I am more excited about Aprizi than ever, and my confidence is very much tied to the rigor of our customer development work throughout our short lifespan as a company. Onward and upward!

Awesome! Sounds like you are well on your way to a killer product. Have you found any disadvantages to making the beta more open? Are there any customers you talk to and then think, “this is not our target customer?” and then refine your focus?

http://giffconstable.com giffc

I hope so Tristan — if the product doesn't work, I wouldn't blame the process

I have not found any disadvantages to the open beta. Glad we did it this way. We need to make it even more open so you can use product without registering, but there's some dev work involved there. I can understand why some folks try to use celebrity + closed beta (i.e. scarcity) to drive buzz, but we're not famous enough to use that technique!

We definitely talk to people who are not our target customer — or at least our *initial* target customer. Men, for example, typically shop in a different way. Shopping isn't entertainment the way it is for many women. We have brainstormed on ways to make men more psyched about the product, but said “Hold on, we need to do one thing well before we try to make everyone happy.”

http://blog.startupsquare.com Tristan Kromer

Your approach seems 100% correct given your product.

I see some companies only let in enough users to get a significant test (could be quantitative or qualitative) for whichever hypothesis they are testing. We've tended towards this approach as well. Since we have user profiles, search results can get cluttered by blank or minimal profiles if there are a lot of casual users. So we don't let people in until they are “qualified leads”. At least until we have a better filtering mechanism.

I still have some difficulties with time management when speaking to customers. Every call tends to go an hour or more. I have quite the backlog now. How many customer interviews do you conduct each week?

http://twitter.com/scottarosenberg Scott Rosenberg

Giff, great post – great approach and thought process, and thanks for sharing.

http://hapnin.com/users/2 theschnaz

Sounds you like guys doing lean the right way. I could echo many of the points you made as they relate to my experiences.

Do you have any tips for getting feedback online? I'm a big fan of copilot.com + the phone. With copilot + a phone, I can see a user's screen and talk to them as they use my site.

http://giffconstable.com giffc

I actually haven't tried to do remote “observation” usability studies although eventually want to start using usertesting.com or related services. I've been more focused on follow-up interviews either by phone or email. Thanks for the tip about copilot — will look into that.

http://giffconstable.com giffc

your notes on blank profiles totally make sense — and it's a great example of why people should never get dogmatic about startup “rules”. Everything needs to be driven by context.

My number varies per week depending on time constraints. I think last week I had approx. dozen conversations, half via email. I'm not doing many at all this week — hunkering down on product redesign instead. I think more important than the number is the pace and richness of learning — seeing patterns, drawing conclusions.

Just wanted to say that I am immensely enjoying your MVP post series. They are always insightful and grounded in reality. I look forward to future posts – I'm taking notes!

-Brian

http://giffconstable.com giffc

Thanks Brian. Glad to hear it might be useful. We're not perfect over here, but we're trying!

http://hapnin.com/users/2 theschnaz

Copilot seems to be as close to in-person as I can get w/o being in-person. It's also cheap.

Usertesting.com is cool too. It's nice to setup a script and see if someone can make it though w/o you helping at all.

http://amitklein.com amitklein

great post, but how do you know if you're product is really a market fit? im assuming some users will love what you are doing and others wont. have you had any aha moments yet during your beta which has led to drastically change course / pivot?

http://giffconstable.com giffc

Yes I didn’t go into details of our actual decisions because it was already too long, but short answer: our conclusion was that we were not quite at market fit, but that we were indeed on to something important. One big decision was to focus more on entertainment as a driver than on utilitarian motivation, which changes both messaging as well as some elements of design. Another is a rethinking in how we make will make money, which is a leap I am excited about. Long term vision, as I wrote, is surprisingly consistent but I feel like there have been big changes in how we will get off the ground.

http://giffconstable.com giffc

Yes I didn't go into details of our actual decisions because it was already too long, but short answer: our conclusion was that we were not quite at market fit, but that we were indeed on to something important. One big decision was to focus more on entertainment as a driver than on utilitarian motivation, which changes both messaging as well as some elements of design. Another is a rethinking in how we make will make money, which is a leap I am excited about. Long term vision, as I wrote, is surprisingly consistent but I feel like there have been big changes in how we will get off the ground.