Making sense of numbers: How to do quantitative analysis

Are you building the right product? It’s an important question whether you are a startup or a big company. Good research can help guide you. Doing it incorrectly and you’ll go down the wrong path.

There are two basic types of research: qualitative and quantitative. Qualitative generally involves asking people what they want or their experiences with existing products. Quantitative using hard numbers from your users.

Quantitative research can help you answer questions like “What features do I need to add to my product?” “What features can I remove from my product?” “How is my user base generating revenue?” “Where is there fraud and abuse?” (There is some overlap; I’ll do a separate post on qualitative research.)

Some caveats to look out for when doing quantitative analysis:

  • Data talk, but people hear it in different ways. Given the same set of facts, people can come to multiple interpretations.
  • Interpretation of some metrics can and should change over time. At the very least, acceleration will change over time.
  • A single metric can easily be gamed, either by accident or intent (conning investors).

These are some of my favorite things in quantitative analysis. This is by no means a complete list. 

A/B testing

This is commonly used to test different messages or designs. Two variants (A and B) are presented to different users. Marketing emails commonly use A/B testing. Take a small portion of your subscriber list and send one subject line to half and another subject line to another half. With the data on open rates that you get from these emails, you can send the one with the better conversion to the rest of your list. There can be more than two; you can have A/B/C.

1% testing

This is a variant of A/B testing. It’s commonly used to test different features, especially in complicated products or products so well established that you don’t want to change the experience overnight.

Take Facebook’s News Feed. This is a product that is used by billions of users around the world. Adding a new feature without testing can cause a lot of grief and negative feedback. Before you roll it out widely, you present the new feature to a tiny percent of the user and track how it performs. Do people use it? How often do people use it? Does it add or subtract from other features people use. (I call it 1% testing, but in Facebook’s case, it might be 0.001% testing.)

Market segmentation

One of the challenges with data is that averages can mask important differences. You can dig into data to identify segments that you want to go after. If you’re running a credit card business and find that 15% of your overall spend is travel, that tells you one thing. But when you look deeper, you find that a group of customers spend $50,000 a year on travel. This might lead you to create products for that lucrative customer.

You can also use data to figure out who your profitable and unprofitable customers are. In many products, you’ll find that some customers are unprofitable. They could be doing too many returns. (E-commerce.)

 Fraud/abuse analysis

Detecting fraud (illegal behavior) and abuse (legal behavior but not within your business model) is a great way to use data.

I worked for a company that sold long distance calls. When we looked at the usage data, we found that we had a very large amount of usage to Tuvalu. Given that it’s a tiny nation, this didn’t make sense. A closer look found that there was an error in our rate tables and we were selling something for 10 cents that cost us $2.00. As you’d expect, people from Tuvalu told each other about it and we became the calling service of choice for them. (Some of the details here have been changed.) 

Another use case is finding the outliers in all-you-can-eat plans. Think about cell phone data plans. In the AYCE model, some customers might use 1 GB of data and others use 100GB. Your business model and network capacity is based on average usage of 5 GB of data. The 100GB user hogs capacity and slows things down for everyone else. With data, you can develop new policies: the * that says data rates will be slowed down after 25 GB of use.

Search analysis

Looking at what people search for but you haven’t delivered is important to product planning and improving the experience. After all, they came to you for it.

Let’s say you run a ride app. When someone launches the app, they might be an area where you don’t offer service. Tracking those requests gives you insight on markets that you might want to look at when developing expansion plans.

It’s also a way to improve the product to suggest alternatives that the user might want. If someone is in New York City and searches for “In-N-Out,” you might respond “There are no In-N-Out burgers in New York City, but here are some McDonald’s.” Just kidding. I’d probably return Shake Shack, but In-N-Out is so much better.

Cohort analysis

The key to a successful business is that lifetime value is greater than customer acquisition cost. (Often written as LTV > CAC). You want to make sure that, on average, you make more from customers over their lifetime than it costs to get them.

Look at the customers that signed up for your service 1 year ago and how much they spent and when. For customers that sign up today, can you use the historical data to model what the new customers are likely to do?

When looking at data, you also have to weigh the cost of the analysis against the value of the data. If you’re using data to analyze how people navigate through your site, it may be sufficient just to track data on a small subset of users. Adding too much tracking can add to latency in your site or app.

Also, if you’re trying to decide whether or not to implement something that will take 2 days, it doesn’t make sense to spend 2 weeks to build a system to get the data.

There are a variety of tools you can use for quantitative analysis, depending on what you’re trying to get at. Marketing tools like HubSpot handle A/B testing for email campaigns. Google and Facebook have their own tools for ad performance analytics. Google Analytics and Adobe Analytics allow you to analyze user behavior. For complex feature-level data, you will likely have to create your own database and run SQL queries against it.

COIVD-19 Caveat: For most businesses, I don’t recommend doing quantitative research based on data beginning in March 2020, unless you’re using it to compare the impact of COVID-19. If you try to project based on data from March 2020 on, you’re likely to over or underestimate behavior post COVID-19. 

Grammar trivia: Data is plural, not singular. (Plural of datum.) It’s one of those weird English things that doesn’t seem right, but is. Like how a person who runs a restaurant is a restaurateur, not a restauranteur.

How to keep customer feedback from destroying your startup

You’ve launched your product. You’re getting some adoption. People seem to like it. You start to get some feedback. TechCrunch does a write up on your site. You get lots more feedback.

Sounds great, right? In many ways, yes. But feedback can destroy your company, if you let it causes you to thrash in what you are doing.

Your earliest adopters tend to be the most geeky, the techies who probably don’t represent your target customers. (Unless your target customer is techies.) The feedback they give you isn’t going to matter to the general market.

I heard Stewart Butterfield, the founder of flickr, talk about this early on. Some of his users wanted that ability to geotag photos by uploading tracklogs from Garmin hiking GPSes and syncing their timestamps with the timestamps on their pictures. Great idea. I loved it and would’ve used it. But me and 10 other people. It turned out not to be necessary in the long term because cell phones (which are the source of the vast majority of pictures online) automatically embed GPS information.

In the short term, geeks like me found tools like GeoSetter and iTag to embed GPS date into EXIF fields on photos.

There are a few other problems with customer feedback:

  • It’s not prioritized, even by the person giving feedback. Generally they don’t tell you whether it’s really important to them or just an “it would be nice.” Would it affect their likelihood of sticking around? Would they pay more for your service? Would it help you get the word out to more people? Who knows? Most feedback doesn’t come with that level of detail.
  • You don’t know who they are how they use the product. Without understanding the customer’s persona, feedback is less valuable.
  • It’s rarely the case that more small features will make or break your product. There are rapidly diminishing returns on many features. In a lot of cases, there are negative returns.
  • They are unlikely to know what your business model and goals are.

In order to make the most of the feedback you get, you need to organize it. See how much of a certain type of feedback you get. Group similar feedback. Have your product and business teams decide where it falls within the business goals.

This is especially true of business-to-business companies. You’re having one-on-one interactions with the buyer. He’s giving you some interesting ideas. Sales tells you that if you had “one more feature,” they’d close the deal.

Wait. When there are a small number of big customers, you risk becoming a captive development arm. You definitely don’t want to be that. It makes it easy to lose track of your other customers. If your product becomes too specialized for a specific customer, it can hurt your valuation.

You want to take in feedback and harmonize it with feedback you get from other customers and potential customers.

It can be hard when you’re working closely with someone to say “no.” Your sales team might keep nagging you for that feature. (In a future post, I’ll discuss how to deputize your sales team and sales engineers into mini product managers.)

Stay strong. Make sure what you’re doing aligns with your roadmap and will appeal to other customers.

There are somethings early customers will ask for that are no-brainers. Examples are GDPR compliance and SSO. You don’t have to do these right away, but be sure to have an answer prepared on how you will solve these problems.

Yesterday, I sent some feedback to the CEO of a company I’m looking to invest in. It’s an incredible feature. I know it will help them sell to new customers. Of course he should take my feedback: I’m a potential investor, reasonably well-known writer, have decades of product experience and I run UX quizzes on Twitter that people seem to value.

His response?

Sounds like a good idea.

That’s something that we can think about adding to the system. Thanks for the feedback.
That’s exactly the right response. It increases the likelihood that I will invest.