Category: product management

“When will AI be able to replace a product thinker like me?”

Rakesh Agrawal

Whenever I meet people, they tell me that AI will get rid of most jobs—except theirs.

I know that there is a good chance that at some point AI will develop skills that will help it get closer to my skill level.

I’ve written about how AI can replace PMs with up to 7 years of experience. So I asked ChatGPT “When will AI be able to replace a product thinker like me?”

The TL;DR: Keep building the future. AI can (currently) only do what is in its training set. If you do things that haven’t been done before, you can stay ahead of the tsunami. If you can manage complex organizations to ship. If you’re a system-level thinker.

Is this good advice? That’s TBD. But it’s also the best answer that I’ve been able to come up with when talking with many long-term tech execs.

The following is a transcript of ChatGPT’s answer. When drafting this, I told ChatGPT that I’d put my own assessment in italics. It misunderstood and put its assessment of what my assessment would be. They were spot on, so I left them in.

Executive Summary

AI can already outperform junior and mid-level PMs at execution. It writes PRDs, prioritizes roadmaps, handles tradeoff matrices, and synthesizes user feedback—all faster, cheaper, and more thoroughly than most human PMs. But it still struggles with judgment, taste, and navigating political, temporal, and organizational complexity. Those are the layers where top product thinkers still thrive.

I’m watching this happen in real time. What used to take days of back-and-forth now takes a well-structured prompt and 30 seconds. But the AI still doesn’t know when to say no—or when to break the rules.

This memo breaks the question into three timeframes: what AI can do today, what it will likely master soon, and what may remain uniquely human.

1. What AI Can Already Do (Today)

AI now matches or outpaces $150K+/year PMs at:

  • Feature scoping & prioritization
  • Drafting PRDs and user stories
  • Competitive teardown and synthesis
  • UX tradeoff analysis
  • Roadmap justification and market segmentation
  • Edge-case anticipation
  • Implementation feasibility checks (tooling, latency, systems integration)

Key insight: For structured thinking and repeatable processes, AI is already a high-functioning, tireless, no-ego PM.

In side-by-side comparisons, I’ve seen AI outperform real PMs in breadth, speed, and edge-case awareness. It’s humbling—and a little unnerving.

2. What AI Will Likely Master Soon (1–3 Years)

  • Modeling political/organizational dynamics: understanding who needs buy-in and where blockers live
  • Multi-system reasoning: how product changes ripple through infrastructure, marketing, and user trust
  • Value-based prioritization: balancing revenue, delight, risk, and cost with contextual nuance
  • Taste mimicry: copying the instincts of great product thinkers based on exposure to their prior work

Key risk: If you stay in the realm of optimization and extrapolation, you’ll get outpaced by a machine that never sleeps and never second-guesses.

It’s already good at seeing the chessboard. It’s learning how the players behave. Soon it will start asking which games are worth playing.

3. What’s Hard to Replace (Maybe Never)

This is the true job of senior product leadership:

a. Knowing when to ignore the data

“The metrics say it’s fine, but I know it’s not.”

b. Calling organizational bullshit

“This isn’t a feature request. It’s a political workaround.”

c. Pattern recognition across domains

“This reminds me of something that failed five years ago in a different vertical.”

d. Taste

You can’t A/B test your way to Pixar. You don’t user research your way to vision. You feel when it’s right.

Key moat: Judgment, taste, and courage remain hard to automate—especially when success requires going against the grain.

These are the moments where senior PMs still earn their title. When to kill a darling. When to greenlight a moonshot. When to say “trust me.” AI doesn’t have that instinct—yet.

Final Assessment

If your job is to execute, AI is already competitive.

If your job is to bet, guide, interpret, and influence—it’s still your game to lose.

The PMs who will survive and thrive are those who:

  • Integrate systems thinking and storytelling
  • Understand power and incentives
  • Apply historical and cross-domain reasoning
  • Develop and defend taste

AI isn’t coming for your job. It’s coming for the parts of your job that weren’t your differentiators anyway.

ChatGPT: A $150,000 PM for $20 a month

Rakesh Agrawal

I have a new product manager that outperforms most mid-level PMs I’ve worked with. It’s faster, more thorough, and has ideas veteran PMs miss. It’s also an AI. As someone who has recruited and managed PMs for over a decade, this is what keeps me up at night.

To see how far AI has really come, I started feeding ChatGPT the same product design and strategy questions I use to interview human candidates to see how it would do. The answer: great. For most of the tasks, it has easily out-performed entry level PMs and PMs with 5-7 years of experience. It has come up with solutions that even veteran PMs haven’t. All for the low, low price of $20/month. And, of course, it does it faster.

The humble volume buttons

Here’s one example: In the latest hardware refresh, Google moved the volume buttons on the remote for their TV streamer from the side of the remote to the face.

New (left) and old Google streaming remote

ChatGPT came up with the expected answers: the buttons on the side have become very familiar to users because that’s the way cell phone buttons work. It also lets the remote be smaller.

Putting the buttons on the face is more equivalent to traditional remote controls in terms of discoverability. That’s where they’ve always been. But it makes the remote substantially bigger. (See picture above.)

That’s where most PMs would stop. ChatGPT went into the details of tooling and manufacturing costs.

The absurdity test

I also did something I frequently do with PMs: suggest absurd ideas to see if 1) they understand that they are absurd 2) they are willing to push back.

I suggested doing a split test, with 5,000 units with the volume buttons on the side and 5,000 units with the buttons on the face.

Many junior PMs say “Sure, sounds like a good experiment.” They are trained to be data-driven.

Although that works well in a software environment, that’s a really bad idea for hardware. Doing a split run is prohibitively expensive due to tooling costs. You’d also have to come up with different packaging and marketing materials.

ChatGPT came up with the idea I was looking for: 3D print a few samples and bring in people to test them.

Absent that, ChatGPT recommended putting the volume controls on the side. So did Gemini. (If I meet the team who designed the new remote, I will definitely ask about the reason for the swap – and the swap of the home and assistant buttons.)

What does it mean for entry-level PMs?

I’m afraid the answer isn’t great. I can get $150k of productivity for $20/month. That’s not a tough call.

That begs the question: if there isn’t a pipeline for entry-level and mid-level PMs, where do senior level PMs come from? The best answer for now is that PMs need to expand their breadth to be able to handle more complexity: integrate design, development, business and systems level thinking into their repertoire.

As Scott Belsky says, taste becomes more important than ever.

So does the ability to see what the AI doesn’t: power dynamics, company incentives, unquantifiable friction — and what’s not on the roadmap, but should be.

A snippet of the ChatGPT response is below.

Three things I got right as a PM leader

Rakesh Agrawal

Previous post: Three things I got wrong as a PM leader.

Understanding customer psychology is key

The best products come from the intersection of technology and psychology. Part of the fun of creating new products is trying to figure out things other people haven’t. Imagine someone dumped a pile of small, multi-colored plastic shapes that interlock in front of you in 1948. Dump them in front of someone and they’ll think it is junk.

Put a picture of a houses or airplane on the box and they’ll be able to fill in the gaps. This is what I can do with those Legos. You’ve provided people a framework for understanding and sparking their creativity.

Understanding psychology includes using all of the senses. Incorporate sight, sound, touch, smell and taste. (OK, smell and taste aren’t necessarily applicable to online products.)

I was at a ski resort and their lift ticket scanners would beep when the ticket was scanned. But the beep was just a confirmation that it was scanned, not an indicator of whether it was valid. The liftie had to look at the display to see the ticket status. It could mean moving gloves out in the cold. If I were designing it, the scanner would beep differently based on whether the ticket was valid or not. There would also be big green and red lights on top the scanner.

Haptics are often overlooked, but they can be very useful. When you’re using walking directions, Apple Watch will tap you on the wrist to indicate that you need to make a turn. What they could do better: have a different tap pattern based on whether you need to make a left turn or right. You wouldn’t have to look down at the watch to see the arrow.

Price is not everything

Yes, price matters. But understanding and being able to contextualize price is important. 

We had a feature-rich product that you could use in a lot of different ways — making phone calls, checking email, storing files and sending faxes (!). It was a great set of features, but because it was a new product, people had no understanding of how much it should cost. In fact, we were underpricing it. I was able to create bundles of features that were more widely understood and comparable to how competitors priced things. We were able to double prices and double adoption.

Think carefully about whether you want to charge at all. There is a much bigger psychological difference between $0.00 and $0.01 than between $0.01 and $1.00.

Simplicity of payment also matters. In the Bay Area, there are more than two dozen transit agencies. Each has its own pricing and fare structure. Passes are different. Not only did you have to figure out how much it cost, your had to figure out how to pay. The payment part was simplified by having an NFC card that worked across the systems. 

Some systems have gotten even simpler. In NYC and London, you can use your contactless credit card. No more having to find and buy a separate card.

If you’re shipping physical products, it’s a giant mistake to not incorporate Apple Pay. Apple created a great system to minimize friction in online commerce. Use it. This is especially true if you have low frequency customers.

Whoever sets the defaults controls the world

In general people want to do the least amount of effort, especially things that they aren’t super interested in. They will do whatever is easiest. 

The new tablet-based point-of-sale systems make it easy to tip 15%, 18%, 20% etc. (depending on the system). You can tip less or more, but that usually requires going to a submenu and entering an amount. Not only is picking the pre-filled amounts easier, it tells users that they should tip one of those amounts. (Hey cheapskate!)

Think about walking through a supermarket. The big brands make it convenient to buy their products. They pay slotting fees to grocers to ensure that their products are at eye level or on the end caps. The better values, either in terms of quality or price, aren’t at eye level.

By setting the right defaults, you can push the metrics you want toward your preferred direction.

Three things I got wrong as a PM leader

Rakesh Agrawal

Listening to customers

Listening to your customers can lead you down the wrong path, whether you are talking about consumer or enterprise customers. People don’t necessarily know what they want. People don’t know what’s possible. Some people want the kitchen sink.

I find focus groups to be essentially useless when testing innovation. If you want to test new fragrances for Tide, go ahead. But if you want to test a brand new concept, focus groups won’t get it. 

Listening to your early adopters is especially dangerous. They might seem like the “best” customers because they came to you first. Unfortunately, that’s a small base and likely unrepresentative of your target market if you’re looking for mass scale.

Better than listening to your customers is understanding your customers. You can watch what potential customers do. Be in their environment. Building a product for restaurant kitchens? Go work the line for a day. Think of it as a mini Undercover Boss. You can often learn more in observing for 15 minutes than two hours of conversation.

I sat looking over the shoulders of people using one of my products. I watched as they cut-and-pasted data from one window to another and then made minor edits. It was tedious. I went back and redesigned it so that my system (the new one) pre-populated the data from the older system.

If you’re testing a new product with real users, watch them use it and see where they get stuck. Try not to help them – you won’t be there to help them when they’re using your product in real life.

Adding too many features

As a nerd, I’ve always wanted more features. I’m the person who went through all of the settings screens to customize every new product or service to exactly my taste. 

What I discovered early on is that most people don’t want more features. In an early product, we had search results pages for news stories. I added in controls to allow you to pick how many stories showed up on a page and how large an excerpt from each story you wanted to see. 

Terrible idea. It complicated the page for users. They had the additional cognitive load of looking at those controls versus seeing what they came for. It also made it harder and slower to render the page. We were better off just stripping all of that code out and delivering the pages faster. 

Product managers might not think of “speed” as a feature, but it’s one of the most critical ones. Take too long to render and people will go elsewhere.

In an early test at Google, customers said they wanted 30 results per page. When it was rolled out, the page with 10 results was the winner. It rendered 0.5 seconds faster.

My own needs have changed over time. I don’t want more features. I don’t want more settings pages. I want things to work out of the box.

Believing you can’t fight City Hall

This one will undoubtedly be controversial. Sometimes the biggest innovation comes from pushing the boundaries. If you had paid strict attention to copyright laws, you might avoid building Google. The search engine literally copied almost everything online, word-for-word. 

A friend had an idea for an on-demand transportation company. After thinking about it, he decided that it would violate local taxi ordinances, employment laws and be a liability nightmare and decided not to pursue it. Uber said “screw it” and went for it. It’s now a $140 billion business.

This is an area where startups have a huge advantage over large companies. BigCo lawyers will say “no” to anything that presents a sizable risk to their core business. It takes 10 people to say “yes” and 1 person to say “no.” See my post “Could YouTube have come from a large company?” (The post is from 2006, so a lot of the questions I asked then have been answered.)

Sometimes, as with YouTube, getting sued can be great for the business.

Startups also need to be wary of people who have spent all of their time in big companies. It may be tempting to hire someone who has 25 years of payments expertise for your payments company. You just need to make sure they don’t have all of the rules and “we can’t do that” baked in.

As my friend and noted angel investor Gokul Rajaram says, it’s important to push the boundaries, just don’t to things that might have you end up in jail. If you do go over the line, you might be joining Elizabeth Holmes and Sam Bankman-Fried.

Part 2: Three things I got right…

I asked ChatGPT to write a PRD…

Rakesh Agrawal

When I talk to people about AI, they’re convinced that AI will take everyone’s job but theirs.

So I asked ChatGPT to write a PRD based on these broad requirements on streaming.

This is how it went, with my analysis in italics.

>: create a prd

Here’s a condensed version for a Product Requirements Document (PRD) based on the article’s suggestions:

Title: Streaming Service User Experience Enhancements

1. Objective: Improve user experience for streaming services by addressing current pain points related to navigation, search, and content accessibility.

2. Features:

  • Standardized Status Bar: Implement a universal status bar for content streaming, showing key information like time left, volume, etc.
  • Improved Search Functionality: Prioritize free content in search results and maintain uniform keyboard layouts across platforms.
  • Enhanced Content Accessibility: Ensure closed captioning settings are retained across sessions, and respect language preferences.
  • User Interaction Improvements: Introduce a ‘Like’ button for content, allow searches by video length, and streamline temporary connections for guest users.
  • Profile Management: Simplify profile switching and ensure settings are maintained per profile.

3. Requirements:

  • Compatibility: Ensure features are compatible across different devices and platforms.
  • User Settings: Allow users to save and easily modify their preferences.
  • Design: Follow a user-friendly design for all new features to ensure easy navigation and interaction.

4. Success Criteria:

  • Increased user engagement and satisfaction.
  • Positive feedback from usability testing.
  • Decrease in user complaints related to navigation and content accessibility.

This document outlines the initial requirements for the proposed enhancements and will be updated as feedback is received and additional research is conducted.

Point 1 is a reasonable summary of the post. Point 2 is a decent regurgitation of the requirements. Point 3 is “mom and apple pie,” and could be cut-and-pasted in to any PRD. I didn’t expect it to generate success criteria, but that is a reasonable output. (Though still mom and apple pie.)

All in all, not a PRD, but a reasonable outline for a junior PM to build off of.

>: what kind of testing do I need to do

For your streaming service enhancements, you should conduct the following types of testing:

  1. Usability Testing: Evaluate the user interface and user experience with real users to identify any issues in navigation, content accessibility, and overall interaction.
  2. Functional Testing: Verify that all features work as intended across different devices and platforms.
  3. Compatibility Testing: Ensure the service works seamlessly on various devices, operating systems, and web browsers.
  4. Performance Testing: Assess the application’s performance, especially during peak usage times, to ensure it can handle high traffic without degradation.
  5. Security Testing: Check for vulnerabilities, especially in user data handling and payment systems.
  6. A/B Testing: Compare different versions of features, like search algorithms or UI layouts, to determine which ones perform better in terms of user engagement and satisfaction.

Again, generic. Mom and apple pie. But it is a solid checklist. I would add accessibility testing.

>: would a focus group be helpful?

Yes, a focus group would be helpful. It allows for in-depth discussions on user experiences, preferences, and feedback on the new features. This qualitative method complements quantitative data from A/B testing and usability testing, providing richer insights into user needs and behaviors.

This is an accurate answer in that it explores the differences between qualitative and quantitative research. But a lot of things don’t benefit from focus group. The biggest breakthroughs will fail in focus groups because people have a hard time conceptualizing drastic changes.

For consumer online products, I rarely find focus groups useful. If I were running a CPG and testing different fragrances for my laundry soap, that’d be different.

>: on which features?

A focus group would be particularly helpful for features such as the standardized status bar, improved search functionality, user interaction improvements like the ‘Like’ button, and profile management. These are areas where subjective experiences, preferences, and ideas can significantly vary among different users, making them ideal topics for in-depth discussions in a focus group setting.

The status bar is best analyzed using A/B testing. Search result quality on usage metrics such as CTR and increased engagement. Profile management is where you’ll get the most bang for the bunk in focus groups.

Partly because it’s very complicated and expensive to implement and partly because there are so many ways people could use profiles.

>: stack rank the features

Here’s a suggested stack ranking for the features based on potential impact on user experience and implementation complexity:

  1. Improved Search Functionality: Directly impacts user ability to find and enjoy content.
  2. Enhanced Content Accessibility: Important for inclusivity and user satisfaction.
  3. Standardized Status Bar: Improves user navigation and control during streaming.
  4. User Interaction Improvements (e.g., ‘Like’ button): Increases engagement and personalization.
  5. Profile Management: Enhances user experience for households with multiple users.
This is a bit of a trick question. Despite it being in vogue, I am not a big fan of stack ranking. It is too hard, and not particularly useful, to stack rank.

In a reasonably large organization, there will be multiple teams. For this example, you’d probably have separate teams for player, search, personalization and partnerships. At best, you’re ranking within a team’s responsibilities.

Many features are grouped in the way they are implemented. Once you’re under-the-hood of the player, it’s easier to make all the player fixes than to make one fix, close the hood, come back, open the hood, make another fix, etc.

Degree of difficulty and time to value delivered are also a factor in prioritization. I could probably knock the time progression change quickly and deliver compelling, immediate value.

Prioritizing tasks is definitely important. But stack ranking, as it is commonly explained, is mostly a futile exercise.

Making sense of numbers: How to do quantitative analysis

Rakesh Agrawal

Are you building the right product? It’s an important question whether you are a startup or a big company. Good research can help guide you. Doing it incorrectly and you’ll go down the wrong path.

There are two basic types of research: qualitative and quantitative. Qualitative generally involves asking people what they want or their experiences with existing products. Quantitative using hard numbers from your users.

Quantitative research can help you answer questions like “What features do I need to add to my product?” “What features can I remove from my product?” “How is my user base generating revenue?” “Where is there fraud and abuse?” (There is some overlap; I’ll do a separate post on qualitative research.)

Some caveats to look out for when doing quantitative analysis:

  • Data talk, but people hear it in different ways. Given the same set of facts, people can come to multiple interpretations.
  • Interpretation of some metrics can and should change over time. At the very least, acceleration will change over time.
  • A single metric can easily be gamed, either by accident or intent (conning investors).

These are some of my favorite things in quantitative analysis. This is by no means a complete list. 

A/B testing

This is commonly used to test different messages or designs. Two variants (A and B) are presented to different users. Marketing emails commonly use A/B testing. Take a small portion of your subscriber list and send one subject line to half and another subject line to another half. With the data on open rates that you get from these emails, you can send the one with the better conversion to the rest of your list. There can be more than two; you can have A/B/C.

1% testing

This is a variant of A/B testing. It’s commonly used to test different features, especially in complicated products or products so well established that you don’t want to change the experience overnight.

Take Facebook’s News Feed. This is a product that is used by billions of users around the world. Adding a new feature without testing can cause a lot of grief and negative feedback. Before you roll it out widely, you present the new feature to a tiny percent of the user and track how it performs. Do people use it? How often do people use it? Does it add or subtract from other features people use. (I call it 1% testing, but in Facebook’s case, it might be 0.001% testing.)

Market segmentation

One of the challenges with data is that averages can mask important differences. You can dig into data to identify segments that you want to go after. If you’re running a credit card business and find that 15% of your overall spend is travel, that tells you one thing. But when you look deeper, you find that a group of customers spend $50,000 a year on travel. This might lead you to create products for that lucrative customer.

You can also use data to figure out who your profitable and unprofitable customers are. In many products, you’ll find that some customers are unprofitable. They could be doing too many returns. (E-commerce.)

 Fraud/abuse analysis

Detecting fraud (illegal behavior) and abuse (legal behavior but not within your business model) is a great way to use data.

I worked for a company that sold long distance calls. When we looked at the usage data, we found that we had a very large amount of usage to Tuvalu. Given that it’s a tiny nation, this didn’t make sense. A closer look found that there was an error in our rate tables and we were selling something for 10 cents that cost us $2.00. As you’d expect, people from Tuvalu told each other about it and we became the calling service of choice for them. (Some of the details here have been changed.) 

Another use case is finding the outliers in all-you-can-eat plans. Think about cell phone data plans. In the AYCE model, some customers might use 1 GB of data and others use 100GB. Your business model and network capacity is based on average usage of 5 GB of data. The 100GB user hogs capacity and slows things down for everyone else. With data, you can develop new policies: the * that says data rates will be slowed down after 25 GB of use.

Search analysis

Looking at what people search for but you haven’t delivered is important to product planning and improving the experience. After all, they came to you for it.

Let’s say you run a ride app. When someone launches the app, they might be an area where you don’t offer service. Tracking those requests gives you insight on markets that you might want to look at when developing expansion plans.

It’s also a way to improve the product to suggest alternatives that the user might want. If someone is in New York City and searches for “In-N-Out,” you might respond “There are no In-N-Out burgers in New York City, but here are some McDonald’s.” Just kidding. I’d probably return Shake Shack, but In-N-Out is so much better.

Cohort analysis

The key to a successful business is that lifetime value is greater than customer acquisition cost. (Often written as LTV > CAC). You want to make sure that, on average, you make more from customers over their lifetime than it costs to get them.

Look at the customers that signed up for your service 1 year ago and how much they spent and when. For customers that sign up today, can you use the historical data to model what the new customers are likely to do?

When looking at data, you also have to weigh the cost of the analysis against the value of the data. If you’re using data to analyze how people navigate through your site, it may be sufficient just to track data on a small subset of users. Adding too much tracking can add to latency in your site or app.

Also, if you’re trying to decide whether or not to implement something that will take 2 days, it doesn’t make sense to spend 2 weeks to build a system to get the data.

There are a variety of tools you can use for quantitative analysis, depending on what you’re trying to get at. Marketing tools like HubSpot handle A/B testing for email campaigns. Google and Facebook have their own tools for ad performance analytics. Google Analytics and Adobe Analytics allow you to analyze user behavior. For complex feature-level data, you will likely have to create your own database and run SQL queries against it.

COIVD-19 Caveat: For most businesses, I don’t recommend doing quantitative research based on data beginning in March 2020, unless you’re using it to compare the impact of COVID-19. If you try to project based on data from March 2020 on, you’re likely to over or underestimate behavior post COVID-19. 

Grammar trivia: Data is plural, not singular. (Plural of datum.) It’s one of those weird English things that doesn’t seem right, but is. Like how a person who runs a restaurant is a restaurateur, not a restauranteur.

What it means to be a product leader

Rakesh Agrawal

There has been a lot written about how to be a great product person. The skills for being a product leader are different. Here are some of the things I’ve tried to do in my time as a product leader. (Many of these skills apply to leadership in general.)

  • Represent product in leadership meetings. Any organization will have many competing priorities. This includes business strategy, customer needs, marketing campaigns and financial situation. All of these contribute to what the organization needs from the product organization. It’s your job to listen to these competing needs and provide direction to the rest of the organization to ensure that the expectations from executive leadership are realistic.
  • Go beyond buzzwords. The CEO wants AI in the product? Sounds sexy. But what does that actually mean? Delve into the details. If the CEO wants something that can’t be built with today’s technology, it’s the product leader’s job to say that.
  • Clearly communicate priorities. Once priorities are decided, it’s up to you to make sure that everyone on your team knows what they are and how they fall into the schedule and affect the roadmap. Priorities can and will change and the changes need to be communicated quickly to the team, with direction on what other work needs to be delayed or canceled.
  • Empower people. “Empowerment” is a buzzword too often used in job descriptions, with little to back it up. As a product leader, you can’t make all the  product decisions yourself — there are way too many. If you do, you will fail at the other aspects of your job such as working with company executives on overall strategy.
    Let product managers make decisions. Many decisions often have little consequence to the overall success of the product. Is the right timeout for a login 1 day or 3 weeks? In most cases, you won’t know until you try it. Clearly, the right answer isn’t 10 seconds or 10 months. The goal is to build it in such a way that you can change it later.
    Be prepared to step up into the weeds when something critical comes up, but don’t stay there. I’ve had to do QA when the QA team couldn’t replicate a problem that a large enterprise customer was experiencing. That should be the exception. On average, it should be a QA person making $70,000 a year do that work instead of a leader getting paid $400,000.
  • Make decisions quickly. Inevitably, there will be disagreement among the members of the product team or between product and design. When these come about, it’s your job to make a decision. It’s better to resolve these quickly with the knowledge that most decisions are reversible once you have more knowledge on how customers respond. Don’t let decisions drag out with weeks or months.
  • Elevate the skills of your team. Every team member will have strengths and weaknesses. Help your team members up their game. I’ve spent a lot of time teaching teammates on elements of product design, how to do customer research and how to communicate, among many other things. Some of this had been in the form of classes for the team, others have been in writings.  (You can also find a lot of my writings on product management and customer experience on this blog.)
  • Congratulate in public, criticize in private. You’ll have successes and failure. Celebrate the successes in public so that the team can share in the joy and excitement. When things go wrong, don’t criticize or berate people in pubic. Things will fail; that’s the nature of launching products. Failure is great, if you learn from it.
  • Embrace risk taking. If people are afraid of taking risks because they fear public flogging, your products may end up being too saccharine.
  • Hire great product people. If your company is growing, you’ll need to hire more. As important as skills are, fit is more important. You can teach someone how to do market research; it’s much harder to convince them to care.
    There are different types of roles in product organizations. Some are more structured and repetitive. Some require creativity. Put the wrong person in the wrong role and they will be unhappy and unproductive.
  • Say “No.” This is probably the most underrated. Humans generally want to make others happy. Product leadership requires balancing every aspect of the organization. It’s easy to string people along, but it’s a bad idea. It’s better to clearly say no. It’s not possible with the schedule. The technology doesn’t exist. You’d need to spend a lot of money on AWS. Whatever the reason, put it out there. If someone doesn’t accept your “no,” then take it to the executive team for a resolution. You’ll get more done faster and are less likely to burn relationships.

These are, of course, general guidelines. You need to be aware of the specifics of your situation. If you’re in a highly regulated industry, it may have serious consequences if you fail; it’s the product leader’s job to understand that.

How to keep customer feedback from destroying your startup

Rakesh Agrawal

You’ve launched your product. You’re getting some adoption. People seem to like it. You start to get some feedback. TechCrunch does a write up on your site. You get lots more feedback.

Sounds great, right? In many ways, yes. But feedback can destroy your company, if you let it causes you to thrash in what you are doing.

Your earliest adopters tend to be the most geeky, the techies who probably don’t represent your target customers. (Unless your target customer is techies.) The feedback they give you isn’t going to matter to the general market.

I heard Stewart Butterfield, the founder of flickr, talk about this early on. Some of his users wanted that ability to geotag photos by uploading tracklogs from Garmin hiking GPSes and syncing their timestamps with the timestamps on their pictures. Great idea. I loved it and would’ve used it. But me and 10 other people. It turned out not to be necessary in the long term because cell phones (which are the source of the vast majority of pictures online) automatically embed GPS information.

In the short term, geeks like me found tools like GeoSetter and iTag to embed GPS date into EXIF fields on photos.

There are a few other problems with customer feedback:

  • It’s not prioritized, even by the person giving feedback. Generally they don’t tell you whether it’s really important to them or just an “it would be nice.” Would it affect their likelihood of sticking around? Would they pay more for your service? Would it help you get the word out to more people? Who knows? Most feedback doesn’t come with that level of detail.
  • You don’t know who they are how they use the product. Without understanding the customer’s persona, feedback is less valuable.
  • It’s rarely the case that more small features will make or break your product. There are rapidly diminishing returns on many features. In a lot of cases, there are negative returns.
  • They are unlikely to know what your business model and goals are.

In order to make the most of the feedback you get, you need to organize it. See how much of a certain type of feedback you get. Group similar feedback. Have your product and business teams decide where it falls within the business goals.

This is especially true of business-to-business companies. You’re having one-on-one interactions with the buyer. He’s giving you some interesting ideas. Sales tells you that if you had “one more feature,” they’d close the deal.

Wait. When there are a small number of big customers, you risk becoming a captive development arm. You definitely don’t want to be that. It makes it easy to lose track of your other customers. If your product becomes too specialized for a specific customer, it can hurt your valuation.

You want to take in feedback and harmonize it with feedback you get from other customers and potential customers.

It can be hard when you’re working closely with someone to say “no.” Your sales team might keep nagging you for that feature. (In a future post, I’ll discuss how to deputize your sales team and sales engineers into mini product managers.)

Stay strong. Make sure what you’re doing aligns with your roadmap and will appeal to other customers.

There are somethings early customers will ask for that are no-brainers. Examples are GDPR compliance and SSO. You don’t have to do these right away, but be sure to have an answer prepared on how you will solve these problems.

Yesterday, I sent some feedback to the CEO of a company I’m looking to invest in. It’s an incredible feature. I know it will help them sell to new customers. Of course he should take my feedback: I’m a potential investor, reasonably well-known writer, have decades of product experience and I run UX quizzes on Twitter that people seem to value.

His response?

Sounds like a good idea.

That’s something that we can think about adding to the system. Thanks for the feedback.
That’s exactly the right response. It increases the likelihood that I will invest.

A startup’s guide to doing research on the cheap – usability testing

Rakesh Agrawal

User research is an important part of the product design process. It can help you make sure you’re building the right thing for the right people and to continue to learn from what you’ve built.

I divide companies into two types: those that are driven by users and those driven by features. User-driven companies pay intense attention to the needs of their users and build products for them. Feature-driven companies focus on what they can do with technology and build those products. You don’t want to be the second.

Needs can be expressed or unexpressed. Some needs that users express may actually be counter to what they want. Important needs may go unexpressed because users don’t know exactly how to express what they want, or the research tools are poor.

There are many types of tools that you can use for research: usability testing, focus groups, ideation sessions, ethnographic research, user feedback, survey research, eye tracking, etc. Too often product managers confuse the roles of these tools and end up with bad interpretations.

Research can cost from nearly nothing to tens of thousands of dollars. In this series, I’ll go through research tools I think provide the biggest bang for the buck for startups.

Today’s topic: usability testing.

You need to assess whether people can understand what your product will do. Ideally, you’d do this shortly after the user flows have been built. Bring people in 1 by 1 and watch them accomplish a series of key tasks. Have the designer and product manager sit with the interview subject.

I like to just give the subject a task and see what happens. (After all, no one is going to be sitting with them while they are actually using the product.) “Order a large pepperoni-mushroom pizza and have it delivered to your house.” Then watch as he goes through the user flow. Have the interview subject articulate their thought process as they perform the task. Where did his eye go first? What was the overall flow?

Only when the subject gets stuck should the interviewer intervene. “Where did you expect to find it?” “Maybe try scrolling further down the page?”

Although ordering a pizza is a relatively simple task, there are a number of steps: finding the restaurant nearest your house, selecting your pizza, adding toppings, special instructions, delivery information, payment, etc.

Some people like to do the testing one step of the process at a time. I prefer to do it all in one pass. There are two primary reasons:

  • Time to completion is a key part of user success. If you stop the subject at every step, you aren’t going to be get a sense of how long he would have taken with your user interface. If it would have taken him 10 minutes to order a pizza, your interface has failed.
  • You get a sense of what other flows the user might see. That’s not necessarily a bad thing, but it’s important to get that sense.

Once the first pass is done, you can repeat the test step-by-test and ask for detail on what was confusing or not.

Usability testing can be done with static images linked together, but I really prefer doing it with live or close to live code.

You can also test a couple of different flows.

It’s important that the subject is told at the outset that he is not the one being tested; it is the user interface that’s being tested. There is no right or wrong answer.

Whom you choose to participate is key. On HBO’s Silicon Valley, the initial response to the compression tool was amazement. All of their friends loved it. Except for Monica, who hated it. Then, we saw the user interface:

0

The people who loved it, the initial test group, were friends of engineers who were also engineers. They loved this type of interface because it gave them control over every little detail of the compression. Most users don’t want this.

You want to get people who don’t do tech for a living, if at all possible. You want to get people who are as close to your target audience as possible. If your product is designed to be used by senior citizens, don’t have millennials do the testing. (In this example, you’ll miss things like font sizes.)

Usability testing doesn’t need to be a formal process and doesn’t require special equipment.

You can recruit test subjects via craigslist. $75 to $150 for 40 minutes is a reasonable amount to pay. If you sell a product, some of this could be in service credit. For example, a $75 AmEx gift card and $50 to use on your app. I’m also comfortable asking friends and family, if they would be in the target user group.

Usability testing isn’t survey research; you don’t need to interview 1,000 people. With ten people, you should be able to identify the biggest obstacles in your interface.