Category: Design

ChatGPT: A $150,000 PM for $20 a month

Rakesh Agrawal

I have a new product manager that outperforms most mid-level PMs I’ve worked with. It’s faster, more thorough, and has ideas veteran PMs miss. It’s also an AI. As someone who has recruited and managed PMs for over a decade, this is what keeps me up at night.

To see how far AI has really come, I started feeding ChatGPT the same product design and strategy questions I use to interview human candidates to see how it would do. The answer: great. For most of the tasks, it has easily out-performed entry level PMs and PMs with 5-7 years of experience. It has come up with solutions that even veteran PMs haven’t. All for the low, low price of $20/month. And, of course, it does it faster.

The humble volume buttons

Here’s one example: In the latest hardware refresh, Google moved the volume buttons on the remote for their TV streamer from the side of the remote to the face.

New (left) and old Google streaming remote

ChatGPT came up with the expected answers: the buttons on the side have become very familiar to users because that’s the way cell phone buttons work. It also lets the remote be smaller.

Putting the buttons on the face is more equivalent to traditional remote controls in terms of discoverability. That’s where they’ve always been. But it makes the remote substantially bigger. (See picture above.)

That’s where most PMs would stop. ChatGPT went into the details of tooling and manufacturing costs.

The absurdity test

I also did something I frequently do with PMs: suggest absurd ideas to see if 1) they understand that they are absurd 2) they are willing to push back.

I suggested doing a split test, with 5,000 units with the volume buttons on the side and 5,000 units with the buttons on the face.

Many junior PMs say “Sure, sounds like a good experiment.” They are trained to be data-driven.

Although that works well in a software environment, that’s a really bad idea for hardware. Doing a split run is prohibitively expensive due to tooling costs. You’d also have to come up with different packaging and marketing materials.

ChatGPT came up with the idea I was looking for: 3D print a few samples and bring in people to test them.

Absent that, ChatGPT recommended putting the volume controls on the side. So did Gemini. (If I meet the team who designed the new remote, I will definitely ask about the reason for the swap – and the swap of the home and assistant buttons.)

What does it mean for entry-level PMs?

I’m afraid the answer isn’t great. I can get $150k of productivity for $20/month. That’s not a tough call.

That begs the question: if there isn’t a pipeline for entry-level and mid-level PMs, where do senior level PMs come from? The best answer for now is that PMs need to expand their breadth to be able to handle more complexity: integrate design, development, business and systems level thinking into their repertoire.

As Scott Belsky says, taste becomes more important than ever.

So does the ability to see what the AI doesn’t: power dynamics, company incentives, unquantifiable friction — and what’s not on the roadmap, but should be.

A snippet of the ChatGPT response is below.

Making local real time

Rakesh Agrawal
Monk creating a map

Keeping location data accurate is a Herculean (almost Sisyphean) task. Stores go out of business and new ones take their place. Snowstorms and pandemics temporarily shutter a business. Bars stay open later on a Sunday for the Super Bowl.

It used to be that business information was updated once a year. Data companies would ship Yellow Pages overseas and people would type in the information and send it back to the companies to sell them. Consumers could buy them as CDs or DVDs to generate mailing lists.

That bar has gone up dramatically. With everything else being on their phones being real-time, consumers expect location data to be as well. Plenty of friends (including tech savvy friends) assume that what Google, Apple, OpenTable, etc. tell them is gospel.

It’s not so. I tell them to call to verify. I’ve been to plenty of “fully booked” restaurants that are actually quite empty. OpenTable charges restaurants for each reservation. That gives restaurants a disincentive to make those times available when they would otherwise be full. Bowling alleys might be full of people, but that’s because it’s league night. The typical customer can’t walk in and bowl.

I’ve been to plenty of places where Google’s “live now” data is inaccurate. (In both directions.) Here is a recent example:

The location on the map (POI for mapping nerds) shows that the McDonald’s is “Permanently closed,” but the “live” information says it is “Open”. Both things can’t be true. If you walk by (what local nerds call “ground truth”), you will find that it is permanently closed.

There are a number of ways to improve timeliness and data quality:

  • Have the facilities update their information individually. Google has an extensive set of tools at Google My Business for companies to manage their profiles. Small business owners and government agencies are usually heavily pressed for time or don’t know about these tools.
  • Have users submit corrections. There are a lot more users than there are business owners. I submitted the McDonald’s correction to Apple and it was updated within a few days. The challenge with this approach is that there is fraud. Competing businesses might report a business closed. People create fake emergency rooms (yes, this a real example). Yelp has been frequently bombed with reviews when a business is in the political spotlight
  • Get information feeds from businesses and government. Chains could submit corrections through feeds. But even this information isn’t timely. The McDonald’s app still listed the above McDonald’s as open for at least a week after it closed. Ironically, this is one of many ways Mapquest blew it. Their initial business was store locators; they would charge businesses to put a store search on the store’s Web site. This presented a channel conflict: they didn’t want to feed it in to the consumer site because it could potentially cannibalize the store locator business. (There is precedent for doing this right: Transit agencies provide real-time data through GTFS.)
  • Use anonymized cell-phone location data to predict the number of people are at a business. A key problem is that in many urban areas there is so much density that even the most advanced GPS isn’t good enough. Indoor spaces are another problem.

The best way to keep data “live” is to use real-time transaction data. In the developed world, most businesses take credit cards. A lot of cash businesses use point-of-sale systems like Square. Restaurants are using online ordering. (Disclosure: I was an early board advisor to Olo until its IPO, which powers the ordering for Google Food, as well as restaurants like P.F. Chang’s, Five Guys, Qdoba.)

In the McDonald’s example above, the store locator was inaccurate, but if you tried to order something, you were told it was closed.

There are many advantages to transaction data:

  • If people are transacting, there is a very strong likelihood that a business open. If you don’t see any transactions, you can make a strong inference that the business is closed.
  • You can approximate cost based on transaction value. For restaurants, you could make reasonable estimates for groups of 2, 4, etc.
  • If the data source has SKU information (like from Olo or Square), you can get the full menu and the actual most popular items.
  • Depending on the level of anonymity, you can determine how frequently people visit a place. Frequent visits is a good indicator of NPS.

The rapid uptake of Apple Pay gives Apple an advantage here.

Transaction data won’t tell you if the trail you wanted to hike is closed due to snow, but business search will be a lot better than it is today.

Out of business? Wouldn’t you like to know?

Rakesh Agrawal

You’ve done your research, you found the place you want to go, you get there. It’s closed!

What went wrong? It turns out the map is out of date. We didn’t have this expectation when we had paper maps. But online is different! It should be in real-time! (In a future post, I’ll talk about how to make it near real-time.)

I was reminded of this the other day when I tried to go to a McDonald’s. The sign had been dismantled and there was a letter on the door that they had closed. (I only go to McDonald’s for their cheap Diet Coke!)

Even though data sources exist, they didn’t make them to the maps.

Even when the map platforms know that the business is closed, they don’t render it optimally.

Broadly, there are two types of searches in local: category and business name. If the user is searching for a category, such as “restaurants 10018”, closed businesses should be left out. Why show something that you know the user can’t use?

On the other hand, if a user has searched for a specific business name, it’s helpful for the user to know that the business has closed. One thing to keep in mind is that people don’t often know the exact name of the business.

I did a search for “Alexanders books.” The top search results on Apple Maps don’t reflect my search. I get a list of bookstores; only after scrolling through dozens of bookstores do I see Alexander listed and that it is permanently closed.

Google does a better job. (Note that Google pulled up the correct business, even though I didn’t ask for it exactly.) This is the result I got:

But even that isn’t a great experience. Does the user really want to get directions, navigate to or call a business that is permanently closed? Probably not. After learning that their preferred store is closed, the most likely thing the user wants to do is find another bookstore.