Discussion about this post

User's avatar
Kevin Dick's avatar

I question the whole Cowen/Gross premise.

I think the Null Hypothesis for interviewing is as strong as it is for educational interventions.

When I was in graduate school in the early 90s, my Organizational Behavior course (taught by Bob Sutton of "No Asshole Rule" fame) covered the research on interviewing in great depth. After grinding through all the studies showing that unstructured interviews don't work very well in terms of predicting job performance, Sutton asked, "How many of you would hire or be hired without such an interview?" All the hands stayed down except mine and two others (in a class of roughly 50).

So I learned _two_ lessons: (1) unstructured interviews don't work and (2) people have trouble accepting that. I was also taking a class from Amos Tversky at the time and had internalized his famous saying that, "The human mind is a machine for jumping to conclusions." So I was pretty convinced that people are just built to fool themselves into thinking that interviews work. Also, as Robin Hanson might say, "Interviews aren't actually about evaluation, they're about status."

Then, when I started my existing company, we faced the problem of evaluating a lot of people in a short period of time. I revisited the literature. The definitive paper seemed to be:

The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings”

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.172.1733&rep=rep1&type=pdf

Adding an unstructured interview to a measure of general mental ability increased validity by 0.04. Simply counting the number of years of work experience increased it by 0.03. Reference checks by 0.06. All of these from a base of 0.51 for the GMA measure alone. From a time efficiency perspective, interviews seem pretty useless.

Now, you may be thinking, "But what if super smart guys like Tyler ask really special questions?" Well, Google used to ask really special interview questions and people seem to think that Googlers are super smart. In fact, I live a few miles from Google's headquarters and occasionally get invited to interesting statistics or machine learning talks. Twice, I've met PhDs who work in the HR department and study various aspects of personnel performance. They take this topic very seriously and scientifically.

So what did Google learn from trying to ask really special questions? They reported several years ago that the approach didn't work. They now ask much more typical procedural questions:

https://qz.com/378228/google-is-over-those-ridiculous-brainteasers-but-some-employees-didnt-get-the-memo/

But you might think the type of talent Cowen and Gross are looking for are different. Well, sure. This time is always different. Perhaps they have a well designed study showing their techniques work? I haven't actually read their book... yet. But given the excerpts that Tyler is promoting, my prior is very low. I expect a lot of anecdotes about successful examples of talent they picked and how they did it. However, if they have well designed studies on their particular approach, I'd love the pointer so I can check them out now rather than whenever I get to the book.

As it happens, I'm actually in the same business as Gross. I have almost certainly made at least as many investments as he. Very likely more. My firm has made 1400 since I co founded it in 2012, about 2/3 of those with me as the primary contact (meaning that I'm the one primarily evaluating the company). This year alone, I'll be the primary contact on perhaps 150. YC is one of the few organizations in the world that has done more investments than my firm, but Gross was only there a few years and they have waaay more people than we do.

We spend essentially zero time trying to learn about founder talent through interviews. I mean, we know when they founded the company, so we just look at how much demonstrable progress they've made since then. That tells us far more about their true capability to make more progress in the near term than any of the techniques Tyler has promoted from his book so far. And if you think people can systematically predict anything performance-related about startups farther out than the near term, I'd refer you to Taleb. I'm having trouble thinking of a better example of "Extremistan" than startups!

I like Tyler. I've been reading MR since reading blogs became a thing. I've read most, if not all of his general audience books so far. I think Fast Grants is great and, in fact, our investment process has a lot of similarities. But I am very skeptical of this approach to screening for talent. It seems like exactly the type of thing your brain would think it was good at when in fact it wasn't. And the smarter you are, the better your brain would be at producing justifications.

Expand full comment
Hunmeister's avatar

I'd like to push back on "being ambitious is like playing poker" and how linking business success to poker. Poker is a zero-sum game. If you win, someone else loses. Business success doesn't have to be that way. An entrepreneur can succeed without someone else being a loser. For example, Edison's success benefited many but were there any proportionate losers?

Expand full comment
22 more comments...

No posts