24 Comments

I question the whole Cowen/Gross premise.

I think the Null Hypothesis for interviewing is as strong as it is for educational interventions.

When I was in graduate school in the early 90s, my Organizational Behavior course (taught by Bob Sutton of "No Asshole Rule" fame) covered the research on interviewing in great depth. After grinding through all the studies showing that unstructured interviews don't work very well in terms of predicting job performance, Sutton asked, "How many of you would hire or be hired without such an interview?" All the hands stayed down except mine and two others (in a class of roughly 50).

So I learned _two_ lessons: (1) unstructured interviews don't work and (2) people have trouble accepting that. I was also taking a class from Amos Tversky at the time and had internalized his famous saying that, "The human mind is a machine for jumping to conclusions." So I was pretty convinced that people are just built to fool themselves into thinking that interviews work. Also, as Robin Hanson might say, "Interviews aren't actually about evaluation, they're about status."

Then, when I started my existing company, we faced the problem of evaluating a lot of people in a short period of time. I revisited the literature. The definitive paper seemed to be:

The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings”

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.172.1733&rep=rep1&type=pdf

Adding an unstructured interview to a measure of general mental ability increased validity by 0.04. Simply counting the number of years of work experience increased it by 0.03. Reference checks by 0.06. All of these from a base of 0.51 for the GMA measure alone. From a time efficiency perspective, interviews seem pretty useless.

Now, you may be thinking, "But what if super smart guys like Tyler ask really special questions?" Well, Google used to ask really special interview questions and people seem to think that Googlers are super smart. In fact, I live a few miles from Google's headquarters and occasionally get invited to interesting statistics or machine learning talks. Twice, I've met PhDs who work in the HR department and study various aspects of personnel performance. They take this topic very seriously and scientifically.

So what did Google learn from trying to ask really special questions? They reported several years ago that the approach didn't work. They now ask much more typical procedural questions:

https://qz.com/378228/google-is-over-those-ridiculous-brainteasers-but-some-employees-didnt-get-the-memo/

But you might think the type of talent Cowen and Gross are looking for are different. Well, sure. This time is always different. Perhaps they have a well designed study showing their techniques work? I haven't actually read their book... yet. But given the excerpts that Tyler is promoting, my prior is very low. I expect a lot of anecdotes about successful examples of talent they picked and how they did it. However, if they have well designed studies on their particular approach, I'd love the pointer so I can check them out now rather than whenever I get to the book.

As it happens, I'm actually in the same business as Gross. I have almost certainly made at least as many investments as he. Very likely more. My firm has made 1400 since I co founded it in 2012, about 2/3 of those with me as the primary contact (meaning that I'm the one primarily evaluating the company). This year alone, I'll be the primary contact on perhaps 150. YC is one of the few organizations in the world that has done more investments than my firm, but Gross was only there a few years and they have waaay more people than we do.

We spend essentially zero time trying to learn about founder talent through interviews. I mean, we know when they founded the company, so we just look at how much demonstrable progress they've made since then. That tells us far more about their true capability to make more progress in the near term than any of the techniques Tyler has promoted from his book so far. And if you think people can systematically predict anything performance-related about startups farther out than the near term, I'd refer you to Taleb. I'm having trouble thinking of a better example of "Extremistan" than startups!

I like Tyler. I've been reading MR since reading blogs became a thing. I've read most, if not all of his general audience books so far. I think Fast Grants is great and, in fact, our investment process has a lot of similarities. But I am very skeptical of this approach to screening for talent. It seems like exactly the type of thing your brain would think it was good at when in fact it wasn't. And the smarter you are, the better your brain would be at producing justifications.

Expand full comment
author

I think that if your goal is to find *undervalued* people, which is Tyler's very explicit goal, then you almost have to have an unusual strategy. I think of this as like the stock market. Your goal is to find undervalued stocks, even though there are a lot of empirical studies that say "strategy X does not beat the market." And then you have Warren Buffet. I think that job performance can be at least as hard to predict as the stock market (in part because job performance is hard to measure). My guess is that a few people have a strategy that works, although it may work only for them, and only in the context in which they apply it. My guess is that Tyler is one of those people. Honestly, I thought I was one of those people when I did hiring.

Expand full comment

That's fair. But then I question the wisdom of promoting a practice with a popular audience book that inherently only a few people can do. In the absence of a validated methodology, it seems like you could easily make things worse. Does anyone seriously think reading a book by Warren Buffet would allow any significant number of people to identify undervalued stocks? But how many people might it lead to do something foolish?

Expand full comment

I read The Snowball, and it seemed like Buffet was practicing Value Line style investing at first, to have better than average returns. Then he added his own evaluations of managers, industry trends, & his relative expertise. Plus close view of market timing. He lost very little in the dot.com bubble pop. Gained a LOT by having cash available in 2008 to buy distressed assets.

Probably most successful investors are trying to emulate both the Value Line company metrics. But so many that it's become the standard "market" behavior.

Expand full comment

Based only on Arnold's post, I'd say these questions are designed to screen for people the questioner would like to hang out with rather than anything specific to productivity.

Expand full comment

"And the smarter you are, the better your brain would be at producing justifications."

All smart folk are smart enough to know how to lie to themselves, and believe the untruths. (If believed, are the untruths really lies?).

Your emotions decide, and your Big Brain comes up with rational reasons. Rationalization.

Expand full comment

Whenever I read about how interviews don't work that well, I think about what I could do instead of interviews, and conclude that interviews are like democracy - it's the worst way to do it, except for all the others.

Expand full comment

Well, I think you're suffering from a lack of imagination. The paper I cited lists a bunch of options and their relative value.

And in fact, for certain jobs, like programming, you see a strong shift to tests. You could also do this for other functions like sales, marketing, and accounting. Sales and marketing might be less objective than programming, but you could at least _try_.

Then you could also get more extreme and perhaps try probationary employment, with a large bonus if the candidate survives probation to compensate for the risk. Claiming we should give up when you can't immediately come up with better alternatives doesn't seem like very strong evidence.

Expand full comment

When a test is a realistic help, I'm all for testing. But when I use testing, it's to inform the interviewing/hiring process, and not as a replacement for interviews.

Likewise, I always hire into probationary employment. But that's not a replacement for interviewing, either. That's just a way to hedge the bet you made after the interview.

It doesn't really matter how many studies show that interviewing is not predictive of performance. Sensible managers are not going to hire without interviewing.

Just like it doesn't matter how many studies show that humans are chock full of cognitive biases, logical fallacies, and subconscious short-cutting. People are still going to bring their rational faculties to bear on whatever they're dealing with, because there's seldom any realistic alternative to thinking.

Expand full comment

It occurs to me that you are in an unusually good position to run the experiment. You could choose half your candidates based only on non-interview information (but still conduct interviews as a form of blinding) and then see if their probationary evaluations are significantly different

Expand full comment

"Sensible managers are not going to hire without interviewing."

I consider myself a sensible manager and I do so. Both with screening entrepreneurs and our internal hiring. I know a few others that do as well. You may mean "most". But that's fine. The argument that people will generally reject useful knowledge isn't actually evidence against the knowledge being useful.

Just as Arnold pointed out that Tyler's goal is to identify undervalued talent, the point of knowing that unstructured interviews don't work is to arbitrage all those people that mistakenly believe in them.

"...because there's seldom any realistic alternative to thinking."

This is simply an incorrect framing as a false choice. When you're in a decision environment where biases are likely to be highly detrimental, use an alternative approach. There's a whole literature on alternatives. None of them say "don't think". They say, apply your thinking using a structured instrument of some sort.

Expand full comment
founding

I'd like to push back on "being ambitious is like playing poker" and how linking business success to poker. Poker is a zero-sum game. If you win, someone else loses. Business success doesn't have to be that way. An entrepreneur can succeed without someone else being a loser. For example, Edison's success benefited many but were there any proportionate losers?

Expand full comment

This is very true, in global economic terms. But how many successful business starters do you know? Success requires winning the orders from the customers, who always have alternative offers. For every customer who buys something, it IS zero-sum between the competing businesses, with only the bought stuff maker winning (the customer), and all the alternatives losing.

For everybody and org on a budget, buying anything means less money for anything else. The budget is zero-sum, for the decision maker, at the time of making the decision. It can lead to increasing wealth when the selling org makes a profit.

Profit is a measure of the increase in wealth for society from the buyer's choice. Profit, over time, is how the budget constrained zero-sum decisions lead to positive sum economic growth.

And every successful business knows they have to sell to make profit - and most successful salesfolk are competitive win seekers.

Expand full comment

The word "ambition" can mean so many different things that used by itself it loses its meaning. Arnold, you have written about the satisfaction you derive from having grandchildren. For you I suspect that forms a part of your own "ambition, and something that you would not give up for the ambition that Thiel means.

Expand full comment

> it comes down to the right people don’t have the right jobs for human progress

Indeed. Some of those people are, to use an already old quote, inventing better ways to make people click on ads, but I suspect that most of them are applying their talents to completely unproductive pursuits. As this screencap (https://twitter.com/CandideIII/status/1525202309252141056) jokingly puts it:

>> My brother literally listens to classical music and plays roller coaster tycoon all day

> Dudes like this used to run entire colonial governments 200 years ago

It's a deplorable and inexcusable - in the long term, quite possibly fatal - waste of human capital (note that such people have little chance in today's marriage market). The man who figures out how to mobilize and organize this capital will be King, or whatever the title is going to be called.

Expand full comment

On Gross's LikedIn questions, "'what are things you are trying to hide?'" on your LinkedIn profile. It's a rather mean spirited question. It puts honest people on the spot. An honest person can't say she has nothing to hide. Surely, every person hides some things from public profiles. The interrogation >shames< a person into parting with a secret--perhaps an embarrassing one--to Gross's apparent delight. He says he "loves asking" the question. A quick-witted person would realize that Gross's question is really rather dumb. Dumb in the sense that a LinkIn profile isn't mean as a tell-all autobiography. Rather, it is a short, concise statement relevant to one's job or professional characterotics. LinkedIn is a job and professional network--not TikTok. Secrets and dancing grab eyeballs on TikTok, but aren't relevant on LinkedIn. Lots of stuff--public or private--get left out of a LinkedIn profile. No one wants to read you conception to now autobiography on LinkedIn. Yeah, folk dancing gets left because it's not really relevant to professional capabilities or productivity.

Expand full comment

‘Another interesting one that I learned from a friend of mine is, do you have any good conspiracy theories that you’re liking lately? ’

Conspiracy theories aren’t what they used to be. As the last two years have shown, today’s conspiracy theory is tomorrow’s fact, Govt will introduce vaccine mandates, vaccine passports will be required to live a normal life, CoVid vaccines don’t stop disease or transmission but do cause deaths and injuries… for example.

Expand full comment

Competitive ambition is just one application of ambition, but it is the one most celebrated in western culture. Laziness is a great social problem. Society doesn't necessarily need more Michael Jordans or Jeff Bezos. We need more people who appreciate the principle of self-determination and who are willing to strive to obtain for themselves a better life.

Expand full comment

" I doubt that I am ambitious enough." That's almost certainly true, now.

Your Fantasy Intellectuals Team is a great idea, but needs to get better to scale, and in particular needs self-scoring and self-rating so that there is not "Kling referee".

It was and remains, so far, not tested in a form that could be scalable. But you've somehow lost drive to improve it - tho it remains in your mind. (Mine too!)

Your blogging has gotten better - nearly daily allows us to hang out here and pontificate. I now think blog comments are often a lot like Twitter tweets, but better (intellectually), tho far worse in getting read by masses of people.

I really enjoy the Monday night talks (Tues 2am for me), but they're genuinely not nearly as special as FIT. Nor is your Network University realistic in many specifics, altho it has lots of great ideas.

Why not try some brainstorm FIT zoom meeting? You could invite your commenters AND those who were owners last year, in April or May, and get ideas for improvement.

Preferably one meeting for just brainstorming - coming up with ideas, NO evaluation of those ideas. In an IRL meeting, it could be done with Post-It notes; there are virtual whiteboards that allow it (don't remember the name); not sure it's so much better than all sending ideas in on a chat - with all getting the chat file after the 30-70 minutes of talking (& joking? Nahh, we're too serious).

Success is not ambition as much as willingness to focus on one tough thing, and give up all the other interesting, fun, and less tough things. Certainly for me.

Expand full comment

“If anyone other than Tyler asked me that in an interview, my reaction would be “WTF?”

I have a similar reaction to a lot of the specific interview questions Tyler proposes. Many of them are interesting but am I really going to feel comfortable entertaining a right-wing inflected conspiracy theory in a job interview for a Silicon Valley tech company where 90+% of donations are going to democratic candidates?

Maybe the idea is that by asking the question you signal that you’re not the type of person to cancel an interviewee for a non-pc opinion. Still, I don’t think I would trust it in most cases.

Expand full comment

It could be safe to discuss historical controversies, such as Stonehenge or Atlantis. But I agree. Who in their right mind would risk contamination of their reputation by engaging strangers on topics that might greatly offend?

Modern conspiracy theories are controversial ideas & questions that the mainstream media dismisses as fringe and unwelcome in public discourse. As such, a different way to pose this question is to ask: "What fringe ideas, offensive to conventional wisdom, do you entertain and deem worthy of more consideration?"

Now, if the question is constrained to fringe ideas that are non-threatening to the mainstream, then I suppose that could be worth pursuing. But what are these? Non-threatening thoughts trend to be banal and trivial. For example, why are hot dogs sold in packages of ten and hot dog buns sold in packages of eight? It's a conspiracy!

Expand full comment

It’s been 50 years. Now try Argentine tango. Same dances (and music) repeated at every dance (milonga) worldwide. Not easy, but a good way to meet girls. PS. I’d wait until the Covid risk subsides before dancing tango cheek-to-cheek.

Expand full comment

A conspiracy is a secret plan by a group, two or more people, to do something unlawful or harmful.

Expand full comment

I actually saw a Romeo and Juliet based play on this exact premise. They get a happy ending, but then they turn out to have a bad relationship once the honeymoon is over. However, one gets in trouble and during the long adventure of rescuing each other they develop an actual bond based on shared experience and sacrifice that leads to a deeper and more robust love.

Expand full comment