Justin Lahart of The WSJ reports,
Harvard University economist Raj Chetty has spent more than a decade working to understand what makes mobility possible, and why in some places the children of poor parents have been more able to move up than in others.
Whenever Chetty mines data, he discovers golden ways for government to reduce poverty. Pay kindergarten teachers more! Change people’s neighborhoods! Social engineering can work—follow the science!
Analyzing data covering a near universe of Americans born from 1978 to 1992, the researchers found that when employment among the poor parents of children in a community improves, those children are better off economically as adults. It is a dynamic that some researchers have suspected, but that has never been shown systematically. Importantly, it doesn’t rely on whether a child’s own parents are employed: Outcomes also improve for children who simply grow up in a neighborhood where more parents have jobs. In other words, their own parents might be unemployed, but if their schoolmates’ parents work, their outcomes will be better.
Whenever I see a study like this, I worry about the influence of selection. The study will claim that characteristic X causally affects result Y. I wonder if characteristic X actually selects for people with personality A that causes result Y.
In this case, why do some children grow up in neighborhoods with higher rates of employment than in other neighborhoods? If this is purely random, then it is plausible that differences in outcomes reflect causality.
But what if people who choose neighborhoods with higher rates of employment tend to have more cognitive ability, higher conscientiousness, or perhaps other traits that are completely unobservable but affect outcomes? And what if they pass those traits along to their children? In that case, differences in outcomes will be due to selection.
If the result of this research is that policy makers try to increase employment rates in poor areas, I see no harm, even if the policy fails to affect the outcomes for children in the way that the research would predict. But the general idea that data miners and social engineers have the answers for reducing poverty is one to be wary of. If “social desirability bias as an influence on publication” were an entry in the encyclopedia, Chetty’s picture would be next to it.
Growing up in a neighborhood in which working adults are normal probably makes a big difference. What children see and experience as they grow up becomes “normal” to them. More often than not, they will do what is “normal” to them when they become adults.
It has always seemed to me down the years that - in mainstream media reporting of social science research - misplaced conflation of correlation with causation is the norm and not the exception. The resulting public consciousness ocean of what Twain called "the things you know for sure that ain't so" has been incalculably large.