There is a tendency to prioritize variables that can be objectively measured over qualitative variables. This preference may arise from the belief that quantifiable data is more objective or simply because such data is easier to collect and analyze. As a result, vital information that is not easily captured by numerical measurements may be neglected or undervalued.
For example, P. T. Bauer observed that development economists tend to focus on physical and financial resources, which can be measured, while ignoring individual, cultural, social, and political factors that cannot be measured but that have a profound effect on a nation's productivity.
A common issue stemming from this "quantitative bias" is the evaluation of a program's or policy's efficacy based on measurable inputs rather than qualitative outcomes. For example, improvements in education are often assessed by dollar expenditures and class size instead of gains in students’ knowledge or their ability to think critically.
Goodhart’s Law states that any quantifiable indicator used as a proxy for a non-quantifiable goal will eventually become the goal, making it useless as an indicator. For example, the manager of a large IT department decided to measure production by the number of completed work orders, or “tickets.” Overnight, tasks that previously required a single ticket were split up into multiple tickets – one for each subtask. Productivity fell as time and resources were diverted to filling out and completing tickets instead of designing and writing software. What you measure is what you get; bean counters get beans.
Jones says "average national IQ matters more [economically] than individual IQ" and, I think, wants that to have the counterintuitive implication that how intelligent you personally are doesn't make such a big difference to your personal economic outcomes. But if (as seems plausible) individual IQ score is a noisier measure of individual intelligence than average population IQ is of population average intelligence, that would have the measurement error effect you describe.
No, the measurement issue and increased precision of statistics driving from larger sample sizes does not undermine Jones' analysis or conclusions. The individual's intelligence correlates with higher income and life success, but there's a lot more than intelligence that contributes to those things. On the other hand the fraction of very smart people above 130 or 140 or more matters a lot to both the rate of innovation and the competency of leadership, management, administration and performance of other g- loaded tasks the gains from which are hard to fully capture and which tend to be broadly shared and increase a society's level of security, efficiency, wealth, and rate of growth.
Re: "Measurement error comes from whatever we 'leave out' when we pick a real-world measuring instrument."
In a sense, a measurement can be accurate but misleading if plural variables counteract one another. Let me sketch a notional example.
Suppose that we want to measure the effect of pre-election polls on voter preferences. And suppose voters report their preferences sincerely to poll-takers. And suppose that we find that the published poll indicates 51% for candidate A and 49% for candidate B. And suppose that the election outcome then mirrors the poll.
Even if there isn't any measurement error, it would be misleading to infer that the poll didn't change voter preferences. It might be the case that a psychological 'bandwagon effect' caused 2% of voters to switch from B to A, and that a psychological 'underdog effect' caused 2% of voters to switch from A to B. The two effects would counteract each other.
In this notional example, polls have no effect on the election outcome, but substantial effects on voter preferences. However, it would be hard to track and measure the effects on voter preferences.
Weak aggregate effects may hide strong underlying mechanisms that neutralize each other.
"This becomes really important when more than one explanatory variable is involved. The variable that is measured with the least error will take some of the explanatory power away from the other variable."
While the above is correct, it misses an important part of statistical analysis. In the context of regression analysis (econometrics), the above statement means that variables measured with less noise can contribute more to the explanatory power as measured by "R-squared." However, the coefficient of such a variable might be economically unimportant, i.e. a "beta" or "load factor" closer to zero than the betas or loads of other variables.
If a noisy variable is measured without bias, i.e. the noise on average is zero, then the average value of such a variable is accurate (but not precise). Then the beta or load of such a variable in a regression will also be unbiased, that is, accurate, but might be imprecise. Such a noisy variable may be economically important (have a high beta value or load) but not statistically significant as measured in the given regression and given the amount of noise, i.e. a low t-statistic or "significance" value and so is measured imprecisely. Put another way, the coefficient for this variable could have a high magnitude but also a wide range of possible values.
Sailer’s point is also proven by simply comparing self-reported ancestry with standard ancestry measurements. It turns out that people of African descent (as measured by relatively objective genetics tests) almost always report they’re black, people of European descent (as measured by relatively objective genetics tests) almost always report they’re white, etc.
Not so much a disagreement as an observation, but I find that tends to only hold true for Far/South East Asians, Africans south of the Sahara, and the French/Poland/Finland/UK square Europeans. Everyone else I tend to find reports whatever is most advantageous to them. Works for big buckets but gets progressively worse when the muts get big enough to impact numbers.
Re “…self-reported race measures something. If it were all noise, it would not correlate with anything.” But are there no spurious correlations? I seem to recall a statistics writer once noting a very tight fit between time-series variations in the number of storks nesting on Danish chimneys and the number of babies born in Hyderabad.
we can be sure that the number of storks is not noise. we can be sure that the number of babies born is not noise. what we cannot be confident of is that there is any causal relation between the two (of course there isn't). But causality is a separate issue from what I am talking about.
Leaving out a relevant variable in an N- dimensional problem using as N-X dimensional fitting function will create a false model of reality, by definition. Environmental activists play this game all the time. Think "conic sections" < https://en.wikipedia.org/wiki/Conic_section > where you can get an infinite number of shapes describing a 3-D cone in 2-D.
I'm not saying any of that is wrong but I find the definition of noise a little unclear.
Be that as it may, I found the example of intelligence and character to be enlightening. It also reminded me of the question of race vs culture. We can measure race reasonably well but culture might be even more difficult to measure than character. Seems likely that's why we attribute discrimination to race and not culture.
And it isn't just measurement error- is the propagation of those errors in any calculation using the original measured data set- something that far too many researchers themselves completely ignore.
One difference is that measurement error in the explained variable (income, say) only widens the confidence interval around the effect of the explanatory variable(s). You measure the effect correctly, but you seem less statistically certain of your results. Measurement error in the explanatory variable (character, say) biases the measured effect of the variable toward zero. You understate how strong the effect really is.
Right. There is another kind of error in the explained variable and that is when a proxy is used for the "real" thing like arrests for crime, doctor visits/hospital stays for illness.
There is a tendency to prioritize variables that can be objectively measured over qualitative variables. This preference may arise from the belief that quantifiable data is more objective or simply because such data is easier to collect and analyze. As a result, vital information that is not easily captured by numerical measurements may be neglected or undervalued.
For example, P. T. Bauer observed that development economists tend to focus on physical and financial resources, which can be measured, while ignoring individual, cultural, social, and political factors that cannot be measured but that have a profound effect on a nation's productivity.
A common issue stemming from this "quantitative bias" is the evaluation of a program's or policy's efficacy based on measurable inputs rather than qualitative outcomes. For example, improvements in education are often assessed by dollar expenditures and class size instead of gains in students’ knowledge or their ability to think critically.
Goodhart’s Law states that any quantifiable indicator used as a proxy for a non-quantifiable goal will eventually become the goal, making it useless as an indicator. For example, the manager of a large IT department decided to measure production by the number of completed work orders, or “tickets.” Overnight, tasks that previously required a single ticket were split up into multiple tickets – one for each subtask. Productivity fell as time and resources were diverted to filling out and completing tickets instead of designing and writing software. What you measure is what you get; bean counters get beans.
AIUI this is one of the points made in this old review of "Hive Mind":
https://slatestarcodex.com/2015/12/08/book-review-hive-mind/
Jones says "average national IQ matters more [economically] than individual IQ" and, I think, wants that to have the counterintuitive implication that how intelligent you personally are doesn't make such a big difference to your personal economic outcomes. But if (as seems plausible) individual IQ score is a noisier measure of individual intelligence than average population IQ is of population average intelligence, that would have the measurement error effect you describe.
No, the measurement issue and increased precision of statistics driving from larger sample sizes does not undermine Jones' analysis or conclusions. The individual's intelligence correlates with higher income and life success, but there's a lot more than intelligence that contributes to those things. On the other hand the fraction of very smart people above 130 or 140 or more matters a lot to both the rate of innovation and the competency of leadership, management, administration and performance of other g- loaded tasks the gains from which are hard to fully capture and which tend to be broadly shared and increase a society's level of security, efficiency, wealth, and rate of growth.
Re: "Measurement error comes from whatever we 'leave out' when we pick a real-world measuring instrument."
In a sense, a measurement can be accurate but misleading if plural variables counteract one another. Let me sketch a notional example.
Suppose that we want to measure the effect of pre-election polls on voter preferences. And suppose voters report their preferences sincerely to poll-takers. And suppose that we find that the published poll indicates 51% for candidate A and 49% for candidate B. And suppose that the election outcome then mirrors the poll.
Even if there isn't any measurement error, it would be misleading to infer that the poll didn't change voter preferences. It might be the case that a psychological 'bandwagon effect' caused 2% of voters to switch from B to A, and that a psychological 'underdog effect' caused 2% of voters to switch from A to B. The two effects would counteract each other.
In this notional example, polls have no effect on the election outcome, but substantial effects on voter preferences. However, it would be hard to track and measure the effects on voter preferences.
Weak aggregate effects may hide strong underlying mechanisms that neutralize each other.
"This becomes really important when more than one explanatory variable is involved. The variable that is measured with the least error will take some of the explanatory power away from the other variable."
While the above is correct, it misses an important part of statistical analysis. In the context of regression analysis (econometrics), the above statement means that variables measured with less noise can contribute more to the explanatory power as measured by "R-squared." However, the coefficient of such a variable might be economically unimportant, i.e. a "beta" or "load factor" closer to zero than the betas or loads of other variables.
If a noisy variable is measured without bias, i.e. the noise on average is zero, then the average value of such a variable is accurate (but not precise). Then the beta or load of such a variable in a regression will also be unbiased, that is, accurate, but might be imprecise. Such a noisy variable may be economically important (have a high beta value or load) but not statistically significant as measured in the given regression and given the amount of noise, i.e. a low t-statistic or "significance" value and so is measured imprecisely. Put another way, the coefficient for this variable could have a high magnitude but also a wide range of possible values.
Sailer’s point is also proven by simply comparing self-reported ancestry with standard ancestry measurements. It turns out that people of African descent (as measured by relatively objective genetics tests) almost always report they’re black, people of European descent (as measured by relatively objective genetics tests) almost always report they’re white, etc.
Not so much a disagreement as an observation, but I find that tends to only hold true for Far/South East Asians, Africans south of the Sahara, and the French/Poland/Finland/UK square Europeans. Everyone else I tend to find reports whatever is most advantageous to them. Works for big buckets but gets progressively worse when the muts get big enough to impact numbers.
Re “…self-reported race measures something. If it were all noise, it would not correlate with anything.” But are there no spurious correlations? I seem to recall a statistics writer once noting a very tight fit between time-series variations in the number of storks nesting on Danish chimneys and the number of babies born in Hyderabad.
we can be sure that the number of storks is not noise. we can be sure that the number of babies born is not noise. what we cannot be confident of is that there is any causal relation between the two (of course there isn't). But causality is a separate issue from what I am talking about.
Leaving out a relevant variable in an N- dimensional problem using as N-X dimensional fitting function will create a false model of reality, by definition. Environmental activists play this game all the time. Think "conic sections" < https://en.wikipedia.org/wiki/Conic_section > where you can get an infinite number of shapes describing a 3-D cone in 2-D.
I'm not saying any of that is wrong but I find the definition of noise a little unclear.
Be that as it may, I found the example of intelligence and character to be enlightening. It also reminded me of the question of race vs culture. We can measure race reasonably well but culture might be even more difficult to measure than character. Seems likely that's why we attribute discrimination to race and not culture.
And it isn't just measurement error- is the propagation of those errors in any calculation using the original measured data set- something that far too many researchers themselves completely ignore.
Measurment error. Good pointing general but you need to distinguish error in measuring explanatory variables vs explained variables.
Can you explain?
One difference is that measurement error in the explained variable (income, say) only widens the confidence interval around the effect of the explanatory variable(s). You measure the effect correctly, but you seem less statistically certain of your results. Measurement error in the explanatory variable (character, say) biases the measured effect of the variable toward zero. You understate how strong the effect really is.
Right. There is another kind of error in the explained variable and that is when a proxy is used for the "real" thing like arrests for crime, doctor visits/hospital stays for illness.