The research is using SAT score as a proxy for general intelligence... I wonder if this sort of heuristic short-cutting actually correlates with test-taking ability more than it correlates with intellgence.
A lot of "test-taking" training basically consists of saving time by training away from full reasoning, in favor of cheap-and-good-enough heuristics. Furthermore, those heuristics are over-fitted to the particular problem types on standardized tests. I wonder how much of this study is actually measuring their ability to trigger test-taking instincts on problem types they're not designed for.
This was my first thought as well. Especially since most standardized tests do not include "trick questions" of these sorts (or if they do, they are often painfully obvious), the more "intelligent" are just answering questions quickly thinking they are all straightforward.
The test itself aside, I feel as if often in real life people correlate speed with intelligence, and the "smart" people that we know are the ones that are able to come up with (usually) correct answers quickly. Therefore it makes sense that these smart people would have a lot of heuristics that allow them to do so. To be clear though, the causation here is more complex than what is implied in the article. People who have heuristic strategies are characterized as smart, as opposed to the other way around.
I was thinking something along these lines too. I would guess that people who make these sorts of mistakes when they're in test mode might not make them in a real-life situation
When you're taking a test, you have the expectation that the problem was designed by a human who wants you to demonstrate a particular piece of knowledge. You look for subtle clues in the words that point to which core problem was in the mind of the person who wrote the question.
The experimental setup seems like it would catch out people who make the assumption that the problems were designed to test knowledge.
The research is using SAT score as a proxy for general intelligence.
Despite a number of statements to the contrary in the various comments here, taking SAT scores as an informative correlate (proxy) of what psychologists call "general intelligence" is a procedure often found in the professional literature of psychology, with the warrant of studies specifically on that issue. Note that it is standard usage among psychologists to treat "general intelligence" as a term that basically equates with "scoring well on IQ tests and good proxies of IQ tests," which is why the submitted article has a point.
"Indeed, research suggests that SAT scores load highly on the first principal factor of a factor analysis of cognitive measures; a finding that strongly suggests that the SAT is g loaded (Frey & Detterman, 2004)."
"Furthermore, the SAT is largely a measure of general intelligence. Scores on the SAT correlate very highly with scores on standardized tests of intelligence, and like IQ scores, are stable across time and not easily increased through training, coaching or practice."
"Numeracy’s effects can be examined when controlling for other proxies of general intelligence (e.g., SAT scores; Stanovich & West, 2008)."
As I have heard the issue discussed in the local "journal club" I participate in with professors and graduate students of psychology who focus on human behavioral genetics (including the genetics of IQ), one thing that makes the SAT a very good proxy of general intelligence is that its item content is disclosed (in released previous tests that can be used as practice tests), so that almost the only difference between one test-taker and another in performance on the SAT is generally and consistently getting all of the various items correct, which certainly takes cognitive strengths.
I still think Stanovich's point is interesting that there are very strong correlations with IQ scores and SAT scores with some of what everyone regards as "smart" behavior (and which psychologists by convention call "general intelligence") while there are still other kinds of tests that plainly have indisputable right answers that high-IQ people are able to muff.
(Disclosure: I enjoy this kind of research discussion partly because I am acquainted with one large group of high-IQ young people
This is a wonderful comment. However, what psychologists know as g is the result of over-interpretation of a descriptive method known as factor analysis (which is very similar to principal components analysis).
Also, the SAT grew from IQ tests, so it really isn't surprising that the two measures are correlated, given that the questions for SAT were probably kept on the test because they correlated with IQ tests.
That being said, the work of Kahneman (And the new paper which I should probably read instead of commenting on HN) is pretty rock solid as far as it goes. It is worth noting that there is a converse position in this field, that of Gerd Gigerenzer who argues that these heuristics exist because they are useful, and only go wrong in artificial situations.
www.cogsci.msu.edu/DSS/2007-2008/Todd/environments_that_make_us_smart.pdf
Personally, I incline far more towards the views of Gigerenzer than those of Kahneman, especially given that Gierenzer and colleagues attempt to model the mind given their theories computationally, which is something that psychology could do with more of.
Full disclosure: I'm a psychologist who's very frustrated with the lack of statistical sophistication and interpretation in my field.
Isn't there also another argument that could be made?
Suppose Y is only weakly correlated to X, Y might still be used to show that Z and X correlated if we can show that Y and Z are not otherwise correlated.
If only some smart people score high on SAT and most who score high on SAT are more easily are easily fooled by certain questions, it might indicate most smart people are easily fooled by these question but you would want to engage in further investigation to be sure...
It seems very strange to me that they would see these biases as inherently harmful, rather than the root difference between smart people (who can think effectively and quickly using non-linear reasoning) and stupid people (who can't.)
In general I would be reasonably satisfied with a definition of intelligence which described how usefully one is able to employ intuitive cognition.
Not especially, although being able to comprehend rules and structure is a biproduct of systems-level thinking. However, the "non-linear" part of my definition also implied an ability to question, circumvent or simply ignore such systems as well, and includes the ability to reason within a set of rules without believing that set of rules to be accurate or true.
Systems and rulesets can be a useful method to structure memory and reason, but being bound by them is detrimental to long-term correctness and comprehension. Being able to write syntactically-correct programs is useful, but neither necessary nor sufficient to be an excellent programmer. A meta-understanding of the effect such syntax has on the program at hand is more useful than always correctly following it.
I'm surprised to read that people who did better on the SAT did worse with the bat-and-ball question in the article. That sounds like exactly the sort of simple trap I'd expect to find in an SAT math question.
"...test-taking" training basically consists of saving time by training away from full reasoning...
A quote from one of my professors:
"Tests test what tests test."
Tests end up serving as an observable criteria for identifying intelligence. As long as their proper function is understood, they are useful. They are not not necessarily helpful in identifying who will be most successful in business, the political arena, etc. Because of the increased attention on testing, most reasonably informed people develop test taking ability as a somewhat independent skill.
It is ridiculous to suppose that one's score on a multiple choice test is an accurate measure of innate ability or real-world intelligence.
The SAT, ACT, IQ tests, and all standardized test like them are socially constructed concepts that ATTEMPT a method of measuring intelligence. Intelligence (in the real world) reaches far beyond one's abilities to answer multiple choice reading comprehension, basic math and writing. Not to mention that problem solving in the real world has no time constraints.
Beethoven would not have gotten a perfect score on his SAT's. However, we all can attest to his innovation, creativity and musical genius. How can a multiple choice test measure the creative abilities of people like Sir Richard Branson, Steve Jobs, or Pablo Picasso?
The idea that "smarter people... were slightly more vulnerable to common mental mistakes" is a nonsensical conclusion. These findings are completely worthless.
How on earth would any test be anything more than a "socially constructed concept that ATTEMPTs [sic] [to be] a method of measuring intelligence". That just sounds like a definition to me, not an indictment. Do you have a better test?
No one claims the SAT, or any other test, is the final word. But at the same time pretty much everyone accepts that that "general intelligence" (or something like it) exists, and that tests are a reasonably good proxy for detecting it. To first approximation, students who do well on the SAT are successful in other ways associated with "intelligence".
And that -- the fact that the SAT correlates with something under study -- is all that is needed for good science. Even poor correlations can be enlightening if the data (and scientist) is good enough.
The conclusion that "Smart People are Stupid" is wholly inaccurate. My point was that there is no test that can accurately measure intelligence. As we know, intelligence is often intangible and abstract.
Wikipedia's Definition of intelligence:
Intelligence has been defined in many different ways, including the abilities, but not limited to, abstract thought, understanding, self-awareness, communication, reasoning, learning, having emotional knowledge, retaining, planning, and problem solving.
Sure, many people that scored well on the SAT are intelligent, but that doesn't mean that those that didn't score well are not just as intelligent or capable.
The findings more accurately reflect the conclusion that: "Those With High Scores on the SAT Are Stupid"
The title and your point are both just playing on the semantics of the words. The study is measuring "general intelligence", which is a better defined (if still poorly understood and somewhat controversial) subject amenable to scientific study. Basically everyone in the field accepts that it's real. Even informally, think back to your school peers: I'm willing to bet good money that, on balance, the ones that everyone called "smart" got the best grades, got the best test scores and ultimately got the best jobs. All those things are correlations, and they can be measured scientifically. And they're real.
Basically, if you want to quibble with the headline of the post, then I'll grant that it's a little confusing (intentionally so, as are most good headlines), even if IMHO that point is a little specious. If you really want to claim "there is no test that can accurately measure intelligence" as a matter of scientific fact, you're just plain wrong, sorry.
First off, I agree that the SAT is not a good indicator of intelligence. That being said, this argument is missing the point:
> Beethoven would not have gotten a perfect score on his SAT's. However, we all can attest to his innovation, creativity and musical genius. How can a multiple choice test measure the creative abilities of people like Sir Richard Branson, Steve Jobs, or Pablo Picasso?
These tests are not meant to measure creative abilities. They are meant to measure the ability to solve math, reading comprehension, and writing problems. The SAT is used as one of several criteria in the college admissions process, NOT to try to predict who will be on a short list of history's most innovative people.
My point was that intelligence isn't easily defined. And that using the SATs as a measure of intelligence is absolutely ridiculous. The title "Why Smart People are Stupid" is misleading. The title "Why Good Standardized Test Takers are Stupid" more accurately reflects the findings of the research.
I completely agree to the point that any IQ-test like test is just a proxy to meassure what is commonly called intelligence. And these tests are just proxies, they bear in themselves the risk that any studiy based on them is more analysing the proxy (in this case the IQ-test, SAT or whatever) than the real thing (in this case intelligence).
On the orther hand I completly agree that is damn hard to meassure intelligence correctly. But as as long as you don't have proxy that works as well for "educated" westerners as it does for "uneducated" bush or jungle tribesmen you still have quite a high risk of error in your studies. Just my 5 cents.
P.S.: Upvoted rstevensons posts, don't see any reason to down vote him for being critical about multiple choice tests as a basis for such studies.
The questions, judging from the ones in the article, seem to be about testing one's logical reasoning skills. The SATs, at least the qualitative portion, attempt to measure one's math and logical reasoning skills. It seems entirely reasonable to me to assume that someone who scored highly on the SATs should not be tripped up by these kind of questions, and the finding that they are actually more inclined to be tripped up surprises me.
It might be better just to look at a student's qualitative portion of the exam, since one could score highly on the SATs while still getting a (relatively) poor score on the section most similar to these kinds of questions.
Which is why the paper used "various cognitive measurements" which together could be taken as a proxy for intelligence instead of relying on only on S.A.T. scores.
warning: reading this post is a loss of time unless you're rstevenson542.
> all standardized test are socially constructed concepts that ATTEMPT a method of measuring intelligence
as opposed to ones that don't attempt and have not been constructed by a society? What are we hoping for here, exactly: some ray of light shone upon us by god almighty which will let us know that, without doubt, those men are smart and those other ones are stupid?
> The idea that "smarter people... were slightly more vulnerable to common mental mistakes" is a nonsensical conclusion. These findings are completely worthless.
> Intelligence (in the real world) reaches far beyond one's abilities to answer multiple choice reading comprehension, basic math and writing.
Good thing I'm in La-La-Land, then, so I can answer multiple-choice reading comprehension all day long.
> Beethoven would not have gotten a perfect score on his SAT's
And you know this because you dug up his skeleton and it wouldn't mark answers? I'm not sure what to make of your assertion.
You hadn't quoted any passages from the original study that would display inadequate methodology or statistical error. Your findings are completely worthless.
I don't have access to Kahneman's full paper to scrutinize (yet - I'm asking around) but it seems possible to me that they wanted to place themselves in opposition to the general practices of education at the time. Notice that since his work began, education has changed drastically, several times over. Several examples of approaches tried are the classical lecture being jotted down to study at home; study groups with two-way communication; exam preparation; test preparation; collaborative group work; coaching; a specialist could go on.
Bear in mind that 50 years ago the SAT was still in flux and a lot of it was being experimented with and new. Given that it was one of the biggest, and most repeatedly renewed concerns for researchers like Kahneman, there's little to doubt he's not only an expert on the SAT, but also knows as many of its downsides as anyone.
Therefore, I believe this paper is a sort of "lessons learned" story which shows that his approach is better than the form of education being undertaken by schools.
⁂
I find it ESPECIALLY curious that neither the article nor anyone in this whole thread (and have I tried reading most of it) has commented on the process, and consequences, of being caught on trick questions like bat and ball.
It is my understanding that our intelligence has evolved through use in situations where its impact was more or less immediately visible, and where this feedback could be acted upon.
Example: shaping tools. Is the flint stone sharp? No. Mash it against rocks. Is it sharp now? It's a bit sharper, but not sharp enough. Mash it against rocks some more. Is it sharp enough now? OK, you're done shaping your spear head.
Example: hunting. Approach the prey. It runs away. You don't notice why, you hadn't taken wind into account when looking for clues. No feedback, therefore you couldn't act upon it.
Example: hunting. Approach the prey. It runs away. You notice it did after wind turned and gave away your position. You got feedback from the grass and leaves moving in the wind. Next time you'll be able to act upon this.
Example: trying to shake fruit off a tree. You find a low-hanging branch and try to shake it. First you try this way, then that way, and finally you find the best way to get fruit without making too much fall down.
The last example extends to any sort of experimenting, tinkering, happy-hacking.
It is however notable that the bat and ball question does not test that. There's a question, and you give an answer. There's no feedback before it becomes final, and once it does that is the clear cut-off. This represents the stone-cold, immovable, monolithic machinery displayed by many technical subjects, such as science, mathematics, and some forms of computer programming, but also strategy, and some forms of art. There is no iterative process, you get one try, based on which you can in no way build a tangible mental model of how something works and what parameters of your thinking you need to adjust in order to better yourself. One very stupid example is when someone in a job interview asks you about standard library function names and argument orders. Either you remember, or you don't. I'll call this feedback the feedback gap.
Bear in mind I have mentioned "happy-hacking" above and "computer programming" below. In fact, they're both computer programming. The difference? If I'm presented with a python program where I can use an iterative process, a repl, and its help() command that immediately gives me access to documentation, then I can very easily build up a mental model of what's going on. Exceptions and errors give me constant, constructive feedback which comes immediately. This immediacy is extremely important and even a slight delay makes the learning process slower. Additionally, if some things aren't available as immediate feedback, I can find out. For example, when trying to get at the fruit, I saw immediately where the fruit was in the tree branches. When typing out python, I don't have this, I don't see what the functions are, so I need to use help(). That works well enough. Some people like intellisense for that. Works well too. It all fills the feedback gap.
There's a similar difference in ease of progress between experimental mechanics and similar physics, versus branches where experimentation cannot happen. That's feedback gap again.
It is my belief that this sort of immediate feedback is needed in other technical subjects, especially mathematics and physics. Approaches such as theorem provers are helpful in mathematics, but they're nowhere near being complete, and nowhere near the utility and immediacy of a repl. I am fairly sure there are other ways in which the feedback gap can be filled. Perhaps different methodology, or differently structured theories, can give us more immediate feedback? Perhaps mathematical systems in which theorems are easier to tentatively prove or disprove can become more successful in breeding new results?
Or perhaps the theories are not to blame, but we need more tools. For example, my abstract geometry teacher kept reminding us that we needed to come up with such quick checks. Non-linearities were always useful. His favourite was the binary distance function, which was 0 for two identical points and 1 for different points. A lot of stupid theorems can be disproved by checking some examples with this.
Can someone else comment on any such tools?
Writing this comment definitely came with an insight or two for me. If you read it, thanks for going on this trip with me.
> It is my belief that this sort of immediate feedback is needed in other technical subjects ... Perhaps different methodology, or differently structured theories
The traditional feedback method is teamwork: the next student, or colleague, makes a claim about physics, and you exclaim: "Bullshit! As the mass of the pencil reduces to zero, the whole universe gets pulled off course." Traditionally, physicists and hackers were thick-skinned freaks; the triumph you felt more than made up for your bruised ego when the tables were turned. Now, normal people do these things; their feelings, more of resentment than glory, distract them inefficiently. They react the same way when computers call their bullshit.
A lot of "test-taking" training basically consists of saving time by training away from full reasoning, in favor of cheap-and-good-enough heuristics. Furthermore, those heuristics are over-fitted to the particular problem types on standardized tests. I wonder how much of this study is actually measuring their ability to trigger test-taking instincts on problem types they're not designed for.