Raw Thought

by Aaron Swartz

Trials of Testing

Since many readers complained about the previous piece we have pulled it and provided this replacement. The things we do for you people!

It takes little courage to denounce men who believe they can harness the power of their minds to fly and use a space of universal consciousness to create world peace. And, in the long run, it is of little consequence. No one can recall the obscure psuedoscientific claims of yesteryear.

But take the idea that underneath the skull lie a series of organs for human traits like acquisitiveness and amorousness which bulge and change the shape of the head with dominance. The idea seems equally preposterous but it held real sway in its era — the Massachusetts Medical Association and the president of Harvard threw their weight behind it (Paul, 7) and phrenology continues to be remembered today.

Such absurd ideas are not remnants of a bygone past — just replace “organs” with “genes” and you’ll have the new “science” of evolutionary psychology (formerly sociobiology), an absurdity which Harvard University’s own president has thrown the institution’s weight behind. And yet one rarely sees “pro-science skeptics” challenging its claims. Indeed, scientific magazines write them up with only minor questioning, saving their ire for those who dare criticize the ideas.

But at least such fields have critics (and I count myself among them). There are related claims, however, that exercise much more power over our lives and (perhaps as a result) are far less challenged. One of their creators explained that they would “promote personal development”, “manage conflict”, and “increase human understanding worldwide.” (Paul, 121) But instead of Vedic science, she was talking about here creation: the Myers-Briggs personality test.

I have written before about the failures of experiments to provide evidence in favor of our concepts of personality or intelligence and how despite this many continue to believe in them. One can discuss how even studies by proponents find that IQ lacks validity and that 47% of people have a different Myers-Briggs personality type on a second administration of a test. But this somehow seems not to convince. So let us try another tack: let us look at how these tests are made.

The history of the IQ test — along with a number of other supposed ways of measuring “intelligence” — is detailed in Stephen Jay Gould’s classic The Mismeasure of Man. It was originally created by Alfred Binet to find children in French schools who might need special tutoring. Binet thought that by locating and helping these students, one could make sure that everyone learned all the material. Binet composed the test by throwing together whatever questions came to mind: things about shapes and numbers and words. He just wanted to see if some kids were having trouble, he made no attempt to make sure the result was a balanced measure of “intelligence”.

Lewis Terman, a professor at Stanford University, imported the Binet test to America, added some more random things and mixed it all up a little, and called the result the Stanford-Binet intelligence test (a name which is still used today).1 One of the test’s first applications was American Psychological Association president Robert Yerkes’s attempt to classify the people recruited for the Army. Among the questions:

Crisco is a: patent medicine, disinfectant, toothpaste, food product
The number of a Kaffir’s legs is: 2, 4, 6, 8
Christy Mathewson is famous as a: writer, artist, baseball player, comedian

Recent immigrants, whose command of English might be understandably weak, were allowed to take a pictorial version: drawing “a rivet in a pocket knife, a filament in a light bulb, a horn on a phonograph, a net on a tennis court, and a ball in a bowler’s hand (marked wrong, Yerkes explained, if an examinee drew the ball in the alley, for you can tell from the bowler’s posture that he has not yet released the ball).” (Gould, 230)

Terman, meanwhile, conducted a longitudinal study of the people his IQ test marked as “gifted”. Joel Shurkin, based on exclusive access to the records, documented the full story in his book Terman’s Kids. Among the study’s participants was a man named Jess Oppenheimer. “Gave the impression of being very pushy and forward although he did not show these characteristics during the interview,” wrote one of Terman’s assistants. “I could detect no signs of a sense of a humor.” (Shurkin, 54) Oppenheimer went on to create and write the shows I Love Lucy and Get Smart.

The story of personality tests is little better. In her book The Cult of Personality (recently republished as The Cult of Personality Testing), Annie Murphy Paul (a former senior editor for mass bi-monthly Psychology Today) describes the history of all the major personality tests. Take the Minnesota Multiphasic Personality Inventory (MMPI), which was created in a similar way to the IQ test.

The test was created by psychologist Starke Hathaway and neuropsychiatrist J. Charnley McKinley by simply coming up with a bunch of true-or-false statements that they thought might indicate whether the respondent had a mental illness. Among them:

I have never had any black, tarry-looking bowel movements.
I have had no difficulty starting or holding my urine.
I have never indulged in any unusual sexual practices.
There is something wrong with my sex organs.
I believe there is a Devil and a Hell in the afterlife.
Everything is turning out as the Bible said it would.
I think I would like to belong to a motorcycle club.
Often I feel as if there were a tight band around my head.
I loved my father.
I like to flirt.
I believe my sins are unpardonable.
I have a good appetite.
I think Lincoln was greater than Washington.
Women should not be allowed to drink in cocktail bars.
A large number of people are guilty of bad sexual conduct.
If the money were right, I would like to work for a circus or carnival. (Paul, 53)

The resulting test was administered to the patients at the University of Minnesota mental hospital as well as the (presumably sane) staff there (all white, Protestant, Minnesotans who came to be known as the “Minnesota Normals”). Statistical analysis was then done to determine which questions more accurately predicted whether the user had a mental illness and more specifically, what kind.2

This was quickly generalized: people who scored above-average on the scales for Hysteria or Depression (but not high enough to actually have a mental illness) could be said to have hysterical or depressive personalities, even though there was absolutely no evidence to support this leap (not that it was on particularly sturdy ground to begin with).

The resulting test was used to analyze people in business, the army, court, high school, and at the doctor’s. It was “used to screen job applicants, offer vocational advice, settle custody disputes, and determine legal status.” (Paul, 58f) And while the test engendered some backlash, it continues to be used frequently today, often as the a requirement for getting or keeping a job. Paul notes “the MMPI (in an updated version) is employed by 86% of clinical psychologists and administered, by one estimate, to 15 million Americans each year.” (63) For example, it is used by 60% of police departments to evaluate prospective officers. Meanwhile, studies show that such tests can reject as high as 60% of healthy applicants.

This is but one example — and one chapter in Paul’s book — but all the others all have similar stories. An absurd test, concocted through absurd means, completely untested, ends up becoming a powerful societal force. All the more reason for us to speak out about them.


  1. Incidentally, although Terman did not put his name on the test, his family continues to have a presence at Stanford. His son Frederick Emmons Terman was a professor of engineering (and later provost); the Terman Engineering Center, which was across the street from my dorm, is named in his honor. And down the hall from me in my dorm lived his daughter, who, in full disclosure, I ate meals with a couple times. 

  2. Not that this methodology is necessarily flawed, although it leads to some interesting conclusions. Paul writes that in one experiment, the question “that yielded some of the most useful information” about whether someone had a fascist personality was: “Obedience and respect for authority are the most important virtues children should learn.” (Paul, 147) 

You should follow me on twitter here.

October 28, 2005

Comments

Ok, that was a great post. I don’t know if (or why) you would care what I think, but in case you do:

This post has informational content that I had never seen before. The previous post didn’t.

This post criticizes something that goes under the radar of most people (myself included). The previous post’s criticisms were literally sophomoric - everyone expounds or rebutts those arguments from time to time.

posted by Mark on October 29, 2005 #

My favorite anectote about these sort of tests is in “Surely You’re Joking, Mr. Feynman!”. One of our greatest theoretical physicists was rejected from the army because he failed the standard psycological tests. Apparently he was classified as insane on the basis of his honest answers to questions - despite having previously been given high level security clearance to work on the Manhattan Project!

posted by Ian Gregory on October 29, 2005 #

Ha! And the tests pretty much assume that everyone’s “personality” (or whatever they call it) is completely static — maybe this is true, but it should be tested, not assumed. In any case, I don’t think it’s true.

posted by bi on October 31, 2005 #

Aaron:

Your post is appreciated as I wasn’t aware of the background of some of these tests. Obviously, an un-scientific test that is given the weight and backing of science is a flawed way to approach psychological testing (or any other kind of “scientific testing”). That is, I believe, self-evident.

In regards to “throwing the baby out with the bathwater”; however, there are ways to approach psychological typing that can be extremely useful in a day-t0o-day way. My wife is a therapist and her command of the distinctions of the Meyers-Brigg system makes it quite useful for her in working with her clients (don’t ask me, I don’t quite get the whole thing myself). In our company, we both use and provide for our clients a system called “Insights” based on Jung’s work in psychology types. It can be very useful as a way to understand one’s “preferences”. It is; however, not useful as something to pidgeon-hole people but truly as something useful to understand someone else’s preferences in order to facilitate better communication and interactions.

posted by Eric Morrison on October 31, 2005 #

1) ” Not that this methodology is necessarily flawed, although it leads to some interesting conclusions. Paul writes that in one experiment, the question “that yielded some of the most useful information” about whether someone had a fascist personality was: “Obedience and respect for authority are the most important virtues children should learn.” (Paul, 147)”

So you approve of this methodology when it supports your concepts of the “fascist” metality?

2) Do you really not think that “genes” influence personality and cognition and that these are worthy areas of study?

davidwat

posted by davidwat on November 2, 2005 #

I said that the statistical methodology was not necessarily flawed. Obviously looking for statistical predictors can be instructive in some instances. It’s problematic here for many reasons, the most obvious being that the subject gets to choose their responses.

2) Do you really not think that “genes” influence personality and cognition and that these are worthy areas of study?

I’m not sure what it means to be a “worthy” area of study. Surely everyone is justified in studying things that interest them. But I’ve seen no evidence that such a thing as personality exists, let alone that genes determine it in any meaningful sense. If you have some, do let me know.

posted by Aaron Swartz on November 8, 2005 #

It was “used to screen job applicants, offer vocational advice, settle custody disputes, and determine legal status.” (Paul, 58f) And while the test engendered some backlash, it continues to be used frequently today, often as the a requirement for getting or keeping a job.

Thats not really true. The MMPI isn’t used at all today. The new version (MMPI-2) has a different normative sample and some deleted/added items. The MMPI-2 is used in relatively limited personnel selection, such as you mentioned, with police officers. It is illegal to use measures of mental health in traditional personnel selection. Normal personality as a selection device, such as the NEO-FFI, is legal and often results in less adverse impact against minorities than “cogntive tests”.

As for the reliability of the MBTI, it depends on how it is scored and how skilled the person administering it is. It’s typically favored more as a developmental tool.

posted by Matt on January 1, 2006 #

You can also send comments by email.

Name
Site
Email (only used for direct replies)
Comments may be edited for length and content.

Powered by theinfo.org.