In defense of standardized testing

asteriskea · February 23, 2007, 4:57pm

Science publishes findings in defense of standardized testing as a prime predictor of success - this time on the graduate and professional school level:

The last year hasn’t been a good one for the standardized testing industry, what with SAT scoring errors and more colleges dropping the test as a requirement. But on Thursday, the journal Science published a study backing the reliability of standardized testing in graduate and professional school admissions.

The study, a “meta-analysis” examining thousands of data sets on a range of tests, found that test scores are a better way to predict graduate and professional school success than are college grades, which may be influenced by grade inflation or the relative competitiveness of different student bodies. The study concluded that the most reliable way to admit students to graduate and professional school is a combination of using test scores and college grades. In addition, the study found that these tests predict just as well for minority and white students…

While authors of the new study have ties to testing entities, no testing company paid for the study and its publication in the prestigious journal Science was seen as a coup for the testing industry, some of whose officials have been talking excitedly about this research for months…

Bob Schaeffer, public education director of the National Center for Fair & Open Testing, disputed the findings, which he called “a meta-analysis of pro-testing meta-analyses” and said that the analysis ignored “considerable research that reaches opposite conclusions.” He compared the study to research sponsored by the tobacco industry to demonstrate that cigarettes do not cause cancer…

<a href=“http://www.insidehighered.com/news/2007/02/23/tests[/url]”>http://www.insidehighered.com/news/2007/02/23/tests</a>

emeraldkity4 · February 23, 2007, 5:33pm

I have a issue with standardized testing for grad school
My oldest has a learning disabilty- she has processing speed issues as well as some other things .
For the SAT she was allowed to both use a calculator ( as everyone is) and to have extended time to take the test as well.
This allowed her to more fully show what she was capable of in college ( she was well prepared for college- and her IQ is quite high)
In college, she had accomodations also, and and recently graduated from one of the most demanding schools in the country ( all students are required to write a thesis for graduation and must undergo an orals board as well for example)

She would like to attend graduate school, and requested extended time on teh GRE as she received on SAT as well as use of a calculator ( it is not allowed otherwise & one of the disadvantages of her processing issues are that she needs a lot of time for simple calculations without one)
All her requests were denied.
The reason seemed to be- that as she was about at the same level as the general population without accomodations- she didn’t need them.
However, if they actually looked at her test scores, they might have been able to see that while her overall IQ was in the neighborhood of 160± and that her averaged “performance” without accomodations was still within boundaries of normal intelligence, her ability in the areas where she needed accomodations was quite a bit below “average”.
That is what “learning disability” means.
That you have some areas that are far below others and not representative of your ability overall
This makes a huge difference in the process.
Not only was she denied accomodations by College Board for GRE although she was allowed accomodations a few years earlier for SAT by COllege Board.
But forcing her to take the test without accomodations, will probably result in a much lower score than she would have otherwise received, resulting in fewer offers of admission,and probably fewer offers that come with funds- which as a 1st gen college student, are sorely needed.

asteriskea · March 1, 2007, 8:01am

EK4- Since I am a parent squeezed by PSATs, SATs, and now impending GREs and the GMAT, I sympathize with your D’s situation and wonder why LD accommodation is granted for the SAT and not for the GRE. Perhaps the reasoning here is in order to get a “less noisy” result, the testing field has to be even - one would hope that individual graduate departments will take into account extenuating circumstances, letters of recommendation, and all those other “intangibles” such as character and ambition, for those students who show enough exceptional promise in a field to go on to do work on the graduate level but have less than stellar standardized stats.

…More than prior grades, courses taken and recommendations, the final study - a composite of 2,000 smaller, independent studies reflective of approximately half a million students - showed standardized tests scores best correlate to students’ future scholastic success.

The study was compiled over a six-month period by authors Nathan Kuncel, assistant professor of psychology, and Sarah Hertzell, a senior research scientist at Personnel Decisions Research Institutes.

Most of the data was drawn from studies conducted at postsecondary education institutions across the country to determine students’ success in graduate and professional programs.

“Individual studies can be very noisy,” Kuncel said. “When we compile them together, we can get a very robust estimate of what is actually going on.”

Hertzell said the vast application of the study’s findings across disciplines was somewhat surprising, though many test administration companies have conducted research with results paralleling the study’s findings.

Educational Testing Services administers the Graduate Record Examination, and senior research director Cathy Wendler said she was glad to see the study’s results, but not surprised by them.

She said standardized test scores are most valuable because they are a precise, consistent measurement among many variables.

“Graduate or undergraduate types of education can be different from one department to the next, even in the same university,” Wendler said. “Is an ‘A’ always an ‘A’? Is a ‘B’ always a ‘B’?” 

Dave Wilson, president and chief executive officer of the Graduate Management Admission Council, which offers the Graduate Management Admission Test, said grade point averages, letters of recommendation and essays can not be guaranteed to offer unbiased, accurate assessments of students.

“While our (test) performance is not improving, our grades are. Grade inflation is a reality,” he said. “Measurement does matter, and if we’re going to improve as a nation in the world, we need to have an objective measure of our own improvement.”

As a former member of an admissions committee at the University of Texas at Austin, Wilson found standardized tests to be the most helpful part of students’ portfolios in deciding admission. 

Ellen Julian, associate vice president of the Association of American Medical Colleges and director of the Medical College Admission Test, said standardized tests are imperative, but leave other factors unacknowledged.

“I think that admission tests do a great job at what they do, but I fear they’re often overused because they’re only assessing one part of an applicant’s entire profile,” she said. “It’s the part we know how to test, but that doesn’t make it more important than the things we don’t yet know how to test.”

Julian said noncognitive traits like ambition, altruism and compassion are crucial to success in the medical field, but are difficult to quantify with standardized testing…

For the University graduate programs requiring scores, a holistic review of various application elements is done alongside exam score analysis, according to Andrea Scott, director of the graduate school office of admissions.

“There are many factors and they all are very important,” she said. “But to say the GRE score outweighs anything else, I don’t think that can be said about our programs.”

Scott listed undergraduate transcripts and research, and GPA trends as other considered aspects of the applicant’s profile.

Only two-thirds of University graduate programs require students to report their GRE scores.

“The programs that are not using (scores) are probably missing out on information that is most valuable in choosing students who are most likely to succeed in their programs,” Hertzell said.

<a href=“http://www.mndaily.com/articles/2007/02/28/70965[/url]”>http://www.mndaily.com/articles/2007/02/28/70965</a>

hsseniorlooking · March 1, 2007, 8:16am

EK4:

Your Daughter should take the Grad School standardized tests. Then in her G school application attach a statement about her issue. I would think with a nice statement explaining her test issue and he excellent body of UG work that she should have no troble with G-school applications.

You don’t seem to have much of a choice as her accomodations were denied.

Good Luck!

marite · March 1, 2007, 8:43am

I don’t know about professional schools, which, I understand, are more stats -driven than grad schools-- but grad school admissions vary enormously in the emphasis that is placed on standardized testing. Some departments and some individuals may put lots of emphasis on standardized test scores, others very little. Since most admissions are handled at the departmental level, the personal perspectives of individual members of admissions committees also play a large role. GREs test scores may be the first of the last thing that a reader looks at.

mathmom · March 1, 2007, 8:53am

Many eons ago when I was a student rep on the architecture school admissions committee at Columbia I read the application of a kid with dyslexia. He had abyssmal GRE scores, decent grades, a nice portfolio and stellar recommendations all of which told us to ignore the GRE score. In the end we did, but it bothered me, that I had no way of really knowing how having dyslexia might impact his architectural work.

originaloog · March 1, 2007, 9:11am

Regards to the study cited in the OP, my reaction is “Duh”. In almost all situations more information is better in evaluating something. A gpa of 3.7 from XYZ College gives you part of the puzzle. The courses on the transcript provide another piece. A GRE score of 2150 gives you yet another. And faculty LoR’s, yep another piece. And the students work/research experience rounds out the package. Take away any one and the evaluator has less information and has more difficulty in making a valid decision.

The one(perhaps minor) benefit of the GRE is that it eliminates instiutional biases and factors. Is the transcript influenced by grade inflation, is the LoR’s and research ginned up in any way?

And yes there are specific cases which mathmon and ek note which admissions people need to take into account, to the extent of even dismissing poor GRE scores altogether.

marite · March 1, 2007, 10:00am

I had pretty absymal GRE scores. The courses I had taken in my own fields did not cover the materials that were being tested on the GRE exam, the existence of which I did not even know until some time in the fall of my senior year. I will forever be grateful that the graduate admissions committee chose to ignore those scores. I had straight As throughout grad school–better grades than I did in college.

xiggi · March 1, 2007, 10:34am

[quote]
Bob Schaeffer, public education director of the National Center for Fair & Open Testing, disputed the findings, which he called

asteriskea · March 1, 2007, 10:35am

Marite: Here is an interesting tidbit culled from the GRE site in regard to the OP GRE validity study:

<a href=“http://www.ets.org/portal/site/ets/menuitem.c988ba0e5dd572bada20bc47c3921509/?vgnextoid=fd0c964275cb6010VgnVCM10000022f95190RCRD&vgnextchannel=b195e3b5f64f4010VgnVCM10000022f95190RCRD[/url]”>http://www.ets.org/portal/site/ets/menuitem.c988ba0e5dd572bada20bc47c3921509/?vgnextoid=fd0c964275cb6010VgnVCM10000022f95190RCRD&vgnextchannel=b195e3b5f64f4010VgnVCM10000022f95190RCRD</a>

Hanna · March 1, 2007, 10:52am

“I don’t know about professional schools, which, I understand, are more stats -driven than grad schools”

This is enormously true of law schools; very true of med schools, and less true of business schools. We have plenty of arguments in the law school admissions world about the LSAT and whether it deserves its primary place.

weenie · March 1, 2007, 11:22am

I’ve actually become a huge fan of standardized testing. The more I see the huge discrepency in grade inflation/deflation between schools the more I think standardized tests add an important measurement into the equation.

marite · March 1, 2007, 11:36am

The subject test was what did me in. Luckily, I was switching fields in grad school and the admissions committee 1. realized I was a foreign student; 2. valued what I knew in my new field; 3. valued recs. My GPA was not great, owing to language barriers (“I sympathize, but we can’t give the economics test in French”) and cultural unfamiliarity (“what the heck are stocks and bonds?” ).
With so many more international students applying to grad school nowadays, reliance on GREs is even more on a case-by-case basis.

katliamom · March 1, 2007, 2:14pm

Hey, mathmom - did you ever find out how that dyslexic student did in grad school? Just curious.

molliebatmit · March 1, 2007, 8:42pm

Personally, as someone who took the GRE last year, I think it’s absurd to suggest that it’s a great predictor of success in graduate school. Success in graduate school has so much to do with nonacademic factors – do you care enough about your research to stick it out, did you choose a good thesis advisor, did you pick a fruitful research topic.

Also, given that 8% of GRE takers get a perfect score in math, I don’t see how GRE score could possibly be a reliable predictor of success in technical graduate programs – almost everybody who even applies to science and engineering grad programs has a great math score. The range of variation in math is very small, and verbal scores are rarely a factor unless they’re really bad.

asteriskea · April 3, 2007, 7:06am

NY Times article on the ETS cancellation of the revamped GRE:

…The new version, planned to be the biggest overhaul in the test’s history, was designed to prevent cheating and to produce a more accurate measure of students’ ability. But it would have been longer, more expensive and more difficult to administer…

It included revised sections on verbal reasoning, quantitative reasoning and analytical writing, totaling four hours in length. The current version is two and a half hours.The announcement yesterday caught many in the education establishment off guard. U.S. News & World Report published its widely read annual “America’s Best Graduate Schools” issue this week and included an article warning graduate students about the updated test set to make its debut in the fall.

Susan Kaplan, the director of graduate programs at Kaplan Test Prep and Admissions, said Kaplan had been advising its clients to take the current version of the G.R.E. before the new version was put in place.

“We do think the new test would have been more challenging,” Ms. Kaplan said, adding that she welcomed the announcement. “It’s a positive thing that E.T.S. realized that the change would do more harm than good,” she said.

Robert Schaeffer, public education director at FairTest, a group that opposes the broad use of standardized testing, said E.T.S. had still left the security problems of its test unsolved.

“They’re now using a system that they’ve admitted is a security risk,” Mr. Schaeffer said. “It certainly is a further chink in their armor and undermines the credibility of their product.”

While the risk still remains, Mr. Payne said, he believes it has so far been limited to four countries in Asia.

“If that same behavior spread to the English-speaking world, that would pose a very serious security risk for us,” Mr. Payne said. ?We monitor our G.R.E. scores regularly, and we see no evidence that there’s this type of organized sharing of memorized items. We consider it a threat that we’re constantly vigilant about."

<a href=“http://www.nytimes.com/2007/04/03/education/03test.html?_r=1&oref=slogin[/url]”>http://www.nytimes.com/2007/04/03/education/03test.html?_r=1&oref=slogin</a>

inverse · April 3, 2007, 7:24am

Perhap the link followed could make you feel better. Paul MacCready, the Engineer of the Century, also have dyslexia,and as I recalled, according to his book" Doing more with Less", it make him more careful about his work.

<a href=“http://www.achievement.org/autodoc/page/mac0int-1[/url]”>http://www.achievement.org/autodoc/page/mac0int-1</a>

mathmom · April 3, 2007, 9:28am

No I wish I could have. Only last year students were allowed to be on the committee so we never found out anything about the class we had let in. I’d be curious to know too, how much of a liability dyslexia would be to an architecture career. I’m sure it depends on the severity and what aspects of the field you get involved in.

AnonyMom · April 3, 2007, 8:45pm

Maybe I’m missing something here, but as I understand it, the ETS provides accomodations for learning disable students not only because testing them without accomodation fails to provide accurate information about their potential or achievement, but because the law requires it. In California, when the AMA decided to deny accomodation on the MCAT wholesale, the courts intervened on behalf of the LD students. (Take a look at <a href=“http://www.dralegal.org/cases/educat...ner_v_aamc.php[/url]”>http://www.dralegal.org/cases/educat...ner_v_aamc.php</a> ) This being the case, how does it make sense educationally, or pass muster legally, for the GRE to deny accomodation?

asteriskea · April 4, 2007, 7:18am

AnonyMom, I don’t think you are missing all that much here - the ETS does indeed provide accommodation for LD students BUT the problem here seems to be that for the GRE calculators are not allowed - period. It appears that LD accommodation for the GRE does not cover math dyslexia or dyscalculia in any shape or form and is not even considered to be at issue. I sympathize greatly with EK4 and wish her D the best of luck for grad school.

In defense of standardized testing

CONNECT WITH US