Study shows that admissions test predicts later success at NYC's elite high schools

[jserraglio]

WSJ —> NEW YORK CITY — Some, including Mayor de Blasio, want to scrap the [SHSAT] admissions test in favor of a mix of applicants’ course grades and state test scores

A new puzzle piece emerged Friday in the fractious debate over how to fairly admit students to some of New York City’s most sought-after public high schools.
The city Department of Education released a 2013 report by a consulting firm it hired to analyze whether the entrance exam for Stuyvesant and seven other specialized high schools was a valid predictor of academic achievement.

Its conclusion: Yes, particularly in math and science, based on accepted students’ high school grades, state Regents results and Advanced Placement exams.

Mayor Bill de Blasio’s administration wants to scrap the admissions test in favor of a mix of applicants’ course grades and state test scores, saying that would find talented students who don’t test well, and diversify schools with few black and Latino students. Supporters of the test of math, reading and writing say it is the most objective, fair method for determining merit.

The Specialized High School Admissions Test was revised for fall 2017, to add more grammar and align questions more closely to the middle school curriculum. It is unclear exactly how much the 2013 analysis would relate to the current exam, which retained many core features. Backers of the admissions test are likely to hail the report as evidence that a single test can select top achievers.
The report by Metis Associates looked at all eighth-graders who took the Specialized High School Admissions Test every year from fall of 2005 through fall of 2009. The firm followed those who scored high enough to get into the exam schools, and those who missed the cutoff, comparing how well each group did through two years of high school, no matter what public school they ended up attending.
The study found those accepted to the exam schools had a mean grade point average of 3.1 after two years, compared with 2.4 for those who weren’t accepted.
Accepted students also got higher mean scores on state Regents exams, ranging from 83 to 93 out of 100 points, depending on the subject. Students who weren’t accepted had mean scores ranging from 69 to 79. Accepted students also fared better on two Advanced Placement exams that enough students took to analyze.
“It’s not at all surprising that a kid who did well on the test turns out to be good high school student,” said Toya Holness, a spokeswoman for the city Department of Education. “What the validity study misses is the kid who didn’t do as well on the test, or didn’t take it, but still stands an excellent chance of being successful in these high schools if they had the opportunity.”
The mayor wants to admit eighth-graders performing in the top 7% of each middle school in the city. The department says this year those students had an average state test score of 3.9, almost the same as the 4.1 average of students offered specialized high school seats, on a scale of 1 to 4.5.
Officials at Metis Associates couldn’t be reached Friday, and had referred previous requests for comment to the department.
The study was originally commissioned in response to a civil rights complaint about the admissions method. Reporters requested the study, but the department declined to release it until Friday, saying it had required legal review.
Supporters of the test said the mayor unfairly had suppressed a study that backed their argument. David Lee, education chair of the Chinese American Citizens Alliance of Greater New York, said in a release: “It’s an outrageous scandal that Mayor de Blasio hid from the public for more than four years a study proving the validity of the SHSAT, while he led a political campaign to eliminate” it.
Shael Polakow-Suransky, who was the department’s chief academic officer in 2013 and now is president of Bank Street College of Education, said the entrance exam was relatively cheap and “you get what you pay for.”
The test differentiates students on a narrow band of math and reading, he said.
“It is sufficient to do what it is designed to do, but I don’t think it should be held up as indicative of the full capacities you might want from a search for the most talented kids in the city,” Mr. Polakow-Suransky said.
The revised admissions test in 2017 eliminated the unpopular “scrambled paragraphs” that asked students to put a group of sentences in the right order, and the logistical reasoning questions, akin to word puzzles. Critics thought those challenges didn’t reflect what students learned in school and were especially susceptible to gaming through test prep.
Scott Overland, a spokesman for the test vendor, said by email that “Pearson works diligently with our state and district partners to create fair, valid and reliable assessments aligned to their needs.”

[jbuck919]

Oh, a standardized test predicts academic success? What news!

[jserraglio]

Oh, a standardized test predicts academic success? What news!

Yes, but actually it might be even bigger news if it did.

[jbuck919]

Oh, a standardized test predicts academic success? What news!

Yes, but actually it might be even bigger news if it did.

For generations, colleges have relied on these tests, which have proven reasonably reliable in predicting academic success. Although the people who produce the SAT and the ACT would deny it, these standardized norm-referenced tests are essentially IQ tests that assume exposure to subject matter at a certain level. As you know, SAT used to stand for Scholastic Aptitude Test, though it now stands for Scholastic Assessment Test. Of course, they were never the only criterion, though I can remember being offered a full scholarship at Michigan State of all places solely on the basis of my test scores. Your PC remark represents more and more the rule, but IMO universities will abandon this criterion at their peril, not to mention the peril of society. All other criteria can be rigged. Test scores do not lie.

[jserraglio]

Test scores do not lie.

That certainly has been received wisdom and, as far as it goes, appears plausible. The University of Chicago now thinks otherwise. I suspect Chicago offers students more than a four-year curriculum based on taking tests.

My only real problem is with the SAT verbal, not math. Much of the verbal test is rendered suspect by the fact that a savvy test-taker can answer a significant number of questions simply by skimming a passage and using a POE. I will concede, though, that measures have gotten more valid ever since they dispensed with the lengthy descriptions before the passage, which gave a student enough context in simple expository prose to allow her to answer w/o even reading the passage.

Actually I am not PC at all on this matter. The SAT and its lesser brethren are a fact of life. Over the years, I have loved teaching SAT prep mini-courses (I taught my first one at Fordham Prep and many iterations thereafter at my school in the first decade of this century) — how to beat the test — esp. to weaker students who are clueless about how to attack it. Very rewarding when and if they are able to improve significantly on their second try. Of course the weaker the student the easier it is to get a significant improvement. I usually encouraged the 90th percentile-and-up contingent to drop the class. They didn't need it.

[jbuck919]

Test scores do not lie.

That certainly has been received the received wisdom and, as far as it goes, appears to be a plausible. The University of Chicago now thinks otherwise. I suspect Chicago offers students more than a four-year curriculum based on taking tests.

My only real problem is with the SAT verbal, not math. Much of the verbal test is rendered suspect by the fact that a savvy test-taker can answer a significant number of questions simply by skimming a passage and using a POE. I will concede, though, that measures have gotten more valid ever since they dispensed with the lengthy descriptions before the passage, which gave a student enough context in simple expository prose to allow her to answer w/o even reading the passage.

Actually I am not PC at all on this matter. The SAT and its lesser brethren are a fact of life. Over the years, I have loved teaching SAT prep mini-courses (I taught my first one at Fordham Prep and many iterations thereafter at my school in the first decade of this century) — how to beat the test — esp. to weaker students who are clueless about how to attack it. Very rewarding when and if they are able to improve significantly on their second try. Of course the weaker the student the easier it is to get a significant improvement. I usually encouraged the 90th percentile-and-up contingent to drop the class.

I'm afraid for once you've gotten the better of me. POE? Power over the Internet? You know perfectly well that all standardized tests are strictly proctored so that there can be no cheating using electronic devices. A savvy test-taker, and I am certainly one, does not need any aid to apply test-wise knowledge. The wonder is how few have any test wiseness to enhance their scores, though I have tried to teach it many times and continue to do so as a substitute teacher.

The last test I took, and it was criterion-referenced rather than norm-referenced, was in contemplation of being a census taker in 2010. I didn't follow through, though I may try it again in 2020 if I live that long. Well of course (of course he says, ha ha) I got a perfect score, but most of the people in the room failed it. How, I thought, can anyone be so dumb as to fail a test like that one? Well, we elected Donald Trump, didn't we?

[jserraglio]

I got a perfect score, but most of the people in the room failed it. How, I thought, can anyone be so dumb as to fail a test like that one?

Power over the internet? That acronym, I suspect, would be POI, no?

POE —> process of elimination. Perfectly legal.

I agree the SAT verbal provides a valid measure of test-wisdom, as does the AP Eng Lit multiple choice test for that matter. More than that I won't commit to.

I also agree that Donald Trump inspires stupidity in others: after all, as you know, he's a qualified ILSC (Ivy League Snake Charmer) being a DAWD himself (Dumb Ass With Degree).

[living_stradivarius]

You mean using Harvard’s admissions criteria is preferable here? Damned if you do, damned if you don’t. There’s always an unhappyminority

[jbuck919]

I got a perfect score, but most of the people in the room failed it. How, I thought, can anyone be so dumb as to fail a test like that one?

Power over the internet? That acronym, I suspect, would be POI, no?

POE —> process of elimination. Perfectly legal.

I agree the SAT verbal provides a valid measure of test-wisdom, as does the AP Eng Lit multiple choice test for that matter. More than that I won't commit to.

I also agree that Donald Trump inspires stupidity in others: after all, as you know, he's a qualified ILSC (Ivy League Snake Charmer) being a DAWD himself (Dumb Ass With Degree).

Thanks for the clarification. There was a time when I could write multiple choice tests with plausible distractors (i.e., for those who don't know, the wrong answers), but it is a heck of a lot of work. What always drove me crazy was to encounter tests where the question, for instance, might be "What was the last name of the first President of the United States?" and the choices are maybe Smith, Jones, and Washington.

You would think, wouldn't you, that anyone who can score an 800 on the math SAT would also do well on the verbal. In fact, I had an absolutely brilliant math student who went on to become a cryptologist at the National Security Agency (which was also almost next door to the school where I taught), who barely made 600 on the SAT verbal.

[/quote]

[jserraglio]

You would think, wouldn't you, that anyone who can score an 800 on the math SAT would also do well on the verbal ... who barely made 600 on the SAT verbal.

I am not surprised. I think the SAT math test is what it claims to be — a test, and a good one, of mathematical reasoning.

I would hesitate to make a parallel claim for the SAT verbal test or even for the AP English multiple choice (the AP essays are another matter, presenting validity problems of a different sort).

One cannot know for sure, but your student's math brilliance may have actually worked against him when he faced the redundancies and inconsistencies that are part and parcel of every multiple choice test of critical reading I've ever seen. Of course, there might be any number of better explanations for the discrepancy between his two scores.

So I would not go so far as to call SAT verbal a fraud as the college-prof dad of one of my students did to challenge me for parroting the party line about the SAT's being a valid predictor of future academic success, but I am skeptical.

Now what is fun to do is to map invalidity on specific AP Eng lit tests (the SAT is not so much of a challenge to deconstruct) and show that invalidity to students during tutorial sessions so they can take advantage of it later on the real test.

So I would say that I respect the math test (maybe ETS should weight math equally with verbal) but I LOVE the verbal. It practically invites you to discover logic 'cheats' that are characteristic of one question but applicable by analogy to other questions of a similar type. I find this to be quite stimulating.