Reopening a controversy: Rating schools on test scores

Critics: New policy may be misleading, distract from learning

Posted: Sunday, April 29, 2001

If you don't like judging a student by tests, try judging a whole school.

An upcoming rating system for Alaska's schools will rely on scores on standardized tests and the difference in scores from the previous year.

Whether that is fair, or helpful, is likely to be the next education dispute in Alaska, following a similar debate over the value of the high school graduation exam.

"I'm really concerned that we'll lose our attention on the whole child and get too narrow in our focus," said Juneau fifth-grade teacher Linda Frame.

In August 2002, Alaska will join a national trend in designating individual schools by quality. The idea is to identify poorly performing schools and spur them to improve. Each year, Alaska schools will be labeled as distinguished, successful, deficient or in crisis.

"Finally somebody somewhere will put something in writing and say this school needs help," said Vilette Knight, a Fairbanks parent on a committee that is recommending the designation criteria.

"I think there are parents out there who think their schools are just fine and don't have a clue the low level they're operating in," said School Designators Committee member Mari-Anne Gross, a former Kenai Peninsula school board member from Homer.

But critics of school ratings say averaging out students' test scores is a crude measurement of a school.

The public forgets that a school's average test score includes a lot of students above and below it, said Jim Cox, an educational consultant in Anaheim, Calif. Student scores overlap a lot among schools that are rated differently, he said.

"And yet the media will report this one score, and folks will take this one score and draw some incredible conclusions from it," Cox said.

The rating system likely will reopen all the old wounds about standardized tests. Some educators say those tests don't tell parents whether a school is good and can harm its quality.

"I have a deep aversion to this obsession with standardized testing to judge 'quality' education," said Juneau-Douglas High School English teacher Alison McKenna. "So much of what I think is essential in a genuine, successful, holistic education cannot be tested by filling in bubbles: Curiosity, growth, empathy, a sense of self-worth.

"It seems that legislative members and people who are not directly working with young people seem obsessed with quantifying everything," McKenna said. "How can we provide 'quality education' when we are incessantly asked to give up valuable classroom instruction time to administer these tests that we find arbitrary and irrelevant to what truly matters?"

What gets tested tends to be low-level material, and schools end up teaching to that level, said Monty Neill, executive director of the National Center for Fair and Open Testing in Cambridge, Mass.

High-scoring middle-class suburbs in Massachusetts and Connecticut have rebelled against standardized tests because the tested content isn't as good as the curriculum, he said.

A few states have much more complicated rating programs than Alaska is proposing. In Kentucky even elementary schools are judged on student performance in reading, math, social studies, science, basic skills, arts and humanities, practical skills, writing as demonstrated in a portfolio, attendance and the rate of students held back a grade.

Teaching to the test

Teachers in Alaska "will be forced to focus on those skills that can be measured with a pencil and paper so our schools score well," said JDHS science teacher Erik Lundquist, "and give up teaching our students to produce logical arguments, defend a position, work in cooperative teams, present to an audience or learn technology applications or create a long-term research project."

But it might not have to be that way, according to a three-year study of 2,000 school assignments for 5,000 elementary and middle school children in Chicago.

Students had larger gains on standardized tests of the basics if they received intellectually challenging classroom assignments, the report for the Chicago Annenberg Research Project found.

Researchers scored assignments on how much they required students to interpret knowledge or use it in a novel way, integrate factual knowledge with an in-depth understanding of a problem, communicate about it in elaborated writing, and connect academic knowledge to issues in the students' lives.

Students may have shown greater gains in basic skills, which are essentially vocabulary tests, because challenging assignments require students to think about and use words and concepts to solve real problems, the report said.

"They're probably internalizing a lot of vocabulary as a result of having to explain themselves and having to examine different ways to solve problems," said one of the researchers, Fred M. Newmann of the University of Wisconsin. Students also are more interested and motivated to do the work because it's more connected to their personal experience, he said.

Standardized tests often don't measure the more rigorous standards, agreed Jennifer Vranek, director of benchmarking and state services for Achieve, a nonprofit organization of state governors and corporate executives interested in education.

"But they are the only way we have right now to independently verify if students are learning," Vranek said from her Washington, D.C., office. Schools can become complacent, and their courses and grades can reflect that complacency, she said.

No test of success

Schools already have assessments, of course.

"They're called mid-terms and finals, quizzes and essays, oral presentations and demonstration projects," said JDHS English teacher Bill Ralston.

"When I come into contact with former students who are now working in the community, I notice it is often not those who fared the best on standardized tests who are successful contributing members of society. Usually it is kids who show a positive attitude and a good work ethic that make it," he said.

All of the states that rate schools use standardized test scores to do so, and some include dropout and attendance rates, said Brian Gong, a consultant to the School Designators Committee.

Are test scores enough to judge quality?

"I think most people would say no, but the limitation is it's hard to get good agreement and good measure of the less tangible features of what goes into good quality schools," said Gong, associate director of the nonprofit National Center for the Improvement of Educational Assessment, based in Portsmouth, N.H.

 

Off the charts: Erik Lundquist, left, a science teacher at Juneau-Douglas High School, explains a chemistry experiment to freshman Rye Walters during a lab class on Thursday. Lundquist says this type of learning wont be measured by Alaskas school rating system.

MICHAEL PENN / THE JUNEAU EMPIRE

Some parents and teachers in Massachusetts have recommended a decentralized accountability plan based on schools' own assessments, a limited use of standardized tests as a check, and reviews every few years by outside experts.

It would cost more than a system of standardized tests, Neill said, "but the results would be more powerful because it would engage teachers in thinking about what they're doing."

But teachers easily could coach students to reproduce past high-scoring essays and other assessment work, said Walt Haney, a researcher at the Center for the Study of Testing at Boston College.

Committee member Gross said standardized tests were "about the only valid data we're able to come up with." The committee didn't want to use course grades because those are subjective, she said.

"We have students that come through with straight As and have test scores that are hardly able to get into college," Gross said.

Classroom grades don't necessarily match well with performance on standardized tests, as the Juneau School District has learned.

About three-quarters of students getting an A in math met expectations on the district's math assessments in grades seven, eight and 10 last school year. But only a third to a half of B students met expectations. Meanwhile, about 15 percent to 20 percent of D and F students did OK on the tests.

"Semester marks did not appear to be a predictor of success on district and state assessments," a school district report said.

But it's also questionable whether standardized tests, especially those that compare students with a national sample, mean as much as classroom grades.

Juneau found that 59 percent of eighth-graders who got Ds in math nonetheless scored in the upper half of the California Achievement Test, and 43 percent of students with Fs did so. Those students may look proficient in a school rating system based on tests.

No fair?

Critics of school rating systems also say it's unfair to judge schools on what they can't control. Students' socioeconomic status is the strongest correlation with standardized tests scores nationally.

Kids bring more than lunch from home. Students come with their parents' expectations, attitudes toward school and English-language skills. They are affected by their parents' alcoholism and drug abuse. They are distracted by hunger or damaged by child abuse and fetal alcohol syndrome, among other baggage.

A school's program quality is only part of what determines test scores, said Cox, who has advised Alaska educators on how to interpret the state's new standards-based tests.

Scores are affected by student turnover, English proficiency and family income, Cox said. That shouldn't be used to excuse low scores, he said, but a rating system should account for variables the school can't control.

"Schools serve whomever they have, and that is really their charge," countered Gong. "We are trying to make it so it's a fair system. We're trying very hard not to say if they have these types of students it's OK if they don't learn as much."

The change in student scores from the previous year would be given two-thirds of the weight in determining the ratings. Emphasizing growth could lessen the effect of social and income differences among schools, the committee believes.

"To judge how well a school is doing, you need to take that out of it, and growth is the way to do it," said committee member Gross.

Schools from poor areas can do well in an improvement-based model. The schools that showed the greatest growth in Kentucky's rating index were in the poorest region, said Lisa Gross, a Kentucky education spokeswoman.

But a system that places the continuum of student achievement into four categories isn't going to reflect all improvement, critics said.

A student could grow a lot but still be working below grade level and score poorly on a standardized test, said Frame, a Juneau teacher. Students grow at their own rate, she added, and looking at one year's growth may not be significant enough.

Alaska intends to rate each school as one pool of students, although it will publish scores broken down by factors such as gender, ethnicity and poverty. But there's an argument to be made for taking into account such subgroups in the ratings themselves. If a school is rated as one pool, it could raise its scores by improvement mostly among white middle-class students.

Many of Alaska's schools are so small it wouldn't be statistically valid to break down a class by ethnicity, said Richard Smiley, administrator for assessments at the state Department of Education.

Texas requires schools to meet targets for students overall and in subgroups of ethnicity and poverty. To be rated acceptable, for example, at least half the school's students overall and in each group must pass state tests.

Texas has raised student scores overall and among all ethnic or income groups since it started rating schools in 1994, and the gap between white scores and minority scores has narrowed. At the same time, it has raised the bar for schools to be rated acceptable.

But the success is more apparent than real, test researcher Haney suggests. Texas didn't count special education students until 1999. The dropout rate has increased, and only half of minority students have progressed from ninth grade to graduation since 1990, he said.

It's too soon to say what impact school rating systems will have, said Kathy Christie of the Education Commission of the States, a nonprofit organization that advises state officials.

"Parents and the community are some of the biggest supporters," she said. "They really do appreciate being able to get clear information on how the schools in their community are doing, and I hear that always from parents. It's more the staff and school boards that have reservations."

Eric Fry can be reached at efry@juneauempire.com.



CONTACT US

  • Switchboard: 907-586-3740
  • Circulation and Delivery: 907-586-3740
  • Newsroom Fax: 907-586-3028
  • Business Fax: 907-586-9097
  • Accounts Receivable: 907-523-2230
  • View the Staff Directory
  • or Send feedback

ADVERTISING

SUBSCRIBER SERVICES

SOCIAL NETWORKING