Archive for the ‘Assessment’ Category

What Dropping American Government Could Mean for Maryland

Thursday, February 24th, 2011

Why are Governor O’Malley and Maryland legislators going out of their way to anger the state’s school teachers and jeopardize students’ reading performance? The budget that has been offered by the Governor and is now in the General Assembly drops the state’s high school exit exam in American government. This is a test Maryland teachers are enthusiastic about, in part because the existence of the test gives the subject of government standing in the curriculum and keeps it from getting sidelined by more test prep in reading and math skills.

Also, if Maryland wants to maintain its impressive performance on the NAEP reading test the state would be wise to hang on to the American government test. Evidence shows that it is by increasing students’ knowledge of a wide range of subjects, including civics, that students become better readers. Dropping the test may not only guarantee that Maryland high schoolers will graduate knowing less about our nation’s government, but that they will be poorer readers, too. Is that a gamble Maryland’s elected leaders are willing to take?

Lynne Munson

STEM Spelled with a “T” Alone

Wednesday, February 9th, 2011

“I think we have overemphasized science at the expense of engineering and technology,” says National Assessment Governing Board member Alan Friedman, who is former head of the New York Hall of Science.

Really? Well, let’s look at the evidence that American schools have “overemphasized science.”

  • U.S. students’ NAEP science scores are abysmal.
  • Schools are reporting less participation in science fairs.
  • The U.S. ranks alongside Mexico and the Slovak Republic, near the bottom of the PISA and TIMSS science test.
  • The U.S. provides thousands of H1-B visas to scientists in other nations to bring their knowledge here.

Despite data indicating that taking more science courses can increase student performance (on NAEP’s own science assessment), we increasingly see “STEM” being implemented without the “S” (science) or “M” (mathematics).  In fact, even the “E” (engineering) in STEM is being contorted to fit an an almost solely technology-driven agenda.

Look at NAGB’s plan for its new Technology and Engineering Literacy assessment. It is all “T” (technology)–with a dose of sociology.

Here are the three major areas the new NAEP test will assess:

Technology and Society: the effects that technology has on society and the natural world and the ethical questions that rise from those effects.

Design and Systems: the nature of technology, the engineering design process by which technologies are developed, and basic principles of dealing with everyday technologies, including maintenance and troubleshooting.

Information and Communication Technology: computers and software learning tools, networking systems and protocols, hand-held digital devices, and other technologies for accessing, creating, and communicating information and for facilitating creative expression.

So NAEP’s definition of “engineering” doesn’t include, say, understanding how a dam works or why a skyscraper stands. This new test will assess only students’ understanding of the engineering related to the design, maintenance, and “troubleshooting” of technology (e.g. computers and hand-held digital devices). This list of topics reads like it was drawn up by the human resources folks at Apple, Dell, and Intel. The new NAEP test is a narrow skills assessment that will make a strange bedfellow among NAEP’s other, core-subject assessments in the arts, geography, history, civics, and the like. We hope NAGB will rethink its approach to this test and make it more compatible with its other assessments.

Lynne Munson

Forgetting Homer Plessy

Monday, January 31st, 2011

In Plessy v. Ferguson, the United States Supreme Court’s interpretation of the Fourteenth Amendment

  1. supported segregation
  2. protected the rights of slave owners
  3. guaranteed the right to an attorney
  4. upheld freedom of expression

This question* comes directly from Maryland’s American government exam, a graduation requirement ensuring that students learn about segregation and the Civil Rights movement, about political systems, geography, and human interdependence, for starters.

But Maryland is poised to cut this assessment. A blow to learning from the governor of a state with Ed Week’s #1 education ranking.

When Maryland adopted this requirement (along with a biology test) three years ago, the state endowed subjects beyond NCLB-tested math and reading with a guaranteed place in high school curricula … and with all-important funding.

So Baltimore schools’ head rightly asks: “[W]hat does it signal, that government is suddenly less relevant than the other subjects? Why government and not the other tests?” We’d like to hear Governor O’Malley’s answer.

Although Maryland does hope to join with other states to develop a common social studies assessment, the state wouldn’t assess the subject again until at least 2015. We wonder—what happens to social studies in the four + year void? Could government and history courses be the new arts and foreign language programming—branded disposable in state budgets, and, consequently, by education officials?

Shame on Maryland and its governor for cutting Government from its budget. Let’s hope other states choose to support deep learning—teaching their students the history of “separate but equal,” its legacy, and its overturning.

*Answer 1 is the correct choice.

Stephanie Porowski and Meagan Estep

Page 50 of the Nation’s Report Card

Wednesday, January 26th, 2011

Buried deep in the report on the 2009 NAEP science assessment is what is likely the key to raising science achievement (and achievement in other subjects, for that matter). Past scores of data on race/ethnicity, state/jurisdiction, parental education, gender, school location and the rest, is a chart—figure 46—that shows how much simply taking more science courses improves student performance:

What they mean by “coursetaking category” is that they broke down 12th grade student performance according to how many (and what type of) science courses students took. The bottom bar on the chart indicates the performance of students who took just one science course in high school (either biology or another science course, most likely earth science). The middle bar shows the performance of students who took biology and chemistry. And the top bar illustrates the performance of students who took biology, chemistry, and physics.

The difference in performance is significant. Students who took both biology and chemistry scored 15 points higher than those who just took biology or any other single science course, and those who took physics in addition to biology and chemistry scored 33 points higher than single science course-takers.  A quick analysis shows (see our math and chart below) that this amounts, approximately, to an 11% improvement for each additional science course taken.  So students who took three science courses scored 22% higher than those who took just one.

The great disappointment—and the key reason, NAEP’s data suggests, for poor overall student performance on the science assessment—is that so few students are taking enough science classes. Just one-third of 12th graders who took the NAEP had taken biology, chemistry, and physics.  Thirty-eight percent had taken two courses and 28% just one.

So why doesn’t this finding—that taking more science vastly improves student performance on the NAEP—driving the discussion of the science NAEP scores? The course-taking data is relegated to the 9th paragraph (of 12) of the National Assessment Governing Board’s press release and page 50 of its 79-page report. And NAGB didn’t even bother computing the percentage gains associated with taking additional science courses.  As far as we can find, no one who has written about this report thus far has even mentioned the course-taking data.

For decades, our discussion about student performance has revolved around data points (race, location, etc. as listed above) that are irrelevant to what actually improves student achievement:  increasing student knowledge. It is time to put the content of education at the center of the reform discussion. The longer content, course-taking, and curriculum remain on the sidelines, the further our students and our nation will fall behind.

Lynne Munson

Click here for more Common Core commentary on NAEP.

China Wins!!!

Tuesday, December 7th, 2010

The new PISA results are out and they are fascinating, though not because the US’s lackluster performance has changed much. We’re still well below average in math (tying Ireland to rank 31st and well behind Estonia, Slovenia, and three different testing regions in China). We remain solidly average in reading, coming in 17th (deadlocked with Poland and Iceland). Our “big news” is that we moved up to a single point above average in science. America’s 15-year-olds now rank 23rd in the world in science.

The Chinese, meanwhile, participating in PISA for the first time, tested at the top in every subject. And it wasn’t even close. Students in Shanghai bested the previous top placeholder on the PISA math test–Singapore–by almost 40 points. US students fell 113 points behind their Chinese peers in math and 56 points behind in reading. And even our new and improved science score trailed China’s by 73 points.

Arne Duncan calls these scores “a wake-up call.” You think? I’m going to go out on limb here and predict that the Obama administration will use these scores as further fuel to do more of the same–push charters, promote merit pay, and spend more and more. We don’t oppose spending more on education or evaluating teachers more accurately, or creating more ways to deliver education. But there’s no evidence these activities are going to boost our students’ performance.

The Chinese are almost exclusively focused on getting their students to meet high standards in a wide range of subjects that have been spelled out in clear curriculum guidance that has been provided to all schools and teachers. They focus on what students should learn. While we are focused on how, where, when, why they learn.

Until we pour our attention and resources into high quality curricula that are proven to work we’ll continue to be average.

Lynne Munson

Lowering The Bar

Thursday, October 29th, 2009

The National Center for Education Statistics’ big report on state proficiency standards was released this morning. (Also see Fordham’s previous work on the subject.)

From the report:

In grade 4 reading, 31 states set grade 4 standards for proficiency (as measured on the NAEP scale) that were lower than the cut point for Basic performance on NAEP (208). In grade 8 reading, 15 states set standards that were lower than the Basic performance on NAEP (243).”

Math proficiency standards were not much better:

In grade 4 mathematics, seven states set standards for proficiency (as measured on the NAEP scale) that were lower than the Basic performance on NAEP (214). In grade 8 mathematics, eight states set standards that were lower than the Basic performance on NAEP (262).”

The report also finds:

“Most of the variation (approximately 70 percent) from state to state in the percentage of students scoring proficient or above on state tests can be explained by the variation in the level of difficulty of state standards for proficient performance. States with higher standards (as measured on the NAEP scale) had fewer students scoring proficient on state tests.”

The report (PDF) can be read here.

Off the Stump

Monday, August 24th, 2009

We’ve been thinking about Richard Rothstein’s commentary published two weeks ago in EdWeek. We’re particularly happy that Rothstein restated our concerns about the sampling in the 2008 NAEP arts assessment and that Rothstein believes, as we do, that an education that leaves out the full range of liberal arts and sciences simply isn’t an adequate education.

What struck us most, though, are the statements Rothstein unearthed from then-candidate Obama about schools narrowing their curriculum to boost student scores on standardized tests.  Rothstein quotes Obama lamenting the plight of today’s students: “All they can do is just study math and reading every day, all day long. They’ve eliminated recess, they’ve eliminated art and music.” He goes on:  “So part of the solution is changing No Child Left Behind, so that the assessment is one that takes into account all the factors that go into a good education.”

Clearly candidate Obama believed that detrimental curriculum narrowing had already taken place and should be addressed.  But does President Obama?  Not according to the priorities and policies of his Department of Education.  We’ve not heard a word of concern about curriculum narrowing from Arne.  And the only mention of curriculum at all in the proposed Race to the Top Guidelines is STEM (it is listed as a competitive priority).  But students need more than STEM and reading.  Candidate Obama knew that.  Maybe President Obama should review some of his stump speeches and share them with Arne.

James Elias and Lynne Munson

The NAEP Arts Assessment: Not the NAEP You Know

Wednesday, June 17th, 2009

How would you describe NAEP?  Here’s what comes to mind for me:  a rigorous achievement test given regularly to a statistically significant number of students in a large number of schools in every state.  That description certainly fits NAEP’s reading and math assessments.  In 2007 NAEP’s reading and math tests each were given to approximately 350,000 4th and 8th graders at more than 14,000 schools.

Now let’s look at the NAEP arts assessment.  The 2008 test was given to just 7,900 students in 520 schools.  Now, for analysis purposes, cut that number in half because the test was actually two tests-one each in music and visual arts-and half the sample took each. So fewer than 4,000 students in 260 schools took each test.  That’s about 80 kids in five schools in each state.  Also, it was given just to 8th graders-no 4th or 12th graders need apply.  And, this is only the second time the arts assessment has been given in over 25 years. Reading has been given 13 times during that same period.

NAEP’s arts assessment is a different class of test than the NAEP tests we talk about most often.  Let’s not be confused about the quality of data the arts test represents.

In fact, it may be what’s absent from the NAEP arts assessment that is most instructive.  In 1997, both the first and last time the arts test was given, they tried to administer it in the four arts disciplines selected by the National Assessment Governing Board:  music, visual arts, dance, and theater.  But an initial field test determined that there weren’t enough dance programs in the schools to create a nationally representative sample of students who had received dance instruction.  So they dropped dance from the test.  A 2008 field assessment found that there were now not enough theater programs to produce a reliable sample of testable students.  So neither dance nor theater was tested in 2008.  Well, that’s revealing.  What subjects will they be able to test a decade from now?  Any?

Lynne Munson

Imperiling Education in the Bay State – Under Cover of Night

Monday, May 11th, 2009

Today Massachusetts students rank at the top on NAEP and outscore students from most other nations-and other states-on TIMSS. But all of that could be changing.Our readers know that Massachusetts is one of ten states currently working to reshape their standards and assessments around the framework put forth by the Partnership for 21st Century Skills. We and many others, including the folks at the Pioneer Institute, a Massachusetts-based think tank, and the editorial boards of both the Boston Globe and Boston Herald, are surprised that a state with such a successful education system would want to imperil it by embracing the latest education fad. And-in part because this effort has come under such withering criticism-we’ve been watching closely to see how the commonwealth implements P21′s program. What we didn’t expect is for the commonwealth’s Department of Elementary and Secondary Education (DESE) to sneak 21st century skills into Massachusetts classrooms basically under cover of night.

Last week the Pioneer Institute learned that bureaucrats at the DESE have snuck 21st century skills into the statewide Massachusetts Comprehensive Assessment System (MCAS) without telling anyone-effectively changing the content of Massachusetts education without public or legislative input. Last fall DESE incorporated a 21st century skills emphasis into an RFR they issued to update and administer MCAS. They did this without gaining the approval of the state’s chief education policymaking group, the Board of Elementary and Secondary Education (BESE), or the state legislature. Prospective contractors were required to include in their bids assessments of 21st century skills that are aligned with the hotly debated recommendations of the state’s 21st Century Skills Task Force, even though the BESE has not adopted them. Indeed, the RFP was developed last summer and released last fall, before the BESE had received the task force’s recommendations. The five-year contract for $146 million was just awarded to Measured Progress, an organization that sits on the board of the Partnership for 21st Century Skills.

The state’s two largest newspapers have already registered their shock at DESE’s overreach. A handful of fad-chasing bureaucrats should not be allowed to dictate what Massachusetts students should know and be able to do. This decision cannot stand.

Lynne Munson

Ravitch Blogs Here on NAEP Results: Data Suggests NCLB Slows Student Progress

Monday, May 4th, 2009

Readers of this blog know we’re NCLB skeptics. So we were more than a little suspicious when we saw NCLB boosters crediting the law for the recent uptick in NAEP scores. Common Core co-chairman Diane Ravitch also was skeptical, so she mined the data and found the real story: NCLB is slowing achievement. Especially for minorities. Read her analysis here:

There are many ways to interpret the latest test scores released by the National Assessment of Educational Progress, which is why press reports have been so inconsistent. The New York Times pointed out that racial gaps were not narrowing, which was the point of No Child Left Behind. The scores showed higher performance since the 1970s, but the biggest gains were made by black and Hispanic students in the 1970s and 1980s. Margaret Spellings saw the scores as vindication of NCLB, saying that the results were best in grades 3-8, where the law focused.

The first thing that the public should understand is that there are two different versions of NAEP. There is “Main NAEP,” which tests American students (national, states, and certain cities) every other year in reading and mathematics at grades 4 and 8 (and occasionally tests other subjects as well, such as science, history, civics, writing, etc.). The other version of NAEP is called “Long Term Trend,” which tests only reading and math and is given nationally (not by state and not by city) every four years. Unlike Main NAEP, Long Term Trend NAEP tests students at ages 9, 13, and 17. The latest release is not Main NAEP (which will report later this year on the 2009 testing) but Long Term Trend NAEP. The results of Main NAEP and Long Term Trend NAEP are not precisely the same, because the tests are not precisely the same. Whereas Main NAEP is periodically revised to reflect state frameworks and professional consensus, Long Term Trend NAEP has essentially the same questions and content across many assessments (content has been periodically updated to remove obsolete items, such as a reference to “green stamps” or some technology that has disappeared).

So let’s take a look at the scores. At age 9, reading scores were up significantly from 2004 to 2008 by four points, from an average of 216 to 220. But the scores preceding NCLB increased even more from 1999 to 2004 (five years rather than four) by seven points. So the rate of progress did not increase and possibly decreased after NCLB. At age 13, reading scores were up significantly from 2004 to 2008, from 257 to 260. This is a finding that conflicts with Main NAEP, where eighth grade scores have been flat for the past 20 years. It should be noted, however, that the average score for age 13 (260) is identical to the score posted in 1992. So while there was a gain from 2004 to 2008, that gain takes achievement back to where it was 17 years ago. At age 17, reading scores are also up significantly, from 283 to 286. But too soon to pop the champagne corks, because this score is lower than it was in several previous assessments. In the late 1980s and early 1990s, it was 290.

In mathematics, there was a significant gain for 9-year-olds, whose scores rose four points from 2004 to 2008. But this was a smaller gain than the one posted from 1999-2004, which went from 232 to 241. The 13-year-olds saw a significant gain too of two points (from 279 to 281), which was smaller than the previous assessment, which was five points higher (from 276 to 281). The scores of 17-year-olds did not change. Indeed, they have barely budged since 1973. With the exception of a brief fall and rise in the 1980s, math scores at this age have been flat.

As for the racial gaps, the biggest improvement occurred between 1999 and 2004, when the gap between black and white students was reduced by 9 points, from 35 to 26 points. Since NCLB, from 2004 to 2008, the gap has shrunk by 3 points, from 27 to 24 points. At age 13, the black-white gap shrunk by four points, from 25 to 21; but from 1999 to 2004, it narrowed by seven points, from 29 to 22. At age 17, the black-white gap is 29 points, larger than in 2004, and much larger than it was in 1988, when it was only 20 points.

Thus, when one looks at the patterns, it suggests the following: First, our students are making gains, though not among 17-year-olds. Second, the gains they have made since NCLB are smaller than the gains they made in the years preceding NCLB. Third, even when they are significant, the gains are small. Fourth, the Long Term Trend data are not a resounding endorsement of NCLB. If anything, the slowing of the rate of progress suggests that NCLB is not a powerful instrument to improve student performance.

Diane Ravitch