The report reviewed nearly 700 submissions of "evaluation documents", culled these down to 192 primary sources and sought to determine if the quality of the studies could be used to establish the effectiveness of the programs.
"This discussion of how to establish curricular effectiveness in mathematics is particularly relevant in light of President Bush's State of the Union address and budget, which includes significant expenditures on improving mathematics and science education in this country," Confrey said. "If the funds are to make a difference, thorough, valid and fair evaluations of materials will be critical."
Confrey presented "Evaluation Framework and Comparative Analysis" Feb. 17, 2006, at the annual meeting of the American Association for the Advancement of Science in St. Louis. "We concluded that due to methodological weaknesses and inadequate numbers in the evaluation studies, no determination of the effectiveness of 19 selected curricula could be made," Confrey said.
However, the committee made specific recommendations on how such evaluations should be conducted to provide proper information about curriculum effectiveness to decision-makers. In contrast to the perspective argued for by the Institute for Educational Sciences, the committee suggested not only comparative study, using experimental or quasi-experimental methods, but that one also needs a content analysis by experts in mathematics and mathematics education and case studies to shed light on what is working and how it works.
"Effectiveness requires an integrated judgment involving values, theory and empirical evidence on a cadre of studies," Confrey said.
At the AAAS conference, members of the committee presented their work with responses from others who have conducted some of the evaluation studies in mathematics studied by the committee.