# Conclusions based on PISA results deserve further attention

October 24, 2016The tests results of the Programme for International Student Assessment (PISA), which often informs the development of academic policies in various countries, often receive rather simplified interpretations. As such, analysis of PISA data does not reflect the entire 'package' of school students' knowledge in one key area - mathematics. This is the opinion of researchers from National Research University Higher School of Economics (Russia), Stanford University (USA), and Michigan State University (USA).

The Organization for Economic Co-operation and Development (OECD) is the leading authority currently monitoring PISA educational assessments (since 2000). This process can uncover changes and developments in various education systems throughout the world, while also evaluating the effectiveness of strategic decision-making with respect to education. However, this type of analysis does not include data on teachers and the 'educational histories' of children and youth. Thus, the results of the OECD's monitoring may be inaccurately interpreted, thus hindering countries' ability to properly develop and establish educational policies.

The PISA measures the knowledge of 15-year old students in mathematics and the natural sciences, as well as their ability to work with various types of text. However, according to researchers in Russia and the USA, this monitoring can only provide a 'snapshot' of the knowledge of such students and, thus, does not consider earlier assessments of their abilities and grades, as well as the conditions affecting one's education (e.g., the cultural standards and progress of an entire class, the education level of a student's parents, a teacher's professional abilities and qualifications, etc.). At the same time, all of these factors may influence the movement (up or down) of the grades on the PISA. Furthermore, the researchers cited the results of math tests, which showed that such additional factors can significantly alter the conclusions drawn from the OECD's monitoring data.

During math tests, students usually must solve three types of problem. The most important for PISA evaluations are assignments in applied mathematics (e.g., with real-life applications such as measuring a frame for a photograph). A large amount of attention is paid to 'real life' problems since PISA tests aim to determine the depth of students' ability to apply their knowledge in order to solve essential problems. The second type of assignment is text-based problems with a large amount of additional information. In order to solve such problems, students must read through the text, determine what is most important, and then quickly adapt to the unfamiliar format of the given question. The third type are 'formal' mathematical problems (algebra and geometry), which require the application of complex formulas. PISA tests pay less attention to such problems, even though 'formal' mathematics are much more difficult than applied mathematics, often elevating mathematical thinking to an entirely new level. Furthermore, those students more often engaged in solving such problems usually have better PISA results that those whose teachers tend to focus on practical 'real life' examples.

In addition, the cultural sophistication of a student's family is a key factor (e.g., standard of living, linguistic abilities, what is done in their leisure time, etc.). Students from better educated families with access to many books at home usually show the best test results. Also, such students have the conditions in place enabling them to excel academically. At the same time, classes with many students from educated families are more likely to acquire the skills necessary to solve formal algebraic and geometrical tasks, and thus are able to succeed in PISA testing.

Better qualified teachers can be found in classes where there is a higher average level of culture, and, as such, they tend to give their students more assignments dealing with formal mathematics. During their analysis, researchers also discovered that teachers with a mathematical non-pedagogical background tend to give better instruction, thus helping to boost the average grades of high school seniors.

Thus, best suited for PISA test are those learners who have successfully mastered formal mathematics. The researchers stress that cramming for questions on PISA tests are simply not enough. Thus, in order to ensure a more objective assessment of students' knowledge and abilities in international monitoring, the factors influencing their success should be considered more fully. At the same time, in its recommendations, the OECD overestimates the role of schools in enhancing academic achievements, but underestimates other 'areas of influence' affecting evaluations of students' knowledge.

The results of this analysis were published in the article 'Revisiting the Relationship Between International Assessment Outcomes and Educational Production: Evidence From a Longitudinal PISA-TIMSS Sample' http://aer.sagepub.com/content/53/4/1054.full.pdf+html in the

*American Educational Research Journal*. The article's authors include: Andrey Zakharov (Deputy Head of the HSE Institute of Education/International Laboratory for Education Policy Analysis), Martin Cornoy (Academic Supervisor, Leading Research Fellow at the HSE Institute of Education/International Laboratory for Education Policy Analysis, distinguished Professor of Stanford University), Tatiana Khavenson (Research Fellow at the HSE Institute of Education/International Laboratory for Education Policy Analysis), Prashant Loyalka (Professor at Stanford University) and William H. Schmidt (distinguished professor at Michigan State University).

-end-

National Research University Higher School of Economics

**Related Mathematics Articles:**

People can see beauty in complex mathematics, study shows

Ordinary people see beauty in complex mathematical arguments in the same way they can appreciate a beautiful landscape painting or a piano sonata.

Ordinary people see beauty in complex mathematical arguments in the same way they can appreciate a beautiful landscape painting or a piano sonata.

Improving geothermal HVAC systems with mathematics

Sustainable heating, ventilation, and air conditioning systems, such as those that harness low-enthalpy geothermal energy, are needed to reduce collective energy use and mitigate the continued effects of a warming climate.

Sustainable heating, ventilation, and air conditioning systems, such as those that harness low-enthalpy geothermal energy, are needed to reduce collective energy use and mitigate the continued effects of a warming climate.

How the power of mathematics can help assess lung function

Researchers at the University of Southampton have developed a new computational way of analyzing X-ray images of lungs, which could herald a breakthrough in the diagnosis and assessment of chronic obstructive pulmonary disease (COPD) and other lung diseases.

Researchers at the University of Southampton have developed a new computational way of analyzing X-ray images of lungs, which could herald a breakthrough in the diagnosis and assessment of chronic obstructive pulmonary disease (COPD) and other lung diseases.

Mathematics pushes innovation in 4-D printing

New mathematical results will provide a potential breakthrough in the design and the fabrication of the next generation of morphable materials.

New mathematical results will provide a potential breakthrough in the design and the fabrication of the next generation of morphable materials.

More democracy through mathematics

For democratic elections to be fair, voting districts must have similar sizes.

For democratic elections to be fair, voting districts must have similar sizes.

## Best Science Podcasts 2019

We have hand picked the**best science podcasts**for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.

**Now Playing: TED Radio Hour**

**Erasing The Stigma**

Many of us either cope with mental illness or know someone who does. But we still have a hard time talking about it. This hour, TED speakers explore ways to push past â and even erase â the stigma. Guests include musician and comedian Jordan Raskopoulos, neuroscientist and psychiatrist Thomas Insel, psychiatrist Dixon Chibanda, anxiety and depression researcher Olivia Remes, and entrepreneur Sangu Delle.

**Now Playing: Science for the People**

**#537 Science Journalism, Hold the Hype**

Everyone's seen a piece of science getting over-exaggerated in the media. Most people would be quick to blame journalists and big media for getting in wrong. In many cases, you'd be right. But there's other sources of hype in science journalism. and one of them can be found in the humble, and little-known press release. We're talking with Chris Chambers about doing science about science journalism, and where the hype creeps in. Related links: The association between exaggeration in health related science news and academic press releases: retrospective observational study Claims of causality in health news: a randomised trial This...