# Conclusions based on PISA results deserve further attention

October 24, 2016The tests results of the Programme for International Student Assessment (PISA), which often informs the development of academic policies in various countries, often receive rather simplified interpretations. As such, analysis of PISA data does not reflect the entire 'package' of school students' knowledge in one key area - mathematics. This is the opinion of researchers from National Research University Higher School of Economics (Russia), Stanford University (USA), and Michigan State University (USA).

The Organization for Economic Co-operation and Development (OECD) is the leading authority currently monitoring PISA educational assessments (since 2000). This process can uncover changes and developments in various education systems throughout the world, while also evaluating the effectiveness of strategic decision-making with respect to education. However, this type of analysis does not include data on teachers and the 'educational histories' of children and youth. Thus, the results of the OECD's monitoring may be inaccurately interpreted, thus hindering countries' ability to properly develop and establish educational policies.

The PISA measures the knowledge of 15-year old students in mathematics and the natural sciences, as well as their ability to work with various types of text. However, according to researchers in Russia and the USA, this monitoring can only provide a 'snapshot' of the knowledge of such students and, thus, does not consider earlier assessments of their abilities and grades, as well as the conditions affecting one's education (e.g., the cultural standards and progress of an entire class, the education level of a student's parents, a teacher's professional abilities and qualifications, etc.). At the same time, all of these factors may influence the movement (up or down) of the grades on the PISA. Furthermore, the researchers cited the results of math tests, which showed that such additional factors can significantly alter the conclusions drawn from the OECD's monitoring data.

During math tests, students usually must solve three types of problem. The most important for PISA evaluations are assignments in applied mathematics (e.g., with real-life applications such as measuring a frame for a photograph). A large amount of attention is paid to 'real life' problems since PISA tests aim to determine the depth of students' ability to apply their knowledge in order to solve essential problems. The second type of assignment is text-based problems with a large amount of additional information. In order to solve such problems, students must read through the text, determine what is most important, and then quickly adapt to the unfamiliar format of the given question. The third type are 'formal' mathematical problems (algebra and geometry), which require the application of complex formulas. PISA tests pay less attention to such problems, even though 'formal' mathematics are much more difficult than applied mathematics, often elevating mathematical thinking to an entirely new level. Furthermore, those students more often engaged in solving such problems usually have better PISA results that those whose teachers tend to focus on practical 'real life' examples.

In addition, the cultural sophistication of a student's family is a key factor (e.g., standard of living, linguistic abilities, what is done in their leisure time, etc.). Students from better educated families with access to many books at home usually show the best test results. Also, such students have the conditions in place enabling them to excel academically. At the same time, classes with many students from educated families are more likely to acquire the skills necessary to solve formal algebraic and geometrical tasks, and thus are able to succeed in PISA testing.

Better qualified teachers can be found in classes where there is a higher average level of culture, and, as such, they tend to give their students more assignments dealing with formal mathematics. During their analysis, researchers also discovered that teachers with a mathematical non-pedagogical background tend to give better instruction, thus helping to boost the average grades of high school seniors.

Thus, best suited for PISA test are those learners who have successfully mastered formal mathematics. The researchers stress that cramming for questions on PISA tests are simply not enough. Thus, in order to ensure a more objective assessment of students' knowledge and abilities in international monitoring, the factors influencing their success should be considered more fully. At the same time, in its recommendations, the OECD overestimates the role of schools in enhancing academic achievements, but underestimates other 'areas of influence' affecting evaluations of students' knowledge.

The results of this analysis were published in the article 'Revisiting the Relationship Between International Assessment Outcomes and Educational Production: Evidence From a Longitudinal PISA-TIMSS Sample' http://aer.sagepub.com/content/53/4/1054.full.pdf+html in the

*American Educational Research Journal*. The article's authors include: Andrey Zakharov (Deputy Head of the HSE Institute of Education/International Laboratory for Education Policy Analysis), Martin Cornoy (Academic Supervisor, Leading Research Fellow at the HSE Institute of Education/International Laboratory for Education Policy Analysis, distinguished Professor of Stanford University), Tatiana Khavenson (Research Fellow at the HSE Institute of Education/International Laboratory for Education Policy Analysis), Prashant Loyalka (Professor at Stanford University) and William H. Schmidt (distinguished professor at Michigan State University).

-end-

National Research University Higher School of Economics

**Related Mathematics Articles:**

More democracy through mathematics

For democratic elections to be fair, voting districts must have similar sizes.

For democratic elections to be fair, voting districts must have similar sizes.

How to color a lizard: From biology to mathematics

Skin color patterns in animals arise from microscopic interactions among colored cells that obey equations discovered by Alan Turing.

Skin color patterns in animals arise from microscopic interactions among colored cells that obey equations discovered by Alan Turing.

Mathematics supports a new way to classify viruses based on structure

New research supports a structure-based classification system for viruses which could help in the identification and treatment of emerging viruses.

New research supports a structure-based classification system for viruses which could help in the identification and treatment of emerging viruses.

US educators awarded for exemplary teaching in mathematics

Janet Heine Barnett, Caren Diefenderfer, and Tevian Dray were named the 2017 Deborah and Franklin Tepper Haimo Award winners by the Mathematical Association of America (MAA) for their teaching effectiveness and influence beyond their institutions.

Janet Heine Barnett, Caren Diefenderfer, and Tevian Dray were named the 2017 Deborah and Franklin Tepper Haimo Award winners by the Mathematical Association of America (MAA) for their teaching effectiveness and influence beyond their institutions.

Authors of year's best books in mathematics honored

Prizes for the year's best books in mathematics were awarded to Ian Stewart and Tim Chartier by the Mathematical Association of America (MAA) on Jan.

Prizes for the year's best books in mathematics were awarded to Ian Stewart and Tim Chartier by the Mathematical Association of America (MAA) on Jan.

The mathematics of coffee extraction: Searching for the ideal brew

Composed of over 1,800 chemical components, coffee is one of the most widely-consumed drinks in the world.

Composed of over 1,800 chemical components, coffee is one of the most widely-consumed drinks in the world.

Even physicists are 'afraid' of mathematics

Physicists avoid highly mathematical work despite being trained in advanced mathematics, new research suggests.

Physicists avoid highly mathematical work despite being trained in advanced mathematics, new research suggests.

Mathematics and music: New perspectives on the connections between these ancient arts

World-leading experts on music and mathematics present insights on the connections between these two ancient arts, especially as they relate to composition and performance, as well as creativity, education, and geometry.

World-leading experts on music and mathematics present insights on the connections between these two ancient arts, especially as they relate to composition and performance, as well as creativity, education, and geometry.

Kindergarteners' mathematics success hinges on preschool skills

In a study funded by the National Science Foundation, researchers at the University of Missouri discovered that preschoolers who better process words associated with numbers and understand the quantities associated with these words are more likely to have success with math when they enter kindergarten.

In a study funded by the National Science Foundation, researchers at the University of Missouri discovered that preschoolers who better process words associated with numbers and understand the quantities associated with these words are more likely to have success with math when they enter kindergarten.

First international mathematics research institute launched in Australia

World leaders in the mathematical sciences are visiting Melbourne for a series of research programs at Australia's first international research institute for mathematics and statistics.

World leaders in the mathematical sciences are visiting Melbourne for a series of research programs at Australia's first international research institute for mathematics and statistics.

**Related Mathematics Reading:**

## Best Science Podcasts 2019

We have hand picked the**best science podcasts**for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.

**Now Playing: TED Radio Hour**

**Jumpstarting Creativity**

Our greatest breakthroughs and triumphs have one thing in common: creativity. But how do you ignite it? And how do you rekindle it? This hour, TED speakers explore ideas on jumpstarting creativity. Guests include economist Tim Harford, producer Helen Marriage, artificial intelligence researcher Steve Engels, and behavioral scientist Marily Oppezzo.

**Now Playing: Science for the People**

**#524 The Human Network**

What does a network of humans look like and how does it work? How does information spread? How do decisions and opinions spread? What gets distorted as it moves through the network and why? This week we dig into the ins and outs of human networks with Matthew Jackson, Professor of Economics at Stanford University and author of the book "The Human Network: How Your Social Position Determines Your Power, Beliefs, and Behaviours".