The Program for International Student Assessment (PISA) is a worldwide study and its results are widely used to analyze and compare education levels across nations. PISA, developed and administered under the auspices of the Organization for Economic Cooperation and Development (OECD), is a system of international assessments in over 80 countries and economies that measures 15-year-old students’ capabilities (mostly in grades 9-10) in reading literacy, mathematics literacy, and science literacy every three years.
In conducting this type of international study, the students who participate are usually selected from all schools of the country for the results to be representative and valid, that is, to reflect the state of the country as accurately as possible. That is, there should be a balanced representation of urban, rural, lowland and remote village schools. And along with this, students should be randomly selected to ensure that the results of the study will be statistically valid. It is in accordance with these requirements that the results of this study reflect the situation in the country’s education system as a whole, subject to some degree of the statistical margin of error.
Azerbaijan participated in PISA 2006 and 2009. But in the 2009 PISA, Azerbaijan, on average, placed second to last among the 65 participating countries. As a result, the government withdrew from the Program, accusing PISA of bias and double standards. Nevertheless, Azerbaijan renewed participation in the PISA in 2018. On 5 December 2022 the most recent results were announced. Compared to the previous results from PISA 2018, Azerbaijan in 2022 experienced a considerable drop in performance: the average score of students for proficiency in reading in Azerbaijani fell from 389 to 365 (-24), in mathematics from 420 to 397 (-23), and in science from 398 to 380 (-18).
The purpose of PISA is to conduct studies and comparisons specifically by country, not by city or province. In fact, the participating countries themselves spend considerable funds on this expensive assessment in order to obtain an accurate sample from across the country. Until 2018, China was the only nation to break this tradition in the PISA study. As a rule, China is represented in this study through 3 or 4 provinces where the richest and best schools are located, contrary to rules of the assessment described above. Naturally, China limits the scope of the assessment to improve its position but also because covering 1.4 billion people with the assessment presents particular logistical challenges.
Because Azerbaijan in previous years (2006 and 2009) participated in PISA studies according to the rules of country-wide selection we described above, we can confidently say that our country’s results for those years are comparable to those of other countries. But Azerbaijan changed the selection process beginning in 2018. That year, the Ministry of Education decided not to select a country-wide cross-section of students like the other 75 participating countries, but instead, followed authoritarian China, selecting students from Baku, where the level of education is the highest. Even under war conditions, Ukraine in the name of maximum representativeness, secured the participation of 18 out of 27 regions in its version of the study. However, Azerbaijan, which has more than 4,000 schools, was represented in the 2018 study only by sample schools drawn from among 300 schools in Baku. Naturally, the results are artificially inflated.
Thus, 6.827 students from 197 schools selected from 300 schools in Baku participated in the 2018 PISA survey. The number of Baku-based schools selected by the government was slightly reduced from 197 to 178 for the 2022 PISA study. The study was scheduled for 2021 but was conducted in 2022 due to the coronavirus pandemic.
The purpose of this sample manipulation is to artificially exaggerate the results of the survey in Azerbaijan. For example, thanks to the manipulation, the country’s 2018 results were, although only slightly, higher than those of Kazakhstan. In 2009 Azerbaijan lagged far behind Kazakhstan. However, in the 2022 study, the concentrated Baku sample did not improve the country’s metrics, because Azerbaijan again lagged behind Kazakhstan as well as Georgia, who had higher scores than Azerbaijan in two out of three subject areas. Nevertheless, pro-governmental media and education experts deceptively approached the already manipulated data, stating that Azerbaijan ranks ahead of Georgia and Uzbekistan, outperforming regional countries and ranking 3rd among CIS countries.
As the tables show, despite these efforts, Azerbaijan actually failed to make good progress in the ranking. While the other post-Soviet countries on the list (Estonia, Latvia, Lithuania, Moldova, Ukraine) take up position above or closer to the OECD average (even Estonia ranks first among European countries), Azerbaijan, more specifically, Baku lags behind the OECD average by at least 100 points in almost all indicators. This also means that students in Baku lag behind their peers from other countries by about 1,5-2 years in terms of the skills and habits measured by the study.
The government’s main objective in selecting students to participate in this study from the best schools across the country has been to artificially exaggerate the results and create the impression that there have been improvements in education. Had the selection of schools and students in Azerbaijan been done in a representative manner, the results likely would have been lower than they are now. However, limiting this prestigious and costly international survey to Baku was actually a trivial and wasteful move for several reasons. Firstly, the results are not representative of the entire country. Secondly, since these results encompass Baku only, they cannot be compared with the previous years (2006 and 2009 results). Thirdly, it is wrong to compare indicators in one city—Baku—with those in other countries.
When the decision was made to renew participation in the PISA program in 2018, the current Minister of Education Emin Amrullayev was one of those who led that effort at that time as head of department. He said in an interview then: “The good news is that we have renewed participation in the PISA because, after its results, we will understand to what extent our education is aligned with global standards.” Namely, the main purpose of participating in this study is to analyze the state of the country’s education system, not just Baku schools.
Undoubtedly, thanks to the PISA study in 2006 and 2009, it had been possible to obtain reliable results that more objectively reflect the current state of Azerbaijan’s education system and are comparable with the results of more than 80 countries. However, contrary to what Education Minister Emin Amrullayev said above, a limitation of the research scope made it impossible to obtain objective results across the country. If the minister, exaggerating the results of the research through manipulation, intended that they prove the development of Azerbaijani education and the success of earlier educational reforms, today even the exaggerated results for the city of Baku make it impossible to take such claims seriously.
Overall, a review of country-level educational reforms over the past 20 years shows that there haven’t been major changes in the authoritarian nature of education in Azerbaijan. True, the education sector is run by relatively young professionals who have studied in Western European countries. Additionally, schools are being renovated, modern equipment purchased, class layouts changed, exquisite textbooks published, and STEM centers opened. However, reforms in both education management and curriculum have been very limited, and modern education in Azerbaijan continues the harmful authoritarian traditions of the Soviet education system.
