Achieve Study Methodology
From APEC HRDWG Wiki
On behalf of the Asia-Pacific Economic Cooperation (APEC) and the United States Department of Education, Achieve, Inc. conducted an analysis of the mathematics and science standards. The Analysis aims to qualify how the standards are organized, and to identify common content and performance expectations of APEC member economies’ standards. This page provides insight into how Achieve developed and implemented the study.
Contents |
Participating Economies
The twelve economies in this study volunteered for participation and have maintained involvement throughout the process, providing English-language copies of their standards, data about their students and explaining their educational systems and the approach under girding their standards. The following list details the twelve participants:
- Australia
- Canada
- China
- Chinese Taipei
- Hong Kong
- Japan
- Korea
- Malaysia
- New Zealand
- Singapore
- Thailand
- United States
The Coding Framework
The method of analysis used for this study was modeled on that used by Michigan State University in their 1997 study of content standards and textbooks. Detailed content and performance expectation frameworks, developed for use in the Third International Mathematics and Science Study (TIMSS), were applied by trained content analysts. Analysts assigned multiple content and performance codes to each block of text defined as a standard. Procedures were put into place to calibrate coders and monitor for bias or "drift" from established protocols. Experts from the member economies were invited to review the coding results and provide input.
The coding framework includes two components: a.) content categories, which address the topics covered, and b.) performance expectations, which address what students are expected to do with the content.
The content coding framework provides a detailed, comprehensive taxonomy of content for each subject. Broad categories are broken down into smaller units to allow for finer-grained comparisons. Coders were asked to code standards to the highest degree of specificity possible.
Coders used a similar taxonomy for performance, or cognitive skill, expectations. The performance skill codes are arranged in categories that approximate increasing levels of cognitive demand.
The Coded Standards
The standards analyzed by researchers are the national education standards in all economies, with the exception of a few economies. Australia, Canada and the United States do not publish a single set of national standards, but instead allow states or provinces to develop their own regional standards.
The research was conducted in English; therefore, member economies submitted translations of their standards where necessary. Because of the challenges inherent in conducting research on translations, linguistic nuances or differences may not be fully captured. Certain words may carry a meaning in one language that they do not carry in another. For example, according to Japanese content experts, "to know" connotes a different, more robust meaning in Japan than in the United States. For the purposes of this study, coders used a low-inference coding approach, coding what was obvious and evident in the printed word.
Some economies were unable to provide literal translations of all subjects, courses or grade levels. Therefore, some standards documents submitted for the study were summaries of the content and performance expectations, rather than word for word translations of the documents.
The Grade Span Approach
The grade-level organization of standards varies considerably by economy. To facilitate comparison of standards across economies, Achieve grouped standards into three best-fit grade spans for each subject.
Despite the obvious limitations to examining broader grade spans-- namely, less specificity about when topics are taught-- grouping by grade span facilitates examination of the accumulated content and skills taught by the end of the designated grade spans.
The Common Topics
The main focus of this analysis is the identification of topics that are common across the participating APEC economies. For purposes of this analysis, the common topics from the mathematics coding frameworks are defined as topics addressed by 67 percent or more of participating economies in this study. Sixty-seven percent represents a strong but reasonable consensus of economies (two-thirds or more) upon which to focus the analysis.
The Common Pathway
By far the most complex factor in an analysis of expectations for secondary school students is the enormous variation in course requirements and options. In addition to variations in course requirements, there are often several types of schools that students may attend at the secondary level, ranging from university-preparation schools to technical schools. Each type of school has its own set of courses and course standards. Even within a single economy different schools or regions may offer or require different courses. The standards and expectations for students depend on the schools they attend and the courses they take.
In order to determine the course sequence and corresponding standards followed by a majority (more than 50 percent) of students enrolled in secondary school in each member economy, analysts surveyed member economies on course requirements and clarified remaining doubts in direct communication with mathematics education experts in each economy. The standards for courses in which more than 50 percent of students enroll were then analyzed, as indicated by economies.
