Introduction to the Concepts of Complex Sampling
in Large-Scale Assessments in Education
Time: 1pm-4pm, March 10th Room: Newberry/ Ascot
1. Objectives of the workshop:
The objective of the workshop is threefold:
a) To provide an introduction to the sampling methodology applied in large scale educational surveys, its background and significance. The concepts of complex sampling, stratification, and unbiasedness are defined and illustrated. The selection process is illustrated by means of examples. Further challenges in cross-national comparative studies are highlighted, e.g. the careful definition of target populations and the impact of reduced coverage and participation rates. The necessity of using weights when analyzing data from complex sampling designs is demonstrated and the calculation of sampling weights is explained.
b) To examine the implications for data analysis, particularly the need for specific methods of variance estimation, are examined. Jackknife repeated replication (JRR) and balanced repeated replication (BRR) as the most common variance estimation methods applied in large scale assessments are introduced and participants are pointed to software that can handle these methods. It should be noted that many of the introduced concepts apply also to national assessments.
c) To give participants the opportunity to apply their acquired knowledge to a practical example. They will be asked to develop a sampling design for a fictive study under consideration of the specific circumstances in their own country, and will be encouraged to present their results for discussion with the group.
2. Rationale for Workshop:
When analyzing or interpreting sample data, researchers should always consider the sampling design used to select the targeted populations (e.g., schools, classes, students, teachers etc.). This is of particular importance when focusing on complex random sample data, such as the ones used in large scale assessments in education (e.g. TIMSS, PIRLS, PISA). Failing to apply correct sampling weights and variance estimation procedures could lead to biased outcomes or/and misinterpretation of results. Researchers interested in secondary analysis of large scale assessment data will learn in this workshop about the particularities of these datasets caused by the fact that they come from complex samples and how to address these particularities.
In many large scale assessments, the samples are selected by sampling statisticians related with the international study centers. These experts select the samples after thorough consultation with the national representatives. In order to develop a sampling design that fits not only the international study goals but also specific national research interests, it is necessary for the contact persons to have a basic understanding about the concepts of sampling. The workshop will build this understanding, enabling participants to discuss sampling designs in sophisticated ways with executing sampling experts.
3. Target Audience:
The workshop is aimed at researchers interested in the logic of statistical sampling; researchers interested in secondary data analysis with large scale assessment data; and researchers involved in the implementation of sample surveys. Participants are expected to have a basic working knowledge of statistics. Technical items (e.g., computer or specific software) are not required for workshop attendees.
Maximum number of course participants: 30
4. Instructional goals:
After the training, participants will
• have a basic understanding of the general concepts of random sampling (randomness, unbiasedness, representativeness),
• have become familiar with specific considerations that apply to cross-national comparative studies (target population definitions, coverage/exclusions, participation rates)
• have become acquainted with different random sampling methods (simple, systematic, multistage, cluster, sampling with probabilities proportional to size),
• have learned about purposes and advantages of stratification,
• have been introduced to the technical selection process,
• have learned how to calculate selection probabilities, design weights and non-response adjustments,
• understand why the use of sampling weights is essential when analyzing sample data,
• be introduced to the use of correct variance estimation methods,
• have learned how to interpret analysis results under the presence of sampling error.
In summary, the course will enable participants to understand the implications that specific sampling designs applied in large-scale educational studies have on data analysis and interpretation. The workshop aims to transfer conceptual knowledge rather than to build practical capacities to actually conduct the full sampling process. Participants will be qualified to consider advantages and disadvantages of specific sampling designs in discussion with executing sampling statisticians.
5. Planned workshop schedule and activities:
• Sampling: Theory and underlying concepts (60 min)
• Sampling: Technical selection process (30 min)
• Calculation, use and impact of sampling weights (45 min)
• Variance estimation and interpretation of results (45 min)
• Hands-on training: Apply acquired knowledge to a practical example (30 min)
• 2 breaks, 15min each
6. Instructional Staff and Institutional Affiliation:
Dr. Sabine Meinck
IEA Data Processing and Research Center, Hamburg, Germany
Diego Cortes, M.Sc.
Position: Statistician
Institution: IEA Data Processing and Research Center
e-mail: diego.cortes@iea-dpc.de |
|
