The researchers will develop and validate a set of new computer-based, online assessments embedded into the Assessing Data Modeling and Statistical Reasoning curriculum that are designed to provide diagnostic information about middle school students' learning of statistics and modeling. After an initial iterative development process, the team will complete a series of studies to determine the reliability and validity of the new online assessment system.
Setting
The study will be conducted in urban, suburban, and rural middle schools in Arkansas.
Sample
The study sample will consist of a diverse sample of 9 middle school teachers and approximately 450 of their students.
The assessments will undergo an iterative process of development and refinement. The researchers will focus on: (1) refining the Berkeley Evaluation and Assessment Research Software Environment (BEAR SE) to support the development, calibration, use and training of the new assessment system; (2) embedding the relevant ADM curriculum materials into BEAR SE, constructing computer-deliverable and computer-scorable task equivalents of the current paper-and-pencil items, and developing new computerized reports and support materials for teachers; and (3) investigating the usefulness of the new software in the context of the ADM curriculum. Nine teachers familiar with the ADM curriculum will be selected to participate in the assessment development activities. The first usability and validity study of the computer-based assessments and software tools will be conducted with curriculum developers, teachers, and students. In this phase of the study, the researchers will begin to explore evidence related to reliability and validity, including interviewing students as they work with items in the BEAR SE. Once revisions to the new ADM technology enhancements and online formative assessments are complete, the system will be re-tested with 9 teachers and their students (approximately 450 students) to assess its reliability and validity.
Control condition
Due to the nature of the research design, there is no control condition.
Key measures
The key measures for the study include students' responses on the assessments, and student and teacher interviews, observations, and responses to questionnaires.
Data analytic strategy
Analyses of reliability and validity will be carried out using item response modeling of student data with the Multidimensional Random Coefficients Multinomial Logit model. Wright Maps of the levels of item responses will be produced, along with differential item functioning analysis by gender and ethnicity. The relationship of the various constructs, which are part of the learning progression, will be examined by estimating the connection between the constructs and by estimating the overall difficulty of the achievement levels across the different constructs.
Products: Products include a fully developed and validated computerized formative assessment system focusing on statistics and modeling in middle school, a website for teachers to access the materials, and peer-reviewed publications.
Book chapter
Lehrer, R., Kim, M.J., Ayers, E., and Wilson, M. (2013). Toward Establishing a Learning Progression to Support the Development of Statistical Reasoning. In J. Confrey, and A. Maloney (Eds.), Learning Over Time: Learning Trajectories in Mathematics Education . Charlotte, NC: Information Age Publishers.
Lehrer, R., Kim, M-J., Ayers, E., and Wilson, M. (2014). Toward establishing a learning progression to support the development of statistical reasoning. In A. Maloney, J. Confrey and K. Nguyen (Eds.), Learning over Time: Learning Trajectories in Mathematics Education (pp. 31-60).
Journal article, monograph, or newsletter
Fisher, W. P., Jr., and Wilson, M. (2015). Building a Productive Trading Zone in Educational Assessment Research and Practice. Pensamiento Educativo: Revista de Investigacion Educacional Latinoamericana, 52 (2): 55-78.
Fisher, W.P., Jr., and Wilson, M. (2015). Building a productive trading zone in educational assessment research and practice. Pensamiento Educativo: Revista de Investigacion Educacional Latinoamericana, 52 (2): 55-78.
Irribarra, D.T., Freund, R., Fisher, W., and Wilson, M. (2015). Metrological Traceability in Education: A Practical Online System for Measuring and Managing Middle School Mathematics Instruction. Journal of Physics: Conference Series, 588.
Wilson, M., Mari, L., Maul, A., and Torres Irribarra, D. (2015). A Comparison of Measurement Concepts Across Physical Science and Social Science Domains: Instrument Design, Calibration, and Measurement. Journal of Physics: Conference Series, 588.
Co-Principal Investigator: Richard Lehrer, Vanderbilt University
