Project Activities
In Years 1 and 2, researchers will create the student modeling engine and incorporate it into Rimac using an iterative cycle which will consist of alternating phases of software development, field testing, and refinement. In Year 3, researchers will conduct a final field test followed by the pilot study. The pilot study will compare Rimac with the student modeling engine to Rimac without the student modeling engine.
Structured Abstract
Setting
Participating high schools will be located in urban and suburban areas in Pennsylvania. The participating university will be located in an urban area in Pennsylvania.
Sample
Approximately 80 undergraduate students who have taken physics in high school but not in college and 120 high school students currently enrolled in physics will participate in field trials (about 20 undergraduate students and 30 high school students per field trial). Approximately 160 high school students enrolled in physics will participate in the pilot study. Researchers will include students from a range of high schools to ensure a socioeconomically diverse sample.
Rimac is a tutorial dialogue system that presents students with conceptual questions about just-solved physics problems and provides support to students as they answer those questions. The enhanced version of Rimac, which will be developed in this project, will include a student modeling engine. This engine will support the generation of adaptive tutorial dialogues by assessing a student's ability and determining the right level of support to provide to students.
Research design and methods
In Years 1 and 2, researchers will create the student modeling engine and incorporate it into Rimac using an iterative cycle which will consist of alternating phases of software development, field testing, and refinement. One field trial will compare alternative approaches to dialogue selection and student model updating approaches. Students will take a pre-test to initialize the student model and then will interact with the system. To determine the best dialogue selection and student model updating approaches, system output will be compared to data collected from human judges on the same tasks (dialogue selection and student model updating). Human judges will be provided with the same information the system receives about each student and will be asked to choose the dialogues most appropriate for the student (dialogue selection) and to rate whether students received the appropriate level of support (model updating). The system approach that most closely matches the human judges' response will be deemed the 'best' approach. Researchers will run four additional field trials to refine the system design. Students will take a pre-test, interact with the system, and then take a post-test. In Year 3, researchers will conduct a randomized controlled trial. The team will randomly assign students to a group that uses Rimac with the student modeling engine or to a group that uses Rimac without the student modeling engine. All students will take a pre-test, interact with the system, and then take a post-test. The study will take place over two to four sessions depending on classroom schedules. Students will fill out a "user satisfaction" survey after completing the study.
Control condition
In the control condition, students will interact with a version of Rimac that does not include the student modeling engine. They will solve the same set of problems in the system, view problems in the same order, and engage in the same set of reflective dialogues as students in the intervention condition. In this version of Rimac, dialogues will not adapt to what students know or do not know.
Key measures
Primary measures will include researcher-developed pre- and post-tests, which will draw upon items from a standard instrument of physics conceptual understanding, the Force Concept Inventory. Researchers will measure prior knowledge using students' Preliminary Scholastic Aptitude Test score. The research team will use percent of correct responses during reflective dialogues to see how well the tutor kept the discussion at the right difficulty level. Researchers will use log files to track student engagement and fidelity of implementation. The "user satisfaction" survey administered to students will include questions about the usability of the system.
Data analytic strategy
The research team will perform a one-way analysis of variance for each outcome variable of interest to compare the two versions of the Rimac system. The research team will use multiple regression to study the possible moderating role of student prior knowledge and student aptitude.
People and institutions involved
IES program contact(s)
Products and publications
The products of this project include an enhanced version of Rimac, a tutorial dialogue system for physics intended for high school physics students, and peer-reviewed publications.
Publications:
Albacete, P., Jordan, P., Katz, S., Chounta, I. A., & McLaren, B. M. (2019). The impact of student model updates on contingent scaffolding in a natural-language tutoring system. In Artificial Intelligence in Education: 20th International Conference, AIED 2019, Chicago, IL, USA, June 25-29, 2019, Proceedings, Part I 20 (pp. 37-47). Springer International Publishing.
Albacete, P., Jordan, P., Lusetich, D., Chounta, I. A., Katz, S., & McLaren, B. M. (2018). Providing proactive scaffolding during tutorial dialogue using guidance from student model predictions. In Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27–30, 2018, Proceedings, Part II 19 (pp. 20-25). Springer International Publishing.
Albacete, P., Silliman, S., & Jordan, P. (2017, June). A tool to assess fine-grained knowledge from correct and incorrect answers in online multiple-choice tests: An application to student modeling. In EdMedia+ Innovate Learning (pp. 988-996). Association for the Advancement of Computing in Education (AACE).
Chounta, I. A., Albacete, P., Jordan, P., Katz, S., & McLaren, B. M. (2017). The “Grey Area”: A computational approach to model the Zone of Proximal Development. In Data Driven Approaches in Digital Education: 12th European Conference on Technology Enhanced Learning, EC-TEL 2017, Tallinn, Estonia, September 12–15, 2017, Proceedings 12 (pp. 3-16). Springer International Publishing.
Jordan, P., Lusetich, D., & McLaren, B. M. (2018, June). What Does It Mean to Provide the Right Level of Support During Tutorial Dialogue?. In Intelligent Tutoring Systems: 14th International Conference, ITS 2018, Montreal, QC, Canada, June 11–15, 2018, Proceedings (Vol. 10858, p. 450). Springer.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
