Â鶹´å

Research Questions:

RQ 1: Can a relatively short, asynchronous, self-paced set of genAI learning modules improve genAI competency (i.e., knowledge, skills, and self-efficacy)?

RQ 2: Are impacts of the intervention experienced equitably by all students?

Study Participants: 

1,368 students across 53 different courses taught by 46 instructors:

• Control (no modules): n = 610 students
• Treatment (modules): n = 758 students 

With input from faculty, staff, and student genAI subject matter experts, four learning modules were designed by the office of the Vice Provost for Teaching & Learning Innovation and the Eberly Center for Teaching Excellence & Educational Innovation to foster genAI competency. GenAI competency is defined as the combination of knowledge, skills, and self-efficacy (Chiu et al., 2024). 

Students independently engaged with these 90-minute modules asynchronously. The learning experience included instruction, examples, practice, and immediate feedback to support the following learning objectives: 

(LO1) Describe the basic mechanisms behind how generative AI tools are built and how they work, 

(LO2) Explain why students’ decisions about and applications of generative AI tools will differ across individuals, contexts, tasks, and goals, 

(LO3) Analyze the ethical implications and other concerns of these tools to be a responsible user and/or creator, 

(LO4) Identify and apply strategies to appropriately and responsibly use generative AI for a given educational task, and 

(LO5) Report an increased level of self-efficacy for appropriately using generative AI tools in educational situations.

In Fall 2024, 46 instructors across the university volunteered to enroll at least one course in this study. Each course was randomly assigned to either the treatment or control condition. At the very beginning of the semester, students in both conditions completed a pre-test assessing knowledge, skills, and self-efficacy. Students in the treatment condition then had 48-hour access to the genAI learning modules before taking a post-test 4 days later. Students in the control condition did not have access to the modules until after they had completed the post-test during the same time window as the treatment condition (see Figure 1).

Figure 1. Study design for the genAI modules experiment.

Data Sources: 

1. Pre and post:
   1. Knowledge measured by 4 questions about genAI (LO1) and 5 questions about its responsible use (LO3)
   2. Skills measured by authentic tasks allowing students¹ to demonstrate:
      1. Authentic Task 1: Prompt engineering
      2. Authentic Task 2: Evaluation of output
   3. Attitudes measured by self-efficacy surveys regarding genAI competency 
2. Student demographic data obtained from the university registrar

¹We randomly selected a subset of n = 174 students and two independent raters rubric-scored their responses while unaware of condition and whether it was a pre or post response.

Findings:

RQ1: Engaging with the learning modules significantly improved students’ knowledge for how LLMs work (LO1), prompt engineering skills (authentic task 1), and self-efficacy above and beyond control students who did not complete the modules (see Figure 2). The modules did not affect students’ knowledge for responsible use (LO3) above and beyond what was experienced by the control group, nor did they affect overall output analysis skills, as measured by the second authentic task (see Figure 3).

Figure 2. Accounting for nesting of students within instructors’ courses, there was a significant main effect of time (pre-to-post change), F(1,1364.584) = 257.70, p < .001. There was a significant main effect of condition, F(1,27.993) = 6.272, p = .02. Importantly, there was a significant time x condition interaction, F(1,1364.584) = 45.867, p <.001. While both the treatment and control students improved over time, the modules significantly improved treatment students’ genAI knowledge on LO1 above and beyond other influences on students’ performance.

Figure 3. There was no significant main effect of time, F(1,171) = .23, p = .63, η_p² = .00. There was a significant main effect of condition, F(1,171) = 6.76, p = .01, η_p² = .04, indicating that  students in the treatment condition performed better overall (M = 56.40%, SE = 2.01%) than students in the control condition (M = 49.04%, SE = 1.99%). However, the predicted time x condition interaction was not significant, F(1,171) = 1.53, p = .22, η_p² = .01.

RQ2: There were equitable impacts of the learning modules across student demographics (discipline, sex, race/ethnicity, class year, and first-generation status). In the case of self-efficacy, the learning modules corrected a pre-existing inequity in female students, helping them to catch up to their male peers.

Eberly Center’s Takeaways: 

RQ1 & RQ2: A set of widely accessible, discipline-agnostic learning modules designed with university subject-matter experts improved components of students’ generative AI competency (i.e., knowledge for how LLMs work, self-efficacy, and prompt engineering skills). Students experienced these positive impacts equitably. We also observed increases in LO1 knowledge (how LLMs work) and self-efficacy for the control condition, likely due to practice and testing effects, and/or other ambient learning opportunities. However, the experimental nature of this study means that engaging with the modules significantly improved genAI knowledge and self-efficacy above and beyond other influences on students’ genAI competency. However, we did not see evidence of an effect of the modules on students’ knowledge of responsible use (LO3). A stronger learning intervention may be needed to impact this specific knowledge.

Students’ genAI competency is increasingly important to employers (Cengage Group, 2024) and educators (Watson & Rainie, 2026). Rather than relying on individual instructors’ efforts to foster genAI competency, Â鶹´å designed these modules to be a widely accessible and scalable intervention to support students’ educational and work-force development at any point in the curriculum. Consequently, Â鶹´å has incorporated portions of these modules into a required online course taken by all first-year undergraduate students. We also provide an updated version of these resources for faculty interested in foundational upskilling regarding genAI.

The present study advances genAI competency research by targeting skills and self-efficacy in addition to the more typical knowledge in order to prepare students more holistically to engage with genAI tools. Future directions for our team include further updating the modules’ content to reflect new genAI developments and strengthening the instruction, practice, and feedback on the responsible use of these tools and critical analysis of output. 

References

Chiu, T. K., Ahmad, Z., Ismailov, M., & Sanusi, I. T. (2024). What are artificial intelligence literacy and competency? A comprehensive framework to support them. Computers and Education Open, 6, 100171.

Cengage Group (2024). 2024 graduate employability report: Preparing students for the genAI-driven workplace. Cengage Group.

Watson, C. E., & Rainie, L. (2026). The AI challenge: How college faculty assess the present and future of higher education in the age of AI. American Association of Colleges and Universities & Elon University’s Imagining the Digital Future Center.