Â鶹´å

Research Question:

To what extent do asynchronous interventions, with and without genAI, regarding audience awareness impact students’ learning outcomes, above and beyond what is typically taught in a course? 

Study Participants: 

592 students across 8 courses of various disciplines and two academic semesters (see Table 1).

Table 1. Courses studied.

*Courses receiving pre/post surveys.

Experimental Conditions & Study Design:

Students in all courses had a pre-existing writing assignment that asked students to write for a specific audience. Prior to that assignment, we manipulated both the amount and type of audience awareness practice (above and beyond any instruction traditionally employed by each instructor) using the following three conditions for each course (see Figure 1 and Table 2 for additional condition details). 

GenAI+ and GenAI- Conditions: 

Almost all instructors implemented the genAI+ and genAI- conditions synchronously during their 2025 Fall Semester course (one instructor implemented the conditions in Spring 2026). We randomly assigned students in each course to one of the two practice conditions in which students either used a customized ChatGPT-4o (GenAI+) or did not (GenAI-) (see Table 2). 

Control Condition:

Each course used a historical control condition (i.e., the course taught by the same instructor during a prior semester). Besides the genAI+ and genAI- interventions, instructors held many course features constant across semesters (e.g., writing assignment used as data source, whether or not peer feedback was provided on the writing assignment).

Figure 1. Study design.

Table 2. Comparison of online homework activities from genAI+ and genAI- conditions.

Data Sources:

1. Students’ performance on an extant writing assignment, designed by their course’s instructor and scored via a standardized rubric to evaluate audience awareness skills (i.e., vocabulary, tone, level of expertise, and use of evidence). Note. All scoring was done after both semesters in a randomized order, deidentified for student and condition.
2. Students’ prior academic performance (as a covariate in statistical models) measured as cumulative Quality Point Average (QPA) from the university registrar (similar to GPA or grade point average).

Findings:

RQ: To what extent do asynchronous interventions, with and without genAI regarding audience awareness impact students’ learning outcomes, above and beyond what is typically taught in a course? 

We observed significantly lower total rubric scores (~4%) in the control condition compared to both the genAI- and genAI+ conditions (F (2,588) = 3.28, p =.04, ɳp2= .01; see Table 3). However, the genAI- and genAI+ conditions did not differ from each other. When individually analyzing the rubric’s four component skills, we found no significant impact of condition on students’ performance regarding use of vocabulary, tone, and level of expertise, but, students demonstrated lower performance regarding use of evidence in control conditions than genAI- and genAI+ conditions (F (2,588) = 5.72, p =.003, ɳp2= .02). As with total rubric scores, use of evidence did not differ between genAI- and genAI+ conditions.

Table 3. Rubric Scores.

*significant effect of condition (p < .05)

Eberly Center’s Takeaways (*adapted from submitted manuscript): 

This study demonstrates the need for audience awareness instruction across disciplines. On average, students in our control condition scored 67% on our audience awareness rubric, indicating that students are not at ceiling. This finding is notable given the control sample context: highly motivated students at a highly selective institution enrolled in courses with substantive audience awareness instruction beyond our intervention.

Here, relatively short, asynchronous, scaffolded activities (both with and without genAI use) targeting audience awareness skills resulted in small but significant increases in students’ writing ability, especially regarding the use of evidence. Therefore, customizing genAI agents as  “thought partners” (Pottmeyer et al., 2026) may create beneficial practice and feedback opportunities before students begin drafting a writing assignment. However, our data also demonstrate that students may receive the same benefits if instructors simply add targeted, well-aligned practice that leverages evidence-based principles of learning. This finding aligns with other studies in which replacing a mechanism of practice or feedback with genAI leads to comparable student performance regarding student writing (Li & Collins, 2026; Pottmeyer et al., 2026). 

Consequently, we recommend that instructors align instructional strategies with learning outcomes based on learning principles, rather than the purported affordances of particular technologies alone. And, we advocate for more research to identify the contexts in which practice and feedback with genAI creates positively impactful and unique learning opportunities regarding students’ composition skills, especially at scale.  

Creating customized chatbots is not necessarily difficult, and tools continue to evolve to be more user friendly. However, please note that our pedagogical implementation using ChatGPT-4o agents required considerable effort regarding user testing and quality assurance of outputs. Therefore, we would not expect students using off-the-shelf genAI tools with their own prompt engineering to achieve the same results reported here.

References: 

Li, A.W., & Collins, P. (2026). Formative feedback across sources: Student perceptions and writing outcomes with instructor, peer, and AI-generated feedback. Reading and Writing.

Pottmeyer, L., Hershock, C., Wetzel, D., & Mineroff, Z. (2026). Impact of using generative AI as a “feedback partner” on students’ writing quality. Journal on Excellence in College Teaching, 37.