Â鶹´å

Assistant Teaching Professor
Tepper School of Business
Spring 2024

70-340 Business Communications (14-week course)

Research Question(s): 

1. To what extent does genAI usage and scaffolding impact the quality of a student writing deliverable?
   • Section A - used generative AI, with instructor-provided LLM scaffolding
   • Section B - used generative AI, without instructor-provided LLM scaffolding
   • Section C - no generative AI, no scaffolding
2. To what extent does detailed generative AI-related scaffolding influence students’ perceptions about the utility of LLMs for assisting their growth and development as communicators across an entire semester? (Sections A and B only)

In one of three course sections, DeJeu scaffolded four mini-lectures showcasing genAI use cases in professional communication contexts (section A). Specifically, these lessons provided instruction and modeling on using ChatGPT or Copilot to revise a document, create model documents, identify "lexical bundles" (i.e., phrases and sentences that are used often in particular genres of writing), and generate ideas. Mini-lectures occurred in tandem with each of the four major writing assignments that included reflection questions and documentation regarding the writing process. Students were instructed to use genAI on their first writing assignment and were allowed to choose whether or not to use it for all subsequent assignments. In a second section (section B), DeJeu also instructed students to use genAI on the first writing assignment with permitted use on subsequent assignments, but she did not provide scaffolded instruction regarding genAI.

This study had three sections, two of which were taught by DeJeu (sections A and B), and the third was taught by a colleague (section C). In one of DeJeu’s two sections, students received in-class scaffolding for ethical and effective genAI tool use (section A) while students in the other did not (section B). In both sections, students were instructed to use genAI on the first writing assignment. DeJeu compared performance on the first writing assignment, and students’ global perceptions of genAI at the beginning and end of the semester across the two sections. In a third section taught by a colleague (section C), students were not permitted to use genAI on the first writing assignment. DeJeu compared writing performance on this same assignment across all three sections.

Sample size: Section A (24 students); Section B (21 students); Section C (23 students)

Data Sources:

1. One writing assignment, scored with a rubric by three trained coders who were unaware of the study and students’ section. This assignment was scored for various writing skills (e.g., use of rhetorical strategies, concision, coherence).
2. Pre/post surveys of students’ perceptions of genAI’s utility to influence their growth and development as communicators in terms of familiarity, helpfulness, and efficiency (sections A and B only).

1. RQ1: There was a significant difference in performance on the writing assignment among the three sections. Follow-up comparisons showed no difference between sections A and B, and significant differences between section C and both sections A and B. This overall difference was consistent across all rubric criteria.
   
   

   Figure 1. Students’ writing performance was significantly different across the three sections, F (2,64) = 8.33, p < .001, η_p² = .21. Students in section C (M = 9.67, SD = 2.27) performed significantly lower on the assignment than students in both section A (M = 11.96, SD = 1.75), p < .001, and section B (M = 11.67, SD = 2.27), p <. 01 . Error bars are 95% confidence intervals for the means.

2. RQ2: There was a significant increase in students’ perceived familiarity with genAI tools from pre to post across both sections A and B. There was a marginally significant interaction between section and time, suggesting a slightly greater increase from pre to post in section A (scaffolded genAI use) compared to section B (no scaffolding). There were no significant main effects or interactions for perceived helpfulness or efficiency of genAI tools to assist in their development as communicators.
   
   
   

   Figure 2. There was a significant main effect of time, F (1, 38) = 50.51, p < .001,  η_p² = .57, indicating a significant increase in students’ familiarity with genAI tools from the beginning to the end of the semester. The time x section interaction was marginally significant, F (1, 38) = 3.86, p = .06, indicating that the pre to post change was marginally greater for section A, p < .001, compared to section B, p < .01. Error bars are 95% confidence intervals for the means.

Eberly Center’s Takeaways: 

1. RQ1:  Students who were permitted to use genAI (sections A and B) performed significantly better on the writing assignment than students who were not permitted to use genAI (section C), as evaluated by three trained raters who were not informed about the nature of this project. This finding suggests that the use of genAI can help students turn in higher quality deliverables in their communications classes. It is important to note, however, that quality of the deliverable does not necessarily equate to greater learning. Further research is needed to test whether permitted genAI tool use impacts students’ development of underlying writing skills (e.g., on a transfer task completed without genAI), in addition to the quality of a single deliverable.
2. RQ2: Scaffolded instruction on how to use genAI slightly increased students’ perceived familiarity with using genAI to assist with communication-related tasks above what simply using the tool alone did. However, scaffolded instruction and use did not impact students’ perceptions about how helpful genAI is to their growth or efficiency as communicators. This suggests that additional interventions are needed if the goal is to shift students’ perceptions of genAI’s potential to enhance their development as communicators.