Â鶹´å

Assistant Teaching Professor
Heinz College of Information Systems and Public Policy
Fall 2024

94-819 Data Analytics with Tableau (7 week course)

Research Question(s): 

1. To what extent does the early introduction and scaffolded use of generative AI tools for learning Tableau impact students’ performance on course deliverables?
2. How do student self-efficacy in data literacy skills change over time in a course in which a generative AI tool was introduced early?

Teaching Intervention with Generative AI (genAI):

Gongora-Svartzman introduced students to a genAI tool (Explain Data) designed to assist in data exploration. Gongora-Svartzman demonstrated how this Tableau genAI tool can provide an efficient way to view the landscape of potential data analysis pathways in a given project. The teaching intervention provided students with a scaffolded introduction to Explain Data early in the course. In the control course, students were briefly exposed to the tool, without scaffolding, during the last two weeks of the course.

Gongora-Svartzman taught three sections of the course, one control section in Spring 2024 and two treatment sections in Fall 2024. She briefly introduced Explain Data late in the course in the Spring 2024 section (control), whereas she introduced it earlier in the course and in more scaffolded form in the Fall 2024 sections (treatment). She compared data sources from student deliverables across sections. 

Sample size: Treatment (55 students); Control (25 students) 

Data Sources:

1. Student deliverables (in-class exercises, final group projects, and case study challenges) from course assignments that required students to perform data analysis. 
2. Pre-post surveys of students’ self-efficacy regarding their data literacy (treatment sections only).

1. RQ1: Student performance in the course during the Fall 2024 (treatment) semester did not differ significantly from student performance during the Spring 2024 (control) semester on any course deliverables.
2. RQ2: Treatment students’ (Fall 2024) self-efficacy for data analysis skills and use of genAI tools for data analysis significantly improved from pre to post, marking an increase of nearly 50% from baseline.
   
   
   

   Figure 1. In the Fall 2024 (treatment), students’ self-efficacy for data literacy significantly improved from the beginning (M = 59.82, SD = 21.98) to the end (M = 88.40, SD = 9.07) of the semester, t(41) -8.17, p < .001, g = -1.24. Error bars are 95% confidence intervals for the means.

Eberly Center’s Takeaways:

1. RQ1. Student performance did not change when they were introduced in a more scaffolded fashion to the genAI tool Explain Data earlier in the semester compared to a semester in which students received a more cursory introduction to the tool later in the semester. However, students in the Spring 2024 (control) already evidenced very high performance on all deliverables, limiting the ability to detect improvements.
2. RQ2. Students’ self-efficacy for course-related skills (including the use of genAI tools) did significantly improve from the beginning to the end of the course in Fall 2024 (treatment). These data were not collected during the Spring 2024 (control) section, however, so we cannot say to what extent these pre/post increases are attributable to the intervention.