Â鶹´å

Associate Teaching Professor
Civil and Environmental Engineering
College of Engineering
Spring 2024

12-333 Experimental and Sensing Systems Design and Computation for Infrastructure Systems (14-week course)

1. To what extent does utilization of AI tools impact students’ skills for data processing, cleaning, and visualization of large data sets?
2. What are the attitudes, perceptions, and experiences of students regarding AI-powered tools for data processing and visualization?

Ozis introduced genAI (PerplexityAI) as a possible support tool during students’ multi-week, big-data group project. Students had the option to use genAI during two of their data cleaning and visualization tasks, one completed individually and one in a team. They were not restricted in how they could choose to use the tool but were given some possible uses, such as a coach to provide advice or a tool to detect outliers in the dataset or to provide code to create data visualization plots in Python.

Students could choose to opt into using genAI during their data project, creating a self-selected group of genAI users (treatment) to compare to a group of non-genAI users (control) within the course. Ozis also compared students’ work to a previous iteration of the course in which students were not permitted to use genAI.

Sample size: Self-Selected Treatment (19 students); Self-Selected Control (15 students); Previous-Semester Control (18 students); 12 teams across the three conditions 

Data Sources:

1. Students final course grades as well as students’ data visualizations, cleaned datasets, and documentation of process, scored with a rubric for ability to clean, analyze, and visualize large data sets. 
2. Rubric grade for quality of data analysis (following removal of treatment condition and randomization of both semesters’ team deliverable).
3. Students’ reflections on how effective, challenging, and rewarding their data cleaning process was and whether or not they used genAI in their process (treatment semester only).

1. RQ1: Students who chose to use genAI for their data tasks did not perform differently (as measured by final course grades) than students who never chose to use genAI to work with their data nor students who didn’t have the option to use genAI (Figure 1). Additionally, rubric scores for the quality of teams’ data analysis did not differ across conditions.
   
   

   Figure 1. Students’ grades did not differ statistically, whether they self-selected to use genAI (M = 94.0, SD = 7.3), self-selected to never use genAI (M = 91.3, SD = 9.4), or were required to not use genAI (M = 90.6, SD = 3.2) for their data cleaning and analysis (F(2,49) = 1.23, p = .30). Error bars are 95% confidence intervals for the means. 

2. RQ2: When given the option to use genAI for data tasks, 44% of students chose never to use genAI. Their reasons revealed critical thinking about the added value of the tool (e.g., it can be inaccurate) as well as confidence in their own data skills. Students who chose to work with genAI primarily used it for guidance alone (i.e., opted to clean their datasets without genAI). 
   

Eberly Center’s Takeaways: 

1. RQ1: There was no evidence that using genAI (Spring 2024 self-selected treatment) improved or harmed students’ grades in a course that requires cleaning, visualizing, and analyzing data compared to students who never used genAI (Spring 2024 self-selected control and Spring 2023 control). This null result could be due to alternative factors including high course grades across all students, small sample size, weak manipulation strength, self-selection for whether or not to use genAI (Spring 2024), and minimal instruction on ways to leverage genAI for data tasks.
   1. Teams’ final analysis deliverable was evaluated by a coder who was unaware of the semester and condition. Rubric scores for the quality of their presented analysis did not significantly differ across conditions. Due to the team nature of this final task, the sample size for analysis was extremely small making it difficult to interpret the results as meaningful. 
2. RQ2: Student perspectives about the value of genAI and their confidence in their own data analysis skills could play a role in whether or not a student opts to use genAI when permitted. There is an opportunity for more scaffolding to teach the students the affordances and limitations of this tool to better inform their decisions.