Â鶹´å

Assistant Professor
Tepper School of Business
Spring 2024

70-415 Introduction to Entrepreneurship (14-week course)

Research Question(s): 

To what extent does brainstorming with the assistance of generative AI impact: 

1. the number of ideas generated?
2. the quality of ideas generated?
3. students’ self-efficacy regarding generative AI use and course learning objectives?

Teaching Intervention with Generative AI (genAI):

Motley implemented scaffolded brainstorming sessions during class to support ideation for entrepreneurship projects (by individuals). Students then leveraged genAI tools (Copilot) to support both the generation and evaluation of ideas for new business ventures. Individual students created “pitch decks” (slides) to present their ideas to their peers to recruit collaborators to design a business implementation plan. Teams of students then collaboratively designed implementation plans for the entrepreneurship projects chosen.

Study Design:

All students in two concurrent course sections received training on brainstorming techniques. Motley randomly assigned two conditions to sections: students used (treatment) or did not use (control) genAI tools in brainstorming exercises during class. The treatment group received training on brainstorming techniques and genAI use focused on prompt engineering. Control groups received training on brainstorming techniques alone. Data sources were compared between course sections, statistically controlling for variation in students between conditions. 

Sample size: Treatment section (56 students); Control section (43 students)

Data Sources:

1. Artifacts of brainstorming sessions, including google docs (control and treatment) and transcripts from genAI use (treatment) 
2. Students’ pitch decks (slides from student presentations), scored using a rubric with criteria for uniqueness of the problem being solved, the solution, and the customer segment targeted 
3. Pre/post surveys of students’ self-efficacy regarding skills using genAI tools and course learning objectives 

Findings:

1. RQ1: Students did not differ in the number of ideas they generated with or without the help of genAI across two individual brainstorming sessions. However, students who brainstormed without genAI experienced a decline in idea production over time, whereas students who used genAI did not.
   
   
   

   Figure 1. Although the average number of ideas generated did not differ across conditions, F (1, 83) = .59, p = .45, η² = .007, students across conditions experienced a decline in number of ideas generated over time, F(1, 83) = 8.80, p = .004, η² = .10. However, a closer investigation of the significant time x condition interaction, F(1, 83) = 5.04, p = .03, η² = .06 suggests that this decline was only true for the non-genAI condition, F(1, 83) = 11.54, p < .001, η²  = .12, whereas students who used genAI did not experience a decline in number of ideas generated over time, F(1, 83) = .32, p = .58, η² = .001. 

2. RQ2: The quality of entrepreneurial pitches submitted by students did not differ in uniqueness, feasibility, or compellingness across conditions using and not using genAI (see Figure 2). The subset of students who critically evaluated (“filtered”) ideas early on that genAI produced (i.e., they did not automatically submit all ideas suggested by genAI) pitched marginally higher quality ideas to their peers (see Figure 3). 
   
   
   

   Figure 2. Students’ entrepreneurial pitch deck scores (uniqueness, feasibility, and compellingness total, out of 6 pts.) did not differ when students used genAI (M = 4.30, SD = .81) or not (M = 4.19, SD = .96) for idea generation. Error bars are 95% confidence intervals for the means. An independent-samples t-test showed that the mean difference was not significant, t(97) = -.66, p = .51.
   
   
   

   Figure 3. In the condition that used generative AI, entrepreneurial pitch deck scores (uniqueness, feasibility, and compellingness total, out of 6 pts.) were marginally higher when students critically evaluated and filtered genAI-generated ideas during the first brainstorming phase (M = 4.40, SD = .77) than when they retained every genAI-produced idea (M = 3.75, SD = .89), t (8.82) = -1.94, p = .08, g = -.82. Error bars are 95% confidence intervals for the means. 

3. RQ3: Across a single class session (i.e., two individual brainstorming sessions), students’ confidence in formulating an idea increased significantly, regardless of genAI use. Engaging in brainstorming with genAI significantly increased students’ self-efficacy for using genAI when compared to the condition that did not use the tool.

Eberly Center’s Takeaways:

1. RQ1: Even though students self-reported that genAI helped them generate more ideas, students who used genAI did not differ in the number of ideas they submitted compared to students who did not use genAI. If anything, students who used genAI submitted slightly (though not statistically significantly) fewer ideas than students who did not use genAI. This finding is in line with existing work that suggests integrating genAI into a brainstorming process does not necessarily offer a safeguard against the kinds of productivity losses experienced in human brainstorming groups. However, use of genAI in the present study enabled students to maintain a level of productivity while students who did not use genAI experienced a pattern of exhausting their ability to generate new ideas. This suggests that genAI may be particularly helpful at later stages of the idea-generating process when human capabilities have been maximized.
2. RQ2: 
   1. There was also no evidence that genAI conferred an advantage when it came to the quality of students’ chosen ideas, as measured by final pitch deck scores. Together with RQ1a, these findings echo other research that suggests students overestimate the benefits of genAI for academic performance .
   2. On a cautionary note, students who retained all the ideas genAI produced without critically evaluating, or “filtering” them, had a tendency to perform worse on pitch deck scores than students who filtered genAI-produced ideas early on. This was true of students who retained all genAI ideas without adding any of their own ideas, and of students who kept genAI ideas and added on to them. In other words, students who did not filter also did not come up with good ideas on their own. This is consistent with emerging research suggesting that academic performance is reduced when students overly rely on genAI and fail to invest sufficient cognitive effort into evaluating genAI output .
   3. Motley is collecting a second semester of data in Spring 2025 to further explore the impact of critical evaluation of genAI ideas.
3. RQ3: Regardless of genAI use, students reported an increase in their confidence in generating a startup idea after a single class session whereas only students who used genAI in their brainstorming showed an increase in confidence to use genAI to produce desired results. However, confidence was assessed right after the brainstorming activities and it is unclear if these differences persist over time.