ChatGPT helps students be more creative...maybe

At the moment, we are living through the hype cycle of generative artificial intelligence (AI) tools like ChatGPT. Claims that generative AI tools going to "change everything" or "replace workers" are louder than concerns about whether such changes or replacements are things people actually want. But, those hyperbolic claims are slowly morphing into more restrained ones, such as "AI won't take your job, someone who knows how to use AI will take your job." So, do generative AI tools like ChatGPT actually help people do valuable things, like be more creative?

Based on Urban et al.'s (2024) recent study, my answer to the question above is: "Maybe." There are definitely things to like about the study: it's a true experimental design where half the university students were asked to solve a creative problem-solving task with no access to ChatGPT (3.5 plus, for those who track such things) and the other half could use ChatGPT if they wished. The creative problem-solving task is a variation of one that has been used quite a bit, and seemed solid. The ChatGPT group, on average, outperformed the other group on three dimensions of the task, with Cohen's d values ranging from .55 to .69 (pretty decent-sized effects). Thus, ChatGPT is awesome and everyone should use it for creative problem-solving, lest they lose their job to someone who does, right?

Hold on, I have some concerns. First, participants in the ChatGPT condition were prompted like this: "[With the assistance of ChatGPT,] you are asked to create three solutions that are both as original as possible and as useful as possible to help Mattel achieve higher sales than the Lego Group" (Urban et al., 2024, p. 5), whereas the other group had the same instructions except they excluded the bracketed text. Further, both groups had access to the Internet, but neither group used it. And the ChatGPT group ended up, on average, spending more time solving the task (36 v 26 minutes). So, I worry the ChatGPT group felt compelled to use generative AI due to the instructions, but the other group were not compelled to, or even aware they could, use the Internet and other tools to help them solve the problem. And using ChatGPT led to people in that group spending more time on the task, which likely affects the quality of their creative problem-solving. A better evaluation of the value of ChatGPT would have pitted it against what most people do when trying to do creative problem-solving at their desk or job: using the Internet.

So, I'm cautiously optimistic that generative AI can help people do better work, if those people have a good sense of the affordances and weaknesses of tools like ChatGPT. But just how much better those people do their work was not settled by this study, due to the way the tasks were phrased. A decent study, overall, but with evidence that was not as compelling as I had hoped. (Note: the study included other measures of interesting phenomena like metacognition, but those findings are subject to the same concerns as the creative problem-solving task, therefore I didn't go into them.)