What's the proper role for AI in systematic review research?

Today, I was struck by this article in Nature on academic publishers developing AI chatbots that, according to a spokesperson from Elsevier, are “intended as a light, playful tool to help researchers quickly get summaries of research topics they’re unfamiliar with…In response to a natural-language question, the bot uses a version of the LLM GPT-3.5 to return a fluent summary paragraph about a research topic, together with cited references, and further questions to explore.” Um, no.

As Emily Bender and others have articulated so well, AI doesn’t know anything and it doesn’t actually synthesize anything. It takes input (e.g., the good stuff but also all the garbage on the Internet), develops a probability model, and then based upon what you ask it, it spits out a series of words most likely to follow from your query, given that probability model. Even for an AI model trained on nothing but great data (e.g., a whole bunch of peer-reviewed journal articles), it’s not at all clear AI will ever be able to accurately synthesize that literature or develop insights from it.

So, I worry about publishers who make a “light, playful tool to help researchers quickly get summaries of research topics they’re unfamiliar with.” That tool will almost certainly be wildly unreliable and inaccurate, but also tempting to those who don’t know enough to dismiss it out of hand. Better to not have the tool at all than to have one that may seduce people into using it and thinking what it produces is valuable.

Now, I’m not anti-AI. I’m on the “AI isn’t going to take my job, but someone who knows how to use AI could take my job” bandwagon. AI can be very helpful. For example, this tool helps researchers vet the often-enormous corpus of articles returned when doing a search of the literature for a systematic review. The key here is that the tool is open source, transparent in how it works, and, critically, involves a “researcher-in-the-loop.” The researcher guides how the AI classifies potential articles as included or excluded from the systematic review corpus. Thus, this tool helps people with the mechanics so those people can get to the only-humans-can-do-this parts of systematic reviewing, i.e., being systematic and…reviewing.

So, I’m all for AI tools that help people think. But AI tools that think for people? No thank you - because AI can’t think.