Elsevier is trialling ChatGPT in SSRN's classification process, assigning research papers to topic areas within SSRN's multi-level taxonomy. Runs on Microsoft Azure OpenAI, with Elsevier building its own production pipeline and testing incrementally for accuracy and rule adherence.
Benefits & outcomes
Smaller classification backlog at lower cost.
Faster, more consistent topic assignment.
In early testing GPT-4 averaged 85% on classification quality against 80% for human classifiers.
Notes & quotes
“We’ve learnt that our taxonomy is too large for ChatGPT to ingest at once.”
“Prompt engineering is really tricky, and we spend a lot of time thinking about how we can simplify what we're asking AI to do.”
The taxonomy runs deep — e.g. Social Sciences > Legal > Consumer Law — which is exactly what makes this harder than it sounds.
Publisher / company: Elsevier
Date of mention: 01/11/2023
Source: https://conferences.necsws.com/wp-content/uploads/2023/11/Conf-2023-Kunal-and-Georgia.pdf