Elsevier is trialling the integration of ChatGPT into SSRN’s classification process to automate and improve the assignment of research papers to topic areas. The goal is to reduce manual labour, improve speed and consistency, and decrease costs. ChatGPT is trained to work with SSRN’s detailed taxonomy and is being incrementally tested for accuracy and rule adherence.
Benefits & outcomes
- Reduced classification backlog and cost
- Faster and more consistent topic classification
- Initial results show ChatGPT outperforms humans on average classification scores (GPT-4: 85%, Human: 80%)
- Clear potential for scalable deployment with lower overhead than human classifiers
Notes & quotes
“We think that ChatGPT will be able to at least match the classification quality of the average human classification.”
“We’ve learnt that our taxonomy is too large for ChatGPT to ingest at once.”
“Prompt engineering is really tricky, and we spend a lot of time thinking about how we can simplify what we’re asking AI to do.”
Elsevier is using Microsoft Azure OpenAI’s version of ChatGPT and building its own production pipeline.
SSRN taxonomy is complex, including multi-level classifications across disciplines (e.g., Social Sciences > Legal > Consumer Law).
Publisher / company: Elsevier
Date of case study: 01/11/2023
Source: https://conferences.necsws.com/wp-content/uploads/2023/11/Conf-2023-Kunal-and-Georgia.pdf