From human-in-the-loop to LLM-in-the-loop for high quality legal dataset

Abstract

Annotating legal documents with rhetorical structures is difficult and time-consuming, especially if done completely manually. This paper explores two methodologies for optimal results: first, a human-in-the-loop approach based on a multi-step annotation process with domain experts reviewing and revising datasets iteratively. To enhance interpretability, eXplainable Artificial Intelligence (XAI) models are incorporated, aiding in understanding decision-making processes. Second, an LLM-in-the-loop method has humans leveraging generative large language models (LLMs) to assist experts by automating repetitive annotation tasks under supervision. Further research is proposed to develop interaction models that effectively balance automation with human guidance and accountability.