ISDA has published a whitepaper that shows generative artificial intelligence (AI) can be used to accurately and reliably extract, interpret and digitize key legal clauses from ISDA’s credit support annexes (CSAs), showing how this technology could increase efficiency, cut costs and reduce risks in derivatives processes that have traditionally been highly manual and resource intensive.

The study – published during ISDA’s Annual General Meeting in Amsterdam – evaluates the performance of eight large language models (LLMs) on their ability to accurately extract and interpret five clauses from a selection of CSAs and digitize them into Common Domain Model (CDM) representations. The CDM is a machine-readable data model that describes financial products, trades and lifecycle events in a standard way, helping to facilitate straight-through processing.

Based on a benchmarking exercise, several LLMs achieved accuracy levels of over 90% when prompted with CSA-specific information, with the simpler clauses seeing accuracy levels of 100% in some cases.

“Our benchmarking study shows how generative AI can be used to accurately extract relevant contract information and digitize it into a standardized CDM format, reducing resource requirements and the potential for errors versus traditional contract data extraction. ISDA will conduct additional research to determine what steps are needed to further improve accuracy levels,” said Scott O’Malia, ISDA’s Chief Executive.

The analysis highlighted several key findings, including:

• Domain knowledge boosts accuracy: Providing LLMs with CSA-specific information, such as the ISDA Documentation Taxonomy and ISDA Clause Library, using prompt engineering techniques (eg, few-shot prompting) consistently enhances performance, especially for clauses that exhibit greater linguistic complexity, such as minimum transfer amounts and threshold clauses.
• Standardized phrasing is easier to extract: Clauses in CSAs that typically use standardized phrasing (for example, base and eligible currency) are easier for LLMs to extract accurately, irrespective of whether the LLMs were prompted with CSA-specific information.
• Open-source models also improve with domain knowledge: Larger proprietary LLMs typically exhibit better baseline performances, but smaller open-source LLMs also benefit from CSA-specific information, offering a viable alternative to financial institutions with stringent data privacy requirements that necessitate on-premises deployment.
• Nuanced clauses remain challenging: 100% accuracy is rarely achieved, especially for more nuanced clauses, due to inherent variations in legal language, subtle distinctions between similar clauses and complex cross-referencing within documents. Further refinements in prompting and additional CSA-specific information may be needed to address these challenges.

Separately, ISDA collaborated with Arizona State University’s Artificial Intelligence Cloud Innovation Center to develop a multi-agent framework for CSA extraction, powered by Amazon Web Services. The proof of concept assigns specific tasks to different AI agents orchestrated by a central coordinator, which improves processing efficiency and reduces management overheads.

ISDA will continue to explore other areas of potential research to determine whether the accuracy of CSA clause extraction, interpretation and digitization can be improved – for example, whether modifications to existing documentation could enhance machine readability.

The full benchmarking study is available here.