ISDA has published a whitepaper that shows generative artificial intelligence (AI) can be used to accurately and reliably extract, interpret and digitize key legal clauses from ISDA’s credit support annexes (CSAs), showing how this technology could increase efficiency, cut costs and reduce risks in derivatives processes that have traditionally been highly manual and resource intensive.
The study – published during ISDA’s Annual General Meeting in Amsterdam – evaluates the performance of eight large language models (LLMs) on their ability to accurately extract and interpret five clauses from a selection of CSAs and digitize them into Common Domain Model (CDM) representations. The CDM is a machine-readable data model that describes financial products, trades and lifecycle events in a standard way, helping to facilitate straight-through processing.
Based on a benchmarking exercise, several LLMs achieved accuracy levels of over 90% when prompted with CSA-specific information, with the simpler clauses seeing accuracy levels of 100% in some cases.
“Our benchmarking study shows how generative AI can be used to accurately extract relevant contract information and digitize it into a standardized CDM format, reducing resource requirements and the potential for errors versus traditional contract data extraction. ISDA will conduct additional research to determine what steps are needed to further improve accuracy levels,” said Scott O’Malia, ISDA’s Chief Executive.
The analysis highlighted several key findings, including:
- Domain knowledge boosts accuracy: Providing LLMs with CSA-specific information, such as the ISDA Documentation Taxonomy and ISDA Clause Library, using prompt engineering techniques (eg, few-shot prompting) consistently enhances performance, especially for clauses that exhibit greater linguistic complexity, such as minimum transfer amounts and threshold clauses.
- Standardized phrasing is easier to extract: Clauses in CSAs that typically use standardized phrasing (for example, base and eligible currency) are easier for LLMs to extract accurately, irrespective of whether the LLMs were prompted with CSA-specific information.
- Open-source models also improve with domain knowledge: Larger proprietary LLMs typically exhibit better baseline performances, but smaller open-source LLMs also benefit from CSA-specific information, offering a viable alternative to financial institutions with stringent data privacy requirements that necessitate on-premises deployment.
- Nuanced clauses remain challenging: 100% accuracy is rarely achieved, especially for more nuanced clauses, due to inherent variations in legal language, subtle distinctions between similar clauses and complex cross-referencing within documents. Further refinements in prompting and additional CSA-specific information may be needed to address these challenges.
Separately, ISDA collaborated with Arizona State University’s Artificial Intelligence Cloud Innovation Center to develop a multi-agent framework for CSA extraction, powered by Amazon Web Services. The proof of concept assigns specific tasks to different AI agents orchestrated by a central coordinator, which improves processing efficiency and reduces management overheads.
ISDA will continue to explore other areas of potential research to determine whether the accuracy of CSA clause extraction, interpretation and digitization can be improved – for example, whether modifications to existing documentation could enhance machine readability.
The full benchmarking study is available here.
For Press Queries, Please Contact:
Nick Sawyer, ISDA London, +44 20 3808 9740, nsawyer@isda.org
Joel Clark, ISDA London, +44 20 3808 9760, jclark@isda.org
Christopher Faimali, ISDA London, +44 20 3808 9736, CFaimali@isda.org
Nikki Lu, ISDA Hong Kong, +852 2200 5901, nlu@isda.org
Documents (1) for ISDA Publishes Paper Exploring Use of Generative AI to Extract and Digitize CSA Clauses
Latest
A Path to Greater CFTC-SEC Alignment
Earlier this week, the Commodity Futures Trading Commission (CFTC) and the Securities and Exchange Commission (SEC) held a roundtable on regulatory harmonization – an initiative we wholeheartedly support. The US regulatory framework has evolved over time to facilitate financial markets...
Updated OTC Derivatives Compliance Calendar
ISDA has updated its global calendar of compliance deadlines and regulatory dates for the over-the-counter (OTC) derivatives space.
Working Towards Tokenized Collateral
One of the lessons learned from recent market shocks – including the 2020 dash for cash and the UK gilt market crisis in 2022 – is that when volatility strikes and market participants must suddenly generate large amounts of cash...
IQ Interview Yazeed Alnafjan
ISDA published new legal opinions in June that recognise the enforceability of close-out netting in Saudi Arabia under regulations published by the Saudi Central Bank earlier this year. In this interview with IQ, Yazeed Alnafjan, the central bank’s deputy governor...