Don’t Believe the Hype – AI Won’t Magically Solve Your Data Issues (Yet)

AI Won't Solve Your Law Firm's Data Issues

Whenever I open LinkedIn, I see headlines touting how AI will magically transform the legal industry’s data issues; however, my experience refining our company’s (CI’s) data solutions dictates otherwise. At CI, we test numerous AI tools upon their release with the goal of enhancing our data normalization process. Specifically, to improve the process of matching people, companies, and matters across different systems—our company’s bread and butter.

Based upon hundreds of hours of research, we concluded that we cannot rely upon AI technology alone for entity normalization.

AI’s Problem: Inaccurate, Incomplete, & Erroneous Results

We gleaned two major issues from AI tools’ results: inaccurate and incomplete data. For instance, we leveraged a well-known AI Agent to assess all mergers and acquisitions related to the law firm, Bryan Cave Leighton Paisner LLP. The results overlooked most the firm’s past acquisitions and only correctly identified one prior merger, resulting in a 16% match success rate. Furthermore, the AI Agent mischaracterized several instances of firm dissolutions. In cases where a firm dissolved and a large group of lawyers joined another firm, the Agent often incorrectly classified such cases as acquisitions. A researcher who relied exclusively on AI would have overlooked the misclassifications and remain blinded to key firm acquisitions. While leveraging AI was a useful starting point, the results proved unreliable and incomplete. This is why CI leverages specialized human researchers at each stage of the data normalization process to build a comprehensive, reliable hierarchy of data, such as a hierarchy for law firm organization changes.

In addition to inaccurate and incomplete data, we also found that AI often provided erroneous results. To enhance our CI Matter Connector solution, we tested several AI solutions to match internal matter IDs to external dockets’ case titles, jurisdictions, and docket numbers. Facially, the AI Engine outperformed purely logic-based matching algorithms. However, the error rate was unacceptably high, resulting in up to 30% false positives. Consequently, we used manual human testing and review to overlay an approach to minimize the number of false positives. We successfully developed a hybrid system that leverages the best AI tools combined with a set of over 110 custom logic-based rules to dramatically increase our match success rate with a level of accuracy that AI alone could not achieve.

CI’s Solution: A Reliable Hybrid Data Approach

To ensure CI’s solutions provide the highest level of comprehensive and accurate results, we determined that relying upon a mixture of AI technology, logic-based rules, and human review for data normalization is essential. While it is not as fast as relying on AI alone, it is significantly more reliable. And within the legal industry, accuracy is paramount. In this sense, using domain expertise as part of a logic-based matching algorithm produces significantly better results.

Given the rapid evolution of AI tools and agents, we continue to stay apprised of the latest technologies. However, we are convinced that human review and domain-based logical rules will continue to remain essential for accurate entity resolution for now.