As AI tools grow and reshape the world, we at Zeidler Group always try to look beyond the shiny, brand-led messaging and promotional jargon from vendors and strive to understand the “real” promise of these products. That way, we can really understand how AI can improve our industry, getting through the bright lights and overzealous promises that haunt generative AI.
While continually scanning the market of legal AI tools, we have noticed an increasing trend of adding more bells and whistles to a product to inflate a tool’s reputation and benefits. Whether it is new LLMs, more legal engineers, additional industries, more board members, or new advisory groups, there seems to be an endless flow of additional “enhancements” being added to these products. While all these can help to improve products and typically create compelling buzzwords for marketing, we have observed there is something this approach severely lacks; something that only humans who have been in the industry can provide: Human Feedback based on practical lived experience.
At the end of the day, the law is the law. Regulation is regulation. This will always be the baseline for legal AI tools and what is fed to the LLMs. But that is only part of the puzzle, and no amount of legal engineers or LLMs can solve the other part
What’s not written in the law – or really written anywhere – is how law and regulations are actually applied and implemented. Call it market practice, call it tolerated practices, call it human experience, honestly call it Bob, the terminology is beside the point. It has many names, but having spent years in the investment management industry, I have witnessed, experienced, and advised how various laws and regulations are actually applied by firms and – spoiler alert – it is not remotely as straightforward as the letter of the law.
As an example, investment management firms (inclusive of their legal, compliance, and risk teams) will comply with marketing rules very differently until such time as a regulator drops in and clarifies exactly how the rules must be applied. Market practice can cover a spectrum of regulatory compliance, ranging from conservative to aggressive, all depending on risk appetite and generally all in line with the regulations.
So, what does this mean for AI legal tools? Very simply, uploading the law and training LLMs on written, static information is simply not sufficient to produce and operate an AI legal tool. Especially in the area of investment management, where risk appetite is inherent in a firm’s approach to finding a competitive advantage, an AI legal tool cannot optimize investment management firms’ day-to-day operations on blackletter alone. Actual improvements must come from actual experience in an industry, internally and from clients.
A successful and beneficial AI legal tool must be trained on human feedback and experience to truly learn how to apply the rules in line with industry. In building AI legal tools, we are stewards of the law in order to find a way to optimize our clients’ business.
This harkens back to a legacy technological issue: garbage in, garbage out. If you train a tool on inadequate data, the result will be equally inadequate. AI legal tools do not need more armies of legal engineers or even benchmark-topping new models. They also need to be trained on data from actual human experience. Data that is curated, contextual, interpretive, and often qualitative. Feedback that comes from granular, real-time application of the law from working with the nuances and peculiarities of the laws. This information and knowledge can only be obtained from a human feedback loop from actual users of the tool.
At Zeidler Group, we pride ourselves on asking every client and every potential client to give us feedback, including negative and positive (and yes, we’re happy to offer complimentary trials. Book one here). We thrive on this feedback, this raw data, because we can take in the actual application of various laws and regulations by firms to improve our tools.
We believe an AI legal product, or even a generic service, cannot exist simply through sheer LLM usage and an hoard of freshly-minted legal engineers. Real progress and results require feedback from those working in the industry, interpreted by lawyers who have lived it and used to continuously refine and train the tool. Legal AI tools must be living, breathing machines, dynamically adapting to the law as well as the market, enabling firms to use the tool as part of their own lifecycle, not just a static dataset at risk of uselessness with one no-action letter.
The output of an AI legal tool that considers the human feedback loops creates results that are leaps and bounds above what a standard, industry-agnostic tool can do. Even industry-specific tools are not sufficient if they are not powered by actual experts in the industry.
If you believe legal AI should be shaped by real-world experience, I personally welcome the opportunity to show you how we build differently. Book a complimentary trial and see how AI performs when it is trained on lived regulatory experience.
