Constructing the ultimate Gen AI chat/copilot experience (Part 1)

At the heart of Clearwater’s customer-facing AI applications is the Clearwater Intelligent Console (CWIC) — a cutting-edge Gen AI platform providing a user experience (UX) for investment professionals. CWIC is powered by layers of building blocks that connect to form a comprehensive cognitive architecture, including key concepts from the Cognitive Architectures for Language Agents (CoALA) framework [1], inspired by decades of research as well as Clearwater’s own innovations. Think of it as the LEGO set for AI minus the late-night misstep onto a stray brick.

Zoom image will be displayed

In this blog post, the first of a four-part series, we’ll cover the foundational building blocks — CWIC Tools, Skills, and Specialists — that power CWIC Chat, a powerful Gen AI assistant or copilot on par with the best in use today. In part two, we will cover how to bind a Specialist to a Gen AI chat window using our web component API as an example. In parts three and four, we will move up the building block pyramid to explain how we have built the next game-changing evolution of Gen AI assistants. Spoiler alert: it’s a quantum leap from tricycle simplicity to jetpack sophistication.

But first, let’s discuss how we think about Gen AI chat value for our clients. We’ve built CWIC to support three levels of integration, in order of increasing client value:

· Knowledge Awareness
· Application Awareness
· Data Awareness

Knowledge awareness

Knowledge Awareness, dubbed “Help on Steroids,” exposes Clearwater’s knowledge bases to users via natural language queries in a chat UX. Using a combination of the best LLM’s, RAG, and fine-tuning techniques, we offer users concise, direct answers — not a list of documents for the user to further scour for answers. Users can access immediate responses to their questions, along with a list of citations for reference and validation. Sample client questions include:

· How does Clearwater calculate book yield?
· Why would book yield be negative?
· How to valuate a bond forward?
· How does treatment of options differ between GAAP and STAT?
· What are the 2024 NAIC guidance updates?

These are typical questions our clients would normally ask a customer support representative, which are now quickly addressable within our applications, thanks to CWIC’s knowledge awareness level.

Application awareness

Modern software often comes with feature-rich, complex user interfaces. For beginners, navigating through multiple menus, buttons, and options can be overwhelming — features can be used incorrectly or missed entirely. Even advanced users must invest time to acquire expert-level proficiency.

Application Awareness is a highly evolved form of context-sensitive help. Using a combination of application-specific knowledge, user context, and UI automation, the chat UX can guide users directly to where they want to go without any perquisite training. CWIC not only understands the application, but also the user, what the user is authorized to access in that application, and what their preferences are. It is a powerful way for us to take novice users or users who aren’t familiar with the application and ramp them up very quickly to become power users.

A good example of application awareness is in our CWIC Chat for the Clearwater Reporting application. It understands what reports a user has access to, where those reports are, and how to provide links to navigate to those reports. Users can also ask questions about the reports in terms of the schema and columns and what that report means. So, if there’s a particular column in a report that you’re not quite sure what it means or what it means for Clearwater, you can ask questions about that column and CWIC will give you all the information. It’s like having GPS for your software.

Example application awareness commands are:

· Show me my portfolio’s performance
· Take me to my report for accounting settings
· What is my lot inventory method?
· Where can I see my exposure by country?
· Help me file a FAS 157 report

Data awareness

In today’s >Examples of data awareness queries for Clearwater include:

· What is my available cash balance?
· Help me tie out book yield
· Summarize this report for me
· Are there any cash flow anomalies in the past year?
· Why does my BACV not match market value?

As we walk up the cognitive architecture building block stack, we will refer to these three levels of Gen AI chat integration. We are now ready to begin that journey.

CWIC tools—LLM portals to the world

LLMs are an amazing evolution of deep learning, but they have three major drawbacks today:

1. They are limited to the data they were trained on
2. They struggle with some forms of logic and reasoning
3. They cannot do anything beyond generating content (probably a good thing!)

CWIC Tools can address each of these:

1. Tools can bring in data beyond the LLM’s training data, including:
   a. New content published after the latest training data
   b. Proprietary client data a shared LLM shouldn’t be trained on
2. Tools can use traditional programming where rigor and accuracy are critical:
   a. A calculator written in Python to do math
   b. A machine learning model to make predictions
3. Tools can provide a bridge to any software or hardware system with an interface (including other LLMs). For example:
   a. Call any authorized API
   b. Run generated code in a sandbox
   c. Ask for human input or approval
   d. Speak with autonomous agents, such as a CWIC Specialist (or robot!)

Here is a visual summary of how an LLM interacts with a Tool:

Zoom image will be displayed

CWIC Tools provide the foundation for all three levels of Gen AI chat integration. For a much deeper dive into CWIC Tools, and how they are secured and semantically routed and/or chained with LLMs, please see our excellent post on MRKL Agent architecture here. That post walks through a Data Awareness example of using CWIC Tools to ask questions about report data.

CWIC skills—Curated procedures

“I don’t know who you are, I don’t know what you want. If you are looking for ransom, I can tell you I don’t have money. But what I do have are a very particular set of skills, skills I’ve acquired over a very long career.”

–Liam Neeson as Bryan Mills in “Taken” …and every AI assistant ever.

CWIC Tools, while powerful on their own, may falter as user prompts grow complex, requiring multiple tools to answer. LLMs start to lose their ability to choose and chain tools together reliably as the number of available tools increases. A CWIC Skill is an LLM prompt that packages Tools together into a series of steps to be performed, thus reducing the cost and uncertainty of choosing between a larger number of potentially unrelated tools when trying to accomplish a task. The Skills prompt can be tested and hardened to provide a perfect answer every time.

CWIC Skills are a good match for any complex task that can be broken down into steps, such as a standard operating procedure (SOP) or support runbook. They’re also more flexible than a single Tool alone, as other Tools can be used to support a main Tool, such as gather input for and converting output from the Tool. Here is an example that enhances a compliance rule extractor Tool by requesting a PDF file from the user and outputting a highlighted version of that PDF:

Zoom image will be displayed

Possible instructions for this Skill:

1. Use file_upload tool to get IPS compliance document (in PDF) from user. Skip this step if a file name is provided or known.
2. Use file_selection tool to use the uploaded or named document.
3. Use parse_document_to_markdown tool to parse the document to markdown.
4. Use compliance_rule_extractor_markdown tool to extract compliance rules from the markdown in json.
5. Use compliance_highlight_pdf tool to create a highlighted version.

NOTE: Although Tools and Skills are created, accessed, and orchestrated using natural language and LLMs, they are best developed using the same best engineering practices as traditional programming, such as single responsibility, clean interfaces, encapsulation, loose coupling, reusability, and clear naming conventions. This will be illustrated in a future post.

CWIC specialists—Ghosts in the machine

With a good arsenal of Tools and Skills spanning Knowledge/Application/Data Awareness, you will have the foundational elements you need to power a competitive Gen AI chat for your application. All that’s missing is giving your Gen AI chat some personality to provide a great conversational user experience — that’s where Specialists come in!

CWIC Specialists are autonomous agents that are a concoction of Knowledge, Tools, Skills, Identity, and Memory. When they are active, they also have a Plan they are currently following to address the application user’s prompt.

Knowledge

We’ve already discussed Tools and Skills. Knowledge is a specialized RAG Tool (or collection of Tools) that is centered on a collection of documents and other unstructured or structured sources that constitute the domain of the Specialist. Knowledge is prioritized higher than other Tools or Skills added to the Specialist and tends to be more tailored to the Specialist, like Identity.

Identity

CWIC Identity is comprised of five properties combined into an advanced role prompt to elicit LLM responses in the style of a professional with a personality:

· Name: Name of the Specialist
· Job Title: Title for their job
· Job Description: Description of their job details
· Public Bio: Information that the Specialist is comfortable sharing with others. This should be like information that would be found in a resume, such as abilities and accomplishments.
· Private Bio: Information that only the Specialist themselves know, such as their personality, default tone, insecurities, and desires.

A Specialist’s Identity — its professed ability — should align with its Tools, Knowledge, and Skills — its actual ability.

Memory

A Specialist’s ability and identity can be further enhanced with short- and long- term memories. This enables a Specialist to build a relationship with an end user and learn from past mistakes. CWIC Memories are filtered and sorted based on recency, relevancy, and importance, then injected into the Specialist’s LLM prompt after Identity.

CWIC Memories are also maintained over time. A Crystallization service compacts verbose memories into succinct ones, consolidates similar memories, and prunes old, unused, or duplicate memories. This process helps keep memories efficient as user interaction grows. Think of it as Marie Kondo for your AI’s mental attic.

Planning

A Plan is simply an action a Specialist is executing or intends to execute in the future. Each Plan often boils down to what Tool to run with what parameters. Specialists may prioritize plans into a backlog, but the most important Plan is the Current Plan, which the Specialist decides on with a Thought.

Thoughts are the atomic connective tissue of the whole cognitive architecture. They provide transparency, traceability, and insights into the mechanicians of Specialists — in a word, explainability.

The process to prioritize and act on plans is called a decision cycle. An example of a simple decision cycle is the MRKL Agent loop, mentioned earlier and covered here, where a tool is chosen by matching user input to a tool description

In a complex decision cycle, the process to execute a Plan is dynamic and iterative and can be influenced by:

· Human guidance
· Pre-authored plans
· Longer-term goals
· Semantic routing
· Skills
· Reactions
· Memories
· Reflections

Zoom image will be displayed

We will cover complex decision cycles in more detail in a future post.

Putting it all together

Once you’ve built a Specialist for your application, with Tools, Knowledge, and Skills for Knowledge/Application/Data Awareness, what’s left to do? Not so much! Only a bit of glue to specify the Specialist, dynamically set application context, and handle actions in your Gen AI chat — then you’re off to the races. In Part two of this series, we will walk through how we do that in our platform — stay tuned!

As we continue to develop and refine our Gen AI systems, we’re not just creating building blocks — we’re crafting digital minds that can think, learn, and interact in increasingly sophisticated ways. The CoALA framework provides a valuable roadmap for creating these complex cognitive architectures [1], and we’re excited to be at the forefront of this Gen AI revolution.

References

[1] Cognitive Architectures for Language Agents. arXiv:2309.02427v3

---

About the author

Jeff Sklaroff is the Software Development Director of Gen AI at Clearwater Analytics with decades of experience building, inspiring, and coordinating talented teams to solve big problems and deliver innovative software solutions to provide deep value to customers.