Kahuna Labs Blog

By Shanu Vashishtha, Deep Learning Engineer, Kahuna Labs

A troubleshooting platform proves itself only when it meets a customer unlike any it has seen before – Different products, different engineers, different ways of writing up a support case, different knowledgebases to read from and a different case management backend to integrate with. Kahuna Labs recently wrapped up a Proof of Value (PoV) that tested this head-on and earned an 8.6 out of 10 rating from support engineers on evaluation cases.

A Different Company, a Different Product, the Same Foundation

Customer profile

• Global leader in Cybersecurity, 70,000 customers worldwide
• Support team: 300 engineers 

Scope

• 2 product lines
• 140,000 historical cases ingested

The product, the customer base, and the support culture were all new to us. The core approach stayed the same. Kahuna AI learned from historical cases and the customer’s knowledgebase and built a Troubleshooting Map™: a structured picture of how problems actually get diagnosed and solved, step by step. The terrain was new, but the way we map it wasn’t. 

Data Quality

As in every engagement, Kahuna began by measuring the quality of the historical case data it would learn from. Two findings stood out:

• About 40% of cases for the first product were missing some troubleshooting steps, from minor to moderate gaps in the documented path to resolution.
• Nearly 60% of cases across both products closed without the customer confirming the issue was resolved.

Neither finding is unusual. Engineers are busy, steps get done on calls and never written down, and many customers simply stop replying once things work. Both gaps matter though. A model that learns from case history without checking it will learn the gaps too. Kahuna scores each case for completeness and takes the difference between an unconfirmed resolution from a confirmed one into account when building the Troubleshooting Map.

Rethinking Retrieval: Starting From the Problem

A different customer meant rethinking one important layer: how Kahuna finds the right past cases and articles for a new case. In this customer’s environment, a search built on a general summary of the case kept finding matches that sounded right but were wrong.

So we added problem-statement retrieval. Kahuna first boils each case down to a clear statement of the problem it’s actually trying to solve, then searches with that as one of several complementary queries. It runs alongside search by meaning and search by keyword, before the results are merged into one ranked set of candidates.

Two other changes came out of the same rethink:

1. Product-aware relevance: A knowledge article only counts as relevant when it covers the same product or component as the case, not just a similar-sounding symptom.
2. Ask, don’t assume. When a likely fix depends on a detail the customer hasn’t confirmed yet, such as a version, platform, or configuration, Kahuna recommends a specific question instead of guessing.

Problem-statement retrieval is now a capability any customer can turn on, which makes the platform better for everyone who comes next. Every engagement adds capabilities to the platform and a new customer starts with everything we’ve already learned.

Guardrails That Follow How the Customer Works

Every support organization has its own rules about who owns what. Some actions belong to the support engineer. Others belong to the end customer as normal use of the product. In this PoV we built the customer’s own responsibility model into Kahuna’s recommendation guardrails, so each suggested action goes to the right owner. Recommendations that would have been technically correct but given to the wrong person are now routed properly instead of discarded.

The support policies, ownership rules, and best practices become part of how Kahuna recommends. The process is not changed to fit the tool.

A First: Review Inside the Kahuna Web App

This was the first PoV where the customer’s subject-matter experts (SMEs) reviewed Kahuna’s recommendations directly in our built-in web app instead of using exported spreadsheets.

That changed how the feedback sessions worked. SMEs could see each case, the recommended next steps, and the evidence behind them in one place, and give feedback in context. Feedback cycle meetings had their focus moved from scrolling complex cases in sheet view to an efficient application built for gathering actionable feedback. 

Speed of Execution

Faster review is a big part of the story, but not all of it. Compared with our earlier engagements, this PoV covered:

• About 4.5 times as many historical cases (about 140,000 versus about 30,000)
• 2 product lines instead of 1 (8 feedback sessions with SMEs reviewing cases)
• About half the calendar time (one month versus about two)

It came from a re-architected codebase that installs cleanly inside a locked-down enterprise network, ingestion pipelines that were already proven, retrieval that can be reconfigured without new code, and evaluation tooling that measures every change against historical cases.

For someone onboarding to our platform, it means seeing results on their data in weeks, not quarters.

Evaluation

For the final evaluation, we drew 80 cases at random for each product line. Kahuna had never seen any of them, and they were taken from every stage of troubleshooting, from the first customer message to well into diagnosis. The customer’s support engineers rated each set of recommendations on a scale of 1 to 10, judged against the other tools they use today.

Product 1: average rating of 8.6 out of 10

Product 2: average rating of 7.1 out of 10, with about one-fifth of the historical case volume of the first product

The comparison wasn’t entirely like for like. For this PoV, Kahuna had access to only part of the customer’s tools. Even so, the engineers rated the recommendations highly, on cases with partial details and at any point in a case’s progress.

The gap between the two products also shows something useful. With about a fifth of the case history, Kahuna still delivered strong recommendations, and the results improve as more data comes in. Every additional case, transcript, and document adds detail to the Troubleshooting Map.

Conclusion

This PoV confirmed what we set out to show: the Troubleshooting Map carries over to a new company, new products, and a new support culture. Adapting it now takes weeks and every customer-specific improvement, like problem-statement retrieval, becomes part of the platform for the next customer.