Kahuna Labs Blog

recent posts

about

  • The Completeness Score™

    The Completeness Score™

    Transforming Support Tickets Into Reusable Knowledge Assets

    By Shanu Vashishtha, Deep Learning Engineer, Kahuna Labs

    The Fundamental Challenge

    In traditional support operations, engineers receive tickets without context. They must begin from uncertainty: requesting diagnostics, gathering specifications, and iteratively narrowing the problem space.

    The critical issue emerges after ticket closure. Closed tickets accumulate with highly variable documentation quality. A ticket might contain initial diagnostic questions, a reference to log collection, followed by “Scheduled a Zoom call” and “Closing as resolved”—with no resolution details, verification steps, or technical narrative.

    This represents a fundamental failure of knowledge capture. The resolution information exists but not where organizational processes require it: within the ticket itself. Incomplete documentation translates to duplicated diagnostic effort, extended MTTR, and the inability to build scalable institutional knowledge.

    The Completeness Score™: A Systematic Quality Metric

    Kahuna’s Completeness Score is a 0-5 rating scale that measures how thoroughly the troubleshooting process was documented. The central question: can another engineer, encountering a similar issue six months later, follow the documented diagnostic trail and resolve the issue based solely on the ticket information?

    The Completeness Scale

    Score 0: No Engineer Engagement – No messages from support engineer, or only automated responses. No troubleshooting information available.

    Score 1: Minimal Engagement – Basic acknowledgment or initial information requests without substantive troubleshooting progression. Confirms the issue existed but provides no pathway toward resolution.

    Score 2: Partial Investigation – Some troubleshooting effort evident, but significant gaps in documentation or resolution verification. Provides directional hints but lacks detail for reliable replication.

    Score 3: Standard Documentation – Reasonable troubleshooting with specific steps and technical detail, but missing some elements. Provides a solid starting point, though engineers may need to supplement with additional diagnostics.

    Score 4: Comprehensive Documentation – Thorough step-by-step troubleshooting with clear progression, multiple diagnostic checks, and strong resolution documentation. Provides a reliable playbook for similar issues.

    Score 5: Exemplary Documentation – Complete, professional-grade diagnostic documentation with all relevant artifacts, clear resolution process, and verified outcomes. Represents a reusable solution pattern requiring minimal adaptation.

    Critical Modifier: When evidence exists that a Zoom call, phone call, or remote session occurred but no transcript or summary was documented, the score is reduced by 1 point (floor of 0). Undocumented synchronous work is functionally equivalent to work that never occurred from an organizational knowledge perspective.

    Operational Applications

    Historical Analysis: Efficient Knowledge Retrieval

    When engineers search historical tickets, result sets frequently contain 50-500 potentially relevant tickets. Without quality metadata, each appears equally promising, requiring sequential evaluation.

    The completeness score enables immediate quality-based filtering:

    – Score 4-5: High-confidence documentation with verified resolutions

    – Score 3: Adequate documentation with some gaps

    – Score 0-2: Insufficient documentation for knowledge transfer

    This transforms historical ticket databases from undifferentiated archives into curated knowledge repositories where high-scoring tickets serve as reusable solution templates.

    Active Ticket Resolution: Pattern Matching

    For newly assigned tickets, similarity searches with completeness scoring return quality-weighted results. Engineers can rapidly assess pattern prevalence, implement validated solution methodologies, identify case-specific variations, and compress resolution timelines from days to hours.

    For eg: An engineer searching for “API 503 errors” finds 12 relevant tickets. Filtering for Score ≥ 4 yields 2 high-quality matches. The highest-scoring ticket documents complete diagnostic steps, root cause analysis, resolution implementation, and customer verification—enabling resolution in hours rather than days.

    Proactive Completeness: Real-Time Quality Improvement

    Beyond retrospective analysis, the completeness scoring system operates proactively during the active ticket’s lifecycle. The system calculates completeness scores in real-time as engineers document their work, identifying gaps and prompting support engineers to provide additional documentation before ticket closure.

    This proactive approach creates a win-win scenario: not only do organizations achieve better-documented tickets for audit and knowledge purposes, but this wealth of information significantly increases the accuracy of the Troubleshooting Map moving forward. When tickets contain comprehensive diagnostic narratives, resolution steps, and verification details, the underlying AI systems can more effectively identify patterns, map issue relationships, and generate actionable recommendations for future similar cases.

    Organizational Impact

    Systematic completeness scoring generates cascading benefits:

    1. Documentation Quality Improvement: Objective criteria transform abstract directives into measurable standards
    2. Knowledge Accumulation: High-scoring tickets function as reusable templates, creating compound returns on diagnostic effort
    3. Data-Driven Assessment: Quantitative evaluation of documentation capabilities independent of other performance dimensions
    4. Customer Experience: Reduced time-to-resolution through rapid access to validated solutions
    5. Documentation Debt Visibility: Aggregate metrics reveal systemic documentation issues for targeted improvements

    Conclusion

    The Completeness Score introduces systematic quality metadata to historical ticket data, enabling efficient knowledge retrieval, validated solution reuse, visible quality standards, and organizational learning. This transforms support operations from an experience-based model (knowledge in individual memory) to a documented model (knowledge systematically captured and retrievable).

    The cumulative impact: reduced resolution times, decreased duplicated effort, improved knowledge transfer, and systematic accumulation of institutional expertise in retrievable, actionable form.

    Implementation Note

    The completeness scoring system employs AI-based evaluation to automatically assess every ticket against objective criteria: troubleshooting steps, diagnostic artifacts, hypothesis testing, remote session documentation, knowledge base references, resolution steps, and customer confirmation. This automated approach provides scalability that manual review cannot achieve, making historical knowledge systematically accessible for future diagnostic work.

  • Transforming Support with AI: Build vs. Buy

    Transforming Support with AI: Build vs. Buy

    Build vs. Buy is a common dilemma with AI projects. There are a lot of factors at work: company culture, privacy concerns, and cost of buying and implementing technology. In this webinar, Kahuna Labs CEO, Sanjeev Gupta, dives into the build vs. buy challenge, citing examples from Kahuna Labs customers, to help support leaders navigate this decision to arrive at the best outcome for the company, the department, and ultimately, your customers.

    If you are already building in-house AI, learn how you can achieve the right build+buy collaboration.

  • Why RAG Search Is Not Enough

    Why RAG Search Is Not Enough

    By Sanjeev Gupta, CEO, Kahuna Labs

    Retrieval-augmented generation, or RAG search, is often the starting point for introducing GenAI into the support experience. Search vendors have promoted RAG as a breakthrough that leverages large language models to improve how we find information and answer questions.

    But while RAG looks good in demos, it doesn’t solve the real problems faced by enterprise support organizations. In most complex technical environments, RAG alone is not enough.

    Where RAG Works — and Where It Does Not

    RAG works well when questions are generic, use cases are limited, and information already exists neatly in documents. For example, in B2C scenarios or simple Q&A environments, RAG can retrieve relevant content, collate and rephrase it, and provide a good answer.

    However, when the required information does not exist in documents — which is often the case in technical troubleshooting — RAG relies on incomplete or outdated information to produce incorrect answers with confidence. This sends both the customer and the support engineer down the wrong diagnostic path, increasing time to resolution and frustrating everyone involved.

    Vendors often respond by blaming the lack of documentation and urging companies to hire more people to document more thoroughly — usually resulting in negative ROI for the support leader.

    The key question is simple:
     Can RAG help if you don’t have complete troubleshooting documentation for every scenario, product version, and customer configuration?

    Unfortunately, the answer is no.

    Why Case Summaries Don’t Fix the Problem

    Many support organizations try to overcome RAG’s limitations by summarizing old tickets and feeding those summaries back into RAG. That still doesn’t work well.

    Case summaries highlight the root cause and resolution for a specific customer scenario. A different case is likely to involve a different context. Searching through summaries might give you a distribution of possible root causes or resolutions — but not the immediate next step: “What diagnostic should I run next to find the actual root cause in this new case?”

    Key Limitations of RAG:

    • No awareness of versions or context. RAG search is not aware of product versions or customer context – the ‘retrieval’ treats every piece of information as equally pertinent to the current situation, which it often is not.
    • Incomplete information. RAG is unaware of the incomplete information. In the support landscape, the troubleshooting information for a case may not be documented completely within a ticket or a document – partly due to lack of diligence in documenting steps, and partly because many steps happen outside the case management system. Slack conversations in support channels, zoom calls, references to documents or other cases, escalations to Jira tickets are some examples. Unless you diligently bring them together, and string them in a chronological order, you won’t have all the steps. Even after that, some steps may still be missing – so each piece of information needs to be assessed for completeness and the gaps filled-in from complimentary sources.
    • Unequal credibility of sources. Not all information sources have equal credibility. A past case that was confirmed resolved by the customer due to a specific action has high credibility. A past case that was transient in nature, and went away on its own, or one where the customer never responded has low credibility. A document written last week has a high credibility. A document written 2 years ago, that has not been referred to by anyone in the last 10 months, has low credibility.
    • Evolving problem statements. During the journey of a case, the problem statement evolves – as you collect more diagnostics, the problem statement becomes more clear. Sometimes the problem completely changes direction. In order to find the correct next step, you need to look at the specific next steps taken in the past cases that were at the same stage of problem clarity.
    • Action ownership. The “next step” could be a next step that only a support engineer can execute, or it could be a next step that only a customer can execute. And some customers may have specific situations due to their unique security considerations. Unless you diligently separate out which steps can be performed by the support engineer, and which ones by the customer, or which ones are the questions and which ones are the actions, the recommendations will not be actionable.
    • High noise levels. Past cases are highly noisy. We have seen examples where RAG search recommended getting a coffee or escalating to Nicole as a next step, just because one of the past issues had a mention of that. This noise must be removed, if you’d like the next step recommendations to be accurate.
    • PII and confidentiality. Past cases contain PII and confidential data from your customers. Usually, you’d anonymize that before feeding it for RAG search to avoid cross-customer contamination. However, the common ways of anonymizing tend to lose the chain of thought within and across cases from the same customer. So this anonymization needs special treatment before you can consider learning from past cases.
    • No validation of correctness. Since RAG is a linear pipeline, there is no validation or evaluation of the completeness or correctness of the result, before it is presented to your customers. The generation and augmentation steps assume that the retrieval is accurate, even if it is grossly distanced from the original request.
    • No learning loop. RAG search is a read-only operation – it does not learn from the user feedback, or use that to improve over time.

    Troubleshooting complex support issues is not a language problem that can be solved by a large language model. It is not an information retrieval problem. It is a reasoning problem — one that requires constructing a logical troubleshooting map for your specific product.

    How Kahuna Labs Approaches the Problem

    What support engineers really need is a Troubleshooting Map™ — a guided sequence of diagnostic steps showing how to move from one stage of a problem to the next.

    Kahuna AI uses multiple deep models to break each case into problem snapshots — the smallest meaningful representations of an issue and its context. These snapshots are then clustered to identify similar situations, even when steps are missing or documentation is incomplete.

    From there, Kahuna AI reconstructs the most probable step-by-step path to resolution, ranking possible next actions by their credibility and completeness. Each part — identifying missing steps, clustering similar problems, assessing completeness, and ranking credibility — is powered by specialized models.

    With this approach, Kahuna AI can recommend the correct next step for over 80% of all case progressions.

    A Message to Support Leaders

    If your product is simple, stable, and well-documented, RAG may be sufficient — and it shouldn’t cost much. It’s available out-of-the-box from many vendors today.

    However, if your support organization handles complex issues across multiple versions, configurations, and customer environments, RAG won’t be enough. You need AI that reasons, not just retrieves — one that learns from every past case and guides your engineers to the next best step.

    That’s exactly what we’re building at Kahuna Labs.

    Watch the full webinar here to learn more about how Kahuna AI enables support teams to automate complex problem diagnostics and resolution.

  • Moving Support Automation Beyond the Pareto Principle

    Moving Support Automation Beyond the Pareto Principle

    Transforming Support Knowledge Management with Prescriptive AI

    By John Ragsdale, SVP Marketing, Kahuna Labs

    Historically, knowledge management programs and technology for support organizations had one goal: Automate the common, repetitive problems, so support engineers could focus more time on new and unique issues. The general assumption used the Pareto Principle: 80% of issues had occurred before, while 20% of issues were new. The role of knowledge management and automated search technology was to create and find content for those 80% of issues, and let agents run the play for the remaining 20%.

    For B2B enterprise technology, 80% of issues being known and repetitive was never realistic. And the percent of new, unique issues continues to rise, for several reasons:

    • Wide adoption of customer self-service has eliminated many common, or Level 1, issues from ever reaching support.
    • Technology grows increasingly complex, meaning more technical skills and diagnostics are required to solve a great number of issues.
    • For complex technology, customer configurations make reusing documentation for a similar issue at another customer of little value. When customer customizations, configurations, and deployment environments are highly unique, even common issues need a different approach for each customer.

    With these drivers, the number of incoming cases that can’t be solved by existing knowledge articles or documentation can be as high as 70%+ for some technology support organizations.

    This means the traditional approach of leveraging existing content and search technology becomes less valuable to improve support engineer productivity over time.

    Four Fundamental Requirements for Transformative KM

    As the complexity of enterprise technology environments increases, so does the challenge of solving new, high-variability issues. Traditional automation and knowledge systems were never designed for this level of diversity. They excel at handling repetitive tasks, but they fail when confronted with the nuanced, multi-dimensional problems that dominate modern support queues.

    If automation is to evolve beyond the Pareto Principle, new approaches must meet four fundamental requirements.

    1. Deep Context Awareness

    To effectively address complex issues, automation must understand not just what the problem is, but where and why it’s happening. Each customer’s environment has its own unique configuration, integrations, and data dependencies. A generic response or static article can’t account for these differences.

    Modern AI-driven systems must connect the dots across disparate data sources—ticket histories, telemetry, product logs, and customer metadata—to build a full contextual view of every issue. Without that, “automation” is little more than guesswork.

    2. Historical Intelligence

    Every enterprise has already solved the majority of its toughest issues—just not in a way that’s easily reusable. Years of tickets, development conversations, and internal documentation contain a goldmine of insights. The challenge lies in unlocking that tribal knowledge without requiring armies of knowledge workers to curate it manually.

    This is where learning from the past becomes critical. Systems like Kahuna AI leverage Prescriptive AI to retrace every step from historical tickets and construct a Troubleshooting Map™—a data model that captures how similar issues were diagnosed, what paths were successful, and what context mattered most.

    3. Predictive Reasoning and Confidence-Based Actions

    In complex environments, every issue follows a unique journey—but patterns still exist. The right automation platform identifies those patterns and predicts the next best action with a measurable level of confidence.

    When confidence is high, AI can safely automate next steps such as collecting diagnostics, running scripts, or even resolving the ticket autonomously. When confidence is lower, it can instead recommend the most probable next move to the support engineer, complete with a rationale, supporting evidence, and on-the-fly learning content. This confidence-based approach balances precision with control, ensuring reliability while reducing manual effort.

    4. Continuous Learning and Reinforcement

    Unlike traditional automation, which stops evolving once it’s deployed, next-generation systems must learn continuously. Reinforcement learning allows AI to compare its recommendations with the actual steps taken and outcomes achieved, refining its accuracy over time.

    This feedback loop transforms the support organization from a static process center into a dynamic learning system—one that becomes faster, smarter, and more effective with every case resolved.

    From Efficiency to Intelligence

    The ultimate goal isn’t to replace human expertise but to scale it. The next phase of automation will empower engineers to focus on innovation, problem prevention, and customer success—while AI handles the heavy lifting of pattern recognition, diagnostics, and workflow orchestration.

    By embracing systems that combine context awareness, historical intelligence, predictive reasoning, and continuous learning, enterprises can finally move beyond the limitations of the Pareto Principle—and into an era where every issue, no matter how complex, can be solved intelligently.

    Learn More

    Kahuna Labs is pioneering this transformation with Kahuna AI, an enterprise-grade platform purpose-built for complex technical support environments. Learn how it helps organizations reduce resolution times, improve engineer productivity, and elevate customer experience at http://www.kahunalabs.ai.

  • The Art of Getting Technical Support “Right Enough”

    The Art of Getting Technical Support “Right Enough”

    A Story of Precision, Recall, and the Path Forward

    By Shanu Vashishtha, Deep Learning Engineer, Kahuna Labs

    The Problem

    Imagine you’re a support engineer staring at a freshly opened ticket. A customer’s backup job has failed. The disk is showing strange behavior—high utilization, a rare error code but overall a puzzle that has missing pieces. You’ve seen hundreds of these cases, but each one turned out unique in their journey. What do you do next?

    With the advent of GenAI systems in place, everyone is aware that AI promises to help out in this situation. Depending on the maturity of the assistant in place, the system helps the Support Engineer by analyzing the case history, matching to one of the thousands of resolved tickets it has learned from, and suggests next steps: “Check the cluster logs for errors,” “Ask the customer about recent configuration changes,” “Run diagnostic command X to gather more information.”

    But here’s the million-dollar question: How do we know if the steps suggested by the AI are actually accurate?

    In this blog post, we are going to describe one of our attempts at answering this ‘precisely’ where we describe components of an AI powered Evaluation System. 

    Ground Truth in the Wild

    The start is always with a deceptively simple idea: take real support cases that have been resolved, look at what the predictions were early on, and compare it to what actually happened. The ground truth is right there in the past ticket conversations—the questions engineers asked, the actions they took, the steps they recommended to customers.

    Sounds straightforward, right?

    Well, maybe. 

    Challenge #1: Extracting Signal from Noise

    Support conversations are messy. They’re filled with:

    • “Thanks for the update!”
    • “Can we schedule a Zoom call?”
    • “I’m out of office until Monday”
    • Email signatures, legal disclaimers, and marketing footers

    But buried in that noise are the gems:

    • Support Engineer Actions: “I will review the cluster logs and apply the latest patch”
    • Customer Actions: “Please run `<command with arguments>` and share the output”
    • Probing Questions: “When did this issue first occur? Which version of the product are you on?”

    The first component of the Eval system is about becoming an archaeologist – carefully sifting through email threads to extract these three types of elements from the ground truth. Apart from removing the messy stuff, these conversations require stitching ticket-related content that is not in the ticketing system (e.g. some of the steps may have come from a Zoom call transcript or a Slack conversation with a Senior Support Engineer). 

    But the challenge doesn’t end there. Once we’ve filtered out the operational noise, we face two more critical considerations:

    Privacy First: Support tickets often contain sensitive information—customer names, email addresses, system credentials, IP addresses. Before we can use these tickets for evaluation, we need to scrub all personally identifiable information (PII). This isn’t optional; it’s foundational to building trustworthy AI systems.

    Quality Matters: Not all tickets are created equal. We need to assess:

    • Credibility: Is this ticket from last month or five years ago? Best practices change, and more recent tickets generally reflect current reality better than older ones.
    • Completeness: Does the ticket actually document the resolution steps? Or does it say “Resolved on a call with the customer” with no details? A ticket that ends with “Issue resolved, closing ticket” without explaining how doesn’t help us evaluate anything.

    These quality signals become crucial filters. We’re not just looking for any ground truth—we’re looking for credible, complete, privacy-respecting ground truth that can actually teach us something about what works.

    Challenge #2: Precision vs. Recall

    Here’s where it gets interesting. One could theoretically build a prediction system that suggests “everything”:

    > “Check the logs. Ask about their configuration. Verify their credentials. Review the network settings. Inspect the firewall rules. Check for disk space issues. Investigate memory availability. Look at CPU utilization…”

    This system would have a high recall —it would capture almost every action that engineers eventually take. But it would overwhelm users with a firehose of generic suggestions, most of which aren’t relevant to the specific case at hand.

    Or one could build an ultra-conservative AI system:

    > “Check the logs.”

    This would game the Eval framework by scoring high on precision —when it makes a suggestion, it’s probably relevant. But it would miss too many important steps, leaving engineers to figure out the rest on their own.

    The satisfaction happens in the middle—suggesting the correct next steps without overwhelming or underwhelming the Engineers who are using the prediction system.

    The second component of the eval system then is about computing these precision and recall numbers for the system predictions against the extracted ground-truths. But, before we compute the numbers, we need to know what is a ‘Match’ between predictions and ground-truths.

    Challenge #3: What Does a “Match” Really Mean?

    This is where traditionally human judgment enters the equation. Consider this example:

    • Prediction: “Please try reinstalling the Backup software’s patch on the server as you mentioned”
    • Ground Truth (from actual resolution): “Thank you. It worked after the re-installation of the patch.”

    These match. The system under evaluation identified the correct action, even if the phrasing differs.

    Now consider this:

    • Prediction: “Wait until the retention period expires to successfully unregister the bucket.”
    • Ground Truth: “We will not be able to unregister external targets if they are referenced by any data locks. So for now we will not be able to unregister the bucket until all the data locks expire.”

    Again, a match! The prediction captured the essence: you need to wait for data lock’s expiration, even though the ground truth provides more technical context.

    But here’s a non-match:

    • Prediction: “Check if you can log in to the cluster using the same credentials.”
    • Ground Truth: “Please advise of any recent changes made to your cluster configuration.”

    Both are reasonable diagnostic steps, but they’re exploring different hypotheses. One is verifying authentication; the other is checking for configuration drift.

    Challenge #4: Real World Worry

    When evaluating a real support scenario, suggesting the right probing question at the right time can be the difference between:

    • A case that resolves in 2 hours (because you asked the customer about that recent configuration change that caused everything)
    • The same case that is dragging on for 2 days (because you went down three wrong diagnostic paths first).

    But here’s the paradox: There are often multiple valid paths to resolution

    The prediction might suggest: “Check if port 443 is open.”

    The engineer might ask: “When did you last update your firewall rules?”

    Both could lead to discovering the same root cause. Are these a match from the eval framework’s perspective? Sometimes yes, sometimes no—it depends on the context. When it is not a match, we head deeper into the realm of a ‘usefulness’ evaluation of this discovery, something we will explore in a future blog post.

    For now, we evaluate three categories separately:

    1. Support Engineer Actions: What will the Support Engineer do to help?

    2. Customer Actions: What do we need to ask them to do?

    3. Probing Questions: What information do we need to gather?

    Each category has its own precision-recall tradeoff.

    Enter the LLM-Judge in the Loop

    To determine these matches in each of the categories, the eval framework comprises an LLM judgement component—a judge that understands technical context and semantic similarity. It evaluates each pairwise comparison:

    “Given the case context about backups failing and the attempts that didn’t work so far, does the predicted action of ‘reinstalling the backup software’s correct version’ match the ground truth action mentioned in the resolution?”

    The judge returns a binary verdict: match or no match. From a collection of these judgments, we build our precision and recall scores.

    The caveat here being – who judges the judges? How do we capture the essence of a match when asking an LLM to judge? A single judge may not work for every instance, product or company. We should understand the current situation of a ticket, the nature of the product, the domain the company operates in to identify an optimal set of judges.

    Challenge #5: Do the numbers tell us the story we are looking for?

    When we run our evaluation:

    “`

    Overall SRE Actions Precision: 0.742

    Overall SRE Actions Recall: 0.681

    Overall Probing Questions Precision: 0.658

    Overall Probing Questions Recall: 0.591

    “`

    These numbers tell us: Our predictions get about 2 out of 3 suggestions right. It’s missing about 1 in 3 actions that engineers eventually take.

    Is this good enough? 

    • For a busy support engineer –  having 2-3 relevant suggestions immediately available might save 20 minutes of thinking through initial diagnostic steps.
    • For a new engineer – this could be invaluable guidance on where to start.

    The Path Forward

    Balancing the Scales – We’re constantly tuning this precision recall balance. The answer isn’t to maximize both (that’s mathematically impossible in most cases). The answer is to understand the cost of each type of error:

    • False positives (low precision): Suggesting irrelevant actions wastes engineer time and attention
    • False negatives (low recall): Missing critical actions delays resolution and frustrates customers

    Different contexts might demand different balances. For critical P1 incidents? Maybe we want higher recall—suggest more possibilities, don’t miss anything. For routine cases? Higher precision might be better—just tell the most likely next step.

    Our Learnings

    This evaluation framework isn’t just about measuring model performance. It’s about understanding the nature of technical support itself.

    Every time we analyze a batch of cases, we learn something new about how to evolve the Evaluation we have in place to become a better system in the entire support process. We discover patterns in what works, what doesn’t, and why.

    Conclusion: The Quest for “Right Enough”

    As we continue to refine our evaluation methods, we’re not chasing perfection. We’re chasing usefulness. We’re asking:

    • “Do these precision recall numbers mean more meaningful suggestions for the Support Team?”
    • “Does this capture the essential next steps without drowning users in noise?”

    The precision-recall tradeoff forces us to think critically about what matters, to understand the costs of different types of errors, and to build systems that are genuinely helpful in the messy, complex reality of technical support.

    And that’s a story worth telling.