Kahuna Labs Blog

By Chaitanya Potluri, Co-Founder, Kahuna Labs

Enterprise technical support is a full-context problem. When a customer’s environment fails, the resolution almost always depends on the intersection of their specific implementation, the patches they’ve applied, the product version they’re running, and the history of prior issues on that account. The knowledge needed to resolve a case is rarely in one place, and the relevance of any given source depends on all of these factors at once.

As enterprises have looked to AI for help with technical support, most have reached for RAG (Retrieval-Augmented Generation), the conventional approach for grounding language models in company data. The pitch seems compelling: connect an LLM to your company’s documents, let it retrieve what’s relevant, and generate accurate answers grounded in real data instead of hallucinations.

For many use cases, RAG is useful: internal doc search, policy Q&A, employee onboarding: anywhere the question is clear and the answer lives in a single, well-written document. But in enterprise technical support for complex products, that describes a small fraction of incoming cases. KB coverage is structurally thin.

The combinatorial space of product versions, customer configurations, and deployment environments is too large to document comprehensively. Products evolve faster than documentation, engineers resolve cases and move on, and the knowledge accumulates in ticket histories and tribal memory. In cases where RAG has been force-fitted into technical product support, the results have been consistently disappointing. The cases that drive escalations, consume senior engineering time, and frustrate customers, stubbornly remain out of reach.

The reason is architectural, and not a matter of tuning.

Failure Mode 1: Support Tickets Are Not Documents

RAG was designed to work with clean, authored text: knowledge base articles, product documentation, policy manuals. Support tickets are none of these things. A typical enterprise ticket is a messy, multi-turn conversation containing:

• The customer’s initial report, often incomplete or imprecise.
• The engineer’s diagnostic questions and the customer’s replies, sometimes with log files pasted inline, sometimes with screenshots that are unreadable by RAG.
• Follow-up threads, internal notes, and eventually, if the ticket was resolved, a description of what worked.
• Noise layered throughout: email signatures repeated in every reply, auto-generated headers, PII, boilerplate disclaimers, and previous email chains quoted in full.
• Sensitive customer data embedded throughout: IP addresses, credentials, account-specific configuration details, and in some industries, protected health information, all of which creates a compounding cross-customer contamination risk when past tickets feed recommendations for different accounts.

Beyond what’s in the ticket, significant troubleshooting activity happens outside the ticketing system entirely: in Zoom calls, Remote Sessions, Slack threads, linked Jira or Linear tickets, and referenced documents. RAG can only retrieve what is indexed. If the ticket is the only indexed artifact, all of the reasoning and decisions that occurred in those other channels are invisible. The system is working from an incomplete record, and the recommendations it produces reflect that incompleteness.

When a RAG system retrieves a past ticket as a relevant document, it retrieves all of this. The language model then has to make sense of a conversation that may span weeks, extract the actual resolution from an abundance of noise, and determine whether that resolution applies to the current situation, all in a single generation pass. In practice, the model often either surfaces generic advice or latches onto an irrelevant detail buried in the noise. Neither outcome helps the engineer.

Tickets also have internal structure that RAG overlooks. A ticket is a conversation between at least two actors, the customer and the support engineer, with different roles and different levels of access. RAG retrieves this as an undifferentiated block. It cannot distinguish customer-reported symptoms from engineer-recommended actions, or unanswered questions from confirmed resolution steps. And when a single ticket covers multiple distinct problems over weeks, RAG retrieves the entire conversation as one unit rather than matching at the level of individual problem states.

This structural blindness has a safety consequence. Some troubleshooting steps are destructive or irreversible: restarting a production service, clearing a cache that triggers a full rebuild, modifying firewall rules. Because RAG cannot distinguish who performed each action in a past case, it also cannot assess whether a recommended step is safe for the person who will execute it now. A step that is routine for a senior engineer with direct infrastructure access may be catastrophic if followed by a customer or a new engineer working through a remote session. RAG retrieves what was done and presents it as what should be done, with no assessment of the risk.

The real-world diagnostic patterns buried in past tickets are genuinely valuable. But extracting them requires techniques purpose-built for this kind of data, not a general-purpose retrieval pipeline treating tickets as if they were authored documents.

Failure Mode 2: Blind to Trust, Relevance, Reliability, and Coverage

Even when RAG retrieves a relevant document, it has no way to assess how much that document should be trusted.

RAG ranks by similarity. It has no mechanism to filter or weigh results by the customer’s product version, configuration, or environment. A resolution that applies to version 4.2 is retrieved with the same weight as one for version 3.1, even when those versions are incompatible. Customer context, the very thing that makes a troubleshooting step relevant or irrelevant, plays no role in retrieval.

Reliability is similarly invisible. A two-year-old ticket on a deprecated version where the customer never confirmed the fix is ranked by the same criterion as a recently validated resolution on the current release. Whether the resolution was ever confirmed, how frequently it has been reused across other cases, and how recently it was validated are factors RAG does not consider.

Coverage is the most deceptive blind spot. Tickets frequently have gaps: critical steps that happened on a remote session with no transcript, diagnostics collected but never documented, engineers who resolved an issue over a call and closed the ticket with a summary that omits key reasoning. RAG retrieves these incomplete records and presents them as if they were full resolutions. An incomplete resolution pattern is worse than no pattern at all, because it gives engineers false confidence in a partial answer.

Failure Mode 3: The Query Problem

RAG assumes the question is known. A query goes in, the system retrieves documents that match, and the model answers. This works when the problem is well-defined: “What is the default timeout for service X?” has an answer that a good query will find.

But in troubleshooting, the problem statement itself is often the thing that needs to be figured out. A customer reports intermittent network latency after applying a recent patch. That could be:

• a configuration conflict introduced by the patch,
• a known issue with that build on the customer’s OS version,
• a network topology problem unrelated to the patch, and/or
• interaction between the patch and the customer’s load balancer settings.

The initial query that a RAG system would construct from this description is almost guaranteed to retrieve documents about network latency in general rather than the specific intersection of factors at play.

Expert troubleshooters handle this by iterating: they form a hypothesis, consult a source, refine their understanding, and consult a different source. A single retrieval pass does not capture this reasoning process. By the time the RAG pipeline returns results, it has already committed to one interpretation of the problem.

Failure Mode 4: One Ranked List and Many Knowledge Types

Enterprise support artifacts are not monolithic. They are spread across fundamentally different types of sources, each with its own structure, signal-to-noise ratio, and reliability characteristics:

• Official documentation is authoritative but often lags behind the actual product. It describes how things should work, not always how they do work.
• Past resolved tickets contain real-world patterns but are noisy, incomplete, and may reflect workarounds rather than root-cause fixes.
• Compatibility matrices and release notes are precise but narrow: they answer specific factual questions but do not provide diagnostic reasoning.
• Internal runbooks and tribal knowledge capture expert judgment but are rarely written down comprehensively.

RAG pipelines collapse all of these into a single ranked list. An internal runbook written by a senior engineer sits alongside a two-year-old ticket where the customer never confirmed the fix, and both are ranked by the same criterion: semantic similarity to the query. The ranking has no way to account for whether a source type is appropriate for the kind of problem being investigated, or whether the source itself is reliable.

Even splitting retrieval into separate indexes, one per source type, does not solve the fundamental problem. Each index still returns results ranked by similarity rather than diagnostic relevance, and nothing in the pipeline arbitrates across them.

Failure Mode 5: Conflicting Information Without Arbitration

Perhaps the most damaging failure mode is what happens when retrieved documents disagree. A KB article written for v3.2 recommends one approach. A resolved ticket from v4.0 followed a different path. An internal note warns against the approach in the KB article for certain configurations.

In a single-pipeline RAG system, these sources are concatenated into the model’s context window and the Language Model is left to sort it out. Often it hedges, producing a response that tries to reconcile contradictions and ends up being too vague to act on. Occasionally it confidently picks the wrong one.

An expert engineer would recognize these conflicts and apply judgment: which source is most relevant given the customer’s specific product version and configuration? Which has the higher credibility? Which is more recent? A single-pass retrieval-and-generation pipeline has no mechanism for this kind of arbitration.

The Architectural Gap

These failure modes share a common root cause. RAG treats troubleshooting as an information retrieval problem, but troubleshooting is fundamentally a multi-source reasoning problem. The answer to a complex support case is rarely contained in any one document. It must be assembled from different knowledge types, weighted across several dimensions including relevance, reliability, and coverage, with conflicts explicitly resolved rather than glossed over.

There is also a pattern problem that cuts across all five failure modes. If 40 past cases all resolved the same class of problem with slightly different descriptions, RAG retrieves the top few by match score and misses the pattern they collectively represent. The most reliable troubleshooting signals often lives in the across many imperfect matches, not in any single well-matched result. It desperately needs a mechanism to detect these patterns or produce predictive signals from aggregate data: anticipated case complexity, likely resolution path, or resources needed.

At Kahuna, we recognize that RAG was one component but an insufficient architecture, and built our troubleshooting platform around a multi-agent approach that addresses these gaps directly.

In a companion post, The Excruciating Need for an Ensemble of Agents, we describe how that architecture works in detail. RAG is a useful idea, but an incomplete one. Every hard case that RAG cannot reach means a longer resolution cycle, another frustrated customer, and another senior engineer spending time rediscovering what the organization already knew. For the cases that matter most, incomplete is not good enough.