Evidence-Grounded AI Failure Investigation for InterSystems IRIS Regression Workflows

Summary

Test failure investigation had been a fully manual, laborious, and unstandardized process. A subject-matter expert would inspect unit test logs, often rerun the test, analyze application and system logs, consult product documentation and class reference material, read the test code, review historical results, and reason about product changes that landed between the last pass and current failure.

I recognized that this workflow was a strong fit for an evidence-grounded LLM system: the hard part was not asking a model a generic question, but engineering the right context, retrieval, evidence boundaries, structured output, and human review flow so SMEs could validate an investigation instead of rebuilding the whole picture manually.

Context

Large regression runs can cover 10,000+ tests and commonly produce 5-30 failures per run. Each failure can represent a product issue, test issue, infrastructure issue, or interaction between those categories.

The SMEs assigned to failures do not always write every test they own, and even when they do, they cannot keep every test, superclass, product behavior, platform detail, and historical result in working memory. Investigation required assembling scattered evidence into a coherent technical explanation.

Problem

The pre-AI workflow consumed active SME time and delayed build readiness decisions. Engineers had to manually cross-reference logs, documentation, test code, class reference material, test result history, and potentially relevant product changes before deciding why a test failed and what remediation might be appropriate.

The process was especially costly because the analysis was synchronous: the SME had to perform the investigation directly. My goal was to make the investigation asynchronous, standardized, evidence-grounded, and ready for SME validation by the time the failure ticket reached the reviewer.

Constraints

• No public source code, raw logs, internal ticket IDs, internal repo paths, or proprietary implementation details.
• The AI system could investigate and enrich failure tickets, but it could not modify product code or test code.
• Relevant test code was supplied through a deterministic context pipeline rather than allowing the model to browse source repositories.
• Product documentation and class reference material were accessed through a controlled vectorized index, not open internet access.
• Human SMEs retained ownership of remediation decisions and source changes.

My role

• Identified the failure-investigation workflow as a high-leverage AI opportunity for Quality Development.
• Designed, prototyped, piloted, deployed, and now maintain the investigation system.
• Built pipelines to preserve application-level and system-level logs for failed unit test jobs.
• Built deterministic context collection for test code, including referenced superclass context needed to understand the test behavior.
• Created a vector index over relevant documentation and class reference material with product and version filtering.
• Integrated the investigation output into the existing regression harness ticket-reporting workflow.

System design

The system builds an investigation package for each failed unit test job. That package can include preprocessed unit test logs, preserved application-level logs, preserved system-level logs, relevant test code context, retrieved documentation and class reference material, historical test result signals, and potentially relevant product changes since the last passing run on the same platform.

The AI component is called from the regression testing harness that already reports failures into Jira. Instead of replacing the ticketing flow, it enriches the reported failure ticket with a structured investigation report that the assigned SME can validate.

AI workflow

• Collect and preprocess failure artifacts from the failed unit test job.
• Preserve and extract relevant application-level and system-level log context instead of dumping entire log files into the prompt.
• Gather deterministic test code context, including relevant superclass context.
• Retrieve product documentation and class reference material using a version-aware retrieval loop.
• Check historical test results to identify the last passing run on the same platform.
• Analyze potentially relevant product changes between the last pass and current failure.
• Produce a structured investigation report for SME review and ticket enrichment.

Retrieval and evidence grounding

Retrieval was treated as a core reliability feature. I created a vector index for documentation and class reference material, with rows tagged by product and product version so retrieval could account for differences across IRIS and IRIS for Health versions.

The system used an agentic RAG loop: it generated a query from the observed failure, retrieved candidate documents, critiqued whether the retrieved material was relevant, kept relevant results, avoided already-seen material, and refined the query when additional context was needed.

The report cited retrieved documentation and class reference material with links. It also cited relevant unit test, application-level, and system-level log entries, and surfaced product changes that appeared relevant to the observed failure. When product changes existed but none could be confidently tied to the failure, the system listed them separately for SME follow-up.

Structured output

The system produced a comprehensive investigation report for each failed unit test job. A single job could include one or more test failures, so the output grouped failures by signature when multiple observed failures appeared to share one or more root causes.

The structured output included failure type classification, failure signatures, correlations between application-level and system-level events, focused application-level analysis, focused system-level analysis, supporting evidence, potentially relevant product changes, and remediation steps for the SME to review.

Evaluation and validation

The workflow was evaluated against an eval set of historical analyses performed by SMEs. That made the evaluation closer to the real task: not simply whether the model could produce plausible prose, but whether it could assemble evidence and produce a useful first-pass investigation aligned with expert analysis.

Provider-hosted frontier models were eventually used to improve output quality, structured output adherence, and parallelism. The system needed to run as part of the regression harness, where a run could produce 5-30 failures and one to three runs could occur per day, making local-only inference impractical for latency and throughput.

Human review

The AI system was allowed to investigate and enrich the Jira ticket associated with a reported test failure. It was not allowed to update product code or test code. Any action on source code remained human-owned.

That boundary was deliberate. The useful deployment unit was not an autonomous fixer; it was an evidence-grounded investigation workflow that gives SMEs the full picture faster while preserving reviewer judgment for remediation.

Impact

The system moved test result analysis from several synchronous minutes-to-hours of manual SME investigation to a few asynchronous minutes at most. Instead of manually gathering context across logs, code, docs, history, and product changes, the SME can validate a standardized investigation report.

In practice, failures that might previously stay open for days because of analysis complexity or shifting priorities can now be closed within minutes when the AI-produced report gives the SME enough evidence, analysis, and remediation direction to validate the result.

Tradeoffs

The design favors traceability, context control, and reviewer trust over maximum autonomy. The system does not browse the codebase or internet freely; it receives curated context from deterministic pipelines and controlled retrieval sources.

This choice reduces flexibility but improves auditability. In a failure-investigation workflow, a bounded assistant that cites evidence and hands off to an SME is more useful than a broader assistant that can produce unsupported certainty.

Limitations

A small minority of tests may require artifacts that the current context package does not yet capture. Extending artifact support is the path to covering those cases more completely.

The current system investigates from preserved test artifacts. It does not enter a live preserved test environment and inspect it interactively. A future extension could apply the same context-engineering approach to live or reproduced test environments for on-demand investigation.

The system also stops short of implementing remediation itself. A future version could generate a proposed fix, rerun the test in a reproduced environment, and open a change review for the SME when the rerun succeeds.

What I learned

The deployment challenge was less about proving that an LLM could summarize a log and more about building the surrounding system: artifact preservation, context assembly, version-aware retrieval, structured outputs, evals, ticket integration, and human review.

The work reinforced that useful AI systems need the same reliability instincts as infrastructure automation. The model is one component; the workflow around it determines whether engineers trust and adopt the result.

What I would improve next

• Add support for the remaining test artifact types needed by unusual failure cases.
• Extend the approach to live or reproduced test environments where the system can investigate on demand.
• Explore human-reviewed remediation generation, test reruns, and change-review creation after successful validation in a reproduced environment.