Regression Harness Engineering for InterSystems IRIS

Summary

This is the core infrastructure-backed automation thread of my formal role - the practical bridge between QD users who need reliable product-quality workflows and TAE-provided capabilities for VM provisioning, InterSystems IRIS deployment, environment configuration, and job execution.

The harness runs a huge list of regression tests on new InterSystems IRIS builds across active branches. Depending on branch activity, one to three sets of these large regression runs could be active on any given day.

Context

The tests are owned by different teams within Quality Development. Stakeholders include those QD teams, their corresponding teams in Systems Development, and InterSystems IRIS Data Platforms Product Management.

A primitive nightly regression system already existed when I started my role, but the operational burden was high. False positives and infrastructure-related errors made it difficult to distinguish product failures from harness, environment, resource, or governance issues, adding friction to the build qualification process.

Problem

The old system could run tests, but it lacked enough control, customizability, and post-run automation to operate well at scale. It had ineffective resource management, insufficient platform targeting, limited system-level environment configurability, coarse exclusion and inclusion governance, and no meaningful post-test-run triage automation.

Jira tickets and failure reports had to be created manually. Test governance and failure reporting rules were not standardized enough across stakeholders, which meant triage consumed a large share of active engineering time.

My role

• Own the regression testing harness end-to-end after initially joining the work as a junior engineer.
• Translate QD testing and environment requirements into automation workflows built on TAE infrastructure capabilities.
• Build custom harness-level logic for resource querying, job queueing, platform targeting, and reliability management.
• Work with stakeholders across QD, Systems Development, Product Management, and TAE to understand and standardize test governance and failure reporting rules.
• Build the pipeline that automated major parts of post-run triage, including failure reporting, recurrence detection, recurrence-based test-exclusion, and ticket-resolution-aware test re-inclusion.
• Train other individual contributors to work on the system while remaining the technical lead and primary owner.

Operating model

My operating model is to make internal test infrastructure usable for QD workflows: understand how engineers needed to run and govern tests, map those needs to TAE capabilities, build the missing automation layer, expose practical configuration controls, and keep iterating based on real nightly regression behavior.

When a QD testing requirement is not directly supported by TAE, I work with TAE to either extend platform capabilities or design a workaround that satisfies QD's testing needs without overstraining shared infrastructure.

System/workflow pattern

A typical workflow begins with a QD testing need, becomes a set of platform, environment, branch, and release requirements, then maps to TAE-provided capabilities for VM provisioning, IRIS deployment, environment configuration, and job execution.

On top of those capabilities, I built the harness-level logic that decided which resources were valid for a job, when jobs should be submitted, how to avoid overloading shared infrastructure, and how to minimize harness-level failures.

Custom resource management layer

The resource-querying layer resolves each regression run's configured platform target against the platforms currently offered by TAE and supported by the IRIS version under test. Targets could be narrow, such as a specific minor OS version, or broad, such as multiple OS families.

For broad targets, the harness selects from high-availability supported platform versions so individual unit test jobs could spread across valid resources instead of concentrating on scarce or unavailable targets.

The queueing layer controls when and how jobs are submitted into shared infrastructure. The goal is to avoid direct bulk submission patterns that could turn resource contention, platform mismatch, or infrastructure availability problems into false-positive harness failures.

Capabilities I built into the regression harness

• Large-scale regression automation covering 10,000+ tests per run, with one to three sets of runs possible nightly across active branches.
• Targeting and configurability across Unix, Windows, containerized environments, and release/version dimensions.
• Platform-level granularity such as targeting a specific supported minor OS version instead of only targeting a broad OS family.
• Programmatic determination of supported platform versions so supported targets could evolve without hard-coding every case.
• Randomized target selection within supported families so broad coverage requests could spread individual unit test jobs across valid high-availability targets.
• System-level environment configurability for networking, installed packages, and test-specific requirements.
• Failure lifecycle automation for Jira creation, recurrence detection, exclusion/re-inclusion workflows, bug-association checks, and status monitoring.
• Managed resource querying and job queueing that reduced regression harness job error rate by approximately 80%.

Reliability and ownership

Post-implementation, I am responsible for maintenance and operational reliability. This requires monitoring failure modes, tuning queueing behavior, clarifying user expectations, documenting behavior, and adjusting the workflow as product, platform, and infrastructure constraints change.

A normal run could produce 5-30 failures requiring triage, recurrence analysis, exclusion governance, or bug association. The harness needs to reduce environmental noise and harness-level failures so that reported failures are more likely to represent legitimate test failures requiring triage.

Impact

• Reduced manual regression triage from 70%+ of active time to under 10% by automating major parts of post-run failure reporting and governance workflows.
• Reduced regression harness job error rate by approximately 80% through custom resource querying and job queueing built on top of TAE infrastructure.
• Extended supported platform targeting across OS families, OS versions, containers, and release/version dimensions.
• Added more granular platform targeting, including specific supported major and minor OS versions when required.
• Added randomized platform selection within supported target families to improve broad coverage.
• Improved failure lifecycle automation around test failure reporting, recurrence detection, exclusions, re-inclusions, bug-association checks, and status monitoring.

Related automation work

Beyond the main regression harness, I periodically work with QD engineers to turn manual tests into automated workflows running on TAE's platform. Those efforts combine the test owner's domain expertise, my TAE platform fluency, and the available capabilities for VM provisioning, IRIS deployment, and environment configuration.

Some of those workflows require capabilities TAE does not directly support. In those cases, as with the regression harness, I work with TAE to extend the platform or develop a workable path that meets QD's test requirements while respecting infrastructure constraints.

What I have learned

This work taught me how to turn complex platform capabilities into usable workflows: understand what engineers are trying to accomplish, expose the right controls, map those controls to the underlying infrastructure's real supported options, and iterate from real operational usage.

I learnt how to sit between users and a platform team: understand user intent and platform capability, bridge the gap, and deliver solutions that work for both sides.

What this shows

The project shows end-to-end ownership, technical thought partnership with users and platform teams, pragmatic tradeoffs, cross-functional coordination, and scaling an internal platform workflow from basic execution toward reliable operational adoption.