TestMu AI is addressing the critical visibility gap in agentic testing by consolidating fragmented execution data into a single, portable format. The company, formerly known as LambdaTest, has introduced evidence packs for its Kane CLI, a terminal-first tool designed for natural language browser and mobile app testing. This update aims to solve the "black box" problem inherent in AI-driven testing, where engineers often struggle to reconstruct the specific actions and environmental states that led to a particular test outcome or failure.
Consolidating Agentic Execution Data via Evidence Packs
The introduction of evidence packs allows every Kane CLI run to generate a single, sealed .evidence file. This file serves as a comprehensive audit trail, capturing the test definition, a result summary, and per-step screenshots that include annotations of the elements the AI agent interacted with. To provide deeper technical context, the packs also bundle browser console logs, network logs, and CLI runner logs. For mobile application testing, the file includes specific metadata such as the device model and OS version.
When a test fails, the evidence pack provides a per-step record containing the error message and the exact page state at the moment of failure. This structure is intended to allow developers to move from a failed run to a root cause analysis in minutes. The company is positioning this as a way to make AI agent behavior transparent and easily shareable across engineering teams.
Operationalizing the .evidence Format in CI/CD
TestMu AI is implementing these packs through a new set of Kane CLI commands designed to manage the lifecycle of testing data. The evidence serve command allows users to view sealed packs via a hosted viewer from a localhost-only server, while evidence validate checks the integrity of a pack against a specific profile. Additionally, the evidence merge command enables teams to combine multiple packs into a single file.
The integration extends to broader infrastructure; runs that upload to Test Manager or are executed via replayed runs automatically attach these sealed packs. For remote executions, the pack is sealed on the grid and downloaded upon job completion. This automation is designed to ensure that even interrupted runs do not leave manual cleanup requirements for the user, as Kane CLI clears incomplete packs during the next startup.
Key Takeaways
- TestMu AI has launched evidence packs for Kane CLI, which bundle test definitions, screenshots, logs, and failure records into a single .evidence file.
- The new feature includes specific commands—evidence serve, evidence validate, and evidence merge—to manage and verify these portable files.
- Evidence packs are produced by default on every Kane CLI run and are automatically attached to runs uploaded to Test Manager.
TechInsyte's Take
In our view, TestMu AI is correctly identifying that the primary barrier to widespread enterprise adoption of agentic AI in QA is not the ability to run tests, but the ability to trust and audit them. By moving away from fragmented dashboards and toward a portable, single-file standard, the company is treating AI test results as verifiable artifacts rather than ephemeral logs. This shift toward "evidence-based" testing suggests a move toward more rigorous compliance and debugging standards in automated environments. If the .evidence format gains traction, it could set a new benchmark for how transparency is handled in the growing field of autonomous software quality engineering.
Questions & Answers
How does the new evidence pack format improve the debugging workflow for engineering teams?
The format consolidates disparate data points—including annotated screenshots, network logs, and console logs—into one .evidence file. This allows developers to see exactly what the AI agent saw and did at each step, reducing the time required to identify the cause of a failure.
What specific technical metadata is captured during mobile application testing runs?
For mobile runs, the evidence packs include critical environmental context, specifically the device model and the OS version, ensuring that the execution environment is fully documented alongside the test results.
Can these evidence packs be integrated into existing continuous integration (CI) workflows?
Yes. Because the packs are single, portable files that can be merged and validated, they are designed to be easily shared with teammates or kept as a permanent record within CI pipelines.
How does TestMu AI handle incomplete or interrupted test runs to prevent data clutter?
The Kane CLI is designed to automatically clear incomplete packs during its next startup, ensuring that interrupted executions do not require manual cleanup by the engineering team.
Source: TestMu AI
