Image: slack.engineering · rights & removal

Agentic Testing: Where Agents Fit in the E2E Testing Stack

Reporting by Slack EngineeringRead the original at slack.engineering

Executive Summary

Agent-driven end-to-end testing involves exploring how agentic workflows fit into existing testing stacks, contrasting them with traditional deterministic tests. Experiments involved over 200 agentic E2E workflows using Playwright MCP and CLI across two defined flows: Thread Reply and Search Discovery. The results show differences in reliability, speed, and cost based on the execution model and input format. The Agent + Playwright MCP approach demonstrated the highest reliability for more complex flows, while Generated Playwright Tests were the fastest but showed lower reliability on complex tasks. Cost analysis indicated that execution environment (MCP vs. CLI) significantly impacted runtime and failure rates, with the structure of context accumulation—how agent interaction is bundled—being a primary driver of cost rather than model choice alone.

Facts Only

* Agent-driven E2E tests involved over 200 agentic workflows using Playwright MCP and CLI.
* Tests validate goals rather than fixed journeys: goal $\rightarrow$ agent adapts $\rightarrow$ verify result.
* Three execution models were tested: Agent + Playwright MCP, Agent + Playwright CLI, and Generated Playwright Tests.
* The Playwright MCP configuration showed a failure rate of 0% for thread reply and $\sim 12\%$ for search discovery.
* Generated Playwright Tests had an $\sim 8\%$ failure rate for thread reply but $\sim 48\%$ for search discovery.
* Average runtimes ranged from 3 minutes (Generated Tests) to 11 minutes (CLI).
* The cost analysis showed that execution model mattered more than the underlying LLM model, with MCP approaches generally costing less than CLI and Code Generation approaches for equivalent flows.
* Failures in CLI-based runs were often caused by authentication or navigation instability, not agent reasoning errors.
* Playwright MCP execution allowed easier parallelization compared to CLI-based runs.

Full Take

The findings suggest a fundamental tension between deterministic validation and exploratory exploration in UI testing. The core pattern observed is that increased workflow complexity exacerbates reliability gaps across all methods, indicating that the state management mechanism implemented by the execution layer is a critical variable. Agentic workflows succeed not by enforcing fixed journeys, but by modeling the necessary agent-state transitions, which requires flexibility agents possess. The split between MCP and CLI highlights how execution philosophy—real-time state retention versus step-by-step command execution—directly impacts stability and cost efficiency. Furthermore, the token usage analysis reveals that the overhead is less about complex reasoning and more about the compounding cost of context retransmission during sequential interaction turns, suggesting future optimization efforts must focus on efficient state aggregation rather than solely optimizing model inference. This implies that agentic testing should be positioned as an advanced layer for debugging and exploration layered atop reliable deterministic regression suites, where deterministic tests establish the boundaries and agentic tests explore behaviors beyond them.

From the original · Slack Engineering

Abstract Agent-driven end-to-end (E2E) tests add a new exploratory layer to testing, but should they replace traditional deterministic tests? We ran more than 200 agentic E2E workflows using the Playwright MCP, Playwright CLI, and agent-generated Playwright tests in test workspaces using non-production data to find out how agentic testing could fit into both our and your testing stacks.

Read the full story at slack.engineering

Sentinel — Human

Confidence

The text presents a detailed analysis of agent-driven testing workflows, grounding its claims in specific experimental metrics regarding reliability, speed, and cost across different execution models.

Signals Detected

Natural variation in sentence structure and occasional digressions.

Logically structured argument building toward a synthesized conclusion based on empirical results.

Use of specific, cited experimental metrics and clear structuring of comparative data; the narrative flows from problem to method to results naturally.

The technical depth regarding token cost breakdown and execution model differences suggests domain expertise present in a real research context.

Human Indicators

Use of specific, complex terminology (e.g., Playwright MCP, Action Signatures, testing pyramid) that reflects deep familiarity with software engineering and agentic workflows.

The presentation of nuanced results where reliability gaps are tied to execution methodology (MCP vs. CLI) rather than just model performance is characteristic of empirical research reporting.

Acknowledgement of practical limitations, cost analysis derived from token usage, and explicit focus on *how* different systems interact (state handling, context accumulation).