Selector diagnostics: giving the agent evidence from a failed test#
Before reaching for an interactive browser connection, I wanted the agent to have useful evidence from the test itself. The debugging helpers printed controls, labels, roles, and page context into the output. The agent could compare that evidence with the failed selector, investigate the relevant helper, and rerun the test after a change.
This page explains the original logAllBoxes, the enhanced diagnostics, and the failure hook that connected them to a test run. Output and integration examples labeled illustrative show how the pieces work together.
What the original logAllBoxes collected#
The complete orchestration function is below. It delegates discovery to smaller functions, scans roles, then checks labels, placeholders, and common action text.
export async function logAllBoxes(page: Page) {
await logAllTextboxes(page);
// await page.pause();
const roles = [
'alert', 'alertdialog', 'application', 'article', 'banner', 'blockquote', 'button', 'caption',
'cell', 'checkbox', 'code', 'columnheader', 'combobox', 'complementary', 'contentinfo', 'definition',
'deletion', 'dialog', 'directory', 'document', 'emphasis', 'feed', 'figure', 'form', 'generic', 'grid',
'gridcell', 'group', 'heading', 'img', 'insertion', 'link', 'list', 'listbox', 'listitem', 'log', 'main',
'marquee', 'math', 'menu', 'menubar', 'menuitem', 'menuitemcheckbox', 'menuitemradio', 'navigation',
'none', 'note', 'option', 'presentation', 'progressbar', 'radio', 'radiogroup', 'region', 'row', 'rowgroup',
'rowheader', 'scrollbar', 'search', 'searchbox', 'separator', 'slider', 'spinbutton', 'status', 'strong',
'subscript', 'superscript', 'switch', 'tab', 'table', 'tablist', 'tabpanel', 'term', 'textbox', 'time',
'timer', 'tooltip', 'tree', 'treegrid', 'treeitem'
];
for (const role of roles) {
await logAllByRole(page, role);
}
await logAllByLabel(page);
// await page.pause();
await logAllByPlaceholder(page);
// await page.pause();
// Optional: Search for some common texts (expand as needed)
const commonTexts = ['Submit', 'Cancel', 'OK', 'Save', 'Delete'];
for (const text of commonTexts) {
await logAllByText(page, text);
}
}
One of those smaller functions shows how a discovered textbox becomes a line in the output:
export async function logAllTextboxes(page: Page) {
const textboxes = await page.locator('[role="textbox"], input, textarea').elementHandles();
for (const textbox of textboxes) {
const role = await textbox.evaluate(node => (node as HTMLElement).getAttribute('role') || (node as HTMLElement).tagName);
const name = await textbox.evaluate(node => {
const el = node as HTMLElement & { labels?: NodeListOf<HTMLLabelElement> };
if (el.getAttribute('aria-label')) return el.getAttribute('aria-label');
if (el.getAttribute('aria-labelledby')) {
const ids = el.getAttribute('aria-labelledby')!.split(' ');
return ids.map(id => document.getElementById(id)?.innerText).join(' ');
}
if (el.labels && el.labels.length) return Array.from(el.labels).map(l => l.innerText).join(' ');
// For input/textarea, use .name and .id if available
return (el as HTMLInputElement).name || el.id || '';
});
console.log(`Found textbox: role=${role}, accessible name="${name}"`);
}
}
The label lookup uses attributes, associated labels, and fallbacks such as the input’s name or ID. It is a practical diagnostic approximation of a useful name, not a complete accessibility-name computation.
To invoke the original scan directly during an investigation, keep the failing page open:
import { logAllBoxes } from '../../HelperFiles/playwright-debug';
await logAllBoxes(page);
Illustrative output in the original helper’s format:
Found textbox: role=INPUT, accessible name="Email Subject"
[getByRole] role="button", index=0, accessible name="Save Template"
[getByLabel] label="Email Subject", for="email-subject", control="<input id="email-subject">"
[getByPlaceholder] <INPUT> placeholder="Search templates"
If the failing locator expected a button named exactly Save, this output supplies a candidate explanation: the available control is labeled Save Template. The next step is to confirm its role in the workflow. A similar label by itself is not enough to justify selecting a different control.
What the enhanced version added#
logAllBoxesEnhanced prioritizes interactive roles before structural content. For each role it inspects up to ten matching elements, adds visibility and disabled state, includes nearby text and parent context, and prints candidate selectors. It then performs additional discovery for textboxes and other page information. logActionableElements offers a smaller scan of buttons, links, menu items, and tabs, up to five elements per role.
This substantial source excerpt is the inner processRoleWithMatching routine. It closes over page and a getElementContext function defined earlier in logAllBoxesEnhanced; it is not a standalone replacement for the enhanced module.
// Process roles with exact and fuzzy matching
async function processRoleWithMatching(role: string, isPriority: boolean = true) {
const roleElements = page.getByRole(role as any);
const count = await roleElements.count();
if (count === 0) return;
console.log(`\n--- ROLE: ${role.toUpperCase()} (${count} elements) ---`);
for (let i = 0; i < Math.min(count, 10); i++) { // Limit output
const element = roleElements.nth(i);
try {
// Get basic attributes
const name = await element.getAttribute('aria-label') ||
await element.getAttribute('aria-labelledby') ||
await element.textContent() || '';
const id = await element.getAttribute('id') || '';
const className = await element.getAttribute('class') || '';
const isDisabled = await element.isDisabled().catch(() => false);
const isVisible = await element.isVisible().catch(() => false);
// Get context information
const context = await getElementContext(element);
console.log(`\n[${i}] ${role}:`);
console.log(` Name: "${name.trim().substring(0, 50)}"`);
console.log(` State: ${isVisible ? 'visible' : 'hidden'}, ${isDisabled ? 'disabled' : 'enabled'}`);
if (id) console.log(` ID: "${id}"`);
if (className) console.log(` Class: "${className.split(' ')[0]}..."`);
// Context information
if (context.parent) console.log(` Parent: ${context.parent}`);
if (context.nearbyText) console.log(` Near: "${context.nearbyText}"`);
if (context.siblings > 0) console.log(` Siblings: ${context.siblings}`);
// Suggest selectors
console.log(` Selectors:`);
// 1. Exact match selectors
if (name.trim()) {
console.log(` // Exact match:`);
console.log(` page.getByRole('${role}', { name: '${name.trim()}', exact: true })`);
}
// 2. Fuzzy match selectors
if (name.trim() && name.trim().length > 5) {
const fuzzyName = name.trim().substring(0, 20);
console.log(` // Fuzzy match:`);
console.log(` page.getByRole('${role}', { name: /${fuzzyName}/i })`);
}
// 3. Context-aware selectors
if (context.parent) {
console.log(` // With parent context:`);
console.log(` page.locator('${context.parent}').getByRole('${role}', { name: '${name.trim()}' })`);
}
// 4. Filter-based selectors
if (context.nearbyText) {
console.log(` // With nearby text filter:`);
console.log(` page.getByRole('${role}').filter({ hasText: '${context.nearbyText}' })`);
}
// 5. Index-based selector (last resort)
console.log(` // By index (least preferred):`);
console.log(` page.getByRole('${role}').nth(${i})`);
// Special handling for specific roles
if (role === 'button' && name.includes('icon')) {
console.log(` // Icon button alternative:`);
console.log(` page.locator('button').filter({ has: page.locator('.icon, mat-icon') })`);
}
if (role === 'textbox') {
const placeholder = await element.getAttribute('placeholder');
if (placeholder) {
console.log(` // By placeholder:`);
console.log(` page.getByPlaceholder('${placeholder}')`);
}
}
if (role === 'combobox') {
const expanded = await element.getAttribute('aria-expanded');
console.log(` // Combobox state: ${expanded === 'true' ? 'expanded' : 'collapsed'}`);
}
} catch (error) {
console.log(`[${i}] Error processing ${role}: ${error}`);
}
}
}
The generated selectors are suggestions to inspect. For example, the enhanced name fallback uses the raw aria-labelledby attribute rather than resolving its referenced text. The generated strings also need checking before being copied into code. A visible, enabled element with the expected context is stronger evidence than a name alone.
The original module re-exports the enhanced helpers, so the call site can stay short:
import {
logAllBoxesEnhanced,
logActionableElements
} from '../../HelperFiles/playwright-debug';
// Use a broader scan when role, state, and surrounding context matter.
await logAllBoxesEnhanced(page);
// Or start with the smaller actionable-elements scan.
await logActionableElements(page);
Illustrative excerpt of enhanced output:
--- ROLE: BUTTON (1 elements) ---
[0] button:
Name: "Save Template"
State: visible, enabled
ID: "save-template"
Parent: DIV.template-actions
Near: "Save Template Cancel"
Siblings: 1
Selectors:
// Exact match:
page.getByRole('button', { name: 'Save Template', exact: true })
// Fuzzy match:
page.getByRole('button', { name: /Save Template/i })
// With parent context:
page.locator('DIV.template-actions').getByRole('button', { name: 'Save Template' })
This gives an agent a way to compare the failed selector with the available controls before opening an interactive investigation. It can also reveal a disabled button or the wrong screen, in which case renaming a selector would not fix the cause.
Connect the output to the failure#
The historical debugAfterEach hook captures the test title, file, status, duration, error, diagnostics, and a failure screenshot. Here is the complete exported handler, including its deprecation notice:
/**
* @deprecated Use Playwright's trace and screenshot on failure.
*/
// Export the afterEach handler function
export const debugAfterEach = async ({ page }, testInfo) => {
// Check if the test failed
if (testInfo.status === 'failed' || testInfo.status === 'timedOut') {
console.log('\n=== TEST FAILED - RUNNING logAllBoxes ===');
console.log(`Test: ${testInfo.title}`);
console.log(`File: ${testInfo.file}`);
console.log(`Status: ${testInfo.status}`);
console.log(`Duration: ${testInfo.duration}ms`);
if (testInfo.error) {
console.log(`Error: ${testInfo.error.message}`);
console.log(`Stack: ${testInfo.error.stack}`);
}
console.log('-------------------------------------------\n');
try {
// Wait a bit to ensure the page is in a stable state
await page.waitForTimeout(500);
// Run logAllBoxes to dump all page elements
// Use enhanced version if DEBUG_ENHANCED env var is set
if (process.env.DEBUG_ENHANCED === 'true') {
console.log('Using enhanced role-based debugging...');
await logAllBoxesEnhanced(page);
} else {
await logAllBoxes(page);
}
// Additionally run quick actionable scan if requested
if (process.env.DEBUG_ACTIONABLE === 'true') {
console.log('\nRunning quick actionable elements scan...');
await logActionableElements(page);
}
// Also take a screenshot for additional debugging
const screenshotPath = `test-failures/${testInfo.title.replace(/[^a-z0-9]/gi, '_')}_${Date.now()}.png`;
await page.screenshot({
path: screenshotPath,
fullPage: true
});
console.log(`\nScreenshot saved to: ${screenshotPath}`);
} catch (error) {
console.error('\nError running logAllBoxes:', error);
console.error('The page might have been closed or navigated away.');
}
console.log('\n=== END logAllBoxes OUTPUT ===\n');
}
};
DEBUG_ENHANCED=true chooses the enhanced scan. DEBUG_ACTIONABLE=true adds the smaller actionable scan. The hook catches diagnostic errors so it can report a closed page or another collection failure.
The hook is marked deprecated in favor of traces and failure screenshots. Verbose logging also caused test hangs, so the cost of broad dumps motivated more targeted collection.
A complete example using the correct page#
The Proof of Address suite creates its own shared page in beforeAll. Registering test.afterEach(debugAfterEach) unchanged would give the hook Playwright’s separate page fixture. The following illustrative integration passes the actual workflow page explicitly and closes the context after the suite.
import { test, type BrowserContext, type Page } from '@playwright/test';
import { columnNames } from '../../HelperFiles/cit-app-config.js';
import { setupTestPage } from '../../HelperFiles/components/settings-helpers.js';
import { clickViewActionInTable } from '../../HelperFiles/components/common.js';
import { verifyFieldsPresent } from '../../HelperFiles/components/verification_checks.js';
import { debugAfterEach } from '../../HelperFiles/debug-hooks';
test.describe.serial('Proof of Address diagnostic example', () => {
let context: BrowserContext;
let page: Page;
test.beforeAll(async ({ browser }) => {
({ context, page } = await setupTestPage(browser, {
menuItem: 'Account',
subMenuItem: 'Proof of Address'
}));
});
test.afterEach(async ({}, testInfo) => {
if (page && !page.isClosed()) {
await debugAfterEach({ page }, testInfo);
}
});
test.afterAll(async () => {
await context?.close();
});
test('Open a record and inspect its detail fields', async () => {
await clickViewActionInTable(page);
await verifyFieldsPresent(page, columnNames.ProofOfAddress.DetailFields);
});
});
Diagnostics are collected before cleanup, from the page that performed the failing action. The test body continues to describe the workflow using existing helpers.
Logs first, with later Playwright MCP use#
During the 2025 work I generally limited MCP use because of my security concerns at the time, and started selector investigations with repository and test output. I eventually also used Playwright MCP for interactive browser investigation.
An MCP-assisted investigation can use browser_snapshot in place of logAllBoxes, then rerun the programmatic test. Playwright MCP is Microsoft’s Playwright project, with documented Claude Code support. Official Playwright MCP repository.
For the project’s progression, see the timeline. Continue to docs as code, or return to Functions explained.