Heuristic-driven and LLM-assisted Auto-Healing Engine

The Challenge

Web test cases often fail because selectors and page structures drift as product teams ship changes. The challenge was to detect recoverable failures, find the most likely replacement target, and update the test flow without relying on expensive or opaque healing every time.

Engineering Approach

• Designed a heuristic matching layer that compared stable DOM signals such as text, attributes, structure, and element proximity.
• Core components were a sementic analyzer, an attribute scorer, a visual matcher and a spatial proximity matcher that worked together to find the most likely replacement target for a failed selector.
• Computed Levenshtein distances and cosine similarity scores to compare text, attributes, and visual features of elements.
• Implemented a lightweight LLM-based fallback that could reason about the page structure and suggest likely replacements when deterministic signals were insufficient.
• Based on weighted scores, the system could determine whether a change was high-confidence and could be auto-healed or low-confidence and needed human review.
• An assisted fallback to LLM when deterministic signals were insufficient, allowed system to scale at low cost while still being able to handle edge cases and complex page structures.

Impact

• Reduced cost as compared to other fully LLM-based solutions by using deterministic signals first and falling back to LLM only when necessary.
• Had a high impact on users by reducing the number of flaky test failures and improving confidence in automation results.