QA Automation Manager

The candidate will have responsibilities across the following functions:

QA Strategy and Leadership:

Define and drive the overall QA and Quality Engineering strategy for complex products and programs.

Establish quality goals, automation strategy, test coverage targets, release criteria, and quality metrics.

Drive shift-left testing and ensure quality is embedded throughout the software development lifecycle.

Own quality for critical projects and ensure timely delivery without compromising product quality.

Identify quality risks early and establish mitigation plans.

Drive continuous improvement in testing processes, tools, and engineering practices.

Test Automation - Playwright:

Lead the adoption and implementation of Playwright-based automation for UI, API, and end-to-end testing.

Design scalable, maintainable, and reusable automation frameworks.

Define automation architecture, coding standards, test design patterns, and best practices.

Drive migration from legacy automation frameworks such as Selenium to Playwright where applicable.

Establish automated regression suites and integrate them into CI/CD pipelines.

Improve automation reliability, execution speed, coverage, and maintainability.

Review automation code and provide technical guidance to QA engineers.

Drive automation toward meaningful coverage rather than simply increasing test-case counts.

LLM / AI Testing:

Lead QA strategy for applications and features powered by Large Language Models (LLMs).

Define testing approaches for prompt quality, response accuracy, relevance, hallucination, consistency, robustness, safety, guardrails, context handling, regression, and latency.

Develop automated testing approaches for LLM-based applications.

Work closely with AI/ML and engineering teams to establish measurable quality standards.

Evaluate AI features across different models, prompts, configurations, and datasets.

Evals / AI Evaluation:

Demonstrate strong practical knowledge of LLM Evals and AI evaluation methodologies.

Define evaluation criteria, datasets, test scenarios, and quality thresholds for AI applications.

Design and implement automated evaluation pipelines for LLM outputs.

Work with metrics such as accuracy, relevance, faithfulness, toxicity, safety, and consistency as applicable.

Establish golden datasets and benchmark test suites for continuous AI evaluation.

Analyse evaluation results and convert findings into actionable engineering improvements.

Integrate Evals into CI/CD so AI quality is continuously monitored across releases.

Establish repeatable processes for evaluating prompt, model, RAG, and application changes.

Critical Project Ownership:

Take end-to-end ownership of quality for business-critical and high-risk projects.

Lead QA planning from requirements through production release.

Build risk-based test strategies for complex projects with multiple dependencies.

Identify critical business flows and ensure comprehensive E2E validation.

Lead release readiness reviews and provide quality recommendations to leadership.

Establish war-room processes for critical releases when required.

Coordinate cross-functional teams during critical production issues.

Drive RCA, corrective actions, and preventive actions for major defects.

Ensure critical projects are delivered with predictable quality and minimal production risk.

QA Process and Engineering Excellence:

Establish standardised QA processes across teams and projects.

Define and monitor test strategy, automation coverage, defect leakage, regression effectiveness, production incidents, release quality, automation stability, and test execution efficiency.

Improve defect prevention and early defect detection.

Establish quality gates for development and release processes.

Drive continuous improvement using metrics and data.

Ensure QA processes scale effectively as products and teams grow.

Team Leadership and Mentoring:

Lead, mentor, and develop QA engineers and automation engineers.

Build a high-performing Quality Engineering organisation.

Establish technical and career development plans for team members.

Conduct technical reviews and provide feedback on automation and testing practices.

Encourage adoption of AI-assisted testing and modern QA engineering practices.

Promote engineering ownership, accountability, and quality culture.

Cross-Functional Collaboration:

Partner closely with Engineering, Product, DevOps, Data Science, AI/ML, and Business teams.

Collaborate with architects and developers to ensure testability and observability are built into the product.

Communicate quality risks and release readiness clearly to senior leadership.

Represent QA in architecture, design, planning, and release discussions.

Influence engineering teams to adopt quality-first development practices.

The core requirements for the job include the following:

Automation:

Strong hands-on experience with Playwright.

Strong understanding of browser automation and E2E testing.

Strong programming skills in JavaScript/TypeScript or another suitable language.

Experience designing automation frameworks from the ground up.

Strong API automation and integration-testing knowledge.

Experience with CI/CD tools such as Jenkins, GitHub Actions, or GitLab CI.

Strong knowledge of Git and software engineering practices.

AI / LLM Testing:

Strong understanding of LLM applications and AI-based systems.

Strong practical knowledge of LLM testing and evaluation.

Understanding of prompts, context windows, embeddings, RAG, agents, and model behaviour.

Experience designing test datasets and evaluation scenarios.

Knowledge of automated Evals and continuous AI quality monitoring.

Understanding of AI-specific failure modes such as hallucination, inconsistency, irrelevant responses, and prompt sensitivity.

Testing Quality:

Strong understanding of Functional, Integration, API, E2E, Regression, Performance, Risk-based, and Production Validation testing.

Strong understanding of SDLC, STLC, Agile, Scrum, and CI/CD practices.

Leadership Competencies:

Strong ownership and accountability.

Ability to drive projects independently.

Excellent analytical and problem-solving skills.

Ability to work effectively under tight deadlines.

Strong stakeholder management skills.

Ability to challenge decisions constructively.

Strong communication and presentation skills.

Ability to manage ambiguity and rapidly changing requirements.

Strong decision-making capability during critical releases.

Ability to balance speed, risk, quality, and business priorities.

Experience:

15+ years of experience in Software Quality Assurance / Quality Engineering, with significant experience in automation.

5+ years of experience in QA leadership/management or technical leadership.

Strong experience with modern automation frameworks, preferably Playwright.

Experience owning quality for complex, enterprise-scale applications.

Experience working with AI/ML or LLM-based applications is highly preferred.

Practical knowledge of LLM Evals / AI evaluation is strongly preferred.

Experience managing multiple teams/projects and driving critical releases.

Preferred Qualifications:

Experience migrating legacy automation frameworks to Playwright.

Experience building automation frameworks and QA platforms.

Experience with AI-assisted testing and test-case generation.

Experience with RAG, AI agents, or conversational AI testing.

Experience implementing automated Evals in CI/CD.

Experience with cloud platforms such as AWS, Azure, or GCP.

Experience with observability, quality metrics, and production monitoring.