Staff Product Development Engineer

The candidate will have responsibilities across the following functions:

Technical Leadership and Architecture:

Define reliability architecture standards for AI workloads across Ad Platforms.

Establish latency SLOs and reliability targets for AI-dependent services.

Design and implement circuit breaker, retry, fallback, and graceful degradation strategies.

Lead multi-provider failover strategies across Azure, OpenAI, Bedrock, and future model providers.

Define production readiness review standards for AI workloads.

Runtime and Scalability Engineering:

Lead concurrency modelling and throughput validation for model-dependent services.

Design scalable backend services that support high-volume AI traffic.

Partner with Infrastructure Engineering on capacity planning for AI workloads.

Drive resilience testing and load validation before beta and production releases.

Operational Standards and Governance:

Establish AI production reliability best practices.

Lead post-incident analysis for AI-related outages or degradations.

Identify systemic reliability risks and drive architectural improvements.

Partner with AI Core to ensure shared AI services meet enterprise production standards.

Engineering Excellence:

Contribute to critical reliability and runtime code paths.

Mentor engineers on distributed systems and resilience patterns.

Raise the bar for operational rigour and runtime reliability across the organisation.

Requirements:

8+ years of backend software engineering experience building distributed systems at scale.

Strong proficiency in Python (or similar backend language).

Deep experience designing resilient microservices architectures.

Strong understanding of retry semantics, circuit breakers, caching, and graceful degradation.

Experience defining and enforcing service-level objectives (SLOs).

Experience working with cloud environments such as AWS, Azure, or GCP.

Proven ability to influence architectural decisions across multiple teams.

Experience operating AI-powered or model-dependent services in production.

Familiarity with LLM provider constraints (rate limits, latency variability, concurrency caps).

Preferred Qualifications:

Experience designing multi-provider failover strategies.

Background in high-throughput, low-latency service architectures.

Experience participating in or leading incident response for distributed systems.

Scalable distributed backend systems and microservices operating at high throughput.

Experience with:

Designing and enforcing service-level objectives (SLOs) and reliability standards.

Circuit breakers, retry semantics, graceful degradation, and resilience engineering patterns.

Multi-provider integration strategies (e. g., Azure, OpenAI, Bedrock, external APIs).

Concurrency modelling, rate limiting, and performance profiling.

Observability systems including metrics, logging, tracing, and incident response workflows.

Cloud-native architectures in AWS, Azure, or GCP environments.

Production incident postmortems and systemic reliability improvement initiatives.