Associate Delivery Manager - Data Engineer

Job Summary

To effectively design, develop, and manage data solutions using ETL technologies such as Azure Databricks (ADB), Azure Data Factory (ADF) and SQL along with leading 3 to 5 members of developers

Duties and Responsibilities

Lead the end-to-end design, development, and delivery of scalable data engineering solutions using Azure Databricks, ADF, PySpark, SQL, and Delta Lake.

Convert business requirements into robust technical designs, architecture documents, data models, and implementation plans.

Own technical delivery of data integration, ETL, lakehouse, semantic layer, and AI/BI enablement initiatives.

Guide and mentor data engineers on coding standards, design best practices, performance optimization, and reusable framework development.

Review technical designs, code, pipelines, and deployment plans to ensure quality, scalability, maintainability, and compliance.

Drive architecture decisions for batch and near-real-time data pipelines across Bronze, Silver, and Gold layers.

Ensure data quality, reconciliation, anomaly detection, and timely resolution of production issues through effective RCA and permanent fixes.

Optimize data pipelines, Databricks jobs, SQL queries, and storage usage to improve performance and reduce cost.

Implement CI/CD practices, version control, automated deployments, and environment management across Dev, QA, and Production.

Collaborate with PMO, business stakeholders, BI teams, InfoSec, DevOps, and external partners for smooth project execution.

Establish SOPs, engineering standards, reusable components, monitoring frameworks, and documentation practices.

Track delivery progress, manage technical dependencies, prioritize work, and ensure timely closure of project milestones.

Support adoption of modern data platforms, semantic modeling, metrics layer design, and GenAI/BI capabilities.

Ensure compliance with data governance, security, access control, audit, and enterprise data management standards.

Act as the technical escalation point for critical issues, complex solutioning, and cross-team dependency resolution.

Key Decisions / Dimensions

Define semantic layer design and metric definitions

Prioritize data vs AI optimization trade-offs

Handle production issues with RCA and long-term fixes

Drive architectural decisions for lakehouse + Data integration

Major Challenges

Ensuring Data Delivery within TAT

Driving adoption of GenAI-based BI over traditional dashboards

Balancing performance, cost, and scalability

Managing dependencies across data engineering, AI, and business teams

Required Qualifications and Experience

Must Have

Azure Databricks - PySpark, SQL, Delta Lake

Strong experience in Semantic Modeling & Metrics Layer design

Hands-on with Databricks workflows

Pyspark (Pandas, PySpark, FastAPI)

Azure Data Factory (ADF) for ETL pipelines

Strong SQL and data modeling skills

Good to Have

Cosmos DB / MongoDB (NoSQL concepts)

Azure Data Explorer (KQL)

Data Stack (Mandatory for Screening)

Databricks Lakehouse - PySpark, SQL, Delta Lake

AI for BI - Databricks Genie, Genie Rooms, Instructions, Agents

ETL & Orchestration - Azure Data Factory

Programming - Pyspark

Cloud Platform - Azure (Preferred)

DevOps - CI/CD Pipelines, Git