Data Engineer - CB2

Data Engineer

Overview Of Role

Design, develop, and maintain scalable, high-performance data engineering solutions using AWS,

Databricks, Spark, and PySpark, leveraging strong technical acumen to solve complex data and

engineering challenges.

Build robust data pipelines following Medallion Architecture (Raw, Silver, and Gold layers), covering data

ingestion, transformation, processing, data modeling, validation, quality checks, and reconciliation.

Develop and optimize data solutions using AWS services including S3, Glue, Aurora/RDS, Lambda, and

Step Functions, along with Databricks, Spark, and PySpark to improve performance, scalability, reliability,

and cloud cost efficiency.

Implement reliable orchestration and operational workflows covering scheduling, dependencies, retries,

error handling, monitoring, and failure recovery, while troubleshooting complex data and production

issues and driving root-cause resolution.

Develop reusable, maintainable, and production-ready code following engineering standards and best

practices; contribute to code reviews, testing, CI/CD, deployment, automation, and continuous

improvement.

Support data migration and modernization initiatives across AWS and Databricks, including legacy

platform migrations, source-to-target mapping, data validation, reconciliation, and production readiness.

Collaborate with Product, Business, Architecture, QA, API, and Engineering teams in a cross-functional

Agile environment, contributing to technical design discussions, estimation, sprint planning, backlog

refinement, and delivery.

Provide technical guidance and mentorship to other engineers as applicable, promote reusable

frameworks and engineering standards, and contribute to resolving complex technical challenges.

Required Qualifications

5+ years of experience in Data Engineering, ETL/ELT, and developing large-scale data pipelines, with

experience providing technical guidance or leadership as applicable.

Strong hands-on experience with AWS services including S3, Glue, Aurora/RDS, Lambda, and Step

Functions, with a strong understanding of cloud-native data engineering practices.

2+ years of hands-on experience with Databricks, Spark, and PySpark, including Spark performance

optimization, partitioning, joins, caching, file formats, data skew, and efficient job design.

Strong SQL and Python/PySpark skills, with experience in data modeling, database design, schema

mapping, ETL/ELT, and source-to-target transformations.

Experience designing and optimizing data pipelines and orchestration workflows, including scheduling,

dependencies, retries, error handling, monitoring, failure recovery, and performance optimization.

Strong understanding of software engineering practices including coding standards, code reviews, testing,

Git, CI/CD, Azure DevOps, reusable frameworks, and automation.

Experience with data quality, validation, reconciliation, monitoring, troubleshooting, and root-cause

analysis.

Understanding of REST APIs, API request/response flows, and integration with APIs and third-party source

systems.

Experience creating, reviewing, and maintaining functional and technical documentation throughout the

delivery lifecycle.

Strong communication and collaboration skills, with experience working across Product, Business,

Architecture, QA, API, and Engineering teams in Agile environments.

Strong understanding of Agile practices, including sprint planning, backlog refinement, estimation,

iterative delivery, and production support.

5

Preferred Qualifications

Experience with API development and integration, including hands-on experience working with API-driven

data solutions.

Experience with Redis or other in-memory databases and caching technologies. Experience with Databricks and advanced Spark performance optimization techniques. Experience developing reusable frameworks, automation, and engineering standards. Relevant AWS certifications in Cloud, Data Engineering, or Solutions Architecture. Experience developing, debugging, and supporting data solutions within large, cross-functional

engineering teams.

Strong analytical and problem-solving skills, with the ability to manage multiple priorities in a deadlinedriven

environment.

Strong attention to detail and commitment to data quality, reliability, and engineering excellence.