Principal Product Development Engineer

The Product Engineering team is part of Ad Platforms, with a mission to help advertisers reach the right audience and deliver personalized, optimized experiences tailored to customers' needs and interests. Our Audience Platform offers a unified solution for audience data exchange and enrichment, as well as high-concurrency, low-latency audience targeting services powered by advanced data technologies. This is a dynamic, cross-domain team delivering end-to-end solutions across diverse technical areas, including machine learning, big data, microservices, and data visualization.

The Principal Product Dev Engineer role will focus on providing engineering and architectural direction for Disney's Advertising Data environment. The right person should possess a deep understanding of data technology, features, functions, and services coupled with a positive, solution-oriented attitude to drive the team forward with high-quality technical deliveries and improved processes and best practices. This person will be an important leader and influencer with strong communication skills that will work closely with a cross-functional team to deliver performant solutions for our world-class Ad Sales business partners. The right candidate must be fluent in all aspects of data development (including data pipelines, datasets, observability, lineage, data quality, logging, and alerting), which are a vital key to success. The right person should also blend their engineering abilities with a strong sense of craftsmanship.

We're looking for a passionate, hands-on, technical engineering expert who wants to come here to do their very best work and make their mark and add their chapter to the long and storied history of The Walt Disney Company. Someone who holds themselves and their teammates accountable in a professional, collaborative manner. A collaborative technologist who seeks to bring the best out of themselves and those around them. Someone who can provide a fresh perspective and innovative insight to our initiatives.

Responsibilities:

Lead the team to design, develop, maintain, operate, and support resilient data pipelines and data services in Scala, Python / Spark, and Java while maintaining strict uptime SLAs.

Collaborate with the teams to extend the functionality of current Operational Data platform data services and build new integrations with APIs both internal and external to the Data organization.

Drive and provide guidance to teams on architecture, design, development, integrations, and technical work breakdown for complex applications and data flows and work to align SLAs and data delivery to the business strategies, principles, and practices.

Review both the architecture and architectural standards with an eye to propose changes to update and improve both in order to optimize and automate where possible.

Lead the team to implement quality coding practices across all development efforts related to our batch and streaming data pipelines using Scala, Python, Databricks, and other key technologies.

Able to mentor, coach, and educate a team of skilled Data Engineers and Software Engineers on the best use of data capabilities with development best practices (fault tolerance, reliability, reuse, observability, etc. ) while cultivating a focus on continuous improvement.

Work to implement shared libraries in Scala and Python that abstract complex business logic to allow consistent functionality across all data pipelines across the data organization.

Lead the resolution of critical incidents and provide leadership in proactively addressing data/pipeline issues; continuously assess technology to build more stable, scalable, and resilient data flows; promote and expand on the use of the CI/CD pipelines to improve the deployment and build process.

Lead and facilitate engineering solutioning sessions to provide technical guidance to solve any technical roadblocks for the team; contribute to development planning and implementation; help the team to understand and break down work.

Collaborate and lead the development and documentation of both internal and external standards and best practices for pipeline configurations, naming conventions, partitioning strategies, and more.

Ensure high operational efficiency and quality of the data platform datasets to ensure our solutions meet SLAs and project reliability and accuracy for all our stakeholders (Engineering, Data Science, Operations, BI, and Analytics teams).

Engage with and understand our customers, forming relationships that allow us to understand and prioritize both innovative new offerings and incremental platform improvements.

Drive standards for detailed documentation of the work and changes to support data quality and data governance requirements.

Leadership and Collaboration:

Mentor engineers, conduct design/code reviews, and enforce data engineering best practices.

Collaborate with infra, product, and governance teams to align on priorities and safe adoption of AI.

Drive delivery of high-profile AI applications by balancing execution speed with system reliability.

Requirements:

Tech stack includes Airflow, Spark, Flink, Databricks, Delta Lake, Apache Iceberg, Snowflake, Scala, and Python.

Hold a Bachelor's degree in Computer Science, Information Systems, Electrical Engineering, or another comparable field of study.

Have 10+ years of experience in software development or software engineering in a large enterprise environment using modern languages, frameworks, and data platforms/tools, with at least 5 years of data engineering experience developing large data pipelines.

Strong algorithmic problem-solving expertise.

Have demonstrable expertise with relevant data technologies such as MSK, Kinesis, S3 Flink, Spark, nifi, SingleStore, Snowflake, AWS Glue, Terraform, K8s, EKS, Lambda, Airflow, DataDog, and Grafana for observability with advanced Scala and Python programming skills.

Have a deep understanding of data architecture, data management, data governance, security, and automation tools.

Have a track record of championing quality engineering: be adamant that all code is written in a way that can be tested with unit tests.

Have hands-on production environment experience with distributed processing systems such as Spark and data pipeline orchestration systems such as Airflow for creating and maintaining data pipelines.

Have experience working with high-performing teams using Agile and Lean methodologies and frameworks.

Care about your craft and have opinions about the right way to do things with technology.

Experience with: Python, Java, Databricks, LangChain, Vector Stores (Pinecone, Weaviate, FAISS, pgvector), SQL, AWS big data tech stack (like S3 Glue, MWAA).

Required Education: Bachelor's or above in computer science or a related quantitative field, or equivalent practical experience demonstrating it.

Preferred Qualifications:

Holding a Master's degree in Computer Science, Software Engineering or a related technical discipline is highly desirable.

Previous work experience in Ad Serving or Ad Technology platforms.

Excellent leadership and communication skills.

Experience with processing large amounts of data at the petabyte level, with experience with machine learning and AI.

Strong curiosity about how Disney delivers the Magic and a desire to be a part of it.