Sr. Data Engineer

The Data Platforms Team, a segment under the Disney Entertainment and ESPN Product and Technology (DEEPT) organization, is looking for a Senior Data Engineer to join our Sports and Preferences Data Team. Product Performance Data is critical to making product-level decisions for the DEET Portfolio whether it's related to product design, measuring advertising effectiveness, helping users discover new content or building new businesses in emerging markets. This data is deeply valuable and gives us insights into how we can continue improving our service for our users, advertisers, and our content partners. Our Sports and Preferences data Engineering team is seeking a highly motivated Sr. Data Engineer with a strong technical background who is passionate about designing and building systems to process data at scale to provide business value, create models and features that unlock powerful insights, and work across software and data disciplines to solve difficult, unique problems at scale. Our tech stack includes AWS, Databricks, Airflow, Spark, and Databricks, and languages include Scala and Python.

Responsibilities:

Contribute to the architecture, design, and growth of our Data Products and Data pipelines in Scala and Python/Spark while maintaining uptime SLAs.

Develop scalable solutions, building ETL pipelines in Big Data environments (cloud, on-prem, hybrid).

Implement the Lakehouse architecture, working with key partners to shift towards a Lakehouse-centric data platform.

Our tech stack includes AWS, Snowflake, Spark, Databricks, Delta Lake, and Airflow, and languages include Python and Scala.

Collaborate with Data Product Managers, Data Architects and Data Engineers to design, implement, and deliver successful data solutions.

Maintain detailed documentation of your work and changes to support data quality and data governance.

Ensure high operational efficiency and quality of your solutions to meet SLAs and support commitment to our customers (Data Science, Data Analytics teams)

Be an active participant and advocate of agile/scrum practice to ensure health and process improvements for your team.

Be a problem solver; when presented with new challenges, you are expected to research and network to find solutions.

Seek out answers to business problems and look for opportunities to automate processes and optimize Cost.

Engage with and understand our customers, forming relationships that allow us to understand and prioritize both innovative new offerings and incremental platform improvements.

Requirements:

Minimum and Preferred. Inclusive of Licenses/Certs (include functional experience as well as behavioral attributes and/or leadership capabilities).

At least 5 years of data engineering experience developing large data pipelines.

Strong hands-on Experience with Cloud technologies like AWS (S3 EMR, EC2) and building data pipelines.

Strong algorithmic problem-solving expertise.

Hands-on experience with Databricks for building pipelines and scheduling the same.

Strong SQL skills and ability to create queries to extract data and build performant datasets.

Hands-on experience with distributed systems such as PySpark to query and process data.

Strong fundamental Scala and Python programming skills.

Experience with at least one major MPP or cloud database technology (Snowflake, Redshift, BigQuery).

Solid experience with data integration toolsets (i. e., Airflow) and writing and maintaining Data Pipelines with Databricks.

Hands-on experience in writing test cases and performing automated unit testing scripts.

Experience in Data Modeling techniques and Data Warehousing standard methodologies and practices.

Familiar with Scrum and Agile methodologies.

Bachelor's degree in computer science, Information Systems, Software, Electrical or Electronics Engineering, or comparable.

Demonstrated excellent interpersonal skills and communication skills, including the ability to partner with others and build consensus in a cross-functional team toward a desired outcome.

Preferred Qualifications:

Experience with at least one major Massively Parallel Processing (MPP) or cloud database technology (Snowflake, Redshift, or BigQuery).

Experience in Data Modeling techniques and Data Warehousing standard methodologies and practices.

Experience with Software Application development.

Good data acumen to understand domain / functional knowledge and to be able to articulate analysis of data points in business context for stakeholders.

Demonstrated excellent interpersonal skills and communication skills, including the ability to partner with others and build consensus in a cross-functional team toward a desired outcome.

Experience working with a test-first mentality.

Deep Understanding of AWS or other cloud providers as well as infrastructure as code.

Familiar with Scrum and Agile methodologies.