Reporting and Data Migration lead

Key Responsibilities

ETL Design and Development

Design, implement, and maintain ETL processes to extract, transform, and load data from diverse sources into data warehouses or other storage systems. Optimize ETL workflows for performance, reliability, and scalability. Build reusable ETL components and enforce data quality and observability across pipelines.

Data Integration

Develop and manage data integration solutions, integrating structured and unstructured data from multiple sources. Work with APIs, databases, and flat files to retrieve, process, and stage data. Ensure data lineage, cataloging, and metadata are properly maintained.

Data Transformation

Clean, transform, and standardize data to meet business and analytics requirements. Apply validation, enrichment, and aggregation techniques to produce reliable datasets. Implement error handling and data reconciliation mechanisms.

Data Warehouse Development

Support the design and development of schemas (star/snowflake) and dimensional models. Implement and manage database objects such as tables, views, stored procedures, indexes, and partitions. Collaborate with BI/Analytics teams to ensure models meet reporting and performance needs.

Monitoring and Troubleshooting

Monitor ETL processes for failures, inconsistencies, and SLA breaches; troubleshoot issues proactively. Implement alerting, retry, and recovery mechanisms; maintain runbooks and SOPs. Track pipeline health via dashboards and logs; conduct root cause analysis (RCA) and performance tuning.

Mandatory Skills

ETL Tools/Frameworks: Hands-on with at least one (e.g., Talend, Informatica, Pentaho, SSIS, dbt, Apache Airflow/Luigi/NiFi). SQL & Data Modeling: Strong SQL, dimensional modeling (star/snowflake), query optimization. RDBMS: Experience with Oracle and PostgreSQL (DDL/DML, indexing, partitioning, performance tuning). Kafka: Practical experience with topics, producers/consumers, offsets, schema management, and cluster monitoring. Scripting: Python or Shell for automation, data parsing, and ETL orchestration. APIs & Files: Integration using REST APIs, JSON/CSV/Parquet, and file-based ingestion patterns. Monitoring & Observability: Experience with pipeline monitoring, logging, and alerting (e.g., Grafana/Prometheus/ELK). Version Control & CI/CD: Git and CI/CD for ETL deployments.

Desirable Skills

Cloud Data Platforms: AWS/Azure/GCP (e.g., S3/ADLS, Glue/Data Factory, Lambda/Functions). Streaming & Batch: Experience with Kafka Connect, ksqlDB, or Spark/Flink for streaming ETL. Orchestration: Airflow DAG design best practices; SLA management and backfills. Data Quality: Great Expectations / Deequ; data validation frameworks. Performance Tuning: SQL and pipeline performance tuning at scale. Security & Compliance: Row/column-level security, encryption at rest/in transit, data governance.

Behavioral & Professional Attributes

Strong analytical and problem-solving abilities with a data-driven mindset. Ownership-oriented; able to work independently and in cross-functional teams. Clear and concise communication with stakeholders (engineering, BI, product). Detail-focused with strong documentation habits (runbooks, SOPs, design specs). Comfortable working in fast-paced environments with shifting priorities and SLAs.

Nice-to-Have (Tools & Ecosystem)

Experience with dbt, Snowflake/Redshift/BigQuery, Oracle GoldenGate, or AWS DMS. Exposure to data catalog tools and metadata management.