Data Engineer

Responsibilities:

Set up processes for data management, working on automated analytical modules.

Continuously focus on improvement and automation by partnering with different teams.

The person would lend data engineering support to the creation and execution of omnichannel campaigns for the bank.

Prepare data layer and support for the analytics and data science team.

Help in the integration of new sources of data and pipelines.

Lead the data development of new analytical modules and automated customer journeys.

Design, build, and optimise data pipelines using AWS S3 EMR, and Glue.

Develop and maintain scalable ETL workflows for campaign, customer, deposit, and credit card domains.

Automate tasks that are repetitive, and design ETL pipelines to reduce manual efforts.

Bring in the innovation quotient that can assist in driving business digitally.

Write and maintain complex data transformation logic using Python or PySpark scripts for ETL, data quality, and automation tasks.

Monitor, troubleshoot, and resolve issues in scheduled data jobs (root cause analysis, escalations, bug fixes) using Airflow.

Automate failure notifications, job monitoring, and maintenance history tracking.

Coordinate with the source and Datalake teams for missing data or fixes.

Design and implement automated data quality checks and reporting solutions.

Investigate and resolve data quality issues.

Collaborate with QA and production support teams for continuous process improvement.

Work on the end-to-end data lifecycle from the data ingestion, data transformation and data consumption layers.

Versed in API and its usability.

Ability to work independently and handle your own development efforts.

Excellent oral and written communication skills.

Learn and use available analytic technologies.

Identify key performance indicators and create an educational/deliverables path to achieve the same.

Use background in data engineering and perform analysis.

Work with BI analysts/engineers to create prototypes.

Engage in the delivery and presentation of solutions.

Requirements:

2-5 years of work experience in data engineering with experience in cloud and big data.

Strong hands-on knowledge of Python and PySpark.

Has experience in processing large amounts.

Proficiency in writing SQL queries is a must.

Worked on the AWS/Azure/GCP cloud platform. AWS preferred.

Experience in Airflow or similar orchestrations would be an added advantage.

Experience with Jira or similar project management tools.

Banking domain experience is a plus.

Graduation: Bachelor of Science (B. Sc. ) / Bachelor of Technology (B. Tech. ) / Bachelor of Computer Applications (BCA).

Post-Graduation: Master of Science (M. Sc. ) / Master of Technology (M. Tech. ) / Master of Computer Applications (MCA).

Technical Stack:

Programming: PySpark, Python/Scala.

AWS: S3 EC2 EMR, Glue, Athena.

Orchestrator: Apache Airflow.

Data Integration: Kafka, API, flat files, vendor feeds.

DevOps: GoCD, Bitbucket, Jira.