So… why is it free?

Employers pay when they hire through Elip. You're not the product — you're the talent it finds. No contracts, no card. Not useful? One message wipes everything. No hidden Pro tier. No “contact sales.” This is the entire pricing page.

Data Engineer (Real - Time Streaming & Big Data) at IDFC FIRST Bank — Bengaluru

Experience3–7 yrs
SalaryNot disclosed
Work modeonsite
Levelmid
Posted2 Sept 2026

Apply for this job

About IDFC FIRST Bank

IDFC FIRST Bank is hiring in Bengaluru in financial services. This role looks for around 3+ years of experience.

Skills

Apache Flink
Spark Structured Streaming
Apache Spark
PySpark
Python
SQL
AWS
Amazon EMR
Amazon S3
AWS Glue
AWS Lambda
Real-time streaming
ETL/ELT
Distributed systems
Big data engineering

The role

A data engineer at a banking technology company designs real-time data pipelines using Apache Flink, Spark Structured Streaming, and PySpark for low-latency analytics and large-scale processing. The role builds AWS data pipelines and optimizes production streaming systems.

Full job description

We are looking for a hands-on Data Engineer with strong experience in big data engineering and real-time streaming systems. You will work closely with Data Platform, Analytics, and Machine Learning teams to design and build scalable, reliable, and low-latency data pipelines that enable real-time analytics and decision-making.

Responsibilities:

Design, develop, deploy, and own real-time streaming applications using Apache Flink.

Build and maintain Spark Structured Streaming pipelines for high-volume, real-time data processing.

Tune and troubleshoot Flink and Spark workloads in production, including: Checkpointing, State management, Backpressure handling, Resource and performance optimization.

Develop and optimize batch and streaming ETL/ELT pipelines using PySpark and Python.

Write efficient and optimized SQL for data transformation, validation, and downstream analytics consumption.

Build and deploy cloud-native data pipelines on AWS, leveraging services such as EMR, S3 AWS Glue, and Lambda.

Design scalable solutions capable of meeting defined latency, throughput, reliability, and data accuracy SLAs.

Implement monitoring, schema validation, error handling, and data quality checks to ensure pipeline reliability.

Apply strong programming, data structures, and algorithms fundamentals to build performant, maintainable, production-grade systems.

Work effectively across a complex ecosystem involving technology teams, vendors, business stakeholders, and other engineering teams.

Contribute to Agile projects spanning multiple teams, business units, and organizations.

Requirements:

3-5 years of professional experience in Data Engineering / Big Data Engineering.

Strong hands-on experience with Apache FlinkMandatory.

Experience building and supporting production-grade real-time streaming applications.

Strong experience with Apache Spark / Spark Structured Streaming.

Hands-on expertise in PySpark and Python.

Strong understanding of streaming architectures, distributed data processing, and large-scale data systems.

Strong working knowledge of SQL and data transformation.

Experience building data pipelines on AWS, particularly with: EMR, S3 AWS Glue, Lambda.

Experience with performance tuning, debugging, monitoring, and optimization of production data pipelines.

Ability to work across multiple stakeholders, teams, and technology ecosystems.

Good to Have:

Programming experience in Java or Scala.

Strong understanding of Data Structures and Algorithms (DSA).

Experience working in Agile environments across multiple organizations or business units.

Key Skills: Apache Flink, Spark Structured Streaming, Apache Spark, PySpark, Python, SQL, AWS, EMR, S3 AWS Glue, Lambda, Real-Time Streaming, ETL/ELT, distributed systems, and big data.