Assistant Director-Databricks Platform and Data Engineer at Acuity Knowledge Partners, Bengaluru

Open. Posted .

Databricks platform and data engineering spans production batch and real-time pipelines, AWS administration, Unity Catalog governance, security, and performance optimization. The role is based in Bengaluru.

Apply on Acuity Knowledge Partners's site for Assistant Director-Databricks Platform and Data Engineer

Applying opens the employer's page. Want jobs like this sent to you each morning? Open Elip.

Key facts
LocationBengaluru, Karnataka
Work modeOn-site
Experience8–12 years
PayNot stated by the employer
RoleData Engineer
Skills asked forDatabricks, Python, SQL, PySpark, Delta Lake, Apache Kafka, Spark Structured Streaming, AWS
Posted11 Nov 2026

Elip's summary

Builds and operates data platforms for a financial services company. Designs Databricks data pipelines and real-time streaming, and administers AWS platform security and Unity Catalog governance using Databricks, PySpark, and Apache Kafka. Optimizes Spark performance and automates deployments with Git-based CI/CD.

The job, in the employer's words

Job purpose

The Senior Databricks Platform & Data Engineer will design, build, deploy, administer, and optimize enterprise data solutions using the Databricks Data Intelligence Platform on Amazon Web Services. The role combines advanced data engineering with platform administration and requires hands-on ownership of batch and realtime pipelines, platform security, Unity Catalog governance, compute optimization, deployment automation, monitoring, and production support. The engineer will work closely with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, and firewall teams to deliver secure, scalable, reliable, and cost-effective data platforms.

Key responsibilities

Design, build, and maintain production-grade batch and real-time data pipelines using Databricks, Python, SQL, PySpark, Delta Lake, Apache Kafka, and Spark Structured Streaming.

Administer, configure, deploy, monitor, and support Databricks environments on AWS, including workspaces, compute resources, permissions, jobs, and environment-specific configurations.

Tune and scale Spark workloads, SQL queries, Delta tables, streaming jobs, and compute resources to meet performance, reliability, latency, and cost objectives.

Evaluate serverless, classic, and hybrid compute models, recommending the appropriate approach based on workload characteristics, security, cost, scalability, performance, and operational complexity.

Build fault-tolerant streaming pipelines using Kafka and Structured Streaming, including checkpointing, schema evolution, failure recovery, latency monitoring, and streaming-performance optimization.

Implement Unity Catalog for centralized governance, metadata management, lineage, data discovery, access control, auditing, and secure data sharing.

Automate data ingestion, transformation, testing, deployment, monitoring, and recovery using Databricks workflows, Lakeflow Jobs, Auto Loader, Git-based CI/CD, and Databricks Asset Bundles where applicable.

Establish secure connectivity between source systems, AWS services, Kafka, and Databricks, working with infrastructure, network, cybersecurity, and firewall teams.

Configure and troubleshoot AWS networking and security, including VPCs, subnets, routing, security groups, private connectivity, IAM roles and policies, encryption keys, and secrets.

Integrate Databricks with AWS services such as Amazon S3, EC2, CloudWatch, KMS, and Secrets Manager, with exposure to Kubernetes and Amazon EKS where applicable.

Diagnose and resolve production issues involving source availability, data ingestion, streaming latency, Spark performance, compute capacity, permissions, networking, firewalls, and platform security.

Document architecture decisions and technical trade-offs, clearly explaining the impact of design choices on cost, security, performance, availability, scalability, and maintainability.

Key competencies

Technical Competencies

Databricks Platform Engineering: Advanced hands-on capability in administering, configuring, deploying, monitoring, tuning, scaling, and supporting Databricks on AWS.

Python, SQL and PySpark Engineering: Strong programming capability for developing, testing, optimizing, and troubleshooting production-grade pipelines.

Kafka and Structured Streaming: Expertise in real-time ingestion, event-driven processing, fault tolerance, checkpointing, schema evolution, latency management, and streaming recovery.

Unity Catalog, Security and Governance: Experience implementing data lineage, metadata governance, permissions, auditing, data isolation, secure sharing, and least-privilege access.

AWS Infrastructure and Networking: Working knowledge of IAM, VPCs, subnets, routing, security groups, private connectivity, S3, EC2, CloudWatch, KMS, Secrets Manager, and firewall dependencies.

Performance, Scalability and Cost Optimization: Ability to tune Spark jobs, SQL queries, streaming pipelines,

Delta tables, clusters, and serverless compute while managing cost and reliability.

Deployment and Automation: Experience with Git, CI/CD, Databricks Asset Bundles, automated testing, deployment automation, workflow orchestration, and operational monitoring.

Architecture and Trade-off Analysis: Ability to assess serverless, classic, and hybrid approaches based on security, cost, performance, scalability, supportability, and operational risk.

Behavioral Competencies

Production Ownership and Problem-Solving: Ability to independently diagnose and resolve complex data, platform, networking, security, streaming, and performance issues.

Cross-functional Collaboration and Communication: Ability to work effectively with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, firewall, and source-system teams.

About Acuity Knowledge Partners

Headquartered in London, founded in 2003.

People in Elip's index who work here
7,133
Typical time at Acuity Knowledge Partners
2.2 years
Typical career length so far
7.7 years

Largest teams

  • Program and Project Management1,277 people
  • Consulting995 people
  • Operations910 people
  • Research779 people

Where people here worked before

  • Moody's Analytics Knowledge Services244 people
  • Evalueserve155 people
  • Cians Analytics141 people
  • EY97 people
  • Genpact94 people

Where people went next, since 2025

  • Evalueserve82 people
  • Deloitte40 people
  • EY39 people
  • Wells Fargo23 people
  • HSBC21 people

Totals from public professional profiles in Elip's index; any group under ten people is left out.

All open jobs at Acuity Knowledge Partners

Similar open jobs

More jobs in Bengaluru or all jobs on Elip

About this page

Elip lists jobs that employers publish on their own careers sites, never copies from job boards. This one comes from Acuity Knowledge Partners's listing on Darwinbox, posted 11 Nov 2026. The summary is Elip's; the description is the employer's. When the job closes, this page says so.