Senior Data Engineer – Azure Databricks & Scala Spark - 53

Job Description & Summary:

We are looking for a Senior Data Engineer with strong hands-on experience in PySpark development on

Azure Databricks. The role involves end-to-end ownership of data pipeline development, from coding and

version control to build, deployment, and production validation in a modern lakehouse environment. You

will work primarily on PySpark and Scala codebases, build Spark JARs using Maven, and execute/validate

them on Azure Databricks using Spark Submit and Databricks Workflows.

Level: Senior Associate

Location: Bangalore

Skill: Apache Spark on Azure Databricks, Azure Databricks, Pyspark, SQL, Scala and Hadoop

Job Position Title: Senior Data Engineer – Azure Databricks & Scala Spark

Responsibilities: • Design, develop, and maintain scalable batch data pipelines using Apache

Spark on Azure Databricks.

• Work on Spark repositories, implementing new features and

enhancements.

• Use Git for day-to-day development (branching, committing, merging, pull

requests).

• Build Spark executable JARs using Maven and manage dependencies.

• Execute and validate Spark jobs using Spark Submit on Azure Databricks

with custom runtime arguments and I/O paths.

• Configure, orchestrate, and monitor Databricks Workflows, Jobs, and

Pipelines for reliable batch execution.

• Implement and manage Delta Lake components (Delta Tables, Delta Live

Tables).

• Work with Hive Metastore and Unity Catalog for metadata, governance,

and access control.

• Optimize Spark jobs for performance and cost; troubleshoot failures and

performance bottlenecks.

• Ensure code quality, stability, and best practices in version control, build,

and deployment.

• Collaborate with architects, data modelers, and business stakeholders to

translate requirements into technical solutions.

Mandatory skill sets: • Hands-on Execution (must be able to do independently):

• Working with Git repositories: branching, committing, pushing, merging,

and handling pull requests.

• Making code changes in Scala/PySpark projects and taking them through

to production execution.

• Building Spark JARs from Scala/Spark repositories using Maven.

• Executing and validating Spark jobs on Azure Databricks using Spark

Submit (including handling arguments and I/O paths).

• End-to-end ownership of changes: from development to successful

execution and validation.

• Core Technical Skills:

• 6+ years of overall experience in Data Engineering, with strong focus on:

• Apache Spark (core APIs, transformations/actions,

RDD/DataFrame/Dataset APIs).

• Spark performance optimization (partitions, caching, joins, shuffle tuning,

skew handling, etc.).

• Debugging Spark jobs and resolving runtime and data issues.

• Databricks platform:

• Databricks Workflows, Jobs, and Pipelines.

• Operational monitoring, alerting, and troubleshooting in Databricks.

Good To Have skill sets: • Proficiency in Pyspark and Scala for Spark development (primary

language for repositories)., • Strong SQL skills for complex queries,

transformations, and performance tuning., • Solid understanding of

advanced Data Engineering concepts (e.g., data modeling in lakehouse,

incremental/batch patterns, CDC, SCDs, data quality frameworks)., • Strong

working knowledge of Git and branching/versioning best practices.

Years of experience required: 6-9 years

Education qualification: BE B.tech