Lead Data Engineer (Databricks & Spark)

We are building a next-generation analytics platform and need a Lead Data Engineer to shape scalable ingestion, transformation, and governed data products on Databricks. You will combine hands-on delivery with technical leadership and architecture to enable self-service analytics and AI-assisted engineering.

Responsibilities

• Design scalable data ingestion and transformation frameworks on Databricks

• Define and enforce Databricks best practices for medallion layers and operational excellence

• Build batch and streaming pipelines using Apache Spark and PySpark

• Deliver end-to-end data products from ingestion through governed consumption layers

• Develop trusted analytical datasets and semantic models for reporting and self-service analytics

• Implement data quality, lineage, monitoring, and observability standards

• Lead proof-of-concepts and evaluate emerging capabilities in the Databricks ecosystem

• Mentor engineers and drive reusable frameworks adopted across teams

Requirements

• 5+ years of data engineering experience with production-grade platforms

• Strong Databricks experience in enterprise production environments

• Lead-level technical leadership skills to drive standards and mentor engineers

• End-to-end delivery experience across ingestion, transformation, semantic layers, and reporting

• Advanced Apache Spark and PySpark expertise for scalable pipeline development

• Advanced SQL skills for transformation, modeling, and performance tuning

• Upper-Intermediate English proficiency (B2) for leading technical discussions

Nice to have

• Data architecture experience for enterprise lakehouse patterns

• Data solution architecture skills to define scalable platform designs

• Scala proficiency for Spark development and optimization

We offer

• International projects with top brands

• Work with global teams of highly skilled, diverse peers

• Healthcare benefits

• Employee financial programs

• Paid time off and sick leave

• Upskilling, reskilling and certification courses

• Unlimited access to the LinkedIn Learning library and 22,000+ courses

• Global career opportunities

• Volunteer and community involvement opportunities

• EPAM Employee Groups

• Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn