Senior Data Engineer

About the Role

We are looking for a Senior Data Engineer with deep Databricks expertise to design, build, and optimize scalable data platforms. This is a hands-on engineering role — you will architect and implement production-grade solutions, drive modernization from legacy data warehouses to Lakehouse architectures, and act as a technical advisor both internally and directly with clients. If you thrive on solving complex distributed systems challenges and enjoy mentoring others, this role is for you.

What You’ll Do

Architecture & Development

• Design and implement scalable data pipelines using Apache Spark (PySpark) on Databricks.
• Build robust, production-grade solutions — not just scripts.

Optimization

• Analyze and optimize complex Spark jobs for performance and cost-efficiency through partitioning, z-ordering, and Photon engine utilization.

Modernization

• Lead the migration of legacy data warehouses to Lakehouse and cloud-native architectures using Delta Lake.

Engineering Standards

• Enforce high code quality standards through rigorous code reviews and CI/CD implementation via Databricks Asset Bundles.

Mentorship & Client Collaboration

• Share knowledge and best practices with the broader engineering team.
• Act as a technical advisor for clients, translating complex data engineering concepts into clear, actionable guidance.

Requirements

Experience

• 5+ years in Data Engineering with a strong focus on distributed systems.
• Minimum 3 years of hands-on experience with the Databricks platform, including deep understanding of Delta Lake internals.

Technical Skills

• Production experience with Azure (ADLS Gen2, ADF) or AWS (S3, Glue).
• Advanced proficiency in Python and SQL.
• Hands-on experience with Unity Catalog implementation and modern data governance practices.

Communication

• Fluent English at C1+ level — mandatory for direct client-facing collaboration.

Nice to Have

• Snowflake governance: row access policies, dynamic data masking, tag replication.
• AI/ML for data classification — LLM-based PII detection and automated sensitivity tagging.
• Experience with Delta Live Tables (DLT).
• Databricks Apps development.
• Databricks Certified Data Engineer Professional certification.
• Experience with dbt (data build tool).

What Makes a Strong Candidate

The ideal candidate combines deep technical expertise in Databricks and distributed data systems with the communication skills and client mindset to operate effectively in a consulting-style environment. You are someone who takes ownership, raises engineering standards, and helps others grow — while remaining hands-on in building and optimizing complex data platforms.