Key Skills

Job Description

Position Summary This role is for professionals seeking a long-term career with us; subcontractor applications will not be considered. We are seeking an experienced Senior Data Engineer to design, build, and optimize modern data platforms using the Databricks Lakehouse Platform. The ideal candidate will have 10+ years of data engineering experience and deep expertise in Lakehouse Architecture, Medallion Architecture, cloud-native data solutions, and enterprise-scale data governance practices. This role will be responsible for architecting scalable data solutions, implementing robust data pipelines, enforcing data quality standards, and enabling analytics and AI workloads across the organization. The successful candidate will collaborate closely with business stakeholders, data architects, analytics teams, and platform engineers to deliver high-quality, secure, and reliable data products. Key Responsibilities Data Platform Architecture & Engineering Design and implement scalable, high-performance data platforms using the Databricks Lakehouse architecture. Lead the development of enterprise data solutions leveraging Bronze, Silver, and Gold layers following Medallion Architecture principles. Build and maintain batch and streaming data pipelines for large-scale data processing. Develop reusable data frameworks, accelerators, and engineering best practices. Optimize data storage, processing, and query performance across cloud environments. Databricks Development Design, develop, and maintain Databricks workflows using PySpark, SQL, and Python. Implement and manage Delta Live Tables (DLT) pipelines. Utilize Unity Catalog for centralized governance, metadata management, access controls, and lineage tracking. Support advanced analytics, machine learning, and AI workloads on the Databricks platform. Monitor and troubleshoot production data pipelines and platform performance. Data Governance & Quality Establish and enforce enterprise data governance standards and policies. Implement data quality frameworks, validation rules, reconciliation processes, and monitoring solutions. Ensure compliance with security, privacy, regulatory, and audit requirements. Drive metadata management, lineage, cataloging, and data stewardship initiatives. Collaborate with business and governance teams to improve trust and usability of enterprise data assets. Cloud Engineering Design and deploy cloud-native data solutions on Azure, AWS, or GCP. Implement scalable and secure cloud architectures aligned with enterprise standards. Optimize cloud resource utilization, performance, and cost management. Integrate cloud-native services with Databricks-based solutions. DevOps & Automation Implement CI/CD pipelines for data engineering workloads. Automate deployment, testing, monitoring, and operational processes. Apply Infrastructure as Code (IaC) practices using tools such as Terraform or similar. Promote engineering excellence through code reviews, testing strategies, and release management processes. Leadership & Collaboration Provide technical leadership and mentorship to junior and mid-level engineers. Collaborate with architects, analysts, data scientists, product owners, and business stakeholders. Contribute to platform roadmaps, architectural decisions, and technology evaluations. Lead technical discussions and establish best practices across the data engineering organization. Required Qualifications Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field. 10+ years of experience in Data Engineering, Data Warehousing, or Big Data platforms. 5+ years of hands-on experience with Databricks in enterprise environments. Strong expertise in: Lakehouse Architecture Medallion Architecture Delta Lake Delta Live Tables (DLT) Unity Catalog Data Governance and Data Quality frameworks Data Modeling and Data Warehousing concepts Extensive experience with at least one major cloud platform: Microsoft Azure Amazon Web Services (AWS) Google Cloud Platform (GCP) Advanced programming experience with: Python SQL PySpark Strong understanding of CI/CD methodologies and DevOps practices. Experience implementing Infrastructure as Code (Terraform). Expertise in performance tuning, optimization, and troubleshooting of large-scale data pipelines. Strong understanding of security, access controls, and compliance requirements in cloud environments. Preferred Qualifications Databricks certifications such as 'Databricks Certified Data Engineer Professional' Cloud certifications (Azure, AWS, or GCP). Experience with real-time streaming technologies and event-driven architectures. Experience supporting AI/ML and Generative AI data platforms. Familiarity with data observability tools and modern data quality frameworks. Experience implementing enterprise metadata management and data lineage solutions. Hiring Process We aim to complete the interview process in October-November. The targeted onboarding is January 2027 or earlier.

Core Responsibilities

Design, build, and optimize enterprise-scale Databricks Lakehouse platforms, including batch and streaming pipelines, governance, data quality, and cloud-native solutions. Provide technical leadership and mentorship while collaborating with stakeholders to deliver secure, reliable data products that support analytics, machine learning, and AI workloads.

Requirements

Requires a bachelor's or master's degree in a relevant field, 10+ years of data engineering experience, and 5+ years of enterprise Databricks experience. Candidates need advanced Python, SQL, and PySpark skills; expertise in Lakehouse and Medallion architectures, Delta Lake, DLT, Unity Catalog, governance, and data quality; and experience with a major cloud platform, CI/CD, and Terraform.