Senior Azure Databricks Engineer(ID: 4002)
Key Skills
Job Description
- Design, develop, and maintain scalable and reliable data processing solutions using Azure Databricks.
- Build and manage robust batch and streaming data pipelines within Databricks environments.
- Develop and optimize data transformation and processing solutions using Python, PySpark, and SQL.
- Design and optimize data models to support scalable processing, performance, and reliability.
- Manage multiple parallel data processing workflows and shared data sources efficiently.
- Implement and maintain CI/CD pipelines using Azure DevOps and YAML-based configurations.
- Apply Infrastructure as Code (IaC) using ARM/Bicep for deployment and infrastructure automation.
- Monitor, troubleshoot, and optimize data processing workloads and Databricks environments.
- Collaborate with cross-functional engineering and business teams to deliver reliable and maintainable data solutions.
- Contribute to Agile development practices and continuously improve engineering standards, system stability, and performance.
- Strong hands-on experience with Azure Databricks as a core data engineering platform.
- Strong proficiency in Python, PySpark, and SQL.
- Hands-on experience developing batch and streaming data pipelines.
- Experience with data modeling, transformation, and optimization within Databricks environments.
- Good understanding of Azure cloud services relevant to data engineering.
- Experience with Azure DevOps, CI/CD, and YAML-based pipeline configurations.
- Hands-on experience with Infrastructure as Code, particularly ARM/Bicep.
- Experience working in Agile engineering and delivery environments.
- Understanding of modern cloud-based data architectures and end-to-end data engineering solutions.
- Strong communication, collaboration, troubleshooting, and problem-solving skills.
- Build scalable, high-performance, and reliable data processing solutions using Azure Databricks.
- Develop efficient PySpark and SQL-based data transformations.
- Design and manage complex batch and streaming workloads.
- Optimize data pipelines, processing performance, and resource utilization.
- Troubleshoot complex data engineering issues and improve system reliability.
- Implement automated deployment and infrastructure management practices.
- Make pragmatic technical decisions while maintaining scalability and maintainability.
- Work effectively with engineering, architecture, and business stakeholders.
- Drive continuous improvement and maintain high standards of code and solution quality.
- Opportunity to work on enterprise-scale Azure and Databricks data engineering initiatives.
- Exposure to modern cloud-based data platforms and engineering practices.
- A collaborative Agile environment focused on technical excellence and innovation.
- Opportunities to work with advanced data processing, pipeline engineering, and cloud technologies.
- Continuous learning and opportunities for technical and professional growth.
- A culture focused on quality, ownership, scalability, and sustainable engineering solutions.
Core Responsibilities
Design, develop, and maintain scalable data processing solutions and pipelines using Azure Databricks. Collaborate with cross-functional teams to optimize data models and ensure system reliability through CI/CD and infrastructure automation.
Requirements
Requires strong hands-on experience with Azure Databricks, Python, PySpark, and SQL. Candidates must have a background in developing batch and streaming pipelines within Agile environments.
Benefits
- Continuous learning
- Professional growth opportunities
- Collaborative environment
About STAFIDE
Industry: Staffing and Recruiting
Company size: 11-50 employees
Headquartered in the Netherlands, STAFIDE specializes in Niche Tech Recruitment across Europe, connecting top talent with outstanding opportunities in the IT sector. Our data-driven recruitment model ensures the ideal match between candidates and employers, enhancing both skill alignment and cultural fit. With a strong focus on Skill-Based Hiring, we provide candidates with an in-depth understanding of the organization—whether for permanent positions or temporary assignments (interim opdrachten) like secondment (detachering)—strengthening employer branding for our clients as a recruitment company (uitzendbureau). In today’s fast-evolving tech landscape, we recognize the unique challenges companies face in sourcing highly skilled talent. Our advanced, AI-driven talent intelligence system (ATS) is specifically designed to prioritize Skill-Based Hiring for niche IT roles, empowering organizations to address talent shortages by identifying professionals with the expertise to drive innovation and growth. Whether you’re a company in search of exceptional talent or a professional seeking your next ICT role, STAFIDE is your strategic recruitment (werving en selectie) partner. Our deep expertise in the Dutch market, combined with a steadfast commitment to excellence, makes us the preferred choice for all your recruitment needs.