Azure Databricks Engineer (ID: 4047)
Key Skills
Job Description
- Design, develop, and maintain robust and reliable data processing solutions using Azure Databricks, Python, and PySpark.
- Build and manage both batch and streaming data processing workflows within Databricks.
- Design appropriate data models and processing architectures within the Databricks environment.
- Manage multiple parallel data processes working on shared source data.
- Develop scalable and maintainable data solutions while ensuring high standards of quality, reliability, and performance.
- Implement and maintain CI/CD pipelines using Azure and YAML.
- Work with Infrastructure as Code (IaC) using ARM/Bicep.
- Continuously improve existing solutions, reduce technical debt, and make pragmatic technical decisions.
- Collaborate effectively within an Agile engineering environment.
- 8+ years of overall professional experience in data engineering/software engineering.
- Strong hands-on experience with Azure Databricks as a core technology.
- Deep expertise in Python, PySpark, and SQL.
- Proven experience designing and managing data processing solutions within Databricks.
- Strong experience handling both streaming and batch data.
- Experience designing data models specifically for Databricks-based processing environments.
- Solid experience with Microsoft Azure cloud services.
- Hands-on experience with CI/CD pipelines using YAML.
- Experience with Infrastructure as Code, particularly ARM/Bicep.
- Experience working in an Agile environment.
- Build stable, scalable, reliable, and maintainable Databricks solutions.
- Apply strong problem-solving and analytical skills to complex data-processing challenges.
- Make well-founded technical decisions in complex and evolving architectures.
- Work effectively with large-scale batch and streaming data processes.
- Manage and coordinate multiple parallel data-processing workloads.
- Identify and address technical debt proactively.
- Continuously improve data-processing solutions and engineering practices.
- Collaborate and communicate effectively with technical and cross-functional teams.
- Follow a structured and pragmatic engineering approach without unnecessary over-engineering.
- Balance engineering best practices with practical business and technical requirements.
- An opportunity to work on complex Azure Databricks and data-processing solutions.
- Exposure to modern cloud, data engineering, CI/CD, and Infrastructure as Code practices.
- A collaborative Agile engineering environment focused on quality and continuous improvement.
- Opportunities to work with both batch and streaming data-processing architectures.
- A technically challenging environment where pragmatic, scalable, and maintainable solutions are valued
Core Responsibilities
Design, develop, and maintain robust data processing solutions using Azure Databricks, Python, and PySpark. Manage both batch and streaming data workflows while implementing CI/CD pipelines and Infrastructure as Code.
Requirements
Requires 8+ years of professional experience in data or software engineering with deep expertise in Azure Databricks, Python, and SQL. Candidates must have proven experience in designing data models and managing large-scale data processing architectures.
About STAFIDE
Industry: Staffing and Recruiting
Company size: 11-50 employees
Headquartered in the Netherlands, STAFIDE specializes in Niche Tech Recruitment across Europe, connecting top talent with outstanding opportunities in the IT sector. Our data-driven recruitment model ensures the ideal match between candidates and employers, enhancing both skill alignment and cultural fit. With a strong focus on Skill-Based Hiring, we provide candidates with an in-depth understanding of the organization—whether for permanent positions or temporary assignments (interim opdrachten) like secondment (detachering)—strengthening employer branding for our clients as a recruitment company (uitzendbureau). In today’s fast-evolving tech landscape, we recognize the unique challenges companies face in sourcing highly skilled talent. Our advanced, AI-driven talent intelligence system (ATS) is specifically designed to prioritize Skill-Based Hiring for niche IT roles, empowering organizations to address talent shortages by identifying professionals with the expertise to drive innovation and growth. Whether you’re a company in search of exceptional talent or a professional seeking your next ICT role, STAFIDE is your strategic recruitment (werving en selectie) partner. Our deep expertise in the Dutch market, combined with a steadfast commitment to excellence, makes us the preferred choice for all your recruitment needs.