Azure Data Engineer (ID: 4069)
Key Skills
Job Description
- Design, develop, and maintain complex and scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks.
- Build and optimize large-scale data processing solutions using PySpark and Databricks.
- Develop robust ETL/ELT solutions in Python for data extraction, transformation, and loading.
- Collaborate with cross-functional teams to understand data requirements and deliver efficient data engineering solutions.
- Implement and maintain CI/CD pipelines using Azure DevOps for automated testing, integration, and deployment.
- Develop, test, and maintain reliable data solutions aligned with technical and business requirements.
- Monitor, troubleshoot, and optimize data pipelines to ensure performance, scalability, and reliability.
- Work with large datasets and use SQL for data querying and data management.
- Ensure data solutions follow applicable security, compliance, and engineering best practices.
- Stay updated with the latest trends, technologies, and best practices in Azure cloud data engineering.
- 8–10 years of overall professional experience in data engineering, cloud data engineering, or related technology roles.
- Strong hands-on experience with Azure Data Factory for data orchestration, movement, and transformation.
- Strong experience with Azure Databricks for developing and managing scalable data processing solutions.
- Strong programming skills in Python.
- Hands-on experience with PySpark for large-scale data processing.
- Experience implementing CI/CD pipelines using Azure DevOps.
- Good knowledge of SQL and experience working with large datasets.
- Experience developing and maintaining ETL/ELT data pipelines.
- Understanding of data security, compliance, and cloud data engineering best practices.
- Strong problem-solving and analytical skills.
- Experience working independently as well as collaboratively within cross-functional teams.
- Design and develop scalable data pipelines using Azure Data Factory and Azure Databricks.
- Develop efficient PySpark applications for processing and transforming large volumes of data.
- Write clean, maintainable, and reusable Python-based data engineering solutions.
- Build and manage automated CI/CD workflows in Azure DevOps.
- Perform data extraction, transformation, loading, validation, and integration across different data sources.
- Optimize data pipelines and Databricks workloads for performance and scalability.
- Monitor pipelines, identify failures or bottlenecks, and implement appropriate solutions.
- Troubleshoot data processing and pipeline-related issues effectively.
- Write SQL queries for data analysis, transformation, and validation.
- Work with stakeholders to understand requirements and translate them into technical data solutions.
- Follow appropriate data security, governance, compliance, and development standards.
- Document technical solutions, data processes, pipelines, and deployment procedures.
- Adapt to new Azure services, cloud data engineering technologies, and industry best practices.
- An opportunity to work on enterprise-scale Azure cloud data engineering projects.
- Exposure to Azure Data Factory, Azure Databricks, PySpark, Python, Azure DevOps, and SQL.
- Opportunities to design and optimize large-scale data pipelines and processing frameworks.
- A collaborative environment involving cross-functional technical and business teams.
- Opportunities to contribute to cloud data modernization, automation, and CI/CD initiatives.
Core Responsibilities
Design, develop, and maintain complex, scalable data pipelines using Azure Data Factory and Databricks. Collaborate with cross-functional teams to implement robust ETL/ELT solutions and ensure data performance and security.
Requirements
Requires 8-10 years of professional experience in data engineering with strong proficiency in Azure cloud services. Candidates must have hands-on expertise in Python, PySpark, SQL, and CI/CD pipeline implementation.
About STAFIDE
Industry: Staffing and Recruiting
Company size: 11-50 employees
Headquartered in the Netherlands, STAFIDE specializes in Niche Tech Recruitment across Europe, connecting top talent with outstanding opportunities in the IT sector. Our data-driven recruitment model ensures the ideal match between candidates and employers, enhancing both skill alignment and cultural fit. With a strong focus on Skill-Based Hiring, we provide candidates with an in-depth understanding of the organization—whether for permanent positions or temporary assignments (interim opdrachten) like secondment (detachering)—strengthening employer branding for our clients as a recruitment company (uitzendbureau). In today’s fast-evolving tech landscape, we recognize the unique challenges companies face in sourcing highly skilled talent. Our advanced, AI-driven talent intelligence system (ATS) is specifically designed to prioritize Skill-Based Hiring for niche IT roles, empowering organizations to address talent shortages by identifying professionals with the expertise to drive innovation and growth. Whether you’re a company in search of exceptional talent or a professional seeking your next ICT role, STAFIDE is your strategic recruitment (werving en selectie) partner. Our deep expertise in the Dutch market, combined with a steadfast commitment to excellence, makes us the preferred choice for all your recruitment needs.