Key Skills

Job Description

Must have skills –. Key Responsibilities: • Design, develop, and maintain complex data pipelines using Azure Data Factory and Azure Databricks. • Build and optimize large-scale data processing solutions using PySpark and Databricks. • Collaborate with cross-functional teams to understand data requirements and deliver efficient data solutions. • Implement CI/CD pipelines in Azure DevOps for automated testing, integration, and deployment of data solutions. • Develop, test, and maintain robust data solutions in Python for data extraction, transformation, and loading (ETL) processes. • Monitor, troubleshoot, and optimize data pipelines for performance and reliability. • Ensure data security and compliance with company standards and best practices.• Stay updated with the latest trends and best practices in cloud data engineering and Azure services. Required Skills and Qualifications: • Experience with Azure Data Factory for orchestrating and automating data movement and transformation. • Proficiency in Azure Databricks for building and managing scalable data processing frameworks. • Strong programming skills in Python and experience working with PySpark for big data processing • Knowledge of Azure DevOps and experience with CI/CD pipeline implementation.• Familiarity with SQL for querying and managing large datasets.

Core Responsibilities

Design, build, and maintain scalable data pipelines and processing solutions using Azure Data Factory, Azure Databricks, PySpark, and Python. Implement CI/CD, collaborate with teams on data needs, and monitor, troubleshoot, optimize, and secure data solutions.

Requirements

Candidates need experience with Azure Data Factory and Azure Databricks, strong Python skills, and experience using PySpark for large-scale data processing. Familiarity with Azure DevOps and CI/CD implementation, as well as SQL, is also required.