Key Skills

Job Description

Requirements - Must have skills –. Key Responsibilities: • Design, develop, and maintain complex data pipelines using Azure Data Factory and Azure Databricks. • Build and optimize large-scale data processing solutions using PySpark and Databricks. • Collaborate with cross-functional teams to understand data requirements and deliver efficient data solutions. • Implement CI/CD pipelines in Azure DevOps for automated testing, integration, and deployment of data solutions. • Develop, test, and maintain robust data solutions in Python for data extraction, transformation, and loading (ETL) processes. • Monitor, troubleshoot, and optimize data pipelines for performance and reliability .• Ensure data security and compliance with company standards and best practices. • Stay updated with the latest trends and best practices in cloud data engineering and Azure services.Required Skills and Qualifications: • Experience with Azure Data Factory for orchestrating and automating data movement and transformation. • Proficiency in Azure Databricks for building and managing scalable data processing frameworks. • Strong programming skills in Python and experience working with PySpark for big data processing. • Knowledge of Azure DevOps and experience with CI/CD pipeline implementation. • Familiarity with SQL for querying and managing large datasets. • Strong problem-solving skills and the ability to work independently or as part of a team. •

Core Responsibilities

Design, develop, and optimize scalable data pipelines and processing solutions using Azure Data Factory, Azure Databricks, PySpark, and Python. Implement CI/CD in Azure DevOps, collaborate with teams on data needs, and monitor pipelines to ensure performance, reliability, security, and compliance.

Requirements

Requires experience with Azure Data Factory and Azure Databricks, strong Python and PySpark skills, and knowledge of Azure DevOps and CI/CD implementation. Familiarity with SQL, strong problem-solving ability, and the capacity to work independently or collaboratively are also required.