Senior Data Engineer

Job description
Key Responsibilities:

Develop, optimise, and maintain data pipelines using Python and SQL within Azure Databricks Notebooks.
Design and implement ETL/ELT workflows in Azure Data Factory, ensuring efficient data transformation and loading.
Apply Kimball dimensional modelling and Medallion architecture best practices for scalable and structured data solutions.
Collaborate with team members and business stakeholders to understand data requirements and translate them into technical solutions.
Implement and maintain CI/CD pipelines using Azure DevOps, ensuring automated deployments and version control with Git.
Monitor, troubleshoot, and optimise Databricks jobs and queries for performance and efficiency.
Work closely with data analysts and business intelligence teams to provide well-structured, high-quality datasets for reporting and analytics.

Essential criteria



1. Demonstrated experience in .NET and Azure application development, including Function Apps, Logic Apps, and Web Apps.


2. Proven ability to design and implement data pipelines using Azure Data Factory, with a preference for candidates experienced in integrating data from Unity Catalog.


3. Experience working collaboratively within agile development teams and with DevOps practices





Desired Skills and Experience

Develop, optimise, and maintain data pipelines using Python and SQL within Azure Databricks Notebooks.
Design and implement ETL/ELT workflows in Azure Data Factory, ensuring efficient data transformation and loading.
Apply Kimball dimensional modelling and Medallion architecture best practices for scalable and structured data solutions.
Collaborate with team members and business stakeholders to understand data requirements and translate them into technical solutions.
Implement and maintain CI/CD pipelines using Azure DevOps, ensuring automated deployments and version control with Git.
Monitor, troubleshoot, and optimise Databricks jobs and queries for performance and efficiency.


If you wish to apply for this position, please submit your resume by clicking the 'Apply Now' button. For further information please contact Shilpa Kunhikannan Sreeja at Clicks IT Recruitment on +61 430 2+4 828.


At Clicks we embrace diversity, inclusion and equal opportunity.


We provide reasonable adjustments, including alternate formats to the recruitment process for individuals with disability. If you require an adjustment to be made during the recruitment process, please call 1300 254 257 or email adjustments@clicks.com.au