Job Description
Cloud Data Engineer - Scala / Databricks
We are seeking a highly skilled Cloud Data Engineer to join our team. As a remote worker, you will be responsible for designing, implementing, and maintaining robust data pipelines and building scalable data lakes.
Key Responsibilities:
• Design and Development:
• Design, develop, and maintain scalable ETL pipelines using cloud-native tools (AWS DMS, AWS Glue, Kafka, Azure Data Factory, GCP Dataflow, etc.).
• Architect and implement data lakes and data warehouses on cloud platforms (AWS, Azure, GCP).
• Develop and optimize data ingestion, transformation, and loading processes using Databricks, Snowflake, Redshift, BigQuery and Azure Synapse.
• Implement ETL processes using tools like Informatica, SAP Data Intelligence, and others.
• Develop and optimize data processing jobs using Spark Scala.
• Data Integration and Management:
• Integrate various data sources, including relational databases, APIs, unstructured data, and ERP systems into the data lake.
• Ensure data quality and integrity through rigorous testing and validation.
• Perform data extraction from SAP or ERP systems when necessary.
• Performance Optimization:
• Monitor and optimize the performance of data pipelines and ETL processes.
• Implement best practices for data management, including data governance, security, and compliance.
• Collaboration and Communication:
• Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Collaborate with cross-functional teams to design and implement data solutions that meet business needs.
• Documentation and Maintenance:
• Document technical solutions, processes, and workflows.
• Maintain and troubleshoot existing ETL pipelines and data integrations.
Requirements:
• 7+ years of experience as a Data Engineer in a similar role.
• Minimum 3 years of experience specifically working with
