About project

We are seeking highly skilled and experienced DataOps Engineer who will join a team working in a global container liner shipping company. Team is specialising in building Data Lake and leveraging the Databricks platform. This role is an exciting opportunity to lead the development and optimization of data orchestration platform, leveraging Astronomer and Airflow, and enabling CI/CD workflows for a cutting-edge, cloud-based Data Warehouse built on Databricks and AWS.

Possibility of 100% remote work.

Tasks

• Developing and maintaining a scalable Astronomer-based scheduling environment for our Data Lakehouse platform
• Designing and implementing CI/CD pipelines to automate deployments, validate data models, and monitor workflows
• Configuring and managing Astronomer workspaces, deployments, and access control
• Integrating dbt Core with Cosmos for orchestrating dbt models in Airflow
• Building and optimizing Cosmos Airflow DAGs for data ingestion and transformation
• Leveraging AWS services (e.g., S3) and Databricks for performance, scalability, and security
• Ensuring comprehensive documentation of workflows, CI/CD processes, and operational standards
• Finalizing the migration of the existing Data Lake scheduling from Airflow MWAA to Astronomer

• At least 5+ years of experience as a Data Engineer or DataOps Engineer, focusing on CI/CD pipelines and workflow orchestration
• Expertise in configuring and deploying Astronomer or Apache Airflow in cloud environments
• Hands-on experience with Databricks, dbt Core, and AWS services
• Proficiency in Python and Spark for data transformation and pipeline development
• Strong SQL skills and knowledge of relational databases
• Familiarity with GitFlow and CI/CD tools like GitLab or Jenkins
• Knowledge of dimensional modeling, data access patterns, and data storage optimization
• Experience with Terraform for Infrastructure as Code and Airflow DAGs with Cosmos
• Excellent collaboration and communication skills, with a proactive approach to sharing knowledge
• Proficiency in English at the C1 level