About project

We are seeking two highly skilled and experienced Data Engineers who will join a team working in a global container liner shipping company.

Team is specialising in building Data Lake and leveraging the Databricks platform.

Possibility of 100% remote work.

Tasks

• Designing and implementing robust data processing pipelines to efficiently ingest, integrate, and transform raw data from diverse sources such as internal databases, IoT data, fleet data, and third-party sources
• Ensuring seamless integration of data into the Data Lake, adhering to data quality and consistency standards
• Utilizing your strong expertise in Databricks and Apache Spark to develop and optimize data processing workflows within the Data Lake
• Leveraging Spark's capabilities to process data
• Taking ownership of data storage and management within AWS
• Implementing data tokenization mechanisms
• Optimizing data processing pipelines and storage structures to enhance data retrieval speed and overall system efficiency
• Providing expert technical support to the Data Lake team and data users, addressing issues related to data ingestion, processing, and access
• Troubleshooting and resolving any data-related problems

• At least 5+ years of experience as a Data Engineer, with a focus on building and managing Data Lake infrastructures
• Strong expertise in AWS services, including data storage and processing using S3, Databricks, and Spark
• Hands-on experience with Databricks, Apache Spark, and Python for data processing and optimization
• Proficiency in working with big data formats like Parquet, Iceberg, Delta Format
• Solid understanding of data privacy regulations and best practices in handling PII data
• Experience with data visualization and reporting tools like Qlik Sense is a plus
• Strong problem-solving skills, attention to detail, and the ability to work in a collaborative team environment
• Excellent communication skills, with the ability to convey technical concepts to both technical and non-technical stakeholders.
• Experience with containerization technologies and Infrastructure as Code (IaC) using Terraform
• Proficiency with orchestration tools like Airflow
• Proficiency in English at the C1 level