Data DevOps Engineer

As a Data DevOps Engineer, you will serve as a platform expert responsibile for the design, development, automation, testing, support and administration of the Enterprise Infrastructure for Big Data and Fast Data processing.

Requirements

• Background in computer science, engineering, physics, mathematics or equivalent
• Strong background in system administration and shell scripting
• Strong hands on experience with Python, SQL
• Strong problem-solving skills
• Eagerness to learn new approaches and technologies

Nice to have

• Experience with Azure cloud services
• Experience with Databricks and Spark.
• Experience with administration of large clusters
• Experience with Private endpoint configuration
• Understanding of modern Big Data technologies (Hadoop, Spark, Kafka, NoSQL)
• Experience with virtualization
• Experience with containerization (Docker, Kubernetes)
• Experience with CI/CD tools
• Experience with Infrastructure as code
• Experience with configuration or infrastructure management tools

• Communicate effectively with technical and non-technical stakeholders.
• Support and host discussions within a multidisciplinary team, with potentially difficult dynamics.
• Be an advocate for the team externally, and can manage differing perspectives.
• Understand and help teams to apply a range of techniques for data profiling.
• Source system analysis from a different sources.
• Bring multiple data sources together in a conformed model for analysis.
• Establish enterprise-scale data integration procedures across the data development life cycle, and ensure that teams adhere to them.
• Manage resources to ensure that data services work effectively at an enterprise level.
• Identify areas of innovation in data tools and techniques, and recognise appropriate timing for adoption.
• Establish standards, keep them up to date and ensure adherence to them.
• Keep abreast of best practice in industry.
• Understand the concepts and principles of data modelling and can produce relevant data models.
• Recognise opportunities for the reuse and alignment of data models in different organisations.
• Design the method to categorise data models within an organisation.
• Design an appropriate metadata repository and present changes to existing metadata repositories.