Senior Data Engineer (Databricks Expert)

Senior Data Engineer (Databricks Expert)

REQUIREMENTS:

• Minimum 5 years of experience in designing and building Business Intelligence, ETL/ELT, Data Warehouse, Data Lake, Data Lakehouse, Big Data and OLAP-class solutions, with Databricks used as the core platform in production environments
• Practical knowledge of various relational and non-relational database engines, applied in the context of building and operating Databricks-based data platforms
• Strong proficiency in SQL and Python
• Familiarity with data engineering and orchestration tools, particularly Spark/Databricks (including structured streaming mechanisms, DLT, etc.), Hadoop/CDP, Azure/Fabric Data Factory, Apache Flink, Apache Kafka, Apache Airflow, dbt, Debezium, and more
• Understanding of data governance, data quality, and batch/streaming data processing challenges
• Knowledge of architectural patterns in data, including Data Mesh, Data Vault, Dimensional Modeling, Medallion Architecture, and Lambda/Kappa Architectures
• Proficiency in using git repositories (Bitbucket, GitHub, GitLab)
• Experience with cloud data services on Azure and/or AWS, with Databricks used as a key component of the data platform
• Flexibility, self-reliance, and efficiency, with a strong sense of responsibility for assigned tasks

RESPONSIBILITIES:

• Designing new solutions and coming up with initiatives for improvements to existing solutions within data platforms - both as part of orders coming from the business (functional changes) and from technology (architectural changes)
• Development of data platforms and ETL/ELT processes: technical support and active participation in the development of data platforms
• Work on building and optimizing ETL/ELT processes that are responsible for processing large data sets
• Implement processes to ensure optimal data processing, using data engineering best practices
• Standardize and streamline technical processes: implementing and optimizing code, test and documentation management standards
• Selecting and configuring tools and development environments that support data engineering processes to maintain code quality and facilitate code scaling
• Ensure standards compliance and code review: responsible for applying existing platform development standards, initiating new guidelines where improvements are needed, and monitoring the quality of delivered solutions by conducting regular code-reviews
• Work directly with technology as a Data Engineer and Data Analyst to maintain a high level of technology sophistication
• Act as a mentor to the team