Senior Data Engineer (Python & Databricks) (Remote)

This role supports a large-scale transformation from SQL Server–based systems to a Databricks / Delta Lake platform. The focus is on enterprise-grade data engineering and software development, not analytics or reporting. The project is SQL2Databricks migration, it involves 3500-4000 SQL DBs (2TB), replicating data in different shapes/ schemas to Databricks.

• Support transformation from SQL Server–based systems to a Databricks / Delta Lake platform
• Transform business-critical SQL logic (stored procedures) into clean, maintainable, and scalable Python / PySpark code
• Redesign and implement this logic in Python / PySpark within Databricks
• Contribute to a large, long-running data engineering codebase used
• Develop production-grade transformation code (packages, modules, reusable components)
• Design and evolve data models within a Medallion Architecture (Bronze / Silver / Gold) across multiple data layers
• Ensure software engineering quality, reusability, and long-term maintainability
• Apply software engineering best practices (clean code, OOP, modularization, refactoring)
• Work with very large data volumes and highly parallel, event-driven transformations
• Actively participate in code reviews and technical design discussions
• Support orchestration workflows (e.g., Azure Data Factory)

• Very strong Python and PySpark skills; proven experience with Databricks and Delta Lake
• Experience working in large, shared codebases (beyond notebooks)
• Strong SQL skills, especially reading and understanding complex logic
• Solid object-oriented programming experience, clean code principles
• Strong data modelling background (transactional and analytical)
• Experience in redesigning models during platform migrations
• Familiarity with layered data architectures (Bronze / Silver / Gold)
• Very good English skills