Data Engineer (Remote)

Responsibilities

• Build a FinOps Data Platform – design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data to produce a cloud cost chargeback dataset.
• Develop robust ETL (Extract, Transform, Load) processes to integrate data from diverse sources into our data ecosystem.
• Implement data validation and quality checks to ensure accuracy and consistency.
• Design and maintain data models, schemas, and database structures to support analytical and operational use cases.
• Optimize data storage and retrieval mechanisms for performance and scalability.
• Evaluate and implement data storage solutions, including relational databases, NoSQL databases, data lakes, and cloud storage services.
• Build and maintain integrations with internal and external data sources and APIs.
• Implement APIs and web services for data access and consumption.
• Ensure compatibility and interoperability between different systems and platforms.
• Ability to build reliable, observable data workflows.
• Knowledge with cloud platform (Azure data factory, Synapse, AWS S3, BiqQuery, etc.)
• Scheduling, dependencies, retries, backfills and data validation.

Requirements

• Bachelor’s degree in computer science, Engineering, Information Systems, or related field.
• 6-8 years of experience in data engineering, software development, or related roles.
• Proficiency in programming languages is commonly used in data engineering (e.g., Python, Java, Scala, etc.).
• Strong knowledge of database systems, data modeling techniques, and SQL proficiency.
• Proficiency with ETL tools commonly used in data engineering (e.g., Databricks, Azure Data Factory).
• Experience with big data technologies and frameworks (e.g., Hadoop, Spark, Kafka, etc.).
• Familiarity with cloud platforms and services (e.g., AWS, Azure, GCP, Snowflake, etc.).
• Excellent problem-solving skills and attention to detail.

Nice to have

• Experience with TBM Studio.
• Experience with Cloudability Data Platform.