Senior Data Engineer

We are looking for a a Data Engineer for one of our renowned customer and you would be working alongside with Data Scientist / AI Architect in the team to develop scalable and production ready Advanced Analytics and AI software and products. Additionally, to develop different technical tools/services to enable large scale machine learning solutions.

You should believe in a non-hierarchical culture of collaboration, transparency, safety, and trust. Working with a focus on value creation, growth and serving customers with full ownership and accountability. Delivering exceptional customer and business results.

Responsibilities

- Design, develop and build real-time data pipelines from a variety of sources (streaming data, APIs, data warehouse, messages etc.)

- Leverage the understanding of software architecture and software design patterns to write scalable, maintainable well-designed and future-proof software

- Manage existing pipelines and create new pipelines from a variety of sources (relational, XML, etc.)

- Actively apply best practices within CI/CD

- Propose and implement solutions for data pipeline stabilization and data quality checks

- Coordination with other teams to design optimal patterns for data ingest and egress, as well as lead and coordinate data quality initiatives and troubleshooting

- Design and build solutions to track data quality, stabilize data pipeline, etc. to ensure reliable operations

- Ensure best practices are followed across architecture, codebase and configuration

- Eliminate waste

- Deliver on time

Competences

- Ability to establish with clear goals and responsibilities to achieve a high level of performance.

- Ability to evaluate different options proactively and ability to solve problems in an innovative way. Develop new solutions or combine existing methods to create new approaches.

- Comfortable in working with external product teams to establish the optimal data integration patterns/solutions

Functional Knowledge

- PySpark

- Python

- SQL

- Hadoop

- Jenkins

- Docker

- Kubernetes

- Git

- Azure Data Factory

- PowerShell

- Bash

- DevOps

- CI/CD

- Azure

- GCP

- Architecture Principles Design

- Agile Architecture Delivery

- React