Data Engineer
About Candidate
Introduction:
The candidate is a highly experienced Senior Data Engineer and Cloud Architect with a strong background in data platform engineering, DevOps, and cloud infrastructure across Azure, AWS, and GCP. They have led the development of scalable, high-performance data platforms for various organizations, implementing Medallion architecture, Infrastructure as Code (IaC), and MLOps workflows.
As a team leader, they have managed and mentored data engineers, overseeing HR, sales, and learning initiatives while driving business expansion strategies. Their expertise includes CI/CD pipeline automation, Terraform refactoring, and optimizing data workflows, significantly improving deployment efficiency and scalability.They have engineered simulation models for large-scale data processing, developed greenfield data platforms, and implemented cloud-native solutions using AWS Batch, RDS, API Gateway, PostgreSQL, and GIS technologies. Additionally, they played a key role in enhancing game data pipelines, recommender systems, and financial planning models in hybrid cloud environments.With a solid academic foundation in scientific programming and computational simulations, they have also contributed to research, teaching, and the development of automated data analysis tools. Their technical acumen, combined with leadership experience, positions them as an innovative problem-solver and strategic architect in the data engineering and cloud ecosystem
Responsibilities:
- Designed and implemented Azure-based data platforms using Data Factory and Databricks to support multiple teams.
- Developed long-term architecture strategies, optimizing Terraform scripts and Infrastructure as Code (IaC).
- Led cloud migration and hybrid cloud transitions, integrating AWS, Azure, and GCP environments.
- Built and maintained data lakehouses and unified streaming pipelines for large-scale data ingestion and processing.
- Engineered simulation models for electrical grid planning, executing 35,000+ simulations efficiently.
- Designed and integrated CI/CD pipelines, reducing release cycles and improving deployment efficiency.
- Implemented Infrastructure as Code (IaC) using Terraform, Ansible, and CloudFormation to streamline cloud operations.
- Developed MLOps workflows for scalable machine learning pipeline management.
- Enhanced data lineage, cost optimization, and pipeline automation to improve operational efficiency.