Data Engineer

Introduction:

A highly skilled and experienced Data Engineer with over 10 years in the IT industry, specializing in big data, cloud platforms, and data analytics. Expertise in designing and developing scalable data pipelines, end-to-end ETL processes, and data lake architectures using Azure and Google Cloud platforms. Proficient in PySpark, Spark SQL, Hive, and SQL for data processing and transformation. Strong knowledge of Azure services, including Databricks, Synapse Analytics, ADF, Logic Apps, and ADLS, as well as Google Cloud tools like BigQuery, GCS, Dataflow, and Pub/Sub. Experienced in optimizing data workflows, implementing Spark transformations, and working with Delta tables for efficient data management. Skilled in analyzing complex datasets, automating data processing tasks, and improving business processes through data-driven solutions. Adept at developing and maintaining data warehouses, enhancing performance using partitioning and indexing techniques, and implementing data governance frameworks. Strong background in mentoring teams, defining best practices, and collaborating with cross-functional stakeholders to deliver high-quality data solutions. Holds multiple certifications in Azure, Databricks, and AI, demonstrating a deep understanding of modern data engineering technologies and methodologies.

Responsibilities:

• Design and develop scalable data pipelines and end-to-end ETL processes for data ingestion and transformation.
• Architect and implement data lake solutions with structured layers (Raw, Cleansed, and Curated) using cloud platforms.
• Optimize data workflows using Spark, PySpark, and SQL for efficient data processing and transformation.
• Develop and manage data warehouse solutions, ensuring performance tuning and query optimization.
• Automate invoicing and surcharge processes to improve operational efficiency and reduce manual effort.
• Implement Spark transformations, including window functions, pivoting, and complex aggregations.
• Utilize Azure services such as Databricks, Synapse Analytics, ADF, ADLS, and Logic Apps for cloud-based data solutions.
• Leverage Google Cloud technologies like BigQuery, GCS, Dataflow, and Pub/Sub for data processing and analytics.
• Enhance data processing performance through optimization techniques such as Delta tables, partitioning, and indexing.
• Conduct data analysis and implement business logic to derive meaningful insights for decision-making.
• Collaborate with cross-functional teams to define data strategies, architecture, and governance frameworks.
• Provide technical expertise in Azure and Google Cloud environments, including assessments, POCs, and best practices.
• Mentor junior developers and data engineers, guiding them in implementing modules and adhering to best practices.
• Support and troubleshoot data pipelines, ensuring minimal downtime and efficient data flow.
• Work on data migration projects, moving data across platforms and optimizing storage structures.
• Develop and maintain data analytics protocols, standards, and documentation for enterprise-wide data initiatives.
• Perform unit testing and integration testing to validate data accuracy and system performance.
• Implement email configurations, security measures, and compliance standards within data solutions.
• Handle production support, bug fixes, and performance enhancements to ensure system reliability.
• Participate in Agile development processes, contributing to sprint planning, code reviews, and continuous improvement.