Senior Data Engineer (AWS)

Roles & Responsibilities

Data Engineering Excellence

1. Design and implement data pipelines using formats like JSON, Parquet, CSV, and ORC, utilizing batch and streaming ingestion.

Cloud Data Migration Leadership

1. Lead cloud migration projects, developing scalable Spark pipelines.

Medallion Architecture

1. Implement Bronze, Silver, and Gold tables for scalable data systems.

Spark Code Optimization

1. Optimize Spark code to ensure efficient cloud migration.

Data Modeling

1. Develop and maintain data models with strong governance practices.

Data Cataloging & Quality

1. Implement cataloging strategies with Unity Catalog to maintain high-quality data.

Delta Live Table Leadership

1. Lead the design and implementation of Delta Live Tables (DLT) pipelines for secure, tamper-resistant data management.

Customer Collaboration

1. Collaborate with clients to optimize cloud migrations and ensure best practices in design and governance.

Educational Qualifications

Experience

1. Minimum 5 years of hands-on experience in data engineering, with a proven track record in complex pipeline development and cloud-based data migration projects.

Education

1. Bachelor’s or higher degree in Computer Science, Data Engineering, or a related field.

Skills

1. Proficiency in Spark, SQL, Python, and other relevant data processing technologies.
2. Expertise in on-premises to cloud Spark code optimization and Medallion Architecture.
3. Strong knowledge of Databricks and its components, including Delta Live Table (DLT) pipeline implementations.
4. Familiarity with AWS services (experience with additional cloud platforms like GCP or Azure is a plus).

Soft Skills

1. Excellent communication and collaboration skills, with the ability to work effectively with clients and internal teams.

Good to Have

Certifications

1. AWS/GCP/Azure Data Engineer Certification.