Lead Data Engineer

Job Title: Lead Data Engineer

About the Role:

We are seeking a highly skilled Lead Data Engineer to join our growing team. In this role, you will lead the design, development, and maintenance of scalable data architectures and pipelines. You will be responsible for driving data initiatives, managing a high-performing team, and collaborating with cross-functional stakeholders to support data-driven decision-making across the organization.

Responsibilities:

1. Leadership & Team Management

Lead and mentor a team of data engineers, fostering collaboration and continuous learning.

Define goals, assign responsibilities, and oversee the delivery of data engineering projects.

Conduct code reviews, enforce coding best practices, and ensure high-quality output.

Identify team skill gaps and coordinate training or recruitment to address them.

Act as a liaison between the data engineering team and business or cross-functional teams.

2. Data Architecture & Design

Design scalable, efficient, and secure data architectures (data lakes, warehouses, NoSQL, etc.).

Define data models, schemas, and storage strategies aligned with business goals.

Choose appropriate tools, technologies, and frameworks for data storage and processing.

Support both batch and real-time data processing architectures.

Ensure compliance with data security and regulations (e.g., GDPR, CCPA).

3. Data Pipeline Development

Develop and maintain robust ETL/ELT pipelines from diverse data sources.

Optimize pipelines for performance, scalability, and cost-efficiency.

Implement data validation, quality checks, and monitoring mechanisms.

Automate data workflows and integrate with CI/CD for seamless deployment.

4. Cloud Infrastructure Management

Design and manage data infrastructure on cloud platforms (AWS, Azure, or GCP).

Use Infrastructure-as-Code (IaC) tools (e.g., Terraform) for provisioning and scaling.

Monitor and optimize cloud resource usage and costs.

Ensure high availability and disaster recovery across all data systems.

5. Stakeholder Collaboration

Partner with business stakeholders, product managers, and analytics teams.

Translate business requirements into scalable technical data solutions.

Act as a technical SME in architecture and data engineering discussions.

Use Agile/Scrum to manage projects, milestones, deliverables, and risks.

6. Innovation & Continuous Improvement

Keep up with the latest trends and tools in data engineering and cloud technology.

Evaluate new technologies and implement improvements to the data ecosystem.

Drive innovation through proof-of-concepts and modern engineering practices.

Continuously refine and enhance data systems for efficiency and scale.

7. Documentation & Knowledge Sharing

Create and maintain technical documentation for architectures and data flows.

Build a team knowledge base and lead internal training and workshops.

Promote a culture of learning and transparency within the data team.

Required Qualifications:

Bachelor’s or Master’s in Computer Science, Data Engineering, or a related field.

10+ years of experience in data engineering, including 3+ years in a leadership or architectural role.

Strong hands-on experience with:

Big data tools: Apache Spark, Kafka, etc.

Programming: Python, SQL

Cloud platforms: AWS, Azure, or GCP and associated data services

Experience in designing and implementing scalable ETL/ELT pipelines and data warehouses.

Proficient in data modeling, real-time/batch data processing, and distributed systems.

Familiarity with DevOps tools and practices: Docker, Kubernetes, Terraform, etc.

Deep understanding of data governance, security, and compliance.