Lead Data Engineer

Skills this role screens for

Also

Job Description

As a Data COE Lead Data Engineer I, you will have the opportunity to design and build scalable, secure, and cost-effective cloud-based data solutions. You will develop and maintain data pipelines to extract, transform, and load data into data warehouses or data lakes, ensuring data quality and validation processes to maintain data accuracy and integrity. You will ensure efficient data storage and retrieval for optimal performance, and collaborate closely with data teams, product owners, and other stakeholders to stay updated with the latest cloud technologies and best practices.

Experience: Data Engineering: 9+ years with high expertise in required tech stack.

Notice Period: 0-3 Months allowed

Who they want

Technical Requirements

• Programming: Python, PySpark, SQL
• ETL & Integration: Databricks (LDP, Lakeflow Connect, Notebooks), AecorSoft/DataSphere
• Data Warehousing: SCD Types, Fact & Dimensions, Star Schema, ETL
• Data Modeling: Dimensional modelling understanding, Erwin tool (Optional)
• Medallion Architecture: Bronze, Silver, Gold (Databricks Medallion Architecture)
• AWS Cloud Services: S3, Lambda, EC2, Glue, Redshift
• GenAI exposure: Genie/Kiro/Gemini etc.
• Scheduling Technologies: Lakeflow Jobs, Airflow
• Visualization (Optional): PowerBI (Optional), GenBI (Optional)

The work

Role & Responsibilities

• Design and Build: Develop and implement scalable, secure, and cost-effective cloud-based data solutions.
• Manage Data Pipelines: Develop and maintain data pipelines to extract, transform, and load data into data warehouses or data lakes.
• Ensure Data Quality: Implement data quality and validation processes to ensure data accuracy and integrity.
• Optimize Data Processing & Storage: Ensure efficient data processing of pipelines & data storage for optimal retrieval performance.
• Collaborate and Innovate: Work closely with data architects, data modelers, product owners, and stay updated with the latest cloud technologies and best practices.
• Team Lead: Create & lead a team of data engineers (external/internal) to ensure timely deliverables fulfilling all data engineering guidelines & standards.

Who they want

Soft Skills

• Problem-Solving: The ability to identify and solve complex data-related challenges.
• Communication: Effective communication skills to collaborate with Product Owners, analysts, and stakeholders.
• Analytical Thinking: The capacity to analyze data and draw meaningful insights.
• Attention to Detail: Meticulousness in data preparation and pipeline development.
• Adaptability: The ability to stay updated with emerging technologies and trends in the data engineering field.