Technical Lead

Job Description

Senior Data Engineer:

Incedo is a US-based consulting, data science and technology services firm with over 2,000 people helping clients from our six offices across US and India. We help our clients achieve competitive advantage through end-to-end digital transformation. Our uniqueness lies in bringing together strong engineering, data science, and design capabilities coupled with deep domain understanding. We combine services and products to maximize business impact for our clients in telecom, financial services, product engineering and life science & healthcare industries.

Working at Incedo will provide you an opportunity to work with industry leading client organizations, deep technology and domain experts, and global teams. Incedo University, our learning platform, provides ample learning opportunities starting with a structured onboarding program and carrying throughout various stages of your career. A variety of fun activities are also an integral part of our friendly work environment. Our flexible career paths allow you to grow into a program manager, a technical architect or a domain expert based on your skills and interests.

Role Description:

We are seeking a skilled professional to maintain and support batch jobs in a legacy environment. The role involves managing and monitoring ETL processes, addressing issues, and enhancing existing PL/SQL scripts. The ideal candidate will have strong expertise in Informatica, SQL Server, and data warehousing concepts, along with experience in troubleshooting and improving batch job performance.

Key Responsibilities:

• Design and implement robust ETL pipelines using AWS Glue, Lambda, and S3.
• Monitor and optimize the performance of data workflows and batch processing jobs.
• Troubleshoot and resolve issues related to data pipeline failures, inconsistencies, and performance bottlenecks.
• Collaborate with cross-functional teams to define data requirements and ensure data quality and accuracy.
• Develop and maintain automated solutions for data transformation, migration, and integration tasks.
• Implement best practices for data security, data governance, and compliance within AWS environments.
• Continuously improve and optimize AWS Glue jobs, Lambda functions, and S3 storage management.
• Maintain comprehensive documentation for data pipeline architecture, job schedules, and issue resolution processes.

Required Skills and Experience:

• Strong experience with Data Engineering practices.
• Experience in AWS services, particularly AWS Glue, Lambda, S3, and other AWS data tools.
• Proficiency in SQL, python , Pyspark, numpy etc and experience in working with large-scale data sets.
• Experience in designing and implementing ETL pipelines in cloud environments.
• Expertise in troubleshooting and optimizing data processing workflows.
• Familiarity with data warehousing concepts and cloud-native data architecture.
• Knowledge of automation and orchestration tools in a cloud-based environment.
• Strong problem-solving skills and the ability to debug and improve the performance of data jobs.
• Excellent communication skills and the ability to work collaboratively with cross-functional teams.
• Good to have knowledge of DBT & Snowflake

Preferred Qualifications:

• Bachelors degree in Computer Science, Information Technology, Data Engineering, or a related field.
• Experience with other AWS data services like Redshift, Athena, or Kinesis.
• Familiarity with Python or other scripting languages for data engineering tasks.
• Experience with containerization and orchestration tools like Docker or Kubernetes.

We are an Equal Opportunity Employer:

We value diversity at Incedo. We do not discriminate based on race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.

Job Classification

Contact Details: