Data Engineer

In this role, the Data Engineer is responsible for providing technical direction and lead a group of one or more developer to address a goal.

• Develop, implement, and optimize data pipelines using Snowflake, with a focus on Cortex AI capabilities.
• Extract, transform, and load (ETL) data from various sources into Snowflake, ensuring data integrity and accuracy.
• Utilize Snowflake's advanced features, including Snowpark, Streams, and Tasks, to enable data processing and analysis.
• Develop and maintain data documentation, best practices, and data governance protocols.
• Ensure data security, privacy, and compliance with organizational and regulatory guidelines.
• Understand and translate business discussion into design requirements and solutions.
• Able to liaise with offshore team for quality and on-time delivery.

Roles and Responsibilities:

• Bachelor’s degree in Computer Science, Data Engineering, or a related field.
• Experience in data engineering, with experience working with Snowflake.
• Proven experience in Snowflake Cortex AI, focusing on data extraction
• Strong proficiency in SQL, Python, and data modeling.
• Good to have experience with data integration tools (e.g., SSIS, Matillion, Talend, Informatica).
• Knowledge of cloud platforms such as AWS, Azure, or GCP.
• Excellent problem-solving skills, with a focus on data quality and performance optimization.
• Strong communication skills and the ability to work effectively in a cross-functional team.
• Proficiency in using DBT's testing and documentation features to ensure the accuracy and reliability of data transformations.
• Understanding of data lineage and metadata management concepts, and ability to track and document data transformations using DBT's lineage capabilities.
• Understanding of software engineering best practices and ability to apply these principles to Snowflake DBT development, including version control, code reviews, and automated testing.
• Should have experience building data ingestion pipeline.
• Should have experience with Snowflake utilities such as SnowSQL, SnowPipe, bulk copy, Snowpark, tables, Tasks, Streams, Time travel, Cloning, Optimizer, Metadata Manager, data sharing, stored procedures and UDFs, Snowsight, OpenFlow and Snowpark.
• Should have good experience in implementing CDC or SCD type 2
• Proficiency in working with Airflow or other workflow management tools for scheduling and managing ETL jobs.
• Good to have experience in repository tools like Github/Gitlab, Azure repo

qualifications

B.E./ Masters in Computer Science, Information technology, or Computer engineering or any equivalent degree with good IT experience and relevant of working experience as a Data Engineer with DBT+Snowflake skillsets