AI Quality Analyst

• Job Title: EIS QA Engineer II
• Duration: 5+ months (Total Hours: 936.00)
• Location: 2100 Meadowvale Blvd, Mississauga, Ontario L5N 7J9 (Hybrid)
• Hours: 8/day, 40/week

Job Specification:

AI Quality Analyst

• Job Summary
• The AI Quality Analyst is a critical role responsible for ensuring the performance, safety, and reliability of our cutting-edge AI/ML models. You will be at the forefront of our development lifecycle, designing and executing comprehensive evaluation strategies to identify model weaknesses, potential biases, and critical edge cases. This role requires a blend of analytical rigor, technical aptitude, and a deep curiosity for how AI models behave in real-world scenarios. You will not just find bugs, but provide the actionable insights that drive model improvement and guide our research and development efforts.

Responsibilities:

• Evaluation Strategy & Benchmark Development: Design, develop, and maintain a comprehensive suite of test cases and evaluation benchmarks. Proactively identify potential model failure points, including edge cases, adversarial inputs, and sources of bias.
• Error Analysis & Failure Triage: Conduct systematic error analysis to categorize model failures and identify underlying patterns. Triage defects, prioritize them based on severity and impact, and work with the development team to ensure resolution.
• Data Sourcing & Curation: Source, curate, and manage high-quality datasets for model evaluation and testing. This includes performing data annotation and validation to ensure the integrity of our ground-truth data.
• Exploratory & Adversarial Testing (Red Teaming):Perform unscripted, exploratory testing to discover unexpected model behaviors. Participate in red teaming exercises to intentionally challenge our models and identify potential safety and security vulnerabilities.
• Test Environment Management: Set up, maintain, and troubleshoot testing and demonstration environments to ensure a stable and reliable evaluation pipeline.
• Reporting & Insights: Analyze and synthesize test results into clear, actionable reports for both technical and non-technical stakeholders. Translate complex findings into concrete recommendations for model improvement.
• Process Improvement: Actively participate in post-hoc evaluation reviews and contribute to the continuous improvement of our testing methodologies, tools, and overall quality assurance processes.
• Qualifications and Skills
• Proven experience in a quality assurance, testing, or data analysis role, preferably within the AI/ML domain.
• A deep understanding of the machine learning lifecycle and the common failure modes of AI models.
• Hands-on experience with data annotation, data validation, and managing large datasets.
• Meticulous attention to detail and a methodical approach to problem-solving.
• Strong analytical skills with the ability to identify patterns in data and draw meaningful conclusions.
• Expertise with industry-standard test automation tools and libraries (e.g., Selenium, Playwright, Cypress, REST-assured).
• Experience in testing across different platforms (e.g. comprehensive testing of mobile Android/iOS and web applications)
• Experience with bug tracking systems (e.g., Jira) and test case management tools.
• Scripting skills (e.g., Python) for test automation and data manipulation.
• (Preferred) experience in testing AI systems, including evaluating agentic responses, model performance metrics, and data integrity.
• Excellent communication skills, with the ability to clearly document bugs and articulate complex technical issues.
• Familiarity with computer vision or other specific AI domains relevant to our work.