Product Analyst – AI Agent Evaluation & Quality(Only w2)

• Product Analyst – AI Agent Evaluation & Quality(Only w2)
• Location: Austin, TX(Onsite)
• Minimum 8+ Years of Experience required

• Role Overview
• We are seeking a Product Analyst to join the Channel Strategy & Operations team and support the development of strategic AI agents and an enterprise agent marketplace/platform. This role will own the evaluation, testing, measurement, and quality of AI-powered products.
• Key Responsibilities
• Design and execute evaluations to measure the accuracy, quality, reliability, and performance of AI agents and agentic workflows.
• Develop test cases, evaluation criteria, guardrails, and measurement frameworks to validate AI agent outputs against expected outcomes.
• Analyze evaluation results, identify failure patterns, edge cases, and quality issues, and provide actionable recommendations to Product and Engineering teams.
• Use Claude or similar agentic AI/coding tools to automate testing, evaluation, analysis, and quality workflows.
• Partner with Product and Engineering teams to validate requirements, test new functionality, troubleshoot issues, and improve agent performance.
• Develop reports, dashboards, and analytics to track product quality, performance trends, and improvement opportunities.
• Required Qualifications
• 2–5 years of experience in Testing, Measurement, Evaluation, Quality Assurance, or a related field.
• Hands-on experience designing and executing evaluations for accuracy, quality, reliability, or performance.
• Experience developing test cases, evaluation frameworks, and measurement methodologies.
• Strong analytical and problem-solving skills with the ability to translate test results and data into actionable insights.
• Hands-on experience with Claude, ChatGPT, or other agentic AI/coding tools, including using AI to automate testing, analysis, or operational workflows.
• Strong experience identifying failure patterns, edge cases, and quality issues.
• Excellent cross-functional collaboration and communication skills with Product and Engineering teams.
• Preferred Skills
• Experience testing AI/ML, LLM, GenAI, or agentic applications.
• Familiarity with AI evaluation metrics, guardrails, hallucination testing, prompt testing, and model output validation.
• Experience with SQL, Python, analytics, dashboards, or automation.
• Experience working in Agile/Scrum product development environments.

If I missed your call ! Please drop me a mail.

• Thank you,
• Harish Jannela
• Accounts Manager/Talent Acquisition
• Astir IT Solutions, Inc - An E-Verified Company
• Email:[email protected]
• Direct : 7326946000*788
• 50 Cragwood Rd. Suite # 219, South Plainfield, NJ 07080
• www.astirit.com