- Role: Test Engineering Analyst – ML/AI Chatbot
- Industry: Healthcare IT
- Location: Baltimore/Canton, MD or Washington, DC
- Work Arrangement: Hybrid—onsite 1–2 days per week, typically in Baltimore
- Contract: 6+ months with strong extension potential
- Interview Process: 1–2 video interviews followed by a mandatory in-person interview
- Position Overview
- We are seeking a Test Engineering Analyst to evaluate AI-powered chatbots, Generative AI applications and machine-learning models. This is a hands-on testing and analysis position—not a lead role.
- The analyst will test chatbot responses for accuracy, identify issues such as hallucinations or inconsistent outputs, and communicate findings to senior engineers and business stakeholders for correction.
- Key Responsibilities
- Test AI chatbots to confirm that responses are accurate, relevant and consistent.
- Validate Generative AI outputs, prompts, LLM responses and basic ML predictions.
- Create test strategies, test cases and evaluation frameworks for AI applications.
- Perform manual, exploratory and automated frontend, backend and API testing.
- Test AI/ML inference endpoints and prompt-processing pipelines.
- Use Selenium with Java and BDD frameworks for test automation.
- Analyze model outputs and report problems to engineers, data scientists and business teams.
- Detect hallucinations, bias, model drift and unexpected chatbot behavior.
- Review business requirements and functional specifications.
- Write SQL queries to validate data across relational databases.
- Test Salesforce applications containing AI-powered features.
- Support validation of data pipelines and model-data sources.
- Document test results and defects through Jira and Confluence.
- Participate in CI/CD testing processes.
- Mandatory Requirements
- At least 3 years of software testing or test-automation experience.
- Approximately 3 years of exposure to Generative AI/ML products.
- Hands-on experience testing Generative AI or chatbot applications.
- Strong understanding of chatbot testing and response-accuracy validation.
- Selenium, Java and BDD framework experience.
- Python and SQL experience.
- API, frontend and backend testing experience.
- Experience with Jira and/or Confluence.
- Experience with a major cloud platform: AWS, Azure or GCP.
- Basic understanding of machine learning concepts.
- Experience testing microservices and containerized applications.
- Ability to attend onsite work and an in-person interview.
- Must live within approximately one hour of Baltimore or Washington, DC.
- Technical Environment
- Languages: Python, Java and SQL
- Automation: Selenium, Playwright and BDD frameworks
- AI/ML: Generative AI, LLMs, LangChain, Pandas and ML libraries
- Databases: Oracle, Snowflake and Databricks
- Cloud: AWS, Azure or GCP
- AWS Tools: Bedrock and SageMaker
- CI/CD: CloudBees or Jenkins
- Collaboration: Jira and Confluence
- Additional Environment: Salesforce, APIs, microservices and containers
Mention you found this on Data First Jobs — it helps us bring you more roles like this.
Test Engineering Analyst – ML/AI Chatbot
Holistic Partners, Inc
Similar Analytics Jobs
View all Analytics jobs→Cambium Learning Group
Senior Financial Analyst, FP&A
New
RemoteUSA
Adams Communication & Engineering Technology (ACET, Inc.)
Senior Budget Analyst - NAS Patuxent River
New
Maryland (USA)
BMO
Senior Analyst, Data Analytics & Automation Solutions
New
Toronto, Ontario (Canada)
Actinvision
Consultant Tableau - Data Analyst F/H
New
Montreal, Quebec (Canada)
Clean Energy Alliance
Rates Specialist / Rates Analyst
New
Carlsbad, California (USA)$80,000 - $150,000
ROR
Marketing Analyst
New
Superior, Colorado (USA)
Like this role? Get carefully selected jobs like it, twice a week, straight to your inbox.
Free, no spam. Unsubscribe anytime.