AI Test Automation Engineer

Project description (VR-120860)

Join our Development Center in Bucharest, and become a member of our open-minded, progressive and professional team. In this role you will be working on projects for one our world famous clients.

You will have a chance to grow your technical and soft skills, and build a thorough expertise of the industry of our client.

On top of attractive salary and benefits package, Luxoft will invest into your professional training, and allow you to grow your professional career.

Responsibilities

We're seeking an experienced AI Quality Assurance Engineer to ensure the quality, reliability, and performance of our AI-powered solutions built on Google's Gemini platform and CX Agent Studio automation systems. You'll play a critical role in testing and validating conversational AI experiences and customer experience automation workflows

Key Responsibilities

You'll design and execute comprehensive testing strategies for Gemini-powered applications and CX Agent Studio automation workflows, focusing on conversational AI quality, intent recognition accuracy, and end-to-end customer journey validation. Working alongside AI developers and CX designers, you'll establish quality benchmarks for chatbot responses, voice interactions, and automated customer service flows.

Your work will include creating automated test suites for Gemini API integrations, validating multi-turn conversations and context handling, and testing CX Agent Studio automation scenarios across various customer touchpoints. You'll monitor deployed conversational AI agents for response quality degradation, hallucinations, and edge case failures, while also evaluating prompt engineering effectiveness and model response consistency.

You'll develop testing frameworks to assess natural language understanding accuracy, entity extraction, and sentiment analysis within CX Agent Studio workflows. Additionally, you'll conduct adversarial testing to identify prompt injection vulnerabilities and unintended behaviors, analyze conversational outputs for bias and inappropriate content, and collaborate with product teams to define acceptance criteria for AI-driven customer experiences.

Skills

• 3+ years of experience in software quality assurance: API test automation, testing AI/ML systems, preferably conversational AI or customer experience platforms.
• Strong understanding of large language models, prompt engineering, and generative AI evaluation is essential; familiarity with dialogue flow validation.
• Experience with Google Cloud Platform services.
• Experience with API testing tools such as Postman, Newman, or equivalent.
• Experience with programming languages such as Python or Java.
• Experience with CI/CD tools like GitHub Actions, TeamCity.
• Experience with Agile/Scrum software development methodologies.
• Excellent communication and collaboration skills.
• Good knowledge of software QA methodologies, tools, and processes.
• Hands-on experience testing Google Gemini models or similar LLMs (GPT, Claude, etc.) is highly valued.
• Direct experience with CX Agent Studio, Dialogflow CX, or comparable conversational AI platforms.
• Knowledge of contract automation frameworks, multi agent testing strategies.
• Experience with AI safety testing, content moderation, and responsible AI practices will strengthen your application.
• Familiarity with customer experience metrics (CSAT, NPS, containment rate) and understanding of voice AI, speech-to-text, and text-to-speech quality assessment is a plus.