best practice in chatbot testing

A Chatbot allows consumers to communicate with it via SMS, messenger applications and/or voice.

A successful Chatbot is based on customer centric requirements which reflect and predict customers’ needs. These needs should then be communicated in a direct and concise language which may reflect a pre-determined chatbot personality. All of which should result in a positive customer experience.

The World Quality Report 2017–2018, released by Capgemini, Sogeti, and Micro Focus, identifies several trends in software quality and testing for chatbots.

Here we examine key techniques and best practices that are transforming the QA industry specifically in the way chatbot testing needs to be planned and implemented – effectively and efficiently.

 

1. Chatbot Specific Testing

For a great chatbot, usability comes first.

Here are some of the top features to test for a new chatbot to create an engaging experience and make users come back for more.

• Interaction: Most users, even when they are aware they are talking to a chatbot, have the tendency to treat the app as a real person due to the novelty of this technology and our brains being accustomed to human interaction. Hence testing should be focused on the consistency of keeping the same voice over the entire conversation. Testing should also be performed in high-stress conditions to detect the system’s limits.

 

• Answering: Following the defined testing methodology, the chatbot should perform according to the input table. Testing should detect any ambiguous commands or duplicate keywords. Testing should play with different inputs and variations of the same input to identify the system’s ability to understand.
• Navigation:Compared to a website or an app, a chatbot is less intuitive regarding navigation, and the user’s wish to go back to a previous step could create frustration and quick abandonment. Hence test if users can change their selected topic, start over, or look for something else.

• Handling Errors:  Chatbot needs to understand when the user made a mistake or to get the frustration arising from not getting the answer they are looking for. Test Analyst’s need to define different types of errors such as no answer in the database, wrong topic, invalid response and so on.
• Agile and Continuous Testing:The minimum viable product can be enriched during each iteration with new phrases captured by the error management functions. To ensure no bugs are crawling into the bot, perform testing at each iteration. While in the first stages manual testing ensures business logic, in later phases automation can save time and help developers and QA teams get new and improved versions to market.

 

2. Define Chatbot Testing Methods

Testing a chatbot should address each of its components below:

• starting with an input,
• the knowledge base,
• intelligence and reasoning,
• taking into consideration the infrastructure where the bot is hosted,
• other premises like connectivity and voice communication.

To test usability, it is suggested to create a list of possible user inputs, together with the chatbot’s expected answer as well as potential problems such as alternative spellings or misspelling to ensure it still produces the same correct outcome or ask for further clarification. There can be an upfront disclaimer about the chatbot’s abilities, limitations and preferred way of interaction for best results. 

3. Define Metrics

All testing should be based on clear expectations defined by metrics. Having clear KPIs can speed up the development process significantly.

Possible useful parameters include:

• the number of steps to perform a specific request,
• the percentage of returning visitors,
• the average time spent by the user in one session
• retention rates
• click-through rates
• handling of confusion

Best Practices

Recommended best practices include:

• Agile and Continuous Testing
• Understanding the Chatbots capability and requirements
• Test scenarios encompassing conversation and voice testing i.e., the ability to recognize speech patterns and interpret non-verbal cues.
• Test scenarios should be designed with variations of the same input. Scenarios to handle multiple instructions in a request, conversations with background noises, different dictions, and localization needs are a must.
• Tests to validate the ability of the chatbot to help in user navigation and the ability to handle errors are also vital.
• Omni channel compatibility tests to ensure the same look and feel and responses are required if the chatbot is expected to be used across multiple channels.
• Performance testing i.e., the speed at which the chatbot responds
• Security testing including authentication, authorization, encryption of conversations, and adherence to compliance are key.

Sogeti uses proven methodologies to prove the correct functional behavior of software.

Contact Sogeti  to learn how to approach and test chatbots, and use our cognitive and smart solutions to make your testing processes faster, smarter and more efficient than ever before.