Test the Design of the Interaction Model for Your Skill

Once a skill has been built, test it to ensure the best customer experience. By testing your skill before it goes live, you can help ensure that the skill has good speech recognition accuracy, that requests go to the correct intent and slot value, and that the skill responds appropriately to each request.

Recommendations for testing your skill

  • Test your invocation name for any speech recognition accuracy concerns. Consider changing the name if Alexa frequently fails to recognize it. For more information about invocation names, see Choose the Invocation Name for a Custom Skill.
  • Test your sample utterances to ensure the recognition is accurate and the correct intent and slots are triggered.
  • Test any example phrases provided by the description page and prompts for the skill. Be sure to double check there are exact matching sample utterances for these example phrases in the interaction model.
  • Review the detail page for the skill in the Amazon Alexa App (under the Skills option). Make sure the icon, description, example phrases, and other fields provide meaningful information to help users understand the capabilities you are adding to Alexa.
  • If you can observe other users during testing, note the phrases that they speak to invoke each intent and update the sample utterances accordingly.

Tools available for testing

The Alexa Skills Kit provides a range of tools for testing, such as:

  • Alexa Simulator - The Simulator allows utterances to be input by text or voice, and returns the response from the skill. The tool also shows the information that is sent to and from the skill, to aid in troubleshooting. This is best utilized to test the full customer experience that your skill provides. Please be aware of these Alexa Simulator Limitations. For details about testing on the developer console, see Test Your Skills page.

  • Natural Language Understanding (NLU) evaluation tool - In the skill builder tool we can create a list of utterances with intents and slots we would expect to be invoked for each utterance. An NLU Evaluation can then be run on these utterances to determine how well the model of the skill performs against your expectations. More information can be found on the NLU Test page.

  • Utterance Profiler - Utterance Profiler allows you to quickly test what intent and slots a given utterance will route to. It is independent of the skill code, so you can begin testing your interaction model before developing the backend. When an utterance does not invoke the right intent, sample utterances can be updated and retested, all before writing any code for the skill. More information can be found on the Test Your Utterances as You Build Your Model page.

  • Intent History - Intent History provides a list of frequent requests being made of the skill, listing which intent and slot was hit, and what action was triggered from the skill as a result. If a frequent request is not resolving in a way that’s expected, the tool allows you to quickly map the request to the desired intent, so all future instances of that request will be directed to the expected intent and trigger the correct action.

    As the Intent History tool displays aggregated, anonymized frequent utterances and the resolved intents, use it to look for utterances in the Intent History that were matched to incorrect intents, or appear to have been matched with only LOW or MEDIUM confidence. These utterances are likely to be missing from the interaction model of the skill, so consider adding them (and slight variations of them) to relevant intents.

    The Intent History tool lists the frequency users have said certain utterances from most-said to least-said (information on the exact number of times the utterance has been directed to the device is not available). Use this information to focus on the most common utterances with friction to provide a better user experience. More information can be found on the Review Intent History page. You can also visit Tips for Using Intent History to Enhance Your Alexa Skill and Discover How Customers Engage With Your Alexa Skill Using Intent History.

  • ASK CLI test tools - The Alexa Skills Kit Command Line Interface (ASK CLI) is a tool used to manage Alexa skills and related resources. Use the ask run command to start a session to test your skill.

    The Alexa Skills Kit Command Line Interface (ASK CLI) can be used to interact with any of these APIs.

    With these skill testing features in the Skill Management API (SMAPI), the skill can be tested programmatically rather than using the Alexa Skills Kit Developer Console.

    More information can be found on the Quick Start Alexa Skills Kit CLI and Skill Testing Operations pages.

Making use of device testing

Test on devices that your customers are likely to use, as this is the best way to see what their experience would be like. Device testing also provides additional benefits not offered by the Alexa Simulator. For instance, this allows you to test the accuracy of speech recognition, NLU interpretation accuracy and the responses coming from the skill. Device testing is also recommended for any skill that uses Alexa Presentation Language. For details, see Alexa Simulator Limitations.

Making use of beta testing

The skill beta testing tool can be used to test an Alexa skill in beta before releasing it to production. The skill beta testing tool can also be used to test changes to an existing skill, while still keeping the currently live version of the skill available for the general public. You can invite friends or family, your social network contacts, or other people for whom you have an email address to test your skill and provide feedback. Using Beta testing you can get enough interactions with a skill to get an Intent History. This helps get an idea of how the ASR works, get feedback from users about what worked or didn't work. See Beta Testing.

Was this page helpful?

Last updated: Mar 20, 2024