Video Skill Testing Guide
Before you submit your video skill for certification, you must perform functional testing. Perform the recommended functional tests manually to test your skill from a customer perspective. Complete the recommended tests that apply to the capabilities that your skill supports.
After you perform functional testing, review the Smart Home and Video Skills Certification Guide for certification guidance.
Verify the minimum configuration
To test the end-to-end flow of your skill, you must complete the minimum configuration for a smart home skill. Use the left-hand navigation of the Build page in the developer console to navigate to the configuration item.
- In VIDEO, you must define the following items for your smart home skill:
- A valid AWS Lambda ARN service endpoint.
- Each country that your skill supports.
- A skill invocation name for each language that your skill supports.
- In ACCOUNT LINKING, configure the required fields.
To test and publish your skill, you need an account configured for your device cloud and discoverable devices for that account.
Deploy your skill code
After you verify the minimum configuration for the skill in the developer console, deploy your skill code Lambda function. For details, see Create a Lambda Function.
Enable your skill in the developer console
Complete the steps to enable your skill for testing in the developer console.
Response performance requirements
As you complete the functional tests, make sure that your skill adheres to the following performance requirements.
| Call percentage | Latency limit in milliseconds (ms) |
|---|---|
|
50 percent |
50 ms |
|
90 percent |
100 ms |
|
99 percent |
200 ms |
Video skills functional tests
These tests covers video-specific functionality, such as discovering devices, playing content or searching content. The tests are organized by functional area, such as device discover or search or play content
Device discovery
These tests cover user account linking, device pairing and discovery.
Prerequisite: Skill is not enabled, no user account is linked, and no device is paired.
| Test | Expected Results |
|---|---|
| Enable the skill in the Alexa app, open the skill and account link the skill with an account in your system. Pair the skill to a device. | The Alexa app indicates a device was paired. |
Prerequisite: Skill is enabled, a user account is linked and the skill is paired to a device.
| Test | Expected Results |
|---|---|
| Open the skill in the Alexa app and remove the device pairing. Disassociate the user account. | The Alexa app indicates a device pairing was removed. |
Search content
Test the search functionality using different entity types.
Prerequisite: Skill is enabled, a user account is linked, and the skill is paired to a device.
| Test | Expected Results |
|---|---|
| Ask, "Alexa, show me franchise movies" where franchise is the title of a movie franchise. For example, "The Muppets" | Output screen shows list of search results for the requested franchise. |
| Ask, "Alexa, show me sport/league/team games" where sport/league/team is a value such "University of Washington Huskies" | Output screen shows list of search results for the requested sport/league/team. |
| Ask, "Alexa, show me movies with actor" where actor specifies an actor's name. | Output screen shows list of search results for the requested actor. |
| Ask, "Alexa, show me genre movies", where genre is a value such as science fiction or drama | Output screen shows list of search results for the requested genre |
| Ask, "Alexa, find title" where title is a movie or TV show title. For example, "Star Wars: Return of the Jedi" | Shows results for the requested title. |
| Ask "Alexa show me genre movies with actor" where actor is the name of an actor. For example, "show me comedy movies with Tom Hanks". | Search results are shown for the requested genre and actor. For the example, movies could include "Big", but not "The Green Mile". |
| Ask, "Alexa find title movies with actor", specifying a specific title and actor name. For example, "Find Total Recall with Arnold Schwarzenegger" | Search results are shown for the requested actor and title. For the example, the result would be the original 1990 version, but not the 2012 remake. |
| Ask, "Alexa, find genre movie title/partial title", where title/partial title can include the whole title or a word or two from a title. For example, "Find comedy movie Saving" | Search results are shown for the requested genre and title. The example returns "Saving Mr. Banks", but not "Saving Private Ryan" |
| Ask, "Alexa find genre movie title/partial title with actor" specifying at least a part of a content title. For example, "find comedy movie Saving with Tom Hanks" | Search results are shown for the requested genre, partially-matched title and actor. The example returns "Saving Mr. Banks", but not "Saving Private Ryan". |
| Ask, "Alexa find character movie", where character specifies the name of a character in the movie, such as "find Snow White movies". | Search results are shown for the requested character. |
| Ask, "Alexa find character movie title", where character specifies the name of a character in a specific movie title, such as "find the James Bond movie Skyfall". | Search results are shown for the requested character and title. |
| Ask, "Alexa find title", where title is a TV show. For example, "Alexa, find Transparent" | Search results are shown for the requested TV show with the requested title. |
| Ask, "Alexa find title", where title content that needs to be purchased. | Search results include the detail page for the requested title that allows the user to purchase. |
| Ask, "Alexa, find title", where title may include non-video content such as games | Search results are shown for a standard search, which may include app and game results, and is not limited to video content. |
Play content
Test content playback using a variety of search and play entity types.
Prerequisite: Skill is enabled, a user account is linked, and the skill is paired to a device.
| Test | Expected Results |
|---|---|
| Ask, "Alexa, watch franchise movies", where franchise is the title of a movie franchise such as "The Muppets". | A video from the requested franchise begins playing, if applicable. |
| Ask, "Alexa, watch sport/league/team games" where sport/league/team is a value such "University of Washington Huskies" | A video from the requested sport/league/team begins playing, if applicable. |
| Ask, "Alexa, watch episode of show". Where episode provides the season and episode. For example, "Alexa, watch season 2, episode 4 of Mozart in the Jungle" | The requested episode begins playing, if applicable. |
| Ask, "Alexa, watch a genre", where genre is a value such as "comedy" or "action" | A video of the requested genre begins playing, if applicable. |
| Ask, "Alexa, watch an actor", where actor specifies an actor's name. | A video with the requested actor begins playing, if applicable. |
| Ask, "Alexa, watch a genre movie with actor", specifying a genre and an actor's name. For example, "Play a comedy with Tom Hanks" | A video in the requested genre with the requested actor begins playing, if applicable. |
| Ask, "Alexa, watch title with actor", specifying a specific title and actor name. For example, "Find Total Recall with Arnold Schwarzenegger" | Search results are shown for the requested actor and title. For the example, the result would be the original 1990 version, but not the 2012 remake. |
| Ask, "Alexa, watch genre title/partial title" specifying at least part of a content title. For example, "watch the comedy Saving" | A video in the requested genre with a partially-matching title begins playing, if applicable. For the example, valid matches could include "Saving Christmas", but not "Saving Private Ryan". |
| Ask, "Alexa, watch the genre title/partial title with actor" to specifying at least part of a content title, the genre and an actor's name. For example, "watch the comedy Saving with Tom Hanks". | A video of the requested genre, partially-matching title and actor begins playing, if applicable. For the example, valid matches could include "Saving Christmas", but not "Saving Private Ryan". |
| Ask "Alexa watch character movies" where character provides the name of a character that appears in the movie. For example, "watch Snow White movies". | A video with requested character begins playing, if applicable. |
| Ask "Alexa watch character movie title", specifying a character name that appears in a specific movie title, such as "watch James Bond movie Sky Fall". | A video with requested character and title begins playing, if applicable. |
| Ask, "Alexa, watch title" where title is a movie title. For example, "watch Star Wars: Return of the Jedi". | The requested title begins to play, if applicable. |
| Ask, "Alexa, watch title", where title is a TV show. For example, "watch Transparent" | A TV show with the requested title begins to play, if applicable. |
| Ask, "Alexa, watch title", where title is content that needs to be purchased. | Detail page for the requested title is displayed allowing user to purchase, if applicable. |
| Ask, "Alexa, watch title", where title may include non-video content such as games. | Search results are shown for a standard search, which may include app and game results, not just videos |
Change channel
Test changing channels using channel number, name, and affiliateCallSign.
Prerequisite: Skill is enabled, a user account is linked, and the skill is paired to a device.
| Test | Expected Results |
|---|---|
| Ask, "Alexa, go to channel number", where number is a value like 35 or 105. | Video changes to requested channel. |
| Ask, "Alexa, go to channel name", where name is a value like "ABC" or "Food Channel". | Video changes to requested channel. |
| Ask, "Alexa, go to channel name", where name is a provider or location-dependent value like "KOMO" in the Seattle area. | Video changes to requested channel. |
| From the Alexa app, disable the skill, and issue any Alexa video command such as "Change the channel to ABC" | Error message indicating no devices are available to support the command. |
Control playback
Test controlling playback using various commands.
Prerequisite: Skill is enabled, a user account is linked, and the skill is paired to a device.
| Test | Expected Results |
|---|---|
| Ask, "Alexa, watch title" where title specifies a content title. For example, "watch Star Wars: Return of the Jedi". | Video begins to play. |
| Ask, "Alexa, pause" | Playback pauses. |
| Ask, "Alexa, resume" | Playback resumes from where it paused. |
| Ask, "Alexa, stop" | Playback stops. |
| Ask, "Alexa, play" | Playback resumes. |
| Ask, "Alexa, fast forward" | Playback position moves forward. |
| Ask, "Alexa, fast forward two minutes" | Playback position moves forward two minutes. |
| Ask, "Alexa, rewind" | Playback position moves back. |
| Ask, "Alexa, rewind 10 seconds". | Playback position moves back 10 seconds. |
| Ask, "Alexa, play title season 1, episode 1," where title is a TV series with multiple episodes. | TV series episode 1 begins to play within 10 seconds. |
| Ask, "Alexa, next" | TV series episode two begins to play. |
| Ask, "Alexa, previous" | TV series episode one begins to play. |
Record video
Test video recording by using a variety of search and play entity types.
Test results may vary depending on whether the content is available in your content library. You should provide the best match(es) possible or provide feedback to the user that indicates no matches are available.
Prerequisite: Skill is enabled, a user account is linked, and the skill is paired to a device.
| Test | Expected Results |
|---|---|
| Ask, "Alexa, record the league/team game" where league/team is a value, such as "University of Washington Huskies". | A video from the requested league/team begins recording or gets scheduled for recording, if applicable. |
| Ask, "Alexa, record episode of show". Where episode provides the season and episode. For example, "Alexa, record Bosch season 2, episode 4". | The requested episode begins recording or gets scheduled for recording, if applicable. |
| Ask, "Alexa, record title" where title is a movie or TV show title. For example, "record Star Wars: Return of the Jedi". | A movie or TV show with the requested title begins recording or gets scheduled for recording, if applicable. |
| Ask, "Alexa, record this" while a video is playing. | Begin recording the video that's currently playing. |
| Ask, "Alexa, stop recording" while a video is playing and being recorded. | Stop recording the video that's currently playing. |
| Ask, "Alexa, stop recording title", where title is a movie or TV Show. For example, "stop recording Star Wars: Return of the Jedi". | The requested title stops recording. |
| Ask, "Alexa, cancel recording of next week's episode of title", where title is a TV show. For example, "Alexa, cancel recording of next week's episode of The Grand Tour". | The scheduled recording for the requested title is canceled. |
| Ask, "Alexa, delete the recording of title", where title is a movie or TV show. For example, "Alexa, delete the recording of The Grand Tour". | The recorded video for the requested title gets deleted from the DVR library or storage. |
Launch app and shortcut navigation
Test app launch and shortcut navigation by using the target app or shortcut's name and identifier.
Prerequisite: Skill is enabled, a user account is linked, and the skill is paired to a device.
| Test | Expected Results |
|---|---|
| Ask, "Alexa, launch appName", where appName is the name of an application. For example, "Alexa, launch Twitch". | The "Twitch" application is launched. |
| Ask, "Alexa, launch appName app", where appName is the name of an application. For example, "Alexa, launch Prime Video app". | The "Prime Video" application is launched. |
| Ask, "Alexa, go to shortcut", where shortcut is a shortcut location supported on the user interface. For example, "Alexa, go to settings". | The settings home page is presented. |
Additional tests
In addition, you should perform some of the same tests that are applicable to custom skills. Following are links to functional tests for custom skills that are also relevant to video skills.
Policy testing
Test your skill to make sure that its content meets Amazon's policy guidelines. For more details, see Policy Requirements.
Security testing
Test the Lambda portion of your skill to make sure that it meets Amazon's security requirements. For more details, see Security Testing for an Alexa Skill.
Example phrases
To pass skill certification, you must provide example phrases. Use the sample utterances in the documentation for the Alexa interfaces that your skill supports. Example phrases should be grammatically correct and correctly punctuated. Also, make sure:
- Your first example phrase starts with the wake word "Alexa"
- Your example phrases don't include symbols, variables, placeholders, instructions, or punctuation other than a comma after the wake word or a question mark or period at the end of the phrase.
Skill description and detail page
Test the description that teaches users how to use your skill. For more details, see Skill description and detail page tests.
Account linking
Make sure that you have a test account in your system for account linking. Verify that your skill implements the solution described in Account Linking for Smart Home and Other Domains. Use the Test Instructions to provide test credentials for the device cloud that your skill interacts with.
Test environment
To certify your video skill, the certification team requires access to compatible video equipment. You can use an application like Slingbox to provide the certification team remote access to a compatible set-top box. When you submit the skill, provide a skill test account and a valid set of credentials for Slingbox (or similar application) in the Testing Instructions so the certification team can verify account linking and full functionality of your skill.
In addition, you must make sure that the device cloud for the test account contains at least one discoverable device for testing purposes. In the testing instructions, include test credentials for each language version of your skill.
Duplicates of samples or templates
Make sure that your skill isn't an exact duplicate of any of the Amazon-provided samples or templates. For more details, see Duplicates of samples or templates tests.
Provide content in each language
A skill offered in multiple languages must provide an excellent experience for users that speak those languages. Test to verify that your skill works in its target languages and that you provide the following Store Preview information for each target language in that language:
- One sentence description
- Detailed description
- Example phrases
- Small and large icons
- Keywords
- Privacy questions
- Testing instructions
In addition, Amazon recommends that you provide the following in the skill's target language:
- Any supporting pages, such as Privacy Policy or FAQ
Support for the Alexa APIs varies by region. Make sure that these items don't reference functionality that isn't available for that language version of the skill. For example, if you create a Japanese version of your skill, the Japanese descriptions, phrases and support pages should discuss features that are available in the Japanese skill only. For details about providing a full skill description in the Provide a detailed full skill description.
Related topics
Last updated: May 01, 2024