Music, Radio, and Podcast Skill Testing Guide

Note: Sign in to the developer console to build or publish your skill.

Important: Anyone can build a music skill for public distribution in the United States. However, radio and podcast skills are currently in developer preview. To register your radio or podcast skill for the preview, contact your Alexa Music, Radio, or Podcast representative.

Before you submit your music, radio, or podcast skill for certification, you should test your skill. To test your skill, you can do any or all of the following:

Run validation tests
Perform functional tests
Run a beta test

We recommend that you test your skill in all of these ways before you submit it for certification.

Run validation tests
Perform functional tests
Run a beta test
Additional tests
Podcast customer experience overview

Run validation tests

You can use the Alexa Skill Validation API to help determine if your skill's AWS Lambda function correctly supports the music and radio skill APIs. The API runs several tests and produces an overall status of either SUCCESSFUL or FAILED. To pass certification, your skill must have an overall status of SUCCESSFUL. For more information, see supported validations.

You can use the validation API via the Alexa Skills Kit Command Line Interface (ASK CLI). For more information, see validate in the ASK CLI command reference.

When the validation is complete, you receive the overall status and the results for each test that was run. The number of tests that are run depends on which APIs your skill has implemented.

The following examples show the different results you might receive for each test when you use the validate command in the ASK CLI.

A required test passed

{
    "title": "<test name>",
    "status": "SUCCESSFUL",
    "importance": "REQUIRED"
}

A required test failed

A test can fail for many reasons. The description of a failed test explains the reasons for the failure. To pass certification, your skill must pass all of the required validation tests.

{
    "title": "<test name>",
    "status": "FAILED",
    "importance": "REQUIRED"
}

A recommended test passed

{
    "title": "<test name>",
    "status": "SUCCESSFUL",
    "importance": "RECOMMENDED"
}

A recommended test failed

Your skill can fail a recommended validation test and still pass certification. However, in this case some APIs might not work as expected and might negatively impact users using your skill. We recommend that you fix all failed validation tests before submitting your skill for certification.

{
    "title": "<test name>",
    "status": "FAILED",
    "importance": "RECOMMENDED"
}

Perform functional tests

Before you submit your music or radio skill for certification, you should perform functional testing to make sure your skill works as you intend.

The following are our recommended functional tests for music skills. You should perform these tests manually to evaluate your skill from the end-user perspective. You should complete the test cases that correspond to features that your skill supports. Your music and radio skills should pass these tests before you submit the skill for certification.

For the "play by …" tests (track, album, artist, stations and so on), make sure to test a variety of items for each catalog type to make sure the items play accurately. During certification, testers try a wide variety of items and expect each one to play accurately.

Test	Utterance	Expected result
Play music	"Alexa, play music on skill name."	Songs play without any additional input from the user.
Play by track	"Alexa, play track name on skill name."	The requested track plays.
Play by album	"Alexa, play the album album name on skill name."	The requested album plays.
Play by artist	"Alexa, play artist name on skill name."	A song by the requested artist plays.
Play by genre	"Alexa, play genre on skill name."	Songs from the requested genre play.
Play by playlist	"Alexa, play the playlist playlist name on skill name."	Songs from the requested playlist play.
Play by station	"Alexa, play station name on skill name."	The requested station plays.
Play popular	"Alexa, play popular songs on skill name."	A playlist of popular songs plays.
Play latest	"Alexa, play the latest songs on skill name."	A playlist of the latest songs plays.
Pause	(When music is playing) "Alexa, pause."	Music pauses.
Resume	(After pausing music) "Alexa, resume."	Music resumes playing.
Stop	(When music is playing) "Alexa, stop."	Music stops playing.
Next	(When music is playing) "Alexa, next."	The next song in the queue plays.
Previous	(When music is playing) "Alexa, previous."	The previous song in the queue plays.
Loop	(When a queue is playing) "Alexa, loop."	After the final track in the queue finishes, the queue plays again beginning with the first item from the queue.
Repeat	(When music is playing) "Alexa, repeat."	After the current songs finishes, the same song plays again.
Shuffle	(When music is playing) "Alexa, shuffle."	The order of songs in the queue changes randomly.
Accept positive feedback	(When music is playing) "Alexa, I like this song."	The skill accepts the positive feedback gracefully.
Accept negative feedback	(When music is playing) "Alexa, I don't like this song."	The skill accepts the negative feedback gracefully.
Play continuously for a long duration	"Alexa, play [station name or playlist name or genre or artist name] on skill name."	Music plays for a long duration (at least 15 minutes).
Explicit language filter	(After enabling the explicit language filter) "Alexa, play the song name of a song with explicit lyrics on skill name."	The clean (not explicit) version of the requested song plays, if one is available. If a clean version isn't available, no content plays.

Run a beta test

You can set up and run a beta test to learn about your skill's user experience from real users while the skill is still in development, before you submit it for certification and make it available to the general public. Your beta testers can include your friends or family, social network contacts, existing users, or anybody for whom you have an email address and that is willing to test your skill and provide feedback. You can invite up to 500 beta testers per skill. At any time you can add testers, remove testers, or end a test. You can see tester feedback in the Alexa Skills Kit developer console, and can view feedback throughout the testing cycle.

For more details about setting up and running a beta test for your skill, see Skill Beta Testing for Alexa Skills.

Additional tests

In addition to the tests described in the preceding sections, you should perform some of the same tests that are applicable to custom skills. Following are links to functional tests for custom skills that are also relevant to music and radio skills.

Policy guidelines

Make sure your skill's content meets Amazon's policy guidelines. For more details, see Policy Requirements.

Security requirements

Test your skill code (Lambda function) to make sure it meets Amazon's security requirements. See Security Testing.

Skill description and detail page

Check the description that teaches users how to use your skill. See Skill Description and Detail Page in the functional testing guide for custom skills.

Account linking

Make sure you can successfully account link your skill. See Account Linking in the functional testing guide for custom skills.

Duplicate of samples or templates

Make sure your skill isn't an exact duplicate of any of the Amazon-provided samples or templates. See Duplicates of Samples or Templates in the functional testing guide for custom skills.

Podcast customer experience overview

The following sections detail the end-to-end expected customer experience for podcast skills, starting from a GetPlayableContent request and ending with the end of a customer's listening session on Alexa.

Note: The prompt examples might not be exact, because prompts are subject to change.

Basic playback and state persistence

Utterances:

"Alexa, play <programName> (program/podcast) (on <provider>)."
"Alexa, play my podcast."
"Alexa, play <provider>."
"Alexa, resume my podcast."
"Alexa, resume <provider>."

#	Condition	Prompt example (TTS)	Customer experience
1.1	Program found without a resume marker	"Playing <programName> from <provider>. Here's (the latest episode) <episodeTitle>."	First program (by publish date) plays for serialized program series. Latest on demand program (by publish date) plays for episodic program series.
1.2	Program found with a resume marker	"Playing <programName> from <provider>. Resuming (the latest episode) <episodeTitle>."	Program resumes from the most recent resume marker recorded.
1.3	Program not found	"Hmm, I didn't find <programName>."	Media resumes if playing, else, nothing
1.4	Error in program playback	"Sorry, I'm not able to play that podcast right now."	Media resumes if playing, else, nothing
1.5	Request is to "resume / play "	"Here's <provider>."	Resume media last played on provider
1.6	Request is to "resume podcast" and there is nothing to resume	"Hmm, I'm not sure."	Media resumes if playing, else, nothing

Continuous play

#	Condition	Customer experience
1.1	Episodic program series plays and isn't interrupted by the customer	Programs are played in reverse chronological order. Programs that have been completed are skipped over and programs that contain a resume marker are resumed.
1.2	Serialized program series plays and isn't interrupted by the customer	Programs are played in chronological order. Programs that have been completed are skipped over and programs that contain a resume marker are resumed.

Variable Seek

Utterances:

"Alexa, fast-forward / rewind <time>."
"Alexa, skip ahead / back <time>."
"Alexa, go forward / back <time>."

#	Condition	Prompt example (TTS)	Customer experience
1.1	Time seek successful	Success earcon (chime)	Program seeks requested time period and plays
1.2	Not enough time left in the program to fast-forward the requested amount.	"There's only <time> left in this episode."	Program resumes
1.3	Not enough time left in the program to rewind the requested amount.	"Restarting <episodeTitle>."	Program restarts

Restart a Program

Utterances:

"Alexa, restart / start over."
"Alexa, restart this episode / podcast."
"Alexa, restart <programName>."
"Alexa, play / restart this (episode) from the beginning."

#	Condition	Prompt example (TTS)	Customer experience
1.1	Program is playing and restart is allowed	"Restarting <episodeTitle>."	Program restarts

Play the latest episode

Utterances:

"Alexa, play the latest episode of <programName>."
"Alexa, play the latest (episode)."
"Alexa, Play the most recent (episode)."

#	Condition	Prompt example (TTS)	Customer experience
1.1	Latest program is found and isn't playing	"Playing <ProgramName> from <provider>. <Here's / Resuming> the latest episode, <episodeTitle>."	Latest program (by publish date) plays / resumes
1.2	Latest program is found and is playing	"You're currently playing the latest episode."	Latest program resumes

Play the latest season

Utterances:

"Alexa, play the latest season of <programName>."
"Alexa, play the most recent season."

#	Condition	Prompt example (TTS)	Customer experience
1.1	Latest season is found and isn't playing	"Playing <ProgramName> from <provider>."	Latest season (by publish date) plays / resumes

Episode Search by Timeframe

Utterances:

"Alexa, play <programName> (program/podcast) from <timeframe/date> (on <provider>)."
"Alexa, play <timeframe/date> episode/podcast/program of <programName> (on <provider>)."

#	Condition	Prompt example (TTS)	Customer experience
1.1	Program found	"Okay. Playing <programName> from <provider> . <Here's / Resuming> episode, <episodeTitle>."	Within the date range provided, the provider to determine what program plays. If multiple matches returned, the provider to determine the best match. Our recommendation is to play / resume the latest program (by publish date) of the program returned within the provided timeframe.
1.2	Program not found	"Hmm, I didn't find an episode from that date, but you can ask me to play the latest."	Media resumes if playing, else, nothing

Episode Search by Topic

Utterances:

"Alexa, Play (the) <programName> (episode) <qualifier> <topic> (on <provider>)."
"Alexa, Play (the) <programName> podcast/program (episode) <qualifier> <topic> (on <provider>)."
"Alexa, play me something <qualifier> <topic> on <programName> (on <provider>)."
"Alexa, play the <topic> interview on/in <programName> (podcast/program) (on <provider>)."
"Alexa, play <topic> on <programName> (podcast/program) (on <provider>)."

#	Condition	Prompt example (TTS)	Customer experience
1.1	Program found	"I found an episode of <programName> from <provider> that you might like. <Here's / Resuming> the <releaseDate> episode, <episodeTitle>."	Provider to determine what program plays. If multiple matches returned, Provider to determine the best match.
1.2	Program not found	"I didn't find an episode about <topic> on <programName>, but you can ask me to play the latest."	Media resumes if playing, else, nothing

Episode Search by Season and Episode Number

Utterances:

"Alexa, play episode <#>."
"Alexa, play episode <#> of <ProgramName>."
"Alexa, play season <#>."
"Alexa, play season <#> of <ProgramName>."
"Alexa, play season <#>, episode <#>."
"Alexa, play season <#>, episode <#> of <ProgramName>."
"Alexa, play the first episode of <programName>."

#	Condition	Definition	Prompt example (TTS)	Customer experience
1.1	An exact match is found	The customer's request matches all slots defined in the RSS feed for `<itunes:season>` and/or `<itunes:episode>`	"Playing <programName> from <provider>." "<Here's / Resuming> (season <seasonNumber>), episode <episodeNumber>, <episodeTitle>."	One episode matched Program matched plays or resumes if in-progress Season only requested If program number isn't provided, the requested season plays from the most recent resume marker in the season, else, the first published episode for a serialized program series or the most recently published episode for an episodic program series. Multiple matches If multiple episodes are returned, playback; 1. the returned episode nearest the episode currently playing / in focus else; 2. the returned episode nearest to the resume marker found; else: 3. the earliest (by publication date) of the returned episodes for a serialized program series or the latest (by publication date) of the returned episodes for an episodic program series.
1.2	A partial match is found	The customer's request matches the `<itunes:episode>` slot but may be missing the `<itunes:season>` slot or requests "season 1" when the program series doesn't define seasons in the RSS feed. Examples of partial matches include: - The program series has one season with ten programs and the customer asks "play episode 5". While season isn't matched, program 5 is and is expected to play. - The program series has three seasons with ten programs each and the customer asks "play episode 5". While season isn't matched, episode 5 returns three matches and is expected to follow the multiple match scenario in 1.1. - The program series has zero seasons with 200 episodes. The customer asks "play season 1, episode 75". As the program series technically has no seasons, this can be interpreted as a single season. Episode 75 is expected to play.	"Playing <programName> from <provider>. <Here's / Resuming> (season <seasonNumber>), episode <episodeNumber>, <episodeTitle>."	Follow the customer experience outlined in 1.1 for partial matches.
1.3	No match is found	The customer's request doesn't match the slots defined in the RSS feed for `<itunes:season>` and/or `<itunes:episode>`	"I only found <#> episode(s) (in season <seasonNumber>). I'll <play / resume> (season <seasonNumber>), episode <episodeNumber> of <programName> from <provider>. <episodeTitle>."	Season requested Play nearest season to the one requested. If there is a resume marker in that season, resume playback from the most recent resume marker in that season; else: - Play from the latest episode published for episodic program series - Play from the first episode published for serialized program series Episode only requested Resume playback from the most recent resume marker found in the program; else: - Play from the latest episode published for episodic program series - Play from the first episode published for serialized program series
1.4	Season and episodes aren't defined in the RSS feed	`<itunes:season>` and `<itunes:episode>` tags aren't found in the RSS feed	"Hmm, I didn't find any seasons or episodes by number for <programName>. But you can ask me for the next or previous episode."	If the program series is playing, the program in focus resumes. If the program series isn't playing, the most recent in-progress program resumes; else: - Play the latest program published for episodic program series - Play the first episode published for serialized program series
1.5	Feature not supported by provider	Provider hasn't implemented the search by episode and season number feature	"Searching by season and episode number isn't supported on <serviceName> right now."	If the program series is playing, the program in focus resumes. If the program series isn't playing, the most recent in-progress program resumes; else: - Play the latest program published for episodic program series - Play the first program published for serialized program series

Subscription playback

Utterances:

"Alexa, play my subscribed podcasts on <provider>."
"Alexa, resume my subscribed podcasts on <provider>."
"Alexa, play the latest from my followed podcasts."

Condition	Customer experience
Customer has episodes to play from their subscription playlist	"Okay. I've lined up the episodes you haven't heard, from your followed podcasts on provider. Navigate by asking for the next podcast. Now playing (or resuming) the episode, programName.episodeTitle."
Next/Previous navigation	"Playing programName.episodeTitle."
Customer hasn't subscribed to any podcasts	"To play the latest episodes from the podcasts in your library, you'll first need to follow podcasts in your provider app."
No new episodes to play due to prior consumption	"You're up to date with all the episodes from the podcasts you're subscribed to."
Feature not yet supported by provider	"Provider does not support your podcast library right now, but you can ask for a podcast by name."
No content found for provider	"Provider didn't find any episodes in your podcast library."

Was this page helpful?

Provide feedback

Last updated: Nov 27, 2023