Alexa.UIController Interface (VSK Fire TV)

The UIController interface takes into account your app's UI state in order to more intelligently respond to user requests. To use the UIController interface, you need to send Alexa your UI state, listing all elements on the page and the elements in focus. These details are then included in the directives that Alexa sends to your Lambda.

If you're implementing UIController, consider also implementing MediaDetailsNavigator, which is another Alexa interface that uses the UI state report to provide directives for "show more details about video title" requests.

Note: The UIController interface supports only cloudside integrations.

Detailed Explanation
UIController versus KeypadController
Supported Utterances
Sending UI State Reports
- VSK-Enabled Apps
  - Step 1 - Add the UI Controller to Alexa Client Library Supported Capabilities
  - Step 2 - Send UI Reports through Alexa Client Library
- VSK-Enabled Devices
Data Structures
- Schema for uiElements versus focusedUIElements
Payload Definitions
- Enum<EntityTypes>
- Enum<uiSupportedActions>
ActionOnUIElement Directive Example
Examples of UI State Reports and UIController Directives
Sample App Integration
MediaDetailsNavigator

Detailed Explanation

For Alexa to more intelligently respond to user's requests, Alexa needs to know what elements the customer sees on the screen and what actions are available for each element. This information is known as the UI state. A UI state has two different dimensions to describe what's shown on the UI — the list of elements shown on the screen (which is reported in the uiElements section of the state report), and the element in focus (which is reported in the focusedUIElement section).

Tip: The "focused element," as it's sometimes called, refers to the currently highlighted or selected UI element on screen. When this element is highlighted/selected, it's considered to be "in focus."

To let Alexa know your UI state, you (as the app developer) first send a UI report to Amazon (posting the event as a ChangeReport to a specific Alexa gateway endpoint) so that Alexa can dynamically target different screen elements. For example, after launching your app, your app would send a structured list of UI elements on screen (including names, aliases, position, and supported actions for those objects). You send your UI state to the Alexa endpoint whenever your UI changes.

When customers make requests related to on-screen UI navigation or selection (e.g., "Alexa, play number three," "Alexa, select continue watching," "Alexa, play harry potter," "Alexa, play this" ), Alexa sends an ActionOnUIElement directive from the UIController interface to your Lambda. The directive contains specific details from the UI state that you previously sent to Alexa. For example, if you have a title on the screen called "Popcorn Movie" in the second position in a list, the UIController interface sends a directive with those details to your Lambda.

Important: In addition to navigation and selection requests, the UI state will also be used to perform non-UI-navigation actions such as "Watch video title," "Launch video title," and so on through the corresponding capability interfaces such as RemoteVideoPlayer: SearchAndPlay. In other words, the UIController interface doesn't always send ActionOnUIElement directives. Sometimes it sends other directives such as RemoteVideoPlayer: SearchAndPlay as appropriate. As a result, the Supported Utterances section below also includes the utterances for SearchAndPlay Directives as well.

Overall, the UIController interface lets customers interact in a richer way with voice in your app's UI, such as selecting their profile by voice or selecting a specific item on the screen by voice.

UIController versus KeypadController

UIController differs from KeypadController in that UIController considers your UI state and includes details from the UI state in the directive it sends. In contrast, KeypadController is stateless and doesn't receive any specific details about your app's UI. As a result, UIController is a lot more powerful in the types of utterances it can support.

You can use both KeypadController and UIController together. UIController will take precedence over KeypadController whenever the UI state is available. If the UI state is not available, KeypadController will be used as a fallback. If you don't have KeypadController implemented, the fallback would be the in-app voice scrolling and selection functionality that is built-in to Fire TV.

Supported Utterances

Alexa sends an ActionOnUIElement directive to your Lambda when users make a navigation or selection request related to an element on the screen. Unlike the other utterances, because UIController has access to your UI state, there aren't a rigid set of supported utterances. The supported utterances and features can be dynamically controlled based on the on-screen context you provide in your UI state report (described below). The utterances your app supports depend on the elements you provide in your UI state reporting.

For example, if you describe an element called "Jake's watchlist" as an ordinal in the third position, then Alexa would support utterances such as "Go to Jake's watchlist" or "select the third one." (Ordinals are identifiers that indicate position, such as number one, number two, etc.)

Depending on the detail in your state reporting, the following utterances and scenarios can be supported:

You can scroll a list by saying, "Scroll/Page right/left/up/down." (Note: "Scroll back" is not supported.)
You can see more/all the titles that are not visible in the shortlist by saying "Show/See more." (Note: "Show/see all" is not supported.)
You can select an item from the list by saying "Select/Pick/Choose this/that/the first one/number one," or ask Alexa to play a title from the list, like "Select/Play/Choose/Watch Mockingjay."
You can also use your voice to emulate a d-pad controller on your remote by saying "Go right/left/up/down" followed by "Select/Choose/Watch/Play."
On a video detail page, you can take the following actions showing on the screen: "Add to watchlist," "Watch Now," "Watch Trailer," etc.
You can navigate to any UI location, such as recently watched, watch lists, recommended videos.
You can respond to a dialog screen and pick an option without needing to reach for the remote control.

Tip: To support requests such as "Show more details about video title", you need to implement another interface called MediaDetailsNavigator.

For specific examples of utterances in multiple locales, see the following list. These utterances include utterances from KeypadController and RemoteVideoPlayer: SearchAndPlay.

Feature	Sample Utterances	Expected Response
Watch this Required for certification	Watch this	The media in focus begins to play
Play this Required for certification	Play this	The media in focus begins to play
Directional navigation - go Required for certification	Go up go down go left go right	The app goes to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional navigation - move Required for certification	Move up move down move left move right	The app moves to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional scrolling Required for certification	Scroll up scroll down scroll left scroll right	The app scrolls left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional pagination Required for certification	Page up page down page left page right	The app pages left, right, up, or down, similar pressing the same direction on the D-pad controller.
Verbless directional navigation Required for certification	Up Down Left Right	The app shifts left, right, up, or down, similar pressing the same direction on the D-pad controller.
Select/choose this (that) Required for certification	Select this	Whatever is in focus gets selected.
Show/see more Optional	Show more	More items are shown. For example, if a list shows only a limited number of items, more will be retrieved and shown.
Show/see details (info) Optional	Show details	More details are shown. For example, if information is truncated, more information are shown.
Get Next Page Required for certification	Next	The next page of results appears on the screen.
Watch [title] Required for certification	Watch breaking bad	The requested media plays. If there are multiple matches for the content, the user is prompted to select the desired media.
Watch by actor Required for certification	Watch sean connery	Either media with the actor plays (through an algorithm that selects the best match for the user), or a list of results is presented to the user.
Watch by genre name Required for certification	Watch comedy	Either media from the requested genre plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by franchise [title] Required for certification	Watch star trek	Either the top recommended result for the requested franchise plays, or the user is presented with a list of matching franchises.
Watch by media type Optional	Watch a movie	Either media matching the non-title request plays (through an algorithm that selects a recommended video for the user), or a list of results is presented to the user.
Watch [title] with season number Optional	Watch breaking bad season two	The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with episode number Optional	Watch breaking bad episode five	The correct season is identified by the user's last-watched episode. The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with season and episode number Optional	Watch breaking bad season two episode five	The content corresponding to the requested season and episode plays.
Watch by character name Optional	Watch james bond	Either media matching the character name plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports team Optional	Watch seahawks	Either media matching the sports team plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports type Optional	Watch soccer	Either media matching the sports type plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports league Optional	Watch n. b. a. league	Either media matching the sports league plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports event Optional	Watch soccer game	Either media matching the sports event plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Launch [partner] app Required for certification	Launch netflix app	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Launch special character title Required for certification	Launch a. t. and t. u-verse app	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Go to control settings	Go to control settings	The view changes to the control settings.
Go home Optional	Go home	The view changes to the home screen.
Close Optional	Close [3P App] e.g. Close Netflix	The app or skill closes.
Exit Optional	Exit [ 3p App] Exit Netflix	The app or skill is exited.
Quit Optional	Quit [ 3p App] Quit Netflix	The app or skill is quit.
Open Optional	Open	The app or skill opens.

Feature	Sample Utterances	Expected Response
Watch this Required for certification	Not supported	The media in focus begins to play
Play this Required for certification	joue ça	The media in focus begins to play
Directional navigation - go Required for certification	va vers le haut monte vers le haut remonte va en bas descends faire défiler vers le bas va à gauche aller à gauche va à droite aller à droite	The app goes to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional navigation - move Required for certification	va vers le haut monte vers le haut remonte navigue en bas navigue vers le bas Naviguer à gauche Naviguer vers la gauche Naviguer à droite Naviguer vers la droite	The app moves to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional scrolling Required for certification	fais défiler vers le haut fais défiler en haut faire défiler vers le bas défiler vers le bas se déplacer vers le bas fais défiler à gauche faire défiler à gauche défiler à gauche se déplacer à droite fais défiler à droite faire défiler à droite défiler à droite se déplacer à droite	The app scrolls left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional pagination Required for certification	page vers le haut Page vers le bas Page vers la gauche Page vers la droite	The app pages left, right, up, or down, similar pressing the same direction on the D-pad controller.
Verbless directional navigation Required for certification	va en haut monte faire défiler vers le haut va en bas descends faire défiler vers le bas fais défiler à gauche va à gauche faire défiler à gauche se déplacer à gauche aller à gauche défiler à gauche fais défiler à droite va vers la droite fais défiler vers la droite déplace-toi vers la droite aller à droite défile à droite	The app shifts left, right, up, or down, similar pressing the same direction on the D-pad controller.
Select/choose this (that) Required for certification	sélectionne ça choisis ça	Whatever is in focus gets selected.
Show/see more Optional	montre moi en plus montre en plus encore	More items are shown. For example, if a list shows only a limited number of items, more will be retrieved and shown.
Show/see details (info) Optional	montre des détails montre moi des détails voir plus d'infos	More details are shown. For example, if information is truncated, more information are shown.
Get Next Page Required for certification	prochain(e) suivant(e) next	The next page of results appears on the screen.
Watch [title] Required for certification	mets / joue / lance / pars [VideoName] sur [AppName]	The requested media plays. If there are multiple matches for the content, the user is prompted to select the desired media.
Watch by actor Required for certification	regarde un film avec [actor] mets un film avec [actor] joue un film avec [actor] pars un film avec [actor]	Either media with the actor plays (through an algorithm that selects the best match for the user), or a list of results is presented to the user.
Watch by genre name Required for certification	regarder un [GenreName] regarder une [GenreName] mets un [GenreName] mets une [GenreName] joue un [GenreName] joue une [GenreName	Either media from the requested genre plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by franchise [title] Required for certification	regarde [title] mets [title] joue [title] pars [title]	Either the top recommended result for the requested franchise plays, or the user is presented with a list of matching franchises.
Watch by media type Optional	regarde un film mets un film joue un film pars un film	Either media matching the non-title request plays (through an algorithm that selects a recommended video for the user), or a list of results is presented to the user.
Watch [title] with season number Optional	regarde la saison [x] de [title] mets la saison [x] de [title] joue la saison [x] de [title] pars la saison [x] de [title] regarde [title] saison [x] mets [title] saison [x] joue [title] saison [x] pars [title] saison [x] joue la saison [x] de [title]	The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with episode number Optional	regarde l'épisode [x] de [title] mets l'épisode [x] de [title] joue l'épisode [x] de [title] pars l'épisode [x] de [title] joue [title] épisode [x] mets [title] épisode [x]	The correct season is identified by the user's last-watched episode. The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with season and episode number Optional	Same as fr-FR	The content corresponding to the requested season and episode plays.
Watch by character name Optional	Not supported	Either media matching the character name plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports team Optional	Regarde le [team] Regarde le match de [team]	Either media matching the sports team plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports type Optional	Not supported	Either media matching the sports type plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports league Optional	Mets [league] Joue [league] Regarde [league]	Either media matching the sports league plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports event Optional	Not supported	Either media matching the sports event plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Launch [partner] app Required for certification	pars [AppName]	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Launch special character title Required for certification	pars [AppName]	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Go to control settings	vas à la configuration vas aux paramètres vas à la configuration des contrôles vas aux paramètres des contrôles ouvres la configuration ouvres les paramètres ouvres la configuration des contrôles ouvres les paramètres des contrôles	The view changes to the control settings.
Go home Optional	retourne au home page affiche-moi l'écran d'accueil retourne à l'écran d'accueil affiche l'accueil affiche-moi l'accueil ouvre l'accueil va à l'accueil affiche la page d'accueil retourne à la page d'accueil rentre à l'accueil accueil	The view changes to the home screen.
Close Optional	Not supported	The app or skill closes.
Exit Optional	sors de là	The app or skill is exited.
Quit Optional	quitte [AppName]	The app or skill is quit.
Open Optional		The app or skill opens.

Feature	Sample Utterances	Expected Response
Watch this Required for certification	Not supported	The media in focus begins to play
Play this Required for certification	joue ça	The media in focus begins to play
Directional navigation - go Required for certification	va vers le haut monte vers le haut remonte va en bas descends faire défiler vers le bas va à gauche aller à gauche va à droite aller à droite	The app goes to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional navigation - move Required for certification	va vers le haut monte vers le haut remonte navigue en bas navigue vers le bas Naviguer à gauche Naviguer vers la gauche Naviguer à droite Naviguer vers la droite	The app moves to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional scrolling Required for certification	fais défiler vers le haut fais défiler en haut faire défiler vers le bas défiler vers le bas se déplacer vers le bas fais défiler à gauche faire défiler à gauche défiler à gauche se déplacer à droite fais défiler à droite faire défiler à droite défiler à droite se déplacer à droite	The app scrolls left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional pagination Required for certification	page vers le haut Page vers le bas Page vers la gauche Page vers la droite	The app pages left, right, up, or down, similar pressing the same direction on the D-pad controller.
Verbless directional navigation Required for certification	va en haut monte faire défiler vers le haut va en bas descends faire défiler vers le bas fais défiler à gauche va à gauche faire défiler à gauche se déplacer à gauche aller à gauche défiler à gauche fais défiler à droite va vers la droite fais défiler vers la droite déplace-toi vers la droite aller à droite défile à droite	The app shifts left, right, up, or down, similar pressing the same direction on the D-pad controller.
Select/choose this (that) Required for certification	sélectionne ça choisis ça	Whatever is in focus gets selected.
Show/see more Optional	montre moi en plus montre en plus encore	More items are shown. For example, if a list shows only a limited number of items, more will be retrieved and shown.
Show/see details (info) Optional	montre des détails montre moi des détails voir plus d'infos	More details are shown. For example, if information is truncated, more information are shown.
Get Next Page Required for certification	prochain(e) suivant(e) next	The next page of results appears on the screen.
Watch [title] Required for certification	mets / joue / lance [VideoName] sur [AppName]	The requested media plays. If there are multiple matches for the content, the user is prompted to select the desired media.
Watch by actor Required for certification	regarde un film avec [actor] mets un film avec [actor] joue un film avec [actor]	Either media with the actor plays (through an algorithm that selects the best match for the user), or a list of results is presented to the user.
Watch by genre name Required for certification	regarder un [GenreName] regarder une [GenreName] mets un [GenreName] mets une [GenreName] joue un [GenreName] joue une [GenreName	Either media from the requested genre plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by franchise [title] Required for certification	Regarde [title] Mets [title] Joue [title]	Either the top recommended result for the requested franchise plays, or the user is presented with a list of matching franchises.
Watch by media type Optional	regarde un film mets un film joue un film	Either media matching the non-title request plays (through an algorithm that selects a recommended video for the user), or a list of results is presented to the user.
Watch [title] with season number Optional	Regarde la saison [x] de [title] Mets la saison [x] de [title] Regarde [title] saison [x] Mets [title] saison [x] Joue [title] saison [x] Joue la saison [x] de [title]	The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with episode number Optional	Regarde l'épisode [x] de [title] Mets l'épisode [x] de [title] Joue l'épisode [x] de [title] Joue [title] épisode [x]	The correct season is identified by the user's last-watched episode. The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with season and episode number Optional	Regarde l'épisode [x] de la saison [x] de [title] Joue l'épisode [x] de la saison [x] de [title] Mets l'épisode [x] de la saison [x] de [title]	The content corresponding to the requested season and episode plays.
Watch by character name Optional	Not supported	Either media matching the character name plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports team Optional	Regarde le [team] Regarde le match de [team]	Either media matching the sports team plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports type Optional	Not supported	Either media matching the sports type plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports league Optional	Mets [league] Joue [league] Regarde [league]	Either media matching the sports league plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports event Optional	Not supported	Either media matching the sports event plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Launch [partner] app Required for certification	lance [AppName]	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Launch special character title Required for certification	lance [AppName]	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Go to control settings	ouvre les paramètres de contrôles ouvre les paramètres va aux paramètres	The view changes to the control settings.
Go home Optional	affiche-moi l'écran d'accueil retourne à l'écran d'accueil affiche l'accueil affiche-moi l'accueil ouvre l'accueil va à l'accueil affiche la page d'accueil retourne à la page d'accueil rentre à l'accueil accueil	The view changes to the home screen.
Close Optional	Not supported	The app or skill closes.
Exit Optional	sors d'ici	The app or skill is exited.
Quit Optional	quitte [AppName]	The app or skill is quit.
Open Optional		The app or skill opens.

Feature	Sample Utterances	Expected Response
Watch this Required for certification	das anschauen/angucken	The media in focus begins to play
Play this Required for certification	spiel das ab?	The media in focus begins to play
Directional navigation - go Required for certification	geh/e nach oben nach oben/rauf/hinauf/hoch gehen gehe hoch geh rauf/hinauf nach unten/runter/hinunter gehen geh runter/nach unten/hinunter geh nach links geh nach rechts	The app goes to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional navigation - move Required for certification	nach oben bewegen nach unten bewegen nach links bewegen nach rechts bewegen	The app moves to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional scrolling Required for certification	scroll hoch scroll nach oben scroll rauf scrolle hinauf scrolle hoch scroll hier/dahin/dorthin runter scroll zum letzten runter scroll runter/hinunter/nach unten runter/hinunter/nach unten scrollen scroll nach links scroll links scroll nach rechts scroll rechts	The app scrolls left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional pagination Required for certification	seite nach oben seite rauf seite hinauf seite hoch blätter hoch blätter runter seite runter/hinunter/nach unten blätter links seite nach links seite links blätter rechts seite nach rechts seite rechts	The app pages left, right, up, or down, similar pressing the same direction on the D-pad controller.
Verbless directional navigation Required for certification	nach oben bewegen bätter hoch geh hoch scroll hoch scroll nach oben nach unten bewegen blätter runter scroll hier/dahin/dorthin runter geh runter/nach unten scroll hinunter/nach unten runter/hinunter gehen seite runter/hinunter/nach unten nach links bewegen blätter links seite nach? links scroll nach? links geh nach links nach rechts bewegen blätter rechts seite nach? rechts scroll nach? rechts geh nach rechts	The app shifts left, right, up, or down, similar pressing the same direction on the D-pad controller.
Select/choose this (that) Required for certification	nimm das wähl das aus? klick die an diese	Whatever is in focus gets selected.
Show/see more Optional	mehr ansehen/sehen zeig mir mehr zeig mir mehr davon zeig mir mehr auf [AppName] zeig mir mehr filme an?	More items are shown. For example, if a list shows only a limited number of items, more will be retrieved and shown.
Show/see details (info) Optional	details ansehen/sehen zeig details an liefere/gib mir mehr details mehr details	More details are shown. For example, if information is truncated, more information are shown.
Get Next Page Required for certification	nächste nächstes das nächste spring/geh vor spring/geh vorwärts vor überspring das eins weiter/vorwärts weiter spring nach vorn spring nach vorn auf [AppName]/[DeviceBrand] nächstes im [DeviceLocation] nächstes auf [DeviceLocation]? [DeviceType]/[DeviceBrand] gib mir das nächste von [AppName] nächstes auf allen [DeviceType]	The next page of results appears on the screen.
Watch [title] Required for certification	ich möchte [VideoName] gucken spiel [VideoName] spiel watchlist ab von [VideoName]	The requested media plays. If there are multiple matches for the content, the user is prompted to select the desired media.
Watch by actor Required for certification	spiel [VideoName] (mit [ActorName]) auf [AppName] streame einen film mit [ActorName] auf [AppName] streame einen [ActorName] film auf [AppName] streame eine serie mit [ActorName] auf [AppName] streame eine [ActorName] serie auf [AppName]	Either media with the actor plays (through an algorithm that selects the best match for the user), or a list of results is presented to the user.
Watch by genre name Required for certification	spiel eine [GenreName] [MediaType] auf [DeviceType] ab spiel eine [GenreName] [MediaType] auf [AppName] ab [GenreName] [MediaType] schauen spiel einen [GenreName] [MediaType]	Either media from the requested genre plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by franchise [title] Required for certification	ich möchte [VideoName] gucken spiel [VideoName] spiel watchlist ab von [VideoName]?	Either the top recommended result for the requested franchise plays, or the user is presented with a list of matching franchises.
Watch by media type Optional	spiel einen [MediaType] spiel [MediaType] aus der/meiner watchlist streame den [MediaType] [VideoName] auf [AppName]/[Device] spiel trailer von [VideoName] spiel die neueste folge von [VideoName]	Either media matching the non-title request plays (through an algorithm that selects a recommended video for the user), or a list of results is presented to the user.
Watch [title] with season number Optional	spiel [VideoName] [Season] [Season_Numer] spiel [VideoName] [Season] [Season_Numer] auf [AppName]	The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with episode number Optional	spiel die [Episode_Number] [Episode] von [VideoName] spiel [VideoName] [Episode_Number] spiel von [VideoName] [Episode] [Episode_Number]	The correct season is identified by the user's last-watched episode. The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with season and episode number Optional	spiel [Season] [Season_Numer] [Episode] [Episode_Number] von [VideoName] spiel [VideoName] [Season] [Season_Numer] [Episode] [Episode_Number] spiel die [VideoName] [Season] [Season_Numer] [Episode] [Episode_Number] auf [AppName] ab	The content corresponding to the requested season and episode plays.
Watch by character name Optional	Not supported	Either media matching the character name plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports team Optional	spiel [SportsTeam] [Event] auf [AppName] [SportsTeam] [Event] anschauen/ansehen das [SportsTeam] [Event] auf [AppName] anschauen/ansehen spiel das [SportsTeam] [Event] auf meinem [Device]	Either media matching the sports team plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports type Optional	spiel das [Sport] [Event] auf meinem [Device] das [Sport] [Event] ansehen	Either media matching the sports type plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports league Optional	spiel [SportsTeam] [Event] auf [AppName] [SportsTeam] [Event] anschauen	Either media matching the sports league plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports event Optional	spiel das [Sport] [Event] auf meinem [Device] das [Sport] [Event] ansehen	Either media matching the sports event plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Launch [partner] app Required for certification	starte [AppName] spiel/öffne [AppName] app starte [AppName] auf [DeviceBrand]	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Launch special character title Required for certification	starte [AppName] spiel/öffne [AppName] app starte AppName auf [DeviceBrand]	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Go to control settings	Öffne die Einstellungen Einstellungen öffnen Gehe zu den Einstellungen Zu Einstellungen gehen Einstellungen aufrufen Ruf die Einstellungen auf zeige mir die eintellungen zeig mir meine w. lan einstellungen zeig mir meine start seiten einstellungen zeig meine start seiten feed einstellungen zeig mir die (toneinstellungen/geräteeinstellungen/erreichbarkeitseinstellungen/bedienungshilfeeinstellungen) öffne einstellungen öffne einstellungen auf [AppName]	The view changes to the control settings.
Go home Optional	Zum Startbildschirm Zu Start Geh nach Hause Gehe zum Startbildschirm Zurück zu Start bring mich zum start bildschirm bring mich zum home screen geh zum home screen geh zur home seite geh zurück zum start bildschirm zeig home zeige mir die start seite zeig mir den start bildschirm zu start seite geh zur start seite	The view changes to the home screen.
Close Optional	Not supported	The app or skill closes.
Exit Optional	Not Supported	The app or skill is exited.
Quit Optional	Not Supported	The app or skill is quit.
Open Optional	öffne [AppName] öffne [AppName] app öffne [AppName] auf [DeviceBrand]	The app or skill opens.

Feature	Sample Utterances	Expected Response
Watch this Required for certification	इसे देखो इसको देखो	The media in focus begins to play
Play this Required for certification	इसे चलाओ यह चला दो इसे चला दो यह चलाओ	The media in focus begins to play
Directional navigation - go Required for certification	ऊपर जाओ ऊपर करो नीचे जाओ * नीचे जाना * ambiguity with "move down" left जाना बाएं जाओ * left पे जाओ * ambiguity with "move left" right जाना right पे जाओ दायें जाओ ambiguity with "move right"	The app goes to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional navigation - move Required for certification	ऊपर जाओ ऊपर करो नीचे जाओ * नीचे जाना * ambiguity with "move down" left जाना बाएं जाओ * left पे जाओ * ambiguity with "move left" right जाना right पे जाओ दायें जाओ ambiguity with "move right"	The app moves to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional scrolling Required for certification	उपर चलो ऊपर scroll ऊपर करो नीचे scroll करें नीचे की तरफ scroll करो scroll down करो left scroll करो left की तरफ scroll करो बायें की तरफ scroll करो right scroll करो right की तरफ scroll करो दाएँ की तरफ scroll करो	The app scrolls left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional pagination Required for certification	पेज ऊपर करो पेज नीचे करो पेज को left पे ले जाओ पेज को right पे ले जाओ पेज को बाएँ ले जाओ पेज को दाएँ ले जाओ page नीचे करो page down करो page को बाएँ कर दो page left करो page को दाएँ करो page right करो	The app pages left, right, up, or down, similar pressing the same direction on the D-pad controller.
Verbless directional navigation Required for certification	ऊपर ऊपर जाओ उपर चलो ऊपर करो नीचे नीचे जाओ नीचे चलो नीचे करो left बाएँ जाओ बाएँ बाएँ करो Right दाएँ जाओ दाएँ दाएँ करो	The app shifts left, right, up, or down, similar pressing the same direction on the D-pad controller.
Select/choose this (that) Required for certification	इसे चुनो इसको select करो	Whatever is in focus gets selected.
Show/see more Optional	और दिखाओ	More items are shown. For example, if a list shows only a limited number of items, more will be retrieved and shown.
Show/see details (info) Optional	details दिखाओ जानकारी दिखाओ	More details are shown. For example, if information is truncated, more information are shown.
Get Next Page Required for certification	[GroupDeviceBrand] [HomeAutomationType] पर अगला लगाओ[GroupDeviceBrand] पर अगला लगाओ[GroupDeviceBrand] वाले [HomeAutomationType] पर अगला चलाना[AppName] पर इसके बाद वाला दो[AppName] पर मुझे next वाला दे देनाnext recipe अगली recipe[DeviceLocation] पर इसके बाद वाला दो[LocationNonSpecificQuantity] अगला चलाओ अगला/next [Episode] play करो[DeviceLocation] पर DeviceBrand अगला वाला दो	The next page of results appears on the screen.
Watch [title] Required for certification	[VideoName] चलाओ/बजाओ/लगाओ/play/देखो [AppName] पे/पर/में [VideoName] चलाओ/बजाओ/लगाओ/play	The requested media plays. If there are multiple matches for the content, the user is prompted to select the desired media.
Watch by actor Required for certification	[ActorName] का/की/के movies को चलाओ/बजाओ/लगाओ/play/देखो [ActorName] का/की/के movies चलाओ/बजाओ/लगाओ/play/देखो [ActorName] का/की/के [GenreName] को चलाओ/बजाओ/लगाओ/play/देखो [ActorName] का/की/के [GenreName] चलाओ/बजाओ/लगाओ/play/देखो [ActorName] का/की/के [VideoName] को चलाओ/बजाओ/लगाओ/play/देखो [ActorName] का/की/के [VideoName] चलाओ/बजाओ/लगाओ/play/देखो [ActorName] का/की/के/वाला [GenreName] की [VideoName] को चलाओ/बजाओ/लगाओ/play/देखो [ActorName] का/की/के/वाला [GenreName] की [VideoName] चलाओ/बजाओ/लगाओ/play/देखो	Either media with the actor plays (through an algorithm that selects the best match for the user), or a list of results is presented to the user.
Watch by genre name Required for certification	[GenreName] [VideoName] को चलाओ/बजाओ/लगाओ/play/देखो [GenreName] [VideoName] चलाओ/बजाओ/लगाओ/play/देखो	Either media from the requested genre plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by franchise [title] Required for certification	star trek देखो/देखना/देखिए	Either the top recommended result for the requested franchise plays, or the user is presented with a list of matching franchises.
Watch by media type Optional	[VideoName] movies चलाओ/बजाओ/लगाओ/play/देखो	Either media matching the non-title request plays (through an algorithm that selects a recommended video for the user), or a list of results is presented to the user.
Watch [title] with season number Optional	[VideoName] का/की/के [SeasonNumber] को चलाओ/बजाओ/लगाओ/play/देखो [VideoName] का/की/के [SeasonNumber] चलाओ/बजाओ/लगाओ/play/देखो [VideoName] का/की/के [SeasonNumber] को चलाओ/बजाओ/लगाओ/play/देखो [VideoName] का/की/के [SeasonNumber] [SeasonNumber] चलाओ/बजाओ/लगाओ/play/देखो	The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with episode number Optional	[VideoName] का/की/के [EpisodeNumber] को चलाओ/बजाओ/लगाओ/play/देखो [VideoName] का/की/के [EpisodeNumber] चलाओ/बजाओ/लगाओ/play/देखो	The correct season is identified by the user's last-watched episode. The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with season and episode number Optional	[VideoName] season one episode one चलाओ/बजाओ/लगाओ/play करो [VideoName] season एक का episode एक चलाओ/बजाओ/लगाओ/play करो	The content corresponding to the requested season and episode plays.
Watch by character name Optional	[CharacterName] का/की/के/वाला movies चलाओ/बजाओ/लगाओ/play/देखो [CharacterName] का/की/के/वाला movies को चलाओ/बजाओ/लगाओ/play/देखो Watch [CharacterName] movie [VideoName]	Either media matching the character name plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports team Optional	[Sport/League/Seam] games चलाओ/बजाओ/लगाओ/play/देखो [Sport/League/Seam] के games को चलाओ/बजाओ/लगाओ/play/देखो	Either media matching the sports team plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports type Optional	[Sport/League/Seam] games चलाओ/बजाओ/लगाओ/play/देखो [Sport/League/Seam] के games को चलाओ/बजाओ/लगाओ/play/देखो	Either media matching the sports type plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports league Optional	[Sport/League/Seam] games चलाओ/बजाओ/लगाओ/play/देखो [Sport/League/Seam] के games को चलाओ/बजाओ/लगाओ/play/देखो	Either media matching the sports league plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports event Optional	[Sport/League/Seam] games चलाओ/बजाओ/लगाओ/play/देखो [Sport/League/Seam] के games को चलाओ/बजाओ/लगाओ/play/देखो	Either media matching the sports event plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Launch [partner] app Required for certification	netflix launch करो/करना/कीजिए/करिएगा	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Launch special character title Required for certification	facebook app launch करो/करना/कीजिए/करिएगा	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Go to control settings		The view changes to the control settings.
Go home Optional	home पर जाओ/जाना/जाइए/चलो/चलना home पे जाओ/जाना/जाइए/चलो/चलना	The view changes to the home screen.
Close Optional	netflix बंद करना netflix close करो/कीजिए	The app or skill closes.
Exit Optional	1. netflix से बाहर निकलो/निकलना/आओ 2. netflix exit करो/करना	The app or skill is exited.
Quit Optional	netflix quit करना netflix quit कीजिए	The app or skill is quit.
Open Optional	netflix खोलो netflix खोलना netflix open करना	The app or skill opens.

Feature	Sample Utterances	Expected Response
Watch this Required for certification	guarda questo metti questo	The media in focus begins to play
Play this Required for certification	riproduci questo	The media in focus begins to play
Directional navigation - go Required for certification	vai su Vai giù / scendi voglio andare verso sinistra voglio andare verso destra	The app goes to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional navigation - move Required for certification	salta su Passa giù /passa in giù/salta giù / scendi/ salta verso il basso voglio andare verso sinistra voglio andare verso destra	The app moves to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional scrolling Required for certification	scorri verso l'alto scorri giù / scorri verso il basso /scrolla giù scorri verso sinistra scorri verso destra	The app scrolls left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional pagination Required for certification	pagina in su pagina di sotto pagina sinistra/pagina di sinistra pagina destra/pagina di destra	The app pages left, right, up, or down, similar pressing the same direction on the D-pad controller.
Verbless directional navigation Required for certification	su giù verso sinistra verso destra	The app shifts left, right, up, or down, similar pressing the same direction on the D-pad controller.
Select/choose this (that) Required for certification	seleziona questo scegli questo	Whatever is in focus gets selected.
Show/see more Optional	mostra mi di più	More items are shown. For example, if a list shows only a limited number of items, more will be retrieved and shown.
Show/see details (info) Optional	fornisci dettagli mostra più dettagli mostra i dettagli	More details are shown. For example, if information is truncated, more information are shown.
Get Next Page Required for certification	prossima / seguente / avanti / salta / successivo	The next page of results appears on the screen.
Watch [title] Required for certification	metti / trasmetti / riproduci / fammi guardare / fammi vedere + [VideoName] (+ da/su [DeviceName])	The requested media plays. If there are multiple matches for the content, the user is prompted to select the desired media.
Watch by actor Required for certification	guarda un film con [ActorName] metti un film con [ActorName]	Either media with the actor plays (through an algorithm that selects the best match for the user), or a list of results is presented to the user.
Watch by genre name Required for certification	guarda una [GenreName] fammi vedere una commedia mostra mi una commedia metti una commedia	Either media from the requested genre plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by franchise [title] Required for certification	guarda star trek fammi vedere star trek mostra star trek	Either the top recommended result for the requested franchise plays, or the user is presented with a list of matching franchises.
Watch by media type Optional	guarda un film metti un fim trasmetti un film riproduci un film	Either media matching the non-title request plays (through an algorithm that selects a recommended video for the user), or a list of results is presented to the user.
Watch [title] with season number Optional	guarda [VideoName] stagione due metti [VideoName] stagione due riproduci [VideoName] stagione due	The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with episode number Optional	guarda [VideoName] episodio cinque metti [VideoName] episodio cinque riproduci [VideoName] episodio cinque	The correct season is identified by the user's last-watched episode. The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with season and episode number Optional	guarda [VideoName] stagione due episodio cinque metti [VideoName] stagione due episodio cinque riproduci [VideoName] stagione due episodio cinque	The content corresponding to the requested season and episode plays.
Watch by character name Optional	guarda [CharacterName] fammi vedere un film con james bond mostrami un film di james bond	Either media matching the character name plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports team Optional	riproduci la partita del [SportsTeam]	Either media matching the sports team plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports type Optional	not supported	Either media matching the sports type plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports league Optional	trasmetti la partita della [League]	Either media matching the sports league plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports event Optional	not supported	Either media matching the sports event plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Launch [partner] app Required for certification	lancia [AppName] apri [AppName]	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Launch special character title Required for certification	apri [special character AppName]	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Go to control settings	vai alle impostazioni (di controllo) apri le impostazioni (di controllo)	The view changes to the control settings.
Go home Optional	vai alla home	The view changes to the home screen.
Close Optional		The app or skill closes.
Exit Optional	esci	The app or skill is exited.
Quit Optional	Chiudi [AppName]	The app or skill is quit.
Open Optional		The app or skill opens.

Feature	Sample Utterances	Expected Response
Watch this Required for certification	これ見せて	The media in focus begins to play
Play this Required for certification	これを再生して	The media in focus begins to play
Directional navigation - go Required for certification	上に行って下に行って左側に行って右の方へ行って	The app goes to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional navigation - move Required for certification	上に移動して下の方へ移動して左の方へ移動して右側に移動して	The app moves to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional scrolling Required for certification	上にスクロールして下にスクロールして左の方にスクロールして右側へスクロールして	The app scrolls left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional pagination Required for certification	画面の上部へページの下へ画面の左へページの右の方へ	The app pages left, right, up, or down, similar pressing the same direction on the D-pad controller.
Verbless directional navigation Required for certification	上^* 下左^* 右^*	The app shifts left, right, up, or down, similar pressing the same direction on the D-pad controller.
Select/choose this (that) Required for certification	これ選んで	Whatever is in focus gets selected.
Show/see more Optional	もっと見せて	More items are shown. For example, if a list shows only a limited number of items, more will be retrieved and shown.
Show/see details (info) Optional	詳細を表示して	More details are shown. For example, if information is truncated, more information are shown.
Get Next Page Required for certification	次スキップ次に行って[DeviceLocation]で次に行って[DeviceLocation]の[DeviceBrand]で次に行って[LocationNonSpecificQuantity]で次を再生[Episode]を再生して[VideoName]の[Episode]を再生[DeviceBrand]で再生	The next page of results appears on the screen.
Watch [title] Required for certification	[VideoName]を再生して [VideoName]を[AppName]で再生して [AppName]で[VideoName][MediaType]を再生して	The requested media plays. If there are multiple matches for the content, the user is prompted to select the desired media.
Watch by actor Required for certification	[ActorNameName]が出ている[MediaType]を再生	Either media with the actor plays (through an algorithm that selects the best match for the user), or a list of results is presented to the user.
Watch by genre name Required for certification	[AppName]で[GenreNameName][MediaType]を再生[AppName]で[GenreNameName]を再生	Either media from the requested genre plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by franchise [title] Required for certification		Either the top recommended result for the requested franchise plays, or the user is presented with a list of matching franchises.
Watch by media type Optional	[VideoName]を再生して [VideoName]を＜AppName]で再生して [AppName]で[VideoName][MediaType]を再生して [MediaType]を再生して [ActorNameName]が出ている[MediaType]を再生 [Anaphor][MediaType]を流して [AppName]で[GenreNameName][MediaType]を再生 [AppName]で[VideoName][MediaType]を再生して [ChannelName]の[MediaType]を再生	Either media matching the non-title request plays (through an algorithm that selects a recommended video for the user), or a list of results is presented to the user.
Watch [title] with season number Optional	[VideoName]の[Season][Season_Number]の	The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with episode number Optional	[Episode_Number][Episode]を再生	The correct season is identified by the user's last-watched episode. The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with season and episode number Optional	[Episode_Number][Episode]を再生 [VideoName]の[Season][Season_Number]の	The content corresponding to the requested season and episode plays.
Watch by character name Optional		Either media matching the character name plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports team Optional	[SportsTeam]の[Event]を[AppName]で見せて	Either media matching the sports team plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports type Optional	[SportsTeam]の[Event]を[AppName]で見せて	Either media matching the sports type plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports league Optional	[SportsTeam]の[Event]を[AppName]で見せて	Either media matching the sports league plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports event Optional	[SportsTeam]の[Event]を[AppName]で見せて	Either media matching the sports event plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Launch [partner] app Required for certification		The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Launch special character title Required for certification		The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Go to control settings	設定を開いて設定を見せて設定に戻って設定に行って	The view changes to the control settings.
Go home Optional	[TargetName]に行って	The view changes to the home screen.
Close Optional		The app or skill closes.
Exit Optional	[3p App]を閉じて	The app or skill is exited.
Quit Optional	[3p App]を終了して	The app or skill is quit.
Open Optional		The app or skill opens.

Feature	Sample Utterances	Expected Response
Watch this Required for certification	Assistir/assista/assiste isso	The media in focus begins to play
Play this Required for certification	Reproduzir/Reproduza/Reproduz isso	The media in focus begins to play
Directional navigation - go Required for certification	Subir/suba/sobe Vá/vai/ir para baixo Vá/vai/ir para esquerda Vá/vai/ir para direita	The app goes to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional navigation - move Required for certification	Mover/mova/move pra cima Mova/move/mover para baixo Mova/move/mover para a esquerda Mova/move/mover para direita	The app moves to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional scrolling Required for certification	Rolar/rola/role para cima Rola/role/rolar para baixo Rola/role/rolar para a esquerda Rola/role/rolar para a direita	The app scrolls left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional pagination Required for certification	página para cima/ ir para cima na página Página para baixo Pagina para esquerda Página para direita	The app pages left, right, up, or down, similar pressing the same direction on the D-pad controller.
Verbless directional navigation Required for certification	Ir/vai/vá pra cima Ir/vai/vá pra baixo Ir/vai/vá pra esquerda Ir/vai/vá pra direita	The app shifts left, right, up, or down, similar pressing the same direction on the D-pad controller.
Select/choose this (that) Required for certification	escolher isso/esse/essa selecionar isso/essa/ esse esse aqui / esse/ essa aqui/ essa/ isso aqui/ isso	Whatever is in focus gets selected.
Show/see more Optional	Mostrar/mostra/mostre mais	More items are shown. For example, if a list shows only a limited number of items, more will be retrieved and shown.
Show/see details (info) Optional	ver todos os detalhes do progama/show	More details are shown. For example, if information is truncated, more information are shown.
Get Next Page Required for certification	próximo/próxima	The next page of results appears on the screen.
Watch [title] Required for certification	tocar/assistir/reproduzir [VideoName] no/na [AppName]	The requested media plays. If there are multiple matches for the content, the user is prompted to select the desired media.
Watch by actor Required for certification	assistir [ator] colocar [ator] coloque filmes com [ator] encontre filmes com [ator] me mostre filmes com [ator]	Either media with the actor plays (through an algorithm that selects the best match for the user), or a list of results is presented to the user.
Watch by genre name Required for certification	assitir [GenreName] assistir programas de [GenreName] programas de [GenreName] colocar [GenreName] por favor reproduzir [GenreName] reproduzir programas de [GenreName] colocar um programa de [GenreName]	Either media from the requested genre plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by franchise [title] Required for certification	assitir jornada nas estrelas colocar jornada nas estrelas reproduzir jornada nas estrelas você pode passsar jornada nas estrelas por favor reproduzir jornada nas estrelas	Either the top recommended result for the requested franchise plays, or the user is presented with a list of matching franchises.
Watch by media type Optional	assistir um filme colocar um filme nós podemos assistir um filme ver um filme	Either media matching the non-title request plays (through an algorithm that selects a recommended video for the user), or a list of results is presented to the user.
Watch [title] with season number Optional	assistir [title] temporada [x] colocar [title] temporada [x] reproduzir [title] temporada [x]	The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with episode number Optional	assistir [title] episódio [x] colocar [title] episódio [x] reproduzir [title] episódio [x] reproduzir o [title] episódio [x]	The correct season is identified by the user's last-watched episode. The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with season and episode number Optional	assistir [title] temporada [x] episódio [x] colocar [title] temporada [x] episódio [x] reproduzir [title] temporada [x] episódio [x]	The content corresponding to the requested season and episode plays.
Watch by character name Optional	Not Supported	Either media matching the character name plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports team Optional	assisitr [SportTeam] assitir o/a [SportTeam] colocar no/na [SportTeam] colocar o jogo do/da [SportTeam] me mostre o jogo do/da [SportTeam] mostre o jogo do/da [SportTeam]	Either media matching the sports team plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports type Optional	assistir [Sport]	Either media matching the sports type plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports league Optional	assistir [League] colocar [League]	Either media matching the sports league plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports event Optional	not supported	Either media matching the sports event plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Launch [partner] app Required for certification	abrir [AppName]	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Launch special character title Required for certification	abrir [special character AppName]	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Go to control settings	vá para Configurações ir para Configurações abrir Configurações abra Configurações Configurações acessar Configurações acesse Configurações	The view changes to the control settings.
Go home Optional	vá para a página principal/ vá para a tela principal	The view changes to the home screen.
Close Optional	fechar [AppName]	The app or skill closes.
Exit Optional	Sair [AppName] sai [AppName] saia [AppName]	The app or skill is exited.
Quit Optional	sair do [AppName]	The app or skill is quit.
Open Optional		The app or skill opens.

Feature	Sample Utterances	Expected Response
Watch this Required for certification	ver lo^* ver este^*	The media in focus begins to play
Play this Required for certification	reproduce esto reproduce este	The media in focus begins to play
Directional navigation - go Required for certification	ve te arriba ve te abajo ve te a la izquierda ve a la izquierda ve te a la derecha ve a la derecha	The app goes to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional navigation - move Required for certification	muéve te hacia arriba muéve te hacia abajo muéve te hacia izquierda muéve te hacia derecha	The app moves to the left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional scrolling Required for certification	despláza te hacia arriba despláza te para arriba despláza te hacia abajo despláza te para abajo despláza te hacia izquierda despláza te para izquierda despláza te hacia derecha despláza te para derecha	The app scrolls left, right, up, or down, similar pressing the same direction on the D-pad controller.
Directional pagination Required for certification	arriba de la página abajo de la página izquierda de la página derecha de la página	The app pages left, right, up, or down, similar pressing the same direction on the D-pad controller.
Verbless directional navigation Required for certification	ve te arriba ve te abajo izquierda ve te a la derecha ve a la derecha	The app shifts left, right, up, or down, similar pressing the same direction on the D-pad controller.
Select/choose this (that) Required for certification	selecciona esto selecciona este selecciona aquel	Whatever is in focus gets selected.
Show/see more Optional	muestra más muestra qué más hay quiero ver algo más veamos un poco más veamos algo más	More items are shown. For example, if a list shows only a limited number of items, more will be retrieved and shown.
Show/see details (info) Optional	muestra detalles enseña detalles ver detalles	More details are shown. For example, if information is truncated, more information are shown.
Get Next Page Required for certification	siguiente salta muestra el siguiente siguiente en el sonos próxima en el jardín	The next page of results appears on the screen.
Watch [title] Required for certification	pon interstellar reproduce interstellar ver interstellar	The requested media plays. If there are multiple matches for the content, the user is prompted to select the desired media.
Watch by actor Required for certification	pon salma hayek reproduce salma hayek	Either media with the actor plays (through an algorithm that selects the best match for the user), or a list of results is presented to the user.
Watch by genre name Required for certification	pon comedia en netflix (Not supported without AppName)	Either media from the requested genre plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by franchise [title] Required for certification	pon star wars reproduce star wars	Either the top recommended result for the requested franchise plays, or the user is presented with a list of matching franchises.
Watch by media type Optional	pon la película^* reproduce la película^* Note: Not supported in es-ES or es-US	Either media matching the non-title request plays (through an algorithm that selects a recommended video for the user), or a list of results is presented to the user.
Watch [title] with season number Optional	empieza la segunda temporada de los rompecorazones comienza la segunda temporada de los rompecorazones reproduce la segunda temporada de los rompecorazones	The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with episode number Optional	pon el capítulo final de los serrano	The correct season is identified by the user's last-watched episode. The last-watched episode in the series plays, if the episode wasn't completed. If the last-watched episode was completed, the next unwatched episode plays.
Watch [title] with season and episode number Optional	pon el segundo episodio de la primera temporada de anabel reproduce el cuarto episodio de la temporada tres de águila roja	The content corresponding to the requested season and episode plays.
Watch by character name Optional	reproduce algo con james bond	Either media matching the character name plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports team Optional	ver el partido del real madrid	Either media matching the sports team plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of media results.
Watch by sports type Optional	reproduce el partido de fútbol en mi televisor ver el partido de fútbol en mi tele ver el partido de fútbol	Either media matching the sports type plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports league Optional	pon el partido de la a. c. b. en mi tele ver el partido de la a. c. b. en mi televisor	Either media matching the sports league plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Watch by sports event Optional	ver el partido de fútbol	Either media matching the sports event plays (through an algorithm that selects recommended media for the user), or the user is presented with a list of results.
Launch [partner] app Required for certification	lanza la app de netflix lanza netflix	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Launch special character title Required for certification	lanza i. b. e. x. treinta y cinco	The requested app launches and shows a list of available media to play. Multimodal devices have a homepage template that gets populated with media potentially grouped in different categories.
Go to control settings	abre configuración ve a configuración ve te a configuración configuración ir a configuración abrir configuración	The view changes to the control settings.
Go home Optional	página inicial ir a la página principal lléva me a la página principal pantalla de inicio menú de inicio pantalla principal menú principal	The view changes to the home screen.
Close Optional	cierra netflix	The app or skill closes.
Exit Optional	cierra netflix	The app or skill is exited.
Quit Optional	quitar netflix	The app or skill is quit.
Open Optional		The app or skill opens.

Sending UI State Reports

You need to send a UI state report to inform Alexa about the currently visible items on the customer's screen.

VSK-Enabled Apps

For VSK-enabled apps, you send the UI state report via the Alexa Client Library. The UI Controller requires Alexa Client Library version 1.4.9 and later. See Step 3: Integrate the Alexa Client Library for information about downloading and installing the Alexa Client Library.

Step 1 - Add the UI Controller to Alexa Client Library Supported Capabilities

Before you can send UI state reports, you need to add support for the UI Controller in the Alexa Client Library. In the initializeAlexaClient method, add getAlexaUIControllerCapability to the list of supportedCapabilities.

private void initializeAlexaClient() {
     // Retrieve the shared instance of the AlexaClientManager
     final AlexaClientManager clientManager = AlexaClientManager.getSharedInstance();

     // Gather your Skill ID
     final String alexaSkillId = "<insert skill id>";

     // Create a list of supported capabilities in your skill.
     final List<AlexaVideoCapability> supportedCapabilities = new ArrayList<>();
     supportedCapabilities.add(getAlexaChannelControllerCapability());
     supportedCapabilities.add(getAlexaPlaybackControllerCapability());
     supportedCapabilities.add(getAlexaRemoteVideoPlayerCapability());
     supportedCapabilities.add(getAlexaSeekControllerCapability());
     supportedCapabilities.add(getAlexaKeypadControllerCapability());
     supportedCapabilities.add(getAlexaUIControllerCapability()); //add UI Controller support

Step 2 - Send UI Reports through Alexa Client Library

To send your UI state report, call the setUIState method in the AlexaClientManager class of the Alexa Client Library.

public AlexaClientManager setUIState(Map<String, String> states) {
    Log.i(TAG, "setUIState");
    alexaClientEventManager.onUIStateSet(states);
    return this;
    }

The setUIState method takes a states parameter. The states parameter is a map of user interface properties representing elements displayed on screen. A key string, representing the property name to set (i.e. uiElements, focusedUIElement), is matched with a value string, the JSON encoded contents of the value key associated with this property.

VSK-Enabled Devices

For third party devices, you send the UI state report as a POST request to an Alexa event gateway endpoint every time there is a actionable change with your UI State. The steps to send UI state report events are listed below.

Step 1 - Request Permission to Send Events

UI state report events are sent to the Alexa event gateway, which requires an authentication token for each customer. In the Alexa console, find your skill and, select the PERMISSIONS tab on the left, and turn on Send Alexa Events. Record your Alexa Client Id and Alexa Client Secret (shown on this screen), which will be used in the next step to authenticate the customer with Login with Amazon (LwA). For more information, see Configure Permissions for Customer Information in Your Skill in the Alexa documentation. (State reporting is common with the Alexa Smart Home APIs.)

Step 2 - Add Code to Handle `AcceptGrant` Directives

When you request permission to send Alexa events, your skill's Lambda function will receive an AcceptGrant directive for each customer. The AcceptGrant directive contains an authorization code that you will use to obtain an authentication token for the customer. You must add code to obtain and store the tokens in the same region where your customer is located. Here's an example AcceptGrant directive:

{
  "directive": {
    "header": {
      "namespace": "Alexa.Authorization",
      "name": "AcceptGrant",
      "messageId": "<message id>",
      "payloadVersion": "3"
    },
    "payload": {
      "grant": {
        "type": "OAuth2.AuthorizationCode",
        "code": "VGhpcyBpcyBhbiBhdXRob3JpemF0aW9uIGNvZGUuIDotKQ=="
      },
      "grantee": {
        "type": "BearerToken",
        "token": "access-token-from-skill"
      }
    }
  }
}

AcceptGrant directive payload details

Field	Description	Type
`grant`	Information that identifies a user in Amazon Alexa systems.	Object
`grant.type`	The type of grant. Currently, the only valid value is `OAuth2.AuthorizationCode`.	String
`grant.code`	An authorization code for the user.	String
`grantee`	Information that identifies a user in a linked account service or system.	Object
`grantee.type`	The type of grantee. Currently, the only valid value is `BearerToken`.	String
`grantee.token`	The user access token received by Alexa during the account linking process.	String

If the AcceptGrant directive was successfully handled, respond with an AcceptGrant.Response event.

AcceptGrant response event example

{
  "event": {
    "header": {
      "namespace": "Alexa.Authorization",
      "name": "AcceptGrant.Response",
      "messageId": "<message id>",
      "payloadVersion": "3"
    },
    "payload": {}
  }
}

If you can't handle an AcceptGrant directive successfully, respond with an Alexa.Authorization.ErrorResponse event. Here are some reasons why the handling might be unsuccessful:

You are unable to call Login with Amazon to exchange the authorization code for access and refresh tokens.
You are unable to store the access and refresh tokens for the user.
Any other error that occurs while you try to retrieve and store the access and refresh tokens.

Here's an example Alexa.Authorization.ErrorResponse:

{
  "event": {
    "header": {
      "messageId": "abc-123-def-456",
      "namespace": "Alexa.Authorization",
      "name": "ErrorResponse",
      "payloadVersion": "3"
    },
    "payload": {
      "type": "ACCEPT_GRANT_FAILED",
      "message": "Failed to handle the AcceptGrant directive because <your reason>"
    }
  }
}

A user will not be able to enable your skill if an error occurs during AcceptGrant processing.

See the following for more details:

Step 3 - Indicate that Properties Are Proactively Reported During Discovery

Alexa sends a Discover directive through the Discovery Interface to understand your app's capabilities. In addition to declaring support for the UIController interface, during discovery you must also indicate that you will send change reports. You do this by indicating the properties are proactivelyReported in your discovery response.

Here's how you can configure your response to the Discover directive to indicate that you support the UIController capability:

{
    "event": {
        "header": {
            "namespace": "Alexa.Discovery",
            "name": "Discover.Response",
            "payloadVersion": "3.0",
            "messageId": "abc-123-def-456"
        },
        "payload": {
            "endpoints": [
                {
                    "endpointId": "uniqueIdOfEndpoint",
                    "friendlyName": "Living Room TV",
                    "description": "a description that is shown to the customer",
                    "cookie": {

                    },
                    "capabilities": [
                        {
                            "type": "AlexaInterface",
                            "interface": "Alexa.UIController",
                            "version": "3.0",
                            "properties": {
                                "supported": [
                                    {
                                        "name": "uiElements"
                                    },
                                    {
                                        "name": "focusedUIElement"
                                    }
                                ],
                                "proactivelyReported": true,
                                "retrievable": false
                            }
                        }
                    ]
                }
            ]
        }
    }
}

Declaring Support for MediaDetailsNavigator

If you're also planning to support the MediaDetailsNavigator interface, you can declare capabilities for it here. The MediaDetailsNavigator interface lets you handle customer requests to see more details about a specific item on the screen. See the Sending UI State Reports section in MediaDetailsNavigator for details.

For more information about state reporting and authorization in general, see the following:

Step 4 - Send UI State Reports to the Alexa Event Gateway

The final step is to begin sending UI state reports to the Alexa event gateway when a reportable property changes. When your UI state changes for any reason, send a UI state report. Each message to the event gateway should include the authentication token as an HTTP header and in the body of the message.

To send your UI state to the Alexa event gateway, submit the request header as follows:

POST /v3/events HTTP/1.1
Host: api.amazonalexa.com
Authorization: Bearer <access-token-from-Amazon>
Content-Type: application/json

(Change <access-token-from-Amazon to the actual authentication token you obtained in Step 2 above.) The request body must contain a JSON object with your UI state details as described in UI State Report Example and Schema. See the "Example" tab in the next section for details.

For more details about authorization, see the following in the Alexa documentation:

UI State Report Example and Schema

The following tabs provide a UI state report example (which shows sample populated values) and a schema (which shows data types rather than examples). A tree diagram is also included to show the schema's visual hierarchy.

POST /v3/events HTTP/1.1
Host: api.amazonalexa.com
Authorization: Bearer <access-token-from-Amazon>
Content-Type: application/json

{
    "context": {
        "properties": [

        ]
    },
    "event": {
        "header": {
            "messageId": "abc-123-def-456",
            "namespace": "Alexa",
            "name": "ChangeReport",
            "payloadVersion": "3"
        },
        "endpoint": {
            "scope": {
                "type": "AMAZON.BearerToken",
                "token": "access-token-from-Amazon"
            },
            "endpointId": "endpoint-001"
        },
        "payload": {
            "change": {
                "cause": {
                    "type": "PHYSICAL_INTERACTION"
                },
                "properties": [
                    {
                        "namespace": "Alexa.UIController",
                        "name": "uiElements",
                        "value": {
                            "scene": {
                                "sceneId": "sceneId-001"
                            },
                            "elements": [
                                {
                                    "elementId": "elementId_10",
                                    "entity": {
                                        "type": "AMAZON.ItemList"
                                    },
                                    "uiSupportedActions": [
                                        "SCROLL_DOWN",
                                        "SCROLL_FORWARD"
                                    ],
                                    "elements": [
                                        {
                                            "elementId": "elementId_11",
                                            "entity": {
                                                "type": "AMAZON.VideoObject",
                                                "name": {
                                                    "value": "Superbad"
                                                },
                                                "externalIds": {
                                                    "externalIdNamespace": "e99e067f-9112-4642-9853-b6401322964d"
                                                }
                                            },
                                            "ordinal": 1,
                                            "uiSupportedActions": [
                                                "SELECT"
                                            ]
                                        },
                                        {
                                            "elementId": "elementId_12",
                                            "entity": {
                                                "type": "AMAZON.VideoObject",
                                                "name": {
                                                    "value": "Dumb and Dumber"
                                                },
                                                "externalIds": {
                                                    "mediaBrowseId": "d9fd6168-81b8-4a28-a660-ba35141ca5b6"
                                                }
                                            },
                                            "ordinal": 2,
                                            "uiSupportedActions": [
                                                "SELECT"
                                            ]
                                        },
                                        {
                                            "elementId": "elementId_13",
                                            "entity": {
                                                "type": "AMAZON.VideoObject",
                                                "name": {
                                                    "value": "Hangover"
                                                },
                                                "externalIds": {
                                                    "mediaBrowseId": "08cb2390-2511-4df7-84c4-457a98fb738c"
                                                }
                                            },
                                            "ordinal": 2,
                                            "uiSupportedActions": [
                                                "SELECT"
                                            ]
                                        }
                                    ]
                                }
                            ]
                        },
                        "timeOfSample": "2017-02-03T16:20:50.52Z",
                        "uncertaintyInMilliseconds": 0
                    },
                    {
                        "namespace": "Alexa.UIController",
                        "name": "focusedUIElement",
                        "value": {
                            "scene": {
                                "sceneId": "sceneId-001"
                            },
                            "element": {
                                "elementId": "elementId_11",
                                "entity": {
                                    "type": "AMAZON.VideoObject",
                                    "name": {
                                        "value": "Superbad"
                                    },
                                    "externalIds": {
                                        "externalIdNamespace": "e99e067f-9112-4642-9853-b6401322964d"
                                    }
                                },
                                "ordinal": 1,
                                "uiSupportedActions": [
                                    "SELECT"
                                ]
                            }
                        },
                        "timeOfSample": "2017-02-03T16:20:50.52Z",
                        "uncertaintyInMilliseconds": 0
                    }
                ]
            }
        }
    }
}

{
  "context": {
    "properties": []
  },
  "event": {
    "header": {
      "messageId": string,
      "namespace": "Alexa",
      "name": "ChangeReport",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "AMAZON.BearerToken",
        "token": string
      },
      "endpointId": string
    },
    "payload": {
      "change": {
        "cause": {
          "type": "PHYSICAL_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {
              "scene": {
                "sceneId": string
              },
              "elements": [
                {
                  "elementId": string,
                  "entity": {
                    "type": enum<EntityTypes>
                  },
                  "uiSupportedActions": [
                    list<uiSupportedActions>
                  ],
                  "elements": [
                    {
                      "elementId": string,
                      "entity": {
                        "type": enum<EntityTypes>,
                        "name": {
                          "value": string
                        },
                        "externalIds": {
                          map<String>
                        }
                      },
                      "ordinal": Integer,
                      "uiSupportedActions": [
                      list<uiSupportedActions>
                      ]
                    }
                  ]
                }
              ]
            },
            "timeOfSample": string (ISO 8601),
            "uncertaintyInMilliseconds": integer
          },
          {
            "namespace": "Alexa.UIController",
            "name": "focusedUIElement",
            "value": {
              "scene": {
                "sceneId": string
              },
              "element": {
                "elementId": string,
                "entity": {
                  "type": enum<EntityTypes>,
                  "name": {
                    "value": string,
                    "variants": string
                  },
                  "externalIds": {
                    map<String>
                  }
                },
                "ordinal": Integer,
                "uiSupportedActions": [
                  list<uiSupportedActions>
                ]
              }
            },
            "timeOfSample": string (ISO 8601),
            "uncertaintyInMilliseconds": integer
          }
        ]
      }
    }
  }
}

object
├── context object
│   └── properties array[object]
└── event object
    ├── header object
    │   ├── messageId string
    │   ├── namespace enum => "Alexa"
    │   ├── name enum => "ChangeReport"
    │   └── payloadVersion string => "3"
    ├── endpoint object
    │   ├── scope object
    │   │   ├── type enum => "AMAZON.BearerToken"
    │   │   └── token string
    │   └── endpointId string
    └── payload object
        └── change object
            ├── cause object
            │   └── type enum => "PHYSICAL_INTERACTION"
            └── properties array[object]
                ├── namespace enum => "Alexa.UIController"
                ├── name enum => "uiElements"
                └── value object
                    ├── scene object
                    │   └── sceneId string
                    ├── elements array[object]
                    │   ├── elementId string
                    │   ├── entity object
                    │   │   └── type enum => Enum<EntityTypes>
                    │   ├── uiSupportedActions array[enum] => Enum<uiSupportedActions>
                    │   └── elements array[object]
                    │       ├── elementId string
                    │       ├── entity object
                    │       │   ├── type => Enum<EntityTypes>
                    │       │   ├── name object
                    │       │   │   ├── value string
                    │       │   │   └── variants string
                    │       │   └── externalIds map
                    │       ├── ordinal integer
                    │       └── uiSupportedActions array[enum] => Enum<uiSupportedActions>
                    ├── timeOfSample string ISO 8601
                    ├── uncertaintyInMilliseconds integer
                    ├── namespace enum => "Alexa.UIController"
                    ├── name enum => "focusedUIElement"
                    └── value object
                        ├── scene object
                        │   └── sceneId string
                        ├── element object
                        │   ├── elementId string
                        │   ├── entity object
                        │   │   ├── type enum => Enum<EntityTypes>
                        │   │   └── name object
                        │   │       ├── value string
                        │   │       └── variants string
                        │   └── uiSupportedActions array[enum] => Enum<uiSupportedActions>
                        ├── timeOfSample string
                        └── uncertaintyInMilliseconds integer

(The context and properties objects aren't used with Fire TV implementations. The context object would include only updates from other Alexa components such as Alexa.PowerState. Leave this empty if there’s no state update from other components when the Change Report event is triggered.)

Data Structures

This section describes the data structures for the state report schema described above. These tree diagrams are similar to the earlier schema but are paired down here to show the required schema for uiElements and focusedUIElement sections of your UI state report.

Schema for `uiElements` versus `focusedUIElements`

The properties array has two main objects: one with a namespace of uiElements, and another with a namespace of focusedUIElement. Their schemas are slightly different:

uiElements lists all elements on the screen; this section uses an elements array (plural) and also contains a nested array similarly called elements where you can provide an optional list of sub-elements.
focusedUIElement lists the element in focus; this section uses an element object (singular).

The following tabs shows the different schemas for uiElements and focusedUIElement lists.

uiElements Schema
focusedUIElement Schema

├── namespace enum => "Alexa.UIController"
├── name enum => "uiElements"
└── value object
    ├── scene object
    │   └── sceneId string
    ├── elements array[object]
    │   ├── elementId string
    │   ├── entity object
    │   │   └── type enum => Enum<EntityTypes>
    │   ├── uiSupportedActions array[enum] => Enum<uiSupportedActions>
    │   └── elements array[object]
    │       ├── elementId string
    │       ├── entity object
    │       │   ├── type => Enum<EntityTypes>
    │       │   ├── name object
    │       │   │   ├── value string
    │       │   │   └── variants string
    │       │   └── externalIds map
    │       ├── ordinal integer
    │       └── uiSupportedActions array[enum] => Enum<uiSupportedActions>
    ├── timeOfSample string ISO 8601
    └── uncertaintyInMilliseconds integer

├── namespace enum => "Alexa.UIController"
├── name enum => "focusedUIElement"
└── value object
    ├── scene object
    │   └── sceneId string
    ├── element object
    │   ├── elementId string
    │   ├── entity object
    │   │   ├── type enum => Enum<EntityTypes>
    │   │   └── name object
    │   │       ├── value string
    │   │       └── variants string
    │   └── uiSupportedActions array[enum] => Enum<uiSupportedActions>
    ├── timeOfSample string
    └── uncertaintyInMilliseconds integer

Payload Definitions

The following table defines the payload for the UI state reports that you send. Note that elements repeated (e.g., uiSupportedActions) in both the uiElements and focusedUIElement sections have the same definitions.

Payload Descriptions
Field	Description	Data Type
`change` required	Contains details about the change in state.	Object
`cause` required	Provides the reason for the change in state.	Object
`type` required	The type of state change. One option is allowed for Fire TV implementations: `AMAZON.PHYSICAL_INTERACTION`.	Enum
`properties` optional	Contains the details about the state change. The `properties` element contains details for state changes relating to both `uiElements` (a list of elements) and `focusedUIElement` objects.	Array
`namespace` required	The namespace for the payload. For Fire TV implementations, one option is allowed: `Alexa.UIController`.	String
`name` required	The name of the interface for the payload. Two options are allowed: `uiElement` and `focusedUIElement`. If you're providing a list of elements, use `uiElement`. In contrast, if you're reporting about the element in focus, use `focusedUIElement`. See the sample UI State Report Example for details and context. Example: `uiElements`	String
`value` optional	Contains a `scene` object that describes the elements on the customer's screen.	Object
`scene` required	Includes a `sceneId` property containing information about the scene displayed on the customer's UI.	Object
`sceneId` required	ID of the scene rendered on the customer's UI. Example: `sceneId-001`	String
`elements` optional	List of UI elements rendered on this scene. Used only with `uiElements` state reporting.	Array
`element` required	Element that has the visual focus on this scene. Used only with `focusedUIElement` state reporting.	List<Element>
`entity` required	A Thing (a generic object) that describes this element.	Object
`type` required	Type of this entity. See the Enum<EntityTypes> section for details. Example: `AMAZON.ItemList`	Enum<EntityTypes>
`name` optional	Name of this entity. Contains two possible properties: `value` and `variants`.	Object
`value` optional	Preferred name of this entity. Example: `Superbad`	String
`variants` optional	Additional names used to refer this entity. Example: `Extremebad`	String
`externalIds` optional	Identifiers for this entity unique within the scope of this endpoint. The `entity.externalIds` field will be opaque to Alexa and will only be used to include in directives from Alexa. The identifiers reported in `entity.externalIds` are required to be understood by the endpoint on all directive commands that accept an entity structure. Example: `"mediaBrowseId": "d9fd6168-81b8-4a28-a660-ba35141ca5b6"`	Map<String, String>
`elementId` required	Identifier of this UI element unique within the scope of the reported scene. The `elementId` field will be opaque to Alexa and will only be used to include in directives from Alexa to this endpoint. The endpoint can use this field to include information it needs to be able to locate this element on the rendered UI in response to a directive command. Example: `elementId_11`	String
`ordinal` optional	The number of the element that can be used in spoken language selection. For example, if this value is one then "Alexa select number 1" will select this element. Example: `Number one`	Integer
`uiSupportedActions` required	A list of enumerated actions a user can take on this UI element through the `UIController` interface. See Enum<uiSupportedActions> for details about enum options. Example: `SELECT`	Enum
`elements` optional	Optional list of nested UI elements. This element is unique to `uiElements`.	List<ElementWithChildren>
`timeOfSample` optional	The time at which the property value was recorded. Example: `2018-02-03T16:20:49.52Z`	ISO 8601 time
`uncertaintyInMilliseconds` optional	The number of milliseconds that have elapsed since the property value was last confirmed. Example: `0`	Integer

Enum<EntityTypes>

The type enum for entity allows the following options:

AMAZON.VideoObject: Entity type to represent a video item rendered on a UI. This entity is an abstraction for various types of videos (e.g. movie, tv shows, trailers, etc.). Example screen:

Example: “Interstellar” from this screen
AMAZON.ItemList: Entity type to represent a list of items. This entity can be used to represent a collection of entities of same or different types. Example screen:

Example: “Recents” from this screen
AMAZON.SoftwareApplication: Entity type to represent a software application. Example screen:

Example: “IMDb TV” from this screen
AMAZON.Thing: Entity type to represent generic entities whose specific type is either unknown to the API integrator or undefined in the UIController API specification. (Note that MediaDetailsNavigator does not support showing more details about this entity type.) Example screen:

Example: A screen reader application reporting “Captain Fantastic” from this screen

Enum<uiSupportedActions>

The uiSupportedActions enum allows the following options.

Name	Description	Directive Handling Expectation
`SELECT`	Action that selects a UI element	The app should do an equivalent of a single click on the provided element
`EXPAND`	Action that "expands" or otherwise "reveals more" on a UI element.	The app should expand the hidden, obstructed, truncated, or otherwise expandable content of the provided element.
`SCROLL_RIGHT`	Action that scrolls in the right direction on a scrollable element	The app should perform a scroll action on the specified element in the right direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen
`SCROLL_LEFT`	Action that scrolls in the left direction on a scrollable element	The app should perform a scroll action on the specified element in the left direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen
`SCROLL_UP`	Action that scrolls in the upwards direction on a scrollable element	The app should perform a scroll action on the specified element in the upwards direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen
`SCROLL_DOWN`	Action that scrolls in the downwards direction on a scrollable element	The app should perform a scroll action on the specified element in the downwards direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen
`SCROLL_FORWARD`	Action that navigates in the forward direction on a scrollable element	The app should perform a scroll action on the specified element in the forward direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen.
`SCROLL_BACKWARD`	Action that navigates in the backward direction on a scrollable element	The app should perform a scroll action on the specified element in the backward direction such that the sense of the scroll is clear, either through animation or by maintaining some elements on the screen.

ActionOnUIElement Directive Example

When the customer makes a request described in the Supported Utterances, the UIController interface sends a directive with the name ActionOnUIElement to your Lambda. The following is a sample directive. Note that the information you report in your UI state is used in the directive. For example, "sceneId": "sceneId-001" was reported in the UI state that the app sent to Alexa.

{
    "directive": {
        "header": {
            "namespace": "Alexa.UIController",
            "name": "ActionOnUIElement",
            "payloadVersion": "3"
        },
        "endpoint": {
            "scope": {
                "type": "AMAZON.BearerToken",
                "token": "access-token-from-Amazon"
            },
            "endpointId": "endpoint-001"
        },
        "payload": {
            "action": "SELECT",
            "scene": {
                "sceneId": "sceneId-001"
            },
            "element": {
                "elementId": "elementId_11",
                "entity": {
                    "type": "AMAZON.VideoObject",
                    "name": {
                        "value": "Superbad"
                    },
                    "externalIds": {
                        "externalIdNamespace": "e99e067f-9112-4642-9853-b6401322964d"
                    }
                },
                "ordinal": 1,
                "uiSupportedActions": [
                    "SELECT"
                ]
            }
        }
    }
}

Examples of UI State Reports and UIController Directives

The following are examples showing sample screens, the UI state report sent, the customer's utterance, and the UIController directive that Alexa sends to your Lambda.

Named/Ordinal Selection Example

In this example, the screen shows content in different ordinals (positions such as number 1, 2, 3, etc.). The user selects content by ordinal or by title.

Screen the user sees:

{
  "context": {
    "properties": [

    ]
  },
  "event": {
    "header": {
      "messageId": "abc-123-def-456",
      "namespace": "Alexa",
      "name": "ChangeReport",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "AMAZON.BearerToken",
        "token": "access-token-from-Amazon"
      },
      "endpointId": "endpoint-001"
    },
    "payload": {
      "change": {
        "cause": {
          "type": "PHYSICAL_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {
              "scene": {
                "sceneId": "scene-id-004"
              },
              "elements": [
                {
                  "entity": {
                    "type": "AMAZON.ItemList",
                    "externalIds": {
                      "entityId": "webpage-001"
                    }
                  },
                  "elementId": "elementId-001",
                  "uiSupportedActions": [
                    "SCROLL_DOWN",
                    "SCROLL_FORWARD"
                  ],
                  "elements": [
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Iron Man"
                        },
                        "externalIds": {
                          "entityId": "video-001"
                        }
                      },
                      "ordinal": 1,
                      "elementId": "elementId-002",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Interstellar"
                        },
                        "externalIds": {
                          "entityId": "video-002"
                        }
                      },
                      "ordinal": 2,
                      "elementId": "elementId-003",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Captain Fantastic"
                        },
                        "externalIds": {
                          "entityId": "video-003"
                        }
                      },
                      "ordinal": 3,
                      "elementId": "elementId-004",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "The Dressmaker"
                        },
                        "externalIds": {
                          "entityId": "video-004"
                        }
                      },
                      "ordinal": 4,
                      "elementId": "elementId-005",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Gaurdians of the Galaxy"
                        },
                        "externalIds": {
                          "entityId": "video-005"
                        }
                      },
                      "ordinal": 5,
                      "elementId": "elementId-006",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Passengers"
                        },
                        "externalIds": {
                          "entityId": "video-006"
                        }
                      },
                      "ordinal": 6,
                      "elementId": "elementId-007",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    }
                  ]
                }
              ]
            },
            "timeOfSample": "2017-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          }
        ]
      }
    }
  }
}

Customer utterance: "Alexa, select number two" / "Alexa, select interstellar"

{
  "directive": {
    "header": {
      "name": "ActionOnUIElement",
      "namespace": "Alexa.UIController"
    },
    "endpoint": {
      "scope": {
        "type": "AMAZON.BearerToken",
        "token": "access-token-from-Amazon"
      },
      "endpointId": "endpoint-001"
    },
    "payload": {
      "scene": {
        "sceneId": "scene-id-004"
      },
      "action": "SELECT",
      "element": {
        "entity": {
          "type": "AMAZON.VideoObject",
          "name": {
            "value": "Interstellar"
          },
          "externalIds": {
            "entityId": "video-002"
          }
        },
        "elementId": "elementId-003",
        "uiSupportedActions": [
          "SELECT"
        ]
      }
    }
  }
}

"Select This" Example

In this example, the user selected a title ("The Marvelous Mrs. Maisel") to view the details. The app presents the user with the content details. The user then says "select this." This speech pattern is referred to as "anaphora" because the user changes the referent from a title ("The Marvelous Mrs. Maisel") to a shorter abbreviation ("this"); in both cases, the referent refers to the same item.

Screen the user sees:

{
  "context": {
    "properties": [

    ]
  },
  "event": {
    "header": {
      "messageId": "abc-123-def-456",
      "namespace": "Alexa",
      "name": "ChangeReport",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "AMAZON.BearerToken",
        "token": "access-token-from-Amazon"
      },
      "endpointId": "endpoint-001"
    },
    "payload": {
      "change": {
        "cause": {
          "type": "PHYSICAL_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {
              "scene": {
                "sceneId": "scene-id-001"
              },
              "elements": [
                {
                  "entity": {
                    "type": "AMAZON.VideoObject",
                    "name": {
                      "value": "The Marvelous Mrs. Maisel"
                    },
                    "externalIds": {
                      "entityId": "video-001"
                    }
                  },
                  "elementId": "element-010",
                  "uiSupportedActions": [
                    "SELECT"
                  ],
                  "elements": [
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Watch Now with Prime"
                        },
                        "externalIds": {
                          "entityId": "webPageButton-001"
                        }
                      },
                      "elementId": "element-011",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Seasons & Episodes"
                        },
                        "externalIds": {
                          "entityId": "webPageButton-002"
                        }
                      },
                      "elementId": "element-012",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Add to Watchlist"
                        },
                        "externalIds": {
                          "entityId": "webPageButton-003"
                        }
                      },
                      "elementId": "element-013",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "More Ways to Watch"
                        },
                        "externalIds": {
                          "entityId": "webPageButton-003"
                        }
                      },
                      "elementId": "element-013",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },                    
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "CUSTOMERS ALSO WATCHED"
                        },
                        "externalIds": {
                          "entityId": "webPageButton-004"
                        }
                      },
                      "elementId": "element-014",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                  ]
                }
              ]
            },
            "timeOfSample": "2017-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          },
          {
            "namespace": "Alexa.UIController",
            "name": "focusedUIElement",
            "value": {
              "scene": {
                "sceneId": "scene-id-001"
              },
              "element": {
                "entity": {
                  "type": "AMAZON.Thing",
                  "name": {
                    "value": "Watch Now with Prime"
                  },
                  "externalIds": {
                    "entityId": "webPageButton-003"
                  }
                },
                "elementId": "element-013",
                "uiSupportedActions": [
                  "SELECT"
                ]
              }
            },
            "timeOfSample": "2017-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          }
        ]
      }
    }
  }
}

Customer utterance: "Alexa, select this"

{
  "directive": {
    "header": {
      "namespace": "Alexa.UIController",
      "name": "ActionOnUIElement"
    },
    "endpoint": {
      "scope": {
        "type": "AMAZON.BearerToken",
        "token": "access-token-from-Amazon"
      },
      "endpointId": "endpoint-001"
    },
    "payload": {
      "scene": {
        "sceneId": "scene-id-004"
      },
      "action": "SELECT",
      "element": {
        "entity": {
          "type": "AMAZON.Thing",
          "name": {
            "value": "Watch Now with Prime"
          },
          "externalIds": {
            "entityId": "webPageButton-003"
          }
        },
        "elementId": "element-013",
        "uiSupportedActions": [
          "SELECT"
        ]
      }
    }
  }
}

Named Play Example

In this example, the user sees a list of titles on the screen and selects one of the titles by name. Because the user asks to play the title, a RemoteVideoPlayer: SearchAndPlay directive is sent rather than a UIController directive.

Screen the user sees:

{
  "context": {
    "properties": [

    ]
  },
  "event": {
    "header": {
      "messageId": "abc-123-def-456",
      "namespace": "Alexa",
      "name": "ChangeReport",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "AMAZON.BearerToken",
        "token": "access-token-from-Amazon"
      },
      "endpointId": "endpoint-001"
    },
    "payload": {
      "change": {
        "cause": {
          "type": "PHYSICAL_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {
              "scene": {
                "sceneId": "scene-id-004"
              },
              "elements": [
                {
                  "entity": {
                    "type": "AMAZON.ItemList",
                    "externalIds": {
                      "entityId": "webpage-001"
                    }
                  },
                  "elementId": "elementId-001",
                  "uiSupportedActions": [
                    "SCROLL_DOWN",
                    "SCROLL_FORWARD"
                  ],
                  "elements": [
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Iron Man"
                        },
                        "externalIds": {
                          "entityId": "video-001"
                        }
                      },
                      "ordinal": 1,
                      "elementId": "elementId-002",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Interstellar"
                        },
                        "externalIds": {
                          "entityId": "video-002"
                        }
                      },
                      "ordinal": 2,
                      "elementId": "elementId-003",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Captain Fantastic"
                        },
                        "externalIds": {
                          "entityId": "video-003"
                        }
                      },
                      "ordinal": 3,
                      "elementId": "elementId-004",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "The Dressmaker"
                        },
                        "externalIds": {
                          "entityId": "video-004"
                        }
                      },
                      "ordinal": 4,
                      "elementId": "elementId-005",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Gaurdians of the Galaxy"
                        },
                        "externalIds": {
                          "entityId": "video-005"
                        }
                      },
                      "ordinal": 5,
                      "elementId": "elementId-006",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.VideoObject",
                        "name": {
                          "value": "Passengers"
                        },
                        "externalIds": {
                          "entityId": "video-006"
                        }
                      },
                      "ordinal": 6,
                      "elementId": "elementId-007",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    }
                  ]
                }
              ]
            },
            "timeOfSample": "2017-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          }
        ]
      }
    }
  }
}

Customer utterance: "Alexa, play Interstellar"

{
    "directive": {
        "header": {
            "name": "SearchAndPlay",
            "namespace": "Alexa.RemoteVideoPlayer"
        },
        "endpoint": {
            "scope": {
                "type": "AMAZON.BearerToken",
                "token": "access-token-from-Amazon"
            },
            "endpointId": "endpoint-001"
        },
        "payload": {
            "entities": [
                {
                    "externalIds": {
                        "entityId": "video-002"
                    },
                    "value": "Interstellar",
                    "type": "Video"
                }
            ]
        }
    }
}

"Play This" Example

In this example, the user previously asked to see more detail about the title "The Marvelous Mrs. Maisel," so the app showed the content details page. Now the user says "Play this." As with the previous "Select this" example, this is also an example of anaphora. Except in this case, the directive sent is a RemoteVideoPlayer: SearchAndPlay directive rather than a UIController directive.

Screen the user sees:

{
  "context": {
    "properties": [

    ]
  },
  "event": {
    "header": {
      "messageId": "abc-123-def-456",
      "namespace": "Alexa",
      "name": "ChangeReport",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "AMAZON.BearerToken",
        "token": "access-token-from-Amazon"
      },
      "endpointId": "endpoint-001"
    },
    "payload": {
      "change": {
        "cause": {
          "type": "PHYSICAL_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {
              "scene": {
                "sceneId": "scene-id-001"
              },
              "elements": [
                {
                  "entity": {
                    "type": "AMAZON.VideoObject",
                    "name": {
                      "value": "The Marvelous Mrs. Maisel"
                    },
                    "externalIds": {
                      "entityId": "video-001"
                    }
                  },
                  "elementId": "element-010",
                  "uiSupportedActions": [
                    "SELECT"
                  ],
                  "elements": [
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Watch Now with Prime"
                        },
                        "externalIds": {
                          "entityId": "webPageButton-001"
                        }
                      },
                      "elementId": "element-011",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Seasons & Episodes"
                        },
                        "externalIds": {
                          "entityId": "webPageButton-002"
                        }
                      },
                      "elementId": "element-012",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Add to Watchlist"
                        },
                        "externalIds": {
                          "entityId": "webPageButton-003"
                        }
                      },
                      "elementId": "element-013",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                      "type": "AMAZON.Thing",
                      "name": {
                        "value": "More Ways to Watch"
                    },
                      "externalIds": {
                        "entityId": "webPageButton-003"
                      }
                    },
                    "elementId": "element-013",
                    "uiSupportedActions": [
                      "SELECT"
                    ]
                   },                    
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "CUSTOMERS ALSO WATCHED"
                        },
                        "externalIds": {
                          "entityId": "webPageButton-004"
                        }
                      },
                      "elementId": "element-014",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                  ]
                }
              ]
            },
            "timeOfSample": "2017-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          },
          {
            "namespace": "Alexa.UIController",
            "name": "focusedUIElement",
            "value": {
              "scene": {
                "sceneId": "scene-id-001"
              },
              "element": {
                "entity": {
                  "type": "AMAZON.Thing",
                  "name": {
                    "value": "Watch Now with Prime"
                  },
                  "externalIds": {
                    "entityId": "webPageButton-003"
                  }
                },
                "elementId": "element-013",
                "uiSupportedActions": [
                  "SELECT"
                ]
              }
            },
            "timeOfSample": "2017-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          }
        ]
      }
    }
  }
}

Customer utterance: "Alexa, play this"

{
    "directive": {
        "header": {
            "name": "SearchAndPlay",
            "namespace": "Alexa.RemoteVideoPlayer"
        },
        "endpoint": {
            "scope": {
                "type": "AMAZON.BearerToken",
                "token": "access-token-from-Amazon"
            },
            "endpointId": "endpoint-001"
        },
        "payload": {
            "entities": [
                {
                    "externalIds": {
                        "entityId": "video-001"
                    },
                    "value": "The Marvelous Mrs. Maisel",
                    "type": "Video"
                }
            ]
        }
    }
}

On-screen Selection Example

Some applications can read the on-screen content by either scraping the UI or through accessibility frameworks. These applications can read the on screen content text and provide simple on screen selection functionalities without fully understanding content being rendered

These applications can also use the send state UI reports to the Alexa gateway endpoint to report the on-screen information to Alexa. The UIController interface includes a generic entity type AMAZON.Thing that can be used by screen-reader applications to report the on screen text data in the same format as some of the more advanced applications would use. The following is a simple UI state report and interaction.

Screen the user sees:

{
  "context": {
    "properties": []
  },
  "event": {
    "header": {
      "messageId": "abc-123-def-456",
      "namespace": "Alexa",
      "name": "ChangeReport",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "BearerToken",
        "token": "access-token-from-Amazon"
      },
      "endpointId": "avl-endpoint-id-001"
    },
    "payload": {
      "change": {
        "cause": {
          "type": "PHYSICAL_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {
              "scene": {
                "sceneId": "scene-id-004"
              },
              "elements": [
                {
                  "entity": {
                    "type": "AMAZON.Thing"
                  },
                  "elementId": "uiNode-000",
                  "uiSupportedActions": [
                    "SCROLL_FORWARD",
                    "SCROLL_DOWN"
                  ],
                  "elements": [
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Home"
                        }
                      },
                      "elementId": "uiNode-001",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Shows"
                        }
                      },
                      "elementId": "uiNode-002",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "IMDB FREEDIVE MOVIES AND TV-FREE WITH ADS"
                        }
                      },
                      "elementId": "uiNode-010",
                      "uiSupportedActions": []
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Captain Fantastic"
                        }
                      },
                      "elementId": "uiNode-011",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Fringe"
                        }
                      },
                      "elementId": "uiNode-012",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Drive"
                        }
                      },
                      "elementId": "uiNode-013",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Forte"
                        }
                      },
                      "elementId": "uiNode-014",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                    {
                      "entity": {
                        "type": "AMAZON.Thing",
                        "name": {
                          "value": "Padington"
                        }
                      },
                      "elementId": "uiNode-015",
                      "uiSupportedActions": [
                        "SELECT"
                      ]
                    },
                  ]
                }
              ]
            },
            "timeOfSample": "2017-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          }
        ]
      }
    }
  }
}

Customer utterance: "Alexa, select captain fantastic"

{
  "directive": {
    "header": {
      "namespace": "Alexa.UIController",
      "name": "ActionOnUIElement"
    },
    "endpoint": {
      "scope": {
        "type": "AMAZON.BearerToken",
        "token": "access-token-from-Amazon"
      },
      "endpointId": "endpoint-001"
    },
    "payload": {
      "scene": {
        "sceneId": "scene-id-004"
      },
      "action": "SELECT",
      "element": {
        "entity": {
          "type": "AMAZON.Thing",
          "name": {
            "value": "Captain Fantastic"
          }
        },
        "elementId": "uiNode-011",
        "uiSupportedActions": [
          "SELECT"
        ]
      }
    }
  }
}

Sample App Integration

You can find a sample integration of UIController in the cloudside sample app. The sample app includes Lambda code that shows how to declare support with the Discover directive for UIController and how to handle the ActionOnUIElement directives that UIController sends.

The sections below contain detailed use cases for UI Controller APIs from the sample app. You do not need to implement any of the classes or methods below to test the sample app. The code snippets below show you where to implement, and how to use, required methods. You need to use your own logic in your application code.

To test with the sample app, change the UI content of any screen and test voice targeting the new UI contents with Alexa.

MainFragment Class

The MainFragment class is associated with the sample app's home screen. The sample app reports what's on this page and what element is currently in focus.

public class MainFragment extends BrowseFragment implements MainActivity.UIElementAction, MainActivity.MediaDetailsAction {
    public static final String HOME_BROWSER_SCENE_IDENTIFIER = "home-browser-scene-000";

    private static final String TAG = "MainFragment";

    private static final int BACKGROUND_UPDATE_DELAY = 300;
    private static final int NUM_ROWS = 2;
    private static final int NUM_COLS = 4;

    private final Handler mHandler = new Handler();
    private Drawable mDefaultBackground;
    private DisplayMetrics mMetrics;
    private Timer mBackgroundTimer;
    private String mBackgroundUri;
    private BackgroundManager mBackgroundManager;

To see what's in focus and the current UI state reported to Alexa, run an adb (Android debug bridge) command in your terminal

adb logcat | grep MainFragment -i

The sample apps reports the UI State for this screen to Alexa whenever the customer makes a selection on this screen.

private void reportCurrentUIState(final Object selectedItem) {
    final ArrayObjectAdapter rowsAdapter = (ArrayObjectAdapter) getAdapter();
    final List<ElementWithChildren> elements = getUIElements(rowsAdapter);

    final Movie currentMovieSelected = (Movie) selectedItem;
    final Element focusedElement = getUIElement(currentMovieSelected);

    Log.d(TAG, MessageFormat.format("Focused movie element id is {0}", focusedElement.getElementId()));

    final Scene scene = new Scene.Builder().withSceneId(HOME_BROWSER_SCENE_IDENTIFIER).build();
    final UIElements uiElements = new UIElements.Builder().withScene(scene).withElements(elements).build();
    final FocusedUIElement focusedUIElement = new FocusedUIElement.Builder().withScene(scene).withElement(focusedElement).build();

    final Map<String, String> currentUIStateJSON = getUIStateJSON(uiElements, focusedUIElement);

    Log.d(TAG, MessageFormat.format("Reporting home screen UI State to Alexa: {0}", currentUIStateJSON));
    AlexaClientManager.getSharedInstance().setUIState(currentUIStateJSON);
    Log.i(TAG, "Finished reporting home screen UI State to Alexa.");
}

VideoDetailsFragment

In the sample app, the VideoDetailsFragment class is associated with a specific Movie object. This class provides all the Movie details like a trailer, purchase options, and movie description.

public class VideoDetailsFragment extends DetailsFragment implements DetailsActivity.UIElementAction, DetailsActivity.MediaDetailsAction {
    public static final String VIDEO_DETAIL_SCENE_IDENTIFIER = "video-detail-scene-001";

    private static final String TAG = "VideoDetailsFragment";

    private static final int ACTION_WATCH_TRAILER = 1;
    private static final int ACTION_RENT = 2;
    private static final int ACTION_BUY = 3;

    private static final int DETAIL_THUMB_WIDTH = 274;
    private static final int DETAIL_THUMB_HEIGHT = 274;

    private static final int NUM_COLS = 4;

    private Movie mSelectedMovie;

    private ArrayObjectAdapter mAdapter;
    private ClassPresenterSelector mPresenterSelector;

    private DetailsFragmentBackgroundController mDetailsBackground;

To see what's in focus and the current UI state reported to Alexa, run an adb (Android debug bridge) command in your terminal.

adb logcat | grep VideoDetailsFragment -i

The sample apps reports the UI State for this screen to Alexa whenever the customer makes a selection on this screen.

private void reportCurrentUIState(final Object selectedItem) {
    final ArrayObjectAdapter rowsAdapter = (ArrayObjectAdapter) getAdapter();
    final List<ElementWithChildren> elements = getUIElements(rowsAdapter);

    final Movie currentMovieSelected = (Movie) selectedItem;
    final Element focusedElement = getUIElement(currentMovieSelected);

    Log.d(TAG, MessageFormat.format("Focused movie element id is {0}", focusedElement.getElementId()));

    final Scene scene = new Scene.Builder().withSceneId(HOME_BROWSER_SCENE_IDENTIFIER).build();
    final UIElements uiElements = new UIElements.Builder().withScene(scene).withElements(elements).build();
    final FocusedUIElement focusedUIElement = new FocusedUIElement.Builder().withScene(scene).withElement(focusedElement).build();

    final Map<String, String> currentUIStateJSON = getUIStateJSON(uiElements, focusedUIElement);

    Log.d(TAG, MessageFormat.format("Reporting home screen UI State to Alexa: {0}", currentUIStateJSON));
    AlexaClientManager.getSharedInstance().setUIState(currentUIStateJSON);
    Log.i(TAG, "Finished reporting home screen UI State to Alexa.");
}

UI State Report

The sample app uses several classes to generate the UI state event payload:

Element
ElementWithChildren
Entity
EntityType
EntityName
Scene
UIAction
FocusedUIElement
UIElement

You can use these classes and the methods within them to generate the UI state payload with your logic. For example, the Element class defines each UI element on a screen. The sample app defines this behavior in the code snippet below:

package com.amazon.alexauicontroller;
 
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
 
/**
 * Represents an UI Element rendered on a UI
 */
public class Element {
 
    /**
     * Unique Id of this element in a scene
     */
    private String elementId;
 
    /**
     * A Thing that describes this Element
     */
    private Entity entity;
 
    /**
     * The number of the element that can be used in spoken language selection.
     */
    private Integer ordinal;
 
    /**
     * A list of UI actions a user can take on this element
     */
    private List<UIAction> uiSupportedActions;
 
    protected Element() {}
 
    protected Element(final String elementId, final Entity entity, final Integer ordinal,
                   final List<UIAction> uiSupportedActions) {
        this.elementId = elementId;
        this.entity = entity;
        this.ordinal = ordinal;
        this.uiSupportedActions = uiSupportedActions;
    }
 
    /**
     * @return the id of this element
     */
    public String getElementId() {
        return elementId;
    }
 
    /**
     * @return the {@link Entity} which describes this element
     */
    public Entity getEntity() {
        return entity;
    }
 
    /**
     * @return the number of this element
     */
    public Integer getOrdinal() {
        return ordinal;
    }
 
    /**
     * @return the list of {@link UIAction}s that a user can take on this element
     */
    public List<UIAction> getUiSupportedActions() {
        return uiSupportedActions;
    }
 
    /**
     * Sets the identifier for an element
     * @param elementId the element id to be set
     */
    public void setElementId(final String elementId) {
        this.elementId = elementId;
    }
 
    /**
     * Sets the entity for an element
     * @param entity the {@link Entity} to be set
     */
    public void setEntity(final Entity entity) {
        this.entity = entity;
    }
 
    /**
     * Sets the element number
     * @param ordinal the element number to be set
     */
    public void setOrdinal(final Integer ordinal) {
        this.ordinal = ordinal;
    }
 
    /**
     * Sets the UI actions for an element. This operation replaces the existing list of UI actions if already present
     * @param uiSupportedActions the list of {@link UIAction}s to be set
     */
    public void setUiSupportedActions(final List<UIAction> uiSupportedActions) {
        this.uiSupportedActions = uiSupportedActions;
    }

UIElementUtil Class

The UIElementUtil class is a utility class that builds all the UI elements required for generating UI state event payloads.

public class UIElementUtil {
    private static final String TAG = "UIElementUtil";

    public static final String EXTERNAL_ID_KEY = "entityId";
    public static final String EXTERNAL_ID_VALUE_PREFIX_FOR_ACTION = "webPageButton-";
    public static final String EXTERNAL_ID_VALUE_PREFIX_FOR_MOVIE = "video-";
    public static final String EXTERNAL_ID_VALUE_PREFIX_FOR_LIST_ROW = "videos-row-";
    public static final String ELEMENT_ID_PREFIX = "element-";
    public static final String ELEMENT_ID_PREFIX_FOR_ACTION = "action-";
    public static final String ELEMENT_ID_PREFIX_FOR_MOVIE = "movie-";
    public static final String ELEMENT_ID_PREFIX_FOR_VIDEO_DETAIL = "details-";
    public static final String ELEMENT_ID_PREFIX_FOR_LIST_ROW = "list-row-";
    public static final String UI_ELEMENTS_EVENT_KEY = "uiElements";
    public static final String FOCUSED_UI_ELEMENT_EVENT_KEY = "focusedUIElement";

onMessage method

The sample app receives the directives from Alexa through Lambda. The directive is called ActionOnUIElement, in the onMessage method.

protected void onMessage(final Intent intent) {
    Log.d(TAG, MessageFormat.format("Recieved a message from ADM: {0}", intent.toString()));
    
    //...
    
     else if ("ActionOnUIElement".equals(directiveName)) {

                if (jsonTree.isJsonObject()) {
                    final JsonObject jsonObject = jsonTree.getAsJsonObject();

                    final JsonObject jDirective = jsonObject.get("directive").getAsJsonObject();
                    final JsonElement jPayload = jDirective.get("payload");

                    if (jPayload != null && jPayload.isJsonObject()) {
                        final String jPayloadString = jPayload.getAsJsonObject().toString();

                        final ActionOnUIElement actionOnUIElement = gson.fromJson(jPayloadString, ActionOnUIElement.class);
                        Log.d(TAG, MessageFormat.format("The ActionOnUIElement directive is {0}", actionOnUIElement));

                        final boolean isActionOnUIElementValid = validateActionOnUIElementDirective(actionOnUIElement);
                        if (!isActionOnUIElementValid) {
                            Log.d(TAG, "The received ActionOnUIElement directive is invalid. Cannot process it.");
                            return;
                        }

                        final Intent actionOnUIElementIntent = new Intent();
                        actionOnUIElementIntent.setAction(ACTION_ON_UI_ELEMENT);
                        final String packageName = FireTVApp.getInstance().getPackageName();
                        actionOnUIElementIntent.setFlags(Intent.FLAG_ACTIVITY_SINGLE_TOP | Intent.FLAG_ACTIVITY_NEW_TASK);

                        actionOnUIElementIntent.putExtra(EXTRA_UI_CONTROLLER_ELEMENT_ID, actionOnUIElement.getElement().getElementId());
                        actionOnUIElementIntent.putExtra(EXTRA_UI_CONTROLLER_ELEMENT_TYPE, actionOnUIElement.getElement().getEntity().getType());
                        actionOnUIElementIntent.putExtra(EXTRA_UI_CONTROLLER_ACTION, actionOnUIElement.getAction());

                        final Scene scene = actionOnUIElement.getScene();
                        if (scene.getSceneId().equals(HOME_BROWSER_SCENE_IDENTIFIER)) {
                            Log.d(TAG, MessageFormat.format("Setting the destination of actionOnUIElement intent to Home Screen: {0}",
                                    MainActivity.class.getName()));
                            actionOnUIElementIntent.setClassName(packageName, MainActivity.class.getName());
                        } else if (scene.getSceneId().equals(VIDEO_DETAIL_SCENE_IDENTIFIER)) {
                            Log.d(TAG, MessageFormat.format("Setting the destination of actionOnUIElement intent to Detail Screen: {0}",
                                    DetailsActivity.class.getName()));
                            actionOnUIElementIntent.setClassName(packageName, DetailsActivity.class.getName());
                        } else {
                            Log.e(TAG, MessageFormat.format("Unknown scene id {0}. Cannot process ActionOnUIElement directive", scene.getSceneId()));
                            return;
                        }
                        Log.d(TAG, MessageFormat.format("Sending the actionOnUIElement intent: {0}", actionOnUIElementIntent));
                        FireTVApp.getInstance().startActivity(actionOnUIElementIntent);
                        Log.d(TAG, "Finished processing the UIController directive");
                    }
                } else {
                    // Invalid message JSON
                    Log.e(TAG, "Invalid message JSON");
                }
       //...

MediaDetailsNavigator

If you've implemented UIController, consider also implementing MediaDetailsNavigator, as this interface builds on the same UI state report you're already sending to also include support for utterances such as "Show more details about video title".

Last updated: Mar 29, 2021

Alexa.UIController Interface (VSK Fire TV)

Detailed Explanation

UIController versus KeypadController

Supported Utterances

Sending UI State Reports

VSK-Enabled Apps

Step 1 - Add the UI Controller to Alexa Client Library Supported Capabilities

Step 2 - Send UI Reports through Alexa Client Library

VSK-Enabled Devices

Step 1 - Request Permission to Send Events

Step 2 - Add Code to Handle AcceptGrant Directives

Step 3 - Indicate that Properties Are Proactively Reported During Discovery

Step 4 - Send UI State Reports to the Alexa Event Gateway

UI State Report Example and Schema

Data Structures

Schema for uiElements versus focusedUIElements

Payload Definitions

Enum<EntityTypes>

Enum<uiSupportedActions>

ActionOnUIElement Directive Example

Examples of UI State Reports and UIController Directives

Named/Ordinal Selection Example

"Select This" Example

Named Play Example

"Play This" Example

On-screen Selection Example

Sample App Integration

MainFragment Class

VideoDetailsFragment

UI State Report

UIElementUtil Class

onMessage method

MediaDetailsNavigator

Step 2 - Add Code to Handle `AcceptGrant` Directives

Schema for `uiElements` versus `focusedUIElements`