Alexa.UIController Interface 3.1
Implement the Alexa.UIController interface in your Alexa skill so that users can voice-control onscreen navigation. You can voice-enable the user interface of entertainment devices, such as televisions and game consoles. For details about video skills, see Understand the Video Skill API.
When you implement the Alexa.UIController interface, you can send Alexa the full details of all user interface elements that appear on the screen, such as buttons, dialogues with options, menu items, and search results. For example, you can send a ChangeReport when a dialog appears on the screen with the options continue watching and exit. After that, the user can say "Alexa, continue watching." For more scenarios, see the Alexa.UIController Interface for Fire TV.
If you want to voice-enable onscreen navigation without sending the full user interface details, implement the Alexa.KeypadController interface instead.
For the list of languages that the UIController interface supports, see List of Alexa Interfaces and Supported Languages. For the definitions of the message properties, see Alexa Interface Message and Property Reference.
Utterances
The Alexa.UIController interface uses the pre-built voice interaction model. After the user says one of the following utterances, Alexa sends a corresponding directive to your skill.
The following examples show some user utterances:
Alexa, select The Aeronauts.
Alexa, select number one.
If you want users to be able to say "Alexa, go back" or "Alexa, scroll right", also implement the Alexa.KeypadController interface.
Properties and objects
Reportable properties
The Alexa.UIController interface supports the focusedUIElement and uiElements properties. You identify that you support the properties in your discovery response.
FocusedUIElement property
The Alexa.UIController interface uses the focusedUIElement property to represent the element on the user interface that has the focus. The focusedUIElement property value is an object.
The following table shows the definition of the FocusedUIElement object.
| Property | Description | Type | Required |
|---|---|---|---|
|
|
Identifying information about the user interface that currently appears on the user's screen. |
|
Yes |
|
|
Identifies the element in the user interface that has the focus. |
|
Yes |
UiElements property
The Alexa.UIController interface uses the uiElements property to represent all the elements in the current user interface. The uiElements property value is an object.
The following table shows the definition of the UiElements object.
| Property | Description | Type | Required |
|---|---|---|---|
|
|
Identifying information about the user interface that currently appears on the user's screen. |
|
Yes |
|
|
Identifies the element in the user interface that has the focus. |
Array of |
Yes |
Element object
The Alexa.UIController interface uses the Element object to represent an individual item that appears on the user's screen.
The following example shows a video element.
{
"element": {
"elementId": "elementId-003",
"ordinal": 3,
"uiSupportedActions": ["SELECT"],
"entity": {
"type": "AMAZON.VideoObject",
"name": {
"value": "The Dressmaker"
},
"externalIds": {
"entityId": "video-ghi"
}
}
}
}
The following table shows the definition of the Element object.
| Property | Description | Type | Required |
|---|---|---|---|
|
|
Unique identifier for the user interface element in this scene. |
String |
Yes |
|
|
Number of the element when the user can refer to the element by number, such as "Alexa, select number one." |
Integer |
No |
|
|
List of actions that a user can perform on the element. |
Array of strings |
Yes |
|
|
Information about the element. |
|
Yes |
|
|
Child elements of the element. Include the |
Array of |
No |
Entity object
The Alexa.UIController interface uses the Entity object to represent the content of a UI element.
The following example shows a video entity.
{
"entity": {
"type": "AMAZON.VideoObject",
"name": {
"value": "The Marvelous Mrs. Maisel",
"variants": ["Mrs. Maisel"]
},
"externalIds": {
"entityId": "video-1"
}
}
}
The following table shows the definition of the Entity object.
| Property | Description | Type | Required |
|---|---|---|---|
|
|
Describes the UI element. For more details, see Entity type values. |
String |
Yes |
|
|
Name of the entity |
Object |
No |
|
|
Preferred name of the entity, such as "The Marvelous Mrs. Maisel." |
String |
Yes |
|
|
List of alternate names for the entity, such as "Mrs. Maisel." |
Array of strings |
No |
|
|
External identifiers for the entity. |
Map<String, String> |
No |
Entity type values
The following table shows the string values for the type property.
| Value | Description |
|---|---|
|
|
An image or photo. |
|
|
A collection of entities of the same or different types, such as a collection of photos. |
|
|
An app, such as Netflix. |
|
|
A type that you can use when the entity doesn't fit the other type categories. |
|
|
A video item rendered on a UI. This entity is an abstraction for various types of videos, such as movies, trailers, TV shows. |
Scene object
The Alexa.UIController interface uses the Scene object to identify the user interface that currently appears on the user's screen.
The following example shows the home screen.
{
"scene": {
"sceneId": "Home Screen 1234"
}
}
The following table shows the definition of the Scene object.
| Property | Description | Type | Required |
|---|---|---|---|
|
|
Unique identifier for a set of UI elements that appear on the user's screen. |
String |
Yes |
Discovery
You describe endpoints that support Alexa.UIController by using the standard discovery mechanism described in Alexa.Discovery.
Set retrievable to true for the properties that you report when Alexa sends your skill a state report request. Set proactivelyReported to true for the properties that you proactively report to Alexa in a change report.
Alexa.UIController interface doesn't define any retrievable properties. Set retrievable to false in your Discover.Response. Set retrievable to true as appropriate for other interfaces that you implement in your skill.For the full list of display categories, see display categories.
Discover response example
The following example shows a Discover.Response message for a device that supports the Alexa.UIController and Alexa.PowerController interfaces.
{
"event": {
"header": {
"namespace": "Alexa.Discovery",
"name": "Discover.Response",
"payloadVersion": "3",
"messageId": "Unique identifier, preferably a version 4 UUID"
},
"payload": {
"endpoints": [
{
"endpointId": "Unique ID of the endpoint",
"manufacturerName": "Manufacturer of the endpoint",
"description": "Description to be shown in the Alexa app",
"friendlyName": "Device name, displayed in the Alexa app",
"displayCategories": ["TV"],
"cookie": {},
"capabilities": [
{
"type": "AlexaInterface",
"interface": "Alexa.UIController",
"version": "3.1",
"properties": {
"supported": [
{
"name": "uiElements"
},
{
"name": "focusedUIElement"
}
],
"proactivelyReported": true,
"retrievable": false
}
},
{
"type": "AlexaInterface",
"interface": "Alexa.PowerController",
"version": "3",
"properties": {
"supported": [
{
"name": "powerState"
}
],
"proactivelyReported": true,
"retrievable": true
}
},
{
"type": "AlexaInterface",
"interface": "Alexa",
"version": "3"
}
]
}
]
}
}
}
Directives
Alexa sends the following Alexa.UIController interface directives to your skill.
ActionOnUIElement directive
Support the ActionOnUIElement directive so that users can voice-control your user interface.
scene specified in the directive with the scene that currently appears to the user. If the scene in the directive doesn't match the current scene, do not act on the directive and send an Alexa.ErrorResponse event instead of an Alexa.Response event.ActionOnUIElement directive example
The following example shows a ActionOnUIElement directive that Alexa sends to your skill.
{
"directive": {
"header": {
"namespace": "Alexa.UIController",
"name": "ActionOnUIElement",
"messageId": "Unique version 4 UUID",
"correlationToken": "Opaque correlation token",
"payloadVersion": "3.1"
},
"endpoint": {
"endpointId": "Endpoint id",
"cookie": {}
},
"payload": {
"scene": {
"sceneId": "Home Screen 1234"
},
"action": "SCROLL_FORWARD",
"element": {
"elementId": "list-001",
"uiSupportedActions": ["SCROLL_FORWARD", "SCROLL_RIGHT"],
"ordinal": 12,
"entity": {
"type": "AMAZON.ItemList",
"name": {
"value": "Suggested for You",
"variants": ["Suggested"]
}
}
}
}
}
}
ActionOnUIElement directive payload
The following table shows the payload details for the ActionOnUIElement directive.
| Property | Description | Type | Required |
|---|---|---|---|
|
|
Identifies the user interface scene that the user is trying to act on. |
|
Yes |
|
|
Identifies the action that the user wants to perform. |
String |
Yes |
|
|
Identifies the element that the user wants to perform the action on. |
|
Yes |
ActionOnUIElement response
If you handle a ActionOnUIElement directive successfully, respond with an Alexa.Response event. In the context object, include the values of all relevant properties.
The following example shows a ActionOnUIElement response.
{
"event": {
"header": {
"namespace": "Alexa",
"name": "Response",
"messageId": "Unique identifier, preferably a version 4 UUID",
"correlationToken": "Opaque correlation token that matches the request",
"payloadVersion": "3"
},
"endpoint": {
"endpointId": "Endpoint id"
},
"payload": {}
},
"context": {}
}
ActionOnUIElement directive error handling
If you can't handle a ActionOnUIElement directive successfully and the error is specific to video, respond with an Alexa.Video.ErrorResponse event. For general errors, respond with a generic Alexa.ErrorResponse event.
State reporting
Alexa sends a ReportState directive to request information about the state of an endpoint. When Alexa sends a ReportState directive, you send a StateReport event in response. The response contains the current state of all the retrievable properties in the context object. You identify your retrievable properties in your discovery response. For details about state reports, see Understand State and Change Reporting.
Alexa.UIController interface doesn't define any retrievable properties. However, if you also implement other interfaces in the skill for your device, you must participate in state reporting for the properties in those interfaces. For example, you must report the properties from the Alexa.PowerController interface.StateReport response example
{
"event": {
"header": {
"namespace": "Alexa",
"name": "StateReport",
"messageId": "Unique identifier, preferably a version 4 UUID",
"correlationToken": "Opaque correlation token that matches the request",
"payloadVersion": "3"
},
"endpoint": {
"endpointId": "Endpoint id"
},
"payload": {}
},
"context": {
"properties": [
{
"namespace": "Alexa.PowerController",
"name": "powerState",
"value": "OFF",
"timeOfSample": "2020-02-03T16:20:50.52Z",
"uncertaintyInMilliseconds": 0
}
]
}
}
Change reporting
You send a ChangeReport event to report changes proactively in the state of an endpoint. You identify the properties that you proactively report in your discovery response. For details about change reports, see Understand State and Change Reporting.
ChangeReport event example
{
"event": {
"header": {
"namespace": "Alexa",
"name": "ChangeReport",
"messageId": "Unique identifier, preferably a version 4 UUID",
"payloadVersion": "3"
},
"endpoint": {
"scope": {
"type": "BearerToken",
"token": "OAuth2.0 bearer token"
},
"endpointId": "Endpoint id"
},
"payload": {
"change": {
"cause": {
"type": "VOICE_INTERACTION"
},
"properties": [{
"namespace": "Alexa.UIController",
"name": "uiElements",
"value": {
"scene": {
"sceneId": "Home Screen 1234"
},
"elements": [{
"elementId": "list-001",
"ordinal": 12,
"uiSupportedActions": ["SCROLL_FORWARD", "SCROLL_RIGHT"],
"entity": {
"type": "AMAZON.ItemList",
"name": {
"value": "Suggested for You",
"variants": ["Suggested"]
}
},
"elements": [{
"elementId": "elementId-001",
"ordinal": 1,
"uiSupportedActions": ["SELECT"],
"entity": {
"type": "AMAZON.VideoObject",
"name": {
"value": "Captain Fantastic"
},
"externalIds": {
"entityId": "video-abc"
}
}
},
{
"elementId": "elementId-002",
"ordinal": 2,
"uiSupportedActions": ["SELECT"],
"entity": {
"type": "AMAZON.VideoObject",
"name": {
"value": "The Aeronauts"
},
"externalIds": {
"entityId": "video-def"
}
}
},
{
"elementId": "elementId-003",
"ordinal": 3,
"uiSupportedActions": ["SELECT"],
"entity": {
"type": "AMAZON.VideoObject",
"name": {
"value": "The Dressmaker"
},
"externalIds": {
"entityId": "video-ghi"
}
}
}
]
}]
},
"timeOfSample": "2020-02-03T16:20:50.52Z",
"uncertaintyInMilliseconds": 0
},
{
"namespace": "Alexa.UIController",
"name": "focusedUIElement",
"value": {
"scene": {
"sceneId": "Home Screen 1234"
},
"element": {
"elementId": "elementId-003",
"ordinal": 3,
"uiSupportedActions": ["SELECT"],
"entity": {
"type": "AMAZON.VideoObject",
"name": {
"value": "Captain Fantastic"
},
"externalIds": {
"entityId": "video-abc"
}
}
}
},
"timeOfSample": "2020-02-03T16:20:50.52Z",
"uncertaintyInMilliseconds": 0
}
]
}
}
},
"context": {
"properties": [{
"namespace": "Alexa.PowerController",
"name": "powerState",
"value": "ON",
"timeOfSample": "2020-02-03T08:00:50.52Z",
"uncertaintyInMilliseconds": 500
}]
}
}
ChangeReport event example to reset UI
To reset the UIController state, for example, when the user opens an app on the device that you don't control, send a change report with an empty uiElements property.
{
"event": {
"header": {
"namespace": "Alexa",
"name": "ChangeReport",
"messageId": "Unique identifier, preferably a version 4 UUID",
"payloadVersion": "3"
},
"endpoint": {
"scope": {
"type": "BearerToken",
"token": "OAuth2.0 bearer token"
},
"endpointId": "Endpoint id"
},
"payload": {
"change": {
"cause": {
"type": "VOICE_INTERACTION"
},
"properties": [
{
"namespace": "Alexa.UIController",
"name": "uiElements",
"value": {},
"timeOfSample": "2020-02-03T16:20:50.52Z",
"uncertaintyInMilliseconds": 0
}
]
}
}
},
"context": {
"properties": [
{
"namespace": "Alexa.PowerController",
"name": "powerState",
"value": "ON",
"timeOfSample": "2020-02-03T10:00:50.52Z",
"uncertaintyInMilliseconds": 500
}
]
}
}
Related topics
Last updated: Aug 23, 2024