Web API Entity-Sensing Extension

Alexa-enabled devices with entity-sensing capabilities can detect the presence of the user. Using the entity-sensing extension, you can do the following:

  • Retrieve information to determine if the device has detected the user and, if detected, get the position of the user in relation to the device.
  • Respond to changes in user detection. For example, when the user position changes or when another user becomes the most engaged user.

The entity-sensing extension provides you with information about the primary user. The primary user is the user who is most likely engaging with the device. The algorithms to determine the primary user depend on the implementation of the specific device. Factors might include the distance and angle of the user from the device and the sensing capabilities of the microphone and camera. Use the entity-sensing interfaces to retrieve live updates of the primary user position.

Enable the entity-sensing extension in your skill

To enable the entity-sensing extension in your skill, you must add the extension in the skill manifest by using the Alexa Skills Kit (ASK) Command Line Interface (CLI) or the developer console. For more details about how to add extensions to the skill manifest, see Add extensions to your skill.

Add the extension by using the CLI

To enable the entity-sensing extension, add the ALEXA_EXTENSION interface to the skill manifest JSON file. The ALEXA_EXTENSION interface has two properties, autoInitializedExtensions and requestedExtensions. Add the entity-sensing extension to the requestedExtensions property. This extension doesn't have any settings to configure in autoInitializedExtensions.

The URI for the entity-sensing extension is alexaext:entitysensing:10.

The following example shows the skill manifest configured with the entity-sensing extension.

Copied to clipboard.

{
  "apis": {
    "custom": {
      "interfaces": [
        {
          "type": "ALEXA_EXTENSION",
          "requestedExtensions": [
            {
              "uri": "alexaext:entitysensing:10"
            }
          ]
        }
      ]
    }
  }
}

After you save your manifest file, use the Alexa Skills Kit Command Line Interface to deploy the changed manifest.

Add the extension in the developer console

You can also configure the skill manifest in the developer console.

To configure your skill for the entity-sensing extension

  1. Open the developer console, locate the skill you want to configure, and then click Edit.
  2. Navigate to the Build > Interfaces page.
  3. Enable the Alexa Web API for Games interface.
  4. From the Alexa Extensions list, select Entity Sensing v.10.
    When you select this option, you add the ALEXA_EXTENSION interface and the requestedExtensions property in the skill manifest.
  5. Click Save Interfaces, and then, to re-build your interaction model, click Build Model.

Add the extension to your web app

To load the Alexa JavaScript extension library, include the URL in a script tag on your HTML page, as shown in the following example.

Copied to clipboard.

<head>
      <script src="https://cdn.html.games.alexa.a2z.com/extensions/entity-sensing/v10/entity-sensing.js"></script>
</head>

Use the following code to initialize the entity-sensing client to allow the app to talk to the device.
Not all devices support entity-sensing. You can determine whether the device supports the extension by checking the alexa.capabilities interface, as shown in the following example.

Copied to clipboard.

Alexa.create({version: "1.1"})
    .then(async ({alexa, message, createExtensionsMessageProvider}) => {
        if(alexa.capabilities.extensions['alexaext:entitysensing:10']) {
            entitySensing = await EntitySensing.create(createExtensionsMessageProvider);
        }
    });

Entity-sensing objects

EntitySensing

The EntitySensing object exposes information about the user the device detects, if any.

EntitySensing object details

Property Description Type
primaryUser Provides the interfaces to retrieve information about the most engaged user.
For more details, see Get primary user information.		 A PrimaryUser object.

PrimaryUser

The PrimaryUser object represents the user who is most engaged with the device, if any. Sensing considers the entity who has spoken the wake word to be the active user and the detected entity who hasn't spoken the wake word to be the engaged, but not active.

The device assigns an entity id to the primary user. When an entity leaves the field of view and then returns to the field of view, the device makes a best effort attempt to recognize the user as a known entity. When a known entity returns to the field of view, the id remains the same. If the device doesn't recognize the user as a known entity, it assigns a new id. A given id persists for a limited amount of time when the device can no longer see the user. The duration of this time is device-specific. If an id that was previously assigned to an entity is no longer in use, the device doesn't reuse that id.

PrimaryUser object details

Property Description Type
id The identifier of the primary user assigned by the device. The format is device-specific. The string is empty when no user is detected. string
isActive Set to true if the primary user has spoken the wake word. Set to false otherwise. boolean
isSeen Set to true if the device detected the primary user and the user is within the observable range of the device. Set to false if the entity is no longer detected or the entity id isn't assigned. boolean
poise The position of the primary user. An EntityPoise object.

EntityPoise

The EntityPoise object represents the current angular position of the primary user. The EntityPoise object references angles in degrees. Horizontal positive values increase in the counter-clockwise direction with an axis centered at 0 degrees. Devices that don't rotate the screen have the same angular and relative positions.

EntityPoise object details

Property Description Type
absoluteAngle The current angular position of the entity relative to the center of the device. number
relativeAngle The current angular position of the entity relative to the screen. number

EntitySensingState

The EntitySensingState object identifies the functional state of entity-sensing. Errors occur when conditions prevent the device from sensing or the device experiences mechanical failure. Common causes of an error include:

  • Physical obstruction
  • Low light
  • The device is in "Do Not Disturb" mode
  • The user closed the camera shutter

Expect the device to have limited capabilities when errorCode is non-zero. When the device detects an error, it periodically checks to see if the error condition has cleared. After the condition is clear, normal entity-sensing operations resume.

EntitySensingState object details

Property Description Type
error When an error occurs, this property provides descriptive text when the device is experiencing an entity-sensing error.
When the device is operating normally, the error value is an empty string.		 number
errorCode Identifies the functional state of entity-sensing.
When an error occurs, this property reports a non-zero error code.
When the device is operating normally, the errorCode value is 0.		 number

The following errorCode and corresponding error values are available on Alexa devices. The error description and errorCode values are device-specific. A given device might provide additional values. The error string isn't localized.

errorCode Description Sample error text
0 – The camera is working properly.
1 The device can't locate the user due to an issue with the camera. Camera is disabled.
Camera shutter is closed.

Use error and errorCode values for game logic and debug purposes. Your web app should not display the error or errorCode value to the user.

Environment

The Environment object defines the static properties of the device.

Environment object details

Property Description Type
version The entity-sensing extension version. string
verticalFOV The device-specific vertical extent of the entity-sensing field of view in degrees. number
horizontalFOV The device-specific horizontal extent of the entity-sensing field of view in degrees. number

Entity-sensing interfaces

Your web app can get live updates to entity-sensing information by using these interfaces over the lifecycle of your game.

Get device properties

You can get the static properties of the device by checking the properties of the Environment object, as shown in the following example.

Copied to clipboard.

    let environment = entitySensing.environment;

Get primary user information

Get the state of the primary user from the device. Returns the PrimaryUser object, as shown in the following example.

Copied to clipboard.

    entitySensing.primaryUser;

The following table shows the possible states of the primary user:

isActive isSeen Meaning
true true The user spoke the wake word and the device detects the user. poise represents the current position of the user.
false true The device detects the user, but this user didn't speak the wake word. poise represents the current position of the user.
true or false false When the device can no longer sense the user, poise represents the last known position of the user.

Get the entity-sensing state

Get the entity-sensing state of the device. Returns the EntitySensingState object, as shown in the following example.

Copied to clipboard.

    entitySensing.entitySensingState;
Was this page helpful?

Last updated: May 01, 2024