APL Entity-Sensing Extension

Devices with entity-sensing capabilities can detect the presence of the user. Use the entity-sensing extension in your APL response to get information about the detected user, such as the position of the user in relation to the device. Your skill can also respond to changes, such as when the most engaged user changes.

Extension summary

About the entity-sensing extension

The entity-sensing APL extension works with devices that can detect the presence of the user. Use the entity-sensing APL extension to do the following:

• Retrieve properties to determine whether the device has detected the user and get the position of the user in relation to the device.
• Use event handlers to respond to changes in the device state and user detection. For example, you can respond when the device determines that the most engaged user has changed.

The entity-sensing extension provides you with information about the primary user. The primary user is the user who is most likely engaging with the device. The algorithm to determine the primary user is specific to the device and depends on different factors.

Use the live data properties to retrieve information about the primary user.

Enable the entity-sensing extension

To use the entity-sensing extension properties and event handlers, you must complete the following steps:

1. Add the extension to the skill manifest requestedExtensions property.
2. Add the extension to the extensions property of the APL document to explicitly request it in the document.
3. Configure the document-level settings for the extension in the document settings property.

Entity-sensing extension in the skill manifest

The ALEXA_EXTENSION interface has two properties, autoInitializedExtensions and requestedExtensions. Add the entity-sensing extension to the requestedExtensions property. This extension doesn't have any settings to configure in autoInitializedExtensions.

The URI for the entity-sensing extension is "alexaext:entitysensing:10".

The following example shows the skill manifest configured with the entity-sensing extension.

{
  "apis": {
    "custom": {
      "interfaces": [
        {
          "type": "ALEXA_EXTENSION",
          "requestedExtensions": [
            {
              "uri": "alexaext:entitysensing:10"
            }
          ]
        }
      ]
    }
  }
}

For more details about APL extension settings in the skill manifest, see Request an extension in the skill manifest.

You can also configure the skill manifest in the developer console.

To configure your skill for the entity-sensing extension

1. Open the developer console, locate the skill to configure, and then click Edit.
2. Navigate to the Build > Interfaces page.
3. Enable the Alexa Presentation Language interface.

4. From the Extensions list, select Entity Sensing v.10.

   When you select this option, you update the requestedExtensions property in the skill manifest.

5. Click Save Interfaces, and then, to rebuild your interaction model, click Build Model.

Entity-sensing extension in the APL document

To use the entity-sensing event handlers and properties in an APL document, you must request the extension in the document extensions property. This requirement is in addition to adding the extension to requestedExtensions in the skill manifest.

The URI for the entity-sensing extension is "alexaext:entitysensing:10".

The following example requests the entity-sensing extension and names it EntitySensing. You then use the "EntitySensing" name in your document to reference the settings, properties, and event handlers.

{
  "type": "APL",
  "version": "2024.2",
  "extensions": [
    {
      "name": "EntitySensing",
      "uri": "alexaext:entitysensing:10"
    }
  ]
}

For more details about requesting an APL extension in the APL document, see Request an extension in the APL document.

Verify device support for the entity-sensing extension

Not all devices support the entity-sensing extension. Determine whether the user's device supports the extension in your document and in your code.

In your document, use the environment.extension variable to determine whether the device supports the entity-sensing extension. For example, assuming you used EntitySensing for the name in the extensions property, the following data-binding expression returns true when the device supports entity sensing.

${environment.extension.EntitySensing}

You can use a data-binding expression in conditional statements and with the when property.

In your code, the request sent to your skill includes supported extensions in the context.Extensions.available property. This property contains a map in which the key for each item is the URI for the available extension.

The following example shows a request from a device that supports the entity-sensing extension.

{
  "version": "1.0",
  "session": {},
  "context": {
    "Viewports": [],
    "Viewport": {},
    "Extensions": {
      "available": {
        "alexaext:entitysensing:10": {}
      }
    },
    "System": {}
  },
  "request": {}
}

The context.Extensions property includes extensions that meet both of the following criteria:

• You requested the extension in the skill manifest in the requestedExtensions property.
• The device supports the extension.

Extension settings

The following table shows the settings for the entity-sensing extension.

To configure these settings in your APL document, use settings.AssignedName, where AssignedName is the name you assigned the entity-sensing extension in the extensions property.

The following example assigns the extension the name EntitySensing, and then sets the entitySensingStateName and primaryUserName properties in settings.EntitySensing.

{
  "type": "APL",
  "version": "2024.2",
  "extensions": [
    {
      "name": "EntitySensing",
      "uri": "alexaext:entitySensing:10"
    }
  ],
  "settings": {
    "EntitySensing": {
      "entitySensingStateName": "EntitySensingState",
      "primaryUserName": "User"
    }
  },
  "mainTemplate": {}
}

entitySensingStateName

Set the entitySensingStateName property to the name you want to use to access the EntitySensingState in the global data-binding context. When entitySensingStateName is missing or set to an empty string, you can't access the EntitySensingState properties.

The following example shows how to set the entitySensingStateName to MyEntitySensingState, and then access an EntitySensingState property.

{
  "type": "APL",
  "version": "2024.2",
  "extensions": [
    {
      "name": "EntitySensing",
      "uri": "alexaext:entitysensing:10"
    }
  ],
  "settings": {
    "EntitySensing": {
      "entitySensingStateName": "MyEntitySensingState",
      "primaryUserName": "MyPrimaryUser"
    }
  },
  "mainTemplate": {
    "parameters": [
      "payload"
    ],
    "items": [
      {
        "type": "Container",
        "items": [
          {
            "type": "Text",
            "id": "MyTextBox",
            "text": "Entity sensing is ${EntitySensingState.errorCode == 0 ? 'Active' : 'Faulted'}"
          },
          {
            "type": "Text",
            "id": "MyTextBox",
            "text": "Error code: ${MyEntitySensingState.errorCode}"
          },
          {
            "type": "Text",
            "text": "Error: ${MyEntitySensingState.error}"
          }
        ]
      }
    ]
  }
}

primaryUserName

Set the primaryUserName property to the name you want to use to access the PrimaryUser properties in the global data-binding context. When primaryUserName is missing or set to an empty string, you can't access the PrimaryUser properties.

The following example shows how to set the primaryUserName to MyPrimaryUser and then access a PrimaryUser property.

{
  "type": "APL",
  "version": "2024.2",
  "extensions": [
    {
      "name": "EntitySensing",
      "uri": "alexaext:entitysensing:10"
    }
  ],
  "settings": {
    "EntitySensing": {
      "primaryUserName": "MyPrimaryUser",
      "entitySensingStateName": "MyEntitySensingState"
    }
  },
  "mainTemplate": {
    "item": {
      "type": "Container",
      "items": [
        {
          "type": "Text",
          "id": "MyTextBox",
          "text": "${MyPrimaryUser.isSeen ? 'User located at ${MyPrimaryUser.poise.absoluteAngle} degrees.' : 'Nobody here'}"
        },
        {
          "type": "Text",
          "id": "textUserId",
          "text": "id: ${MyPrimaryUser.id}"
        },
        {
          "type": "Text",
          "id": "textUserActive",
          "text": "isActive: ${MyPrimaryUser.isActive}"
        },
        {
          "type": "Text",
          "id": "textUserSeen",
          "text": "isSeen: ${MyPrimaryUser.isSeen}"
        }
      ]
    }
  }
}

Environment properties (static)

The following table shows the static properties that the entity-sensing extension adds to the environment.extension property in the data-binding context.

A sample entity-sensing environment object has the following structure.

{
  "version": "1.0",
  "horizontalFOV": 60,
  "verticalFOV": 34
}

To access these properties in your APL document, use environment.extensions.AssignedName, where AssignedName is the name you assigned the entity-sensing extension in the extensions property. For example, assuming you used EntitySensing for the name in the extensions property, the following data-binding expression returns the value of the horizontalFOV property.

${environment.extension.EntitySensing.horizontalFOV}

horizontalFOV

The horizontalFOV defines the horizontal extent of the entity-sensing field of view. This property is device specific.

version

The version environment property represents the entity-sensing service version. The version can be useful to report build and release details of the installed extension.

verticalFOV

The verticalFOV defines the vertical extent of the entity-sensing field of view. This property is device specific.

Live data properties

The entity-sensing extension adds live data properties to the data-binding context. Live data properties are data objects that can change during the lifecycle of the APL document.

The following table shows the live data objects that the entity-sensing extension adds to the data-binding context.

For details about live data, see Extension live data.

EntitySensingState

The EntitySensingState live data object represents current entity-sensing capabilities of the device.

To access EntitySensingState, you must set the entitySensingStateName property in the extension settings. The EntitySensingState isn't available when entitySensingStateName is missing or contains an empty string.

The following table shows the properties of EntitySensingState.

A complete EntitySensingState data binding context has the structure shown in the following example.

{
  "error": "",
  "errorCode": 0
}

error, errorCode

The EntitySensingState.error and `EntitySensingState.errorCode properties provide information about entity-sensing device errors.

When an error occurs, errorCode reports a nonzero code and error provides descriptive text about the error. When the device is operating without any errors, errorCode returns 0 and error returns an empty string. The error description is device specific.

Errors occur when conditions prevent the device from sensing or the device experiences mechanical failure. Common causes of errors include:

• Physical obstruction
• Low light
• The device is in "Do Not Disturb" mode
• The user closed the camera shutter on the device

The device has limited capabilities when errorCode is nonzero. When the device detects an error, the device periodically checks to see if the errorCode condition has cleared. After the errorCode condition is clear, normal entity-sensing operations resume.

Changes in errorCode values trigger an OnEntitySensingStateChanged event.

The following table shows the errorCode and corresponding error properties that are available on Alexa devices. The error description and errorCode values are device specific. Therefore, a given device might provide additional values.

Use these error values for debugging purposes. The error string isn't localized. Therefore, don't display the error or errorCode value to the user.

PrimaryUser

The PrimaryUser live data object represents the user who is most engaged with the device. The engagement algorithm is device specific.

To access PrimaryUser, you must set the primaryUserName property in the extension settings. The PrimaryUser object isn't available when primaryUserName is missing or contains an empty string.

The following table shows the properties of PrimaryUser.

A sample PrimaryUser data binding context, showing an active user has the following structure.

{
  "id": "Entity1",
  "isActive": true,
  "isSeen": true,
  "poise": {
    "absoluteAngle": 30,
    "relativeAngle": 20
  }
}

The following example shows a sample PrimaryUser data binding context when the device didn't detect a user.

{
  "id": "",
  "isActive": false,
  "isSeen": false,
  "poise": {
    "absoluteAngle": 0,
    "relativeAngle": 0
  }
}

id

The PrimaryUser.id property returns a device-assigned identifier for the user who last spoke the wake word. The property returns an empty string when the device can't detect a user.

A given id persists for a limited amount of time when the device can no longer see the user. The duration of this time is device-specific. When the user leaves the field of view, and then returns to the field of view, the device attempts to recognize the entity as a previously known entity. Then, the device assigns the same id value. If the device doesn't recognize the user, it assigns a new entity ID.

isActive

Returns true if the user has spoken the wake word, and false otherwise.

When the device detects a user and that user hasn't spoken the wake word, the user is considered engaged, but not active.

isSeen

Returns true if the device has detected the user and the user is within the observable range of the device. Returns false if the entity is no longer detected or the entity id is unassigned.

The PrimaryUser.isSeen property also returns false when the EntitySensingState.errorCode is nonzero.

A change in the user isSeen value triggers an onPrimaryUserChanged event.

poise

Returns an object that describes the user's position. The following table shows the properties of the poise object.

The poise.absoluteAngle represents the angular position of the entity relative to the center position of the device. The poise.relativeAngle represents the angular position relative to the screen. Devices that don't rotate the screen return the same angular and relative position.

When the device can't sense the user, the isSeen value returns false, and the poise.absoluteAngle and poise.relativeAngle values represent the last know position of the user. These values are undefined when the entity is unassigned.

Changes in PrimaryUser.poise values trigger an onPrimaryUserChanged event.

Commands

The entity-sensing extension doesn't add any new extension commands.

Event handlers

The entity-sensing extension adds new extension event handlers. To respond to events generated by the entity-sensing extension, add the handler to the top-level properties in your APL document. For the handler name, use AssignedName:EventHandlerName, where AssignedName is the name you assigned the entity-sensing extension in the extensions property and EventHandlerName is the name of the handler.

For example, assuming you named the extension EntitySensing, you would use EntitySensing:OnEntitySensingStateChanged to respond to events from the OnEntitySensingStateChanged handler.

OnEntitySensingStateChanged

Invoked when the properties in EntitySensingState have changed. Changes to any EntitySensingState properties trigger an OnEntitySensingStateChanged event.

The following table shows the properties that the OnEntitySensingStateChanged handler adds the event context:

For the list of EntitySensingState properties, see EntitySensingState.

The following example shows the form of the event generated.

{
  "event": {
    "changed": {
        // All of the changed state properties
    },
    "current": {
        // All of the current state properties
    },
    "source": {
      "type": "Document",
      "handler": "OnEntitySensingStateChanged",
      "id": null,
      "uid": null,
      "value": null
    }
  }
}

The commands in this event handler always run in fast mode.

OnPrimaryUserChanged

Invoked when any of the following PrimaryUser changes occur:

• Changes to any PrimaryUser properties, except for poise.
• The device detects and assigns a new PrimaryUser.
• The device no longer detects the existing PrimaryUser.

The following table shows the properties that the OnPrimaryUserChanged handler adds to the event context.

The following example shows the form of the event generated.

{
  "event": {
    "changed": {
        // Only the changed properties
    },
    "current": {
        // All of the current properties
    },
    "source": {
      "type": "Document",
      "handler": "OnPrimaryUserChanged",
      "id": null,
      "uid": null,
      "value": null
    }
  }
}

Image filters

The entity-sensing extension doesn't add any new image filters.

Related topics

• APL Extensions
• APL smart-motion extension

Last updated: Nov 28, 2023