Natural Language Understanding Evaluation REST API Reference

Note: Sign in to the developer console to build or publish your skill.

Use the Natural Language Understanding (NLU) Evaluation REST API to test the accuracy of the interaction model that you defined with your skill. NLU evaluations assess the accuracy of your skill against an annotation set. To create an annotation set, see NLU Annotation Set REST API Reference. After your annotation set passes the evaluation, rerun the evaluation when you make changes to your interaction model to make sure that your changes don't degrade the accuracy.

If you want to use the NLU Evaluation tool in the Alexa developer console, see Batch Test Your NLU Model.

API endpoint

The endpoint of the NLU Evaluation API is https://api.amazonalexa.com.

Authentication

Each API request must have an authorization header whose value is a bearer token retrieved from Login with Amazon (LWA). For details, see Get an Access Token for SMAPI.

Operations

The NLU Evaluation API includes the following operations.

Operation	HTTP method and URI
Get evaluation results	`GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}?locale={locale}&sort.field={sortField}&testCaseStatus={testCaseStatus}&actualIntentName={actualIntentName}&expectedIntentName={expectedIntentName}&nextToken={nextToken}&maxResults={maxResults}`
Get evaluation status	`GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}/status`
List evaluations	`GET /v1/skills/{skillId}/nluEvaluations?locale={locale}&annotationId={annotationId}&stage={stage}&nextToken={nextToken}&maxResults={maxResults}`
Run NLU evaluation	`POST /v1/skills/{skillId}/nluEvaluations`

Get evaluation results

Get the details results of the specified NLU evaluation. If you want the status of the evaluation only, use Get NLU evaluation status.

Request

To get the results of an evaluation, you make a GET request to the nluEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}?locale={locale}&sort.field={sortField}&testCaseStatus={testCaseStatus}&actualIntentName={actualIntentName}&expectedIntentName={expectedIntentName}&nextToken={nextToken}&maxResults={maxResults}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter	Located in	Description	Type	Required
`skillId`	Path	Identifies the skill. Valid values: 1–255 characters.	String	Yes
`evaluationId`	Path	Identifies the NLU evaluation.	String	Yes
`sortField`	Query	Sort the result on the specified field in alphabetical order. Valid values: `STATUS`, `ACTUAL_INTENT`, `EXPECTED_INTENT`.	String	No
`testCaseStatus`	Query	Filter on the test case status. Valid values: `PASSED`, `FAILED`.	String	No
`actualIntentName`	Query	Filter results that resolve to the specified intent name.	String	No
`expectedIntentName`	Query	Filter results on the expected intent name.	String	No
`nextToken`	Query	Token from the previous response. Include if iterating over a paginated response.	String	No
`maxResults`	Query	Maximum number of results to return in the response. Default: 1000.	Number	No
`access token`	Header	LWA token.	String	Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the evaluation results. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{

    "totalFailed": 0,
    "testCases": [{
        "status": "PASSED",
        "inputs": {
            "utterance": "i want to go to chicago on monday",
            "referenceTimestamp": "2019-08-12T18:46:31.674Z"
        },
        "actual": {
            "domain": "string",
            "intent": {
                "name": "PlanMyTripIntent",
                "confirmationStatus": "NONE",
                "slots": {
                    "toCity": {
                        "name": "toCitySlot",
                        "value": "chicago",
                        "confirmationStatus": "NONE",
                        "resolutions": {
                            "resolutionsPerAuthority": [{
                                "resolution1": {
                                    "authority": "string",
                                    "status": {
                                        "code": "ER_SUCCESS_MATCH"
                                    },
                                    "values": [{
                                        "name": "string",
                                        "id": "string"
                                    }]
                                }
                            }]
                        }
                    }
                }
            }
        },
        "expected": [{
            "domain": "string",
            "intent": {
                "name": "PlanMyTripIntent",
                "slots": {
                    "toCity": {
                        "value": "chicago"
                    }
                }
            }
        }]
    }],
    "paginationContext": {
        "nextToken": "someToken.1",
        "totalCount": "1"
    },
    "_links": {
        "self": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations"
        },
        "next": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations?nextToken=token.1&maxResults=100"
        }
    }
}

Response body properties

Property	Description	Type
`totalFailed`	Number of test cases that failed. A test fails when the utterance doesn't resolve to the expected intent and expected slots.	Integer
`testCases`	Array of test cases runs based on the filter criteria.	Array of objects
`testCases[].status`	Indicates the status of the test case. Valid values: `PASSED`, `FAILED`.	String
`testCases[].inputs`	Inputs to the test case.	Object
`testCases[].inputs.utterance`	Utterance to test.	String
`testCases[].inputs.referenceTimestamp`	Date and time to use as the basis for relative date and time values. Include when the utterance tests. Use this when the utterance tests `AMAZON.DATE` or `AMAZON.TIME` slots with words that represent relative dates and times, such as "today," "tomorrow," and "now." Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String
`actual`	Actual results of the test case.	Object
`actual.domain`	Resolved domain name.	String
`actual.intent`	Resolved intent.	Object
`actual.intent.name`	Name of the intent.	String
`actual.intent.confirmationStatus`	Indicates whether the user confirmed or denied the intent. Valid values: `NONE`, `CONFIRMED`, `DENIED`.	String
`actual.intent.slots`	List of expected slots for the utterance.	Array of objects
`actual.intent.slots.<slotname>`	Name of the slot. There might be one or more slots.	Object
`actual.intent.slots.<slotname>.slotValue.name`	Slot name.	String
`actual.intent.slots.<slotname>.slotValue.value`	Slot value.	String
`actual.intent.slots.<slotname>.confirmationStatus`	Indicates whether the user confirmed or denied the slot. Valid values: `NONE`, `CONFIRMED`, `DENIED`.	String
`actual.intent.slots.<slotname>.resolutions`	Represents the results of resolving the words captured from the utterance.	Object
`actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority`	Identifies the authority for the slot values.	Object
`actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.authority`	Name of the authority for slot values. For custom slot types, the label incorporates the skill ID and slot type name.	String
`actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.status`	Indicates whether the spoken value matched a value or synonym defined in the slot type.	Object
`actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.status.code`	Status of the slot resolution. Valid values: `ER_SUCCESS_MATCH`, `ER_SUCCESS_NO_MATCH`, `ER_ERROR_TIMEOUT`, `ER_ERROR_EXCEPTION`.	Object
`actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.values`	List of resolved values for the slot.	Array of objects
`actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.values[].name`	Resolved slot value.	String
`actual.intent.slots.<slotname>.resolutions.resolutionsPerAuthority.values[].id`	Unique ID for the resolved slot value, based on the IDs in the slot type definition.	String
`expected`	Expected results of the test case.	Object
`expected.domain`	Expected domain name.	String
`expected.intent.name`	Intent that the utterance should trigger.	String
`expected.intent.slots`	List of expected slots for the utterance.	Array of objects
`expected.intent.slots.<slotname>`	Name of the slot. There might be one or more slots.	Object
`actual.intent.slots.<slotname>.slotValue.value`	Slot value.	String
`paginationContext`	Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, `paginationContext` isn't included in the response.	Object
`paginationContext.nextToken`	Identifies the next set of evaluation results to return. The token expires in 24 hours. Use this value for the `nextToken` parameter in a subsequent Get NLU evaluation results request.	String
`_links`	Links for result navigation. Defined in JSON Hypertext Application Language specification.	_links object

HTTP status codes

Status	Description
`200 OK`	Response body contains a list of evaluation results.
`400 Bad Request`	Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty. The response body contains a `violations` array with the errors. The following example shows the response body with a validation error. `{ "message": "Payload validations failed", "violations": [{ "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.", "code": "INVALID_STRING_LENGTH" }] }`
`401 Unauthorized`	Authorization token is invalid, expired, or doesn't have access to the resource.
`403 Forbidden`	The requested operation isn't allowed. The response body contains a `violations` array with the errors.
`404 Not Found`	The requested resource isn't found. For example, the `skillId` isn't found. The following example shows the response body with the error and message. `{ "message": "The skill cannot be found.", "code": "RESOURCE_NOT_FOUND" }`
`409 Conflict`	Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.
`429 Too Many Requests`	User has exceeded the permitted rate limit.
`500 Internal Server Error`	Server has encountered an error.
`503 Service Unavailable`	Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Get evaluation status

Get the status of the specified NLU evaluation. If you want the detailed results of the evaluation, use Get evaluation results.

Request

To get the status of an evaluation, you make a GET request to the nluEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}/status
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter	Located in	Description	Type	Required
`skillId`	Path	Identifies the skill. Valid values: 1–255 characters.	String	Yes
`evaluationId`	Path	Identifies the NLU evaluation.	String	Yes
`access token`	Header	LWA token.	String	Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with the evaluation status. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "startTimestamp": "2019-08-12T18:41:09.684Z",
    "endTimestamp": "2019-08-12T18:41:09.684Z",
    "status": "string",
    "errorMessage": "string",
    "inputs": {
        "locale": "en-US",
        "stage": "development",
        "source": {
            "annotationId": "annotation.set.1"
        }
    },
    "_links": {
        "results": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations"
        }
    }
}

Response body properties

Property	Description	Type
`status`	Indicates the status of the evaluation. A failure status indicates that the evaluation didn't start. Valid values: `PASSED`, `FAILED`, `IN_PROGRESS`, `ERROR`.	String
`startTimestamp`	Start time of the evaluations. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String
`endTimestamp`	Time the evaluation completed. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String
`errorMessage`	(Optional) Human-readable error message included when evaluation fails.	String
`inputs`	Request payload that triggered this evaluation.	Object
`inputs.locale`	Country and language where you offer your skill.	String
`inputs.stage`	Indicates stage of the skill. Valid values: `development`, `live`.	String
`inputs.source`	Identifies the annotation set to use in the evaluation.	Object
`inputs.source.annotationId`	Identifies the annotation set.	String
`_links`	Links for result navigation. Defined in JSON Hypertext Application Language specification.	_links object

HTTP status codes

Status	Description
`200 OK`	Response body contains the evaluation status.
`400 Bad Request`	Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty. The response body contains a `violations` array with the errors. The following example shows the response body with a validation error. `{ "message": "Payload validations failed", "violations": [{ "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.", "code": "INVALID_STRING_LENGTH" }] }`
`401 Unauthorized`	Authorization token is invalid, expired, or doesn't have access to the resource.
`403 Forbidden`	The requested operation isn't allowed. The response body contains a `violations` array with the errors.
`404 Not Found`	The requested resource isn't found. For example, the `skillId` isn't found. The following example shows the response body with the error and message. `{ "message": "The skill cannot be found.", "code": "RESOURCE_NOT_FOUND" }`
`409 Conflict`	Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.
`429 Too Many Requests`	User has exceeded the permitted rate limit.
`500 Internal Server Error`	Server has encountered an error.
`503 Service Unavailable`	Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

List evaluations

Get all completed NLU evaluations for the specified skill. You can add optional filters to the request.

Request

To list NLU evaluations, you make a GET request to the nluEvaluations resource.

Request path and header example

Copied to clipboard.

GET /v1/skills/{skillId}/nluEvaluations?locale={locale}&annotationId={annotationId}&stage={stage}&nextToken={nextToken}&maxResults={maxResults}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter	Located in	Description	Type	Required
`skillId`	Path	Identifies the skill. Valid values: 1–255 characters.	String	Yes
`locale`	Query	Filter on the country and language where you offer the skill. If omitted, the response includes all locales.	String	No
`annotationId`	Query	Filters results to the specified annotation set. If omitted, the response includes all annotation sets.	String	No
`stage`	Query	Filter on the specified skill stage. If omitted, the response includes all stages. Valid values: `development`, `live`.	String	No
`nextToken`	Query	Token from the previous response. Include if iterating over a paginated response.	String	No
`maxResults`	Query	Maximum number of results to return in the response. Valid values: 1 – 100. Default: 10.	Number	No
`access token`	Header	LWA token.	String	Yes

Request body example

The request has no body.

Request body properties

The request has no body.

Response

A successful response returns HTTP 200 OK, along with a list of evaluations. On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "evaluations": [{
        "startTimestamp": "2019-08-12T18:41:09.684Z",
        "endTimestamp": "2019-08-12T18:41:09.684Z",
        "status": "string",
        "errorMessage": "string",
        "inputs": {
            "locale": "en-US",
            "stage": "development",
            "source": {
                "annotationId": "annotation.set.1"
            }
        },
        "id": "evaluation.1"
    }],
    "paginationContext": {
        "nextToken": "someToken.1"
    },
    "_links": {
        "self": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations"
        },
        "next": {
            "href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations?nextToken=token.1&maxResults=100"
        }
    }
}

Response body properties

Property	Description	Type
`evaluations`	Array of evaluation runs based on the filter criteria.	Array of objects
`evaluations[].id`	Identifies the evaluation run.	String
`evaluations[].status`	Indicates the status of the evaluation. A failure status indicates that the evaluation didn't start. Valid values: `PASSED`, `FAILED`, `IN_PROGRESS`, `ERROR`.	String
`evaluations[].startTimestamp`	Start time of the evaluations. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String
`evaluations[].endTimestamp`	Time the evaluation completed. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String
`evaluations[].errorMessage`	(Optional) Human-readable error message included when evaluation fails.	String
`evaluations[].inputs`	Request payload that triggered this evaluation.	Object
`evaluations[].inputs.locale`	Country and language where the skill is offered.	String
`evaluations[].inputs.stage`	Indicates stage of the skill. Valid values: `development`, `live`.	String
`evaluations[].inputs.source`	Identifies the annotation set to be used in the evaluation.	Object
`evaluations[].inputs.source.annotationId`	Identifies the annotation set.	String
`paginationContext`	Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, `paginationContext` isn't included in the response.	Object
`paginationContext.nextToken`	Identifies the next set of evaluation results to return. The token expires in 24 hours. Use this value for the `nextToken` parameter in a subsequent Get NLU evaluation results request.	String
`paginationContext.totalCount`	Number of test cases that matched the request parameters.	String
`_links`	Links for result navigation. Defined in JSON Hypertext Application Language specification.	_links object

HTTP status codes

Status	Description
`200 OK`	Response body contains a list of evaluation runs.
`400 Bad Request`	Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty. The response body contains a `violations` array with the errors. The following example shows the response body with a validation error. `{ "message": "Payload validations failed", "violations": [{ "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.", "code": "INVALID_STRING_LENGTH" }] }`
`401 Unauthorized`	Authorization token is invalid, expired, or doesn't have access to the resource.
`403 Forbidden`	The requested operation isn't allowed. The response body contains a `violations` array with the errors.
`404 Not Found`	The requested resource isn't found. For example, the `skillId` isn't found. The following example shows the response body with the error and message. `{ "message": "The skill cannot be found.", "code": "RESOURCE_NOT_FOUND" }`
`409 Conflict`	Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.
`429 Too Many Requests`	User has exceeded the permitted rate limit.
`500 Internal Server Error`	Server has encountered an error.
`503 Service Unavailable`	Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Run NLU evaluation

Start an NLU evaluation against an existing annotation set. Concurrent evaluation requests for a given skill id aren't supported.

Request

To run an evaluation, you make a POST request to the nluEvaluations resource.

Request path and header example

Copied to clipboard.

POST /v1/skills/{skillId}/nluEvaluations
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json

Request path and header parameters

Parameter	Located in	Description	Type	Required
`skillId`	Path	Identifies the skill. Valid values: 1–255 characters.	String	Yes
`access token`	Header	LWA token.	String	Yes

Request body example

Copied to clipboard.

{
    "stage": "live",
    "locale": "en-US",
    "source": {
        "annotationId": "annotation.set.1"
    }
}

Request body properties

Property	Description	Type	Required
`stage`	Indicates stage of the skill. Valid values: `development`, `live`.	String	Yes
`locale`	Country and language where the skill is offered. For details, seeLocale.	String	Yes
`source`	Identifies the annotation set to be used in the evaluation.	Object	Yes
`source.annotationId`	Identifies the annotation set. Use Create annotation set or List annotation sets to get the ID.	String	Yes

Response

A successful response returns HTTP 200 OK with a Location parameter in the response header. The response body contains the evaluation ID. Use the ID to get the status and results of the evaluation.

On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.

Response body example

{
    "id": "evaluation.1"
}

Response body properties

Property	Description	Type
`id`	Identifies the evaluation run.	String

HTTP status codes

Status	Description
`200 OK`	Empty annotation set created successfully. The response header contains a `Location` parameter that includes the `skillId` and `annotationId`. The `annotationId` is the same as the `id` in the response body. For example: `HTTP 200 OK Content-Type: application/json Location: v1/skills/{skillId}/nluEvaluations/{evaluationId}/status`
`400 Bad Request`	Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty. The response body contains a `violations` array with the errors. The following example shows the response body with a validation error. `{ "message": "Payload validations failed", "violations": [{ "message": "The \"name\" property at property path \"$.name\" is outside the allowed range.", "code": "INVALID_STRING_LENGTH" }] }`
`401 Unauthorized`	Authorization token is invalid, expired, or doesn't have access to the resource.
`403 Forbidden`	The requested operation isn't allowed. The response body contains a `violations` array with the errors.
`404 Not Found`	The requested resource isn't found. For example, the `skillId` isn't found. The following example shows the response body with the error and message. `{ "message": "The skill cannot be found.", "code": "RESOURCE_NOT_FOUND" }`
`409 Conflict`	Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress.
`429 Too Many Requests`	User has exceeded the permitted rate limit.
`500 Internal Server Error`	Server has encountered an error.
`503 Service Unavailable`	Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request.

Object definitions

The NLU Evaluation API defines the following objects.

Locale values

The following table shows valid values for the locale property.

Locale code	Language
`ar-SA`	Arabic (SA)
`de-DE`	German (DE)
`en-AU`	English (AU)
`en-CA`	English (CA)
`en-GB`	English (UK)
`en-IN`	English (IN)
`en-US`	English (US)
`es-ES`	Spanish (ES)
`es-MX`	Spanish (MX)
`es-US`	Spanish (US)
`fr-CA`	French (CA)
`fr-FR`	French (FR)
`hi-IN`	Hindi (IN)
`it-IT`	Italian (IT)
`ja-JP`	Japanese (JP)
`nl-NL`	Dutch (NL)
`pt-BR`	Portuguese (BR)

Was this page helpful?

Provide feedback

Last updated: Aug 01, 2024

Natural Language Understanding Evaluation REST API Reference

API endpoint

Authentication

Operations

Get evaluation results

Request

Request path and header example

Request path and header parameters

Request body example

Request body properties

Response

Response body example

Response body properties

HTTP status codes

Get evaluation status

Request

Request path and header example

Request path and header parameters

Request body example

Request body properties

Response

Response body example

Response body properties

HTTP status codes

List evaluations

Request

Request path and header example

Request path and header parameters

Request body example

Request body properties

Response

Response body example

Response body properties

HTTP status codes

Run NLU evaluation

Request

Request path and header example

Request path and header parameters

Request body example

Request body properties

Response

Response body example

Response body properties

HTTP status codes

Object definitions

Locale values

Related topics

Was this page helpful?