Natural Language Understanding Evaluation REST API Reference
Use the Natural Language Understanding (NLU) Evaluation REST API to test the accuracy of the interaction model that you defined with your skill. NLU evaluations assess the accuracy of your skill against an annotation set. To create an annotation set, see NLU Annotation Set REST API Reference. After your annotation set passes the evaluation, rerun the evaluation when you make changes to your interaction model to make sure that your changes don't degrade the accuracy.
If you want to use the NLU Evaluation tool in the Alexa developer console, see Batch Test Your NLU Model.
API endpoint
The endpoint of the NLU Evaluation API is https://api.amazonalexa.com
.
Authentication
Each API request must have an authorization header whose value is a bearer token retrieved from Login with Amazon (LWA). For details, see Get an Access Token for SMAPI.
Operations
The NLU Evaluation API includes the following operations.
Operation | HTTP method and URI |
---|---|
| |
| |
| |
|
Get evaluation results
Get the details results of the specified NLU evaluation. If you want the status of the evaluation only, use Get NLU evaluation status.
Request
To get the results of an evaluation, you make a GET
request to the nluEvaluations
resource.
Request path and header example
GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}?locale={locale}&sort.field={sortField}&testCaseStatus={testCaseStatus}&actualIntentName={actualIntentName}&expectedIntentName={expectedIntentName}&nextToken={nextToken}&maxResults={maxResults}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
Request path and header parameters
Parameter | Located in | Description | Type | Required |
---|---|---|---|---|
|
Path |
Identifies the skill. |
String |
Yes |
|
Path |
Identifies the NLU evaluation. |
String |
Yes |
|
Query |
Sort the result on the specified field in alphabetical order. |
String |
No |
|
Query |
Filter on the test case status. |
String |
No |
|
Query |
Filter results that resolve to the specified intent name. |
String |
No |
|
Query |
Filter results on the expected intent name. |
String |
No |
|
Query |
Token from the previous response. |
String |
No |
|
Query |
Maximum number of results to return in the response. |
Number |
No |
|
Header |
String |
Yes |
Request body example
The request has no body.
Request body properties
The request has no body.
Response
A successful response returns HTTP 200 OK
, along with the evaluation results.
On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.
Response body example
{
"totalFailed": 0,
"testCases": [{
"status": "PASSED",
"inputs": {
"utterance": "i want to go to chicago on monday",
"referenceTimestamp": "2019-08-12T18:46:31.674Z"
},
"actual": {
"domain": "string",
"intent": {
"name": "PlanMyTripIntent",
"confirmationStatus": "NONE",
"slots": {
"toCity": {
"name": "toCitySlot",
"value": "chicago",
"confirmationStatus": "NONE",
"resolutions": {
"resolutionsPerAuthority": [{
"resolution1": {
"authority": "string",
"status": {
"code": "ER_SUCCESS_MATCH"
},
"values": [{
"name": "string",
"id": "string"
}]
}
}]
}
}
}
}
},
"expected": [{
"domain": "string",
"intent": {
"name": "PlanMyTripIntent",
"slots": {
"toCity": {
"value": "chicago"
}
}
}
}]
}],
"paginationContext": {
"nextToken": "someToken.1",
"totalCount": "1"
},
"_links": {
"self": {
"href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations"
},
"next": {
"href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations?nextToken=token.1&maxResults=100"
}
}
}
Response body properties
Property | Description | Type |
---|---|---|
|
Number of test cases that failed. A test fails when the utterance doesn't resolve to the expected intent and expected slots. |
Integer |
|
Array of test cases runs based on the filter criteria. |
Array of objects |
|
Indicates the status of the test case. |
String |
|
Inputs to the test case. |
Object |
|
Utterance to test. |
String |
|
Date and time to use as the basis for relative date and time values. Include when the utterance tests. Use this when the utterance tests |
String |
|
Actual results of the test case. |
Object |
|
Resolved domain name. |
String |
|
Resolved intent. |
Object |
|
Name of the intent. |
String |
|
Indicates whether the user confirmed or denied the intent. |
String |
|
List of expected slots for the utterance. |
Array of objects |
|
Name of the slot. There might be one or more slots. |
Object |
|
Slot name. |
String |
|
Slot value. |
String |
|
Indicates whether the user confirmed or denied the slot. |
String |
|
Represents the results of resolving the words captured from the utterance. |
Object |
|
Identifies the authority for the slot values. |
Object |
|
Name of the authority for slot values. For custom slot types, the label incorporates the skill ID and slot type name. |
String |
|
Indicates whether the spoken value matched a value or synonym defined in the slot type. |
Object |
|
Status of the slot resolution. |
Object |
|
List of resolved values for the slot. |
Array of objects |
|
Resolved slot value. |
String |
|
Unique ID for the resolved slot value, based on the IDs in the slot type definition. |
String |
|
Expected results of the test case. |
Object |
|
Expected domain name. |
String |
|
Intent that the utterance should trigger. |
String |
|
List of expected slots for the utterance. |
Array of objects |
|
Name of the slot. There might be one or more slots. |
Object |
|
Slot value. |
String |
|
Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, |
Object |
|
Identifies the next set of evaluation results to return. The token expires in 24 hours. |
String |
|
Links for result navigation. |
_links object |
HTTP status codes
Status | Description |
---|---|
|
Response body contains a list of evaluation results. |
|
Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.
|
|
Authorization token is invalid, expired, or doesn't have access to the resource. |
|
The requested operation isn't allowed. |
|
The requested resource isn't found. For example, the
|
|
Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress. |
|
User has exceeded the permitted rate limit. |
|
Server has encountered an error. |
|
Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request. |
Get evaluation status
Get the status of the specified NLU evaluation. If you want the detailed results of the evaluation, use Get evaluation results.
Request
To get the status of an evaluation, you make a GET
request to the nluEvaluations
resource.
Request path and header example
GET /v1/skills/{skillId}/nluEvaluations/{evaluationId}/status
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
Request path and header parameters
Parameter | Located in | Description | Type | Required |
---|---|---|---|---|
|
Path |
Identifies the skill. |
String |
Yes |
|
Path |
Identifies the NLU evaluation. |
String |
Yes |
|
Header |
String |
Yes |
Request body example
The request has no body.
Request body properties
The request has no body.
Response
A successful response returns HTTP 200 OK
, along with the evaluation status.
On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.
Response body example
{
"startTimestamp": "2019-08-12T18:41:09.684Z",
"endTimestamp": "2019-08-12T18:41:09.684Z",
"status": "string",
"errorMessage": "string",
"inputs": {
"locale": "en-US",
"stage": "development",
"source": {
"annotationId": "annotation.set.1"
}
},
"_links": {
"results": {
"href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations"
}
}
}
Response body properties
Property | Description | Type |
---|---|---|
|
Indicates the status of the evaluation. A failure status indicates that the evaluation didn't start. |
String |
|
Start time of the evaluations. |
String |
|
Time the evaluation completed. |
String |
|
(Optional) Human-readable error message included when evaluation fails. |
String |
|
Request payload that triggered this evaluation. |
Object |
|
Country and language where you offer your skill. |
String |
|
Indicates stage of the skill. |
String |
|
Identifies the annotation set to use in the evaluation. |
Object |
|
Identifies the annotation set. |
String |
|
Links for result navigation. |
_links object |
HTTP status codes
Status | Description |
---|---|
|
Response body contains the evaluation status. |
|
Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.
|
|
Authorization token is invalid, expired, or doesn't have access to the resource. |
|
The requested operation isn't allowed. |
|
The requested resource isn't found. For example, the
|
|
Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress. |
|
User has exceeded the permitted rate limit. |
|
Server has encountered an error. |
|
Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request. |
List evaluations
Get all completed NLU evaluations for the specified skill. You can add optional filters to the request.
Request
To list NLU evaluations, you make a GET
request to the nluEvaluations
resource.
Request path and header example
GET /v1/skills/{skillId}/nluEvaluations?locale={locale}&annotationId={annotationId}&stage={stage}&nextToken={nextToken}&maxResults={maxResults}
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
Request path and header parameters
Parameter | Located in | Description | Type | Required |
---|---|---|---|---|
|
Path |
Identifies the skill. |
String |
Yes |
|
Query |
Filter on the country and language where you offer the skill. If omitted, the response includes all locales. |
String |
No |
|
Query |
Filters results to the specified annotation set. If omitted, the response includes all annotation sets. |
String |
No |
|
Query |
Filter on the specified skill stage. If omitted, the response includes all stages. |
String |
No |
|
Query |
Token from the previous response. |
String |
No |
|
Query |
Maximum number of results to return in the response. |
Number |
No |
|
Header |
String |
Yes |
Request body example
The request has no body.
Request body properties
The request has no body.
Response
A successful response returns HTTP 200 OK
, along with a list of evaluations.
On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.
Response body example
{
"evaluations": [{
"startTimestamp": "2019-08-12T18:41:09.684Z",
"endTimestamp": "2019-08-12T18:41:09.684Z",
"status": "string",
"errorMessage": "string",
"inputs": {
"locale": "en-US",
"stage": "development",
"source": {
"annotationId": "annotation.set.1"
}
},
"id": "evaluation.1"
}],
"paginationContext": {
"nextToken": "someToken.1"
},
"_links": {
"self": {
"href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations"
},
"next": {
"href": "/v1/skills/amzn1.ask.skill.1/nluEvaluations?nextToken=token.1&maxResults=100"
}
}
}
Response body properties
Property | Description | Type |
---|---|---|
|
Array of evaluation runs based on the filter criteria. |
Array of objects |
|
Identifies the evaluation run. |
String |
|
Indicates the status of the evaluation. A failure status indicates that the evaluation didn't start. |
String |
|
Start time of the evaluations. |
String |
|
Time the evaluation completed. |
String |
|
(Optional) Human-readable error message included when evaluation fails. |
String |
|
Request payload that triggered this evaluation. |
Object |
|
Country and language where the skill is offered. |
String |
|
Indicates stage of the skill. |
String |
|
Identifies the annotation set to be used in the evaluation. |
Object |
|
Identifies the annotation set. |
String |
|
Indicates there are more matching results. If present, the response contains a subset of the results. When there are no more matching results, |
Object |
|
Identifies the next set of evaluation results to return. The token expires in 24 hours. |
String |
|
Number of test cases that matched the request parameters. |
String |
|
Links for result navigation. |
_links object |
HTTP status codes
Status | Description |
---|---|
|
Response body contains a list of evaluation runs. |
|
Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.
|
|
Authorization token is invalid, expired, or doesn't have access to the resource. |
|
The requested operation isn't allowed. |
|
The requested resource isn't found. For example, the
|
|
Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress. |
|
User has exceeded the permitted rate limit. |
|
Server has encountered an error. |
|
Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request. |
Run NLU evaluation
Start an NLU evaluation against an existing annotation set. Concurrent evaluation requests for a given skill id aren't supported.
Request
To run an evaluation, you make a POST
request to the nluEvaluations
resource.
Request path and header example
POST /v1/skills/{skillId}/nluEvaluations
Host: api.amazonalexa.com
Content-Type: application/json
Authorization: Bearer {access token}
Accept: application/json
Request path and header parameters
Parameter | Located in | Description | Type | Required |
---|---|---|---|---|
|
Path |
Identifies the skill. |
String |
Yes |
|
Header |
String |
Yes |
Request body example
{
"stage": "live",
"locale": "en-US",
"source": {
"annotationId": "annotation.set.1"
}
}
Request body properties
Property | Description | Type | Required |
---|---|---|---|
|
Indicates stage of the skill. |
String |
Yes |
|
Country and language where the skill is offered. |
String |
Yes |
|
Identifies the annotation set to be used in the evaluation. |
Object |
Yes |
|
Identifies the annotation set. |
String |
Yes |
Response
A successful response returns HTTP 200 OK
with a Location
parameter in the response header. The response body contains the evaluation ID. Use the ID to get the status and results of the evaluation.
On error, the response returns the appropriate HTTP status code and includes a response body with an error code and human readable message.
Response body example
{
"id": "evaluation.1"
}
Response body properties
Property | Description | Type |
---|---|---|
|
Identifies the evaluation run. |
String |
HTTP status codes
Status | Description |
---|---|
|
Empty annotation set created successfully. The response header contains a
|
|
Server can't process the request due to a client error. For example, the payload is malformed, the annotation set is malformed or empty.
|
|
Authorization token is invalid, expired, or doesn't have access to the resource. |
|
The requested operation isn't allowed. |
|
The requested resource isn't found. For example, the
|
|
Request to run an evaluation for a given skill conflicts with an evaluation request that's currently in progress. |
|
User has exceeded the permitted rate limit. |
|
Server has encountered an error. |
|
Server is down for maintenance, overloaded, or otherwise unavailable to handle the incoming request. |
Object definitions
The NLU Evaluation API defines the following objects.
Locale values
The following table shows valid values for the locale
property.
Locale code | Language |
---|---|
|
Arabic (SA) |
|
German (DE) |
|
English (AU) |
|
English (CA) |
|
English (UK) |
|
English (IN) |
|
English (US) |
|
Spanish (ES) |
|
Spanish (MX) |
|
Spanish (US) |
|
French (CA) |
|
French (FR) |
|
Hindi (IN) |
|
Italian (IT) |
|
Japanese (JP) |
|
Dutch (NL) |
|
Portuguese (BR) |
Related topics
Last updated: Aug 01, 2024