Developer Console
Sign In
Sign In
Developer Console

Optional: Use Page Collation

 Catalog Ingestion

STEP 1:
Create your Catalog File
→
STEP 2:
Validate Your Catalog File
→
STEP 3:
Set Up Your AWS Account
→
STEP 4:
Upload Your Catalog File to Amazon
→
STEP 5:
Verify Your Uploaded Catalog File
→
STEP 6:
Update your Catalog at Regular Intervals
 
Optional:
Use Page Collation

Page collation gives you the flexibility to submit your catalog in smaller parts rather than submitting one large catalog. With page collation, you can express an entire CDF catalog as separate XML files rather than containing all content in one XML file. To do this, submit individual pages as XML files, along with a manifest that lists the included pages. The manifest requests that the ingestion system combines the pages into one CDF catalog.

Use the guidance on this page to learn how to use page collation for your catalog.

Page collation process

Submit your pages to your S3 bucket under the pages folder, and submit the manifests in the manifests folder. These folders are separate from the catalogs folder in your S3 bucket. When page collation completes, the ingestion system submits the final catalog to the catalogs folder to start an ingestion job.

Pages folder

The pages folder contains the individual pages you want to combine. Each page should be a syntactically valid CDF file with a subset of work items you want to include. Because the end result of page collation is a fully merged CDF file created by Amazon, there are no restrictions on splitting references of items between pages. For example, if on one page, a show references a related episode that is on another page, this doesn't cause issues as long as the overall collection of pages contain all the items referenced. All validation requirements, such as ID uniqueness, referencing, and image requirements, still apply to the set of pages as if a single full catalog file was ingested.

Amazon recommends that you keep the page names unique. You can do this with UUIDs, timestamps, or other methods. Page names can be reused and referenced across different manifests. To make an update to the content of a page, you must upload a new XML file for the page. To prevent a page that is still in use from being cleaned up by the 30-day lifecycle policy, the page collation job refreshes the page metadata if a newer manifest has referenced it.

Manifests folder

The manifests folder listens for the creation of any manifest object in the folder. When you upload a manifest file, it triggers an event to start the page collation job. In the manifest file, list the names of the page files under the pages folder and separate each file name with a new line. Manifest files must have a .manifest extension. Amazon recommends that you use unique names for manifest files following the same naming convention you use for pages.

Folder structure

Your S3 bucket will have the following folder structure.

s3://cdf-{provider-name}/
-> reports/
   -> report.html             // Generated from latest catalog submitted.
    
-> catalogs/
   -> catalog.xml             // Provider submitted catalog.
   -> full_catalog1.xml.zst   // Created by the manifest1.manifest request.
   -> full_catalog2.xml.zst   // Created by the manifest2.manifest request.
   
-> pages/
    -> page1.xml
    -> page2.xml
    -> page3.xml
   
 -> manifests/
   -> manifest1.manifest      // Contains page1.xml, page2.xml.
   -> manifest2.manifest      // Contains page1.xml, page2.xml, page3.xml.

Conditions and policies

The following conditions and policies apply when you use page collation for your catalog.

  • Page size: you can submit a page with a size of up to 250 MB, in compressed or uncompressed format.
  • Supported compression formats: you can compress pages with Zstd or Gzip format.
  • Maximum allowed pages: up to 10,000 pages are allowed per manifest.
  • Page reuse: pages may be reused by new manifests.
  • Overwrite policy: overwriting is allowed under the catalogs folder, but page and manifest objects must not be overwritten. New manifests and pages must be unique, or the job will fail.
  • Expiration policy: pages and manifests expire and are removed from the bucket after 30 days. If a manifest uses a page, it will renew the lifecycle on the page so that it doesn't expire.

Feedback reports

If a page collation job fails, the job generates a report and adds it under the reports folder. If there are no failures, the ingestion job follows the standard process—it creates a CDF file, evaluates the file, and generates a report.

The following image shows an example feedback report when the page collation job fails.

Result: Failed. Ingestion failed because of the issues found in the catalog. Please refer to errors for details. Two errors: Max size exceeded, Page get error.

Flow diagrams

These diagrams show the flow of events for several scenarios you might encounter while using page collation. The first diagram shows the flow when you (the provider) submit pages and trigger the start of a page collation job.

Expand text description.
Page collation job flow

The next diagram shows the flow for adding a new page to an existing catalog.

Expand text description.
Add a new page to an existing catalog

The last diagram shows the flow when there is a failure during page collation.

Expand text description.
Page collation failure

Example

The following is an example of a manifest, the pages it references, and the final construction of the CDF file before Amazon ingests it.

You can download the example files here.

Manifest

File name: manifest1.manifest

File content:

pages/page1.xml
pages/page2.xml
pages/page3.xml

Pages

File name: page1.xml

File content:

<?xml version="1.0" encoding="utf-8"?>
<Catalog xmlns="http://www.amazon.com/FireTv/2014-04-11/ingestion" version="FireTv-v1.3">
    <Partner>partner</Partner>
    <Works>
        <TvEpisode>
            <ID>EPISODE_1</ID>
            <Title locale="en-US">Hello Doggo</Title>
            <Synopsis locale="en-US">Say hello to my little doggo</Synopsis>
            <Language>en-US</Language>
            <Genres>
                <Genre locale="en-US">Comedy</Genre>
            </Genres>
            <!-- ... rest of CDF data -->
            <ShowID>SHOW_1</ShowID>
            <SeasonID>SERIES_2</SeasonID>
            <EpisodeInSeason>1</EpisodeInSeason>
        </TvEpisode>
        <TvEpisode>
            <ID>EPISODE_2</ID>
            <Title locale="en-US">Hello KiyKiy</Title>
            <Synopsis locale="en-US">Say hey to kitty.</Synopsis>
            <Language>en-US</Language>
            <Genres>
                <Genre locale="en-US">Suspense</Genre>
            </Genres>
            <!-- ... rest of CDF data -->
            <ShowID>SHOW_1</ShowID>
            <SeasonID>SEASON_1</SeasonID>
            <EpisodeInSeason>2</EpisodeInSeason>
        </TvEpisode>
    </Works>
</Catalog>

File name: page2.xml

File content:

<?xml version="1.0" encoding="utf-8"?>
<Catalog xmlns="http://www.amazon.com/FireTv/2014-04-11/ingestion" version="FireTv-v1.3">
    <Partner>partner</Partner>
    <Works>
        <Movie>
            <ID>MOVIE_1</ID>
            <Title locale="en-US">Afterwhile Crocodile</Title>
            <Synopsis locale="en-US">Croc has to leave.</Synopsis>
            <Language>en-US</Language>
            <Genres>
                <Genre locale="en-US">Drama</Genre>
            </Genres>
            <!-- ... rest of CDF data -->
        </Movie>
    </Works>
</Catalog>

File name: page3.xml

File content:

<?xml version="1.0" encoding="utf-8"?>
<Catalog xmlns="http://www.amazon.com/FireTv/2014-04-11/ingestion" version="FireTv-v1.3">
    <Partner>partner</Partner>
    <Works>
        <TvShow>
            <ID>SHOW_1</ID>
            <Title locale="en-US">Domestics</Title>
            <Synopsis locale="en-US">Welcome the cats and doggos.</Synopsis>
            <Language>en-US</Language>
            <Genres>
                <Genre locale="en-US">Comedy</Genre>
            </Genres>
            <!-- ... rest of CDF data -->
        </TvShow>
        <TvSeason>
            <ID>SEASON_1</ID>
            <Title locale="en-US">Season 1</Title>
            <Synopsis locale="en-US">Season 1 of Domestics.</Synopsis>
            <Language>en-US</Language>
            <Genres>
                <Genre locale="en-US">Comedy</Genre>
            </Genres>
            <!-- ... rest of CDF data -->
            <ShowID>SHOW_1</ShowID>
            <SeasonInShow>1</SeasonInShow>
        </TvSeason>
    </Works>
</Catalog>

Combined result

File name: full_catalog1.xml.zst (decompressed)

File content:

<?xml version="1.0" encoding="utf-8"?>
<Catalog xmlns="http://www.amazon.com/FireTv/2014-04-11/ingestion" version="FireTv-v1.3">
    <Partner>partner</Partner>
    <Works>
        <TvEpisode>
            <ID>EPISODE_1</ID>
            <Title locale="en-US">Hello Doggo</Title>
            <Synopsis locale="en-US">Say hello to my little doggo</Synopsis>
            <Language>en-US</Language>
            <Genres>
                <Genre locale="en-US">Comedy</Genre>
            </Genres>
            <!-- ... rest of CDF data -->
            <ShowID>SHOW_1</ShowID>
            <SeasonID>SERIES_2</SeasonID>
            <EpisodeInSeason>1</EpisodeInSeason>
        </TvEpisode>
        <TvEpisode>
            <ID>EPISODE_2</ID>
            <Title locale="en-US">Hello KiyKiy</Title>
            <Synopsis locale="en-US">Say hey to kitty.</Synopsis>
            <Language>en-US</Language>
            <Genres>
                <Genre locale="en-US">Suspense</Genre>
            </Genres>
            <!-- ... rest of CDF data -->
            <ShowID>SHOW_1</ShowID>
            <SeasonID>SEASON_1</SeasonID>
            <EpisodeInSeason>2</EpisodeInSeason>
        </TvEpisode>
        <Movie>
            <ID>MOVIE_1</ID>
            <Title locale="en-US">Afterwhile Crocodile</Title>
            <Synopsis locale="en-US">Croc has to leave.</Synopsis>
            <Language>en-US</Language>
            <Genres>
                <Genre locale="en-US">Drama</Genre>
            </Genres>
            <!-- ... rest of CDF data -->
        </Movie>
        <TvShow>
            <ID>SHOW_1</ID>
            <Title locale="en-US">Domestics</Title>
            <Synopsis locale="en-US">Welcome the cats and doggos.</Synopsis>
            <Language>en-US</Language>
            <Genres>
                <Genre locale="en-US">Comedy</Genre>
            </Genres>
            <!-- ... rest of CDF data -->
        </TvShow>
        <TvSeason>
            <ID>SEASON_1</ID>
            <Title locale="en-US">Season 1</Title>
            <Synopsis locale="en-US">Season 1 of Domestics.</Synopsis>
            <Language>en-US</Language>
            <Genres>
                <Genre locale="en-US">Comedy</Genre>
            </Genres>
            <!-- ... rest of CDF data -->
            <ShowID>SHOW_1</ShowID>
            <SeasonInShow>1</SeasonInShow>
        </TvSeason>
    </Works>
</Catalog>

Last updated: Dec 21, 2023