Websites

The Websites page lets you add public web URLs to your agent's knowledge base. The platform visits each URL, extracts the main text content, and indexes it as searchable knowledge, so your agent can reference and cite web pages the same way it does uploaded documents.

In this tutorial you give Atlas, Kirana Group's operations assistant, a public safety page to answer from: the UK Health and Safety Executive (HSE) page "Inspecting the condition of ladders". Site supervisors can then ask Atlas how often ladders must be inspected and get a cited answer.

Before you start

• Read Teach Your Agent for how knowledge turns into answers. If you have files, start with Documents.
• In the CMS, open Knowledge → Retrieval and make sure Enable Reference is on. Without it, the agent ignores everything you add here. If it is off when you save, the CMS offers to switch it on, as on the Documents page. See Retrieval.
• Make sure the page you want to add is public. The platform must be able to open the URL without logging in. Some sites block automated readers altogether; see When a website fails.

Step 1: Enter the website source

1. In the CMS, go to Knowledge → Websites in the left sidebar and click New Website in the top-left. The Add New Website form opens.

2. In Website Source, enter the full URL:

   https://www.hse.gov.uk/work-at-height/ladders/inspecting-condition-of-ladders.htm
   

3. Click Get Metadata to fill in the title and thumbnail from the page. You can change both afterwards.

Only http:// and https:// URLs are accepted. The URL must contain a valid hostname (for example example.com), must not exceed 2,048 characters, and must not point to localhost or a private IP address.

Step 2: Choose how the page is read

1. Set Ingestion Mode to Skimming. It works well for most web pages. The modes are explained in Ingestion Mode below.
2. Set Content Extraction to Single Page. The default is Related Pages, which also follows links from the page. For Atlas you only want this one page, not the rest of the HSE site. See Content Extraction.

Step 3: Fill in the title, author, and code

Title is the display name, and it also appears in citations. Author lists the authors registered in Citations. Code is a short unique identifier you choose. Leave Thumbnail Source and Permissions empty for this tutorial.

Step 4: Save and wait until the website is Active

1. Click Save and confirm with Yes.
2. A success message says the web resource was created and is being processed, and that it becomes active and usable for your agent once processing is done. Close it.
3. The website's detail page opens with its chunking log, so you can follow the progress. You can close the log; processing continues in the background.

When processing is done, the website's badge changes from Processing to Active. On the Websites page, the card shows the title, the code WEB-HSE-LADDERS, the author, and how many chunks are active.

Like documents, websites do not need to be published: once a website is Active, both the Preview and your live agent can use it.

Managing the websites list

The list shows all websites registered to this agent. Each entry represents a single URL that has been, or is being, processed. Use Search Websites to filter by title, code, or status; click it again to collapse the panel. Use the Sort By and Sort Direction controls in the top-right to order entries; your preference is kept across page visits.

Websites use the same status badges as documents: Processing, Active, Inactive, Failed, and Deleted. See Document statuses.

Grounding your agent in real web content reduces the risk of hallucinations by anchoring answers to verifiable sources.

When a website fails

Some sites block automated readers. Wikipedia, for example, returns HTTP 403 to the platform. The chunking log then shows a line such as:

Website validation failed: Website returned HTTP 403

The website's badge changes to Failed. Open the website's detail page and choose one of these:

• Retry Chunking: process the same URL again, for example after a temporary error. Type the website title to confirm.
• Purge Web: remove the website entry. Type the website title to confirm.

If a site keeps blocking the platform, save the page content as a file and add it in Documents instead.

Field reference

Ingestion Mode

Ingestion Mode controls how the extracted web text is split into chunks before it is stored in the knowledge base. Two modes are available for web resources:

Click the ⋯ button next to the dropdown to open the ingestion parameters panel for the selected mode.

Skimming parameters

Semantic parameters

Content Extraction

Content Extraction controls how many pages the platform visits from the starting URL.

When Related Pages is selected, click the ⋯ button to configure the crawl parameters:

Thumbnail Source and Permissions

• Thumbnail Source: an optional cover image, either a URL (paste an HTTPS image link and click Preview) or an Upload.
• Permissions: optional permission codes that limit the website to authorized users. See Knowledge Permissions.

Check the result

1. On the Websites page, the card for WEB-HSE-LADDERS shows Active.

2. In the preview panel on the right side of the page, click the reload button to start a fresh conversation.

3. Ask:

   How often must ladders that are part of a scaffold be inspected?
   

Atlas answers that ladders that are part of a scaffold must be inspected every seven days as part of the scaffold inspection, plus a pre-use check each working day. When citations are on (see Citations), a citation below the answer names the HSE page and links to its URL.

If the answer is missing, look at the extracted text in Chunks. If the website shows Failed, see When a website fails.

Next steps

• Videos: add video transcripts as knowledge (next step).
• Documents: upload file-based knowledge sources.
• Retrieval: tune retrieval settings that apply to all resource types.