Knowledge

Add knowledge

The Knowledge tab has two buttons and one status. Use Manage files to select documents from your account library, and Manage websites to add addresses for the agent to crawl. You can use both on the same agent.

Saving reads every source again from the start and builds a fresh index. The agent keeps answering from its existing index while that runs, so there is no gap in service and nothing to schedule around.

Add files

Files live once in your account library and are selected per agent, so the same document can serve six agents without being uploaded six times. Removing a file from one agent leaves it in the library for the others. See Files for the library itself.

Only completed uploads can be selected. Keep the tab open until an upload finishes. An interrupted upload cannot be used and nothing repairs it, so upload the file again rather than waiting.

Crawl a website

Duvi crawls a website source rather than fetching a single page. The crawler obeys robots.txt and takes its page list from your sitemaps, so your site decides what ends up in the index.

1. Duvi resolves the address first

   The crawler follows a redirect from the bare domain to www before anything else, so you do not get two half crawls of the same site.

2. It reads and obeys robots.txt

   The crawler checks robots.txt before every page. It does not read a path your robots file disallows, whatever that path contains.

3. A sitemap fills the queue, or the address does

   The crawler expands the sitemaps named in robots.txt into its page list. If a site declares none, the crawler starts at the address you gave and follows the links it finds.

4. The address you gave sets the boundary

   A bare domain covers the whole site. An address with a path stays beneath that path, so adding one article indexes that article rather than everything its publisher has posted.

5. It opens each page in a real browser

   The crawler captures content drawn by JavaScript, and describes the images it finds so the agent can show them.

A sitemap is the most useful thing you can give the crawler, because it names the pages you consider worth reading. To narrow a crawl, use a more specific address or a robots rule.

The crawler identifies itself as DuviBot. On a staging site that blocks everything, allow DuviBot explicitly and leave the rest closed:

User-agent: DuviBot
Allow: /

User-agent: *
Disallow: /

Sitemap: https://example.com/sitemap.xml

Indexing states

One status covers the whole agent rather than one per source. It sits beside Index at the top of the Knowledge tab.

There is no partial state. A run either replaces the whole index or leaves the last good one alone, so an agent never answers from half your content.

Fix an empty or incomplete index

Before you change anything, run a retrieval trace from the same tab. The trace searches the index as it stands and shows what comes back, which separates the material was never indexed from the material is there and the agent ignored it. The two have different fixes.

For symptoms across every part of the product, see Troubleshooting.