A crawler that reads everything in order spends its budget where the links happen to lead. One with a goal goes straight there, and keeps every site’s rules.

A goal-driven web crawler. Given a goal and the sites to read, it follows the most promising link next, checks each request against scope, site terms, robots.txt and AI-crawler rules before sending it, and writes down every choice and refusal with its reason. A recorded demo replays its crawls of a generated test corpus.

Agentic Crawl is a web crawler with a goal. Told what to look for and which sites to read, it reads every start page first, then takes whichever queued link comes closest to the goal. Each link is scored on its own words (its text, its address, the title of the page it sits on) and on the page it was found on, by an embedding model that runs where the crawler runs. When two addresses that differ only by a parameter, such as a session id or a print view, return the same page, the parameter is learned and further copies are skipped. After a run of pages well below the best it has found, the crawl ends by itself and says why. What comes back is the pages, their content as an Agentic Dataset that says where each item came from, and the record.

The agent chooses the order; it never fetches anything itself. Every request, for every page and every redirect, passes the same checks before it is sent: the crawl’s scope, the site’s terms, its robots.txt and its rules for AI crawlers, none of which the agent can change. A refusal sends nothing and goes on the record with its reason. Each step is written as a Brain API decision record (the links weighed and their scores, what the rules said, what was taken and why) in a hash-chained log that can be verified afterward, and every extracted page carries where and when it was fetched and under which decision.

Measured on a test corpus built for checking crawlers exactly: nine generated sites and 1,799 pages, with every page’s content, link, rule, trap and change fixed before any crawl starts, and every request read from the sites’ own server logs rather than from the crawler. Across 36 recorded crawls and 39,718 requests, none reached a page that robots.txt, a site’s rules for AI crawlers or its terms put off limits, and each of the 1,210 refusals gave the reason the corpus itself gives. For a goal of 40 product pages, the agent found them all in 90 requests where a breadth-first crawl of the same sites under the same rules needed 1,121; for backup guides, forum advice and library books, 51 vs 297, 61 vs 1,718 and 75 vs 860. Session-id links that never end took two thirds of a breadth-first crawl’s requests; the agent fetched three copies, learned that the parameter changes nothing, and skipped the rest. A second, held-out corpus gives the same picture.

Extraction and change detection were measured the same way. Extracted text scored 0.972 F1 against the exact content of 1,648 pages, and no page lost more than half of it, including pages built from separate sections, which had lost a third of their text before. Re-crawled a day and a week later, all 1,650 changed-or-unchanged verdicts agreed with what had in fact changed, including on servers that kept reporting “unchanged” when it was not.

The demo replays recorded crawls in the browser in three linked views: the crawl as a tree growing from the agent through each site toward the goal’s pages, a request lane per site taken from the servers’ own log, and the decision trace behind each choice, with a Truth tab that holds each run against the corpus’s own answers. The test sites are generated, and their small vocabularies make matching easier than on real sites; a free pilot measures the same things on a customer’s own sites.

Transferable principle: let the agent choose the order, and keep the decision to send a request in checks it cannot change.

References

1. Chakrabarti et al. (1999). Focused crawling: a new approach to topic-specific Web resource discovery. Computer Networks 31(11–16), 1623–1640. doi:10.1016/S1389-1286(99)00052-3
2. Koster et al. (2022). Robots Exclusion Protocol. RFC 9309, IETF. doi:10.17487/RFC9309
3. Chernov (2026). Brain API: An Intent-Aware Control Plane for Policy-Governed Agentic Systems. arXiv preprint. arXiv:2609.21299

agenticcrawl.cademo.agenticcrawl.caAll results, goal by goalDemo video: Zenodo 10.5281/zenodo.23328688