Skip to content
Changelog

Scrape, crawl, map and search the web with AnakinTools

October 12, 20261 min read

Agno's AnakinTools sends scrape, crawl and map requests to Anakin.io as jobs, then polls each job every poll_interval seconds (2 by default) until it completes, so the model gets page content or links back from one tool call. Search is a single synchronous request.

MethodFlagAnakin endpoint
scrape_urlenable_scrapePOST /url-scraper
crawl_websiteenable_crawlPOST /crawl
map_websiteenable_mapPOST /map
search_webenable_searchPOST /search

Only scrape_url is on by default, and all=True turns on all four.

from agno.agent import Agent
from agno.tools.anakin import AnakinTools
 
agent = Agent(tools=[AnakinTools(all=True)], markdown=True)
 
agent.print_response("Scrape https://docs.agno.com/introduction/ and summarize it")
agent.print_response("List the URLs reachable from https://docs.agno.com")

Against a stand-in for the Anakin API, one scrape_url call submitted the job and checked it three times before handing the result back.

# real output: AnakinTools(poll_interval=0.01).scrape_url("https://docs.agno.com/introduction")
POST /url-scraper {'url': 'https://docs.agno.com/introduction', 'generateJson': False, 'useBrowser': False, 'country': 'us'}
GET /url-scraper/job_1
GET /url-scraper/job_1
GET /url-scraper/job_1
scrape_url -> {"status": "completed", "markdown": "# Agno\nBuild agents..."}

max_wait_time (120 seconds by default) caps the wait. A job that fails or runs past it comes back as an error string the model can read.

# real output: AnakinTools(poll_interval=0.01, max_wait_time=0.05) on a job that never finishes
Error scraping https://example.com: Job job_2 was still 'processing' after 0.05s. Retry later or raise max_wait_time.

Scrape and crawl route through a proxy in the country you set ("us" by default), and use_browser=True renders JavaScript-heavy pages in a headless browser.

See the cookbook, and learn more about toolkits in the documentation.

Frequently asked questions

Set ANAKIN_API_KEY, import AnakinTools from agno.tools.anakin and pass tools=[AnakinTools()] to your Agent. It uses httpx, which Agno already depends on, so no vendor SDK is needed.

Only scrape_url. Turn on crawl_website, map_website and search_web with enable_crawl, enable_map and enable_search, or pass all=True for all four.

Nothing until the job settles. The toolkit polls the job and returns the finished JSON. A job that fails, or is still running after max_wait_time seconds (120 by default), returns an error string in place of the result.

Shipped around the same time