Building production-ready web intelligence agents: Agno x Oxylabs integration

Agno
October 16, 2025
8 min read

AI agents are only as smart as the data they can access. Yet giving them reliable access to large-scale web data is one of AI’s hardest challenges, as it requires specialized infrastructure that’s complex and costly to build.

The Agno and Oxylabs integration solves this challenge by combining ultra-fast agent orchestration (up to 10,000× faster processing with 50× less memory) and enterprise-grade web scraping infrastructure. Together, they empower any team to build intelligent web research systems that scale.

In this blog, you'll learn how to configure Agno agents with Oxylabs tools, design multi-agent web intelligence workflows, and deploy production-ready systems that can handle complex research tasks automatically.

The challenge: bringing real-world data to production AI agents

Most AI agent frameworks perform well in controlled demos but fall short when agents need reliable access to real-world web data. At scale, operational challenges compound, including managing concurrent agent and web requests, unblocking websites for agents, maintaining uptime across distributed systems, and ensuring data quality while controlling costs.

Built-in AI web search solutions are often expensive and limited in scope, while custom web scraping infrastructure diverts engineering resources away from core agent development.

Agno workspaces take you from zero to production in minutes, but without reliable and relevant data, even the fastest agents can't deliver meaningful business value.

Oxylabs and Agno integration: benefits overview

Our integration with Oxylabs makes your agentic systems smarter by providing:

  • Reliable global data access: Block-free web access with accurate, JavaScript-rendered, and geotargeted data for your Agno agents
  • High performance & scalability: Enterprise-grade web data infrastructure that amplifies Agno's ultra-fast agent performance and scales effortlessly with demand
  • Seamless integration & flexibility: Native compatibility with Agno's structured output system (Pydantic models) and full support for any LLM provider integrated with Agno
  • Streamlined operations: End-to-end automation of data pipelines with zero maintenance overhead and lower total cost of ownership
  • Enterprise-grade security: Robust security, privacy, and access controls designed for mission-critical deployments

Real-world use case: AI-driven SEO insights

To demonstrate the power of this integration, let’s tackle a practical challenge by building a multi-agent system that fetches web data in real time. SEO teams often need to analyze search engine results and relevant web page content, a process that can take hours when done manually. Key SEO metrics to monitor include:

  • Brand visibility and share of voice
  • Content gaps, search intent clusters, and topical authority
  • Content analysis, including keyword usage, structure, and internal/external links

Solution: Agno and Oxylabs integration

The following Python automation example reduces hours of manual SEO research into minutes of agent-driven analysis, enabling SEO teams to focus on strategy and optimization rather than data collection.

‍

import asyncio
from textwrap import dedent

from dotenv import load_dotenv
from agno.agent import Agent
from agno.team import Team
from agno.models.openai import OpenAIChat
from agno.tools.oxylabs import OxylabsTools


# Load environment variables from a .env file
load_dotenv()

# SERP analysis agent
serp_agent = Agent(
    # Use Oxylabs Google Search scraper
    tools=[OxylabsTools().search_google],

    model=OpenAIChat(id="gpt-5-mini"),
    role="You are Google Search agent that can gather SERP results for given keyword.",
    instructions=dedent("""
        Search Google with a keyword and analyze the SERP result.
        Use only the extracted data to identify: 1) Brand visibility, 2) Top competitors by position, 3) Search intent (informational/commercial/transactional).
        Flag high-value URLs (positions 1-3, competitor pages, relevant content) for deeper analysis.
        Return Markdown-formatted data with keyword, brand visibility score, competitor list, and intent classification.
    """),
    markdown=True
)

# Website analysis agent
web_agent = Agent(
    # Use Oxylabs generic website scraper
    tools=[OxylabsTools().scrape_website],

    model=OpenAIChat(id="gpt-5-mini"),
    role="You are a web scraping agent that can gather data from websites.",
    instructions=dedent("""
        Scrape and analyze each URL one by one. Use JavaScript rendering.
        Extract and analyze: 1) Primary keywords (frequency and prominence), 2) Content structure (headings, word count), 3) Meta tags, 4) Internal/external links count.
        Identify content gaps and assess content quality signals (readability, keyword density, topic coverage depth).
        Return key insights for each URL in a concise and Markdown-formatted report.
    """),
    markdown=True
)

# Team that works together and produces a final report
team = Team(
    model=OpenAIChat(id="gpt-5"),
    members=[serp_agent, web_agent],
    name="SEO Analysis Team",
    role="Coordinate SERP and web analysis agents and draft a final report.",
    instructions=dedent("""
        First, have the SERP agent analyze Google SERP for a given keyword and return URLs for further analysis.
        Then, pass the URLs to the web agent for a deep dive into top-performing and competitor pages.
        In the end, make a joint report of the keyword-by-keyword analysis from the SERP and web agent.
        The report must contain an actionable SEO strategy with Executive Summary, Keyword Analysis, Competitive Landscape, Content Gap Analysis, and Recommendations.
        The final report MUST BE concise and well-structured in Markdown with H1, H2, H3, H4 headers, bolded text, bullets, and tables for clarity.
    """),
    markdown=True,
    share_member_interactions=True
)


async def main():
    # Example SEO analysis prompt
    user_prompt = dedent("""
        Find Google.com SEO insights for 'best running shoes 2025'.
        Brand name for tracking is 'Adidas'.
    """)
    
    result = await team.arun(user_prompt)
    print(result)

    # Save the final report to a Markdown file
    with open("seo_analysis.md", "w") as file:
        file.write(result.content)


if __name__ == "__main__":
    asyncio.run(main())

‍

To run this code, install the required libraries:

pip install -U oxylabs agno openai

Make sure you’ve also saved your Oxylabs API credentials and OpenAI API key in a .env file. To claim a free trial of Oxylabs, register on the dashboard and navigate to Web Scraper API > Pricing.

OXYLABS_USERNAME=oxylabs_api_username
OXYLABS_PASSWORD=oxylabs_api_password
OPENAI_API_KEY=your_openai_key

‍

How it works

This multi-agent workflow automatically:

  1. Scrapes Google SERPs to analyze brand visibility, competitors, and search intent
  2. Scrapes high-value URLs to assess content quality signals

Generates a final report with keyword analysis, competitor insights, content gaps, and actionable recommendations

‍

Here’s a snippet of the final SEO report, saved in Markdown format:

‍

Next steps

With Oxylabs, your Agno agents can scrape thousands of web pages quickly and without blocks, freeing up your team to focus on innovation instead of infrastructure. Use the code example in this post as a blueprint for building your own web intelligence agents, whether you're developing a deep web research system or a product price monitoring solution.

Additional resources:

MCP implementation: Oxylabs MCP, MCP example in Agno