# Page sync completes on Mintlify-style sites

> Agno page sync now finishes large Mintlify sites such as docs.langchain.com as completed, so pruning runs and pages removed from the site leave the search index.

- Published: 2026-10-02
- Author: Kaustubh Shukla
- Categories: Changelog
- Canonical: https://www.agno.com/blog/page-sync-completes-on-mintlify-style-sites
- Markdown: https://www.agno.com/blog/page-sync-completes-on-mintlify-style-sites.md

`Knowledge.sync_pages` in Agno took docs.langchain.com from `partial` with 38 failed pages in 1,147 seconds to `completed` with 0 failed in 243 seconds. Four fixes to discovery and page parsing got it there.

- Discovery skips links to `.json`, `.yaml`, `.xml`, `.csv` and `.pdf` files, images, audio, video, archives and fonts. Pages named like `/guides/node.js` still sync.
- Nested indexes are walked breadth-first, so a root that lists every `/_llms/` index stays within `max_depth`.
- The `mintlify` and `fumadocs` profiles drop top-level MDX `import` and `export` statements. On one LangChain page that was 48 KB of component code. `strip_esm=False` keeps them.
- A listed `.md` page that redirects to another host or to a non-Markdown URL raises `PageMoved` and is skipped. `SyncReport.skipped` and `skipped_paths` report it.

We fed `PageSource.discover()` a small llms.txt with an OpenAPI spec, a `node.js` guide and four `/_llms/` indexes that each link the next.

```text
# Earlier releases (real output)
pages:    /agents/overview.md, /api/openapi.json.md, /deploy/overview.md, /guides/node.js.md, /quickstart.md, /teams/overview.md, /workflows/overview.md
complete: False

# This release (real output)
pages:    /agents/overview.md, /deploy/overview.md, /guides/node.js.md, /quickstart.md, /teams/overview.md, /workflows/overview.md
complete: True
```

Off-host page links still block pruning by design.

See the [cookbook](https://github.com/agno-agi/agno/blob/main/cookbook/05_agent_os/27_public_pages/README.md), and learn more about [syncing published pages](https://docs.agno.com/knowledge/published-pages#synchronize-published-markdown) in the documentation.