Page sync completes on Mintlify-style sites

Knowledge.sync_pages in Agno took docs.langchain.com from partial with 38 failed pages in 1,147 seconds to completed with 0 failed in 243 seconds. Four fixes to discovery and page parsing got it there.

• Discovery skips links to .json, .yaml, .xml, .csv and .pdf files, images, audio, video, archives and fonts. Pages named like /guides/node.js still sync.
• Nested indexes are walked breadth-first, so a root that lists every /_llms/ index stays within max_depth.
• The mintlify and fumadocs profiles drop top-level MDX import and export statements. On one LangChain page that was 48 KB of component code. strip_esm=False keeps them.
• A listed .md page that redirects to another host or to a non-Markdown URL raises PageMoved and is skipped. SyncReport.skipped and skipped_paths report it.

We fed PageSource.discover() a small llms.txt with an OpenAPI spec, a node.js guide and four /_llms/ indexes that each link the next.

# Earlier releases (real output)
pages:    /agents/overview.md, /api/openapi.json.md, /deploy/overview.md, /guides/node.js.md, /quickstart.md, /teams/overview.md, /workflows/overview.md
complete: False
 
# This release (real output)
pages:    /agents/overview.md, /deploy/overview.md, /guides/node.js.md, /quickstart.md, /teams/overview.md, /workflows/overview.md
complete: True

Off-host page links still block pruning by design.

See the cookbook, and learn more about syncing published pages in the documentation.