<h1 align="center">
<a href="https://prompts.chat">
Fix Brave search result classification so snippet-only results are sent to the scraper instead of being treated as already-fetched full content.
Loading actions...
<a href="https://prompts.chat">
TypeScript and ESLint rules that MUST be followed when creating, modifying, or reviewing any file under apps/frontend/, including .ts, .tsx, .js, and .jsx files. Also apply when discussing frontend linting, type safety, or ESLint configuration.
risks
Fix Brave search result classification so snippet-only results are sent to the scraper instead of being treated as already-fetched full content.
Brave search results populate body with snippet text, not full-page content. In ResearchConductor._search_relevant_source_urls(), results were classified as already-fetched whenever len(raw_content) > 100, where raw_content fell back to result["body"]. That caused Brave results with long snippets to bypass scraping entirely.
In the failing reproduction, this manifested as every sub-query logging "Scraping content from 0 URLs" because all Brave hits were treated as prefetched content before the scraper ever saw them.
This PR intentionally contains only the snippet-classification fix. The separate scraped-count tracking used by local safeguard logic belongs to the safeguards PR, not this one.
body as full page content in _search_relevant_source_urls()raw_content to identify retriever results that already contain full textraw_content results remain prefetchedVerified locally with:
uv run python -c "from gpt_researcher.skills.researcher import ResearchConductor; print('ok')"uv run python -m unittest tests.test_research_conductor_retrievalNone.