feat: add Firecrawl backend for WebSearch and WebFetch tools

WebSearch is currently disabled for all non-Anthropic providers (OpenAI
shim, DeepSeek, Ollama, etc.) because those providers have no native
search backend. This adds Firecrawl as a fallback that activates when
FIRECRAWL_API_KEY is set, unlocking web search for every model
openclaude supports.

WebFetch uses basic HTTP + Turndown for HTML-to-markdown conversion,
which fails silently on JS-rendered SPAs and bot-protected pages.
Firecrawl scrape replaces the fetch layer when FIRECRAWL_API_KEY is set,
returning clean markdown that handles dynamic content correctly.

Changes:
- WebSearchTool: add runFirecrawlSearch() using @mendable/firecrawl-js,
  respects allowed_domains (post-filter) and blocked_domains (-site: operators),
  includes result snippets alongside links. shouldUseFirecrawl() ensures
  firstParty/Vertex/Foundry/Codex providers keep their native backends.
- WebFetchTool: add scrapeWithFirecrawl(), drops into the existing
  applyPromptToMarkdown() pipeline so prompt processing is unchanged.
- Remove "Web search is only available in the US" restriction from
  prompt when Firecrawl is active (it works globally).
This commit is contained in:
Leonardo Grigorio
2026-04-02 12:18:20 -03:00
parent 4c1ba35aa1
commit ac4efae870
4 changed files with 112 additions and 2 deletions

View File

@@ -112,7 +112,8 @@
"ws": "8.20.0",
"xss": "1.0.15",
"yaml": "2.8.3",
"zod": "3.25.76"
"zod": "3.25.76",
"@mendable/firecrawl-js": "^4.18.1"
},
"devDependencies": {
"@types/bun": "1.3.11",