Lifestyle
Cloudflare launches Web Search API via AI Gateway, requiring search partners to follow its crawler rules
On October 2, 2026, Cloudflare announced a Web Search API that lets AI agents query live web information through AI Gateway. The first partners are Ceramic.ai, Exa and Linkup. Cloudflare says these partners' crawlers must meet its Verified bots requirements and cite sources. This matters both to developers building AI applications and to website owners whose content may be crawled.
About 5 min read
What Cloudflare announced
On October 2, 2026, Cloudflare announced on its official blog a Web Search API offered through its AI Gateway. An API (application programming interface) is a way for one program to request a service from another; here, it lets software send a search query and get web results back. AI Gateway is Cloudflare's control platform for AI applications, with built-in logging and observability, unified billing, security and access controls. The post, written by Michelle Chen, Sam Else and Gabriel Massadas, is part of the company's Birthday Week series of launches. Cloudflare says its first web search provider partners are Ceramic.ai, Exa and Linkup.
AI agents are AI programs that carry out tasks on a user's behalf. Cloudflare explains that when an AI agent needs to read a live web page, it often guesses a URL and fetches it directly, and a wrong guess returns a 404 Not Found error (page not found). The company also notes that models are usually frozen at a point in time after training and can only use information from before their knowledge cutoff. According to Cloudflare, web search lets an agent search first, as a person would, and then ground its answer in live information.
Crawler rules: the core of this launch
A crawler is a program that automatically reads web page content. According to Cloudflare, partners have committed to meeting Cloudflare's crawler standards. Specifically, there are two requirements. First, the crawlers a search provider uses must meet the public "Verified bots" requirements in Cloudflare's developer documentation. Second, search results must include links to where the crawled content is located. Cloudflare says these providers identify their crawlers, respect robots.txt and provide sources for their search results.
For general readers and website owners, this means that on this platform, as Cloudflare has designed it, AI tools citing web content will show where it came from. Websites can also use robots.txt to signal whether they are willing to be crawled. However, whether these commitments are honored currently rests solely on Cloudflare's own account.
How developers can use it
Cloudflare says web search requests appear in AI Gateway's regular observability logs, query costs are deducted from AI Gateway credits, and administrators can control who may use which search providers. The company says it will label partners that support zero data retention (ZDR), meaning providers that do not retain user data. Web search is offered at partners' public API pricing with no markup. Cloudflare also supports bring your own key (BYOK), letting teams keep using their existing provider account setups.
| Access method | Description (per Cloudflare's announcement) | Current status |
|---|---|---|
| REST API | Query via a standard REST endpoint from existing backends, mobile apps or external services | Available |
| Workers binding | Call web search through a dedicated binding in Cloudflare Workers (Cloudflare says it takes one line of code) | Available |
| Server Tools | Native tools built into AI Gateway, with web search among the first | Coming soon |
What it means for general readers
- If an AI assistant uses this kind of search service, it can answer questions about recent events based on live information rather than training data alone.
- Under Cloudflare's rules, search results must include source links, making it easier for readers to check the original.
- Website owners can state their preferences via robots.txt, which Cloudflare says partners will respect.
Frequently asked questions
What is the Web Search API?
According to Cloudflare's announcement, it is a web search service offered through AI Gateway. AI agents can use it to query live information on the web, and the search results are placed into the model's context for the model to draw on when answering.
Which search providers are taking part?
Cloudflare says the first partners are Ceramic.ai, Exa and Linkup.
How does this relate to websites being crawled by AI?
Cloudflare says partners' crawlers must meet its "Verified bots" requirements, identify themselves and respect robots.txt, and search results must include links to the content's source.
Is there an extra charge?
Cloudflare says web search is offered at partners' public API pricing with no markup; costs are deducted from AI Gateway credits, and bring your own key is also supported.
When will Server Tools launch?
Cloudflare only says it is "coming soon" and that web search will be among the first tools included; the announcement gives no specific date.
Browse the latest news in this topic
Lifestyle
Cloudflare launches Traces in public beta: site operators can follow every step a request takes through the platform on one timeline
On October 2, 2026, Cloudflare announced the public beta of Cloudflare Traces. Website operators and developers using Cloudflare can see a request pass through security rules, caching, routing and the origin server on a single timeline, making it easier to find why a request was blocked or slowed. New pricing takes effect on December 1, 2026. Information comes from the official Cloudflare blog.
Lifestyle
Cloudflare open-sources Streamline: a demo of using its cloud services to add graphics to live streams and burn subtitles into videos
On October 2, 2026, Cloudflare launched and open-sourced Streamline, a developer playground showing how developers can combine Stream, Workers, Containers and Durable Objects to build their own video processing pipelines, such as adding graphics to live streams in real time or adding subtitles to videos. This article explains what it is, how it works, its limitations, and what it means for viewers and developers.
Lifestyle
Google unveils Gemini 4 Argon: cyber defenders get it first, everyone else still has to wait
On September 30, 2026, Google announced Gemini 4 Argon, which it calls its new frontier (most advanced) AI model. For now it is available only to trusted cyber defenders through the Fairwind Program. Here is what Google says the model can do, what it will cost developers, how Google says it is managing the risks, and what it means for everyday users. All figures come from Google itself.
Lifestyle
NVIDIA: CoreWeave Begins Offering Vera Rubin NVL72, With Cognition as First Production Customer
According to the NVIDIA blog, AI cloud provider CoreWeave now offers NVIDIA's next-generation Vera Rubin NVL72 systems, plans to offer the Vera CPU and launched CoreWeave Forge. This matters mainly to companies building AI agents, and could eventually mean faster AI tools for everyday users.
Latest travel guides
GuideTokyo
Where to Stay in Tokyo: Comparing Shinjuku, Ueno, Tokyo Station, Shibuya, Asakusa, Ikebukuro, and Ginza, Plus Airport Access, Accommodation Tax, and Luggage Delivery
Where should you stay in Tokyo? Compare Shinjuku, Ueno, Tokyo Station, Shibuya, Asakusa, Ikebukuro, and Ginza by the same criteria: access from Narita and Haneda, transit routes, nearby attractions, neighborhood character, and who each area suits. Includes a comparison table, a Yamanote Line diagram, Tokyo’s accommodation tax as verified in 2026/9 (changing to 3% in 2027/4), and Airport TA-Q-BIN luggage shipping rules.
- Budget
- Hotels
GuideTokyo
How to Choose Tokyo Transit Passes: Are Suica, Welcome Suica, the Tokyo Subway Ticket, and the JR Pass Worth It?
On a first Tokyo trip, start with an IC card and pay per ride (Welcome Suica has no deposit and is valid for 28 days). If you take four or more subway rides in a day, add a 72-hour Tokyo Subway Ticket for 2,000 yen; a JR Pass is never worthwhile if you stay in Tokyo and do not go to Kansai. See what TOURIST PASMO, Suica on iPhone, and the Tokyo Metro day pass do and do not cover, with a decision chart. Prices verified in September 2026.
- Transport
- Budget
GuideTokyo
Tokyo Disneyland and DisneySea Guide: Ticket Prices, Fantasy Springs, Disney Premier Access (DPA), Standby Pass, and Which Park to Choose for Your First Visit
Tokyo Disney one-day Passport prices vary: most weekdays in 9/2026 cost ¥9,900 and weekends ¥10,900. At 14:00 daily, tickets go on sale for the same date two months later. Free Priority Pass is no longer on the official service list; only paid Disney Premier Access (¥1,000–3,500 per person per use) shortens waits. Covers hours, the 25th anniversary, Standby Pass, Entry Request, Fantasy Springs access and first-visit park choice; checked on the official site in 9/2026.
- Itineraries
- Family