Every developer with a healthy respect for technical infrastructure shares a consensus: building a search engine is extremely difficult. Over the long twenty years spanning the PC internet to the mobile internet, the number of general-purpose search engines with significant global market share could be counted on one hand (Google, Bing, Baidu, Yandex). Achieving high-frequency web-wide crawling, anti-scraping defenses, web spam filtering, and sub-second freshness requires vast hardware, bandwidth, and sunk engineering costs behind the scenes—making general search an exceptionally high-barrier game reserved for tech giants.
Yet in these few months of mid-2026, a seemingly counter-intuitive phenomenon has emerged: not only are Agent-focused search services like Exa and Tavily everywhere, but even AWS released AgentCore Web Search and Bedrock Web Search in quick succession. Search interfaces have suddenly landed on every developer’s desk.
When my team and I were refactoring the external retrieval architecture for our agents, the first question that popped into our minds was: if search is so hard, why does it suddenly feel like any company can jump out and launch a Web Search API in the agent era? Did web-wide search suddenly get cheaper or simpler in the LLM era? When we previously discussed multi-model routing in agent sessions and agent context evolution, we were also keeping a close eye on the true shifts in underlying retrieval costs.
After hooking these APIs into our production environment and unpacking their data flows, we finally understood the true logic behind it: tackling web-wide search hasn’t gotten any simpler, and the heavy lifting at the bottom—crawling web pages, fighting anti-scraping mechanisms, and building indexes—remains extremely hard-core. What has actually happened is a shift in search demand and the unbundling of the supply chain.
To understand why almost no one could challenge traditional search in the past, one must look at how the vertically integrated black box of legacy search engines actually operated.
The supreme difficulty of traditional search lies in the requirement to do three things right simultaneously: index enough of the web, rank the most relevant results in front of humans, and recoup the massive infrastructure costs through advertising. This formed the seemingly invincible business flywheel of old search: secure massive queries via default entry points → transform queries and clicks into training signals → use signals to refine ranking for long-tail and trending queries → support ad impression monetization with precise rankings → use the revenue to buy out browser default placements and sustain a massive crawling team. Data disclosed in the US v. Google antitrust decision shows that the revenue-share Google paid to distributors like browsers and device makers for default search placements totaled over $26 billion in 2021 alone.
The decisive technical bond behind this was an internal system named NavBoost, a feedback mechanism that continuously calibrates relevance using aggregated query and click behavior at scale, as detailed in the DOJ NavBoost testimony document.
This flywheel tightly coupled web crawling, algorithmic ranking, and the ad-based business model. In the past, you couldn’t pull out web crawling or relevance ranking to run as an independent business, because breaking out any single step could not recoup the infrastructure costs. A single tech giant had to handle all the heavy lifting end-to-end, directly outputting web page lists for human consumption and monetizing via ads.
In the agent era, the primary entity initiating search requests shifted from humans to LLMs. Models neither view ads nor type into address bars; they don’t need web page lists rendered with complex HTML styling and ad slots. They only need clean body text, extracted snippets, and authoritative URL citations that can be stuffed directly into the Context Window.
The delivery contract shifted from human eyes viewing web pages to LLMs consuming tokens, causing the previously monolithic pipeline—crawling, retrieval, parsing, and compression—to be unbundled. This split was not driven solely by LLMs; it also relied on the simultaneous maturation of public crawls (such as Common Crawl), open-source search infrastructure, targeted indexing, and direct monetization via cloud billing.
There aren’t dozens of companies attempting to replicate Google; people are actually leveraging shortcuts or external leverage in different ways to make a specific layer of the search stack cheaper and sellable standalone:
The first type is a proxy layer that passes through Big Tech results. Represented by Serper, their engineering implementation is extremely lightweight—they don’t crawl web pages or build any indexes. They simply wrap an API to pass through Google’s raw ranked results. The most hard-core web-wide crawling, anti-spam, and relevance ranking are still being handled on their behalf by Google, backed by billions of dollars in annual infrastructure and engineering investments.
The second type is targeted semantic indexing by narrowing the battleground. Represented by Exa, facing the adversarial pressure of billions of spam pages daily reported by Google (a self-reported vendor figure), Exa chose to prioritize crawling high-information-density sites like GitHub, ArXiv, Wikipedia, and high-quality tech blogs, drastically reducing crawling and storage costs. It searches with high accuracy in specific technical or academic fields, but cannot handle general everyday search across the web; its index scale is self-reported by the vendor and has not been independently audited. Among market players built truly on proprietary general web crawlers and physical indexes, Brave is one of the very few (see Brave’s independent index announcement for details).
The third type is context refinement and hybrid retrieval layer. Represented by Tavily (refer to Tavily Terms of Service), its core selling point is not building an alternative web-wide search engine to Google out of thin air, but focusing on web page denoising, main body extraction, and text compression tailored to LLM RAG needs, specifically outputting high-information-density snippets.
The fourth type is the cloud-native infrastructure layer. This refers to the two distinct product surfaces AWS launched in 2026—AgentCore Web Search Tool (GA in June, as an MCP connector) and Amazon Bedrock Web Search (GA in August, as a model server-side tool)—embedding retrieval capabilities directly into the cloud-side model API runtime.
As for Perplexity, which is commonly used, it is essentially a consumer-facing end-to-end answer engine combining proprietary crawlers with third-party crawling/indexing services, rather than infrastructure middleware that developers directly use for assembly.
Once you understand the layering of the search stack, looking at AWS launching AgentCore Web Search Tool and Bedrock Web Search successively in 2026 reveals the deeper logic behind its market entry.
Many developers might wonder: why Amazon? And why specifically in 2026?
Behind this is not a sudden decision by AWS to compete with Google for the consumer search market, but rather the overlap of two key factors:
First is the secondary commercialization opportunity of internal infrastructure. Before selling search APIs to developers, Amazon had already established a massive Amzn-SearchBot web crawling and retrieval infrastructure internally to power Alexa+ (next-gen voice assistant), Amazon Quick (enterprise business intelligence), and Kiro (cloud development environment). In the past, this was merely an expensive internal cost center; by 2026, as building agents on Bedrock became mainstream among external enterprises, AWS keenly captured this new B2B monetization opportunity—packaging existing internal infrastructure as cloud service offerings. Theoretically, API calls from external developers can help offset fixed crawling and freshness maintenance costs alongside internal workloads; however, AWS has not disclosed backend sharing ratios or unit economics, making this a strategic inference rather than a publicly declared fact.
Second is the fundamental B2B shift in search distribution channels. In the traditional consumer search era, user acquisition relied on winning browser default spots, with Google spending tens of billions of dollars annually to buy out address bars in Safari and Android; but in the agent era, search has transformed into a pure B2B developer product, shifting the entry point of traffic from mobile/browser address bars to model API runtimes on Bedrock or SageMaker.
This represents a massive structural opportunity for cloud giants. AWS has no need to buy out consumer device entry points because it already controls the cloud runtime environments for enterprise agents. Developers are already managing IAM authentication, paying cloud bills, and executing model contexts on AWS, making native Web Search an out-of-box default choice with minimal compliance friction. Partially substituting consumer browser distribution with cloud B2B infrastructure distribution is the core move for cloud giants capturing dividends from the unbundling of the search supply chain.
By examining the restructuring of the entire search supply chain, the migration path of defensive moats becomes clearer.
What got conveniently bypassed were the commercial and interaction stacks tied to traditional SERPs: SERP layout designs, ad bidding systems, search-box-driven traffic distribution, and OS-controlled default search entry points.
However, the engineering hard problems standing in the way remain unchanged: bandwidth and IP resources required for high-frequency web crawling, anti-scraping battles and copyright licensing negotiations, web page freshness indexing, and domain authority calculations for spam filtering. No matter how smart LLMs become, they cannot bypass these physical engineering constraints for you.
At the same time, new moats and data flywheels are beginning to emerge. On the distribution side, model SDKs, MCP protocol standards, and cloud provider model API runtimes have become the new default entry points; on the data side, the signals used to calibrate ranking quality have evolved from human clicks and dwell time into code execution success rates, tool retry rates, and citation link open rates during agent task execution.
For us as architects, when choosing connectivity solutions for agents, we must never be misled by the illusion that anyone can build a search engine. Pure SERP pass-through wrapper layers face deterministic long-term disadvantages against vendors with proprietary indexes and cloud giants; meanwhile, data sources specializing in context refinement or holding independent indexes will retain unique value in specific agent scenarios. Clearly discerning the true position of different vendors within the stack and procuring a combination tailored to your system’s requirements for latency, privacy, COGS, and evidence depth is the most grounded approach to navigate this supply chain decoupling.