Meta’s updated Commercial Terms, which take effect Oct. 30, give the company permission to crawl websites associated with businesses that use its products and to use that content to train those products. The terms apply to any business use of Meta’s products, from running ads to managing a Facebook Page.
Nieman Lab reported this development on Oct. 15. A total of three law professionals have viewed the terms for this media outlet and believe that language in these terms could support an argument by Meta that it is permitted to scrape websites of news sites and use the scraped content to train its AI products.
On Oct. 16, Meta spokesperson Andy Stone responded on X to a post summarizing that report and called the summary false. He wrote that the update is meant to make sure Meta can pull accurate information about products for sale on third-party websites when it makes ad and content recommendations on its platforms, and that it’s “not about scraping news sites.”
Meta’s ad terms already let it crawl advertisers’ websites for uses that include training. The October version reaches businesses that don’t buy ads.
What The New Clause Allows
The new crawling paragraph in the Commercial Terms lets Meta, or anyone acting for it, use web scrapers, bots and other automated tools to crawl and collect content from domains and pages associated with a business’s products or services. It also covers URLs made available through the business’s use of Meta’s products or through users’ use of them.
The clause also lets Meta “create test credentials” to reach those pages. Meta may use the content to develop, improve, research and train its products, among other listed uses.
Meta’s own list of ways businesses can utilize Meta services includes advertising, marketing messages, listing your products on Meta platforms along with Meta-created tools including Ads Manager and Meta Business Suite. The very same crawl permission is included in the revised Self-Serve Ad Terms, effective Oct. 30.
A continued usage of Meta’s service for business purposes will be interpreted by Meta to mean that you have agreed to the new terms. The terms say that if you disagree with these new terms, you may cease utilizing Meta’s products for business purposes.
Media and intellectual property attorney Matt Topic, representing The Intercept in their copyright case versus OpenAI, stated to Nieman Lab that while there are no assurances if a court will ultimately enforce this provision, publishers should assume it will.
How Much Of This Is New
The current Commercial Terms, dated March 6, 2024, have no crawling clause. Meta’s ad terms do.
The current Self-Serve Ad Terms, dated Aug. 22, 2025, say that placing an ad order gives Meta the right to crawl content on the advertiser’s domains, webpages or URLs. The listed uses include developing advertising and other business products, maintaining integrity, and “research and training.”
Meta’s Ad Creative Generative AI Terms (May 6, 2024) permit Meta to crawl URLs provided by advertisers, for use with their ads, as part of using Meta’s generative AI ad tools. Also the Product Catalog Terms allow Meta to periodically update product catalogs from catalog URLs.
In addition, Meta has announced in its Business Help Center notice that the Product Catalog Terms will be moved into the Commercial Terms, and the Ad Creative Generative AI Terms will move into the Self-Serve Ad Terms to “reduce overlap.” The announcement did not reference crawling or training.
Further, unlike the ad terms, which referenced the placement of an advertisement purchase, the new terms tie the granting of permission to any business use, covering URLs related to a business’s products or that were otherwise made available through users’ usage of Meta’s products. Prior ad agreements allowed Meta to access the advertiser’s domain or advertiser-supplied URLs.
Also, Meta’s Seller Agreement, last updated May 26, 2026, already permits Meta to utilize automated software, including creating test credentials, to obtain information from URLs that sellers provide. The agreement says this helps Meta better assist the seller with optimizing their listings.
Where Robots.txt Fits
Meta’s spokesperson told Nieman Lab the new terms won’t change how the company crawls publishers and that “content owners can still manage how their content is accessed by using robots.txt.” Stone made both points in his post as well. Neither the updated Commercial Terms nor the updated Self-Serve Ad Terms mention robots.txt.
The legal experts told the outlet the new language gives Meta an opening to argue that its commercial terms supersede other blocking instructions. Cornell law professor James Grimmelmann said test credentials could technically let Meta sign up as a fake user to get past registration walls. Meta declined to answer whether test credentials would bypass paywalls or other paid content.
Meta’s crawler documentation says Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products by indexing content directly. Meta-WebIndexer improves Meta AI search results, and Meta says allowing it helps the company cite and link to your content in Meta AI responses.
According to the documentation, Meta-ExternalFetcher may bypass robots.txt when fetching pages at a user’s request. FacebookExternalHit, which creates link previews, may also bypass it while performing security or integrity checks.
Why This Matters
Meta’s Oct. 30 crawling and training grant comes with regular business uses of Meta’s products; thus, it includes brands and clients who have a Facebook Page or are in the Business Suite but do not buy ads.
The Robots.txt file is what Meta points to as the method to control its crawling. The Crawler Documentation listed above references Meta-AI training crawlers (Meta-ExternalAgent) and Meta AI Search Crawlers (Meta-WebIndexer); therefore, you may block the Meta-ExternalAgent crawler while allowing the Meta-WebIndexer crawler. Additionally, the new terms do not indicate if a business’s robots.txt settings restrict what it agrees to share by accepting these terms.
Looking Ahead
Meta’s new Commercial Terms are set to go into action on Oct. 30, giving website operators who use Meta for business two weeks in order to make their decision concerning whether or not they will continue to utilize Meta’s products for business. In addition, during this time, operators should review how their robots.txt files treat each Meta crawler.
The European Publishers Council (EPC) is also writing to the European Commission regarding Meta’s new terms. EPC is specifically concerned that Meta may be allowed by its terms to claim permission to use an article when a user shares a link to that story via social media.
