Meta has removed training from the crawling clause in its new Commercial Terms, which take effect Oct. 30. The permission to crawl websites associated with a business’s products or services is still in the agreement.
The revised clause connects crawling to advertising and business products, as well as to the integrity and security of Meta’s products. Because the terms apply to any business use of Meta’s products, they also cover companies that manage a Facebook Page without buying ads.
Nieman Lab reported the revision on Oct. 17, two days after the outlet published legal experts’ view that the original wording could support an argument that Meta may scrape news sites for AI training. I covered the original terms on Oct. 16.
The European Publishers Council objected to the original wording before the revision, and the UK’s News Media Association condemned it in an Oct. 16 statement.
What Changed In The Clause
The original clause, as printed in Nieman Lab’s first report, said Meta “may use any content” it collected “to provide, develop, improve, research, train, and maintain the integrity of” its products, services and features under the Commercial Terms.
The revised clause allows the crawling “in order to provide, develop, improve, research, and maintain advertising and business products.” A parenthetical says those include products that help a business create and deliver ads and connect it with interested users. The clause keeps an integrity purpose, now worded as maintaining “the integrity and security of” Meta’s products.
Meta continues to utilize its prior access language. It may continue to use automated software and “create test credentials” in order to gather information from such websites; similarly, it will be able to periodically retrieve website updates. The new language does not include a reference to robots.txt.
As of Oct. 19, the UK English preview for Meta’s advertising agreement (the Self-Serve Ad Terms) still contained the previous statement indicating that Meta “may use any content” gathered from the aforementioned sites in order to “provide, develop, improve, research, train and maintain the integrity of” its products. The U.S. English version doesn’t.
Meta’s Explanation And The Legal Questions
Meta spokesperson Daniel Roberts told The Verge the crawling terms the company consolidated “only applied to our ads and business products, and we’ve clarified that in the terms themselves.” He said the update “does not change how Meta uses web crawlers” and that “This is not about scraping news sites.”
Meta also told Nieman Lab the first version “wasn’t clear” that robots.txt preferences are respected, and said the updates don’t give it “new rights that supersede those preferences.”
The legal experts from the first report raised concerns about the revised wording. Matt Topic, a media and intellectual property attorney at Loevy & Loevy, said, “I think ‘develop’ is broad enough to include ‘train.’” He added that “business product” doesn’t seem to be a defined term.
Cornell Law School professor James Grimmelmann said the edit does nothing to restrict the “associated with your products or services” language, which he said would let Meta scrape entire sites. He called the purposes language “still broad enough to drive a truck through.”
University of Wisconsin law professor Benjamin Sobel explained that robots.txt signals a site’s preferences. Following those signals is voluntary, and they don’t give the site an enforceable legal right.
Why This Matters
Meta provided a revised version of the stated purposes, but access was left unchanged. The company has cited robots.txt as the method that businesses can use to control which areas of their website they want crawled. In addition, the crawler documentation for Meta lists separate crawlers, one for AI training and one for Meta AI search. Therefore, if a business wants to disallow access for one but allow it for another they may do this with the help of robots.txt. It is unclear from the revised terms whether these methods limit what a business grants when they accept the terms.
Looking Ahead
The revised terms take effect on Oct. 30. Meta has updated the preview since it was first posted, so please check the clause again before then, and review how your robots.txt file handles each Meta crawler.
