The cost of ‘free’: How Wikimedia Enterprise protects Wikipedia in the AI era

The digital commons are under threat: with the advent of generative AI, bots have officially surpassed human traffic on the internet. Last year, the Wikimedia Foundation shared our concerns around drops in human traffic on Wikipedia. We also reported a significant increase in bot traffic to Wikimedia projects - largely coming from crawlers that extract content to train generative AI systems. Wikimedia projects such as Wikipedia, Wikidata, and Wikimedia Commons have been obvious choices for training: they are among the last bastions of high-quality, human-created and curated content. However, the resulting strain on our infrastructure and risk of impact on our user experience requires us to respond to this threat to continue our mission of making knowledge freely accessible to everyone, everywhere.

As the Wikimedia Foundation continues to take steady steps to protect its infrastructure, a big part of creating a sustainable solution for our knowledge mission is Wikimedia Enterprise, a commercial service and product we started working on five years ago.

Building Wikimedia Enterprise

Though it's become a more significant concern in the AI era, commercial reuse of Wikimedia data taxing our servers has been a problem for some time. For over a decade already, Wikimedia content has been crawled to power commercial products like search engines, voice assistants, knowledge panels, and any other service that needs up-to-date information about the world.

But while our content is free, our infrastructure is not. The companies running these services weren't the ones paying for the maintenance and upkeep of the servers they were accessing to get our content. Our individual donors were the ones footing the bill. The Foundation needed to find a way to get those commercial institutions profiting off the digital commons to help carry the burden of sustaining those commons.

Did we think this approach would be easy? Not in the slightest. Nobody wants to pay for something when you're already giving it to them for free. Still, we made a bet that if Wikimedia Enterprise could provide a comprehensive product experience that catered to their business needs of scale and speed with new features and functionality they want, reusers would be more easily drawn to make the switch.

The Wikimedia Enterprise team talked to big tech companies and other commercial reusers to find out how they were using Wikimedia projects in their products. Then we set out to build a product geared towards their use cases: a set of APIs for high-volume, high-frequency access that integrated and normalized the Wikimedia projects' many datasets into a unified offering with a well-structured, extensible data schema.

Wikimedia Enterprise's commercial strategy isn't licensing-based. It's access-based. The content of Wikimedia projects will forever be free and open to anyone. But Wikimedia Enterprise shines when you need access to that data at the volumes that automated scrapers and commercial businesses need: it offers automated bulk and real-time access to Wikimedia's vast and constantly updated store of knowledge, packaged in a product designed to seamlessly support a wide range of commercial use cases.

Five years since the launch of Wikimedia Enterprise, we can safely say: it's working. Since its inception, Wikimedia Enterprise has fully repaid its initial investment and is now contributing back to the Wikimedia Foundation's long-term sustainability. We did this by iterating the product offering and creating incentives for commercial organizations to switch to Wikimedia Enterprise's infrastructure, all without sacrificing the broader goal of keeping the Wikimedia projects free for human users and mission-aligned partners and communities. Income from Wikimedia Enterprise is capped at 30% of the total Wikimedia Foundation annual revenue. This cap was established in order to ensure that the majority of funding continues to come from small donations from millions of readers - a key ingredient to ensuring Wikimedia's ongoing independence. This has allowed the Wikimedia Foundation to continue serving its mission of preserving the open knowledge ecosystem, while forging meaningful partnerships with tech companies and other commercial businesses so that they can do their part to sustain and preserve the ecosystem as well.

Establishing boundaries for commercial reuse

Building the infrastructure of Wikimedia Enterprise and creating incentives for commercial organizations to switch to it is not enough. Bad-faith actors continue to tax our public API infrastructure to crawl and scrape, spoofing their identities, using residential proxies, and more. This has put multiple kinds of increased pressure on the Wikimedia Foundation's limited resources as a non-profit - and ultimately impacts the knowledge commons that the world relies on.

To ensure commercial reusers know they need to switch to Wikimedia Enterprise, we have to use new tactics. We're not switching off our existing public API access, now or ever, but we have begun rate-limiting large-scale reusers of the Wikimedia Foundation public APIs. If a company needs to access our data at a significant enough speed or volume, Wikimedia Enterprise is now the right place for them to go - a must-do rather than a nice-to-have. This rate-limiting is a difficult balancing act of protecting our infrastructure while fostering our community. It's not easy, but we're working on viable solutions that align with our community needs.

We're also continuing to build out the Wikimedia Enterprise platform to make it more valuable to non-commercial reusers who aren't profiting off Wikimedia data but still need access at scale. We're actively forging partnerships with non-profits and startups that align with the Wikimedia Foundation's goals or values to give them free access to the Wikimedia Enterprise APIs. Volunteers can get access to Wikimedia Enterprise through Wikimedia Cloud Services, and we introduced free access for smaller commercial reusers as well.

The Wikimedia Foundation operates within many systems that make the world function. Five years ago, we started Wikimedia Enterprise to translate between the non-profit system of the Wikimedia Foundation and the for-profit system of commercial businesses, so they have an easier pathway to reuse one of the largest high-quality, human-created knowledge resources on the internet. We told them that Wikimedia projects are so valuable that we made them free for the millions of humans who rely on them every day - but, as companies building products based on these vast knowledge commons, we need you to pay for the value you derive from them so we can keep them free forever.

Lane Becker is President of Wikimedia Enterprise and Senior Director of Commercial Partnerships). Jolan Wuyts, Senior Technical Writer at Wikimedia Enterprise, also contributed to this post.