Closing the Gap Between Policy and Practice

Contents

• Where We Started
• What Changed
• Limitations
• Help Us Improve

People come to Runway to tell stories, and many great stories involve conflict, danger or violence. Over the last two years, our moderation has evolved to allow far more of those stories while continuing to block content that causes real harm.

Every moderation system has two parts: the policy that says where the line should be, and the tools that enforce it. We've always aimed to support creative expression and prevent harm with the best tools available to us, and those goals have not changed. What has changed is the tools themselves, and as they've improved, so has our ability to draw and apply lines that make sense for creative work.

Where We Started

Two years ago, we relied on general-purpose tools that were among the best available at the time, but weren't built for creative work. These third-party classifiers scored content in fixed categories like "blood" or "weapons," and we could only choose which categories to use and the score at which to block content. The classifiers couldn't tell a fantasy battle from real violence, and similar content didn't always get similar scores. To make sure we caught the most harmful content, we had to block more content than we wanted to.

Violence shows this most clearly. With the categories available, the only practical way to block violent content was to block blood, and we couldn't precisely control how much blood was too much. This meant that for a time, we blocked content showing even a small drop of blood or a minor scratch, and sometimes blocked things like ketchup or red paint when our tools mistook them for blood.

Examples of content our policies allow but our earlier tools would have blocked.

What Changed

Advances in large language models (LLMs) over the past two years have improved our moderation in two separate ways.

1. We can set policies built for Runway. LLMs let us build a moderation system around our own policy decisions, rather than adhering to a third party's policy lines. We can write policies for the ways people actually use Runway, and update them much more quickly (in a matter of hours or even minutes) as usage patterns evolve.
2. We're better at applying those policies. LLMs can take in more context surrounding a piece of content, and apply more nuanced guardrails. This means that our moderation is more accurate in both directions: we block less content that should be allowed, and we catch more content that shouldn't be.

Going back to violence, these changes allow us to draw a line between creative expression (which we want to allow) and harmful content, such as violent content directed at protected groups or recreations of violent extremist attacks (both of which we block). That nuance allows our users to incorporate sword fights, shootouts, battle scenes and horror in their work. These performance improvements also mean that we can more accurately catch attempts to create sexualized content involving minors or non-consensual intimate imagery (NCII).

Limitations

Even with these substantial improvements, some limitations remain, and it's important to us to be transparent with users about what those are.

Third-party models. Many of the third-party models available on Runway apply their own moderation. A prompt that Runway allows may still be blocked by that model provider. We recently rolled out new error messaging that makes it clearer to users when moderation is due to a third party, and we'll continue to refine this on an ongoing basis.

Automated mistakes. LLMs still make mistakes. For example, they may over-enforce on an ambiguous word or miss harmful content framed as fiction. We are continuously iterating to address these mistakes, and we layer our safeguards so that the most serious harms have the most protection.

Missing context. No matter how strong our moderation systems are, there inevitably will be times when they fail to capture all the necessary context. Even content that nearly everyone agrees we should block can have a legitimate creative use. For example, a film might follow a character who discovers that someone has made sexually explicit deepfakes of them. That's a story worth telling, but a request for that scene can look the same as the abuse we're trying to prevent. When we can't tell the difference, we will sometimes block a valid use case to prevent real harm.

Help Us Improve

If you think we got a decision wrong, report it to us by clicking the "Send Feedback" button at the bottom of the error message on the moderated generation. This helps us refine our policies and systems so that more stories can get made.