How Pinterest built its Trust & Safety team
Maisy Samuelson | Head of Trust & Safety Product
“Inch by inch, row by row, I’m gonna make this garden grow” — Garden Song by Peter, Paul & Mary
Since I started working on Trust & Safety four years ago, the team has grown quickly and learned a lot about how to protect Pinners from spam and account takeovers as well as unsafe content like nudity, self-harm, hate speech, and harassment.
Back in 2017, the Trust & Safety team (like Pinterest itself) was small and spent a lot of time fighting attacks. During attacks we scrambled to figure out product nuances, where and how data was stored, wait for queries to run, write scripts to update production databases and fix issues in user accounts, all under pressure to quickly resolve the issue. It was clear that we needed a larger team with expanded tools and systems, especially because the Pinterest user base was growing rapidly and becoming a larger target for spammers.
Now, the Pinterest T&S team is much more robust across engineering, product and data analysis, as well as Operations and Policy. This article focuses on how you can build out teams, tools and processes for fighting spam, one important part of your overall T&S strategy.
Team building
Your first priority needs to be building the team because without a strong team nothing else matters. As part of building the team, you need to educate leadership about Trust & Safety’s unique challenges.
For example, spammers attack unexpectedly, often on nights, weekends, or holidays when they hope teams will be distracted. Absent adequate people and tools, the on-call burden quickly becomes unsustainable. Leadership must understand these unique T&S challenges to resource and recognize the team accordingly.
Here are some tactics that worked at Pinterest:
“Code Red” Meetings
For the first year, I set up a monthly meeting with the VPs of Product and Engineering to discuss critical team issues, make clear requests, and create urgency.
Roadshows
In the beginning, we used every possible internal forum to educate the company and our leadership about potential T&S problems at Pinterest, why they were important, what our strategy was, and how that audience could help. We also highlighted specific examples where resources were needed and we made sure to explain and celebrate our progress.
Collaborating with Product teams
You’ll need deep collaboration with Product teams to be successful. At Pinterest, we assign a T&S PM, analyst, or engineer to each major product area. The T&S representative is responsible for helping partner teams appropriately integrate T&S signals, logging and defenses, and finding the best balance between safety and engagement. For example, requiring users to verify a phone number in order to create content will reduce spam significantly because it increases the cost of creating fake accounts. However, it will also prevent some good users from creating good content. You need to work with Product teams to decide the best tradeoff between competing goals.
Foundations
It’s tempting to start developing machine learning models immediately. Instead, I recommend that you focus on building a strong foundation first, including:
- Documenting enforceable policies and processes for iterating on them over time
- Making it easy to query what is happening on your platform in real time and safely take action on policy-violating content and accounts
- Building a tool that allows reviewers to see the data they need to make accurate judgments about whether a user or content is policy-violating
- Logging all actions taken by the Trust & Safety team
- Creating appeals flows and process
For example, without clear policies or appropriate review tools, it will be prohibitively expensive for reviewers to create labeled datasets to train your models and measure their performance over time.
Policy & Operations
You’ll want to publish policies that set clear expectations with your users about what’s not permitted in your product. Enforcement guidelines constantly evolve as new trends and nuances arise. If you wait to act until your enforcement criteria cover all possible scenarios, you will never act, and you will expose people to harm. The key is to establish a clear and enforceable starting point and create a process for updating it quickly. Spam tactics change constantly, so you should expect to regularly update your policies and enforcement tools also.
Let’s walk through a hypothetical example: you work at a social network and you want to detect spam accounts. First, you should try to find as many examples of spam users as possible. One obvious place to start is by looking at accounts that users have reported for spam. These examples might reveal a number of spam tactics including:
- Sending messages that contain links to spam websites
- Publishing a large amount of content linking to spam websites
- Creating account profiles that look like someone a user is already connected to in order to trick the user into accepting a friend request and then spamming their feed.
Next, you can use these examples to write v1 of your spam enforcement guidelines and create tools so that reviewers can enforce the policy. The enforcement guidelines will be relatively simple at first. Here’s a possible row in your spam policy:
You might define logic such as “mark the account as spam if it has”:
- One high severity signal, OR
- Two medium severity signals, OR
- Two low severity signals and one medium severity signal.
At first, it’s okay to choose thresholds by looking at a number of examples and using judgment. The key is to start somewhere and iterate quickly. For example, if you find that a threshold of 5 frequently leads to deactivating accounts that you don’t think are actually spam accounts, you can adjust the threshold higher.
While developing v1 of your enforcement guidelines, you’ll also need to build a review tool that allows reviewers to see all the important spam-related data about that account, presented in a way that’s easy to digest and understand quickly. You can read more about Pinqueue, Pinterest’s internal review tool here: Introducing Pinqueue3.0, Pinterest’s next-gen content moderation platform
When developing a new policy or enforcement guideline, you should measure reviewer accuracy and the time it takes for reviewers to make a decision. You can do this by providing examples for the reviewers to evaluate and then measuring how many reviewers got the result that the team expected.
Going through this exercise when developing a new policy will help identify areas you can improve, for example, by one or more of the following:
- Clarifying the policy
- Improving tools to provide more data so that the reviewers can make accurate decisions and/or
- Enhancing training for reviewers.
Data & Tools
In order to shut down bad accounts and content, you need to know what’s happening on your platform right now. Tactics include:
- Create comprehensive anomaly monitoring and alerting: We leverage Druid and our internal tool Statsboard. For example, a large spike in Pin Creates on a Thursday of one week compared to the Thursday of the previous week very likely indicates a spam attack rather than organic user behavior. As Trust & Safety improves you should expect the team to detect all attacks before Product teams flag them. At Pinterest, we’ve developed systems that automatically detect and take action in response to anomalies. You can read more here: Fighting Spam using Clustering and Automatic Rule Creation.
- Make it easy and quick to query data in real time and act on users who meet certain criteria: We built a tool called Guardian for this. You can read more here: Guardian: Fighting Spam with Guardian, a Real time Analytics and Rules Engine.
- Review user reports to find new trends.
- Create tools that allow reviewers to easily see all the information they need to make an accurate decision about an entity. At Pinterest, we call this tool “Lookup.”
Today, Pinterest is better equipped to defend against spam thanks to investments in alerting, automation, process, and tools.
Signals
The heart of any T&S strategy requires identifying signals that differentiate good actors and content from bad. The best way to identify strong signals is to look at a lot of examples of both good and bad entities.
In our hypothetical social network example above, spam signals might include:
- Not many followers or friends
- Followers or friends who joined the network all around the same time (i.e. looks like fake accounts)
- A low friend request acceptance rate
- Shares many suspicious links.
Graph-based signals like these would be hard for a spammer to mutate around because it’s difficult to create a friend graph that mimics a normal user.
Note that you should find examples of a specific problem before you build automated defenses against it. It isn’t feasible (or a good investment of resources) to try to build signals against hypothetical threats for which you can’t find concrete examples. You’re better off monitoring what’s happening on your platform and responding quickly to new threats as they occur.
When deciding whether to develop a signal, you should consider:
- Cost of building and maintaining the signal
- Incremental bad stuff the signal catches (incremental gain in recall)
- Incremental good stuff the signal mistakenly catches (incremental loss in precision).
Evaluating new signals is important work that requires significant process and tooling investments. Finally, you will need to choose between erring on the side of false positives (removing good content) or false negatives (showing bad content).
Partly, this is a philosophical question that depends on your company’s mission. For example, if your mission is to “organize the world’s information,” you might want to err on the side of false negatives and potentially show some spam in searches rather than risk removing good content. By contrast, if your company’s mission is to uplift people, then you may want to err on the side of false positives to avoid recommending potentially bad content. Regardless, you should provide a robust appeals process to help correct any mistaken removals.
Conclusion
Trust & Safety threats are constantly evolving. As a result, one common T&S theme is iteration, and you should design all of your systems with easy-iteration in mind. This means it should be easy to tell why your systems are acting and non-engineers should be able to improve the system whenever possible. For example, Guardian uses a SQL-like query language so that non-engineers can easily QA and write rules.
Feel free to reach out to me via LinkedIn if you’d like to delve deeper into T&S topics.
Acknowledgements
Many people have meaningfully changed the course of spam fighting at Pinterest, but I wanted to especially thank those who made a big difference in the earlier days when the team was fragile and facing significant tooling, infra and resourcing challenges, especially: Beatrice Zhang, Dennis Horte, Preston Guillory, Alok Singhal, Rundong Liu, Honkai Pan, John Thrall, Lawrence Ripsher, Vijay Narayanan, Vanja Josifovski, Kate Flaming, Becky Stoneman, Farran Wang, Sharon Xie, Minli Zang, Attila Dobi, Dan Towne. I’d also like to thank Sev Guardado, Sarah Bromma, Becky Stoneman, Crystal Espinosa, Dan Towne and Sara Rowe for providing feedback on this blog post.
To learn more about engineering at Pinterest, check out the rest of our Engineering Blog, and visit our Pinterest Labs site. To view and apply to open opportunities, visit our Careers page.
