With the continuous shipping nature at GitHub, it’s easy for the most well-intentioned feature to accidentally become the vector of abuse and harassment. The Community & Safety engineering team focuses on building community management tools and maintaining user safety, but we also review new features our colleagues have written to ensure there are no accidental abuse vectors. Similar to Application Security reviews, these Community & Safety reviews hopefully catch any potential problems before they go out, in order to minimize impact on marginalized folks, reduce spam, and encourage healthy communities.
But manually reviewing every pull request doesn’t scale, so we’ve created a handy checklist for folks who haven’t had the privilege of being harassed on the internet for things to look out for.
Our approach focuses on three main areas: ensuring explicit consent, keeping an audit log trail, and minimizing abuse.
On the Community & Safety team, we believe in explicit consent in our daily lives as well as when we build software. Many abuse vectors can be avoided by simply asking: Are all parties involved aware and consenting to this interaction? If everyone is aware and on board with what’s going on, we reduce the number of unpleasant surprises, lower support ticket volume, and increase user trust. A great example of explicit consent is the Repository Invitations feature by @CoralineAda.
Any time you have two or more users interacting, there’s potential for harassment and abuse. Let’s say that Alice has been contacted by Bob using your new feature (i.e. direct messages).
Some example questions we ask to ensure explicit consent include:
Support folks are the unsung heroes of all matters related to Community & Safety. They are often dropped into a battlefield with very little context of what’s going on. Make it easy for your support folks to help your users quickly and with minimal digging by ensuring there’s a clear audit log trail available. Audit logs keep track of your activity, and any organization you own, and can be very helpful to provide context and accountability in the event that something goes wrong. You can read more about audit logs in the documentation.
Some example questions we ask to ensure proper audit trail logs include:
Many sites are optimized for easy account creation, but this often leads to spam or sock puppet (throwaway) accounts that are handy harassment tools. Limiting the amount of features 0-day accounts can access on high-risk features can help limit abuse.
Some example questions we ask to ensure minimal abuse vectors include:
These are just some things to think about that can help your teams curb abuse vectors on new features before they go out. We hope that this checklist will help you build safer products and lead to happier users.