Back in March 2016 we started rolling out bug bounty programs for OX App Suite, PowerDNS and Dovecot. Today we'd like to share how we've been doing, what we learned and how we will go on improving the programs.
A bug bounty program tries to attract external security researchers to gather and share their findings on vulnerabilities with the vendor, rather than putting user data (and themselves) at risk by selling vulnerability information to criminals. Running such a program on our own was an option in the beginning, however we quickly realized that doing so would be rather challenging for a small organization in terms of reach, compensation, availability, responsiveness and analytics. Looking back, it is quite obvious that we would have been drowning in logistical overhead rather than ensuring quick responses and actually solving vulnerabilities.
Therefore we have chosen HackerOne as partner for this endeavour. Their platform provides access to over 300,000 security researchers and is well respected for doing business professionally with more than 1,200 customer programs. Their added services help customers to get the most value out of their bug bounty programs by removing many of the distractions and focusing on solving vulnerabilities. The folks at HackerOne handle logistics (payout, currencies, hacker tax forms), visibility into our programs for us and provide a unified interface to their researcher community. We assigned senior engineers as managers for each of our HackerOne programs to make sure external input gets reviewed, scored and resolved swiftly. To provide reasonable and transparent compensations for reporters, we're using our existing rating system for vulnerabilities, which is based on CVSS. This makes sure reporters are compensated based on actual impact in accordance to our priorities. While minor issues could be compensated with a swag bundle, critical vulnerabilities are compensated with up to $5,000. We also make sure reporters are being attributed on our Hacktivity page, on industry disclosure sites like this one and in our regular release communication, if they wish so; something which turned out to be more rewarding than compensating with money in some cases.
Running a bug bounty program on HackerOne complements our internal efforts to create secure solutions as well as running professional penetration tests with specialized companies. Such a program is less predictable in terms of results but it potentially involves thousands of researchers 24/7 instead of just a few for a limited period of time. As a result quantitative test coverage is much larger and various uncommon attack vectors are being discovered. Receiving a stream of well documented attacks and resulting vulnerabilities through the HackerOne programs really helped us to deal professionally with this topic internally. We've created sessions to review new findings, discuss potential side effects for solutions and think about other ways a specific vulnerability could be exploited. To some degree we even aligned some architectural decisions based on input provided by external researchers.
Looking at some metrics of the OX App Suite program:
- 244 reports were valid and were addressed through Security Patch Releases within days
- 0 reports went public before we solved and shipped them
- Our average time to first response has been 12 hours
- The average time to compensation was 3 days
- Our average time to fix issues internally was 5 days
- We paid out $81,348 in bounties; our MVP has made $22,300 on our program so far
Compensating external parties (and HackerOne) with $81,348 seems like a lot of money, so lets put this into perspective. A bespoke software penetration test costs something north of $30,000 and includes 2-3 weeks of research. Such tests can of course focus much more on specific topics than "unmanaged" researchers would. Therefore we continue to use these to review a specific concept or implementation. We usually get 10 relevant findings out of those test runs, making it a rate of $3,000 per finding regardless of its severity and relevancy. With our bug bounty program we see a rate of $350 per finding while getting a lot more feedback that helps us to protect our customers and learn a lot in the process. The signal-to-noise ratio of bug-bounty responses started quite high and got to almost zero after learning and using the HackerOne platform settings to fit our requirements. Much of the most valuable feedback comes from returning researchers that by now are very skilled in OX specific black-box testing methods as well as analyzing our source code.
Based on these metrics and the extremely positive effect on the security of our products we'll be continuing to invest in this aspect of security research along with managed penetration tests and automated code analysis. Last year we rolled out a "sandbox" environment which gives researchers on HackerOne a ready-to-use OX App Suite and Dovecot Pro platform to execute their research. This platform runs preview versions of our upcoming product versions and include new components to make sure they get attention and public exposure even before day one.
If you're interested in more details or like to participate at our bug bounty programs, please visit:
Let’s wrap this up with a big THANK YOU to all researchers who have participated at those programs and contributed their skills to make our products and by that a part of the internet a safer place for users data.
