Bad Stuff Can Happen

What happens when the world’s largest ad server network goes down? Google Ad Manager’s publisher ad server (previously Doubleclick for Publishers) towers over the ad server market, with an 80%+ market share according to Kevel Tracker research worldwide, and a 90% market share in the US according to Reuters. Well on Thursday, December 8, 2022, Google Ad Manager (GAM) had an outage during the busiest season of the year. No ads were served during the outage - worldwide for over two hours! https://twitter.com/GoogleAdManager/status/1601047271411359744?s=20&t=S9rXaBcZ8MGqBAfC7Lg7sw

https://ads.google.com/status/publisher/incidents/EsQcYmPW6wkQvc9NZ4WB

Obviously, the Google team pieced it all back together and got things back on track in about two and a half hours from start-to-finish. Indeed, it was a herculean restoration task! Kudos to Google’s response team, regardless of the reasons for the outage, they rectified the issues and got everything back online by 10:40 pm ET. That’s only part of the story.

Similar, but smaller and shorter, incidents have happened in the past. “Google Server Crash Leaves Gmail and Google Drive Inaccessible” identified the Google incident that happened on March 12, 2022. Also, the Wikipedia article “Google Services Outages” tracks other outage instances back to 2013. But, that’s not the whole story.

What's The Bottom Line?

The other side of the story is that publishers in the ad ecosystem who utilized GAM, lost all of their ad revenues during that outage. “We and all the publishers we work with are getting no ad revenue. Literally, every company we know has been affected,” said Paul Bannister, Chief Strategy Officer at CMI Marketing Inc’s CafeMedia. “Assuming a consistent number of impressions per hour, the three-hour downtime cost publishers about 12.5% of their daily GAM revenue, or about 0.4% of their monthly GAM revenue.” Using very conservative numbers, publishers lost a minimum of $20 million in ad revenues during the global outage.

Under ordinary circumstances, getting something fixed that broke down is no big deal. We all expect that from online service providers. However, getting something huge fixed in record time is awesome, and as we just mentioned, we believe the Google response team is amazing considering that GAM has the largest market share of any publisher ad server worldwide. The incident highlights that even the largest ad server network in the world can have issues that impact their business and yours - without warning. Since other outages had previously occurred, it was not a total surprise that it could happen again. But the scope and duration of the event did set new record highs.

Metaphor for Malvertising Detection and Response

What was nearly as important as the outage itself, was the speed that the response team was able to mitigate damage and restore the network. Google recognizes that huge ad revenues are at risk on their network every minute. So the lesson we should do well to learn is, things will break in unexpected ways - it is impossible to expect the unexpected - speed to detection is therefore critical. The key is to anticipate that issues will occur regardless of the efforts to protect against them, and to be prepared with the resources to respond rapidly is the best strategic approach.

When we designed Confiant’s system to integrate at multiple points of the ad tech supply chain, it was because we understood that we needed to accurately monitor everything, especially when new threats are constantly surfacing, extremely subtle, and hardest to identify. When there's a shift in the attackers' approach it often creates a massive wave of attacks that are built to overwhelm the industry’s security solutions. Two years ago the number one issue was phishing redirects, last year it was cloaked investment scams, and today it is cloaked tech support scams. On any given day though, one individual platform or publisher may be facing none of those or all at once. Anyone serious about their security needs to ensure they have access to an incident response approach and be prepared with their brightest and fastest resources to respond, just like Google did. Recognizing that it is a technical truth that there is no way to ever identify and block every malvertising attempt helped us strategize that our speed of detection for new threats is more important than maintaining visibility of older threats, and helps us protect against previously unknown threats

“The feedback loop the ad tech supply chain offers malicious actors is 100 times stronger than in other threat vectors,“ Louis-David, ‘LD’, Mangin, Confiant’s CEO contributed. “Effective novel threat detection requires systematically marrying behavioral code analysis with static code analysis and human ingenuity. The speed of the humans’ reaction and their level of ingenuity especially matter”.

Our ad security and quality solution keep clients safe from known attacks, but our threat detection and security experts must always search to identify and rapidly mitigate issues as new attacks arise and escalate. It’s the combination of the solution and the expertise that protects the ad ecosystem from issues that can have a huge impact on businesses, reputations, and revenue streams. When everything is working properly, the solution is nearly invisible to the outside world - everything just functions normally and nothing seems wrong. When issues do arise, it’s about how fast your vendor can help you mitigate any damage the issues can cause.

Learn more at our Malvertising Attack Matrix: https://matrix.confiant.com/