CDN's and the recent trend of Blacklisting Genuine Customers

There has been a recent shift towards using Content Delivery Networks to distribute content rather than hosting in a conventional way, and this brings with it a selection of good and bad. One of the regular issues we receive at the HelpDesk is primarily generated by Amazon Cloudflare which offers 'free' content delivery, making it a popular choice for smaller websites. The most common complaint is the screen right "One more step", which prevents the customer from visiting the website without completing the infamous Google ReCaptcha. Given the serious privacy concerns surrounding Google ReCaptcha would it be Amazon or the website owner who is responsible for *not* highlighting this to the end user? Regardless, our standard answer in this case (and it's a canned response now) is "Unless this website is business critical close the tab and select another website". There is some suggestion that these messages are generated in an attempt to rate-limit or reduce load on either Cloudflare or the vendors website but this is unconfirmed. 

So what causes Cloudflare to blacklist business customers from visiting their vendors websites? Amazon will claim that they blacklist IP addresses that exhibit unusual traffic as well as those on commercial blacklists. That sounds great in theory, but in fact with the vast majority of client IP's being dynamic (including mobile devices) this blacklisting simply prevents customers reaching vendors and for no technically good reason. If their blacklisting wasn't inherently flawed then we would not see the volume of Helpdesk requests on this very issue, with genuine customers trying to reach genuine vendors, and its for this reason that we no longer offer Cloudflare as an option on our hosting services. 

Another example of Cloudflare blacklisting, this time suggesting that the website owner enabled this block is the message to the left with "Error 1005". In this case we're shown that the network AS8560 is blocked from accessing the site. This HelpDesk ticket was raised by a customer who was in fact in Germany, using a tablet in a coffee shop and who wanted to check who had been blowing up their mobile. There are of course other sites which I'm sure satisfied their curiosity but the customer was concerned that the message may have been an issue, because quite honestly to the end user it is a little intimidating. 

In the "Access Denied" message to the left we again have a genuine customer who was trying to access their account on a vendors website, and yet again we're told that its not going to happen, this time suggesting that the client is somehow responsible for an online attack. They are of course not responsible for anything except trying to access their vendors site, but again this sort of message just generates HelpDesk requests, takes time and effort to explain to the customer they've done nothing wrong and that they should consider another vendor in future. In this particular case "Error 1020" indicates that the website operator has established this block as a firewall rule which you would think was intentional but I can't speak for the site or site owner. 

That's enough of Cloudflare, which is after all a free service for most and with that you cannot really complain if you knew it was happening, the very issue here though is that the vendors operating their websites in most cases are unaware that customers are being turned away or impeded from visiting. The prevalence of Cloudflare means that once a customers IP is blacklisted, a good few sites in their daily browsing will all be met with the same resistance. You could say - Contact the vendor, but how do you do that when you can't access their website? 

Cloudflare is not alone and there are a growing number of alternative Content Delivery Networks all bringing their own flavour of issues to the market, preventing customers from visiting vendors and there can be nothing worse to a growing e-business. I understand that protecting the business from 'attack' is a good idea, but in reality we're not protecting them from anything, what is happening is the content delivery network is protecting itself from excessive load at the vendors expense. 

One effective but equally concerning method around this is to use a free proxy server, and the internet is full of them - just search "free proxy server". These servers whilst for the most part are safe, have the ability at the protocol level to intercept your traffic, even that over HTTPS which presents a clear danger. Whilst it's beyond the scope of this article to discuss the technical ramifications of http proxies our recommendation is please do not use them. 

Summary

The idea of CDN's is great and has a mostly positive effect on content delivery and site speed, but when your CDN starts blocking customers, either itself or due to (mis)configuration from visiting your site then you need to asses the overall benefit to the business. In other words what is the likelihood of your website being 'attacked', and in 'attack' we mean an attack that a CDN can block (which is actually very few) verses the potential lost business due to customer rejection. It's a hard one and as CDN's become more popular I think it will be increasingly relevant. 

Looking through 3 months of tickets raised in our Support/Web/Browsing channel and selecting a few from the list I find: 

• analog.com (analog devices) access denied - customer was looking for components for project - went elsewhere. 
• semrush.com : various - customer was trying to access account - gave up trying. 
• moneysavingexpert.co.uk : one more step - customer was following link from google - filled box still rejected. 
• fiver.com : one more step - customer was trying to buy services because we are 'too expensive', got to love tickets - customer went to seoclerks.com instead. 
• yell.com : forbidden - customer was trying to find business phone number - directed them to alternative website. 
• yelp.co.uk : sorry you are not allowed to access this page - customer was trying to check reviews - went elsewhere. 
• scottishpower.co.uk : one small step/not a robot - customer trying to contact company - agent found phone number for customer and advised them to compare prices. 
• rswww.com : permission denied - customer trying to purchase components - customer went to another supplier.
• royal applications.com : An error occurred in retrieving update information - This took 4 hours of helpdesk time to determine that the update url "royaltsx-v4.royalapplications.com" is a cloudflare url and being blocked. 
• rigol.com : one more step - customer was trying to compare equipment specifications - customer attempted to complete captcha but was then told they were blocked. 
• talktalk.co.uk : Request Rejected - customer was trying to report a fault on their service - customer was persuaded to source bb elsewhere. 

There are many more, and a lot of tickets don't actually specify the website but you get the idea, from our small subset of customers 46 of them gave up and were advised to go elsewhere. There's no way to tell how many successful customers were able to access these sites and how many we're presented with stupid rejection messages so our sample set is the only indicative data we have, but its statistically significant in this scenario.