Categories
Security

Cloudflare Data Leak: How to Secure Your Site

Cloudflare has experienced a data leak over a 5 month period that mixed sensitive data between websites and visitors. A visitor to one website using Cloudflare may have seen data from another website using Cloudflare that was being sent to a completely different site visitor.

Some of the leaked data has been indexed by search engines who have been working over the past few days to try and remove the data from their caches.

In this post I am going to explain in simple terms, what occurred and what you need to do about it.

If you are a WordPress user and simply want to know how to secure your site, you can skip to the What Should I Do section below. I have included some information for non-WordPress site owners in that section too.

What Happened in the Cloudflare Data Leak?

Cloudflare provides a firewall and content distribution service. Their servers are between your website visitors and your own web server.

Under normal circumstances, cloudflare returns the data each site visitor requested to that visitor. This may be public or sometimes private information and it is usually done over a secure channel. Each website visitor only sees the data they requested.

From September 22nd, 2016 until February 18th 2017 (last Saturday), Cloudflares servers in some cases mixed data that belonged to one visitor to a website, with data belonging to another visitor that was visiting a completely different website.

The worst data leakage occurred between the dates of February 13th and February 18th when one in every 3.3 million requests to Cloudflare’s servers was leaked.

During the period when the leak occurred, visitors to certain websites would see what appeared to be garbage data mixed into the web page they were viewing. That garbage data was data from a memory leak. The data was in some cases sensitive and included security tokens and other sensitive information.

According to Tavis Ormandy, the researcher who discovered this data leak:

“The examples we’re finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I’ve informed cloudflare what I’m working on. I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”

Data leakage occurred when a site visitor or search engine visited one of 3438 domains hosted behind Cloudflare’s servers, according to Cloudflare’s CTO who posted a comment on Hacker News. However, any of Cloudflare’s customer websites could have had their response data mixed into data returning from those 3438 websites. Tavis Ormandy confirms this in the same Hacker News thread.

You can see an illustration of this data leakage in the diagram below. Any visitor to an ‘affected website’ which is one of the 3438 websites, could have had data from any one of over 5 million Cloudflare customer sites mixed into their response. Website 1 and Website 2 which are not ‘affected’ websites could have experienced data leakage to visitors of the ‘affected’ website.

What Data Was Leaked?

The data that was leaked could include passwords, cookies and authentication tokens. If an attacker is able to access the text of your cookies, they may be able to use them to sign into your website.

Internal Cloudflare private keys used to secure data being transferred between Cloudflare machines were also leaked.

At this point one should assume that there is a small chance that any private data transferred from any Cloudflare customer website to a site visitor may have been leaked between September 2016 and February of this year.

According to Cloudflare, no private SSL customer keys were leaked from the memory of their servers. A private SSL key is a key used to secure visitor connections to your website. If the your private SSL key is leaked, an attacker could listen in on all traffic to and from your website.

Has the Leaked Data Been Stored Somewhere?

Unfortunately Google and other search engines have been crawling the web during the time that this leak was occurring on Cloudflare’s systems. Those search engines stored the leaked data when they indexed one of the 3438 affected websites.

When viewing cached pages in Google, it is still possible at the time of writing (7pm Pacific Time on Feb 23rd) to view cached sensitive data in Google’s search results.

Similarly it is possible to view sensitive Cloudflare data in DuckDuckGo’s search results.

Since this leak was discovered, Google and other search engines have been working to try and remove the sensitive data from their caches. Based on what we are seeing this evening, there is still some data that needs to be removed.

What Should I Do if I use Cloudflare on my Website?

According to a conversation on Hacker News between the Cloudflare CTO and Tavis Ormandy, the security researcher who discovered this, any customer of Cloudflare’s could have been affected by this data leak.

WordPress site owners: Change your wp-config.php salts. This will log everyone out and invalidate cookies and sessions

If you are using WordPress, we recommend you edit your wp-config.php file and change all of the ‘salts’. This will automatically log all of your users out. This protects you and your site members in case any of their cookies have been stolen. Once you make this change, an attacker will no longer be able to use stolen session cookies from your site to sign in.

You need to change the following section in your wp-config.php and save it:

We suggest that you change the highlighted text in your wp-config.php to a long random string of characters and numbers. You can also use the link in the comment above the ‘define’ statements to generate a salt.

Non-WordPress Site Owners: Invalidate Sessions

If you use a different publishing platform, you will need to ensure that all sessions are invalidated. That means that your site visitor login cookies need to be made invalid. You will need to consult the documentation of your particular publishing platform to determine how to do this.

Suggest your site members change their passwords and change your Admin passwords

As a precautionary measure, you should suggest that your site members change their passwords. You should also change any admin level passwords.

You may need to comply with any data breach reporting requirements you have

This bug in Cloudflare’s systems is being described as a “data leak”. It is unclear at this point whether it is considered a “data breach”. A private database of customer personally identifiable information was not stolen. However, private data that may have included customer PII was leaked, no matter how small.

If you have HIPAA, PCI or other reporting requirements that relate to data breaches, you may want to get advice on whether you are required to report this incident.

Check the Search Results

It is difficult to determine if any private information from your site has been stored in the search results. However, we recommend that as a precautionary measure you do a few Google searches with your domain name in quotes. Add the following text to the search:

-site:example.com

Replace example.com with your own site domain. Take note of the minus sign before the word ‘site’ above. This will exclude results from your own website. You can exclude results from other sites using the same operator several times. You can also try adding the following in quotes:

“CF-Host-Origin-IP:”

If you do find any results, report them immediately to Google for removal.

Who Discovered This and How?

Tavis Ormandy discovered this data leak in Cloudflare’s systems. He is a security researcher employed by Google’s Project Zero. Project Zero is a Google team who works on trying to find zero day (previously unknown) vulnerabilities.

Tavis discovered the data leak while analyzing Google search results. He noticed data that appeared to be a raw memory dump and he and his colleagues took a closer look and discovered it was a data leak in Cloudflare’s servers that was leaking data between websites.

Tavis is a well known researcher who has done ground breaking research in the computer security field over the past few years.

Has This Problem Been Fixed by Cloudflare?

The Cloudflare team have fixed the data leak on February 18th which was last Saturday. You can find a detailed technical post on Cloudflare’s blog describing what caused the leak and how it was fixed.

According to Techcrunch, Cloudflare have not notified customers like Uber and OkCupid directly.

Where Else Can I Read About This?

The following are the most authoritative resources discussing this issue:

How can I help?

Please share this information with any other site owners you know to help them secure their websites. Specifically, if you know any WordPress site owners you should suggest that they invalidate their user sessions using the instructions above.

As always I will be around to respond to any comments or questions you post below.

The post Cloudflare Data Leak: How to Secure Your Site appeared first on Wordfence.

Categories
Security

Revslider, MailPoet, GravityForms Exploits Bypass Cloudflare WAF

Last week we blogged about the advantages of endpoint security over a cloud firewall solution. We wrote about how cloud WAFs can be bypassed. We also blogged about how it is more challenging for a cloud WAF provider to write complex firewall rules because cloud WAFs don’t know if a user is signed in or what their access level is.

Part of the forensic research we do at Wordfence involves analyzing attack data we receive from sites that use Wordfence. We use a scaleable database cluster to perform big data analysis on WordPress attack data. We identified many attacks that were bypassing Cloudflare and being blocked by Wordfence. So we dug a little deeper.

Cloudflare Pro provides a web application firewall that is designed to perform a similar function to the Wordfence WAF. We are in that sense, direct competitors. We wanted to evaluate the Cloudflare WAF and to get access to it you have to get a paid ‘Pro’ account for $240 per year or $20/month. So we bought and paid for the Cloudflare WAF.

The default Cloudflare WAF sensitivity setting is ‘Medium’. We increased the sensitivity setting to ‘High’.  That is the highest sensitivity setting before your users have to get through a captcha to access your site.

We also enabled every rule we could find in the Cloudflare WAF. That includes 11 rules in the “Cloudflare ruleset” and 20 rules in the “OWASP ModSecurity Core Rule Set”. We also put that ruleset on “High” sensitivity. We also enabled the “browser integrity check”.

We enabled absolutely everything we could find and put everything on “High” sensitivity.

We then confirmed that we could bypass the Cloudflare Pro WAF with the following attacks using no special techniques:

  • Revolution Slider – We gained a remote shell. This went through completely undetected.
  • MailPoet – We gained a remote shell. Also completely undetected.
  • Gravity Forms – We gained a remote shell. Also completely undetected.
  • Timthumb – Gained a remote shell using the .phtml form of the attack. Detected but not blocked.

These results were surprising. We used off-the-shelf hacker scripts without any special modifications. It’s well known that RevSlider, Gravity Forms and Timthumb are three of the leading causes of hacked websites. According to one report, 25% of hacked sites are hacked through one of these three WordPress exploits. Cloudflare Pro at $240/year with a ‘High’ sensitivity setting and all rules enabled allows these attacks through.

The free version of Wordfence blocks all of these attacks.

Why do these well known attacks bypass Cloudflare?

We don’t know why Cloudflare allows these attacks through, as surprising as it is, but I’d like to share a few observations. Firstly, Cloudflare is not WordPress specific. They are trying to be a firewall for all web platforms which is a difficult, perhaps impossible, challenge. Wordfence is WordPress specific, so we are able to tailor our rules for attacks that we know target that platform specifically.

Cloudflare is a ‘cloud WAF’ and, as we have pointed out previously, because their servers and rules run out on the Internet, they don’t have access to authentication and authorization data to make their rule decisions. Wordfence on the other hand knows if a user is signed in, what their identity is and what their access level is, so we are able to write more complex and stricter rules.

Demonstration

We have created a video demonstrating Cloudflare being bypassed by these exploits. In the first test we have a site that is filtering traffic through Cloudflare Pro with all rules enabled and on a ‘High’ sensitivity setting. In this test we also enable the free version of Wordfence on our target site. This allows us to see the attacks bypassing Cloudflare and being blocked by Wordfence.

cfwf2

In the next test, we remove Wordfence completely from the target site and demonstrate how, without it, the site is exploited by an attacker, completely bypassing the Cloudflare Pro WAF on ‘High’ sensitivity.

 

cfexploited1

 

 

The following is a video demonstration of this attack. In it we use two Linode servers, one as our attacker and another as our ‘Victim’. We use a Cloudflare Pro account on ‘High’ sensitivity with all rules enabled. We also download and configure the free version of Wordfence for the first part of the demo, and then remove it.

Why does this matter?

The free version of Wordfence blocks all of these attacks. They are what we consider “the basics” when it comes to WordPress security. If you pay us $99 a year you also get a real-time feed of emerging threats. Cloudflare are selling a web application firewall for $240 per year that allows through the best known and most dangerous WordPress attacks.

That means you can get better protection by using our free product than by using the $240/year Pro Cloudflare WAF. Then, if you choose to upgrade to our paid option, you know you’re protected against the newest emerging threats against WordPress.

It’s important to us that our customers know this. We feel we would be doing you a disservice if we didn’t share this comparative data. If you want to protect your WordPress site from a hack, you need a WordPress specific firewall that runs on the endpoint and protects you against “the basics” and also against emerging threats.

Conclusion and Technical Notes

We have provided a public Github repository that include tcpflow packet captures from the perspective of the attacker and from the perspective of the victim. The repository also includes the four exploits we used. Note that you will need to edit the exploit files to add in your own target hostnames. In the case of Timthumb you will also need to add a server from which timthumb can download an attack shell.

We have not included the vulnerable plugins or theme. However, we are supplying their versions. They are: Gravity Forms 1.8.1, MailPoet 2.6.4, Revslider 2.3.91 and Timthumb version 1.12. You can find these in the WordPress.org repository, in other repositories on github and elsewhere.

If you do download and test one of these vulnerable products and find you’re not able to exploit them, you may be using an old version that has a back-ported security fix. So please note that you need to find both a vulnerable ‘version’ of the product and one that has not had a back-ported security fix applied.

As always I welcome your comments and questions.

Trademark notice: All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

The post Revslider, MailPoet, GravityForms Exploits Bypass Cloudflare WAF appeared first on Wordfence.