In mid-December we updated our Vulnerability Disclosure Policy to include Service Vulnerabilities. A service vulnerability is any issue with a technology service that represents an exploitable security risk for its users. We made this update in response to a growing trend of security issues we’ve been discovering in commercial services, most often WordPress hosting providers.
Earlier this year we started to see multiple similar cases from three service providers with no immediate explanation. We love a good mystery, and solving this one led to the discovery and correction of a critical vulnerability affecting many of the companies’ customers. The companies were:
- Hostway (and its other brands including Gate, Netnation, Domainpeople, and rebranded services for Qwest/Centurylink)
- Momentous Corp. (includes Rebel and Internic.ca)
- Paragon Group (includes Vidahost, TSOHost, Webfaction and others)
A Note on Disclosure And Responsible Vendors
It’s important to note that vulnerabilities are a fact of life in any service, system or software. Finding, confidentially disclosing and fixing vulnerabilities is how industry works with the information security community to improve the products and services we use and keep the public safe. The process that we use is well established and is widely used by organizations that include Google’s “Project Zero” and Cisco’s “Talos” security group.
When vulnerabilities are found and vendors are responsive, you benefit as a customer of those vendors and can know that your vendor reacts quickly to fix security problems and will likely do so long term, keeping you and your data safe.
A disclosure like this is not an opportunity for “vendor shaming” or a witch hunt. All developers who write enough code write vulnerabilities at some point in their career. It is in fact a moment to celebrate responsive vendors and a well handled incident that left customers and the online community safer.
At Wordfence, we are excited when a vendor works closely with us to fix a vulnerability and responsive vendors garner the greatest respect from our engineering team.
Attacks and investigation
As time went by, we had groups of separate customers suffering identical attacks. At first they were locked out of their administrator account and a user from an unknown IP address had logged into it. Then they saw new administrator accounts appearing from nowhere. Later the attackers started to add malware, spam, or defacement messages to the blog posts or site titles.
In each case we had fairly complete forensic information available to us, but it didn’t lead to any answers about how it was happening. We thoroughly analyzed all of the site files, and didn’t find any malware. We set up detailed logging to show us exactly when the exploits were taking place, and then checked site access logs at those times, and still found nothing. Even the less-used avenues of attack like cron jobs and FTP accounts were clean.
Taken together, all signs pointed to the sites’ databases being compromised. If the attackers have direct access to the database, they can change the existing administrator account’s password, add new accounts, or tamper with site content. It would leave no signs in any of the site’s logs, and would not require them to save malware in any files.
But how were they getting access to the databases? On most hosting platforms, you can only reach the database server through a hosting control panel. Web applications like WordPress can reach it as well. But either way, you still need a site’s particular database username and password in order to make changes to that site. In virtually all WordPress sites, that username and password can only be found in one place: the wp-config.php file.
File permissions
With a shared hosting service, companies use one server to host dozens or even hundreds of websites. Each customer gets a user account on the server, and their site’s files belong to that account. Sometimes groups of users are created, and files may belong to a group as well. In short, every file and folder has both a user and a group which are considered the owners.
Server-level file permissions control who can do what to files and folders. In addition to assigned owners, every file and folder has three permissions settings:
– User (what can the owner do?)
– Group (what can the group do?)
– World (what can any user on the server do?)
It’s possible to make dangerous mistakes with permissions. For example, if you set your website directory to be world-writeable, then any other user on the server can create a file in it. Or, if your website directory and wp-config.php file are both world-readable, then any other user can read the contents of wp-config.php (which include your site’s database username and password). On most servers, PHP scripts run as the customer’s user account, so if a script belonging to user ‘bob’ tries to access a file, it will only succeed if that file’s permissions allow bob (or any of bob’s groups) to do so.
An important note: imagine a folder named home. If your account has “read” permission on it, then you can list the contents of it. If you don’t have “read” permission but you do have “traverse” permission, then you can work with any file or subfolder within home as long as you know their names and have appropriate permissions on them. For example, the server will not tell you that home contains a folder named public_html. But if you already know that public_html exists and you have access to it, then you can still work with it as long as you have “traverse” permission on home.
Network file systems
Managing shared hosting servers can be tricky. Years ago, the hosting company would set up a server with let’s say 500 GB of available disk space. They might choose to limit each customer to 1 GB of usage, and then they could only have 500 customers on that server. But many customers would use only a fraction of the space, resulting in wasted resources. If customers were simply allowed more disk space, the server’s disk might fill up.
Hosting companies have found several ways around this problem. One of them is to use a network file system (NFS) to house customers’ website files. In this setup, an internal group of servers works together to provide disk space, which is shared through the private network with the public-facing servers. So the server that runs your website thinks that it has a massive disk plugged into it, when really it’s just using the shared network drive.
Vulnerable conditions
This is where we found our customers with the mysterious hacks – their websites would run on a server with a network file system. Each of the three hosting platforms used NFS, and had a variation on the same problem. Three conditions existed which combined to make the platforms vulnerable:
- Customer files were stored on a shared file system (in this case, NFS).
- All directories in the shared file system were world-traversable, and customer files were world-readable by default.
- The full path to a customer’s website directory was public or could be guessed.
The vulnerable hosting platforms all stored their customers’ files on a shared file system. The topmost directories were owned by the server root account and not set to be world-readable, so regular user accounts were not able to list their contents. However, for various reasons, all directories were world-traversable.
When hackers compromise a website, it’s very common for them to try to explore the server and see if they can locate other websites where they can create files, or at least read the contents of sensitive files like wp-config.php. The vulnerable servers did not allow for listing of the shared file systems, but their customer directories were set up so that the names of customer directories could be guessed. At some point, the hackers realized this, so they were able to use one hacked site to read sensitive files out of many other sites.
Hostway
On Hostway, the site files were located in a directory structure like this:
/home/14/23/1012314/web
The /home folder contained dozens of directories, and each of those contained multiple directories like 1012314, all following the numerical naming convention. The folder 1012314 and everything in it belong to the user; the directories above it belong to the server root account.
If a hacker took over the website in 1012314, and tried to explore the server by listing /home, they would get a “permission denied” error. They would also be denied if they tried to list /home/14 or /home/14/23. However, they must have realized that all the customer directory names followed a certain pattern:
/home/XX/YY/ZZZYYXX/web
XX and YY are two-digit numbers, and ZZZ is a three-digit number. So the hacker, having compromised just one site, could run a script from it to try all possible numbers and try to list files in each path. For example, it might start with /home/00/00/0000000, then /home/00/00/0010000, etc. Any incorrect guesses would result in a “no such file or directory” error from the server. But when the script correctly guessed an existing directory belonging to some other customer, the server would duly list its contents – because all customer directories were world-readable by default, and all directories above them were world-traversable.
So the hacker could find all the directories in the shared file system, each of which housed at least one website. If the website happened to be WordPress, he could read wp-config.php and get its database username and password. And since he already has control of a legitimate website, he can use it to reach the database server.
Using this method, the hacker could tamper with the databases of virtually any WordPress site running on the same shared hosting platform. This would all happen through the original hacked site, and nothing would ever appear in the logs of the other sites.
Momentous
The Momentous server platform (Rebel et al.) had a similar problem. The names of directories in the network share were random numbers, like /nfsvol-c/54/54321, and they weren’t world-readable but were world-traversable.
But when a website was created, both the site’s directory and the user account associated with it were named after the site. For example, if you set up example.com, then you might end up in the directory /nfsvol-c/12/12345/example.com/public.
A hacker trying to list the contents of /nsfvol-c/12/12345 would be denied. But he would be able to list the contents of /nsfvol-c/12 and see that it contained a directory named 12345 belong to the user account example.com. So he could correctly guess that there was a directory located at /nsfvol-c/12/12345/example.com/public and read the files within it.
Paragon
Likewise, the Paragon platform (Vidahost, TSOHost, et al.) had site directories sorted alphabetically. If you set up example.com on their platform, its files would be in the directory /var/sites/e/example.com/public_html.
Finding other sites in this case was trickier, but still possible. An attacker could take over a site on a Paragon server, and use DNS tools to find other WordPress sites running on the same server IP address. For example, if they found that otherexample.com was running on that server, all they had to do was look in /var/sites/o/otherexample.com/public_html for the files.
Remediation
One of the worst things about this vulnerability was that there was little that a site owner could do to prevent exploitation. In theory, removing world-read permissions from wp-config.php would protect a site, but there was no way to guarantee that some script or process wouldn’t reset it to the default vulnerable permissions. Moreover, only a small minority of site owners would know the adjustment was necessary. The only reliable defense against the exploit was for the hosting companies to fix the vulnerability.
We first contacted Hostway about this in September. At the time we did not have a formal way of dealing with service vulnerabilities. In December we updated our vulnerability disclosure policy to include them. On December 15 we formally contacted Hostway under the new policy, giving them a 14 day disclosure deadline, as the vulnerability was being actively exploited. They deployed a fix on December 21.
We also notified Paragon on December 15. They replied that they were developing a patch already, and requested more time before we published details of the exploit. They didn’t notify us about a deployment, but we’ve noticed that our proof-of-concept code no longer works on their servers, so we assume the patch was deployed some time this month.
Momentous received the disclosure on January 2, and deployed a patch on January 15.
What you need to do
If you use shared hosting on any of the companies we mentioned, use Wordfence to check your site for issues. We’re still getting a few site cleaning cases each week where the customer’s site was exploited before the fix was deployed, and the malicious code has remained on their site since then. The most popular malware we saw was obfuscated Javascript code inserted into every post, like this:
Sometimes cryptomining code was loaded into the page, other times it simply redirected the user to a scareware or spam site as soon as they clicked on anything.
We also saw a lot of sites with cloaked pharma spam links scattered throughout the posts. Their format varied, making them tricky to remove.
If your site has these issues, we recommend using our site cleaning service to fix them.
Conclusion
We are pleased with the positive impact adding service vulnerabilities to our Vulnerability Disclosure Policy is already having. The hosting companies we have worked with have been generally responsive, deploying fixes to issues that were causing many WordPress sites to be hacked.
With the popularity of WordPress today, the security of the WordPress community at large is critically important. We are pleased to see that our new approach is working to support that need and bringing about an improved overall security posture for the community.
Our Security Services Team continues to analyze hundreds of hacked websites each month, so we expect to find more of these on an ongoing basis. We will continue to provide updates here on the blog.
Note: All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.
The post Service Vulnerabilities: 3 Hosting Companies Fix NFS Permissions Problem appeared first on Wordfence.
