Wordfence Launches Short-Circuit Scan Signatures – Up to 6X Performance Increase

In October 2016, the Wordfence team started chatting about a way to radically boost the speed of scans once we grow beyond a certain number of scan signatures. As a reminder, a scan signature is a pattern that recognizes a certain kind of malware.

Today Wordfence has 4,523 signatures available for the free community, and we have an additional 226 new signatures that are only available to Wordfence Premium users. These become free once they are 30 days old.

New Malware Constantly Emerging

Our team continuously adds from 30 to over 100 new scan signatures each week. The site cleaning team constantly discovers new kinds of malware as they clean hacked websites, and each malware sample is turned into a scan signature and released to Wordfence to help it detect that malware.

Wordfence is currently at a total of 4,749 scan signatures for our Premium customers (4,523 free + 226 Premium), and within one year, this will grow to somewhere between 6,000 to 10,000 signatures at the current rate we are discovering new malware.

The constant increase in the amount of malware targeting WordPress is clear, and Wordfence needs to continually grow our scan signatures to keep pace.

Radical Innovation to Address Growth in WordPress Malware

In October last year, Matt Rusnak, who heads up QA for Wordfence, and Ryan Britton, who is our senior core developer for Wordfence, started chatting about a new algorithm to radically speed up the Wordfence scan and make it able to handle a much larger number of signatures.

Matt suggested identifying common patterns across scan signatures, grouping those signatures together and then checking if a file contains the common pattern first before scanning with the signatures in each group.

In theory, if we had 10,000 signatures, and if we are able to identify groups of 100 scan signatures and create a pre-check for each one, we would need to match only 100 scan signatures for every item we scanned instead of 10,000. Ryan dubbed this “short-circuiting.”

Earlier this year, we started work on the project. We created the services to support short-circuiting, then tested and launched them on our back-end servers about a month ago. Support was released for short-circuiting within the Wordfence plugin in the past few weeks. And we have been working to create grouped scan signatures with common “short-circuit” patterns.

Matt Barry, our lead developer who created the Wordfence firewall, worked with Ryan and Matt Rusnak to make this project happen, and they received help from other members of the engineering team.

A 2X to 6X Speedup With Short-Circuit Scanning

When we released Wordfence 6.3.17 late last week, you may have noticed an entry in the changelog which said, “Improvement: Prepared code for upcoming scan improvement which will greatly increase scan performance by optimizing malware signatures.

On Monday this week, we enabled short-circuit scanning on our servers. The speed improvement in Wordfence scans was breathtaking, to say the least.

We are seeing a 2-to-6-times performance increase across our test sites and customer sites. This is an incredible improvement.

On one major hosting provider, scans on one of our large test sites went from an average scan time of 8 minutes per scan to 1 minute and 20 seconds for a Wordfence scan to complete.

Continuous Engineering Innovation in WordPress Security

This is not the first time we have radically improved scan speed. Last year in September we increased scan speed by refactoring the way we perform many operations in the scan.

In July of this year, we further improved scan performance for hosting providers by monitoring scan distribution across hosting provider VPS instances and introducing a smoothing algorithm.

Short-circuiting scan signatures is a powerful new technique the team has created to provide a radical performance improvement on an already fast scan.

While our many of our competitors don’t even provide a firewall and malware scan in their security products, the Wordfence engineering team is at the forefront of engineering innovation, ensuring that you benefit from a powerful firewall and malware scan combination with lighting-fast performance.

Congratulations and thank you to Matt Rusnak, Ryan Britton, Matt Barry and Åsa Rosenberg, who all contributed to bringing short-circuit scanning to our customers.

The post Wordfence Launches Short-Circuit Scan Signatures – Up to 6X Performance Increase appeared first on Wordfence.


Wordfence Scan Gets Faster and Smarter

Wordfence is highly effective at securing your website in part because it is tightly integrated with the WordPress API. We know your visitor identity information, so we can make smart decisions about who gets access and who gets blocked. It’s very different from the way generic firewalls work.

This allows us to create powerful firewall rules and algorithms that have more data, and can therefore improve detection rates while driving false positive rates down to zero.

Over the past few years we have worked closely with our customers to improve Wordfence performance on their websites. Several recent Wordfence releases have shown spectacular gains in performance. Continuous improvement is one of our core goals with Wordfence, and so we went even further.

We have been reaching out to hosting providers to better understand their needs over the last few months. They are, after all, the platform our customers use to serve their site visitors. Our engineering team has worked with some of the largest hosting companies in the world to create radical improvements in the way Wordfence uses resources.

Today we are announcing Wordfence 6.3.14, which is the fastest and smartest Wordfence yet. I’m going to describe a few of the improvements we have made.

Smart Scan Time Distribution Across VPS Instances

Wordfence now centrally monitors scans that are running on the same server. If we see too many scans running on a single physical or virtual server, we’ll temporary defer any new scans. Once the number of concurrent scans decreases, we start the deferred scans.

This will typically only delay your scans from starting by 30 minutes or less, but it has huge benefits for you and your hosting provider. If you graph all the Wordfence scans happening on a single server, you will now see a smooth constant graph of scans instead of seeing spikes that could have harmed overall server performance.

This “smoothing algorithm” helps hosting providers better predict and manage server performance, and it helps our customers by ensuring their sites are always running on a high-performing server.

New Lightweight Scan Introduced

Parts of our scans use almost no server resources. We decided to break these items out from the main Wordfence scan into a separate scan that can be run more frequently.

The lightweight scan checks for:

  • WordPress Core updates
  • Outdated themes
  • Outdated plugins
  • Themes with known vulnerabilities
  • Plugins with known vulnerabilities

The new lightweight scan runs every 24 hours on all Wordfence sites, both Premium and free.

New Scan Schedule for Free Customers

If you are using the free version of Wordfence, we’ve changed the frequency with which your full malware scan will run. Prior to this week’s release, Wordfence ran a full scan every 24 hours for free customers. We’ve changed it to run once every 72 hours.

We will continue to schedule when scans run for free users, and the scheduling now varies based on the number of scans occurring on the shared server that you are on.

Our free users will still receive the new lightweight scan every 24 hours so that they receive time critical alerts about themes, plugins and WordPress core as soon as possible.

Wordfence Premium customers continue to have the ability to schedule unlimited scans to run whenever they want each week. In other words, if you’re a Wordfence Premium customer, your existing scan schedule remains completely unchanged. In general, we recommend a maximum frequency of once every 24 hours, but of course that is up to you.

Wordfence Manual Scans Are Unchanged

For both our free and Premium customers, we have not made changes to your manual scan capabilities. You can still run a manual scan on your site as often as you would like.

The manual Wordfence scan includes all the checks it always has. In other words, it includes the checks that your full Wordfence scan has always done, along with the checks from the new lightweight scan we introduced this week.

Connecting With You

Over the years, we have found that working closely with our community and with hosting providers has yielded huge dividends when it comes to better understanding the needs and challenges that you face and how to better secure our customers. The latest Wordfence release is another great example of the results that come from that ongoing collaboration.

We would like to thank the hosting providers who worked with us to make this release a reality and the users who have provided feedback.

As always, I welcome your feedback and comments below and will be around to reply to you.

The post Wordfence Scan Gets Faster and Smarter appeared first on Wordfence.


Wordfence Integrates Malware Scan Into Firewall

If you’ve been using the Wordfence Firewall for a while, you may have noticed that our firewall ruleset has been growing steadily over the past few months. This happens as we turn new threat intelligence into firewall rules and release them into production to protect your website.

The Wordfence Firewall protects you against attackers hacking into your website using known weaknesses like the vulnerabilities that have been exploited in Timthumb, Mailpoet, Gravity Forms, Slider Revolution and many others.

We also protect against many zero day vulnerabilities that aren’t yet known to the public but are known to us exclusively. These rules protecting against zero day vulnerabilities are unique to Wordfence.

We also protect against vulnerabilities that haven’t yet been discovered by using a smart ruleset that recognizes malicious activity and blocks it.

We knew we could do better

Many firewalls only protect against common attacks that exploit vulnerabilities. One of the things we see when a site is targeted is that an attacker has a goal in mind; They want to upload malicious code so that they can execute that code on your website.

In the security industry we use the phrase “Defense in Depth”. This describes a multi-layered approach to security, so that if one layer of security doesn’t stop an attacker, another will.

We realized if we took a multi-layered approach with our firewall, we would do an even better job of protecting our customers and have a very high probability of stopping attacks.

Announcing a new break-through feature

MalwareWith this in mind we have integrated our scan engine into the Wordfence Firewall. This layered approach means that even if a rule that recognizes an attacker exploiting a vulnerability doesn’t block the attack, our scan rules will block the attack when the attacker tries to upload malicious content.

Last week we quietly rolled Wordfence 6.1.17 into production. This update integrates Wordfence Scan and the Wordfence Firewall. With this update, as traffic passes through the Wordfence Firewall before it hits your website, it is inspected using our full scan capability and if we find any malicious code in a request, it is blocked.

This has the effect of adding a powerful malware and virus scanner to your firewall to complement the already comprehensive ruleset that Wordfence uses to protect you. This new layer of protection is extremely fast and comes with zero performance penalty for your website.

This is a very exciting change because through our forensic research, our scan capability has massively increased over the past few months. This scan capability has now been added to the firewall.

Right now our free Wordfence community users are protected using 402 unique scan signatures, many of which detect multiple malware types. Our Premium Wordfence users are protected using 137 additional malware signatures. As always, these signatures will become available to free customers within 30 days of release.

We also have 163 beta signatures that we are currently testing and will be bringing online for our Premium customers over the next few days and weeks.

This new firewall detection capability has just been added to the Wordfence Firewall in a single release, which has the effect of adding hundreds of new firewall rules at once.

Bringing this new capability online for our customers is a big deal and our team worked hard to make this release happen. I’d like to extend my special thanks to our Dev and QA team who made sure that adding this new detection did not result in any false positives on your website and made sure that, as we rolled this out, the over 1.5 million websites we protect would continue to run fast and flawlessly.

Since our release last Thursday over half a million websites have upgraded to Wordfence 6.1.17 without a hitch. If you haven’t done so already, upgrade now so that you too can benefit from this new capability and protection for your WordPress website.

The post Wordfence Integrates Malware Scan Into Firewall appeared first on Wordfence.


Malware: 139,000 WordPress Sites Saved in 30 Days

Wordfence provides two core security capabilities to the websites we protect. Our firewall prevents your WordPress site from getting hacked. Our malware scanner detects if you have been hacked or if malware has somehow been installed on your system.

Today I’d like to share a few malware statistics, how we put new malware detection into production and a story about one prolific piece of malware we’re seeing.

139,766 Unique WordPress Sites Saved in 30 Days!

During the past 30 days, 139,766 unique websites have detected malware of some kind using Wordfence scan. That’s an incredible number. I like to think of it in terms of baseball stadiums. Our local Safeco Field, home of the Seattle Mariners, can fit just under 50,000 people. If each website belongs to a person, we could fill the stadium almost three times with website owners we’ve helped protect in the last 30 days alone! That’s incredible. 

At Wordfence we provide a free community version of our plugin for WordPress and a paid version. We send out threat intelligence to the plugin via our Threat Defense Feed. This includes new scan ‘signatures’ for new malware that we have discovered during our forensic research. It also includes new firewall rules to protect you against new kinds of attacks.

Wordfence Premium receives the updates in real-time. Our community edition is delayed by 30 days. So new scan signatures only become available to the free users of Wordfence 30 days after they are released.

During the past 30 days the malware signatures that our Premium customers have early access to detected malware on 5.2% of our Premium customer websites.

How many free and premium scan signatures do we currently have?

Wordfence currently has 364 free scan signatures that each detect a different kind of malware. In some cases a signature will detect many different kinds of malware because it’s designed to detect common malware patterns.

When you do a scan with the free version of Wordfence, you are checking every file you scan against all 364 of these signatures to try and detect the most common malware we’re seeing. It’s worth noting that we actually retire some malware signatures because sometimes, a piece of malware just isn’t seen anymore or we improve our detection capability.

For our Premium customers we have an additional 118 signatures (sigs) currently in production. We continue to add to those sigs and the older sigs become available to our free customers 30 days after they are released.

Those 118 Wordfence Premium sigs were added at different times, so they become available to the free community at different times.

  • Thu Sep 08 2016: 38 new Premium sigs will be released to our free customers
  • Sun Sep 25 2016: 24 Premium sigs will be released
  • Wed Sep 28 2016: 15 Premium sigs will be released
  • Thu Sep 29 2016: 41 Premium sigs will be released

How does Wordfence develop, test and release scan signatures?

The way Wordfence adds new scan signatures is we develop and then test them internally. The testing involves running each sig against the top 5,000 WordPress themes and plugins along with multiple versions of WordPress core to ensure there are no false positives. We have developed internal tools to do this very quickly.

Then we add them to our ‘beta’ feed. If you are a Wordfence Premium customer you can enable the Beta Threat Defense Feed which includes our beta scan signatures by going to the ‘Diagnostics’ menu, scrolling to the bottom and checking the ‘beta’ checkbox. This is only available for Premium customers.

Once we’ve verified the new sig is not creating problems for our beta community, we flag the sig as ‘production’ and it enters production for our Wordfence Premium customers. Those customers receive the new sig automatically and don’t need to upgrade or take any action. 30 days later the sig enters the community version of Threat Defense Feed.

Wordfence currently has 112 scan signatures in ‘beta’ and they will enter production for our Wordfence Premium customers over the next few days and weeks. We tend to release scan sigs in batches, as you can see from the date schedule above.

The Premium Threat Defense Feed gets the newest scan signatures which means that those signatures have an unusually high detection rate. The detection rate is high because the new sigs we add are things we have recently seen in the wild.

What does “In the Wild” actually mean? Where do Wordfence scan signatures come from?

So what does it mean when we say we saw an infection ‘in the wild’ and we added it to the Threat Defense Feed as a scan signature?

At Wordfence we have built a team of some of the finest cyber forensic investigators in the industry. We have two major efforts that help us find new WordPress malware samples and threats.

The first is our site cleaning service. Wordfence offers site cleaning at a very reasonable price and one of the things we get from providing that service is on the ground intelligence of what is infecting real WordPress sites. Our site cleaning team has a very efficient process to turn the samples they find on infected sites into scan signatures in production. The process ensures we remove any duplicates and are able to prioritize samples where we’re seeing a widespread infection.

The second way we gather threat intelligence is by mining attack data. When an attack occurs on a WordPress site, that attack is reported to us. Some of those attacks include malware payloads. We are able to aggregate those malware samples into a database and prioritize them by the malware we are seeing most frequently. This has become a huge source of new malware signatures for us which has accelerated the pace at which we are able to add new signatures to our scan engine.

What is the most widespread new malware Wordfence is seeing?

To illustrate how powerful it is to have the newest malware signatures in your scan feed, lets take a look at a specific example. Below we have a scan signature that was added on the 27th of July, last month.

Screen Shot 2016-08-30 at 10.44.22 PM

The image above shows selected database fields for this signature’s record.

Internally at Wordfence this signature has the codename Backdoor: PHP/eawtluil. Not very attractive sounding. Our more technical audience will recognize the scan ‘rule’ as a regular expression or ‘regex’.

During the past 30 days this signature has detected over 1 million infected files across all Wordfence customers. This infection has a habit of infecting a lot of files when it hits a WordPress site. That is more malware infected files than the next 10 signatures combined.

This signature was added on the 27th of July last month. That means it was in our Premium feed for 30 days and only recently became available to the free community users of Wordfence on the 27th of August, 4 days ago. Now that it’s running in our free feed, it is detecting a huge number of malware infected files across the WordPress ecosystem. And that’s just one signature!

So what does this sig actually detect? This is what the malware actually looks like: (click to open a larger image in a new tab)

As you can see the malware is heavily obfuscated (encoded to hide the meaning of the code).

As attackers realize we are detecting their malware, they will use new obfuscation, or code hiding techniques. This signature will become less effective and new signatures we add will have a higher detection rate. That is why we refer to our scan signatures as a ‘feed’ or the Threat Defense Feed. Our forensic work is ongoing and we are continually adding to the feed and optimizing what is in it.

In closing…

I hope this has given you some insight into a small part of our operations at Wordfence – specifically our forensic efforts into gathering threat intelligence and how we turn part of that data into the scan signatures that provide you with early detection of a hack or malware on your WordPress website.

If you would like to install Wordfence free on your WordPress website, simply go to the Plugins menu and use the search box on the top right to search for Wordfence. To upgrade to Wordfence Premium and get your scan and firewall rules in real-time, 30 days earlier than everyone else, you can visit our home page to learn more about the Threat Defense Feed and how to get Wordfence Premium.

As always I welcome your questions or comments. Thanks for using Wordfence and stay safe.

The post Malware: 139,000 WordPress Sites Saved in 30 Days appeared first on Wordfence.