Online Services

Blog

  • 51 Tools for Security Analysts

    Yesterday at Wordfence we had an “all welcome” technology sharing meeting with the entire company – or at least everyone that was available at the time. The meeting became so popular with our team that we had to upgrade the license we use for our real-time collaboration service to accommodate everyone. It is the largest team meeting we have had to date.

    The goal of the call was to have an informal chat about some of the external security and investigative tools that our team finds useful. The meeting included most of our security services team, senior dev staff, security analysts including all senior analysts, team members from customer service and even execs.

    I think we all learned something new and Brad, one of our senior analysts, was kind enough to compile the list of tools we chatted about in a shared document.

    I know that many security analysts, WordPress developers and readers who are interested in security visit this blog. So I thought I’d share the list of tools the Wordfence team came up with so that if you work in security or just want to increase your cyber security literacy, you can benefit from this list too.

    I should emphasize that this is by no means an exhaustive list of security tools. This list merely includes a few of the more interesting cyber security tools that came up in a 40 minute conversation with our team yesterday. We also have a suite of internal tools that are not included. If you have a personal favorite, you are most welcome to share it in the comments.

    Reading this list may be worrying or intimidating for readers who don’t work in the security industry. You should know that all tools on this list are free and publicly accessible. They are also well known within the professional security community and among malicious actors. This list of tools, software and utilities should empower anyone interested in protecting themselves and their online assets by making you aware of the capabilities that exist for analysts and malicious actors. By better understanding the tools that your adversary uses, you can better protect yourself.

    Information gathering and analysis

    Google dorks – Using advanced operators in the Google search engine to locate specific strings of text within search results.

    Using Google for penetration or malicious activity may seem silly or obvious, but Google is incredibly powerful and very popular among analysts and malicious actors alike. “Google dorks”, or google-hacks as they’re also known, are a search query that attackers use on Google to identify targets. If you visit a site like exploit-db.com or any other database of exploits, you’ll find that many of them include Google dorks to help find targets to attack with the exploit.

    Maltego – An interactive data mining tool that renders directed graphs for link analysis.

    Maltego is one of our favorites. It is an investigator’s tool that lets you graphically organize your thoughts and your investigation by creating objects (people, places, devices, events) and link them. It also gives you the ability to run ‘transforms’ on objects. For example, you can run transforms on an IP address to list its malicious activity using external sources of threat intelligence. You can download a free version from Paterva which has some limitations.

    You can see an example of the work we do with Maltego below.

    FOCA – A tool used to find metadata and hidden information in the documents its scans.

    When you create and publish MS Office, PDF, EPS and PS documents online, you may not realize how much information you are leaking to the general public. FOCA is a security analyst’s tool that can be used to extract ‘leaked’ data from documents that have been made public. Using FOCA, an analyst can find things like an organization’s network structure, IP addresses, internal server names, printers, shared folders, access control lists and more. You can watch this video filmed at DefCon 17 for a demo of how FOCA can be used by researchers or malicious actors to perform recon on a target organization or individual.

    http://checkusernames.com/ – Check the use of a brand or username on 160 social networks.

    If you simply want to find a unique username, checkusernames.com is a useful tool. If you are in the security field, it can be a powerful way to attribute an attack to a specific individual. Malware authors occasionally include usernames or ‘hacker names’ in their malware. Using this tool you can search 160 online services to see if they have used the same username somewhere else.

    https://haveibeenpwned.com/ – Check if an account has been compromised in a data breach.

    The term ‘pwned’ is slang for ‘owned’ which in the security industry means “to have your data or system compromised”. So ‘haveibeenpwned.com’ is slang for “Have I been owned dot com”. This is a well known and respected site run by Troy Hunt which finds and aggregates data from data breaches. You can use the service to find out if an account has been compromised by looking up your email or username.

    https://www.beenverified.com/ – Search people & public records.

    This is a general “people search” that is useful to find additional meta-data when researching a target during penetration testing or when researching an attacker.

    Shodan – Search engine for Internet-connected devices.

    This is a very popular service among security researchers. Shodan continually crawls and indexes devices on the internet. We recently used Shodan as part of our research into routers at several ISPs around the world that have been hacked and are now attacking WordPress. You can find a few example searches demonstrating Shodans use on their ‘explore’ page.

    Censys – A search engine that allows computer scientists to ask questions about the devices and networks that compose the internet.

    Censys is similar to Shodan in that it indexes devices and websites connected to the internet. The data is also searchable and differs from Shodan in some ways. Shodan is focused on ports and the services running on those ports. Censys is great at indexing web site SSL certificates among other things. Censys is maintained by a team of computer scientists at the University of Michigan and University of Illinois Urbana-Champaign.

    Gephi – Visualization and exploration software for all kinds of graphs and networks.

    We mentioned Maltego earlier in this post. It uses a ‘graph’ structure which is a diagram of linked objects to represent relationships. Gephi is a tool to analyze graph data at massive scale. We used Gephi to generate the graphical representations of attack data that we published in our February Attack report, seen below.

    Fierce – A DNS reconnaissance tool for finding target IPs associated with a domain.

    Fierce is a tool used to find IP addresses that are potential attack targets associated with a specific domain. It is used by penetration testers when evaluating insecure points on a network.

    BuiltWith – Find out what websites are built with.

    BuiltWith has a search engine-like interface and lets you search for a specific site to find out what tools were used to build it. BuiltWith also aggregates that data so that you can find out what the most popular technologies are on the web or how a specific technology is trending relative to another.

    Wappalyzer – A cross-platform utility that uncovers the technologies used on websites.

    Wappalyzer is another tool that helps you discover what technologies a specific site is using. Like BuiltWith, they also aggregate data to help you determine how technologies are trending. This is their view of the popularity of blog technologies, with WordPress clearly the market leader.

    Wappalyzer Chrome extension

    Wappalyzer also has a browser extension for Chrome that lets you immediately see the technologies a specific site is using. There is also a Python driver available on github called python-Wappalyzer.

    https://aw-snap.info/ – Tools for owners of hacked websites to help find malware and recover their site.

    aw-snap.info includes a suite of tools that may be helpful for site owners who have decided to try to clean their own hacked site. It can help you fetch pages as Google, which sometimes reveals malware. It can also decode base64 obfuscated malware and help find obfuscation in your files that may hide malware.

    http://themecheck.org/ – A quick service that lets you verify WordPress themes for security and code quality.

    ThemeCheck may help you verify your theme integrity by uploading it. It can also help find malware embedded in themes.

    theHarvester – Gather emails, subdomains, hosts, employee names, open ports and banners from different public sources like search engines, PGP key servers and SHODAN.

    This is a tool that performs a variety of reconnaissance operations on an organization and may be useful in the early stages of a penetration test to determine an organization’s overall online footprint.

    Cymon.io – Tracker of malware, phishing, botnets, spam, and more.

    Cymon can help you research a potentially malicious IP or malware hash. Here’s an example of the results for one of our top IPs in the March Wordfence monthly attack report.

    Mnemonic – A passive DNS database.

    Mnemonic is a useful tool that can find which websites are hosted at a specific IP or which IPs host a website.

    Vulnerability scanning and penetration testing

    WPScan – A black box WordPress vulnerability scanner.

    WPScan is a command line tool that is used to remotely scan WordPress sites for vulnerabilities.

    Sqlmap – An open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over database servers.

    SQLMap is widely used among penetration testers and is highly effective at finding and exploiting SQL injection vulnerabilities in target sites.

    BeEF – A penetration testing tool that focuses on the web browser.

    BeEF is a powerful tool that lets penetration testers exploit and control a web browser. Using BeEF you can set up a malicious website, exploit a visiting browser and gain access to the workstation running the browser. You can watch this 2014 KiwiCon video for a demo.

    Firefox Hackbar – A simple security audit / penetration test tool.

    Hackbar is a plugin for Firefox that may help application developers perform security audits on their own web applications. It includes a variety of tools to assist with this task.

    Burp Suite – Software for web security testing.

    Burp Suite is a very well known and powerful framework used to perform security audits and analysis on web applications. It includes a proxy that can intercept traffic and allow you to modify it on the fly. It includes a huge variety of exploit and penetration testing tools.

    OpenVAS – An open source vulnerability scanner and manager.

    You have probably heard of the vulnerability scanning tool Nessus. Back in 2005 Tenable Network Security changed the Nessus open source license to a closed source one. The developers forked the project at that time and created OpenVAS.

    I’ve found that OpenVAS can be quite effective, but it is a bit more challenging to set up than Nessus. OpenVAS does have the advantage of being completely free and open source. The project is well known throughout the online security community.

    Fiddler – A free web debugging proxy.

    Fiddler is a proxy server that lets you intercept requests to a website, view them in different ways, modify the requests and can help debug websites and perform security audits.

    Joomscan – Detect Joomla CMS vulnerabilities and analyze them.

    Joomscan is the Joomla CMS’s equivalent of wpscan.

    Kum0nga – A simple Joomla scan.

    This is another joomla vulnerability scanner.

    Arachni – A feature-full, modular, high-performance Ruby framework aimed towards helping penetration testers and administrators evaluate the security of modern web applications.

    Arachni is a framework to perform detailed vulnerability scanning on web applications.

    Forensics and log analysis

    Lnav – An advanced log file viewer.

    Lnav is short for log file navigator. It automatically detects your log file formats, provides syntax highlighting and a host of other features to view and analyze log files. It can be invaluable when analyzing a compromised website.

    Mandiant Highlighter – A free log file analysis tool.

    Mandiant (now owned by Fireeye) produced this useful product that can help analyze log files. It includes the ability to graphically view a histogram of log files and several other powerful log file analysis features.

    Wp-file-analyser – Find modified, missing and extra files in a WordPress directory.

    This utility can download the original versions of WordPress core and plugin files and can help you compare them against their originals. Wordfence already does this from within WordPress, but this provides a command line tool to perform a similar action.

    Auditd – Access monitoring and accounting for Linux.

    Access monitoring and logging/accounting is very helpful when monitoring a system to see if it is being attacked or performing an investigation after the attack. Auditd can help you improve logging and provide an audit trail on Linux.

    Araxis Merge – Advanced 2 and 3-way file comparison (diff), merging and folder synchronization.

    When responding to a hack, the ability to compare files to originals to determine what has changed is important. Araxis Merge is a powerful tool that can assist with this.

    WinMerge – An Open Source differencing and merging tool for Windows.

    Much like Araxis Merge, WinMerge can help you compare files to examine changes when responding to an incident.

    DiffNow – Compare files online.

    DiffNow is a web based file ‘diff’ tool that can also assist when comparing file differences during incident response.

    Code and malware analysis

    CyberChef – the Cyber Swiss Army Knife

    CyberChef is a tool that is developed by GCHQ, the British intelligence agency. It can help de-obfuscate malware and other code.

    UnPHP – A free service for analyzing obfuscated and malicious PHP code.

    Obfuscating (hiding/garbling) PHP is a favorite tool of hackers, UnPHP can help analyze obfuscated code.

    UnPacker – JavaScript unpacker.

    Jsunpack – A generic JavaScript unpacker.

    ‘Packing’ javascript is a favorite technique of hackers who are dropping malicious javascript on websites. It makes their code more compact and harder to read. Jsunpack can help de-obfuscate JS code to make it more readable so that you can understand how it operates.

    JSBeautifier – An online JavaScript beautifier.

    Much like Jsunpack, JSBeautifier helps improve the readability of packed javascript code.

    https://www.base64decode.org/ – Base64 Decode and Encode

    Base64 encoding is a way to encode anything into an encoded string of (what appears to be) random characters. Anyone who is repairing hacked sites or responding to incidents uses base64 decoding several times a day to expose malicious code that has been base64 encoded. This tool can help decode base64 encoding.

    https://www.urldecoder.org/ – URL Decode and Encode

    URL encoding is also a popular way for hackers to hide their code, through encoding it using this form of encoding. urldecoder.org can help you decode malicious code that has been hidden using urlencoding.

    http://lombokcyber.com/en/detools/decode-sourcecop – Decode SourceCop v3.x

    This is a tool that decodes a specific type of PHP encoding that may prove useful during a hacked site investigation.

    Other tools

    regex101 – Develop and test regular expressions.

    Regex, or regular expressions, are pattern matching routines to find complex patterns in files and code. We use regex extensively at Wordfence to help fix hacked sites and in our software and products.

    regexpal – Another site to develop and test regular expressions.

    Both regex101 and regexpal provide online development environments to help you create or analyze regular expressions.

    HashKiller – Online hash cracking service. Useful to reverse engineer hashes into passwords.

    In most systems, passwords are stored as hashes. Malware authors occasionally use hashing to store their own passwords. In our research we have needed to crack hashes that are used by malware authors in order to read their source code. HashKiller can help reverse a hash into a password if you need to crack a hash as part of your malware analysis.

    Noscript – Noscript  is a Firefox extension that allows Javascript, Java and Flash to only be executed by websites that you define and trust.

    When visiting malicious websites, Noscript can help disable malicious code on that site. Note that you should always visit a malicious site that you are analyzing using a virtual machine that has no important data on it. If the VM gets infected, you can simply destroy it without worrying about important data being leaked. Using Noscript in your browser within your virtual environment can be useful when analyzing the function of a hacked site.

    Other lists of tools

    • Awesome Forensics – A curated list of awesome free (mostly open source) forensic analysis tools and resources.
    • awesome-incident-response – A curated list of tools and resources for security incident response, aimed to help security analysts and DFIR teams.
    • OSINT Framework – OSINT is short for ‘open source intelligence’. This site provides a graphical directory of OSINT resources.

    Kali Linux

    Kali Linux is a linux distribution that is the favorite of penetration testers and security analysts world-wide. It is a linux distribution that comes packed with security analysis tools. If you want to learn about cyber security, Kali should be one of your starting points. If you simply would like to know about some of the more important tools that Kali provides, you can use the list below.

    Kali Linux Tools Listing – All the tools in Kali Linux, a Linux variant used by penetration testers and security analysts.

    Conclusion

    The tools on this page can help you respond to an incident, test the security of your own website and better understand how attackers think and what tools they have available to them. As always I welcome your feedback in the comments and you are most welcome to suggest your own favorite security or analysis tools.

    The post 51 Tools for Security Analysts appeared first on Wordfence.

  • Wordfence Site Cleaning Customer Reviews

    In June last year we officially launched the Wordfence site cleaning service. Our senior analysts Brad and Colette had worked hard to put the processes in place we needed to provide an excellent site cleaning service to our customers that was fast, effective and safe. Since then the site cleaning team and the level of service has evolved tremendously. Today I want to share some of that progress with you.

    When we started offering site cleaning in June of last year, the team was still relatively small. Today the site cleaning team at Wordfence has grown to 15 people who all work together in concert to ensure that you receive the very best in customer service and a fast response time. The team does a great job of ensuring your site is up and running and back in the search results as quickly as possible while preserving the integrity and security of your data.

    Managing growth while providing a ’boutique’ site cleaning service

    We may have grown to a team of 15 people, but I still consider us a ’boutique’ operation because we pride ourselves on providing each customer with the best service possible.

    We currently clean between 100 and 200 hacked sites per week. This is a very comfortable number for us because it is enough volume to give us excellent visibility into the kinds of hacks that the WordPress platform is experiencing on the ground at any moment in time, which assists with our research. With a team of 15 and that kind of volume, we are still able to give our customers individual attention and excellent customer service.

    One of the ways we have optimized our processes to ensure you get the best service is to create a “site cleaning coordinator” role within our organization. We recently brought Jonathan on board who is our full-time site cleaning coordinator. His job is to ensure that customer response times are fast and that customer satisfaction stays high.

    If a customer needs help understanding our process, what the team needs to do their job, or if they have any other issues that are blocking their site cleaning, Jonathan jumps in, gets that unblocked and helps get the cleaning moving forward.

    Priced at $149 and lower, we provide the best value in the industry

    One of the things I am most proud of is that our team has been working since launch to lower the price of our site cleaning service for our customers while actually increasing the value each customer receives. As we have grown, our internal efficiency has improved and we have been able to pass those cost savings on to our customers.

    This has allowed us to lower the price of cleanings from $179 when we launched to just $149 for a site cleaning today. This is an incredible value when you consider that you get a Wordfence Premium license free with your site cleaning, a $99 value. That means that you are only paying an extra $50 for the site cleaning service, which is by far the best value in the industry.

    We also offer awesome bulk discounts. Right now to clean 10 hacked sites it will cost you just $644. What you may not realize is that you get 10 Wordfence Premium API keys for free with those 10 site cleanings. If you had to buy those 10 API keys from us they would cost $429.10.  That means it costs you just an extra $214.90 to clean 10 sites – or just $21.49 per site. That is by far the best value in our industry.

    The reason we decided to go into the site cleaning business is because we wanted the ability to analyze recently hacked sites so that we could better understand how to protect our customers. I went into some detail in our original launch post, explaining why we launched our site cleaning business and how it helps our Wordfence customers.

    We are able to take the forensic data from each site cleaning and use it to improve our products. The synergy between our product development and the forensic data we get from the security analysts that clean hacked sites is another reason our costs and pricing have remained low while our service levels have stayed incredibly high.

    Let’s hear from a few of our customers

    Below I have shared some of the recent feedback the site cleaning team received from our customers. Thank you very much to each of our customers who has sent us their kind feedback. The team very much appreciates it and we are constantly sharing the positive feedback we receive on ‘Slack’, our internal chat system.


    I recently contacted Wordfence to take advantage of their website ‘cleaning’ service after a really annoying hack.

    Kathy at Wordfence was AMAZING! So patient and without going into details of what exactly she did a) – for security reasons and b) – because I didn’t understand it! I feel much happier with the way my sites have been cleaned. Hell they even load faster now too!

    As part of the clean I received a full 4 page guide of what I would need to do after the hack to keep myself safe in future.

    Kathy seriously went above and beyond what she needed to do and helped me out so much.

    In the nicest possible way, I hope I don’t have to contact Wordfence again, but if you need to, I can totally recommend them.

    — Wendy [Blog post]


    Helene is a PhD researcher at a global think tank that researches geopolitical risk. It was our pleasure to help their organization recover from a hacked website.


    I am so happy with the service you provide, the value and the personal assistance.

    — Julie


    Thanks for making it easy and fast!

    — Annie


    I am very satisfied with the quality of your work on my site and all the help you have given me. I am very happy with your services.
    Many thanks.

    — Emmanuel


    David is an author who writes for the Huffington Post, Daily Beast and other publications. It was our pleasure to work with him.


    THANK YOU SO MUCH for your professionalism, customer support and for taking care of my issues. I will spread the word. 🙂

    It helps to have Wordfence under my dashboard to stay on top of it too. You are a true security analyst angel! 🙂

    Thank you again Kathy! I greatly appreciate all of your help.

    — Tracy


    WOW. I commend you guys for your assistance.

    — Derrick


    Danke für die Erklärung –hervorragender Service! [Translation: “Thank you for the explanation -Excellent service!”]

    — Volker


    Thank you, Marco.

    I’ve informed my client about the link potentially being infected and have removed it from the article. I’ll keep an eye on the backup buddy files.

    Thanks again for your speedy help on this. Cheers!

    — Olga


    Shemeka is a keynote speaker, trainer and educational advocate. It was our pleasure to work with her.


    Thanks for all the help! It’s very much appreciated. 🙂

    — Jennifer


    Thanks for your help and follow up, you guys provide a great service.

    — Ross


    Hi Marco

    Thanks for checking in – I’ve had no more scary alerts from scans, hurrah!

    Fingers crossed that’s it!

    Thanks for all your help with this,

    — Amy


    Awesome, thank you Giles!

    Things all seem to be back to normal. I’ve read your report and will quickly work on all the recommendations.

    Very much appreciated. Thanks for getting the website back up and running so quickly.

    I have a number of other websites I manage……if anything ever happens again, it’s nice knowing I can trust Wordfence services.


    Rachel is a blogger who writes about books, lifestyle and parenting. It was our pleasure to work with her.


    Dear Paolo,

    Many thanks for your prompt assistance with this issue.

    Excellent service.

    Kind regards

    — Dan


     

    We would love to hear from you

    Whether you are selling a product or service online, writing about technology, involved in advocacy, blogging for fun or sharing important ideas, a website is just a means of communication and it should not get in the way of the conversation you are having with your audience. A hack definitely should not interfere with that important communication.

    Our site cleaning team loves working with our customers and there is a real sense of satisfaction when we are able to get a customer’s website back up and running and help secure them long term.

    If need our world-class security analysts to help you recover from a hacked website, we would love to hear from you. You can click here to find out more about the Wordfence site cleaning service.

    The post Wordfence Site Cleaning Customer Reviews appeared first on Wordfence.

  • Chrome and Firefox Phishing Attack Uses Domains Identical to Known Safe Sites

    This is a Wordfence public service security announcement for all users of Chrome and Firefox web browsers: 

    There is a phishing attack that is receiving much attention today in the security community.

    As a reminder: A phishing attack is when an attacker sends you an email that contains a link to a malicious website. You click on the link because it appears to be trusted. Merely visiting the website may infect your computer or you may be tricked into signing into the malicious site with credentials from a site you trust. The attacker then has access to your username, password and any other sensitive information they can trick you into providing.

    This variant of a phishing attack uses unicode to register domains that look identical to real domains. These fake domains can be used in phishing attacks to fool users into signing into a fake website, thereby handing over their login credentials to an attacker.

    This affects the current version of Chrome browser, which is version 57.0.2987 and the current version of Firefox, which is version 52.0.2. This does not affect Internet Explorer or Safari browsers.

    We created our own example to demonstrate how an attacker can register their own domain that looks identical to another company’s domain in the browser. We decided to imitate a healthcare site called ‘epic.com’ by registering our own fake site. You can visit our demo site here in Chrome or Firefox. For comparison you can click here to visit the real epic.com.

    Here is what the real epic.com looks like in Chrome:

    Here is our fake epic.com in Chrome:

    And the real epic.com in Firefox:

    And here is our fake epic.com in Firefox:

    As you can see both of these domains appear identical in the browser but they are completely different websites. One of them was registered by us, today. Our epic.com domain is actually the domain https://xn--e1awd7f.com/ but it appears in Chrome and Firefox as epic.com.

    The real epic.com is a healthcare website. Using our unicode domain, we could clone the real epic.com website, then start emailing people and try to get them to sign into our fake healthcare website which would hand over their login credentials to us. We may then have full access to their healthcare records or other sensitive data.

    We even managed to get an SSL certificate for our demonstration attack domain from LetsEncrypt. Getting the SSL certificate took us 5 minutes and it was free. By doing this we received the word ‘Secure’ next to our domain in Chrome and the little green lock symbol in Firefox.

    How is this possible?

    The xn-- prefix is what is known as an ‘ASCII compatible encoding’ prefix. It lets the browser know that the domain uses ‘punycode’ encoding to represent Unicode characters. In non-techie speak, this means that if you have a domain name with Chinese or other international characters, you can register a domain name with normal A-Z characters that can allow a browser to represent that domain as international characters in the location bar.

    What we have done above is used ‘e’ ‘p’ ‘i’ and ‘c’ unicode characters that look identical to the real characters but are different unicode characters. In the current version of Chrome, as long as all characters are unicode, it will show the domain in its internationalized form.

    How to fix this in Firefox:

    In your firefox location bar, type ‘about:config’ without quotes.

    Do a search for ‘punycode’ without quotes.

    You should see a parameter titled: network.IDN_show_punycode

    Change the value from false to true.

    Now if you try to visit our demonstration site you should see:

    Can I fix this if I use Chrome?

    Currently we are not aware of a manual fix in Chrome for this. Chrome have already released a fix in their ‘Canary’ release, which is their test release. This should be released to the general public within the next few days.

    Until then, if you are unsure if you are on a real site and are about to enter sensitive information, you can copy the URL in the location bar and paste it into Notepad or TextEdit on Mac. It should appear as the https://xn--….. version if it is a fake domain. Otherwise it will appear as the real domain in its unencoded form if it is the real thing.

    Spread the word

    The concept of an IDN homograph attack has been around since 2001 when Israeli researchers Evgeniy Gabrilovich and Alex Gontmakher first wrote about it.

    Web browsers have attempted various fixes but the current implementations in Chrome and Firefox are clearly not doing a good enough job. To Chrome’s credit, they are about to fix that. Thankfully there is a manual fix for Firefox.

    We would like to encourage you to spread the word. This new twist on phishing is getting a lot of attention today, Friday April 14th and is making the rounds currently in the security community. Xudong Zheng wrote about this earlier today and it is also being discussed on the netsec subreddit.

    We think here is a high possibility that this may be exploited in phishing attacks before the Chrome fix is released to the general public, which is why we are posting this public service announcement.

    The post Chrome and Firefox Phishing Attack Uses Domains Identical to Known Safe Sites appeared first on Wordfence.

  • IP Blacklist Update: The Launch and Evolution of The Wordfence IP Blacklist

    One of our passion projects at Wordfence has been to find a way to create and run an IP blacklist. We have known for a long time which IPs are attacking the sites we protect and that if we can block those IPs outright, it would be a powerful way to improve the security we provide to our customers.

    Blocking a bad IP completely is more effective and safer than just blocking its malicious requests, because you don’t allow it to gather information about the target website it is about to attack.

    We launched the Wordfence Premium IP Blacklist 1 month ago and it has been incredibly effective at improving the security of the websites that Wordfence protects.

    Today I want to share the story behind why we decided to create the Worfence Premium IP Blacklist, how we created and launched it, how it has evolved rapidly to contain over 4,000 dynamically updated IPs and how quickly it is growing.

    Creating and Launching the IP Blacklist

    When we conceived of the IP blacklist about a year ago the challenge we had was to have a way to block bad IPs accessing our customer sites and let the good guys through without actually distributing a list of bad IPs to customer sites. We wanted to avoid distributing the list so that we didn’t also provide other attackers with a list of possibly compromised targets to attack.

    The trouble with not distributing a list of bad IPs is that the obvious alternative is to have our customer sites look-up whether or not a visitor IP is bad for every request. That would have awful performance implications. It would mean a visitor to a site has to wait until the lookup has completed until their page loads. Yuck!

    After background-processing this problem for several months our team came up with a way of getting the best of both worlds. We realized that we could distribute the list of IPs as “hash prefixes” which achieve both of our goals. Our customer sites would not be slowed down because 99.9% of good requests would not need to do a lookup on Wordfence servers. It also made it very difficult for attackers to reverse engineer the IP blacklist to reveal new targets.

    Blocking baddies without blocking goodies

    Once we had figured out how to implement the list, the dev team started work on it and once complete the feature spent a long time with our quality assurance team. We wanted to do a great job of protecting customer sites while creating zero interference with their legitimate traffic.

    First, we developed a way for anyone blocked by the list to report false positives. Since launch, the number of false positives (good people being blocked accidentally) reported is  very low, which is impressive when you consider that the IP blacklist has processed hundreds of millions of requests since launch.

    Then we took a “phased” approach to the launch.

    Blacklist Phase 1:

    We launched the blacklist with a small number of 100 IPs on the list. Then we gradually added around 100 additional IPs by hand every day or two, until we hit 1000 IPs. We constantly evaluated how the list was doing and if there were any false positive reports. There were none.

    Blacklist Phase 2: 

    Once we hit that point, we had not received any false positive reports, so we switched to “dynamic” mode where the list auto-updated hourly. The algorithms we used in dynamic mode phase 2 to continuously identify bad IPs were “coarse” and only included the worst offenders and excluded some IPs that are engaging in lower frequency attacks.

    Blacklist Phase 3:

    In phase 3 we started optimizing our algorithms and lowering the ‘bar’ for what we considered a bad IP. We did this gradually and are still in this phase as we continue to increase the sensitivity of our algorithms to detect malicious IPs. The list is now fully live and is blocking a large number of requests from malicious IPs on our Premium customer sites every day.

    The Growth of the Wordfence IP Blacklist

    The graph below does a great job of illustrating the phases described above and the gradual way we have grown the IP blacklist and how it continues to grow:

    You can see how we steadily added to the IP blacklist manually until it hit 1000. This took about a week. Then we switched to dynamic mode and you begin to see the fluctuations in the number of IPs on the list as the number of attackers changes over time.

    As we increased the sensitivity of our algorithms there is a steady increase in the number of IPs on the list. We are currently at around 4,500 IPs on the list and we continue to increase sensitivity.

    Changes Over Time: The Blacklist is a Living Breathing Thing

    The Wordfence blacklist isn’t something we just continually add to. It is a living breathing thing which is updated continuously. IPs are continuously added and removed. You’ll notice a dip in the chart above which shows how we gradually removed over 1,000 IPs over a 1 week period as they stopped behaving maliciously.

    IPs that start attacking are immediately added to the blacklist. IPs that stop attacking are removed over time and that time is relatively short – usually a few days, depending on their behavior.

    In the chart below we show how dynamic the list is over a single 24 hour period. We chose the 11th of April for our example. You can clearly see that during the day we are constantly adding malicious IPs as they become active and constantly removing IPs that are no longer behaving badly.

    I know that many of you are “active managers” of your WordPress site security. It can begin to feel like a video game as you’re monitoring live traffic, adding IPs to your block list and blocking networks or user-agents. The blacklist solves that problem comprehensively. It relies on an incredible amount of global attack data and does an excellent job of blocking far more malicious IPs that any administrator could manage. It also unblocks IPs as they stop behaving badly to ensure that false positives don’t occur.

    The Wordfence Premium blacklist blocks new malicious IPs very quickly and any IPs that stop behaving maliciously are quickly unblocked. From the perspective of a WordPress site owner, in my opinion, it is a site admin’s dream come true.

    Ranking Signals for IP Reputation

    We think about IP reputation much the same way as Google thinks about their search algorithm. We use what we refer to as ‘ranking signals’ which are factors that influence how malicious we consider an IP address to be.

    We have no plans to publicly disclose all the ranking signals that we use for IP reputation. We don’t want attackers to be able to completely reverse engineer our methods. But as we grow the list to a significant size, we may make suggestions for network owners to help them ensure that the IPs on their network retain a healthy ranking.

    For example, if you are a hosting provider, a compromised website or hosting account can be used to attack other sites and online targets. That would negatively affect your IP’s ranking on blacklists. Responding quickly to a compromised account or website will ensure that the site IP is not blacklisted and other accounts on the same system are not negatively impacted.

    Next Steps: Optimizing Current Signals, Developing New Signals

    This week we wrote about thousands of home routers that are being used to attack WordPress websites that Wordfence protects. These home routers are engaging in very low volume attacks for short periods of time. This is a new signal that we have discovered and is a relatively weak signal. But when combined with other signals, it becomes strong and clearly reveals IPs that belong on our blacklist.

    We are constantly engaged in R&D like the research we performed to uncover the hacked home routers. This helps us discover and implement new ranking signals to grow and optimize our IP blacklist.  Now that the blacklist is live and dynamic, our goal is to continuously improve, measure and evaluate our algorithms going forward.

    The real-time IP blacklist is one of many great security features in Wordfence Premium. If you aren’t already using Wordfence, try out our completely free (and awesome) plugin today. Then when you are ready to upgrade to Premium, you can visit our home page on wordfence.com to find out more.

    The post IP Blacklist Update: The Launch and Evolution of The Wordfence IP Blacklist appeared first on Wordfence.