Building Public and Private IPset Blacklists

Wouldn’t be nice to combine the world’s Public Blacklists in one centralized database file? Building Public and Private IPset Blacklists will make this possible by taking the blacklist concept to the next level, and show you how to apply Multiple Blacklists effectively as one centralized database using IPset Module, hence, IPtables Firewall can efficiently block thousands or even millions of IP Addresses using light weight IPset database instead of IPtables Simple Lists loaded into RAM.

Objectives:

1. Understanding IPset Blacklists Concept

2. Installing IPset Required Packages

3. Building IPset Blacklists Database

4. Enabling IPset Blacklist Rules

5. Editing IPset Blacklists

Prerequisites:

A. Basic Linux Debian, Ubuntu, or CentOS Knowledge

B. If you haven’t built a VPS yet, login to DigitalOcean or Vultr

C. Linux Hardening Rules and IPtables Firewall Labs

Recommendations:

For better performance, use VPS with at least 2 CPUs, 4G Memory, 1G Bandwidth, and SSD Storage drive.

Understanding IPset Blacklists Concept

IPset is an addon module for IPtables that can be used to create or load a massive long list of bad IP Addresses and Networks. IPset acts as add on or plugin to make IPtables Firewall Manager more efficient, it’s just another Kernel Module to make Blacklist or Whilelist of IP addresses read by IPtables as if they are loaded into the VPS RAM.

IPset allows IPtables to have tiny RAM footprint. If you want to load Millions of IP addresses into your VPS using simple IPtables method without IPset, you need tones of RAM and your VPS will probably crash. Due to tiny piece of Memory used, IPset makes IPtables Firewall extremely efficient to go through thousands of bad IP Addresses.

IPset can utilize unlimited Public sources to combine Blacklists all together. With such Global and Dynamic Multiple Blacklists combined on one database file, bad IP addresses will be collected from different Public Sources around the world using a simple script. IPset blacklist will be used to block bad known IP Addresses against either SSH, HTTP, Mail, FTP, etc…

However, you have to tell IPtables Script about the IPset list by referring to it’s name inside IPtables Script, therefore, the entire list will be checked by IPtables Netfilter using only 2 lines of IPtables rules at remarkable speed.

Note: if you already set Public or Private simple Blacklists or Custom Admin Blacklists using the previous labs, you can disable those simple Blacklists, and have IPset Blacklist Script set as default as main Blacklist for both Public and Private source. However, no need to disable IPv6 simple blacklists from the startup file, since the current IPset script meant only for IPv4.

Installing IPset Required Packages

Run the following steps (1 to 7)

Please make sure the following packages are installed.

1. Install Required Packages

Debian Based

apt-get update

aptitude install curl ipset pv grep

Red Hat Based

yum update

yum install curl ipset pv grep

2. Create IPset Blacklist Directory

mkdir -p /etc/network/iptables/ipset-blacklist

3. Create IPset Modules Loader Script

nano /etc/network/iptables/ipset-blacklist/ipset-modules-check.sh

Open the following file, copy it’s content and paste it inside ipset-modules-check.sh file

IPset Modules Check Script

Save: Ctrl-X, Hit Y Key, and Enter

4. Set Execute Permission

chmod +x /etc/network/iptables/ipset-blacklist/ipset-modules-check.sh

5. Allow only Root Access

chmod 700 -R /etc/network/iptables/ipset-blacklist/

6. Run the IPset Modules Check

/etc/network/iptables/ipset-blacklist/ipset-modules-check.sh

You should see the following Modules getting Loaded:

 Loading IPSET Modules  - Verifying if all IPSET Modules are Loadable
----------------------------------------------------------------------
ip_set, 0
ip_set_bitmap_ipmac, 0
ip_set_hash_ip, 0
ip_set_hash_ipportip, 0
ip_set_hash_net, 0
ip_set_hash_netport, 0
ip_set_bitmap_ip, 0
ip_set_bitmap_port, 0
ip_set_hash_ipport, 0
ip_set_hash_ipportnet, 0
ip_set_hash_netiface, 0
ip_set_list_set, 0
----------------------------------------------------------------------
   Done Loading IPtables Modules.

Note: Most VPS that is based on KVM should load Modules without any problems, however, not all modules might load using OpenVZ VPS.

7. Add IPset Modules Loader to Startup file

nano /etc/init.d/custom-scripts.sh

Add it to the end of the file.

Note: As you can see below, I have disabled only IPv4 individual simple Blacklists since IPset will include all Public and Private Blacklists, but I left all IPv6 blacklists because this IPset Script doesn’t support IPv6 yet.

Note: Very important to add it before the Firewall Script.

[...]
# IPset Modules Check Script
/etc/network/iptables/ipset-blacklist/ipset-modules-check.sh
# IPTables 4 and 6 Firewall Script
/etc/network/iptables/iptfw4and6-single-node.sh
# Blocklist.de SSH List
#/etc/network/iptables/blacklists/blocklist-de-ssh.sh
# IPv6 MYIP.MS Blacklist 
/etc/network/iptables/blacklists/ipv6-myip-blacklist.sh
# Custom Admin Blacklist
#/etc/network/iptables/blacklists/custom-admin-blacklist.sh
# IPv6 Custom Admin Blacklist
/etc/network/iptables/blacklists/v6custom-admin-blacklist.sh
# IPset Modules Check Script
/etc/network/iptables/ipset-blacklist/ipset-modules-check.sh

Save: Ctrl-X, Hit Y Key, and Enter

Building IPset Blacklists Database

Run the following steps (1 to 11)

1. Create the IPset Blacklist Script

nano /etc/network/iptables/ipset-blacklist/ipset-blacklist.sh

Open the following file, copy it’s content and paste it inside ipset-blacklist.sh file

IPset Blacklists Script

Note: I am assuming that you are using Public VPS, however, if you are testing under Virtual Environment using private networks, then you have to disable the following source link as shown below using hash sign under the BLACKLISTS section before you save.

Using nano editor, press Ctrl-w key and search for Bogons, once you find this line, disable it using # sign and save. Do this only if you are testing using Private Network such VirtualBox or VMware Station.

#"http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt" # IP Addresses that should not be used on the internet

Save: Ctrl-X, Hit Y Key, and Enter

2. Set Execute Permission

chmod +x /etc/network/iptables/ipset-blacklist/ipset-blacklist.sh

3. Create Private Custom Blacklist Database File

Note: The following local custom list for your own use, assuming you have an IP or list of IP addresses that are not part of Public source.

touch /etc/network/iptables/ipset-blacklist/ip-blacklist-custom.list

4. Allow only Root Access to the Blacklists

chmod 700 -R /etc/network/iptables/ipset-blacklist/

5. Add bad IPv4 Address to the Private Custom Blacklist

Based on IP Void http://www.ipvoid.com/scan/43.255.190.135/ this address is bad IP address, Copy and paste 43.255.190.135 inside ip-blacklist-custom.list file.

nano /etc/network/iptables/ipset-blacklist/ip-blacklist-custom.list

Save: Ctrl-X, Hit Y Key, and Enter

6. Run the IPset Blacklists Script

/etc/network/iptables/ipset-blacklist/ipset-blacklist.sh

Note: The script can hold 1 Million IPv4 Addresses. As of this writing, IPset Blacklist collected 185,965 Public Bad IPv4 Addresses and Networks. Besides, it collected 1 bad IP address that you have added in the previous step. All of them have been added to the database file under /etc/network/iptables/ipset-blacklist/ip-blacklist.list

####                                                              ####
#    Preparing IPSET Blacklist Loader Script for IPtables Firewall   #
####                                                              ####

----------------------------------------------------------
 Preparing Multi Blacklists into one file...Please wait...
----------------------------------------------------------

 Done! "[#####################################################] (100%)

 Number of Public Blacklist IP/Networks found:  185965


 Number of Private Custom Admin Blacklist IP/Networks found:  1

Note: Some links might be down, so that’s how they look like:

Warning: curl returned HTTP response code 403 for URL http://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1
Warning: curl returned HTTP response code 301 for URL https://www.maxmind.com/en/anonymous-proxy-fraudulent-ip-address-list

7. View IPset Blacklist

ipset -L IPSET-BLACKLIST | less

Press space to move to the next page. Press q to close less command.

Name: IPSET-BLACKLIST
Type: hash:net
Header: family inet hashsize 65536 maxelem 1000000
Size in memory: 2684728
References: 0
Members:
85.20.242.36
173.248.176.187
190.142.249.235
92.222.21.226
121.18.211.70
93.61.100.21
122.224.6.138
173.234.226.187
173.234.227.3
204.92.23.107
[...]

8. Test and Verify by editing IPset Blacklist

 nano  /etc/network/iptables/ipset-blacklist/ip-blacklist.list

That’s my server IPset blacklist below:

9. Add IPset Blacklist Script to Startup file

nano /etc/init.d/custom-scripts.sh

Note: Very important to add it before the Firewall Script.

[...]
# IPset Modules Check Script
/etc/network/iptables/ipset-blacklist/ipset-modules-check.sh
# IPset Blacklist Script
/etc/network/iptables/ipset-blacklist/ipset-blacklist.sh
# IPTables 4 and 6 Firewall Script
/etc/network/iptables/iptfw4and6-single-node.sh

Save: Ctrl-X, Hit Y Key, and Enter

10. Create IPset Blacklist Weekly Update

nano /etc/cron.d/ipset-blacklist-update

Add the following to ipset-blacklist-update file.

MAILTO=root
30 23 * * 7  root /etc/network/iptables/ipset-blacklist/ipset-blacklist.sh

Save: Ctrl-X, Hit Y Key, and Enter

How to Change the Execution time?

*     *     *     *     * 
-     -     -     -     -
|     |     |     |     |
|     |     |     |     +----- Day of week (0-7)
|     |     |     +------- Month (1 - 12)
|     |     +--------- Day of month (1 - 31)
|     +----------- Hour (0 - 23)
+------------- Min (0 - 59)

11. Set Execution Permission

chmod +x /etc/cron.d/ipset-blacklist-update

Enabling IPset Blacklists Rules

Run the following steps (1 to 3)

1. Edit IPtables Firewall Script

Press Ctrl-w and search for IPset Rules section, make sure the following rules under “IPset Blacklist Rules” are enabled by removing the hash sign as shown below. Besides, make sure no hash signs before echo commands.

nano /etc/network/iptables/iptfw4and6-single-node.sh

[...]
# IPset Blacklist Rules
# Note: Enable Only if you have IPSET Plugin for IPtables installed and ipset-blacklist.sh has been already loaded.
echo
echo -e "\x1B[01;92m [+]\x1B[0m" "\x1B[01;89mSetting Public/Private IPset IPv4 Blacklists...\x1B[0m"
echo
BLACKLIST=IPSET-BLACKLIST
$IPT -I INPUT -m set --match-set $BLACKLIST src -j DROP
$IPT -I OUTPUT -m set --match-set $BLACKLIST src -j DROP
[...]

Save: Ctrl-X, Hit Y Key, and Enter

2. Reapply IPtables Firewall Script Again

/etc/network/iptables/iptfw4and6-single-node.sh

You should see the IPset echo message indicating that IPtables loaded IPset Blacklist

####                                                      ####
#    Preparing IPtables v4 and v6 Script for Single VPS...   #
####                                                      ####

 [+] Enabling IPTables v4 and v6 Firewall Rules...

 [+] Setting Global Policy: Dropping All IPv4/IPv6 Traffic...

 [+] Setting Public/Private IPset IPv4 Blacklists...

 [+] Setting Stateful INPUT/OUPUT IPtables Firewall Rules...

 [+] Setting Rules Against TCP and UDP Port Scanning...

 [+] Setting Rules Against IPv4 Finger Printing...

 [+] Setting Rules Against Denial Of Service Attacks...

 [+] Allowing SSH Access with Brute Force Protection...

 [+] Allowing TCP or UDP Services such HTTP, HTTPS, and FTP...

 [+] Stateful IPtables Firewall Rules have been successfully Loaded!

Reapply all the Scripts Again

3. Run the Startup Custom Scripts File

Note: IPtables Firewall Script flushes and recreate IPtables chains again, however, if you still using custom chains such IPv4 or IPv6 Blacklists/Whitelists, then you have to reapply these lists again. The easiest way, is to run the startup file itself.

/etc/init.d/custom-scripts.sh

Your VPS now is protected against thousands of known attackers in effective way using IPtables and IPset Blacklist.

Adjusting IPset Blacklists

a. Editing Online Blacklists Links Inside ipset-blacklist.sh

nano /etc/network/iptables/ipset-blacklist/ipset-blacklist.sh

Press Ctrl-w and search for “URLs” don’t include the quotes. As you can see below, those are the Online resources Links. If you believe you need to disable any link, use hash sign # in front of any link and the script won’t call it. You can even add more links as well using the same format at the end after the last link.

[...]
# List of URLs for IP blacklists. Currently, only IPv4 is supported in this script, everything else will be filtered.
BLACKLISTS=(
"http://lists.blocklist.de/lists/all.txt" # blocklist.de ALL type of Attacks
"http://myip.ms/files/blacklist/csf/latest_blacklist.txt" # BAD IPs MYIP.MS General Blacklist
"http://antivirus.neu.edu.cn/ssh/lists/base_30days.txt" # NEU SSH Blacklist
"http://cinsscore.com/list/ci-badguys.txt" # Cinnscore Bad Guys Blacklist
"http://www.openbl.org/lists/base.txt" # OpenBL.org 30 day List
"http://www.stopforumspam.com/downloads/toxic_ip_cidr.txt" # StopForumSpam
"http://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1" # Project Honey Pot Directory of Dictionary Attacker IPs
"http://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1" # TOR Exit Nodes
"https://www.maxmind.com/en/anonymous-proxy-fraudulent-ip-address-list" # MaxMind GeoIP Anonymous Proxies
"http://danger.rulez.sk/projects/bruteforceblocker/blist.php" # BruteForceBlocker IP List
"http://www.spamhaus.org/drop/drop.lasso" # Spamhaus Don't Route Or Peer List (DROP)
"http://www.stopforumspam.com/downloads/toxic_ip_cidr.txt" # Spamhaus Networks - identified as highly toxic CIDR Networks
"http://cinsscore.com/list/ci-badguys.txt" # C.I. Army Malicious IP List
"http://www.openbl.org/lists/base.txt"  # OpenBL.org 30 day List
"http://www.autoshun.org/files/shunlist.csv" # Autoshun Shun List
"https://www.badips.com/get/list/ssh/5" # Badips.com SSH Level 5
"https://www.badips.com/get/list/apache/1" # Badips.com Apache Level 1
"https://www.badips.com/get/list/nginx/1" # Badips.com Nginx Level 1
"https://www.badips.com/get/list/mail/1" # Badips.com Mail Level 1
"https://www.badips.com/get/list/http/1" # Badips.com HTTP Level 1
"https://www.badips.com/get/list/bruteforce/1" # Badips.com Bruteforce Level 1
"https://www.badips.com/get/list/cms/1" # Badips.com CMS Level 1
"http://www.okean.com/sinokoreacidr.txt" # Okean chinese and korean spammers
"http://www.wizcrafts.net/russian-iptables-blocklist.html" # Wizcrafts Russian botnets, attackers and spammers
"http://www.wizcrafts.net/nigerian-iptables-blocklist.html" # Wizcrafts Nigeria attackers and spammers
"http://www.wizcrafts.net/exploited-servers-iptables-blocklist.html" # Wizcrafts Exploited Servers
"http://www.wizcrafts.net/chinese-iptables-blocklist.html" # Wizcrafts Chinese attackers and spammers
"http://www.wizcrafts.net/lacnic-iptables-blocklist.html" # This is Wizcrafts' Latin America (LACNIC) blocklist
"http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt" # RBN Russian IPs
"http://feeds.dshield.org/block.txt" # Dshield
"http://feeds.dshield.org/top10-2.txt" # Dshield
"https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist" # Bot nets
"https://zeustracker.abuse.ch/blocklist.php?download=badips" # Bad IPs
"https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt" # Emerging Threats Firewall rules
"http://danger.rulez.sk/projects/bruteforceblocker/blist.php" # By Daniel Geržo http://danger.rulez.sk/index.php/about/
"http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt" # IP Addresses that should not be used on the internet
)

If you did modify the script, save: Ctrl-X, Hit Y Key, and Enter

b. Run the IPset Blacklists Script Again

Only if you have modified the IPset Blacklist Script.

/etc/network/iptables/ipset-blacklist/ipset-blacklist.sh

Note: In case you see the following errors: That means you have passed the daily allowed times to pull addresses from the following lists.

Warning: curl returned HTTP response code 000 for URL http://antivirus.neu.edu.cn/ssh/lists/base_30days.txt
Warning: curl returned HTTP response code 000 for URL https://www.badips.com/get/list/ssh/5
Warning: curl returned HTTP response code 000 for URL https://www.badips.com/get/list/apache/1
Warning: curl returned HTTP response code 000 for URL https://www.badips.com/get/list/nginx/1
Warning: curl returned HTTP response code 000 for URL https://www.badips.com/get/list/mail/1
Warning: curl returned HTTP response code 000 for URL https://www.badips.com/get/list/http/1
Warning: curl returned HTTP response code 000 for URL https://www.badips.com/get/list/bruteforce/1
Warning: curl returned HTTP response code 000 for URL https://www.badips.com/get/list/cms/1

c. Editing Private Custom IPset Blacklist Database File

It’s straight forward, but remember that you either need to run the IPset Blacklist Script again, or wait for cronjob to kick in so IPset Blacklist update itself.

nano /etc/network/iptables/ipset-blacklist/ip-blacklist-custom.list

d. Editing IPset Blacklist ALIVE!

How about adjusting the already running IPset Blacklist without reapplying the IPset Blacklist Script again? My favorite part when using IPset Blacklist is: I can add or delete IP addresses from the blacklist while the IPtables Firewall is running, meaning, IPtables Firewall will immediately block or unblock the IP Address/Network.

Adding bad IP Address or Bad Network

E.g. Let’s add 43.255.190.135 to blacklist as per http://www.ipvoid.com/scan/43.255.190.135/

IP Address

ipset add IPSET-BLACKLIST 43.255.190.135

Network

ipset add IPSET-BLACKLIST 210.51.0.0/16

Removing bad IP Address or Bad Network

IP Address

ipset del IPSET-BLACKLIST 43.255.190.135

Network

ipset del IPSET-BLACKLIST 210.51.0.0/16

Subject Related