Building Public and Private IPset Whitelists

Since your VPS might communicate with third party services, Building Public and Private IPset Whitelists is crucial as well. This Lab will take Multiple Whitelists and combine them as one file, which makes whitelist as one centralized database using IPset Module. Trusted IP addresses will be collected from different Public Sources based on your selection using just one script to allow them unrestricted web access through IPtables Firewall rules.

Objectives:

1. Understanding IPset Whitelists Concept

2. Building IPset Whitelists Database

3. Enabling IPset Whitelists Rules

4. Editing IPset Whitelists

Prerequisites:

A. Basic Linux Debian, Ubuntu, or CentOS Knowledge

B. If you haven’t built a VPS yet, login to DigitalOcean or Vultr

C. Linux Hardening Rules and IPtables Firewall Labs

D. Building Public and Private IPset Blacklists

Recommendations:

For better performance, use VPS with at least 2 CPUs, 4G Memory, 1G Bandwidth, and SSD Storage drive.

FYI: IP Addresses can be collected from Cloudflare, Pingdom, MAXCDN, Cloudfront, and many others services as needed. The last thing you want is your IPtables Firewall consider your CDN IP addresses as suspicious intruders, hence, might block them.

Understanding IPset Whitelists Concept

As mentioned, IPset is an addon module for IPtables that can be used to create or load a massive long list of bad IP Addresses and Networks. IPset acts as add on or plugin to make IPtables Firewall Manager more efficient, it’s just another Kernel Module to make Blacklist or Whilelist of IP addresses read by IPtables as if they are loaded into the VPS RAM.

IPset allows IPtables to have tiny RAM footprint. If you want to load Millions of IP addresses into your VPS using simple IPtables method without IPset, you need tones of RAM and your VPS will probably crash. IPset can utilize unlimited Public sources to combine Whitelists all together.

With such Global and Dynamic Multiple Whitelists combined on one database file, good IP addresses will be collected from different Public Sources around the world using a simple script. IPset whitelist will be used to allow trusted known IP Addresses access to either SSH, HTTP, Mail, FTP, or Apps without IPtables Firewall inspection.

IPset Whitelists will be utilized by IPtables to easy the Packet inspection. Since multiple whitelists managed by IPset as ACCEPT rule, IPtables won’t spend time investigating packets from trusted IP addresses; the best part, won’t even block them by mistake.

However, you have to tell IPtables Script about the IPset list by referring to it’s name inside IPtables Script, therefore, the entire list will be checked by IPtables Netfilter using only 2 lines of IPtables rules at remarkable speed.

Note: if you already set Public or Private Whitelists using IPtables, then you can disable those simple Whitelists, and have IPset Whitelists Script set as default as main Whitelist for both Public and Private source. But, no need to disable IPv6 simple Whitelists from the startup file, since the current IPset script meant only for IPv4.

Building IPset Whitelists Database

Run the following steps (1 to 13)

Please make sure the following packages are installed.

1. Install Required Packages

Debian Base

apt-get update

aptitude install curl ipset pv grep

Red Hat Base

yum update

yum install curl ipset pv grep

2. Create IPset Whitelist Directory

mkdir -p /etc/network/iptables/ipset-whitelist

3. Create the IPset Whitelist Script

nano /etc/network/iptables/ipset-whitelist/ipset-whitelist.sh

Open the following file, copy it’s content and paste it inside ipset-whitelist.sh file

IPset Whitelists Script

Note: as of this writing, this IPset Whitelists Script include the following third party URLs only:

"https://www.cloudflare.com/ips-v4" # Cloudflare IP addresses and networks
"https://my.pingdom.com/probes/ipv4" # Pingdom IP Addresses and Networks
"https://www.maxcdn.com/one/assets/ips.txt" # MAXCDN IP Addresses and Networks
"https://ip-ranges.amazonaws.com/ip-ranges.json" # AWS Cloudfront Edge IP Addresses and Networks

Using the same format, please add more URLs as you wish. Use # sign to disable any of the URLs.

Save: Ctrl-X, Hit Y Key, and Enter

4. Set Execute Permission

chmod +x /etc/network/iptables/ipset-whitelist/ipset-whitelist.sh

5. Create Private Custom Whitelist Database File

touch /etc/network/iptables/ipset-whitelist/ip-whitelist-custom.list

6. Allow only Root Access to the Whitelists

chmod 700 -R /etc/network/iptables/ipset-whitelist/

7. Add IPv4 Address to Private Custom Whitelist

Note: Only if needed, the following custom list meant only for your own use to be used by IPset instead of using IPtables Whitelist.

touch /etc/network/iptables/ipset-whitelist/ip-whitelist-custom.list

8. Run the IPset Whitelists Script

/etc/network/iptables/ipset-whitelist/ipset-whitelist.sh

Note: The script can hold 1 Million IPv4 Addresses. It collects good IP addresses that are stored in Private Custom Admin Whitelist. All of them will be added to the database file under /etc/network/iptables/ipset-whitelist/ip-whitelist.list

####                                                              ####
#    Preparing IPSET Whitelist Loader Script for IPtables Firewall   #
####                                                              ####

----------------------------------------------------------
 Preparing Multi Whitelists into one file...Please wait...
----------------------------------------------------------

 Done! "[#####################################################] (100%)

 Number of Public Whitelist IP/Networks found:  546

 Number of Private Custom Admin Whitelist IP/Networks found:  1

9. View IPset Whitelist

ipset -L IPSET-WHITELIST | less

Press space to move to the next page. Press q to close less command.

Name: IPSET-WHITELIST
Type: hash:net
Header: family inet hashsize 65536 maxelem 1000000
Size in memory: 1056664
References: 0
Members:
54.252.254.192/26
54.79.0.0/16
204.246.174.0/23
87.238.80.0/21
50.112.0.0/16
54.244.0.0/16
54.246.0.0/16
54.240.220.0/22
205.251.192.0/21
54.239.96.0/24
107.20.0.0/14
[...]

Note: If one of the Whitelist did not load, try again few times, if still giving you error such the following one:

Warning: curl returned HTTP response code 000 for URL https://my.pingdom.com/probes/ipv4

Then don’t skip it. Browse the list using your Internet browser, copy its IP Addresses and paste it manually into ip-whitelist-custom.list, this way IPset Whilelist Script will load them from your custom list instead. The only draw back of a custom list, that you have to update it manually.

10. Test and Verify by Editing IPset Whitelist

nano /etc/network/iptables/ipset-whitelist/ip-whitelist.list

That’s my server IPset whitelist below:

11. Add ipset-whitelist.sh to Startup

nano /etc/init.d/custom-scripts.sh

Note: Very important to add it before the Firewall Script.

[...]
# IPset Modules Check Script
/etc/network/iptables/ipset-blacklist/ipset-modules-check.sh
# IPset Blacklist Script
/etc/network/iptables/ipset-blacklist/ipset-blacklist.sh
# IPset Whitelist Script
/etc/network/iptables/ipset-whitelist/ipset-whitelist.sh
# IPTables 4 and 6 Firewall Script
/etc/network/iptables/iptfw4and6-single-node.sh

Save: Ctrl-X, Hit Y Key, and Enter

12. Create IPset Whitelist Weekly Update

nano /etc/cron.d/ipset-whitelist-update

Add the following to ipset-whitelist-update

MAILTO=root
30 23 * * 7  root /etc/network/iptables/ipset-whitelist/ipset-whitelist.sh

Save: Ctrl-X, Hit Y Key, and Enter

How to Change the Execution time?

*     *     *     *     * 
-     -     -     -     -
|     |     |     |     |
|     |     |     |     +----- Day of week (0-7)
|     |     |     +------- Month (1 - 12)
|     |     +--------- Day of month (1 - 31)
|     +----------- Hour (0 - 23)
+------------- Min (0 - 59)

13. Set Execution Permission

chmod +x /etc/cron.d/ipset-whitelist-update

Enabling IPset Whitelists Rules

Run the following steps (1 to 3)

1. Edit IPtables Firewall Script

nano /etc/network/iptables/iptfw4and6-single-node.sh

Press Ctrl-w and search for IPset Rules section, make sure the following rules under “IPset Whitelist Rules” are enabled by removing the hash sign as shown below. Besides, make sure no hash signs before echo commands.

[...]
# IPset Whitelist Rules
# Note: Enable Only if you have IPSET Plugin for IPtables installed and ipset-whitelist.sh has been already loaded.
echo
echo -e "\x1B[01;92m [+]\x1B[0m" "\x1B[01;89mSetting Public/Private IPset IPv4 Whitelists...\x1B[0m"
echo
WHITELIST=IPSET-WHITELIST
$IPT -I INPUT -p tcp -m multiport --dport 80,443 -m set --match-set $WHITELIST src -j ACCEPT
$IPT -I OUTPUT -p tcp -m multiport --dport 80,443 -m set --match-set $WHITELIST src -j ACCEPT
[...]

Save: Ctrl-X, Hit Y Key, and Enter

2. Reapply IPtables Firewall Script Again

/etc/network/iptables/iptfw4and6-single-node.sh

You should see the IPset echo message indicating that IPtables loaded IPset Whitelist

####                                                      ####
#    Preparing IPtables v4 and v6 Script for Single VPS...   #
####                                                      ####

 [+] Enabling IPTables v4 and v6 Firewall Rules...

 [+] Setting Global Policy: Dropping All IPv4/IPv6 Traffic...

 [+] Setting Public/Private IPset IPv4 Blacklists...

 [+] Setting Public/Private IPset IPv4 Whitelists...

 [+] Setting Stateful INPUT/OUPUT IPtables Firewall Rules...

 [+] Setting Rules Against TCP and UDP Port Scanning...

 [+] Setting Rules Against IPv4 Finger Printing...

 [+] Setting Rules Against Denial Of Service Attacks...

 [+] Allowing SSH Access with Brute Force Protection...

 [+] Allowing TCP or UDP Services such HTTP, HTTPS, and FTP...

 [+] Stateful IPtables Firewall Rules have been successfully Loaded!

Reapply all the Scripts Again

3. Run the Startup Custom Scripts File

Note: IPtables Firewall Script flushes and recreate IPtables chains again, however, if you still using custom chains such IPv4 or IPv6 Blacklists/Whitelists, then you have to reapply these lists again. The easiest way, is to run the startup file itself.

/etc/init.d/custom-scripts.sh

Or Reboot much better.

Adjusting IPset Whitelists

a. Adjusting Public Whitelists Links Inside ipset-whitelist.sh

nano /etc/network/iptables/ipset-whitelist/ipset-whitelist.sh

Press Ctrl-w and search for “URLs” don’t include the quotes. As you can see below, those are the Public resources Links. If you believe you need to disable any link, use hash sign # in front of any link and the script won’t call it. You can even add more links as well using the same format at the end after the last link.

[...]
# List of URLs for IP Whitelists. Currently, only IPv4 is supported in this script, everything else will be filtered.
WHITELIST=(
"https://www.cloudflare.com/ips-v4" # Cloudflare IP addresses and networks
"https://www.pingdom.com/rss/probe_servers.xml" # Pingdom IP Addresses and Networks
"https://www.maxcdn.com/one/assets/ips.txt" # MAXCDN IP Addresses and Networks
"https://ip-ranges.amazonaws.com/ip-ranges.json" # AWS Cloudfront Edge IP Addresses and Networks
)

If you did modify the script, save: Ctrl-X, Hit Y Key, and Enter

b. Run the IPset Whitelists Script Again

Only if you have modified the IPset Whitelist Script.

/etc/network/iptables/ipset-whitelist/ipset-whitelist.sh

c. Editing Private Custom IPset Whitelist Database File

It’s straight forward, but remember that you either need to run the IPset Whitelist Script again, or wait for cronjob to kick in and IPset Whitelist update itself.

nano /etc/network/iptables/ipset-whitelist/ip-whitelist-custom.list

d. Editing IPset Whitelist ALIVE!

How about adjusting the already running IPset Whitelist without reapplying the IPset Whitelist Script again? My favorite part when using IPset lists is: I can add or delete IP addresses from the whitelist while the IPtables Firewall is running, meaning, IPtables Firewall will immediately trust IP Address or Network.

Adding good IP Address or good Network

E.g. Let’s add pingdom IP 5.178.78.77 to whitelist as per http://www.ipvoid.com/scan/5.178.78.77/

IP Address

ipset add IPSET-WHITELIST 5.178.78.77

Network

E.g. Let’s add Cloudflare Network 199.27.128.0/21 to whitelist as per https://www.cloudflare.com/ips-v4

ipset add IPSET-WHITELIST 210.51.0.0/16

Removing good IP Address or good Network

IP Address

ipset del IPSET-WHITELIST 5.178.78.77

Network

ipset del IPSET-WHITELIST 210.51.0.0/16

Subject Related