Block known dirty hosts from reaching your machine

Great size list.

Can someone help me understand the "!/#|[a-z]/&&/./" portion of the awk string? Would incorporating 'sort' into the command, before feeding to iptables, improve lookup efficiency (the blacklist is not numerically ordered) or is awk accomplishing this? Thank you.

! means not so !/#|[a-z]/ is: ignore anything with a comment and [a-z] is ignore letters. && is an /./

so:

awk '!/# ignore anything with pound sign

|[a-z] ignore any letters

&& and

/./ show me anything with a period

awk '!/ignore_this_string|ignore_that_string/&&/show_this_one/{print $FIELD}'

its the equivalent of grep -v "#|[a-z]"

Sil, great explanation (and fast response) thank you! Thoughts on sorting the list?

To explain things better, I figured I'd show an example: So I created a file called test with 10 lines, the first three are duplicates:

99.69.161.123

99.69.161.123

99.69.161.123

99.68.231.37

99.58.203.81

99.4.136.182

99.38.203.228

99.36.16.156

99.31.123.191

99.27.202.67

Let's number them:

1 99.69.161.123

2 99.69.161.123

3 99.69.161.123

4 99.68.231.37

5 99.58.203.81

6 99.4.136.182

7 99.38.203.228

8 99.36.16.156

9 99.31.123.191

10 99.27.202.67

So we see lines 1, 2, 3 are the same... There are a few ways to sort them uniquely. Since I began using awk, here is how to do so with awk:

Notice how I use the sort command inside awk? There is no reason to re-pipe it through to sort:

awk by itself

iptables -A INPUT -s 99.69.161.123 -j DROP

iptables -A INPUT -s 99.69.161.123 -j DROP

iptables -A INPUT -s 99.69.161.123 -j DROP

iptables -A INPUT -s 99.68.231.37 -j DROP

iptables -A INPUT -s 99.58.203.81 -j DROP

iptables -A INPUT -s 99.4.136.182 -j DROP

iptables -A INPUT -s 99.38.203.228 -j DROP

iptables -A INPUT -s 99.36.16.156 -j DROP

iptables -A INPUT -s 99.31.123.191 -j DROP

iptables -A INPUT -s 99.27.202.67 -j DROP

awk with sort -u (-u is for unique instead of: more filename | sort | uniq)

iptables -A INPUT -s 99.27.202.67 -j DROP

iptables -A INPUT -s 99.31.123.191 -j DROP

iptables -A INPUT -s 99.36.16.156 -j DROP

iptables -A INPUT -s 99.38.203.228 -j DROP

iptables -A INPUT -s 99.4.136.182 -j DROP

iptables -A INPUT -s 99.58.203.81 -j DROP

iptables -A INPUT -s 99.68.231.37 -j DROP

iptables -A INPUT -s 99.69.161.123 -j DROP

Why bother going through this when I can just use sort inside of awk. The uglier and bloated way would be something like:

awk '!/#|[a-z]/&&/./{print "iptables -A INPUT -s "$1" -j DROP"}' filename | sort | uniq

Or even uglier:

curl www.infiltrated.net/blacklisted | ruby -ne 'puts $_ unless $_ == @prev; @prev = $_' | awk '{print "iptables -A INPUT -s "$1" -j DROP"}'

This is exactly what I needed to see! Sil, you rock. The meticulous instruction was so helpful, particularly the placement of sort and its unique switch within awk versus (unnecessary) discreet use. I will use:

Excellent Contribution. Thank you.