February 10, 2015

automating winhelp2002 MVP HOSTS file to pfSense 2.2

Intro

looking to implement - "...HOSTS file to block ads, banners, 3rd party Cookies, 3rd party page counters, web bugs, and even most hijackers."


pfSense uses dnsmasq and we need to convert entries from HOSTS file to the dnsmasq format

0.0.0.0 static.a-ads.com -> address=/static.a-ads.com/127.0.0.1

We will have it scheduled to run daily at midnight to keep the list up to date

noteOn pfSense 2.2, The DNS Forwarder is not active by default. It has been replaced by Unbound as a DNS Resolver. It may still be used, and is still active on upgraded configurations. To use the DNS Forwarder (dnsmasq) on 2.2, first disable Unbound and then enable the DNS Forwarder.Dec 26, 2014
DNS Forwarder - PFSenseDocs
https://doc.pfsense.org/index.php/DNS_Forwarder

Therefore this applies only if you're using dnsmasq

Preparation

install package cron 0.1.8

create folder in ssh 
  # mkdir /usr/local/etc/dnsmasq.d/

go to pfSense: 
Services -> DNS Forwarded
  enable advanced button and enter: 
  conf-dir=/usr/local/etc/dnsmasq.d



Command

fetch -qR http://winhelp2002.mvps.org/hosts.txt /root/hosts.txt && perl -e 'while(<>){ chomp; lc; next if /^#/; if (/^0\.0\.0\.0\s([-a-z0-9.]*)/) { print "address=\/$1\/127.0.0.1\n"; } }' /root/hosts.txt > /usr/local/etc/dnsmasq.d/entries && pfSsh.php playback svc restart dnsmasq

broken down and explained

fetch -qR http://winhelp2002.mvps.org/hosts.txt /root/hosts.txt 
&& 

# -q, --quiet
Quiet mode.
# -R, --keep-output
The output files are precious, and should not be deleted
under any circumstances, even if the transfer failed or was
incomplete.

# saves to /root/hosts.txt
---------------------------------------------------------------------------

perl -e '
  while(<>) {    # for every line in input file (specified as argument below)

    chomp;         # remove newlines

    lc;                 # lowercase all characters

    next if /^#/;  # skip if it matches character '#' at the begining - comments

    if (/^0\.0\.0\.0\s([-a-z0-9.]*)/)     
        # if it matches this 0.0.0.0(space)(url - can only contain a-z0-9 and a dash)
        # example 0.0.0.0 static.a-ads.com

    { 
       print "address=\/$1\/127.0.0.1\n"; 
        # print what we matched $1 and the rest of the text so it looks like: 
        #   address=/static.a-ads.com/127.0.0.1

     } 
  }' 

/root/hosts.txt                                          # our input file
> /usr/local/etc/dnsmasq.d/entries          # save to this file
&& 
---------------------------------------------------------------------------

pfSsh.php playback svc restart dnsmasq   # restart dnsmasq service to reload file


Application


test the line by running it in the SSH - check the output of
/root/hosts.txt
/usr/local/etc/dnsmasq.d/entries


when everything runs and you have the 'entries' file populated correctly, schedule it to run daily 

Schedule


After you install the package, go to pfSense: 
Services -> Cron
Pay attention to add full paths