Intro
looking to implement - "...HOSTS file to block ads, banners, 3rd party Cookies, 3rd party page counters, web bugs, and even most hijackers."
pfSense uses dnsmasq and we need to convert entries from HOSTS file to the dnsmasq format
0.0.0.0 static.a-ads.com -> address=/static.a-ads.com/127.0.0.1
We will have it scheduled to run daily at midnight to keep the list up to date
note: On pfSense 2.2, The DNS Forwarder is not active by default. It has been replaced by Unbound as a DNS Resolver. It may still be used, and is still active on upgraded configurations. To use the DNS Forwarder (dnsmasq) on 2.2, first disable Unbound and then enable the DNS Forwarder.Dec 26, 2014
DNS Forwarder - PFSenseDocs
https://doc.pfsense.org/index.php/DNS_Forwarder
Therefore this applies only if you're using dnsmasq
Preparation
install package cron 0.1.8
create folder in ssh
# mkdir /usr/local/etc/dnsmasq.d/
go to pfSense:
Services -> DNS Forwarded
enable advanced button and enter:
conf-dir=/usr/local/etc/dnsmasq.d
Command
fetch -qR http://winhelp2002.mvps.org/hosts.txt /root/hosts.txt && perl -e 'while(<>){ chomp; lc; next if /^#/; if (/^0\.0\.0\.0\s([-a-z0-9.]*)/) { print "address=\/$1\/127.0.0.1\n"; } }' /root/hosts.txt > /usr/local/etc/dnsmasq.d/entries && pfSsh.php playback svc restart dnsmasqbroken down and explained
fetch -qR http://winhelp2002.mvps.org/hosts.txt /root/hosts.txt
&&
# -q, --quiet
Quiet mode.
# -R, --keep-output
The output files are precious, and should not be deleted
under any circumstances, even if the transfer failed or was
incomplete.
# saves to /root/hosts.txt
---------------------------------------------------------------------------
perl -e '
while(<>) { # for every line in input file (specified as argument below)
chomp; # remove newlines
lc; # lowercase all characters
next if /^#/; # skip if it matches character '#' at the begining - comments
if (/^0\.0\.0\.0\s([-a-z0-9.]*)/)
# if it matches this 0.0.0.0(space)(url - can only contain a-z0-9 and a dash)
# example 0.0.0.0 static.a-ads.com
{
print "address=\/$1\/127.0.0.1\n";
# print what we matched $1 and the rest of the text so it looks like:
# address=/static.a-ads.com/127.0.0.1
}
}'
/root/hosts.txt # our input file
> /usr/local/etc/dnsmasq.d/entries # save to this file
&&
---------------------------------------------------------------------------
pfSsh.php playback svc restart dnsmasq # restart dnsmasq service to reload file
Application
test the line by running it in the SSH - check the output of
/root/hosts.txt
/usr/local/etc/dnsmasq.d/entries
when everything runs and you have the 'entries' file populated correctly, schedule it to run daily
Schedule
After you install the package, go to pfSense:
Services -> Cron
Pay attention to add full paths
Pay attention to add full paths
Thanks for posting this, I have been looking exactly for this! However,
ReplyDeletethis does not work as you have it documented. First, the command does not have paths but your screenshot of Cron does ('/usr/bin/fetch' vs 'fetch'). Second, I get the following error when the command runs: 'fetch: /root/hosts.txt: No such file or directory'. So I manually created an empty /root/hosts.txt file. Third, even after doing this and running the command, both /root/hosts.txt and /usr/local/etc/dnsmasq.d/entries are empty.
Any ideas?
if I manually fetch the hosts.txt file while logged into /root, then the host.txt file populates, then when I run the cronjob, it will copy it over to /usr/local/etc/dnsmasq.d/entries. I verified it works on my client PCs. Yay! But I'm concerned the next time the win2002help.mvps.org updates their host file, will it fetch it and push it to /usr/local/etc/dnsmasq.d?
ReplyDeletehi!
ReplyDeletethe command is to be run from command line and you're right it's different than the cron version as the paths aren't required from the shell.
once you've run it at least once all the files and folders would be created for the cron version with correct permissions
you can test the nightly routine by editing the /usr/local/etc/dnsmasq.d/entries and erasing all the lines. the nightly cron should repopulate
the documentation is primarily for understanding what is happening so you could adapt to your situation easily.
* From the console, I had to install the package manager, just run "pkg".
ReplyDelete* Then I had to install perl5, "pkg install perl5"
* Then I needed to adjust the paths in the script quite a bit.
/usr/bin/fetch -qR http://winhelp2002.mvps.org/hosts.txt -o /root/mvp_hosts.txt && /usr/local/bin/perl -e 'while(<>){ chomp; lc; next if /^#/; if (/^0\.0\.0\.0\s([-a-z0-9.]*)/) { print "address=\/$1\/127.0.0.1\n"; } }' /root/mvp_hosts.txt > /usr/local/etc/dnsmasq.d/entries && /usr/local/sbin/pfSsh.php playback svc restart dnsmasq
fetch was in particular missing the require -o before the output file path...
DeleteWhat do I need to update for this to work with Unbound?
ReplyDelete