OpenWRT adblock implementation

README.md

Others have recently developed packages for this same functionality, and done it better than anything I could do. Use the packages instead of this script:

• Gargoyle package by @lantis1008

• OpenWRT package by @dibdot

Description

In its basic usage, this script will modify the router such that blocked addresses are null routed and unreachable. Since the address blocklist is full of advertising, malware, and tracking servers, this setup is generally a good thing. In addition, the router will update the blocklist weekly. However, the blocking is leaky, so do not expect everything to be blocked.

Setup

The script must be copied to an OpenWRT router (gargoyle firmware works fine, too).

For example, if the router is located at 192.168.1.1:

# scp adblock.sh [email protected]:/etc/adblock.sh

Make the script executable:

# chmod +x /etc/adblock.sh

Basic usage

If you are running the script for the first time:

# sh /etc/adblock.sh -f

There should be status updates in the output, but there should be no errors. If these commands complete without errors, the adblocking is active. You can test it by looking up, say, google analytics.

Whitelists and blacklists

The script supports defining whitelisted urls. That is, urls that will be filtered out of the downloaded blocklists. To whitelist urls, place them (one per line) in /etc/white.list.

Similarly, the script supports defining blacklisted urls - urls that will be added to the downloaded blocklists. To blacklist urls, place them (one per line) in /etc/black.list.

NOTE: The whitelist support is pretty stupid, so don't expect smart filtering (e.g., domain extrapolation). I've found it tedious, but worthwhile, to find the offending url in /etc/block.hosts and copy it to /etc/white.list.

Advanced usage

Toggle on and off

To toggle the blocking on and off, run the script with the -t switch:

# sh /etc/adblock.sh -t

Note: This does not delete the blocklist, whitelist, or blacklist.

Manually update blocklist

To manually update the blocklist, run the script without switches:

# sh /etc/adblock.sh

Full reinstall

To reinstall the current implementation:

# sh /etc/adblock.sh -r

Configuration

The config section of the script has some variables that alter the behaviour of the script.

For example, if you change:

ONLY_WIRELESS="N"

to

ONLY_WIRELESS="Y"

Then only the wireless interface of the router will filter the blocklist.

To change the configuration of an already active installation, I would toggle the adblocking off first, change the script, then toggle it back on. That is,

# sh /etc/adblock.sh -t # turn off
...modify variables...
# sh /etc/adblock.sh -t # turn on

However, if you change certain variables, you must re-update the blocklist because the redirection values will have changed. Other variables require a full restart because they must install or verify dependencies.

Configurable variables:

• ONLY_WIRELESS (Y/N): Only filter on wireless interface
• EXEMPT# (Y/N): Exempt ip range from filtering (between START_ RANGE and END_RANGE)
• IPV6*: (Y/N): Add IPv6 support
• SSL# (Y/N): Install wget with ssl support (only needed for ssl websites)
• TRANS#: (Y/N): Modify router web server to server transparent pixel responses for blocked websites
• ENDPOINT_IP4/IP6*: Define the IP to return for blocked hostnames (IPv4 and IPv6)
• CRON: The cron line to put in the crontab

*: Requires blocklist update.

: Requires full reinstall

adblock.sh

#!/bin/sh
#Put in /etc/adblock.sh
#Block ads, malware, etc.

#### CONFIG SECTION ####

# Only block wireless ads? Y/N
ONLY_WIRELESS="N"

# IPv6 support? Y/N
IPV6="N"

# Need SSL websites?
SSL="N"

# Try to transparently serve pixel response?
#   If enabled, understand the consequences and mechanics of this setup
TRANS="N"

# Exempt an ip range
EXEMPT="N"
START_RANGE="192.168.1.0"
END_RANGE="192.168.1.255"

# Redirect endpoint
ENDPOINT_IP4="0.0.0.0"
ENDPOINT_IP6="::"

#Change the cron command to what is comfortable, or leave as is
CRON="0 4 * * 0,3 sh /etc/adblock.sh"

#### END CONFIG ####

#### FUNCTIONS ####

cleanup()
{
    #Delete files used to build list to free up the limited space
    echo 'Cleaning up...'
    rm -f /tmp/block.build.list
    rm -f /tmp/block.build.before
}

install_dependencies()
{
    #Need iptables-mod-nat-extra installed
    if opkg list-installed | grep -q iptables-mod-nat-extra
    then
        echo 'iptables-mod-nat-extra is installed!'
    else
        echo 'Updating package list...'
        opkg update > /dev/null
        echo 'Installing iptables-mod-nat-extra...'
        opkg install iptables-mod-nat-extra > /dev/null
    fi

    #Need iptable-mod-iprange for exemption
    if [ "$EXEMPT" = "Y" ]
    then 
        if opkg list-installed | grep -q iptables-mod-iprange
        then
            echo 'iptables-mod-iprange installed'
        else
            echo 'Updating package list...'
            opkg update > /dev/null
            echo 'Installing iptables-mod-iprange...'
            opkg install iptables-mod-iprange > /dev/null
        fi
    fi

    #Need wget for https websites
    if [ "$SSL" = "Y" ]
    then
        if opkg list-installed wget | grep -q wget
        then
            if wget --version | grep -q +ssl
            then
                echo 'wget (with ssl) found'
            else
                # wget without ssl, need to reinstall full wget
                opkg update > /dev/null
                opkg install wget --force-reinstall > /dev/null
            fi
        else
            echo 'Updating package list...'
            opkg update > /dev/null
            echo 'Installing wget (with ssl)...'
            opkg install wget > /dev/null
        fi
    fi
}

add_config()
{
    if [ "$ONLY_WIRELESS" = "Y" ]
    then
        echo 'Wireless only blocking!'
        if [ "$EXEMPT" = "Y" ]
        then
            echo 'Exempting some ips...'
            FW1="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -i wlan+ -p tcp --dport 53 -j REDIRECT --to-ports 53"
            FW2="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -i wlan+ -p udp --dport 53 -j REDIRECT --to-ports 53"
        else
            FW1="iptables -t nat -I PREROUTING -i wlan+ -p tcp --dport 53 -j REDIRECT --to-ports 53"
            FW2="iptables -t nat -I PREROUTING -i wlan+ -p udp --dport 53 -j REDIRECT --to-ports 53"
        fi
    else
        if [ "$EXEMPT" = "Y" ]
        then
            echo "Exempting some ips..."
            FW1="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -p tcp --dport 53 -j REDIRECT --to-ports 53"
            FW2="iptables -t nat -I PREROUTING -m iprange ! --src-range $START_RANGE-$END_RANGE -p udp --dport 53 -j REDIRECT --to-ports 53"
        else
            FW1="iptables -t nat -I PREROUTING -p tcp --dport 53 -j REDIRECT --to-ports 53"
            FW2="iptables -t nat -I PREROUTING -p udp --dport 53 -j REDIRECT --to-ports 53"
        fi
    fi

    echo 'Updating config...'

    #Update DHCP config
    uci add_list dhcp.@dnsmasq[0].addnhosts=/etc/block.hosts > /dev/null 2>&1 && uci commit

    #Add to crontab
    echo "$CRON" >> /etc/crontabs/root

    #Update dnsmasq config for Tor
    TOR=`uci get tor.global.enabled 2> /dev/null`
    if [ "$TOR" == "1" ]
    then
        TORPORT=`uci get tor.client.dns_port`
        TORIP="127.0.0.1:$TORPORT"
        uci set dhcp.@dnsmasq[0].noresolv='1' > /dev/null &2>1 && uci commit
        uci add_list dhcp.@dnsmasq[0].server="$TORIP" > /dev/null &2>1 && uci commit
    fi

    # Add firewall rules
    echo "$FW1" >> /etc/firewall.user
    echo "$FW2" >> /etc/firewall.user

    # Provide hint if localservice is 1
    LS=`uci get dhcp.@dnsmasq[0].localservice 2> /dev/null`
    if [ "$LS" == "1" ]
    then
        echo "HINT: localservice is set to 1"
        echo "    Adblocking (and router DNS) over a VPN may not work"
        echo "    To allow VPN router DNS, manually set localservice to 0"
    fi


    # Determining uhttpd/httpd_gargoyle for transparent pixel support
    if [ "$TRANS" = "Y" ]
    then
        if [ ! -e "/www/1.gif" ]
        then
            /usr/bin/wget -O /www/1.gif http://upload.wikimedia.org/wikipedia/commons/c/ce/Transparent.gif  > /dev/null
        fi
        if [ -s "/usr/sbin/uhttpd" ]
        then
            #The default is none, so I don't want to check for it, so just write it
            echo "uhttpd found..."
            echo "updating server error page to return transparent pixel..."
            uci set uhttpd.main.error_page="/1.gif" && uci commit
        elif [ -s "/usr/sbin/httpd_gargoyle" ]
        then
            # Write without testing
            echo "httpd_gargoyle found..."
            echo "updating server error page to return transparent pixel..."
            uci set httpd_gargoyle.server.page_not_found_file="1.gif" && uci commit
        else
            echo "Cannot find supported web server..."
        fi
    fi
}

update_blocklist()
{
    #Delete the old block.hosts to make room for the updates
    rm -f /etc/block.hosts

    # Correct endpoint for transparent pixel response
    if [ "$TRANS" = "Y" ] && [ -e "/www/1.gif" ] && ([ -s "/usr/sbin/uhttpd" ] || [ -s "/usr/sbin/httpd_gargoyle" ])
    then
        ENDPOINT_IP4=$(uci get network.lan.ipaddr)
        if [ "$IPV6" = "Y" ]
        then
            ENDPOINT_IP6=$(uci get network.lan6.ipaddr)
        fi
    fi
    
    echo 'Downloading hosts lists...'
    #Download and process the files needed to make the lists (enable/add more, if you want)
    wget -qO- http://www.mvps.org/winhelp2002/hosts.txt| awk -v r="$ENDPOINT_IP4" '{sub(/^0.0.0.0/, r)} $0 ~ "^"r' > /tmp/block.build.list
    wget -qO- "http://adaway.org/hosts.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
    #wget -qO- http://www.malwaredomainlist.com/hostslist/hosts.txt|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
    #wget -qO- "http://hosts-file.net/.\ad_servers.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list

    #Add black list, if non-empty
    if [ -s "/etc/black.list" ]
    then
        echo 'Adding blacklist...'
        awk -v r="$ENDPOINT_IP4" '/^[^#]/ { print r,$1 }' /etc/black.list >> /tmp/block.build.list
    fi

    echo 'Sorting lists...'

    #Sort the download/black lists
    awk '{sub(/\r$/,"");print $1,$2}' /tmp/block.build.list|sort -u > /tmp/block.build.before

    #Filter (if applicable)
    if [ -s "/etc/white.list" ]
    then
        #Filter the blacklist, supressing whitelist matches
        #  This is relatively slow =-(
        echo 'Filtering white list...'
        egrep -v "^[[:space:]]*$" /etc/white.list | awk '/^[^#]/ {sub(/\r$/,"");print $1}' | grep -vf - /tmp/block.build.before > /etc/block.hosts
    else
        cat /tmp/block.build.before > /etc/block.hosts
    fi

    if [ "$IPV6" = "Y" ]
    then
        safe_pattern=$(printf '%s\n' "$ENDPOINT_IP4" | sed 's/[[\.*^$(){}?+|/]/\\&/g')
        safe_addition=$(printf '%s\n' "$ENDPOINT_IP6" | sed 's/[\&/]/\\&/g')
        echo 'Adding ipv6 support...'
        sed -i -re "s/^(${safe_pattern}) (.*)$/\1 \2\n${safe_addition} \2/g" /etc/block.hosts
    fi
}

restart_firewall()
{
    echo 'Restarting firewall...'
    if [ -s "/usr/lib/gargoyle/restart_firewall.sh" ]
    then
        /usr/lib/gargoyle/restart_firewall.sh > /dev/null 2>&1
    else
        /etc/init.d/firewall restart > /dev/null 2>&1
    fi
}

restart_dnsmasq()
{
    if [ "$1" -eq "0" ]
    then
        echo 'Re-reading blocklist'
        killall -HUP dnsmasq
    else
        echo 'Restarting dnsmasq...'
        /etc/init.d/dnsmasq restart
    fi
}

restart_http()
{
    if [ -s "/usr/sbin/uhttpd" ]
    then
        echo 'Restarting uhttpd...'
        /etc/init.d/uhttpd restart
    elif [ -s "/usr/sbin/httpd_gargoyle" ]
    then
        echo 'Restarting httpd_gargoyle...'
        /etc/init.d/httpd_gargoyle restart
    fi
}
restart_cron()
{
    echo 'Restarting cron...'
    /etc/init.d/cron restart > /dev/null 2>&1
}

remove_config()
{
    echo 'Reverting config...'

    # Remove addnhosts
    uci del_list dhcp.@dnsmasq[0].addnhosts=/etc/block.hosts > /dev/null 2>&1 && uci commit

    # Remove cron entry
    sed -i '/adblock/d' /etc/crontabs/root

    # Remove firewall rules
    sed -i '/--to-ports 53/d' /etc/firewall.user
    
    # Remove Tor workarounds
    uci del_list dhcp.@dnsmasq[0].server > /dev/null 2>&1 && uci commit
    uci set dhcp.@dnsmasq[0].noresolv='0' > /dev/null 2>&1 && uci commit

    # Remove proxying
    uci delete uhttpd.main.error_page > /dev/null 2>&1 && uci commit
    uci set httpd_gargoyle.server.page_not_found_file="login.sh" > /dev/null 2>&1 && uci commit
}


toggle()
{
    # Check for cron as test for on/off
    if grep -q "adblock" /etc/crontabs/root
    then
        # Turn off
        echo 'Turning off!'
        remove_config
    else
        # Turn on
        echo 'Turning on!'
        add_config
    fi

    # Restart services
    restart_firewall
    restart_dnsmasq 1
    restart_http
    restart_cron
}

#### END FUNCTIONS ####

### Options parsing ####

case "$1" in
    # Toggle on/off
    "-t")
        toggle
        ;;
    #First time run
    "-f")
        install_dependencies
        add_config
        update_blocklist
        restart_firewall
        restart_dnsmasq 1
        restart_http
        restart_cron
        cleanup
        ;;
    #Reinstall
    "-r")
        remove_config
        install_dependencies
        add_config
        update_blocklist
        restart_firewall
        restart_dnsmasq 1
        restart_http
        restart_cron
        cleanup
        ;;
    #Default updates blocklist only
    *)
        update_blocklist
        restart_dnsmasq 0
        cleanup
        ;;
esac

#### END OPTIONS ####

exit 0

black.list

#/etc/black.list

#add some server that the list doesn't block
example1.block.com

sysupgrade.conf

#/etc/sysupgrade.conf
#This file is a list of files that should be preserved through upgrades
#OPTIONAL!!!!!
/etc/passwd
/etc/shadow
...
...
/etc/adblock.sh #ADD THIS LINE
/etc/white.list #AND THIS ONE
/etc/block.hosts #AND THIS ONE
/etc/black.list #AND THIS ONE

white.list

#/etc/white.list

#Add whitelisted addresses, when appropriate, etc.
a248.e.akamai.net

when I enable the 'serving with transparent pixel' option then it actually redirects to my openWrt router right? But when it does so, doesn't it interfere with the LUCI login page also listening on port 80 and 443?

@frontalot: From what I understand, as long as you take normal precautions, like not listening on external DNS port, then it shouldn't be a problem. But, yeah, I'm not too knowledgeable about this stuff. =-/

Pushed change for that Tor check. Shouldn't have that error now. =-)

@bonanza123: You are right. =-) Although, what happens is that the redirection has the client request a page from the router, say /advertpage.jsp, and the router cannot find it, so it serves the page_error response (which we've overriden). Consequently, there shouldn't be any problems with the normal operation of the Openwrt web server functionality. However, I have seen some issues. Generally, it's somewhat inelegant and creates subtle issues, but works most of the time. =-)

wget -qO- http://www.mvps.org/winhelp2002/hosts.txt| awk -v r="$ENDPOINT_IP4" '{sub(/^0.0.0.0/, r)} $0 ~ "^"r' > /tmp/block.build.list
wget -qO- "http://adaway.org/hosts.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
#wget -qO- http://www.malwaredomainlist.com/hostslist/hosts.txt|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list
#wget -qO- "http://hosts-file.net/.\ad_servers.txt"|awk -v r="$ENDPOINT_IP4" '{sub(/^127.0.0.1/, r)} $0 ~ "^"r' >> /tmp/block.build.list

Thanks for awesome script! Why links 2 and 4 have "", but 1 and 3 are not? It is important?

@dartraiden: Link 2 could be unquoted (I believe), but link 4 has a weird, literal backslash that must be kept uninterpreted. Although, it's disabled by default. =-)

very useful,any plan support adblock plus's filters

@uneed: Nope. Wouldn't work for a variety of reasons: resource constraints, ssl security, etc. We've talked about this before... somewhere up in the comments. Hehe.

I've noticed that some hosts are added to block.hosts as "0.0.0.0domainnamehere.com", with the space missing. Maybe your awk code needs to add a leading space if the hosts lists contains just domains?

But none of the included lists do not have leading ip's. And looking at my own list I don't observe this behaviour at all.

Yeah, like @lantis1008 said, I don't see that behavior @frontalot. There shouldn't be naked domains on the lists. Hmm...no matter, I'll keep an eye out for it. Thanks!

@teffalump: I'd like to try this great script, what is the license? MIT? GPL?

do u create menu position on openwrt to manage this script?

@n-land: Say GPL, and we'll go from there. Thanks! =-)

@illitum: No, I need to create the package to have easier management and options, like the menu position. But I've been somewhat lazy in that regard. =-)

it would be cool if I can define a path where to store the blacklist.host file instead of storing it on my router /etc folder and make my router's flash goes bad more quickly.

I use a adblock on tomato but I can define to store the host files on my RAM, which is cool.

Yeah yeah, I know, I can edit the source if I want, anyway I hope you can work on that.

Is this just work on incoming traffic from the WAN port? I'm using my phone to tethering via USB port and I still see the ads.

@b4hr4m: I'm having some some trouble understanding what the setup is. The phone is an access point? Or? Basically, if there are unencrypted DNS requests through the router, the ads should be blocked since the router will intercept them. Although there are some unusual cases.

@remlei: Sorry for the delay, but yeah, you have a good point. I should add the ability to define the white and blacklist location. I'll do it when I get back from vacation. In the meantime, yeah, just switch the locations manually. =-)

Thanks for this great plugin. However, wireless only blocking isn't working for me. At first I configured the adblock to block all advertisement, which I later changed to wireless only. And while the script is clearly shows the wireless only option is enabled it still blocks everything on my wired network as well. I've tried reinstalling the script with the -r parameter but that didn't work.

I currently only have two lines in my iptables.

im using mwan3 on openwrt, every time the blocklist updates, mwan3 breaks and need to restarted the service manually.

i thing the next step to do is creating a Web Interface For The Ad-blocking like the pi-hole web https://github.com/Anon135813/dnsblocker-webgui - http://jacobsalmela.com/a-web-interface-for-the-ad-blocking-pi-hole/ source https://pi-hole.net/

@norman25 my plugin already does this.

@lantis1008 wow i see it, but i can't use it.
my old router is not supported by gargoyle.

@norman25: I'm probably not going to add more to this script, the other plugins are way better now anyways - I would move to those. The openwrt package should work for your router, no? =-)

@remlei: Hmm... Yeah, no idea really -- updating is just rereading a new list, don't understand why it would break... I don't know mwan3 that well, although I perused the source. Lol. =-/

adaway now redirects to https. script needs to be updated

root@OpenWrt:/# wget http://adaway.org/hosts.txt
--2017-05-04 16:28:17-- http://adaway.org/hosts.txt
Resolving adaway.org... 2400:cb00:2048:1::6818:6959, 104.24.105.89, 104.24.104.89
Connecting to adaway.org|2400:cb00:2048:1::6818:6959|:80... failed: Permission denied.
Connecting to adaway.org|104.24.105.89|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://adaway.org/hosts.txt [following]
--2017-05-04 16:28:18-- https://adaway.org/hosts.txt
Connecting to adaway.org|104.24.105.89|:443... connected.
ERROR: cannot verify adaway.org's certificate, issued by 'CN=COMODO ECC Domain Validation Secure Server CA 2,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB':
Unable to locally verify the issuer's authority.
To connect to adaway.org insecurely, use `--no-check-certificate'.

Hi Guys

I have a problem with OpenVPN + this script. @dcavni It seems I have the same issue so, I'd like to know how you solved it

root@OpenWrt:~# uci get dhcp.@dnsmasq[0].localservice
0
root@OpenWrt:~# uci show | grep openvpn
...
openvpn.sample_server.server='10.8.0.0 255.255.255.0'

OpenVPN works without running this script

Thanks