From cc06b4432fd81cb21a6f9285425ab5aabd5ee1ad Mon Sep 17 00:00:00 2001 From: Kevin MacMartin Date: Tue, 5 Aug 2014 13:21:27 -0400 Subject: [PATCH] Initial commit with functional behaviour, manpage, README and license --- LICENSE | 21 ++++++ README.md | 43 ++++++++++++ buildhosts | 187 +++++++++++++++++++++++++++++++++++++++++++++++++++ buildhosts.1 | 96 ++++++++++++++++++++++++++ 4 files changed, 347 insertions(+) create mode 100644 LICENSE create mode 100644 README.md create mode 100755 buildhosts create mode 100644 buildhosts.1 diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..4d6f7c9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2014 Kevin M + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..877df88 --- /dev/null +++ b/README.md @@ -0,0 +1,43 @@ +# buildhosts # + +Build and maintain `/etc/hosts` while including lists imported from local and remote hosts list sources. + +## Instructions ## + +**Note:** All instructions should be performed as the **root** user, or using the `sudo` command to run commands with **root** permissions. + +## Enable Remote Sources ## + +1. The first time you run the `set` command, `/etc/hosts` will be moved to `/etc/hosts.core`, and the file `/etc/hosts.sources` will be created using the default sources list. The `/etc/hosts` file will then be generated by combining `/etc/hosts.core` and the resulting contents of the downloaded sources in `/etc/hosts.sources`. +2. You can now open `/etc/hosts.sources` and remove or comment out (by adding `#` in front) any sources you don't wish to use, as well as add new ones. Sources can remote (ie: `http://adaway.org/hosts.txt`) as well as local (ie: `file:///etc/hosts.d/mylist.txt`). +3. If you want to make any changes to the contents of what used to be `/etc/hosts`, you should now make them to `/etc/hosts.core`, and you'll need to rerun the `set` command to have them applied. +4. Once you're done customizing your configuration, run the `set` command again and the `/etc/hosts` file will be regenerated using the current list of sources and your up-to-date `/etc/hosts.core` file. + +## Disable Remote Sources ## + +1. To disable the configured hosts lists, run the `unset` command and `/etc/hosts.core` will be moved back to `/etc/hosts`. +2. The `/etc/hosts.sources` file will continue to exist until it's manually deleted, and it will be used again the next time remote sources are enabled. + +## Notes on /etc/hosts.sources ## + +* On each line of `/etc/hosts.sources`, after `#` is commented/skipped. +* Colour output can be disabled by setting the variable `NOCOL=1`. +* The `$HOSTS_SYSTEM` variable can be set to an alternative location to configure a different file to be used for `/etc/hosts`. +* Similarly, the `$HOSTS_CORE` and `$HOSTS_SOURCES` variables (which default to `$HOSTS_SYSTEM.core` and `$HOSTS_SYSTEM.sources` and will automatically change when `$HOSTS_SYSTEM` is changed), can be set to alternative locations to configure the use of different files. + +## Commands ## + +* `buildhosts set`: generate `/etc/hosts` using `/etc/hosts.core` and the configured sources +* `buildhosts unset`: remove lists imported from hosts list sources and restore `/etc/hosts` +* `buildhosts help`: display the help + +## Credits ## + +Written by Kevin MacMartin: + +* [GitHub Projects](https://github.com/prurigro) +* [Arch Linux AUR Packages](https://aur.archlinux.org/packages/?SeB=m&K=prurigro) + +## License ## + +Released under the [MIT license](http://opensource.org/licenses/MIT). diff --git a/buildhosts b/buildhosts new file mode 100755 index 0000000..41f1c53 --- /dev/null +++ b/buildhosts @@ -0,0 +1,187 @@ +#!/usr/bin/env bash +# +# buildhosts - Build and maintain /etc/hosts while including lists +# imported from local and remote hosts list sources +# +# Instructions (as root): +# +# Enable Remote Sources +# 1. The first time you run the 'set' command, /etc/hosts will be +# moved to /etc/hosts.core, and the file /etc/hosts.sources will +# be created using the default sources list. The /etc/hosts file +# will then be generated by combining /etc/hosts.core and the +# resulting contents of the downloaded sources in +# /etc/hosts.sources. +# 2. You can now open /etc/hosts.sources and remove or comment out +# (by adding # in front) any sources you don't wish to use, as +# well as add new ones. Sources can remote +# (ie: http://adaway.org/hosts.txt) as well as local +# (ie: file:///etc/hosts.d/mylist.txt). +# 3. If you want to make any changes to the contents of what used to +# be /etc/hosts, you should now make them to /etc/hosts.core, and +# you'll need to rerun the 'set' command to have them applied. +# 4. Once you're done customizing your configuration, run the 'set' +# command again and the /etc/hosts file will be regenerated using +# the current list of sources and your up-to-date /etc/hosts.core +# file. +# +# Disable Remote Sources +# 1. To disable the configured hosts lists, run the 'unset' command +# and /etc/hosts.core will be moved back to /etc/hosts. +# 2. The /etc/hosts.sources file will continue to exist until it's +# manually deleted, and it will be used again the next time remote +# sources are enabled. +# +# Notes on /etc/hosts.sources +# * On each line of /etc/hosts.sources, after # is commented/skipped. +# * Colour output can be disabled by setting the variable NOCOL=1. +# * The $HOSTS_SYSTEM variable can be set to an alternative location +# to configure a different file to be used for /etc/hosts. +# * Similarly, the $HOSTS_CORE and $HOSTS_SOURCES variables (which +# default to $HOSTS_SYSTEM.core and $HOSTS_SYSTEM.sources and will +# automatically change if $HOSTS_SYSTEM is changed), can be set to +# alternative locations to configure the use of different files. +# +# by: Kevin MacMartin +# released under the MIT license +# + +# User variables +[[ -z "$HOSTS_SYSTEM" ]] && HOSTS_SYSTEM="/etc/hosts" +[[ -z "$HOSTS_CORE" ]] && HOSTS_CORE="${HOSTS_SYSTEM}.core" +[[ -z "$HOSTS_SOURCES" ]] && HOSTS_SOURCES="${HOSTS_SYSTEM}.sources" + +# Default list of sources (used to generate $HOSTS_SOURCES when it doesn't exist) +DEFAULT_SOURCES=( + 'http://winhelp2002.mvps.org/hosts.txt' + 'http://adaway.org/hosts.txt' + 'http://hosts-file.net/ad_servers.txt' + 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext' +) + +# Set the script name +BUILDHOSTS_SCRIPT="$(basename $0)" + +# Configure colours +if [[ "$NOCOL" -eq 1 ]]; then + HEADINGCOL="|" +else + HEADINGCOL="\e[1;41m" + SUCCESSCOL="\e[1;40m" + FAILCOL="\e[1;40m" + RESETCOL="\e[0m" +fi + +# Help function: display help text +function buildhosts_help(){ + echo -e "${HEADINGCOL} ${BUILDHOSTS_SCRIPT}:${RESETCOL} build and maintain /etc/hosts with local and remote hosts list files\n${HEADINGCOL} ${RESETCOL}" + echo -e "${HEADINGCOL} commands:${RESETCOL}" + echo -e "${HEADINGCOL} ${RESETCOL}${SUCCESSCOL} ${BUILDHOSTS_SCRIPT} set:${RESETCOL} generate /etc/hosts using /etc/hosts.core and the configured sources" + echo -e "${HEADINGCOL} ${RESETCOL}${SUCCESSCOL} ${BUILDHOSTS_SCRIPT} unset:${RESETCOL} remove lists imported from hosts list sources and restore /etc/hosts" + echo -e "${HEADINGCOL} ${RESETCOL}${SUCCESSCOL} ${BUILDHOSTS_SCRIPT} help:${RESETCOL} display this help" + [[ "$1" -eq 1 ]] && exit 0 +} + +# Error function: display error then exit unsuccessfully +function buildhosts_error() { + echo -e "${HEADINGCOL} ${RESETCOL}${FAILCOL} ! ERROR:${RESETCOL}${FAILCOL} ${1}${RESETCOL}" + [[ "$2" -eq 1 ]] && echo -e "${HEADINGCOL} ${RESETCOL}" && buildhosts_help + exit 1 +} + +# The set function to generate or regenerate /etc/hosts using /etc/hosts.core and the list of sources +function buildhosts_set() { + # If $HOSTS_SOURCES doesn't exist, generate one using the default list of sources + if [[ ! -f "$HOSTS_SOURCES" ]]; then + echo -e "${HEADINGCOL} Generating Default Sources:${RESETCOL} ${HOSTS_SOURCES}" + echo '# file:///etc/hosts.d/localhostlist.txt' >> "$HOSTS_SOURCES" + echo '# http://domain.com/remotehostlist.txt' >> "$HOSTS_SOURCES" + for source in "${DEFAULT_SOURCES[@]}"; do + echo "$source" >> "$HOSTS_SOURCES" + done + fi + + # Fail if $HOSTS_SOURCES contains no sources after trimming comments + [[ -n $(sed 's|^\ *#.*$||' "$HOSTS_SOURCES" | tr -d "\n") ]] || buildhosts_error "No sources available in ${HOSTS_SOURCES}" + + # If $HOSTS_CORE doesn't exist and $HOSTS_SYSTEM does, move $HOSTS_SYSTEM to $HOSTS_CORE + if [ -f "$HOSTS_SYSTEM" -a ! -f "$HOSTS_CORE" ]; then + echo -e "${HEADINGCOL} Moving:${RESETCOL} ${HOSTS_SYSTEM} to ${HOSTS_CORE}" + cp "$HOSTS_SYSTEM" "$HOSTS_CORE" + fi + + # Generate the hosts list using the URLs in the $HOSTS_SOURCES + unset TMPHOSTS + while read -r source; do + if [[ -n "$source" ]]; then + echo -e "${HEADINGCOL} Downloading:${RESETCOL} ${source}" + + # Download the the current source into $SRCDATA and fail if the result is empty + SRCDATA=$(curl -C - -s "$source" | tr -d "\r") + [[ -n "$SRCDATA" ]] || buildhosts_error "Could not download list @ ${source}" + + # If this isn't the first source, add a newline at the top + [[ -n "$TMPHOSTS" ]] && SRCDATA=$(echo -e "\n${SRCDATA}") + + # Strip comments, then add the source to the end of $TMPHOSTS + TMPHOSTS=${TMPHOSTS}$(sed 's|\ *#.*$||' <<< "$SRCDATA") + fi + done < <(sed 's|^\ *#.*$||' "$HOSTS_SOURCES") + + # Sort and remove duplicates, change 0.0.0.0 to 127.0.0.1, and remove non-127.0.0.1 redirections + TMPHOSTS=$(sort -u < <(sed 's|\t| |;s|^\ *[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*|127\.0\.0\.1|; + s|^127\.0\.0\.1\ \ *localhost\ *$||; + s|^\ *[^1].*$||' <<< "$TMPHOSTS")) + + # Add the system hosts file to the hosts list and warn if $HOSTS_CORE is missing + echo -e "${HEADINGCOL} Writing:${RESETCOL} ${HOSTS_CORE} and $(wc -l <<< "$TMPHOSTS") host source entries to ${HOSTS_SYSTEM}" + [[ -f "$HOSTS_CORE" ]] \ + && TMPHOSTS=$(cat "$HOSTS_CORE")$(echo -e "\n\n# Generated Host List (${BUILDHOSTS_SCRIPT})")${TMPHOSTS} \ + || echo -e "${HEADINGCOL} ${RESETCOL}${FAILCOL} Warning:${RESETCOL} Core hosts file ${HOSTS_CORE} does not exist" + + # Write $TMPHOSTS to $HOSTS_SYSTEM if it's not empty + [[ -n "$TMPHOSTS" ]] && echo -e "$TMPHOSTS" > "$HOSTS_SYSTEM" + if [[ -n $(cat "$HOSTS_SYSTEM") ]]; then + echo -e "${HEADINGCOL} ${RESETCOL}${SUCCESSCOL} Done!${RESETCOL}" + else + echo -e "${HEADINGCOL} ${RESETCOL}${FAILCOL} Failed...${RESETCOL} (unseting to core hosts file)" + cp "$MAINHOSTS" "$HOSTS_SYSTEM" + buildhosts_error "The generated hosts file @ ${HOSTS_SYSTEM} could not be created" + exit 1 + fi +} + +function buildhosts_unset() { + # Fail if $HOSTS_CORE doesn't exist + [[ -f "$HOSTS_CORE" ]] || buildhosts_error "The core file ${HOSTS_CORE} does not exist, cannot unset" + + # Fail if $HOSTS_CORE is empty when all comments are removed + [[ -n $(sed 's|^\ *#.*$||' "$HOSTS_CORE" | tr -d "\n") ]] || buildhosts_error "The core file ${HOSTS_CORE} contains no information, cannot unset" + + # Move $HOSTS_CORE to $HOSTS_SYSTEM + echo -e "${HEADINGCOL} Moving:${RESETCOL} ${HOSTS_CORE} to ${HOSTS_SYSTEM}" + mv "$HOSTS_CORE" "$HOSTS_SYSTEM" \ + && echo -e "${HEADINGCOL} ${RESETCOL}${SUCCESSCOL} Done!${RESETCOL}" \ + || buildhosts_error "Could not move ${HOSTS_CORE} to ${HOSTS_SYSTEM}" +} + +# Fail if the user isn't root +[[ $EUID -eq 0 ]] || buildhosts_error "You must run ${BUILDHOSTS_SCRIPT} as root" + +# Undo changes to /etc/hosts when the user runs with the disable command +[[ -n "$1" ]] || buildhosts_error "You must run ${BUILDHOSTS_SCRIPT} with a valid argument" 1 + +case "$1" in + set) + buildhosts_set + ;; + unset) + buildhosts_unset + ;; + help) + buildhosts_help 1 + ;; + *) + buildhosts_error "Invalid command" 1 + ;; +esac diff --git a/buildhosts.1 b/buildhosts.1 new file mode 100644 index 0000000..3d6c543 --- /dev/null +++ b/buildhosts.1 @@ -0,0 +1,96 @@ +.if !\n(.g \{\ +. if !\w|\*(lq| \{\ +. ds lq `` +. if \w'\(lq' .ds lq "\(lq +. \} +. if !\w|\*(rq| \{\ +. ds rq '' +. if \w'\(rq' .ds rq "\(rq +. \} +.\} +.ie t .ds Tx \s-1T\v'.4n'\h'-.1667'E\v'-.4n'\h'-.125'X\s0 +. el .ds Tx TeX +.de Id +. ds Yr \\$4 +. substring Yr 0 3 +. ds Mn \\$4 +. substring Mn 5 6 +. ds Dy \\$4 +. substring Dy 8 9 +. \" ISO 8601 date, complete format, extended representation +. ds Dt \\*(Yr-\\*(Mn-\\*(Dy +.. +.TH BUILDHOSTS 1 \*(Dt "buildhosts (git)" "User Commands" +.hy 0 +. +.SH "NAME" +.B buildhosts +\- build and maintain \fB/etc/hosts\fR while including lists imported from local and remote hosts list sources\. +. +.SH "SYNOPSIS" +. +.B buildhosts +.RI [COMMAND] +. +.SH "COMMANDS" +. +.TP +\fBset\fR: generate \fB/etc/hosts\fR using \fB/etc/hosts\.core\fR and the configured sources +.TP +.TP +\fBunset\fR: remove lists imported from hosts list sources and restore \fB/etc/hosts\fR +.TP +.TP +\fBhelp\fR: display the help +. +.SH "INSTRUCTIONS" +\fBNote:\fR All instructions should be performed as the \fBroot\fR user, or using the \fBsudo\fR command to run commands with \fBroot\fR permissions\. +. +.SS "Enable Remote Sources" +.TP +.IP "1." +The first time you run the \fBset\fR command, \fB/etc/hosts\fR will be moved to \fB/etc/hosts\.core\fR, and the file \fB/etc/hosts\.sources\fR will be created using the default sources list\. The \fB/etc/hosts\fR file will then be generated by combining \fB/etc/hosts\.core\fR and the resulting contents of the downloaded sources in \fB/etc/hosts\.sources\fR\. +.TP +.TP +.BR "2." +You can now open \fB/etc/hosts\.sources\fR and remove or comment out (by adding \fB#\fR in front) any sources you don\'t wish to use, as well as add new ones\. Sources can remote (ie: \fBhttp://adaway\.org/hosts\.txt\fR) as well as local (ie: \fBfile:///etc/hosts\.d/mylist\.txt\fR)\. +.TP +.TP +.BR "3." +If you want to make any changes to the contents of what used to be \fB/etc/hosts\fR, you should now make them to \fB/etc/hosts\.core\fR, and you\'ll need to rerun the \fBset\fR command to have them applied\. +.TP +.TP +.BR "4." +Once you\'re done customizing your configuration, run the \fBset\fR command again and the \fB/etc/hosts\fR file will be regenerated using the current list of sources and your up\-to\-date \fB/etc/hosts\.core\fR file\. +.SS "Disable Remote Sources" +.TP +.BR "1." +To disable the configured hosts lists, run the \fBunset\fR command and \fB/etc/hosts\.core\fR will be moved back to \fB/etc/hosts\fR\. +.TP +.TP +.BR "2." +The \fB/etc/hosts\.sources\fR file will continue to exist until it\'s manually deleted, and it will be used again the next time remote sources are enabled\. +.SS "Notes on /etc/hosts\.sources" +.TP +On each line of \fB/etc/hosts\.sources\fR, after \fB#\fR is commented/skipped\. +.TP +.TP +Colour output can be disabled by setting the variable \fBNOCOL=1\fR\. +.TP +.TP +The \fB$HOSTS_SYSTEM\fR variable can be set to an alternative location to configure a different file to be used for \fB/etc/hosts\fR\. +.TP +.TP +Similarly, the \fB$HOSTS_CORE\fR and \fB$HOSTS_SOURCES\fR variables (which default to \fB$HOSTS_SYSTEM\.core\fR and \fB$HOSTS_SYSTEM\.sources\fR and will automatically change when \fB$HOSTS_SYSTEM\fR is changed), can be set to alternative locations to configure the use of different files\. +. +.SH "CREDITS" +Written by Kevin MacMartin: +. +.IP "\(bu" 4 +GitHub Projects \fIhttps://github\.com/prurigro\fR +. +.IP "\(bu" 4 +Arch Linux AUR Packages \fIhttps://aur\.archlinux\.org/packages/?SeB=m&K=prurigro\fR +. +.SH "LICENSE" +Released under the MIT license \fIhttp://opensource\.org/licenses/MIT\fR\.