FORUMS

[HOSTS][ADBLOCK][ROOT][SCRIPT] DiamondJohn Host Builder Script

3,749 posts
Thanks Meter: 2,437
 
By DiamondJohn, Recognized Contributor on 28th November 2018, 04:36 AM
Post Reply Email Thread
This script creates a host file based on multiple host sources, with 2 levels of fallback. ie Morrors, and last known good download

Code:
Your warranty is now void
I take no responsibility for any adverse effects to your phone or you love life
Do not apply / use this mod if you don;t understand it, or there is any risk for damage top your device or well being

YOU are responsible for what YOU do with YOUR own device, the personal information you share, and the processes you follow in regards to any suggestions, mods, or shortcuts offered here on Xda. Please be aware of the potential security and loss of privacy issues sharing information like this can produce.
Remember, each user is solely responsible for everything YOU do with YOUR device, and for the safe keeping of YOUR private information.

Requirements (for running on android)
root
busybox (I have only tested with stericsons busybox
How to use.
NB: This script can be used on the device OR on Linux
To run from Android
Flash the install ZIP (or for Advanced users, extract the containing zip to your desired location, and change the config in "shared_constants.inc")
Open a terminal and type
Code:
su;
sh /data/local/DiamondJohnHostManager/download_hosts.sh;
copy the /data/local/DiamondJohnHostManager/hosts file to /system/etc/hosts

alternatively, I personally have actually symlinked the hosts file so I do not need to flip my system partition between r/w and r/o every time I wish to update my hosts file. (yes, I understand the risks of having a system file on a write-able partition)
Code:
FINAL_DEST=/data/local/DiamondJohnHostManager/hosts;
LINK=$(ls -l /system/etc/hosts | grep "-> $FINAL_DEST");
if [ "$LINK" == "" ]; then
	$bb mount -o rw,remount /system;
	if [ ! -f $FINAL_DEST ]; then
		$bb echo "127.0.0.1 localhost" > $FINAL_DEST;
	fi;
	$bb ln -sf $FINAL_DEST /system/etc/hosts;
	$bb mount -o ro,remount /system;
fi;
If you run the above successfully, every time you update the script, your live host file will be updated. On my phone it means I don't have to reboot after each update.

To run on Linux
Create the directory
mkdir -p ~/android/DiamondJohnHostManager;
expand the containing zip into the folder
open a terminal
run:
Code:
bash ~/android/DiamondJohnHostManager/download_hosts.sh;
Some of the features:
1. combines multiple hosts files. see/edit the contents of the file: /data/local/DiamondJohnHostManager/hosts-sources.txt
2. You can edit which (see point 1)
3. On linux, I had 110 host sources, and it generated a 98MB hosts file with over 3.4M lines/domains, in <8min
4. It allows "fallback" host sources. ie mirrors in case the original source fails. Separate each mirror with a "|". NB: They don't actually have to be a "mirror", but can simply be an "alternative" if the previous one fails.
5. You can have multiple mirrors/fallbacks
6. If all the mirrors fail for a line item, it will then fallback to the last known successful downloaded host for that line
7. handles multiple formats. Some of which:
Code:
0.0.0.0		www.domainname1.com
127.0.0.1 	www.domainname2.com
www.domainname3.com
8. You can modify the whitelist and blacklist. NB: They are explicit hosts, not regex patterns ie no "*" in the domains.. Multiple files for each are supported; so you can logically group them.
9. if a host is "wrongfully" blocked, you can check which source is the culprit. see the "previous" directory, and note the numbers relate to the host_sources.tct file, and start as the first line being 0 (commented lines are not counted)
10. A log is provided in previous/DiamondJohnHostDownload_log.txt the top section is comma seperated and can be pasted into excel and converted to columns, The following section provides details on each individual download / source
11. If you make ANY changes to the host_sources.txt file, all the previous become invalidated.
Some FYI:
I Run with the default 5 hosts sources in the install zip, and it takes 100-120 seconds to update

To monitor the end results/output, you can also use my other app TeMeFI https://forum.xda-developers.com/and...ation-t3806260
you can find the method under: Various Ungrouped Extras>Hosts File Check
The following script is flashed from within recovery. You could also simply manually expand the zip contents to your desired location and edit the "shared_constants.inc" file appropriately
Download: 201901251428 DiamondJohn host download script installer.zip

XDA:DevDB Information
DiamondJohn Host Builder Script, Tool/Utility for all devices (see above for details)

Contributors
DiamondJohn

Version Information
Status: Stable
Current Stable Version: 201811281200
Stable Release Date: 2018-11-28

Created 2018-11-27
Last Updated 2019-09-17
The Following 3 Users Say Thank You to DiamondJohn For This Useful Post: [ View ] Gift DiamondJohn Ad-Free
28th November 2018, 04:36 AM |#2  
OP Recognized Contributor
Flag Sydney
Thanks Meter: 2,437
 
More
Reserved
28th November 2018, 04:36 AM |#3  
OP Recognized Contributor
Flag Sydney
Thanks Meter: 2,437
 
More
Reserved
25th January 2019, 04:46 AM |#4  
OP Recognized Contributor
Flag Sydney
Thanks Meter: 2,437
 
More
Service New build!
New version. Reasonably minor changes. I test with 100+ hosts (on a Unix server), and one of the new ones wasn't downloading correctly in the previous version.

No need to uninstall previous version, simply7 flash the new one in TWRP. HOWEVER, if you are using a custom "host_source.txt", make a backup of your file first, as it will be overwritten. The file format will remain the same. It would possibly be prudent to also make backups of your blacklist and whitelist files as well, if you have modified them.

See OP for download:
The Following User Says Thank You to DiamondJohn For This Useful Post: [ View ] Gift DiamondJohn Ad-Free
1st July 2019, 09:40 AM |#5  
OP Recognized Contributor
Flag Sydney
Thanks Meter: 2,437
 
More
My personal host sources
This is an FYI, NOT a recommendation.

I have been using the following list of host sources probably since the last release. I have not noticed any ads, even when I forget to do it for a week or more. I have been holding off to see how long it would last, and its now been since 17 Jun 2019 of my last re-fresh/re-build. The following list creates a file of about 2.4MB with 72K lines, and takes < 2 minutes on my phone to build (quad core 2.3GHz 2GB Ram, 2013 S4, ie not a powerhouse by today's standards) I built the list way back based on the smaller file sizes which appeared to have a higher change of hosts; ie they seem to catch the latest/newer bad hosts. By using smaller file sizes, it also takes a short time to rebuild the hosts file.

I do not do any social media, so I also block sites like facebook and twitter in my black lists.

In the attached list, apart from "host_sources.txt", if it does not have "whitelist" in the title, I use it as a blacklist.

NB: The one labeled "stop_youtube_ads.txt" is not really to stop ads in youtube, but its one I found on line that does contain a list of advertising hosts.
Attached Files
File Type: txt host_source.txt - [Click for QR Code] (3.2 KB, 232 views)
File Type: txt crashanalytics.txt - [Click for QR Code] (392 Bytes, 58 views)
File Type: txt facebook_black.txt - [Click for QR Code] (2.7 KB, 46 views)
File Type: txt gpstestads.txt - [Click for QR Code] (135 Bytes, 50 views)
File Type: txt stop_youtube_ads.txt - [Click for QR Code] (3.5 KB, 54 views)
File Type: txt whitelist_various_from_inet.txt - [Click for QR Code] (4.8 KB, 34 views)
File Type: txt whitelist_host_sources.txt - [Click for QR Code] (735 Bytes, 74 views)
The Following User Says Thank You to DiamondJohn For This Useful Post: [ View ] Gift DiamondJohn Ad-Free
18th September 2019, 05:43 AM |#6  
OP Recognized Contributor
Flag Sydney
Thanks Meter: 2,437
 
More
Whoops....

Sorry, I must of accidentally deleted the install zip file from my DropBox account.

Sorry.

Its now updated and available for download in the OP
https://forum.xda-developers.com/and...cript-t3872728
The Following User Says Thank You to DiamondJohn For This Useful Post: [ View ] Gift DiamondJohn Ad-Free
11th February 2020, 11:24 AM |#7  
OP Recognized Contributor
Flag Sydney
Thanks Meter: 2,437
 
More
A way to run the script using a UI
It just occurred to me. I have another app that:
1. Makes it easy with a UI to execute the script.using a menu
Click image for larger version

Name:	Screenshot_TeMeFI_20200211-210851.png
Views:	64
Size:	162.5 KB
ID:	4947843
2. Reports on details of your host file, including how old it is
Click image for larger version

Name:	Screenshot_TeMeFI_20200211-005146.jpg
Views:	68
Size:	197.5 KB
ID:	4947839

3. Move common features into a quick and handy location
(see the first image)

The app is called TeMeFI:
https://forum.xda-developers.com/and...ation-t3806260

To set up the above, you need to:
1. Tell it where to find the script
This is done using a shell command (or you could manually edit it with a text file editor)

echo "/sdcard/TeMeFI/scripts,/data/local/DiamondJohnHostManager" > /sdcard/TeMeFI/.config/USER_SCRIPT_LOCATION;

Its basically a text file with a comma separated list of paths of where script files are placed.
But more on that later...

2. To view extra details regarding your host file, you can find that under the menu:
Various Ungrouped Extras>Host file check

3. Now, to place the lkast run script under the Favourites menu (which is straight ioff root, so you do not need to hunt and peck down the menbu, which will be more cumbersome for the running of the host building)

After you have run a method, if you click the menu button (hamburber on the far left), that opens the fly out menu, with the last icon on the far right being a pencil. If you LONG press the pencil, it will ask if you wish to save it as a favourite. Click Yes.

It will now appear under Favourites menu, however, it will be poorly named. Got to home (the home image in the fly-out menu) and then go into TeMeFI Configuration>Manage Favourites>Definition Details (Re-name)

Once in there, the method should appear at the top of the list (the only one if you just installed). Now, if you long press and select rename, you can Enter something like "Host File Check"

Next, we will take advantage of we did for number 2 above.

Go into Dynamic Lists>Run User Scripts

For me, I have a lot of scripts and paths that are searched, so my list os long. To filer it down, LONG CLICK the X icon on the far left (yes, the close button, but you long click it). This enables you to filter done the list (if yours is also long). Type in "download_host". This should get you down top a single menu item. Now, click on that, and it will rebuild your host file.

After running the script we will save this off as a favourite also, as above (long press the pencil icon)

Now go back into renaming as explained above, and call it something like "Rebuild host file"

So it could look something like the following. Note that my favourite list is pretty long, so you would only have the two items.


To more settings but just for visual sake for my preference:
1. Set the toolbar to be at the bottom
TeMeFI Configuration>Default Startup Config>Toolbar at Bottom

2.Go too a dark green CRT screen
TeMeFI Configuration>Default Startup Config>Results Screen Colours>Green CRT Results Screen


Click image for larger version

Name:	Screenshot_TeMeFI_20200211-211926.jpg
Views:	68
Size:	262.2 KB
ID:	4947853
The Following 3 Users Say Thank You to DiamondJohn For This Useful Post: [ View ] Gift DiamondJohn Ad-Free
Post Reply Subscribe to Thread

Guest Quick Reply (no urls or BBcode)
Message:
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes