FORUMS

 View Poll Results: What are you mainly using NetGuard for?

Reducing data usage
 
373 Vote(s)
31.74%
Saving battery
 
264 Vote(s)
22.47%
Increasing privacy
 
591 Vote(s)
50.30%
Blocking ads
 
772 Vote(s)
65.70%

[APP][6.0+] NetGuard - No-root firewall

21,012 posts
Thanks Meter: 44,159
 
By M66B, Recognized Developer on 25th October 2015, 01:33 PM
Post Reply Email Thread
2nd March 2020, 08:03 AM |#11011  
M66B's Avatar
OP Recognized Developer
Thanks Meter: 44,159
 
More
Quote:
Originally Posted by ericJrdn

You're right, i'm wrong. It does not depend on the netguard version.

Android 9 : always blocked
Android 10 + wifi : connexion to ctota3.ospserver.net is seen and blocked
Android 10, 4G : connexion not seen, not blocked.
Connexions are done with UID 1000

I guess this FAQ applies:

https://github.com/M66B/NetGuard/blo...r-content-faq1
2nd March 2020, 02:15 PM |#11012  
Elektron60's Avatar
Senior Member
Thanks Meter: 24
 
More
My OnePulus 6T has lost the AdBlock complete. In the last weeks can a restart solve this problem. But now nothing will help. In the years before it works great. It's a bad world.........

Edit:
I reinstalled Netguard and import an older export. Now it works hopefully for ever and we have a good world.
3rd March 2020, 04:25 AM |#11013  
Junior Member
Thanks Meter: 8
 
More
Quote:
Originally Posted by ericJrdn

You're right, i'm wrong. It does not depend on the netguard version.

Android 9 : always blocked
Android 10 + wifi : connexion to ctota3.ospserver.net is seen and blocked
Android 10, 4G : connexion not seen, not blocked.
Connexions are done with UID 1000

No worries, you experienced different results on Android 10, not sure if using a previous android's NG configuration is recommended. Glad you concluded the issue.

Regards!

P.S. You can use Net Monitor (f-droid) if you're curious about your device connections. On mine that system (1) connection isn't established being blocked by NG, the one at the bottom is required for the wifi to work.
Attached Thumbnails
Click image for larger version

Name:	Screenshot_20200302-224102_Net Monitor.jpg
Views:	270
Size:	236.8 KB
ID:	4963267  
3rd March 2020, 05:26 PM |#11014  
Junior Member
Thanks Meter: 2
 
More
wildcards in filter (RE: FAQ 41)
Hi,

though it is a frequently asked and most probably frequently answered question, I would like to open the following request again:

Please support wildcards in the filter rules!

Why?
  • Performance: The FAQ mentions performance as one reason. I guess that wildcard URLs might even lead to a performance improvement if correctly used by the user. In some cases, I have more than 20 URLs that could be replaced by one single rule. Checking against the list might take more resources than checking against a single wildcard rule. Doesn't it? At which length of the list (what ratio between single URLs vs wildcard URL) is the break-even?
  • URL require no wildcard filter: The argument in the FAQ is that the filter is based on URL not on IP.
    • dynamic ports: Streaming services offer their service under URLs, mostly UDP, with changing high ports. One service may therefore allocated over 65.000 filter rule entries in the config over the years. Examples are: Signal with turn1.whispersystems.org via UDP and the complete high port range.
    • dynamic subdomains: Cloud providers offer dynamic subdomains for different servers that are dynamically asigned to a using service. One example is cloudfront.net. They offer their services under changing URLs in the format like in this URL: d2uq34hri3q.cloudfront.net.
      Some Service Providers have their own server pool: Spotify, for example, tries to access URLs in the format xxxx-accesspoint-x-xxxx.ap.spotify.com. If one is not accessible (as blocked by default by no-root firewall), the Spotify app tries to access the next connection within 30 seconds on a different URL. My access log is full of spotify URLs. I may add them one by one. I would have around 40 URLs or more alone for Spotify in my no-root config.
      Akamai does it the same way with URLs like xxx-xxx-xxx-xx.deploy.static.akamaitechnologies.com as well as Amazon with xxxxxx.compute-1.amazonaws.com, WhatsApp/Facebook with xxxx-chatd-edgex-xxx-xx-xxxx.facebook.com and xx.xx.xxxx.ip4.static.sl-reverse.com.

I understand that this may some fundamental change in the app. But one has to admit that the cases that I listed up above are no edge cases and that these cases are not manageable with filter lists of unique URLs.

Any advice is appreciated.

Best regards,

xdawallah
The Following User Says Thank You to xdawallah For This Useful Post: [ View ] Gift xdawallah Ad-Free
3rd March 2020, 07:18 PM |#11015  
M66B's Avatar
OP Recognized Developer
Thanks Meter: 44,159
 
More
Quote:
Originally Posted by xdawallah

Hi,

though it is a frequently asked and most probably frequently answered question, I would like to open the following request again:

Please support wildcards in the filter rules!

Why?

  • Performance: The FAQ mentions performance as one reason. I guess that wildcard URLs might even lead to a performance improvement if correctly used by the user. In some cases, I have more than 20 URLs that could be replaced by one single rule. Checking against the list might take more resources than checking against a single wildcard rule. Doesn't it? At which length of the list (what ratio between single URLs vs wildcard URL) is the break-even?
  • URL require no wildcard filter: The argument in the FAQ is that the filter is based on URL not on IP.
    • dynamic ports: Streaming services offer their service under URLs, mostly UDP, with changing high ports. One service may therefore allocated over 65.000 filter rule entries in the config over the years. Examples are: Signal with turn1.whispersystems.org via UDP and the complete high port range.
    • dynamic subdomains: Cloud providers offer dynamic subdomains for different servers that are dynamically asigned to a using service. One example is cloudfront.net. They offer their services under changing URLs in the format like in this URL: d2uq34hri3q.cloudfront.net.
      Some Service Providers have their own server pool: Spotify, for example, tries to access URLs in the format xxxx-accesspoint-x-xxxx.ap.spotify.com. If one is not accessible (as blocked by default by no-root firewall), the Spotify app tries to access the next connection within 30 seconds on a different URL. My access log is full of spotify URLs. I may add them one by one. I would have around 40 URLs or more alone for Spotify in my no-root config.
      Akamai does it the same way with URLs like xxx-xxx-xxx-xx.deploy.static.akamaitechnologies.com as well as Amazon with xxxxxx.compute-1.amazonaws.com, WhatsApp/Facebook with xxxx-chatd-edgex-xxx-xx-xxxx.facebook.com and xx.xx.xxxx.ip4.static.sl-reverse.com.

I understand that this may some fundamental change in the app. But one has to admit that the cases that I listed up above are no edge cases and that these cases are not manageable with filter lists of unique URLs.

Any advice is appreciated.

Best regards,

xdawallah

All wildcard rules need to be applied to all connections, which will result in less performance.
There are some optimizations possible at the price of complexity.

Adding wildcards is a significant project and since the support for NetGuard is not significant, I am not willing to undertake this project.
4th March 2020, 06:38 AM |#11016  
lukval's Avatar
Senior Member
Thanks Meter: 12
 
More
I agree with xdawallah!

Maybe someone who is good developer can help us - source code is free, thank you for it M66B!



Sent from my STF-L09 using XDA Labs
7th March 2020, 10:17 PM |#11017  
Member
Thanks Meter: 5
 
More
DNS traffic is blocked when filtering is on and cell data is in use.
I don't really know if this has been discussed. I've searched this thread but have not thoroughly scoured the eleven thousand posts.

EDIT: I've discovered it's only DNS that is not allowed to pass. I've edited this post extensively.

Scenario:
- Andriod 10 (Pixel 3a, fully up to date)
- NetGuard 2.274 (Latest release from github, 2020-03-04, everything unlocked with a donation.)
- NetGuard is active
- Filtering is enabled, using the default hosts list.
- LTE connection is active (Google Fi is my provider)

Expected behavior:
- NetGuard should filter and route traffic over my LTE data connection.

Observed behavior:
- All traffic is blocked.
- DNS traffic is blocked. (I can still ping 1.1.1.1 and 8.8.8.8 just fine, but non-cached hostnames do not resolve.)

Workarounds to recover connectivity:
Choose one:
1. Turn NetGuard off and then back on again. - Result: Works properly over cell data.
1. Disable NetGuard. Traffic is allowed again, and nothing is filtered of course.
OR
2. Disable Filtering. App restrictions are still in place but hostname filtering is not enabled.
OR
3. Reconnect to WiFi. Result: filtering and app restrictions both work properly over WiFi.

Troubleshooting:
- uninstalled the app
- rebooted phone
- installed github release 2.274 (did not bother with the challenge response just yet)
- rebooted phone
- downloaded hosts list
- enabled filtering
- did not yet enable NetGuard
- rebooted phone again
- enabled NetGuard.
- disabled battery optimization for NetGuard as prompted.
- did not change any other settings in NetGuard or in Android.

- I rebooted the phone with WiFi connected, once with and once without NetGuard active.
- in all cases, using cell data with NetGuard active and filtering causes total blockage of DNS service.
- with filtering disabled, cell data is fully working.

- EDIT: I have tried version 2.270 stable from github with the same results as above. (Will attempt the previous stable version as well. I'm completely reconfiguring each time, not reloading a saved settings file.)

QUESTIONS:
Has anyone else run into this?
Are there any solutions?

Thanks much to anyone who has a clue about this!

-------------
Further edits as I investigate
- I switched "Private DNS" in the Android network configuration from "Automatic" to "Off" with no effect.
- In Settings > Advanced Options, I input values for VPN DNS (1.1.1.1 and 8.8.8.8)
---> This allows DNS to function over the cell data network.
... But it also disables my home DNS server for accessing local resources.

SORT OF SOLVED
Since I don't want to treat my very common private IP address as a DNS server on other networks, I've configured an uncommon private address on my home network (think of something like 172.29.35.217). It's not impossible but it is unlikely this address will host a DNS server on other wifi networks I connect to. (In that case I'll usually be using an actual VPN anyway.)

Then for NetGuard, I configure the VPN DNS addresses as such, with my personal obscure DNS server first for when I'm at home, then the public one second.
- VPN DNS: 172.29.35.217
- VPN DNS: 1.1.1.1

Results:
- While at home, everything appears to be working fine. I'm resolving local and internet names.
- While on LTE, lookups occasionally take a little extra time as my "primary" private DNS server is unreachable. Multiple subsequent lookups for uncached names have been pretty speedy.

FEATURE REQUEST:
Allow NetGuard to pick VPN DNS servers based on the connection. This should include the ability to have the defaults, as they are now, as well as being able to set specific DNS servers based on which "saved network" is currently connected. I'd be totally fine with this being a Pro feature.


I hope this post can be found by others who experience the same issues, and furthermore hope it can be helpful as a guide to a workable way around the limitations of NetGuard. If M66B or anyone has other ideas for a more robust way to resolve both private and public names, I'm definitely all ears.

Thanks!
8th March 2020, 07:39 AM |#11018  
M66B's Avatar
OP Recognized Developer
Thanks Meter: 44,159
 
More
Quote:
Originally Posted by Arfyness

I don't really know if this has been discussed. I've searched this thread but have not thoroughly scoured the eleven thousand posts.

EDIT: I've discovered it's only DNS that is not allowed to pass. I've edited this post extensively.

Scenario:
- Andriod 10 (Pixel 3a, fully up to date)
- NetGuard 2.274 (Latest release from github, 2020-03-04, everything unlocked with a donation.)
- NetGuard is active
- Filtering is enabled, using the default hosts list.
- LTE connection is active (Google Fi is my provider)

Expected behavior:
- NetGuard should filter and route traffic over my LTE data connection.

Observed behavior:
- All traffic is blocked.
- DNS traffic is blocked. (I can still ping 1.1.1.1 and 8.8.8.8 just fine, but non-cached hostnames do not resolve.)

Workarounds to recover connectivity:
Choose one:
1. Turn NetGuard off and then back on again. - Result: Works properly over cell data.
1. Disable NetGuard. Traffic is allowed again, and nothing is filtered of course.
OR
2. Disable Filtering. App restrictions are still in place but hostname filtering is not enabled.
OR
3. Reconnect to WiFi. Result: filtering and app restrictions both work properly over WiFi.

Troubleshooting:
- uninstalled the app
- rebooted phone
- installed github release 2.274 (did not bother with the challenge response just yet)
- rebooted phone
- downloaded hosts list
- enabled filtering
- did not yet enable NetGuard
- rebooted phone again
- enabled NetGuard.
- disabled battery optimization for NetGuard as prompted.
- did not change any other settings in NetGuard or in Android.

- I rebooted the phone with WiFi connected, once with and once without NetGuard active.
- in all cases, using cell data with NetGuard active and filtering causes total blockage of DNS service.
- with filtering disabled, cell data is fully working.

- EDIT: I have tried version 2.270 stable from github with the same results as above. (Will attempt the previous stable version as well. I'm completely reconfiguring each time, not reloading a saved settings file.)

QUESTIONS:
Has anyone else run into this?
Are there any solutions?

Thanks much to anyone who has a clue about this!

-------------
Further edits as I investigate
- I switched "Private DNS" in the Android network configuration from "Automatic" to "Off" with no effect.
- In Settings > Advanced Options, I input values for VPN DNS (1.1.1.1 and 8.8.8.8)
---> This allows DNS to function over the cell data network.
... But it also disables my home DNS server for accessing local resources.

SORT OF SOLVED
Since I don't want to treat my very common private IP address as a DNS server on other networks, I've configured an uncommon private address on my home network (think of something like 172.29.35.217). It's not impossible but it is unlikely this address will host a DNS server on other wifi networks I connect to. (In that case I'll usually be using an actual VPN anyway.)

Then for NetGuard, I configure the VPN DNS addresses as such, with my personal obscure DNS server first for when I'm at home, then the public one second.
- VPN DNS: 172.29.35.217
- VPN DNS: 1.1.1.1

Results:
- While at home, everything appears to be working fine. I'm resolving local and internet names.
- While on LTE, lookups occasionally take a little extra time as my "primary" private DNS server is unreachable. Multiple subsequent lookups for uncached names have been pretty speedy.

FEATURE REQUEST:
Allow NetGuard to pick VPN DNS servers based on the connection. This should include the ability to have the defaults, as they are now, as well as being able to set specific DNS servers based on which "saved network" is currently connected. I'd be totally fine with this being a Pro feature.

I hope this post can be found by others who experience the same issues, and furthermore hope it can be helpful as a guide to a workable way around the limitations of NetGuard. If M66B or anyone has other ideas for a more robust way to resolve both private and public names, I'm definitely all ears.

Thanks!

By default NetGuard doesn't touch the DNS server addresses, which means that the Android DNS server addresses will be used which were retrieved via DHCP from the network provider.

Some Android versions do not route DNS traffic correctly when the Android VPN service is being used. The only way to workaround this, is by setting DNS server addresses in the advanced options.
The Following User Says Thank You to M66B For This Useful Post: [ View ]
9th March 2020, 02:50 PM |#11019  
M66B's Avatar
OP Recognized Developer
Thanks Meter: 44,159
 
More
If you want to promote NetGuard, you might want to vote here:

https://android.meta.stackexchange.c...2020/2604#2604

Maybe for FairEmail too.
The Following 2 Users Say Thank You to M66B For This Useful Post: [ View ]
9th March 2020, 09:25 PM |#11020  
Member
Thanks Meter: 5
 
More
Can't figure out how to manually whitelist a hostname.

Instead I must manually download the hosts file, search, and remove the hostname, then apply the host file manually. I have to do this each time I want to update the hosts file. (Also, try going to the URL for the host file, and you don't get the host file, you get redirected to a .md page on github, so you have to use wget or curl, which are unavailable in unrooted Android!!!)

Please observe this example...
s.youtube.com
history.google.com

Problem:
- Both of these resolve to the same address
- Only history.google.com appears in YouTube's access history in NetGuard, and I've marked it as always allow.
- NetGuard is still blocking s.youtube.com but not showing it.
- It does appear in Settings > Advanced Options > Show resolved addresses.
- I believe it's not showing it in the list for YouTube because it's already showing history.google.com which has the same IP address.

1. Feature request: manual global whitelist
2. Feature request: manual per-app whitelist
3. Bug report: logging fails on a blocked hostname when unblocked name resolving to the same IP address is also logged (accessed first?)


If anyone has some other ideas please let me know.

I'd love to be able to run a script that downloads the file (somehow in an Android terminal) then runs a SED command on it to remove that line. Usually it's the second of those that poses the challenge. I'm not used to having an unrooted phone.

EDIT: I've put together a Tasker task.
Downloads the hosts file (action: Net / HTTP Request)
Calls a script (action: Code / Run Shell)
. copies the file to a new one with the date in the name
. calls sed to comment out the offending line
. calls grep to output the edited line (sanity check)
Flashes the script output to let me know it's correctly done.

Then I can just manually import that file in NetGuard.

Still, this is a really long way around just to remove a blocked host.
If anyone is interested in the files, I can export the task and paste the script.

Probably unrelated, I got some other odd behavior in the midst of working on this. Hangouts determined that it was no longer online. Stopping NetGuard put it back online, but then restarting NetGuard took Hangouts back offline. A reboot with NetGuard disabled allowed Hangouts to have its connection, which I only tested after re-enabling Netguard. (I wanted to make sure no DNS was getting cached before testing, even though that didn't seem to matter before.)
The Following User Says Thank You to Arfyness For This Useful Post: [ View ] Gift Arfyness Ad-Free
10th March 2020, 07:46 AM |#11021  
M66B's Avatar
OP Recognized Developer
Thanks Meter: 44,159
 
More
Quote:
Originally Posted by Arfyness

Can't figure out how to manually whitelist a hostname.

Instead I must manually download the hosts file, search, and remove the hostname, then apply the host file manually. I have to do this each time I want to update the hosts file. (Also, try going to the URL for the host file, and you don't get the host file, you get redirected to a .md page on github, so you have to use wget or curl, which are unavailable in unrooted Android!!!)

Please observe this example...
s.youtube.com
history.google.com

Problem:
- Both of these resolve to the same address
- Only history.google.com appears in YouTube's access history in NetGuard, and I've marked it as always allow.
- NetGuard is still blocking s.youtube.com but not showing it.
- It does appear in Settings > Advanced Options > Show resolved addresses.
- I believe it's not showing it in the list for YouTube because it's already showing history.google.com which has the same IP address.

1. Feature request: manual global whitelist
2. Feature request: manual per-app whitelist
3. Bug report: logging fails on a blocked hostname when unblocked name resolving to the same IP address is also logged (accessed first?)

If anyone has some other ideas please let me know.

I'd love to be able to run a script that downloads the file (somehow in an Android terminal) then runs a SED command on it to remove that line. Usually it's the second of those that poses the challenge. I'm not used to having an unrooted phone.

EDIT: I've put together a Tasker task.
Downloads the hosts file (action: Net / HTTP Request)
Calls a script (action: Code / Run Shell)
. copies the file to a new one with the date in the name
. calls sed to comment out the offending line
. calls grep to output the edited line (sanity check)
Flashes the script output to let me know it's correctly done.

Then I can just manually import that file in NetGuard.

Still, this is a really long way around just to remove a blocked host.
If anyone is interested in the files, I can export the task and paste the script.

Probably unrelated, I got some other odd behavior in the midst of working on this. Hangouts determined that it was no longer online. Stopping NetGuard put it back online, but then restarting NetGuard took Hangouts back offline. A reboot with NetGuard disabled allowed Hangouts to have its connection, which I only tested after re-enabling Netguard. (I wanted to make sure no DNS was getting cached before testing, even though that didn't seem to matter before.)

The general support for this project is limited, which means that I will maintain and support this project, but that I won't add new features anymore.

The reported bug is not a bug because the IP address is correctly being blocked because it is listed in the hosts file.

Your hangout problem might be caused by blocking Play services.
Post Reply Subscribe to Thread

Guest Quick Reply (no urls or BBcode)
Message:
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes