Network Neutrality checkers and aggregation : Différence entre versions
(Initial revision, explain what it's all about) |
(→Setting up a server answering to all ports) |
||
(8 révisions intermédiaires par 2 utilisateurs non affichées) | |||
Ligne 1 : | Ligne 1 : | ||
+ | With [http://respectmynet.eu/ Respect My Net], we realized that there are quite a lot of Network Neutrality violations out there, especially for mobile users. This wiki page discusses how we could automate the detection of such violations, agregate the results, and produce a nice-looking visualization of these results. | ||
+ | |||
== Network neutrality checkers? == | == Network neutrality checkers? == | ||
Ligne 13 : | Ligne 15 : | ||
* [http://github.com/zorun/nncheck nncheck.sh] (written in bash… it was merely written for test purpose, but quite usable) | * [http://github.com/zorun/nncheck nncheck.sh] (written in bash… it was merely written for test purpose, but quite usable) | ||
* a [http://crommer.fr/portscan/portscan.php javascript application], which was great but required the user to change the configuration of her browser in a non-obvious way (because browsers themselves block a number of outgoing ports by default) | * a [http://crommer.fr/portscan/portscan.php javascript application], which was great but required the user to change the configuration of her browser in a non-obvious way (because browsers themselves block a number of outgoing ports by default) | ||
+ | |||
+ | ==== Setting up a server answering to all ports ==== | ||
+ | |||
+ | Since http://responder.lqdn.fr is not very reliable, you might want to setup your own server that answers on all TCP and UDP ports. | ||
+ | |||
+ | To do this, you need a two IPv4 addresses: `$IP_MAIN` for normal access to the server (SSH, etc), and `$IP_ANYPORT` that will provide a service on all ports. Then use the following iptables rules: | ||
+ | |||
+ | <pre> | ||
+ | iptables -t nat -A PREROUTING -d $IP_ANYPORT/32 -p tcp -j DNAT --to-destination $IP_MAIN:80 | ||
+ | iptables -t nat -A PREROUTING -d $IP_ANYPORT/32 -p udp -j DNAT --to-destination $IP_MAIN:53 | ||
+ | </pre> | ||
+ | |||
+ | Here, we redirect all TCP ports to port 80, and all UDP ports to port 53. You can of course change this. | ||
=== Other violations === | === Other violations === | ||
Ligne 19 : | Ligne 34 : | ||
* transparent proxy | * transparent proxy | ||
− | * spoofing SMTP servers (any connection on port 25 is redirected to your ISP's mail server) | + | * spoofing SMTP servers (any connection on port 25 is redirected to your ISP's mail server, see this [http://respectmynet.eu/view/185 Respect My Net entry] in French) |
− | * anything | + | * anything else? … |
=== TODO === | === TODO === | ||
Ligne 28 : | Ligne 43 : | ||
* Python program, more portable than bash (and could provide a GUI?) | * Python program, more portable than bash (and could provide a GUI?) | ||
* Setup a reporting & vizualisation tool, eg. a Django app (see below) | * Setup a reporting & vizualisation tool, eg. a Django app (see below) | ||
+ | |||
+ | == Webapp for aggregation == | ||
+ | |||
+ | The idea would be that various clients (python program, Android application, …) send their results to a webapp, allowing to visualize graphically the status of Network Neutrality by country/operator (eg. a table of the most widely used outgoing ports, with a score for each pair of port/operator based on the closed/open state of the port as reported by users) | ||
+ | |||
+ | A simple Django webapp seems fitted for this purpose. | ||
+ | |||
+ | === Reporting API === | ||
+ | |||
+ | Reporting could be done in JSON (simple & widespread) | ||
+ | |||
+ | This is an example of API, feel free to edit/add fields: | ||
+ | |||
+ | ;appname | ||
+ | : a string describing the name of the reporting program (eg '''"nncheck.sh"''') | ||
+ | ;country | ||
+ | : a string representing the country in which the test has been conducted → ''(two-letters code, we need an ISO-**** to make everybody happy)'' | ||
+ | ;operator | ||
+ | : a string representing the operator being tested → ''we need a database for this, or it will be unmaintainable (see below)'' | ||
+ | ;contract | ||
+ | : a string describing the contract → ''idem'' | ||
+ | ;media | ||
+ | : a string, can be either '''"fixed"''' (for ADSL/cable/fiber connection) or '''"mobile"''' (for 3G/4G/…) → ''what about hotspots like Free WiFi or SFR WiFi?'' | ||
+ | ;ports-out | ||
+ | : a list of outgoing ports, of the form: | ||
+ | :;port | ||
+ | :: an integer between 1 and 65536 representing the port being described | ||
+ | :;protocol | ||
+ | :: a string, can be '''"tcp"''' or '''"udp"''' | ||
+ | :;state | ||
+ | :: a string describing the state of the given outgoing port. | ||
+ | :: Can be either: | ||
+ | ::* '''"open"''': this outgoing port is opened, connection to the responder was successful | ||
+ | ::* '''"failure"''': the connection was unsuccessful (without being more accurate on why). If your language/framework is able to distinguish between a connection attempt that is refused and one that timeouts, please use one of the following | ||
+ | ::* '''"closed"''': the connection was refused (probably by someone between the user and the responder, most probably by the ISP) | ||
+ | ::* '''"timeout"''': the connection attempt has timeouted | ||
+ | |||
+ | to be continued… | ||
+ | |||
+ | |||
+ | Additionally, these fields are generated upon reporting: | ||
+ | |||
+ | * '''IP address''' of the reporter → ''is that necessary? It allows to throttle multiple reports from the same source, to avoid spam, but it brings some privacy issues'' → ''mathieui had the good idea of hashing the IP in the db'' | ||
+ | * '''date''' of the report | ||
+ | |||
+ | === The issue of "operator" and "contract" === | ||
+ | |||
+ | It might not be a good idea to let the user chose freely the operator and the contract (it would lead to a lot of almost-duplicates) | ||
+ | |||
+ | Maybe the applications can ask the webapp for a list of operators/contracts to provide to users (allowing them to create new operators/contracts if need be) | ||
+ | |||
+ | === Visualization === | ||
+ | |||
+ | Collecting report data is one thing, but the webapp should also be able to display the data in a processed form. | ||
+ | |||
+ | The idea would be to display nice-looking tables, along the lines of the following example (with '''100%''' entries marked as green, and any other result as red). | ||
+ | |||
+ | <pre> | ||
+ | +-------+-----------------------+---------------+-----------------------+-------+ | ||
+ | | | France | Great-Britain | Germany | ... | | ||
+ | | | | | | | | ||
+ | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | ||
+ | | | Free | SFR |Orange | O2 |Orange | ... | ... | ... | ... | | ||
+ | | | | | | | | | | | | | ||
+ | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | ||
+ | |Overall| 98% | 72% | 40% | | | | | | | | ||
+ | | score | | | | | | | | | | | ||
+ | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | ||
+ | | 22 | 100% | 60% | 66% | | | | | | | | ||
+ | | ssh |(12/12)| (3/5) | (2/3) | | | | | | | | ||
+ | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | ||
+ | | 23 | 100% | 100% | 100% | | | | | | | | ||
+ | |telnet |(12/12)| (5/5) | (3/3) | | | | | | | | ||
+ | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | ||
+ | | 25 | 100% | 100% | 0% | | | | | | | | ||
+ | | smtp |(12/12)| (5/5) | (0/3) | | | | | | | | ||
+ | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | ||
+ | | 110 | 92% | 80% | 33% | | | | | | | | ||
+ | | pop3 |(11/12)| (4/5) | (1/3) | | | | | | | | ||
+ | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | ||
+ | | 6667 | 100% | 20% | 0% | | | | | | | | ||
+ | | irc |(12/12)| (1/5) | (0/3) | | | | | | | | ||
+ | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | ||
+ | </pre> | ||
+ | |||
+ | Of course, we would need two tables for outgoing ports results: one for ''landline'', one for ''mobile''. This is because the results are likely to be very different. | ||
+ | |||
+ | Maybe we can also find a way to display other results, such as transparent proxy or SMTP spoofing. | ||
+ | |||
+ | [[Category:Net Neutrality]] |
Version actuelle datée du 28 août 2014 à 14:52
With Respect My Net, we realized that there are quite a lot of Network Neutrality violations out there, especially for mobile users. This wiki page discusses how we could automate the detection of such violations, agregate the results, and produce a nice-looking visualization of these results.
Sommaire
Network neutrality checkers?[modifier]
They are tools designed to help people seek and report Network Neutrality violations automatically.
Closed outgoing ports[modifier]
This is one of the most concerning issue (especially for mobile users), and one that is relatively easy to check.
Vinci has set up a responder that allows writing tools easily.
Currently, we've developped a few tools to check this:
- nncheck.sh (written in bash… it was merely written for test purpose, but quite usable)
- a javascript application, which was great but required the user to change the configuration of her browser in a non-obvious way (because browsers themselves block a number of outgoing ports by default)
Setting up a server answering to all ports[modifier]
Since http://responder.lqdn.fr is not very reliable, you might want to setup your own server that answers on all TCP and UDP ports.
To do this, you need a two IPv4 addresses: `$IP_MAIN` for normal access to the server (SSH, etc), and `$IP_ANYPORT` that will provide a service on all ports. Then use the following iptables rules:
iptables -t nat -A PREROUTING -d $IP_ANYPORT/32 -p tcp -j DNAT --to-destination $IP_MAIN:80 iptables -t nat -A PREROUTING -d $IP_ANYPORT/32 -p udp -j DNAT --to-destination $IP_MAIN:53
Here, we redirect all TCP ports to port 80, and all UDP ports to port 53. You can of course change this.
Other violations[modifier]
These one might be a bit harder to check, and we haven't implemented them yet:
- transparent proxy
- spoofing SMTP servers (any connection on port 25 is redirected to your ISP's mail server, see this Respect My Net entry in French)
- anything else? …
TODO[modifier]
- Find a cool name for these tools (instead of plain'old "Network Neutrality checker")
- Android application (since we want to target mobile users)
- Python program, more portable than bash (and could provide a GUI?)
- Setup a reporting & vizualisation tool, eg. a Django app (see below)
Webapp for aggregation[modifier]
The idea would be that various clients (python program, Android application, …) send their results to a webapp, allowing to visualize graphically the status of Network Neutrality by country/operator (eg. a table of the most widely used outgoing ports, with a score for each pair of port/operator based on the closed/open state of the port as reported by users)
A simple Django webapp seems fitted for this purpose.
Reporting API[modifier]
Reporting could be done in JSON (simple & widespread)
This is an example of API, feel free to edit/add fields:
- appname
- a string describing the name of the reporting program (eg "nncheck.sh")
- country
- a string representing the country in which the test has been conducted → (two-letters code, we need an ISO-**** to make everybody happy)
- operator
- a string representing the operator being tested → we need a database for this, or it will be unmaintainable (see below)
- contract
- a string describing the contract → idem
- media
- a string, can be either "fixed" (for ADSL/cable/fiber connection) or "mobile" (for 3G/4G/…) → what about hotspots like Free WiFi or SFR WiFi?
- ports-out
- a list of outgoing ports, of the form:
- port
- an integer between 1 and 65536 representing the port being described
- protocol
- a string, can be "tcp" or "udp"
- state
- a string describing the state of the given outgoing port.
- Can be either:
- "open": this outgoing port is opened, connection to the responder was successful
- "failure": the connection was unsuccessful (without being more accurate on why). If your language/framework is able to distinguish between a connection attempt that is refused and one that timeouts, please use one of the following
- "closed": the connection was refused (probably by someone between the user and the responder, most probably by the ISP)
- "timeout": the connection attempt has timeouted
to be continued…
Additionally, these fields are generated upon reporting:
- IP address of the reporter → is that necessary? It allows to throttle multiple reports from the same source, to avoid spam, but it brings some privacy issues → mathieui had the good idea of hashing the IP in the db
- date of the report
The issue of "operator" and "contract"[modifier]
It might not be a good idea to let the user chose freely the operator and the contract (it would lead to a lot of almost-duplicates)
Maybe the applications can ask the webapp for a list of operators/contracts to provide to users (allowing them to create new operators/contracts if need be)
Visualization[modifier]
Collecting report data is one thing, but the webapp should also be able to display the data in a processed form.
The idea would be to display nice-looking tables, along the lines of the following example (with 100% entries marked as green, and any other result as red).
+-------+-----------------------+---------------+-----------------------+-------+ | | France | Great-Britain | Germany | ... | | | | | | | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | | Free | SFR |Orange | O2 |Orange | ... | ... | ... | ... | | | | | | | | | | | | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ |Overall| 98% | 72% | 40% | | | | | | | | score | | | | | | | | | | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | 22 | 100% | 60% | 66% | | | | | | | | ssh |(12/12)| (3/5) | (2/3) | | | | | | | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | 23 | 100% | 100% | 100% | | | | | | | |telnet |(12/12)| (5/5) | (3/3) | | | | | | | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | 25 | 100% | 100% | 0% | | | | | | | | smtp |(12/12)| (5/5) | (0/3) | | | | | | | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | 110 | 92% | 80% | 33% | | | | | | | | pop3 |(11/12)| (4/5) | (1/3) | | | | | | | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+ | 6667 | 100% | 20% | 0% | | | | | | | | irc |(12/12)| (1/5) | (0/3) | | | | | | | +-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
Of course, we would need two tables for outgoing ports results: one for landline, one for mobile. This is because the results are likely to be very different.
Maybe we can also find a way to display other results, such as transparent proxy or SMTP spoofing.