Network Neutrality checkers and aggregation

With Respect My Net, we realized that there are quite a lot of Network Neutrality violations out there, especially for mobile users. This wiki page discusses how we could automate the detection of such violations, agregate the results, and produce a nice-looking visualization of these results.

Network neutrality checkers?

They are tools designed to help people seek and report Network Neutrality violations automatically.

Closed outgoing ports

This is one of the most concerning issue (especially for mobile users), and one that is relatively easy to check.

Vinci has set up a responder that allows writing tools easily.

Currently, we've developped a few tools to check this:

• nncheck.sh (written in bash… it was merely written for test purpose, but quite usable)
• a javascript application, which was great but required the user to change the configuration of her browser in a non-obvious way (because browsers themselves block a number of outgoing ports by default)

Setting up a server answering to all ports

Since http://responder.lqdn.fr is not very reliable, you might want to setup your own server that answers on all TCP and UDP ports.

To do this, you need a two IPv4 addresses: `$IP_MAIN` for normal access to the server (SSH, etc), and `$IP_ANYPORT` that will provide a service on all ports. Then use the following iptables rules:

iptables -t nat -A PREROUTING -d $IP_ANYPORT/32 -p tcp -j DNAT --to-destination $IP_MAIN:80
iptables -t nat -A PREROUTING -d $IP_ANYPORT/32 -p udp -j DNAT --to-destination $IP_MAIN:53

Here, we redirect all TCP ports to port 80, and all UDP ports to port 53. You can of course change this.

Other violations

These one might be a bit harder to check, and we haven't implemented them yet:

• transparent proxy
• spoofing SMTP servers (any connection on port 25 is redirected to your ISP's mail server, see this Respect My Net entry in French)
• anything else? …

TODO

• Find a cool name for these tools (instead of plain'old "Network Neutrality checker")
• Android application (since we want to target mobile users)
• Python program, more portable than bash (and could provide a GUI?)
• Setup a reporting & vizualisation tool, eg. a Django app (see below)

Webapp for aggregation

The idea would be that various clients (python program, Android application, …) send their results to a webapp, allowing to visualize graphically the status of Network Neutrality by country/operator (eg. a table of the most widely used outgoing ports, with a score for each pair of port/operator based on the closed/open state of the port as reported by users)

A simple Django webapp seems fitted for this purpose.

Reporting API

Reporting could be done in JSON (simple & widespread)

This is an example of API, feel free to edit/add fields:

appname

a string describing the name of the reporting program (eg "nncheck.sh")

country

a string representing the country in which the test has been conducted → (two-letters code, we need an ISO-**** to make everybody happy)

operator

a string representing the operator being tested → we need a database for this, or it will be unmaintainable (see below)

contract

a string describing the contract → idem

media

a string, can be either "fixed" (for ADSL/cable/fiber connection) or "mobile" (for 3G/4G/…) → what about hotspots like Free WiFi or SFR WiFi?

ports-out

a list of outgoing ports, of the form:

port

an integer between 1 and 65536 representing the port being described

protocol

a string, can be "tcp" or "udp"

state

a string describing the state of the given outgoing port.

Can be either:

• "open": this outgoing port is opened, connection to the responder was successful
• "failure": the connection was unsuccessful (without being more accurate on why). If your language/framework is able to distinguish between a connection attempt that is refused and one that timeouts, please use one of the following
• "closed": the connection was refused (probably by someone between the user and the responder, most probably by the ISP)
• "timeout": the connection attempt has timeouted

to be continued…

Additionally, these fields are generated upon reporting:

• IP address of the reporter → is that necessary? It allows to throttle multiple reports from the same source, to avoid spam, but it brings some privacy issues → mathieui had the good idea of hashing the IP in the db
• date of the report

The issue of "operator" and "contract"

It might not be a good idea to let the user chose freely the operator and the contract (it would lead to a lot of almost-duplicates)

Maybe the applications can ask the webapp for a list of operators/contracts to provide to users (allowing them to create new operators/contracts if need be)

Visualization

Collecting report data is one thing, but the webapp should also be able to display the data in a processed form.

The idea would be to display nice-looking tables, along the lines of the following example (with 100% entries marked as green, and any other result as red).

+-------+-----------------------+---------------+-----------------------+-------+
|       |        France         | Great-Britain |        Germany        |  ...  |
|       |                       |               |                       |       |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
|       | Free  | SFR   |Orange |  O2   |Orange |  ...  |  ...  |  ...  |  ...  |
|       |       |       |       |       |       |       |       |       |       |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
|Overall|  98%  |  72%  |  40%  |       |       |       |       |       |       |
| score |       |       |       |       |       |       |       |       |       |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
|  22   | 100%  |  60%  |  66%  |       |       |       |       |       |       |
|  ssh  |(12/12)| (3/5) | (2/3) |       |       |       |       |       |       |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
|  23   | 100%  | 100%  | 100%  |       |       |       |       |       |       |
|telnet |(12/12)| (5/5) | (3/3) |       |       |       |       |       |       |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
|  25   | 100%  | 100%  |  0%   |       |       |       |       |       |       |
| smtp  |(12/12)| (5/5) | (0/3) |       |       |       |       |       |       |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| 110   |  92%  |  80%  |  33%  |       |       |       |       |       |       |
| pop3  |(11/12)| (4/5) | (1/3) |       |       |       |       |       |       |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| 6667  | 100%  |  20%  |  0%   |       |       |       |       |       |       |
| irc   |(12/12)| (1/5) | (0/3) |       |       |       |       |       |       |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+

Of course, we would need two tables for outgoing ports results: one for landline, one for mobile. This is because the results are likely to be very different.

Maybe we can also find a way to display other results, such as transparent proxy or SMTP spoofing.