Table of Contents

About

A python bot, that uses Selenium, in order to automatically access the Router/Modem’s web-based setup page and get DSL Link info. It uses Plotly to create an interactive Plot as an HTML web page.

It also supports the export of the data to an InfluxDB Bucket for further monitoring and visualizing.

Currently gets the following info:

  • CRC Errors
  • FEC Errors
  • Showtime_start

Support for the following Routers/Modems:

  • ZTE Nova Routers (H267N, H288A, …)

The need behind this project, was to monitor my DSL Link information over time, to correlate with various internet outages or sudden internet speed issues.

2024 Update: Right now this project is unused, since I have upgraded my internet connection to Fiber and all of these issues are gone 😄.

GitHub GitHub repository: https://github.com/thanoskoutr/DSL-Info-Bot

Technologies

The main technologies used for this project (language, tools, libraries, database):

python logo pandas logo plotly logo selenium logo influxdb logo

Installation

Clone Repository

Clone the repo:

git clone https://github.com/thanoskoutr/DSL-Info-Bot.git
Copy

Change directory:

cd DSL-Info-Bot
Copy

Python Virtual Environment

Install venv

On Debian/Ubuntu systems, you need to install the python3-venv package using the following command:

sudo apt-get install python3-venv
Copy

Create virtual environment

Create a virtual environment on the top directory of the project:

python3 -m venv env
Copy

Activate the virtual environment

source env/bin/activate
Copy

Install required dependencies

python3 -m pip install -r requirements.txt
# or
pip3 install -r requirements.txt
Copy

Minimum Python version tested: Python 3.6.9 Maximum Python version tested: Python 3.8.5

Dependencies

You can install manually, without a virtual environment, the project dependencies:

pip install selenium
Copy
pip install pandas
Copy
pip install plotly
Copy

Download Web-Driver for Selenium

Selenium requires a driver to interface with the chosen browser. For this project the Chrome and Firefox driver is supported, used in headless mode in order to not require a GUI.

Install Chromium

In order to use the chromium driver, chromium or chrome needs to be installed. If you have already a chrome installation in your system, there is nothing to be done. Else, install a chrome version.

For Ubuntu 20.04, use wget to download the latest Google Chrome .deb package:

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
Copy

Install Google Chrome package:

sudo apt install ./google-chrome-stable_current_amd64.deb
Copy

Download Chrome Driver

Download the chrome browser driver, based on the installed chromium version in your system from this link

Check Chrome version:

google-chrome --version
Copy

For the Chrome version 89 driver, for Linux 64-bit, download with:

wget https://chromedriver.storage.googleapis.com/89.0.4389.23/chromedriver_linux64.zip
Copy

Extract the driver, using unzip:

unzip chromedriver_linux64.zip
Copy

Install Mozilla Firefox

In order to use the firefox driver, Firefox needs to be installed. If you have already a Firefox installation in your system, there is nothing to be done. Else, install a Firefox version from here or:

For Ubuntu/Debian Distros

sudo apt install firefox
Copy

For Arch/Arch based Distros

sudo pacman -S firefox
Copy

Download Firefox Driver

Download the Firefox browser driver, based on the installed Firefox version in your system from this link

Check Firefox version:

firefox --version
Copy

For the latest Firefox version driver, for Linux 64-bit, download with:

wget https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-linux64.tar.gz
Copy

Extract the driver, using tar:

tar -zxvf geckodriver-v0.29.0-linux64.tar.gz
Copy

Add driver to PATH

In order for selenium to use the binary driver, it has to be in the PATH environmental variable. The preferred place for local binaries is to place the driver in the /usr/local/bin directory:

# For Chrome Driver
# Run from the location where your binary is downloaded and extracted
sudo mv chromedriver /usr/local/bin
# For Firefox Driver
# Run from the location where your binary is downloaded and extracted
sudo mv geckodriver /usr/local/bin
Copy

Selenium Guides - Documentation

Add Environmental Variables

Create the .env file from .env-test:

cp .env-test .env
Copy

Fill in the environmental variables needed:

  • Router’s web-based setup page (like http://192.168.1.1/)
  • Router’s Username
  • Router’s Password
ROUTER_PAGE=http://192.168.1.1/
ROUTER_USERNAME=username
ROUTER_PASSWORD=password
Copy

Run Program

From the project’s top directory, run:

python3 main.py
Copy
  • A file named dsl_info.csv should be created in the directory that contains the fetched info.
  • A file named dsl_info_headers.csv should be created in the directory that contains the info headers.
  • Firefox is the default browser, running without GUI.

Run Program with arguments

From the project’s top directory, run with help option to see all available options:

$ python3 main.py -h

usage: main.py [-h] [-d {firefox,chrome}] [--headless {True,False}] [-t TIMEOUT] [-f FILE]

DSL-Info-Bot - A python bot, that uses Selenium, in order to automatically access the Router/Modem's web-based setup page and get DSL Link
info.

optional arguments:
  -h, --help            show this help message and exit
  -d {firefox,chrome}, --driver {firefox,chrome}
                        The browser driver to use.
  --headless {True,False}
                        Run without a GUI.
  -t TIMEOUT, --timeout TIMEOUT
                        The maximum time to wait for page loading.
  -f FILE, --file FILE  The file name of the csv where the data will be saved.

Run Plot Script

From the project’s top directory, run:

python3 create_plot.py
Copy
  • Reads the .csv files with the headers and the data
  • Creates a plot.html file that can be viewed on a browser.

Run Plot Script with arguments

From the project’s top directory, run with help option to see all available options:

$ python3 create_plot.py -h

usage: create_plot.py [-h] [-c CSV] [-d CSV_HEADERS]

A plot script in order to visualize the data in the csv file.

optional arguments:
  -h, --help            show this help message and exit
  -c CSV, --csv CSV     The file name of the csv with the data.
  -d CSV_HEADERS, --csv_headers CSV_HEADERS
                        The file name of the csv with the data headers.

Serve Plot - View on Browser

If you are in a headless environment you can serve the .html files using the http.server python module.

By running this command from the project’s top directory, you’ll be able to access the files in your directory through your browser at localhost:8420:

python3 -m http.server 8420
Copy

There are more elegant (and safe) ways to do this, but this is a quick and dirty solution.

Create a Cron Job

In order to run the script after a time interval on a linux machine:

Open crontab:

crontab -e
Copy

In order to run the script every 5 minutes, change the path to the repo accordingly:

*/5 * * * * /usr/bin/env bash -c 'export PATH="/usr/local/bin:$PATH" && source /path/to/DSL-Info-Bot/env/bin/activate && python3 /path/to/DSL-Info-Bot/main.py'
Copy

In order to run the script, update the plot every 5 minutes and serve the html plot, change the path to the repo accordingly:

*/5 * * * * /usr/bin/env bash -c 'export PATH="/usr/local/bin:$PATH" && source /path/to/DSL-Info-Bot/env/bin/activate && python3 /path/to/DSL-Info-Bot/main.py && python3 /path/to/DSL-Info-Bot/create_plot.py'
@reboot cd /path/to/DSL-Info-Bot && python3 -m http.server 8420 > /dev/null 2>&1
Copy

We need to use the bash shell in order to execute the source command and we need to export the PATH variable in order for cron to be able to find the driver.

The dsl-info.csv file should be created or updated in the repo directory.

InfluxDB

In order to add dsl_info.csv to a InfluxDB Bucket, we have to convert theme using the convert_times.py script that converts the showtime_start_value and the current_date values to duration and timestamp data formats that InfluxDB can process.

To do that, create a new dsl_info.csv file:

mv dsl_info.csv dsl_info_influxdb.csv
Copy

And convert the fields on the new dsl_info_influxdb.csv file:

python3 convert_times.py dsl_info_influxdb.csv
Copy

Now we can add them to a DSL-Info bucket using the following commands, by passing only one measurement at each time:

influx write dryrun -b DSL-Info -f dsl_info_influxdb.csv \
--header "#constant measurement,errors" \
--header "#datatype long,ignored,ignored,ignored,ignored,dateTime:RFC3339" \
--header "errors_crc_up,errors_crc_down,errors_fec_up,errors_fec_down,showtime_start_value,current_date"

influx write dryrun -b DSL-Info -f dsl_info_influxdb.csv \
--header "#constant measurement,errors" \
--header "#datatype ignored,long,ignored,ignored,ignored,dateTime:RFC3339" \
--header "errors_crc_up,errors_crc_down,errors_fec_up,errors_fec_down,showtime_start_value,current_date"

influx write dryrun -b DSL-Info -f dsl_info_influxdb.csv \
--header "#constant measurement,errors" \
--header "#datatype ignored,ignored,long,ignored,ignored,dateTime:RFC3339" \
--header "errors_crc_up,errors_crc_down,errors_fec_up,errors_fec_down,showtime_start_value,current_date"

influx write dryrun -b DSL-Info -f dsl_info_influxdb.csv \
--header "#constant measurement,errors" \
--header "#datatype ignored,ignored,ignored,long,ignored,dateTime:RFC3339" \
--header "errors_crc_up,errors_crc_down,errors_fec_up,errors_fec_down,showtime_start_value,current_date"

influx write -b DSL-Info -f dsl_info_influxdb.csv \
--header "#constant measurement,errors" \
--header "#datatype ignored,ignored,ignored,ignored,duration,dateTime:RFC3339" \
--header "errors_crc_up,errors_crc_down,errors_fec_up,errors_fec_down,showtime_start_value,current_date"
Copy