The project is in Work in Progress (WIP) state. A lot of functionality is still missing.

About

gomirror is a CLI mirror ranker that its main goal is to rank or find the best mirror for a Linux distribution. Although it can be extended to be used as a more general ranker for mirrors/servers.

It is build with usability, extensibility, and performance in mind. It can quickly make thousands of requests with the power of Goroutines, to all servers in parallel. Its inputs and outputs are JSON-modeled, to allow for defined but extensible input data and results. As a CLI tool it will display any results in the standard output by default, but with the correct flag it can also output them in various formats. You can find the expected inputs/outputs for the various supported formats in the inputs/in.template.* and outputs/out.template.* directories respectively.

Its main usage is to get the available mirrors for the desired distribution (either through official website or from user input) and output the mirrors ranked or the best mirror based on the server HTTP responses.

Note: This is a hobby project under development. Do not expect any stable functionalities until it has some features implement and some major coding redesigns. There are still a lot of features missing that will make this tool a robust tool that can be used for mirror ranking, under real circumstances.

GitHub GitHub repository: https://github.com/thanoskoutr/gomirror

Techonologies

The main technologies used for this project (language, distributions):

golang logo heroku logo heroku logo heroku logo

Supported Distributions

The supported distributions for now are:

  • Arch (1, 2)
  • Debian (1, 2)
  • Ubuntu (1, 2)

Supported Inputs and Outputs

The supported inputs for the mirrors are:

  • http (web page with mirrors: will be either HTML parsed or JSON parsed)
  • txt (a plain text files which has a mirror URL in every line)
  • json (a JSON file in the expected format - with the mirror URLs and other relevant info)

The supported outputs for the mirrors are:

  • stdout (print results in the terminal - default mode)
  • json (print results in the expected JSON format - just like the input JSON but with the statistics field)
  • csv (same as JSON but in CSV format)
  • txt (print results in the expected mirror format for each distribution - not currently implemented)

Program Internals

This is a brief description of the actions made during the program’s execution:

  • Get mirrors list:
    • If, explicit file input, read from file (TXT, JSON)
    • Else, read mirrors online (parse HTML or JSON page)
      • If, error reading (mirror site, or individual links) fallback to ready-made mirrors lists in source-code (Go maps) - not implemented.
  • Create requests:
    • If country not supplied, find country automatically
    • Find round of requests (currently only 1)
    • Make 1 request per host (filter by host)
    • Generate requests statistics (HTTP response time)
  • Find best distribution mirror server:
    • Based on:
      • http (time)
      • ping (time) - not implemented
      • traceroute (hops) - not implemented
      • location (country) - not implemented
  • Rank distribution mirror servers:
    • Sort them
    • Show Statistics:
      • http (time)
      • ping (time) - not implemented
      • traceroute (hops) - not implemented
  • Produce output:
    • Export in multiple formats: stdout, json, csv, txt

Installation

To run the tool, first we need to clone the repository and build it:

$ git clone https://github.com/thanoskoutr/gomirror.git
$ cd gomirror
$ make

To have the executable available everywhere in the system we can copy it to a location that is available under the $PATH environmental variable (e.g. /usr/local/bin):

$ sudo cp ./gomirror /usr/local/bin

Usage

We can run the tool with the help flag (-h) to view all available options:

$ ./gomirror -h

Testing

To run the tests for all sub-packages (recursively):

$ go test -v ./...

Other similar Tools

Of-course the inspiration for this project was the existence of other similar tools. The motive for this project was that most of the available tools are working for one distribution only, and none of them was written in Go :).

These are the available tool that I want to research further for their functionalities, inputs/flags, outputs results, etc:

Future features

Because this is a unfinished project there are a LOT of things that I want to add/change. Here are some of them categorized:

  • Use 90th percentile stats, along with Average stats
  • Add Heuristics for best mirror decision (speed, locallity, ranking)
  • Support other mirror protocols: (e.g. ftp, rsync)
  • Rerort best mirror and others in sorted order based on internal ranking
  • Update linux distribution mirror on the fly
  • Add support for more distros

To see the full list of the desired improvements and todo items, check the README of the GitHub repository.