Documentation

Installation

Scyros runs on macOS, Linux and Windows. The quickest way to install it is the installer script for your system.

macOS and Linux

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/fxpl/scyros/releases/latest/download/scyros-installer.sh | sh

Windows

In PowerShell:

powershell -ExecutionPolicy Bypass -c "irm https://github.com/fxpl/scyros/releases/latest/download/scyros-installer.ps1 | iex"

Cargo

Scyros is published on crates.io and requires Rust 1.95 or later.

cargo install scyros

Nix

With flakes enabled:

nix profile install github:fxpl/scyros

Prebuilt binaries

The releases page provides archives for macOS (Apple silicon and Intel), Linux (x86-64 and ARM64) and Windows (x86-64, also as an MSI installer).

From source

git clone https://github.com/fxpl/scyros.git
cd scyros
cargo build --release

The binary is written to target/release/scyros.

Basics

A study is a chain of subcommands. Each subcommand reads a CSV file with -i and writes a new one with -o, so every intermediate result is a file that can be inspected, edited or shared. The reference lists the subcommands in the order a study usually runs them, as in the tutorial.

A few subcommands differ: ids takes no input file, download writes its logs with -f and -p instead of -o, and explain takes an error code. Without -o, the output file is named after the input file; each subcommand gives its default output name.

Every output starts with the same repository columns and a link to GitHub, and each subcommand copies them from its input. Every row of every file can thus be traced back to its repository, file or function on GitHub.

Resuming and overwriting

Subcommands that query GitHub (ids, metadata, languages, latest_commit, download and pr) resume where they stopped when rerun with the same arguments. With --force, they restart from the beginning instead.

All other subcommands refuse to overwrite an existing output file unless --force is set. Most subcommands also accept -f for --force, except download, where -f sets the file log.

Random seeds

ids samples at random, and metadata, languages, latest_commit, download, parse and pr process their input in a random order. Each uses a fixed default seed, listed in its options table, so a rerun gives the same result. -s sets another seed. In filter_metadata and duplicate_files, -s is a different option.

Global options

Global options go before the subcommand, as in scyros --debug parse ….

Getting help

This page lists every option of each subcommand. scyros <subcommand> --help describes the options of the installed version.

GitHub tokens

Subcommands that query the GitHub API take a CSV file of personal access tokens with -t. The file has a single column, token, with one token per line:

token
ghp_first_token
ghp_second_token

Create a fine-grained token with access to public repositories only. Scyros only reads public data. The tutorial describes the steps.

One token is enough. Tokens from the same account share a single rate limit, so adding more does not speed up collection. Follow GitHub's rate limits and terms of service.

Never commit or share the tokens file.

Subcommands

The table below summarises all subcommands. File names follow the tutorial.

The options tables below omit -i, -o and --force, which are described under Basics.

Sampling

scyros ids

Samples public repositories at random. Each request starts from an ID drawn uniformly at random, with replacement, and returns the next 100 public repositories. Sampling stops once -n repositories are collected, rounded up to a multiple of 100.

scyros ids -t tokens.csv -n 2000 -o ids.csv

Input: none, apart from the tokens file.

Output: one row per repository. No default name: -o is required. If the file exists, new rows are appended to it.

Options

scyros duplicate_ids

Removes rows with a repository ID that already appeared, which sampling with replacement can produce, and prints the number of duplicates.

scyros duplicate_ids -i ids.csv -o unique_ids.csv

Input: any CSV file with a repo_id column.

Output: the same columns as the input. Default name: input.unique.csv.

Options

scyros forks

Removes forks. A fork duplicates the code of its parent repository, so keeping forks biases every later count.

scyros forks -i unique_ids.csv -o ids_without_forks.csv

Input: a CSV file with a fork column holding 1 or 0, such as the output of ids. Rows without a value are reported as errors.

Output: the same columns as the input. Default name: input.non-forks.csv.

Options

Metadata

scyros metadata

Queries the GitHub API for the metadata of each repository, with one request per repository, in random order.

scyros metadata -i ids_without_forks.csv -t tokens.csv -o metadata.csv

Input: a CSV file with repo_id, repo_name and source columns.

Output: one row per repository. Default name: input.metadata.csv.

Options

scyros filter_metadata

Keeps the repositories that meet thresholds on size, age and language. Popularity metrics such as stars and forks are deliberately not available as filters, because they bias the sample towards popular projects. Rows with an error code are always removed. Runs locally and sends no requests.

scyros filter_metadata -i metadata.csv -s 50 -a 60 --non-code -o filtered_metadata.csv

Input: the output of metadata.

Output: the same columns as the input, except that created and pushed are replaced by age, the number of days between them. A negative age, caused by a wrong clock on the machine that pushed, is set to 0. Default name: input.filtered.csv.

Options

Languages

scyros languages

Queries the GitHub API for the language breakdown of each repository, with one request per repository, in random order.

scyros languages -i filtered_metadata.csv -t tokens.csv -o languages.csv

Input: a CSV file with repo_id, repo_name and source columns.

Output: one row per repository. Default name: input.languages.csv.

Options

scyros filter_languages

Keeps the repositories that contain at least one language listed in a keyword file, and removes the rows with an error code. Language names are case-insensitive.

scyros filter_languages -i languages.csv -l config.json -o java_go_repos.csv

Input: the output of languages, and a keyword file.

Output: the same columns as the input. Default name: input.filtered_lang.csv.

Options

Source code

scyros latest_commit

Pins each repository to the latest commit of its default branch, with one request per repository, in random order. download then fetches these commits, and the links of the next subcommands point to them. This step is optional: without a commit, download takes the default branch and records its commit. Pinning the commits first lets you share the list of repositories and commits before downloading.

scyros latest_commit -i java_go_repos.csv -t tokens.csv -o pinned_repos.csv

Input: a CSV file with repo_id and source columns.

Output: the repository columns, with the commit in latest_commit, then url and error. A commit already in the input is replaced. A repository that cannot be queried keeps the commit of the input and gets an error code, such as E011 for an empty default branch. Default name: input.latest_commit.csv.

Options

scyros download

Downloads each repository, or analyses a copy already on disk. It then deletes every file whose extension is not listed in the keyword files, and every file that contains no keyword. Keywords are matched anywhere in a file, including comments and string literals.

Each row is handled on its own, so an input can mix both kinds of repositories:

• A row with a local_path is analysed in this directory, whatever its source. Nothing is downloaded.
• The other rows are downloaded with their repo_id, at the commit in latest_commit. Without a commit, download takes the default branch and writes its commit in the logs. Rows that are not on GitHub get the error E010.

scyros download -i java_go_repos.csv -t tokens.csv -k config.json -d repos -p projects.csv -f files.csv

Input: a CSV file with a source column, and a repo_id or a local_path column, such as the output of filter_languages or latest_commit.

Output: the downloaded files, a file log and a project log. The file log (-f) has one row per file kept.

The project log (-p) has one row per repository. Repositories that GitHub does not provide, such as deleted ones, and archives that cannot be extracted get an error code and empty statistics. Network failures and server errors are retried a few times first.

Options

scyros duplicate_files

Removes duplicate files, keeping one representative per group. Three strategies are available:

exact

Byte-identical files.

bow

Bag of words: the same tokens with the same frequencies, regardless of order and whitespace.

overlap

A large share of tokens in common. Also detects files that differ by a few statements, but is much slower.

exact and bow hash every file, so their cost grows linearly with the number of files. overlap compares files with one another, using the prefix filtering of SourcererCC to limit the number of comparisons. Two files are duplicates under overlap when the tokens they share reach --threshold times the token count of the longer file; a threshold of 1.0 requires the same tokens with the same frequencies. Files are only compared with files of the same language, given by the keyword files passed with -l: files whose extension is in none of them are left uncompared, and without -l every file is compared with every other.

Files larger than 1 GiB are ignored. Empty files form one group under exact and bow, and are each reported as unique under overlap.

scyros duplicate_files -i files.csv -s bow -o unique_files.csv -m duplicates.csv

Input: a CSV file with a column of file paths (local_path by default), such as the file log of download.

Output: the same columns as the input, plus count, the number of files in the group. Default name: input.unique.csv.

The map (-m) has two columns: local_path (or the column given with --header), a file, and original, the representative of its group.

Options

scyros parse

Parses each file with Tree-sitter and keeps the functions whose body contains at least one keyword. Comments and string literals are ignored. Each function is saved in its own file, named after its start line and column (such as 42-1), in a directory named after its source file with .functions appended.

Supported languages: C, C++, C#, Fortran, Go, Java, Python, Rust, Scala and TypeScript. The language of a file is given by its extension, as defined in the keyword files. Files in other languages are skipped with a warning.

scyros parse -i unique_files.csv -k config.json -o functions.csv -l function_logs.csv

Input: a CSV file with local_path (path on disk), file (path inside the repository) and source columns, such as the output of duplicate_files or the file log of download.

Output: one row per function kept. Default name: input.functions.csv.

The log (-l) has one row per file: the repository columns, local_path, file, url, error (E030 for a file skipped with --failures skip-file, E031 for a file too large to be parsed), language, functions (functions found), functions_with_kw (functions kept), one column per keyword file with the number of functions kept that match it, and parse_error_line and parse_error_column (position of the first parse error in the file, empty without parse error).

Options

Pull requests

scyros pr

Collects the open, closed and merged pull requests of each repository. The discussion and review comments of each pull request are saved in a separate CSV file. This subcommand is not part of the pipeline in the tutorial; it accepts any list of repositories.

scyros pr -i java_go_repos.csv -t tokens.csv -d pull_requests -o pull_requests.csv

Input: a CSV file with repo_id, repo_name and source columns.

Output: one row per pull request. A repository whose pull requests cannot be fetched gets a single row, with an error code and empty pull request columns. Default name: input.pulls.csv.

Each comments file holds the body of the pull request, its discussion comments, code review comments and review summaries, with the columns id (comment ID), error (E002 if the comment could not be read), user, user_id, type (body, discussion, code or review), created_at and body (the text).

Options

Repository columns

Every output starts with five repository columns, so that each row says which repository it comes from. Each subcommand copies them from its input and completes them. Unknown values are empty.

ids only writes repo_id, source and repo_name. In the outputs about files, the repository columns are followed by local_path, the path of the file on disk, and file, its path inside the repository. In the outputs about functions, they are also followed by function_name, line and column.

Links

These columns are followed by url, a link to the repository, file, function or pull request on GitHub. Links point to the commit in latest_commit, or to the default branch without a commit. The link of a function points to its first line, such as https://github.com/alice/chat-server/blob/9f2c1a7…/server/hub.go#L42. Links are empty for repositories that are not on GitHub.

Reading other files

Subcommands that read repositories need a source column. With --infer-source, rows without a source are on GitHub if they have a repo_id, and local otherwise.

Every subcommand with an input file accepts --rename OLD=NEW to read the column OLD of the input as the column NEW. It can be given several times. For example, the output of ids in Scyros 0.4 can be read with --rename id=repo_id --rename name=repo_name --infer-source. If a column is missing, the error lists the columns the subcommand reads.

Errors

A repository, file or function that a subcommand cannot process is still written to its output, with an error code in the error column, right after url. This column is empty for the other rows.

• The cells that could not be computed are empty. The values read from the input, such as the name and the link, are kept.
• The next subcommands copy rows with an error without processing them again, and filter_metadata and filter_languages remove them.
• At the end, each subcommand prints how many rows have each code. With --debug, it also prints every error with its details, such as the message of a network error.
• Resuming does not retry the rows already in the output. To retry temporary errors, such as E001 or E503, remove these rows from the output, or rerun with --force.

Errors returned by GitHub have the code of their HTTP status, such as E404. The other codes start with E0.

Other HTTP statuses have the description HTTP error.

scyros explain E404 explains a code, and scyros explain --help lists all of them.

Keyword files

A keyword file is a JSON file that lists the languages of a study and, optionally, the keywords to search for. It is read by filter_languages (-l), download (-k), parse (-k) and duplicate_files with -s overlap (-l).

Format

{
  "languages": [
    {
      "name": "java",
      "extensions": [".java"],
      "keywords": ["synchronized", "volatile", "Thread"]
    },
    {
      "name": "go",
      "extensions": [".go"],
      "keywords": ["go", "chan", "WaitGroup"]
    }
  ],
  "keywords": ["lock", "unlock"]
}

The top-level object has two fields:

languages

Required. A list with one entry per language. Each entry has the following fields:

name

Required. The language name as GitHub reports it, such as Java, Go, C++ or C#. Case-insensitive.

extensions

Optional. File extensions of the language, with or without the leading dot.

keywords

Optional. Keywords that only apply to files of this language.

keywords

Optional. Keywords that apply to every language.

filter_languages compares each name with the languages GitHub reports. parse uses it to select a parser.

Matching rules

• By default, keywords match whole words and are case-insensitive: Thread matches Thread and thread, but not Threads or ThreadLocal. --case-sensitive makes matching case-sensitive.
• Extensions are always case-sensitive: .c does not match main.C.
• download matches keywords anywhere in a file, including comments and string literals. parse ignores comments and string literals.
• A language with no keywords, neither its own nor top-level, keeps all its files and functions. Use this to study all the code of a language.
• With --regex, each keyword is a case-insensitive regular expression and does not need to match a whole word. Use \b for word boundaries, and escape backslashes in JSON: "\\bAtomic\\w+" matches AtomicInteger and AtomicLong.

Multiple keyword files

download and parse accept multiple keyword files after -k. Each file produces its own column of matches, named after its path as given on the command line, so categories of keywords can be counted separately. A file or function is kept if it matches any of the keyword files. All keyword files must assign each extension to the same language.

scyros parse -i unique_files.csv -k threads.json locks.json -o functions.csv

Short form for filter_languages

filter_languages only reads language names, so it also accepts a plain list:

{ "languages": ["java", "go"] }