Tutorial

Prerequisites

To follow this tutorial, you need Scyros installed and a GitHub personal access token to communicate with the GitHub REST API.

Scyros

Install Scyros following the installation instructions, and confirm that it is on your path:

scyros --version

scyros --help lists the available subcommands, and scyros <subcommand> --help describes all the options for a given subcommand.

GitHub token

To sample repositories, query their metadata or download them, you need a personal access token for the GitHub REST API. In GitHub, go to Settings → Developer settings → Personal access tokens → Fine-grained tokens, generate a token with public repository access only (see GitHub's documentation for details), and save it in a file named tokens.csv with the following format:

token
paste your token here

Do not commit or share this file.

Pipeline overview

Running a study with Scyros involves executing a series of commands that process data in a pipeline. Each step of the pipeline reads the CSV file written by the previous step and writes a new one, so every intermediate stage of the sample can be inspected, reported and rerun independently.

Steps

1. Sample repositories

GitHub assigns repository IDs in roughly chronological order. Scyros samples IDs either at random or in increasing order (--mode linear). Due to the way the REST API works, IDs are retrieved in contiguous chunks: in random mode, each request starts from an ID drawn uniformly at random, with replacement, and returns the IDs of the next 100 public repositories from that point. The --min and --max flags set the range of IDs Scyros queries, and -n sets the number of IDs to collect. In this tutorial, we sample 2,000 repository IDs at random, from the first repository on GitHub up to the default maximum ID, which corresponds to 5 January 2026. To execute this step, run the following command:

scyros ids -t tokens.csv -n 2000 -o ids.csv

The sampled repositories are stored in an output CSV file (here, ids.csv) with the repository's ID, where it comes from (here, always GitHub), its name, a link to it on GitHub, whether it is a fork (1) or not (0), and the request in which it was sampled. Every later step copies the first columns, so each row of each file can be traced back to its repository on GitHub.

repo_id,source,repo_name,url,fork,request_number
412873310,github,alice/chat-server,https://github.com/alice/chat-server,0,0
412873318,github,bob/dotfiles,https://github.com/bob/dotfiles,0,0
412873325,github,carol/spring-petclinic,https://github.com/carol/spring-petclinic,1,0
...

The random seed is fixed (configurable with -s), so running the same command draws the same IDs. In addition, if the process is paused or interrupted, it can be resumed by rerunning the command without losing earlier results or compromising reproducibility.

2. Remove duplicate IDs

Because sampling is with replacement, the same repository can be recorded more than once. To remove duplicate IDs, run the following command:

scyros duplicate_ids -i ids.csv -o unique_ids.csv

The result is stored in unique_ids.csv. With a sample of 2,000 IDs, duplicates are very unlikely.

3. Remove forks

To avoid overrepresenting popular projects, we also remove forks with the following command:

scyros forks -i unique_ids.csv -o ids_without_forks.csv

The result is stored in ids_without_forks.csv. On average, about a third of sampled repositories are forks, so expect this step to shrink the sample by roughly that much.

4. Collect metadata

Run the following command to retrieve each repository's metadata:

scyros metadata -i ids_without_forks.csv -t tokens.csv -o metadata.csv

This metadata includes each repository's primary language, size (in kB), creation, last push and last update dates (as Unix timestamps), popularity counts and licence. This step sends one request per repository and is the slowest of the tutorial. Adding -n 4 sends four requests at the same time; the rows are then written in the order in which the requests complete.

Repositories deleted or made private since sampling cannot be queried. They stay in the output with an error code in the error column, such as E404, and empty metadata. Run scyros explain E404 to see what a code means.

repo_id,source,repo_name,repo_license,latest_commit,url,error,language,created,pushed,updated,fork,disabled,archived,stars,forks,issues,has_issues,watchers_count,subscribers,size
412873310,github,alice/chat-server,MIT License,,https://github.com/alice/chat-server,,Go,1631015230,1712345678,1712345690,0,0,0,4,1,0,1,4,1,612
412873318,github,bob/dotfiles,unknown,,https://github.com/bob/dotfiles,,Shell,1631015302,1633700000,1633700012,0,0,0,0,0,0,1,0,1,18
412873330,github,erin/notes,,,https://github.com/erin/notes,E404,,,,,,,,,,,,,,
...

5. Filter from metadata

Many public repositories are empty, are not software projects, or are tiny projects abandoned shortly after their creation (see Maj et al., The Fault in Our Stars). Since we want to study engineered software projects, we exclude these. Following the authors' recommendations, Scyros provides a command to filter out repositories smaller than a given size (-s, in kB) or younger than a given age (-a, in days). Age is defined as the number of days between the creation date and the last push, so it measures how long a repository was actively developed rather than how long ago it was created. The --non-code flag filters out repositories without a language assigned by GitHub, which most often hold only documentation, data or binaries. The same authors show that selecting repositories by popularity metrics such as stars biases the sample, so we do not filter on them. Here, we keep only repositories that were active for at least 60 days, are at least 50 kB and have a language assigned by GitHub. The filtered dataset is stored in filtered_metadata.csv.

scyros filter_metadata -i metadata.csv -s 50 -a 60 --non-code -o filtered_metadata.csv

filter_metadata runs locally on metadata.csv and sends no requests, so you can compare different thresholds cheaply. In our experience, about 10% of the repositories remain after this step.

6. Query languages

GitHub's primary language understates multi-language repositories: a Kotlin project may also contain Java. This command records the full language breakdown of each repository, in bytes.

scyros languages -i filtered_metadata.csv -t tokens.csv -o languages.csv

repo_id,source,repo_name,repo_license,latest_commit,url,error,languages
412873310,github,alice/chat-server,MIT License,,https://github.com/alice/chat-server,,Go:80312;JavaScript:15420;Dockerfile:310
412873402,github,dan/budget-app,Apache License 2.0,,https://github.com/dan/budget-app,,Kotlin:120344;Java:5120

7. Keep repositories in Java and Go

We now keep only the repositories that contain Java or Go code. To do so, we create a JSON config file that lists the languages included in our study:

{
  "languages": [
    {"name": "java"},
    {"name": "go"}
  ]
}

Language names are case-insensitive, so java matches GitHub's Java. Then run the following command to filter the repositories by language. The output is stored in java_go_repos.csv.

scyros filter_languages -i languages.csv -l config.json -o java_go_repos.csv

Here, the config file is passed with -l because this command only reads its list of languages. Later commands also read its keywords and take the same file with -k.

8. Download the code

The next step is to download the code of the repositories we want to study. The download command fetches each repository and keeps only the Java and Go files that contain at least one concurrency-related keyword. To specify this, we extend the config.json file from step 7 with the file extensions to consider for each language and the keywords that relate to concurrency:

{
  "languages": [
    {
      "name": "java",
      "extensions": [".java"],
      "keywords": ["synchronized", "volatile", "Thread",
                   "ExecutorService", "CompletableFuture"]
    },
    {
      "name": "go",
      "extensions": [".go"],
      "keywords": ["go", "chan", "WaitGroup", "Mutex"]
    }
  ],
  "keywords": ["lock", "unlock"]
}

Each language now has a list of file extensions that Scyros keeps after downloading a repository, and a list of keywords used to filter files. The top-level keywords (lock and unlock) are matched in every language. By default, keywords are matched as whole words and case-insensitively; see the documentation for the flags that change this. Then run:

scyros download -i java_go_repos.csv -t tokens.csv -k config.json -d repos -p projects.csv -f files.csv

Each repository is downloaded from its default branch into the directory given by -d (here, repos/). Files outside the configured extensions, and files containing none of the keywords, are then deleted. The command writes per-repository statistics to the file given by -p (here, projects.csv) and per-file statistics to the file given by -f (here, files.csv). In files.csv, the number of keyword matches is stored in a column named after the relative path of the config file (here, config.json). This filter is deliberately permissive: it also matches keywords in comments and string literals. Manually inspecting a random sample of the matching files helps assess its precision.

Both files record the commit downloaded, in the latest_commit column, so others can retrieve identical code, provided the repository is still public and its history has not been rewritten. Each row of files.csv also gives the path of the file inside its repository and a link to it on GitHub at this commit:

repo_id,source,repo_name,repo_license,latest_commit,local_path,file,url,language,loc,words,config.json
412873310,github,alice/chat-server,MIT License,9f2c1a7e5b0d4c3a8e6f1b2d9c0a7e4f3b5d8c1a,repos/0/412873310/alice-chat-server-9f2c1a7/server/hub.go,server/hub.go,https://github.com/alice/chat-server/blob/9f2c1a7e5b0d4c3a8e6f1b2d9c0a7e4f3b5d8c1a/server/hub.go,go,180,655,9
...

To fix the commits before downloading, for example to share the list of repositories in advance, run scyros latest_commit -i java_go_repos.csv -t tokens.csv -o pinned_repos.csv and give pinned_repos.csv to download, which then fetches these commits.

9. Remove duplicate files

Now that we have repository-level statistics, we turn to files and functions. From this step on, files are treated as a single pool, independent of their repository. As Lopes et al. show in their empirical study on code duplication, GitHub contains substantial cross-repository duplication, such as libraries, generated code and files that users forget to add to .gitignore (e.g. node_modules). Such files end up overrepresented in file- and function-level statistics, so Scyros provides a command to deduplicate files across repositories. With the flag -s bow (bag of words), two files are duplicates when they contain the same tokens with the same frequencies, regardless of order and whitespace. The -s exact option requires byte-identical files, and -s overlap also detects near-miss clones that differ by a few statements, at a much higher cost.

scyros duplicate_files -i files.csv -s bow -o unique_files.csv -m duplicates.csv

The file given by -o (here, unique_files.csv) keeps one representative per group, with a count column giving the group size. The file given by -m (here, duplicates.csv) maps each file to its representative. Feel free to try different deduplication strategies, although at this scale all three produce similar results.

10. Extract the functions

The goal of our example study is to extract functions that use concurrency primitives. For this last step, Scyros provides a command that parses the files with Tree-sitter, extracts their functions and keeps only those whose body contains at least one keyword. Keywords are matched with the same rules as in step 8, but comments and string literals are ignored, so this filter is more precise than the file-level one. Each function is written to its own file, and a summary CSV file (here, functions.csv) records metrics for each function, such as its size, the number of keyword occurrences and the programming constructs it contains (loops, conditionals, function calls), with a link to its first line on GitHub. A file-level summary (function_logs.csv) gives the number of functions found and kept in each file.

scyros parse -i unique_files.csv -k config.json -o functions.csv -l function_logs.csv

Below is an example of a Go function extracted by Scyros. Extracted functions are stored in a .functions directory next to their source file, in a file named after the function's start line and column (here, line 42, column 1).

func broadcastAll(urls []string) []string {
	var wg sync.WaitGroup
	results := make([]string, len(urls))
	for i, url := range urls {
		wg.Add(1)
		go func(i int, url string) {
			defer wg.Done()
			results[i] = fetch(url)
		}(i, url)
	}
	wg.Wait()
	return results
}

Its row in functions.csv links to it on GitHub:

repo_id,source,repo_name,repo_license,latest_commit,local_path,file,function_name,line,column,url,error,local_function_path,language,loc,words,config.json,loop_statements,loop_nestings,if_statements,if_nestings,functions_calls,function_calls_nestings,params,param_kw_match,return_kw_match,parse_error_line,parse_error_column
412873310,github,alice/chat-server,MIT License,9f2c1a7e5b0d4c3a8e6f1b2d9c0a7e4f3b5d8c1a,repos/0/412873310/alice-chat-server-9f2c1a7/server/hub.go,server/hub.go,broadcastAll,42,1,https://github.com/alice/chat-server/blob/9f2c1a7e5b0d4c3a8e6f1b2d9c0a7e4f3b5d8c1a/server/hub.go#L42,,repos/0/412873310/alice-chat-server-9f2c1a7/server/hub.go.functions/42-1,go,13,40,2,1,1,0,0,6,1,1,0,0,,
...

This concludes the tutorial. For more information on the commands, their options and the format of the input and output files, please refer to the documentation. If you find a bug or would like to see a new feature implemented, please open an issue on GitHub.