OONI Docs Documentation

In here you will find information on how to write documentation for docs.ooni.org.

It will include both technical aspects on how the documentation process works, but also some more general principles on why and how we you should document matters of technical nature at OONI.

Principles

1. Documentation is knowledge

Good documentation helps us to share knowledge between members of the team, collaborate effectively and inform our future self about why a particular thing works the way it does.

2. Documentation is community

It also helps us engage with the open source community of volunteers that might want to contribute to the project.

3. Documentation is essential

We should strive to write documentation in a way that it’s not an after-thought, but rather a core piece of the software development lifecycle.

4. Documentation is local

Documentation should live as close as possible to the code that’s it’s documenting. This reduces the friction in keeping it up to date (you don’t have to jump between different repositories) which in turn ensures it’s kept as up to date as possible. It also makes it possible to generate some pieces of the documentation from code.

5. Documentation is pushed not pulled

Documentation that’s generated from code, should follow a “push” approach instead of a “pull” one. This minimizes the likelyhood of it going out of sync and makes it the resposibility of the “pusher” to format and structure it in a way that makes sense.

How documentation is built

For documentation that is cross cutting across projects, you can write it directly in the ooni/docs repo, which will then build and publish it using vercel.

For project specific documentation, you should implement a github action that pushes the documentation to the ooni/docs repo.

Try to keep the pushing logic as simple as possible. Separate the build process from the push process.

Docs build process

We don’t prescribe a specific process, since this can vary wildly depending on the project in question.

What is common though is that the output of the build process shall be a set of simple markdown files, which can then be rendered by the starlight static site generator.

Below is an example of how you might do it through a Makefile.

The build process can be a simple Makefile calling a script, for example:

docs:
  ./scripts/build_docs.sh

clean:
  rm -rf dist/

.PHONY: docs clean

Then scripts/build_docs.sh can look like this:

#!/bin/bash
DOCS_ROOT=dist/docs/
REPO_NAME="ooni/backend"
COMMIT_HASH=$(git rev-parse --short HEAD)

mkdir -p $DOCS_ROOT

strip_title() {
    # Since the title is already present in the frontmatter, we need to remove
    # it to avoid duplicate titles
    local infile="$1"
    cat $infile | awk 'BEGIN{p=1} /^#/{if(p){p=0; next}} {print}'
}

cat <<EOF>$DOCS_ROOT/00-index.md
---
# Do not edit! This file is automatically generated
# to edit go to: https://github.com/$REPO_NAME/edit/master/README.md
# version: $REPO_NAME:$COMMIT_HASH
title: My Documentation Title
description: My Documentation Description
slug: mydocs
---
EOF
strip_title README.md >> $DOCS_ROOT/00-index.md

Key points for the documentation generation:

Always be sure to include in the comment a line saying that the file is automatically generated and include a link to where the source can be edited.
Keep the generating script simple and understandable. Other people will be editing the docs in the future and the build process should apply the minimum amount of transformations to prevent surprises.
Include a version stamp of the hash which the documentation comes from. This is important so we always know which version of docs is published.

Docs pushing process

Once the markdown artifacts have been generated, they need to be pushed to the ooni/docs repo so they can be rendered and published on docs.ooni.org.

These can be generated by means of a github actions task.

Here is an example workflow:

name: build docs
on: push

jobs:
  build_docs:
    runs-on: "ubuntu-20.04"
    steps:
      - name: Check out repository code
        uses: actions/checkout@v3

      - name: Build docs
        run: make docs

      - name: Get current git ref
        id: rev_parse
        run: echo "COMMIT_HASH=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Checkout ooni/docs
        uses: actions/checkout@v2
        with:
          repository: "ooni/docs"
          ssh-key: ${{ secrets.OONI_DOCS_DEPLOYKEY }}
          path: "ooni-docs"

      - name: Update docs
        run: cp -R dist/docs/* ooni-docs/src/content/docs/mydocs/

      - name: Check for conflicting slugs
        run: |
          cat ooni-docs/src/content/docs/mydocs/*.md \
            | grep "^slug:" | awk -F':' '{gsub(/^ +/, "", $2); print $2}' | sort | uniq -c \
            | awk '{if ($1 > 1) { print "duplicate slug for: " $2; exit 1}}'

      - name: Print the lines of the generated docs
        run: wc -l ooni-docs/src/content/docs/mydocs/*

      - name: Commit changes
        # Only push the docs update when we are in master
        if: github.ref == 'refs/heads/master'
        run: |
          cd ooni-docs
          git config --global user.email "github+backend@ooni.org"
          git config --global user.name "OONI Github Actions Bot"
          git add .
          git commit -m "auto: update backend docs to ${{ steps.rev_parse.outputs.COMMIT_HASH }}" || echo "No changes to commit"
          git push origin

Key points for documentation generation:

Add some sanity checks in the push stage to minimize the likelyhood of pushing a broken documentation build.
Include a descriptive message in the commit that pushes the change mentioning where the change came from to faciliate debugging.