Ghaf Infra

This repository declaratively defines the NixOS configuration for the Ghaf CI/CD infrastructure. All host configurations (including secrets) are version-controlled here.

The infrastructure includes:

• Jenkins CI environments (prod, dev, dbg, release) hosted at Hetzner
• Multi-architecture remote builders for x86_64 and aarch64
• On-prem test agents with connected hardware devices
• Supporting services: monitoring, logging, authentication, Nebula overlay network, NetHSM hardware signing, and an OCI container registry
• Secrets management via sops-nix (see architecture overview)

This document assumes you have nix with flakes support.

Clone this repository:

❯ git clone https://github.com/tiiuae/ghaf-infra.git
❯ cd ghaf-infra

Enter the development shell:

❯ nix develop

All commands referenced in the documentation are executed inside the nix-shell.

The dev shell includes pre-commit hooks that run automatically on git commit. See nix/git-hooks.nix for the full list. To run the hooks manually:

nix fmt

This includes the Python tests in tests/test_tasks.py and the Jenkins pipeline tests in tests/jenkins/. Most hooks check all files; pylint checks only changed Python files.

To evaluate the flake without building derivations, using the same command as CI:

nix flake check --option allow-import-from-derivation false --no-build

To run the full check suite including builds:

nix flake check

ghaf-infra
├── docs/               # Documentation (see Documentation section below)
├── hosts/              # NixOS host configurations
│   ├── builders/       # Remote builder machines
│   ├── hetzci/         # Jenkins CI environments (see hetzci/README.md)
│   ├── nethsm-gateway/ # Tampere signing gateways and shared configuration
│   ├── testagent/      # On-prem test agents
│   ├── uae/            # UAE CI, test agents, and signing gateway
│   ├── ghaf-*/         # Supporting services (monitoring, auth, registry, etc.)
│   └── machines.nix    # Canonical host inventory (modules, systems, deploy metadata, IPs, keys)
├── keys/               # Signing public keys and certificates
├── modules/            # Shared NixOS modules
├── nix/                # Flake plumbing (deployments, apps, git-hooks)
├── pkgs/               # Custom packages
├── scripts/            # Operational scripts
├── slsa/               # Provenance definitions and attestation policies
├── tests/              # Python and Jenkins pipeline tests
├── users/              # Admin user configurations
└── tasks.py            # Invoke tasks (see docs/tasks.md)

• Architecture overview: how all the pieces fit together
• Adding a new host: host configuration and first install
• Deployment tasks: install, reboot, and other operational tasks
• Deploying with deploy-rs: how to deploy configuration changes
• Monitoring: Grafana and Prometheus setup
• Nebula overlay network: network connectivity between hosts
• NetHSM hardware signing: hardware-backed signing
• Jenkins authentication: Jenkins auth setup
• Jenkins test agents: on-prem test agents
• Jenkins CI development: developing the CI environment

• Deploy configuration changes: deploy-rs
• Add a new host: adding a host
• Add a remote builder user: add their SSH key to developers.nix, then deploy
• Onboard a new admin: add their user to users/, optionally add their age key to .sops.yaml and run inv update-sops-files, then deploy
• Manage secrets: secrets management
• Install, reboot, and other operational tasks: tasks

Note: Hosts may be reinstalled at any time. Do not store important data outside the configurations in this repository, including in /home directories on the hosts.

This project is REUSE-compliant. See LICENSES/ and the SPDX headers in each file.