soios/contestops/readme.md

# Contest ops

Here are instructions and various scripts and files for running contests.

The setup consists of a machine for each contestant, a machine running the grader, and an admin machine.
All these should be connected through a network, preferably wired.
The grader can be a machine accessible over the internet or in the local network.

## Grader setup

Install an ntp server on the grader machine.
This ensures that the contestant machine clocks are synchronized with the grader clock.
If a firewall is enabled, you may need to open the NTP port.

```bash
sudo apt install ntpsec
```

Configure the grader to accept client certificates.
The CA certificate (`certs/ca.pem`) is generated as part of the admin setup.

## Contestant machine setup

Obtain the contestant ISO, or build it yourself.

Flash the ISO to an USB stick.
All data on the stick will be lost.
For example, with the Gnome Disks utility, select the USB stick, open the menu on the right of the title bar, and click "Restore Disk Image...".

Boot the contestant machine from the USB stick.
Insert the stick and power on the machine.
Then repeatedly press a key to enter the boot menu (which key depends on the model, e.g. F12).
The boot menu may be password protected on machines in computer rooms; in that case you need to know the password.
The OS is loaded into RAM during boot, so you can remove the stick once the boot is finished and boot the next machine.

## Network setup

If there is not already an existing network, you need to set it up yourself.
Connect all contestant machines and the admin machine to a network switch with LAN cables.
If you use multiple switches, don't forget to also link the switches together.

If the grader must be accessed over the internet, you can connect the admin machine to WiFi or USB tethering with a phone.
You can then share the internet with the local network.

If you have Gnome, go to Network settings, click on the gear on the Ethernet connection, go to IPv4 tab, and select "Shared to other computers".

If you have docker installed, this doesn't work yet, because docker blocks routing.
You can fix it by running the following commands.

```bash
sudo iptables -I DOCKER-USER -i en+ -j ACCEPT
sudo iptables -I DOCKER-USER -o en+ -j ACCEPT
```

## Admin setup

This guide assumes that the admin machine is running Debian, Ubuntu or similar.

Invent a password for root on the machines.
Create a password hash for it with the following command.
Put the hash in the `contest_root_password` variable in `os/config/config.toml`.
This must be done before building the ISO.

```bash
sudo apt install whois
mkpasswd
```

Install parallel-ssh.

```bash
sudo apt install pssh
```

Edit `contestants.csv` and fill in the username and real name of each contestant.

Run the script to create a CA and client certificates.

```bash
sudo apt install golang-cfssl
./create-certs.sh
```

Edit `local.ssh_config` and create an entry with hostname and IP address for each contestant machine.
You can get the IP address by running `ip addr` in a terminal on the contestant machine.

Edit `hostlist` and add the hostnames of all contestant machines.

Get ssh host keys.
After rebooting machines, delete `local.known_hosts` and run this command again.

```bash
parallel-ssh -x "-F local.ssh_config" -h hostlist -O StrictHostKeyChecking=accept-new true
```

Test time synchronization.

```bash
parallel-ssh -x "-F local.ssh_config" -h hostlist -i date
```

Edit `config-hosts` and `config-nftables.conf` to fill in the correct IP addresses for the grader.
You can look these up with `host contest.soi.ch`.

Edit `contest-lock.json` to fill in the title and start time of the contest.

Apply the configuration to machines.
If the script gets stuck, press Ctrl+C and run it again.

```bash
./configure-machines.sh
```

Assign users to machines.

```bash
./assign-user.sh contestant01 stofl
./assign-user.sh contestant02 binna1
```

Start periodic backup of contestant machines.

```bash
./backup-create.sh timer
```

## Restore machine from backup

Because machines run from RAM, they will lose all files after rebooting.
Therefore, backups are especially important.

To restore a backup to a spare machine, use the following commands.
Prepare in advance by keeping the user to machine assignment nearby for reference, and
replacing `contestant03` in the commands below with the spare machine hostname.

```bash
./assign-user.sh contestant03 <username>
rsync -e "ssh -F local.ssh_config" -av --chown contestant:contestant backups/contestantxx/xxxx/ contestant03:/home/contestant/
```

## Contest lock screen

The contest lock screen is a gnome extension which can lock the screen and show a countdown until the contest starts.
The screen is unlocked when the contest starts.
The lock screen also displays the user name and a title.
It is configured in the file `/etc/contest-lock.json`.
It watches this file, and when it changes the new configuration is instantly applied.

If there is an error in the config file, it will continue to use the old config and print a message.
To see the logs, run this on a contestant machine:

```bash
journalctl -f -o cat /usr/bin/gnome-shell
```

An additional text can be shown with the `message` field. It can contain newlines (`\n`).


In case there is a problem with the contest lock screen and you can't fix it, the backup solution is to turn off `AutomaticLoginEnable` and set a password instead, that you announce when the contest starts.

```bash
parallel-ssh -x "-F local.ssh_config" -h hostlist 'chpasswd <<< contestant:stofl'
```

**Development notes**

Links:
- https://www.codeproject.com/Articles/5271677/How-to-Create-A-GNOME-Extension
- https://gjs.guide/

Regular lock screen (contest-lock is based on this):
- https://gitlab.gnome.org/GNOME/gnome-shell/-/blob/main/js/ui/screenShield.js
- https://gitlab.gnome.org/GNOME/gnome-shell/-/blob/main/js/ui/unlockDialog.js

Developer commands:
- Open the gnome-shell developer tools: Press Alt+F2, enter `lg`.

## Problems and solutions

Here are solutions to recurring problems.

**User indicator does not appear.**
Fixed by adding the gnome shell version from `gnome-shell --version` to the list of supported versions: `shell-version` in `os/layers/contestant/includes.chroot/usr/share/gnome-shell/extensions/user-indicator@soi.ch/metadata.json`.
The same applies for the contest-lock extension.