soios/contestops/readme.md

# Contest ops

Here are instructions and various scripts and files for running contests.

The setup consists of a machine for each contestant, a machine running the grader, and an admin machine (e.g. your personal laptop).
All these should be connected through a network.
The grader can be a machine accessible over the internet or in the local network.

## Grader setup

Install an ntp server on the grader machine.
This ensures that the contestant machine clocks are synchronized with the grader clock.
If a firewall is enabled, you may need to open the NTP port.

```bash
sudo apt install ntpsec
```

Configure the grader to accept client certificates.
The CA certificate (`certs/ca.pem`) is generated as part of the admin setup below.

## Contestant machine setup

Obtain the contestant ISO, or build it yourself.

Flash the ISO to an USB stick.
All data on the stick will be lost.
For example, with the Gnome Disks utility, select the USB stick, open the menu on the right of the title bar, and click "Restore Disk Image...".

Boot the contestant machine from the USB stick.
Insert the stick and power on the machine.
Then repeatedly press a key to enter the boot menu (which key depends on the model, e.g. F12).
The boot menu may be password protected on machines in computer rooms; in that case you need to know the password.
The OS is loaded into RAM during boot, so you can remove the stick once the boot is finished and boot the next machine.

## Network setup

If there is not already an existing network, you need to set it up yourself.
You can use either WiFi or wired Ethernet.
WiFi has the advantage that you don't need to bring and install all those Ethernet cables, and avoids the ugly cable mess.

**Ethernet:**
Connect all contestant machines and the admin machine to a network switch with Ethernet cables.
If you use multiple switches, don't forget to also link the switches together, such that the entire network forms a tree.

**WiFi:**
The contestant OS already has a WiFi pre-configured.
The SSID is `contest`, and the password is configured in the `contestant_wifi_password` variable in `os/config/config.toml`.
Configure the access point with this SSID and password, and the machines will connect automatically.

For larger contests (10 or more contestants) it's recommended to use a dedicated device as the Internet router.
SOI has a FRITZ!Box 4040 which can be used for this.
Here, you can easily get the list of IP addresses of contestant machines from the web interface.
It's best to connect the admin machine to the router box over LAN instead of WiFi, such that admin traffic only takes one wireless hop instead of two.

For smaller contests, you can also use the admin machine as a router.
For Ethernet, if you have Gnome, go to Network settings, click on the gear on the Ethernet connection, go to IPv4 tab, and select "Shared to other computers".
For WiFi, you can set up a WiFi hotspot in the WiFi settings.
However, with a hotspot enabled, Gnome does not let you connect to a WiFi network for Internet access at the same time, even though the hardware would in many cases support it.
Instead, you can connect a phone to WiFi and to your laptop with USB and enable USB tethering on the phone.

If you have Docker installed, this doesn't work yet, because docker blocks routing.
You can fix it by running the following commands.

```bash
sudo iptables -I DOCKER-USER -i en+ -j ACCEPT
sudo iptables -I DOCKER-USER -o en+ -j ACCEPT
```

## Admin setup

This guide assumes that the admin machine is running Debian, Ubuntu or similar.

Invent a password for root on the machines.
Create a password hash for it with the following command.
Put the hash in the `contest_root_password` variable in `os/config/config.toml`.
This must be done before building the ISO.

```bash
sudo apt install whois
mkpasswd
```

Install parallel-ssh.

```bash
sudo apt install pssh
```

Edit `contestants.csv` and fill in the username and real name of each contestant.

Run the script to create a CA and client certificates.

```bash
sudo apt install golang-cfssl
./create-certs.sh
```

## Before the contest

Edit `local.ssh_config` and create an entry with hostname and IP address for each contestant machine.
You can get the IP address by running `ip addr` in a terminal on the contestant machine.

Edit `hostlist` and add the hostnames of all contestant machines.

Get ssh host keys.
After rebooting machines, delete `local.known_hosts` and run this command again.

```bash
parallel-ssh -x "-F local.ssh_config" -h hostlist -O StrictHostKeyChecking=accept-new true
```

Test time synchronization.

```bash
parallel-ssh -x "-F local.ssh_config" -h hostlist -i date
```

Edit `config-hosts` and `config-nftables.conf` to fill in the correct IP addresses for the grader.
You can look these up with `host contest.soi.ch`.

Edit `contest-lock.json` to fill in the title and start time of the contest.

Apply the configuration to machines.
If the script gets stuck, press Ctrl+C and run it again.

```bash
./configure-machines.sh
```

Assign users to machines.

```bash
./assign-user.sh contestant01 stofl
./assign-user.sh contestant02 binna1
```

You may want to test on one machine that the certificate was properly installed and the grader is accessible.
This should only be done after the machines are configured and users assigned, but with a contest lock start time before the current time.
You can then set the start time and apply the contest lock config again.

```bash
parallel-scp -x "-F local.ssh_config" -h hostlist ./contest-lock.json /etc/contest-lock.json
```

Start periodic backup of contestant machines.
Backups are stored every 2 minutes in the folder `backups`.

```bash
./backup-create.sh timer
```

## Restore machine from backup

Because machines run from RAM, they will lose all files after rebooting.
Therefore, backups are especially important.

To restore a backup to a spare machine, use the following commands.
Prepare in advance by keeping the user to machine assignment nearby for reference, and
replacing `contestant03` in the commands below with the spare machine hostname.

```bash
./assign-user.sh contestant03 <username>
rsync -e "ssh -F local.ssh_config" -av --chown contestant:contestant backups/contestantxx/xxxx/ contestant03:/home/contestant/
```

## After the contest

Stop the periodic backup command with Ctrl+C.
Rename the `backups` folder to e.g. `backups-day1`.

You can shut down all machines.

```
parallel-ssh -x "-F local.ssh_config" -h hostlist poweroff
```

Machines should be shut down and booted again from USB stick between contests (e.g. between practice and actual contest) to ensure all data is erased.

## Contest lock screen

The contest lock screen is a gnome extension which can lock the screen and show a countdown until the contest starts.
The screen is unlocked when the contest starts.
The lock screen also displays the user name and a title.
It is configured in the file `/etc/contest-lock.json`.
It watches this file, and when it changes the new configuration is instantly applied.

If there is an error in the config file, it will continue to use the old config and print a message.
To see the logs, run this on a contestant machine:

```bash
journalctl -f -o cat /usr/bin/gnome-shell
```

An additional text can be shown with the `message` field. It can contain newlines (`\n`).


In case there is a problem with the contest lock screen and you can't fix it, the backup solution is to turn off `AutomaticLoginEnable` and set a password instead, that you announce when the contest starts.

```bash
parallel-ssh -x "-F local.ssh_config" -h hostlist 'chpasswd <<< contestant:stofl'
```

**Development notes**

Links:
- https://www.codeproject.com/Articles/5271677/How-to-Create-A-GNOME-Extension
- https://gjs.guide/

Regular lock screen (contest-lock is based on this):
- https://gitlab.gnome.org/GNOME/gnome-shell/-/blob/main/js/ui/screenShield.js
- https://gitlab.gnome.org/GNOME/gnome-shell/-/blob/main/js/ui/unlockDialog.js

Developer commands:
- Open the gnome-shell developer tools: Press Alt+F2, enter `lg`.

## Problems and solutions

Here are solutions to recurring problems.

**User indicator does not appear.**
Fixed by adding the gnome shell version from `gnome-shell --version` to the list of supported versions: `shell-version` in `os/layers/contestant/includes.chroot/usr/share/gnome-shell/extensions/user-indicator@soi.ch/metadata.json`.
The same applies for the contest-lock extension.