Install and Setup

The Cluster Agent should be installed and executed under a dedicated, non-privileged service account. This account must have sufficient permissions to create and manage Slurm accounts through the Slurm CLI.

useradd --system --create-home keystone-sa
sacctmgr -i modify user name= keystone-sa set AdminLevel=Operator

The Slurm command-line utilities must be available in the agent’s runtime environment. The command below will verify the required Slurm utilities are installed and accessible:

command -v sacctmgr scontrol sinfo

If any Slurm commands are missing, ensure the appropriate client packages are installed and available in the system path.

Package Installation

Before installing the cluster agent, create a new directory under /var/keystone/keystone-agent/. This directory must be owned by the agent service account with permissions restricted to the owner.

AGENT_DIR="/var/keystone/keystone-agent/"
mkdir -p -m 700 $AGENT_DIR
chown keystone-sa:keystone-sa $AGENT_DIR

After the directory has been created with the correct permissions, install the package using pipx.

BHPC_REPO="https://dl.cloudsmith.io/public/better-hpc/keystone/python/simple/"
pip install --extra-index-url=$BHPC_REPO keystone-agent

To verify the installation completed successfully, check for the presence of the kca utility.

kca --help

Launching a Server

The runserver command will launch an agent server on 127.0.0.1:2545. See kca runserver --help for instructions on customizing the target host and port.

kca runserver --slurm

The following unit file is provided as a starting point for system administrators who wish to run the server as a system service.

[Unit]
Description = Keystone Cluster Agent
After = network.target
Wants = network.target

[Service]
Type = simple
User = kca-agent
Group = kca-agent
ExecStart = /usr/bin/kca runserver --slurm
WorkingDirectory = /var/keystone/keystone-agent
Restart = on-failure
RestartSec = 5

[Install]
WantedBy = default.target

After deploying the unit file, load the new configuration and launch the service.

systemctl daemon-reload && systemctl restart keystone-agent

Deploying a Proxy

It is strongly recommended to deploy the Cluster Agent behind a reverse proxy with TLS encryption. The following recipe is provided as a convenient starting point for configuring Nginx.

server {
    listen 443 ssl;
    server_name cluster-agent.example.com;

    ssl_certificate     /etc/ssl/certs/cluster-agent.crt;
    ssl_certificate_key /etc/ssl/private/cluster-agent.key;

    location / {
        proxy_pass http://127.0.0.1:2545;
        proxy_http_version 1.1;

        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 5s;
        proxy_read_timeout 60s;
    }
}

After deploying the configuration, reload Nginx to apply the changes.

nginx -t && systemctl reload nginx