Troubleshooting¶
This page lists the messages you may see, what they mean and what to do. Find yours in the table.
YOUR_LOGIN stands for your university login, JOB_ID for the number of
a job, RUN for the number of a run, FOLDER_NAME for the name of your
project folder and FILE for the name of a file.
| What you see | When | Go to |
|---|---|---|
Cannot reach the cluster, or the test of the setup says not reachable |
connecting | The cluster is not reachable |
SSH asks for a password, or says Permission denied |
connecting | The cluster does not accept your key |
Could not chdir to home directory |
logging in | Your account is not ready yet |
Could not resolve hostname sc-login |
connecting | SSH does not know sc-login |
ssh does not know sc-export yet |
connecting | SSH does not know the server yet |
REMOTE HOST IDENTIFICATION HAS CHANGED |
connecting | "REMOTE HOST IDENTIFICATION HAS CHANGED" |
a warning about post-quantum key exchange |
connecting | A warning about post-quantum key exchange |
There is no job.slurm in this folder, No job was submitted from this folder yet, This folder has no job script |
any sc-starter command | The terminal is in the wrong folder |
runs.txt, line … with is empty, is a comment, ends with \ or has a hidden line break |
submitting | A line of runs.txt is not a run |
setup.slurm (job …) has not finished yet |
submitting | The setup job is still running |
The cluster has a newer copy |
submitting or pushing | "The cluster has a newer copy" |
Batch job submission failed |
submitting | "Batch job submission failed" |
The lockfile at uv.lock needs to be updated |
the setup job | The lock file does not match |
PENDING (Priority) or PENDING (Resources) |
waiting | The job waits |
FAILED with exit code 127 |
running | A command was not found |
ModuleNotFoundError or there is no package called |
running | A package is missing on the cluster |
TIMEOUT |
running | The run reached its time limit |
OUT_OF_MEMORY, Killed or oom_kill |
running | The run needed more memory |
Disk quota exceeded, or The output file … is empty |
pushing or running | Your home directory is full |
unable to start device PNG |
running R | R cannot draw PNG files |
This job has a GPU, but PyTorch cannot use it |
running Python on a GPU | Python: "This job has a GPU, but PyTorch cannot use it" |
Cannot find the requested files in the disk cache |
running Python with a model | The model was not downloaded |
Nothing to pull yet |
pulling | There is nothing to pull yet |
Connecting to the cluster¶
When SSH fails, sc-starter adds a second line that begins with
ssh said: and quotes the message of SSH itself.
The cluster is not reachable¶
sc-starter: Cannot reach the cluster. Is the VPN on? On the university network it works without.
Or the test in step 2 of the setup
says not reachable, or False on Windows. The cluster answers only
from inside the university network, and your computer is outside it.
To fix it:
- Connect the VPN, as in Connect the VPN. Only one device can use your VPN account at a time, so disconnect your phone first.
- On Linux, stop Docker if it runs, and try again. Its networks can cover addresses of the university network.
The cluster does not accept your key¶
SSH asks for a password:
YOUR_LOGIN@login01.sc.uni-leipzig.de's password:
Or it refuses at once:
YOUR_LOGIN@login01.sc.uni-leipzig.de: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Or sc-starter says:
sc-starter: The cluster did not accept your key. Is it loaded in the SSH agent (step 3) and uploaded to the portal (step 4)? See the setup guide, docs/basics/setup.md.
ssh said: YOUR_LOGIN@login01.sc.uni-leipzig.de: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
The cluster did not get a key that it knows. Do not type your password. Press Ctrl+C and go through these points in order:
- Check that the SSH agent has your key.
ssh-add -llists the keys it holds. If it printsThe agent has no identities., give it the key again:- On Linux, where the agent forgets the key at every restart, log in
once by hand with
ssh sc-loginand type your passphrase. - On macOS, type
ssh-add --apple-load-keychain. - On Windows, type
ssh-add "$HOME\.ssh\id_ed25519_sc". If that saysError connecting to agent, the agent is switched off. Turn it on as in Turn on the SSH agent.
- On Linux, where the agent forgets the key at every restart, log in
once by hand with
- Check that the portal has your public key. Compare the line in your
file
id_ed25519_sc.pubwith the key in the SC portal, as in step 4 of the setup. - On Windows, check which terminal you use. RStudio opens Git Bash when Git for Windows is installed. In Git Bash, sc-starter uses the SSH of Git, and that SSH cannot reach your key. Use PowerShell instead.
- Check whether the cluster is in maintenance. SC may block logins then. The SC status page says so.
Your account is not ready yet¶
Could not chdir to home directory /home/sc.uni-leipzig.de/YOUR_LOGIN: No such file or directory
sinfo -s also shows only the line with the column names, and no
partitions. SC has approved your account, but the cluster has not
finished setting it up. Wait a few hours and log in again, as
step 1 of the setup describes.
SSH does not know sc-login¶
ssh: Could not resolve hostname sc-login: Name or service not known
Or sc-starter says:
sc-starter: ssh does not know the name sc-login. Write the SSH settings from the setup guide, docs/basics/setup.md (step 5).
The short names sc-login and sc-export are missing from your SSH
settings. Write them as in
step 5 of the setup.
SSH does not know the server yet¶
sc-starter: ssh does not know sc-export yet. Log in once by hand with `ssh sc-export` and compare the fingerprint (the setup guide, docs/basics/setup.md, step 6).
SSH has not saved the fingerprint of this server yet, so it refuses to connect without asking you. This happens when the first connection to the export node was left out of the setup. Connect once by hand and answer the question about the fingerprint, as in Connect to the export node once.
"REMOTE HOST IDENTIFICATION HAS CHANGED"¶
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Later SSH says Host key verification failed. Or sc-starter says:
sc-starter: sc-login shows another key than the one ssh remembers. Usually SC has renewed it; read "REMOTE HOST IDENTIFICATION HAS CHANGED" in docs/reference/troubleshooting.md before you go on.
The server shows another fingerprint than the one your computer saved at the first login. Usually SC has renewed the keys of its servers, for example during maintenance. Rarely, someone pretends to be the server.
To fix it:
- Compare the new fingerprint with the list on the SC page SSH host keys. If it is not listed, do not connect, and write to SC.
-
If it is listed, make your computer forget the old fingerprint and log in again. Answer the question about the fingerprint as in step 6 of the setup:
Your computerssh-keygen -R login01.sc.uni-leipzig.de ssh sc-loginFor the export node, do the same with
export01.sc.uni-leipzig.deandssh sc-export true.
A warning about post-quantum key exchange¶
** WARNING: connection is not using a post-quantum key exchange algorithm.
** This session may be vulnerable to "store now, decrypt later" attacks.
** The server may need to be upgraded. See https://openssh.com/pq.html
The SSH on your computer is newer than the one on the cluster. The warning concerns the server, and SC has to update it. Everything works, so you need to do nothing.
Submitting and copying¶
The terminal is in the wrong folder¶
Depending on the command, sc-starter says one of these:
sc-starter: There is no job.slurm in this folder.
sc-starter: No job was submitted from this folder yet.
sc-starter: This folder has no job script (*.slurm), so it is not a project. Go to your project folder first, with `cd`.
sc-starter works with the folder that the terminal is in. Move into the
folder of your project with cd, as
Working in the terminal
describes, and try again.
sc-starter pull in a folder that was never submitted says
There is no ~/sc-projects/FOLDER_NAME on the cluster. If sc-starter
says that the folder name cannot be used on the cluster, the name holds
a character other than letters, digits, dots, hyphens and underscores,
such as a space. Rename the folder.
A line of runs.txt is not a run¶
sc-starter: runs.txt, line 4 is empty. Every line is one run, so remove empty lines.
Line N of runs.txt is run N, so every line has to be one complete
command. sc-starter checks this before it submits anything, and names the
line it does not accept:
| The message says | What to do |
|---|---|
is empty |
remove the empty line, also at the end of the file |
is a comment |
remove the line that starts with #, and keep notes elsewhere |
ends with \ |
write the whole command on one line |
has a hidden line break |
delete the line and type it again; text copied from another program can hold invisible characters |
The setup job is still running¶
sc-starter: setup.slurm (job 28404223) has not finished yet. Submit job.slurm once `sc-starter status` shows it COMPLETED.
The packages are still being installed, and a run that started now would
not find them. Wait until sc-starter status shows the setup job as
COMPLETED, then submit again. If it shows FAILED, sc-starter log
shows why.
"The cluster has a newer copy"¶
sc-starter: The cluster has a newer copy of these files than this folder:
checkpoint.pickle
Nothing was uploaded, so that none is overwritten. Read "The cluster has a newer copy" in docs/reference/troubleshooting.md.
sc-starter compares each file with its copy in
~/sc-projects/FOLDER_NAME on the cluster. The copy there has changed
since it was uploaded. A job may have written it, such as a checkpoint,
or you changed it on the cluster. Or this folder is an older copy of the
project, from another computer or a backup.
Decide for each file which copy to keep:
- To keep the copy on the cluster, delete the one in your folder. If you
want it on your computer as well, copy it with its time first:
scp -p sc-export:sc-projects/FOLDER_NAME/FILE FILE. - To send your copy, give it a new time:
touch FILEon Linux and macOS, or(Get-Item FILE).LastWriteTime = Get-Datein PowerShell.
Then push or submit again.
"Batch job submission failed"¶
sc-starter: The command on sc-login failed (exit code 1).
sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit)
Slurm refused the job because of an #SBATCH line in the job script. It
checks the limits when you submit, so a job that asks for too much never
waits in the queue. The end of the message names the cause:
| End of the message | The line to fix |
|---|---|
Requested time limit is invalid |
--time is longer than 2 days |
invalid partition specified |
--partition has a typo |
Requested node configuration is not available |
the job asks for more cores, memory or GPUs than a computer of that partition has |
Compare the line with The cluster at a glance, change it
and submit again. If your account is new and ssh sc-login sinfo -s
shows no partitions, your account is not ready yet, as described in
Your account is not ready yet.
The lock file does not match¶
The setup job of a Python project ends as FAILED, and sc-starter log
shows:
error: The lockfile at `uv.lock` needs to be updated, but `--locked` was provided.
hint: To update the lockfile, run `uv lock`.
You changed pyproject.toml, but uv.lock still lists the packages of
before. The setup job installs only what the lock file lists, and stops
when it does not match. On your computer, in the project folder, type
uv lock, or uv sync, which also installs the packages there. Then
submit the setup job again.
The job waits¶
sc-starter status shows runs as PENDING (Priority) or
PENDING (Resources), for example:
Job 28514701: 2 PENDING, 1 RUNNING
run state exit node elapsed command
1 RUNNING t010 0:41 python run.py 5
2 PENDING (Priority) 0:00 python run.py 10
3 PENDING (Priority) 0:00 python run.py 15
That is normal. Priority means that other jobs
are ahead of yours. Resources means that no node has enough free
cores, memory or GPUs right now. GPUs are in demand and often have a
queue.
Wait, and the runs start on their own. Jobs that ask for less start
sooner, because Slurm can fit them into gaps.
Choosing time and memory shows how
to check what a run really needs, so that you can lower --time,
--mem and --cpus-per-task.
A run fails¶
sc-starter status shows a run as FAILED, TIMEOUT, OUT_OF_MEMORY
or CANCELLED, and writes the numbers of these runs into failed.txt.
sc-starter log RUN shows what the run printed, which usually names the
cause.
After a fix, and once no run of the job waits or runs any more, start
only the failed runs again. submit uploads your fix first:
sc-starter submit --lines failed.txt
Fix lines of runs.txt in place. Run numbers are line numbers, so adding
or removing a line would shift the runs below it. Submit a job without
runs.txt, such as a setup job or a single GPU job, again as a whole,
with sc-starter submit setup.slurm or sc-starter submit.
A command was not found¶
run state exit node elapsed command
1 FAILED 127:0 t003 00:00:00 pyhton --version
sc-starter log 1 shows:
Run 1 on t003: pyhton --version
bash: line 1: pyhton: command not found
The command in that line of runs.txt does not exist, here pyhton
instead of python. The exit code 127 always means that a command was
not found. Correct the line, then submit the failed runs again.
A package is missing on the cluster¶
A Python run stops with a message such as:
ModuleNotFoundError: No module named 'tomotopy'
An R run stops with a message such as:
Error in library(stm) : there is no package called ‘stm’
The package is on your computer but not in the project's environment on the cluster. Usually the setup job has not run since you added the package, or it failed.
To fix it:
- Check that the package is in the lock file:
uv.lockfor Python,renv.lockfor R. If not, add it withuv addor record it withrenv::snapshot(), as Packages: uv and renv explains. - Run the setup job again with
sc-starter submit setup.slurm, and wait until it showsCOMPLETED. - For R, check that the lines of
runs.txtstart withRscriptand not withRscript --vanilla, which skips the project's packages.
The run reached its time limit¶
For a run with --time=00:02:00:
run state exit node elapsed command
2 TIMEOUT 0:0 t003 00:02:15 sleep 600
sc-starter log 2 ends with a line from Slurm:
Run 2 on t003: sleep 600
[2026-10-20T13:24:42.004] error: *** JOB 28408618 ON t003 CANCELLED AT 2026-10-20T13:24:42 DUE TO TIME LIMIT ***
The run needed longer than --time in job.slurm, and Slurm stopped
it. The exit code 0:0 does not mean success here. The state TIMEOUT
counts.
Raise --time. The cluster allows at most 2 days, except in the
long partitions. If a run needs longer, split it
into several runs. Or let it save its progress and continue, as
Runs longer than the time limit
shows.
The run needed more memory¶
For a run with --mem=200M that needed 1 GB:
run state exit node elapsed command
3 OUT_OF_MEMORY 0:125 t003 00:00:00 python3 -c "x = b'1' * 10**9"
sc-starter log 3 shows:
Run 3 on t003: python3 -c "x = b'1' * 10**9"
/var/spool/slurmd/job28408616/slurm_script: line 13: 71798 Killed bash -c "$run"
[2026-10-20T13:22:27.166] error: Detected 1 oom_kill event in StepId=28408616.batch. Some of the step tasks have been OOM Killed.
The run needed more memory than --mem in job.slurm, and Slurm stopped
it. Killed and oom_kill in the log both mean that it ran out of
memory. Raise --mem, and check afterwards what the run really used, as
Choosing time and memory shows.
Your home directory is full¶
sc-starter push stops with lines that end in Disk quota exceeded, or
runs fail with an empty or cut-off log. sc-starter log then says:
The output file python-topic-model-28514701_3.out is empty. If the run failed, your home directory may be full: `ssh sc-export sc_quotas` shows it.
Your home directory on the cluster holds 100 GB, including all projects
and the shared stores of packages and models. ssh sc-export sc_quotas
shows how full it is.
Keep your account tidy shows what takes up space and
how to remove it.
R cannot draw PNG files¶
Error in .External2(C_X11, paste0("png::", filename), g$width, g$height, :
unable to start device PNG
Calls: png
In addition: Warning message:
In png(sprintf("results/grid-%.1f.png", threshold), width = 480, :
unable to open connection to X11 display ''
Execution halted
By default, R draws PNG files through a screen, and the computers of the
cluster have none. Put options(bitmapType = "cairo") at the top of the
script, or write png(..., type = "cairo").
Python: "This job has a GPU, but PyTorch cannot use it"¶
This job has a GPU, but PyTorch cannot use it. See README.md.
The scripts of the GPU code-alongs stop with this message when the job
has a GPU but PyTorch cannot reach it, so that the GPU does not sit
idle.
Usually a version of PyTorch without GPU support was installed. Compare
your pyproject.toml with the one of
Text embeddings on a GPU,
then run sc-starter submit setup.slurm again.
The model was not downloaded¶
Cannot find the requested files in the disk cache and outgoing traffic has been disabled.
The GPU jobs of the code-alongs load their language model from the
folder in which the setup job saved it, and never download it
themselves. So the setup job has not downloaded the model. Perhaps it
failed, or you changed MODEL in run.py since.
Look at its log with sc-starter log --job JOB_ID, using the job ID of
the setup job, and run sc-starter submit setup.slurm again.
Bringing results back¶
There is nothing to pull yet¶
sc-starter: Nothing to pull yet: ~/sc-projects/FOLDER_NAME has no results/ folder and no output files.
No run has started yet, so there is nothing to bring back.
sc-starter status shows whether the runs are still waiting.
Getting help¶
- The SC documentation answers most questions about the cluster.
- SC support helps with your account, the cluster and jobs. Use the form Request support in the SC portal, or write to sc-request@uni-leipzig.de, the address the login node greets you with.
- The service desk of the URZ, the university's computing centre, helps with the VPN and your university login: servicedesk@uni-leipzig.de, phone +49 341 97-33333.
- For problems with sc-starter or this guide, open an issue on GitHub.
When you ask for help, say what you typed, and paste the whole message,
the job ID and the output of sc-starter --version. Never send your
private key or its passphrase. Nobody needs them to help you.