11. Apr 2019
IANVS: /scratch has free inodes, jobs may start again
Dear IANVS users,
the /scratch issue is in the process of being resolved. The repair process will continue through tomorrow. Jobs are allowed to start again on the cluster.
We currently have 18 million free inodes on /scratch, out of 128 million total (you can see the amount yourself with „df -i“). Every file and directory on the /scratch filesystem requires its own inode, so the inode capacity represents the maximum number of files and directories on a filesystem.
Mitigation Steps
The following steps have been taken to migigate the issue:
- We have identified and canceled the job responsible for reaching the inode limit.
- We will implement an inode quota on both /home and /scratch, starting at the next regular maintenance window.
- We have added checks for inode capacity to our monitoring system:
Inode Quota for /home and /scratch
Every user will have a soft upper limit of 20% (i.e., 25M) and a hard upper limit of 25% (i.e. 32M) of all inodes on both /home and /scratch (i.e. 128M), with a grace period of one week. We believe that this should be acceptable even for users whose jobs require creation of a large number of files.
The quota will not come into effect until the next maintenance window. We will also work with users whose current inode counts are above those limits before the quota takes effect.
Please contact us if you have any questions: via e-mail or phone, (0345) 55-21864 or (0345) 55-21861.
Best regards,
ITZ HPC team