M-GRID cluster "ametisti"

More info

• A short summary on M-grid's compilers and MPI libraries can be found at http://www.csc.fi/proj/mgrid/docs/users_guide.html

File system

• Use command df to check exact file system status

  Directories visible to all machines:

  • "/home/" contains user home directories. Meant for software sources and binaries, small input and data files, but not for data storage or running simulations.

  Directories visible to the local machine only:

  • "/tmp" is meant for running all jobs and is a 116 GB (physics), 262 GB (chemistry) software RAID disk array local to each node. Do not run batch jobs on home!!
  • All batch jobs should be done so that the code first does a cd to /tmp/username, then runs below that! This is because the /home directory is an NFS directory, and accessing that is _much_ slower than accessing the local directories. If users access /home frequently over the network it will become horribly clogged down.
  • Files on /tmp are removed automatically after some time. See notes on "automatic removal of files" below.
  • However, users should make sure they do not fill up any local /tmp disk too much themselves
  • A useful command to empty all your own files older than 30 days on all machines is:
    • cluster-fork "find /tmp/$USER -mtime +30 -exec rm -rf {} \;"
    • But be careful with this, a small error may lead to you removing your home directory as well... man find helps.
    • If you just want to locate old files, replace the "-exec rm -rf {} \;" with a "-ls"
  • Long-term data storage is not provided. Use local laboratory systems or the CSC ARCHIVE for this

General hints of usage

• cluster-fork : runs command on all nodes
• cluster-ps : runs ps on all nodes
• cluster-kill : runs kill on all nodes

  All have a built-in help obtained with the "-h" option.

  For instance to check what jobs are really running on the entire node you can do

  • cluster-fork ps auxr | grep -v "ps auxr"

• Automatic removal of files
  • Files on the /tmp disks which should be used for batch jobs are automatically removed after some time. The time may be as small as 240 hours.
  • If you have jobs which run longer than this, data may start vanishing before the job is finished.
  • To avoid this, you are allowed to use the command "touch -a" regularly on all the files which are produced by the running job.
  • However, you should not use touch -a after the job is finished, but rather analyze and/or move the results to some permanent storage!

Batch queue system: SGE

• ametisti uses the Sun Grid Engine (SGE) batch queue system.
• man sge_intro

  How to see what jobs are running?

• qstat
• qstat -f
• qstat -u username

  How to see what queues are available and what they contain

• qstat -g c
• qconf -sq long

How to find out what a specific job is doing

• qstat -f -j (jobid)

  How to submit a job ?

• qsub qsub_samplescript
  • Including the line #$ -q phys.q in the script
  • Submits to "physics" nodes, compute-1-*

    How to kill a job

  • qdel (jobid)
  • qdel -u myuserid - kills all your jobs
• qsub qsub_samplescript
  • Including the line #$ -q chem.q in the script
  • Submits to "chemistry" nodes, compute-0-*

• A sample script for physics nodes is qsub_samplescript
  • The sample script is for a code called "parcas". To run your own code, it may be enough to just change "parcas" to the name of the executable of your own code everywhere in the script.
  • Also remember to change the e-mail address to your own!
• A sample script for chemistry nodes is gaussian-test.job
• As of Jun 1, 2005, there are memory limits on jobs of the same size as the physical memory of the processors - this will prevent the jobs from going into extensive swapping which would slow down running tremendously and hence not be optimal use of the processors.

Miscellaneous hints

• Fortran codes should be compiled statically!
• The default stack size is 8192. If you need to increase it, just add ulimit -s unlimited to your batch job script!
• If you want to run the same could multiple times with a script, simply include the script in the submission script, at the place where you in the previous example would start the executable.
  • This does have the disadvantage you can not use command line options, but you can just put these as variables in the script instead

How to get statistics of cluster usage?

Logging in to individual nodes

Known problems and how to avoid them

• e-mailing from SGE batch job scripts was fixed.
• Other MGRID nodes report system crashes if shared disk usage is high. Therefore it is of utmost importance to set up your jobs such that they use only local disk for all I/O-intensive operations! (KN 27.3. 2005)
• More problems to appear soon.

Rules of usage for the "ametisti" computer cluster

1. The cluster is intended for the use of personnel of the Department of physical sciences, chemistry and HIP.

2. The cluster is administered by Administrators appointed by the Heads of the Departments.

3. Allowed use is research and education in physics and chemistry utilizing efficient simulation and numerical codes. Any large-scale simulations should be run with compiled software, that is, extensive runs using interpreting programs such as Matlab, Mathematica etc. or script languages such as awk and perl are not allowed unless an explicit exception is granted by one of the Administrators. Running password cracking, cryptography, and "seti@home"-kinds of programs on ametisti is naturally strictly prohibited.

4. Research use accounts are given on a group-by basis. Eligible groups are those working at the owning institutions. In unclear cases the Head of the respective owning institutions decides whether a group is eligible for an account.

5. To open a group research account, the group leader should fill in the initial application form, and send it to the Account Administrator. After the initial application has been accepted, new users can be added to the group by a simple e-mail request from the group leader.

6. Educational accounts might be allocated for the lecturer of a computational physics course requiring parallell GRID computing resources for the period of the course, according to a separate agreement with one of the Administrators. But normally the smaller cluster mill should be used for this purpose. The lecturer bears the responsibility that the use of the educational accounts is limited to proper course use, and for guiding the course students into proper use of the cluster.

7. As of now, there are no pre-defined limits for usage. Groups are expected to use the machine in a gentlemanly manner, not attempting to hoard as much computer capacity for themselves as possible at the expense of other groups. All CPU use of each group is logged, and if a single group has used what seems like an obviously unreasonable share of the cluster for a long period of time, the Adminstrators have the right to ask them to limit their use in the future. If after several warnings the group still uses unreasonable amounts of capacity, the group accounts can be closed for a fixed period of time.

8. The use of the machine should take into account hardware limitations such as memory and hard drive space limitations. Hard disk space for public use is allocated on the /home/, and /tmp/ disks. Each user should keep their disk space usage to a reasonable minimum, and clean out stuff they no longer need. All long jobs should put their output to the /tmp/ disks, which are not backuped, and are not intended for long-term storage. Old files from the /tmp/ disks may be removed without prior warning to the user.

9. Although the cluster does have a capability for explicitly (MPI) parallel simulations, the cluster load is expected to be so high from serial jobs that running parallel jobs should be done only after prior consultation and permission from one of the Administrators. Running embarrasingly trivial parallel jobs using scripts is allowed within the limits set by the batch queue system on the number of jobs.

10. The group leader has the responsibility to ensure that the users in the group are aware of these rules, and that they know enough of the use of Unix systems to be able to follow these rules.

11. Any cluster user is allowed and indeed encouraged to report clear violations of these rules to the Administrators.

12. In case of clear violations of these rules, whether intentional or due to negligence or poor understading of the system, the Administrators can issue formal warnings to the group leader or course lecturer. If after two warnings the group still does not comply with the rules, the group account on the cluster will be closed for a fixed amount of time, or permanently.

Persons in charge

The technical administrators are (11.10 2005):

• Tomas Lindén
• Erkki Aalto
• Kai Nordlund
• Jani Kotakoski
• Teemu Pennanen
• Francisco Garcia (Geant4 and Root)
• CSC administrator: Olli-Pekka Lehto