Information/Instructions/Inferno
From C-SAFE Wiki
Contents |
Time limits in Inferno
The following are approximate usage time limits for inferno:
- 1-16 nodes - 48 hours
- 17-32 nodes - 24 hours
- 33-64 nodes - 12 hours
- 64+ nodes - 4 hours
If you plan to use large numbers of processors (48+ or so) for long periods of time, please advise the csafe-homebrew mailing list. Jobs can be run for longer then the above limits, but are subject to being killed if necessary. (Make sure your code has checkpointing to minimize loss of data.)
Before You Start Using Inferno
These instructions are for using the upgraded Inferno cluster.
- You must clean up your old inferno .ssh directory. This only needs to be done one time. (Note, this may cause problems if you are still running on the old Inferno.)
mv ~/.ssh-inferno ~/.ssh-inferno-old logout (of inferno) login to inferno
How to use Inferno
Configure
- Optimized configure line:
../src/configure \ --enable-optimize='-O3 -msse -msse2 -march=pentium4' \ --enable-package="Uintah" \ --with-atlas=/usr/local/atlas \ --with-lapack=/usr/local/atlas \ --with-mpi=/usr/local/lam-mpi \ --enable-assertion-level=0 \ --with-hypre=/usr/sci/projects/Uintah/Thirdparty/1.0.0/Linux/gcc-4.1-lam-32bit/hypre-2.0.0-install \ --with-petsc=/usr/sci/projects/Uintah/Thirdparty/1.0.0/Linux/gcc-4.1-lam-32bit/petsc-install/petsc-2.3.3-p1 \ PETSC_ARCH=linux-gnu-c-opt
- Debug configure line:
../src/configure \ --enable-debug \ --enable-package="Uintah" \ --with-atlas=/usr/local/atlas \ --with-lapack=/usr/local/atlas \ --with-mpi=/usr/local/lam-mpi \ --with-hypre=/usr/sci/projects/Uintah/Thirdparty/1.0.0/Linux/gcc-4.1-lam-32bit/hypre-2.0.0-install \ --with-petsc=/usr/sci/projects/Uintah/Thirdparty/1.0.0/Linux/gcc-4.1-lam-32bit/petsc-install/petsc-2.3.3-p1 \ PETSC_ARCH=linux-gnu-c-opt
- cmake (with Uintah branch)
cmake ../src -DLAPACK_DIR=/usr/local/atlas \ -DMPI_DIR=/usr/local/lam-mpi \ -DHYPRE_DIR=/usr/sci/projects/Uintah/Thirdparty/1.0.0/Linux/gcc-4.1-lam-32bit/hypre-2.0.0-install \ -DPETSC_DIR=/usr/sci/projects/Uintah/Thirdparty/1.0.0/Linux/gcc-4.1-lam-32bit/petsc-install/petsc-2.2.1
Compiling
Please do not use the head nodes to compile sus. Until we get distCC working on the new Inferno, the best way to compile sus is to grab an interactive node for yourself and use that to compile.
qsub -I will get you an interactive node; see also 'llogin' below cd SCIRun/linux make -j3 <other flags here> sus SCI_MAKE_BE_QUIET=true
If your compile fails, you can clean out everything for a full recompile:
make cleanreally
and begin again. Note that it may take hours to do a full recompile!
Submitting the Job (PBS)
- WARNING: for now you can only submit jobs from inferno2.
To run jobs on Inferno, use the PBS batch system. Specifically you will use the qsub command:
qsub script.pbs
Look here for a sample script.
Interactive Login
llogin will get you an interactive session. You will need to either specify the full path, /usr/sci/projects/Uintah/scripts/inferno/llogin, or have /usr/sci/projects/Uintah/scripts/inferno in your PATH. The default is 1 node for 6 hours. Alternatively, you can run
llogin <num nodes>
or
llogin <num nodes> <Time in HH:MM:SS>
Ex.
llogin 12
llogin 4 12:00:00
Notes:
- If you see strange characters on your terminal, you may be able to fix it by doing:
unsetenv LANG
Back to: Main
