General Questions

Who can get an account?

Any faculty member or research scientist at an academic institution in Ohio is eligible for an academic account at OSC. These researchers/educators may request accounts for their students and collaborators. Commercial accounts are also available. More information about applying for both academic and commercial accounts at OSC can be found at https://www.osc.edu/supercomputing/support/account.

Where should a new OSC user begin?

The first thing you should do as a new OSC user is to check the email account that was given to OSC when you were registered. If you are a PI, this will be the email address you entered in the primary contact information on your Project Request application. When your account is created, you will receive an email giving you information on how to start using the services at OSC. Attached will be a letter you can keep for reference that contains your username and temporary password, as well as some other important information about using the systems at OSC. After carefully reading the email, please first go to my.osc.edu to change your temporary password.

Once you are able to connect to our HPC systems, you should start familiarizing yourself with the software and services available from the OSC, including:

How do I register for the Armstrong administration portal (for PIs)?

OSC supports the ARMSTRONG portal for administering your users and monitoring your project balance. ARMSTRONG is located at https://armstrong.osc.edu.You do not need to create your ARMSTRONG account if you choose not to use the features. However, if you wish to use ARMSTRONG, you must first register. You must register by clicking on the link in the paragraph to the right of the login information. Your ARMSTRONG userid and password may differ from those of your HPC accounts.

Password and shell changes have been moved to My OSC (my.osc.edu), and other features of Armstrong will be migrated over in the future as well. Please refer to the FAQ for future information regarding the role of My OSC and Armstrong at OSC.

Do I have to pay for supercomputer use?

OSC receives funding from the state of Ohio. There is no charge for Ohio academic use.

How many supercomputers does OSC have? Which one should I use?

OSC currently has two HPC clusters, which are named Oakley and Glenn. New users should choose Oakley unless they require software that is available only on Glenn. Long-time users should consider migrating some or all of their work to Oakley.

How do I acknowledge OSC in my publications?

Any publication of any material, whether copyrighted or not, based on or developed with OSC services, should cite the use of OSC, and the use of the specific services (where applicable). For more information about citing OSC, please visit www.osc.edu/citation.

Can I receive a letter of support from OSC when I apply for outside funding?

OSC has a standard letter of support that you can include (electronically or in hard copy) with a proposal for outside funding. This letter does not replace the application process for time on OSC's systems. To receive the letter of support, please send your request to oschelp@osc.edu. You should provide the following information: name and address of the person/organization to whom the letter should be addressed; name(s) of the principal investigator(s) and the institution(s); title of the proposal; number of years of proposed project; number of RUs requested per year. Please allow at least two working days to process your request.

How do I register for a workshop?

Where can I find documentation?

For documentation specific to software applications, see Software. For other available documentation, see Manuals.

My question isn't answered here. Whom can I ask for help?

Contact the OSC Help Desk. Support is available 24x7x365, but more complicated questions will need to wait for regular business hours (Monday - Friday, 9am - 5pm) to be addressed. More information on the OSC supercomputing help desk can be found on our Support Services page.

Something seems to be wrong with the OSC systems. Should I contact the help desk?

Information will be coming soon for guidelines on reporting possible system problems.

Account Questions

What are projects and accounts?

An eligible principal investigator heads a project. Under a project, authorized users have accounts with credentials that permit users to gain access to the HPC systems. A principal investigator can have more than one project, and authorized users can have more than one account.

How do I get/renew an account?

For information concerning accounts (i.e., how to apply, who can apply, etc.), see Accounts.

I'm a faculty member. How do I get accounts for my students?

If an eligible principal investigator is new to OSC, he/she can apply for a start-up request and include the contact information for authorized users. Please refer to http://app.osc.edu/cgi-bin/app/startup for the on-line form. If an eligible principal investigator has a current project, he/she can use the ARMSTRONG portal (https://armstrong.osc.edu) (or email OSC Help) to add authorized users. When emailing OSC Help, please include the student's full name, email address, and your project number. Authorized users do not have to be located in Ohio.

I'm continuing the research of a student who graduated. Can I use his/her account?

Yes, but you must have the account transferred into your name. Have your PI send an email to oschelp@osc.edu.

I'm working closely with another student. Can we share an account?

No. Each person using the OSC systems must have his/her own account. Sharing files is possible, even with separate accounts. Ask your PI to request an account for you through the ARMSTRONG portal (alternatively, PI's can email student information to OSC Help for this purpose). Contact oschelp@osc.edu for assistance.

How do I change my password?

You can change your password through the My OSC portal. Log in at http://my.osc.edu, and click the "change password" button at the bottom left corner of the HPC User Profile box. Please note that your password has certain requirements; these are specified on the "change password" portal. Please wait an hour before attempting to log in with the new password. For security purposes, please note that our password change policy requires a password change every 90 days.

I want to use csh instead of bash. How do I change the default shell?

You can change your default shell through the My OSC portal. Log in at my.osc.edu, and use the "Unix Shell" drop-down menu in the HPC User Profile box to change your shell. You will need to log off the HPC system and log back on before the change goes into effect. Please note, that it will take about an hour for the changes to be applied.

What is an RU?

An RU is a resource unit. A resource unit is an aggregate measure of the use of CPU, memory, and file storage. For details on charging algorithms, see Charging.

How do I find my account balance?

NOTE: Accounting is updated once a day, so the account balance is for the previous day.

How do I get more resources?

To request more resources, the principal investigator must prepare a proposal that will be sent for peer review. After the review process is complete, the Allocations Committee of the Statewide Users Group meets to determine the number of resource units to be awarded. The committee meets on the second Thursday of even-numbered months. PIs should submit their proposals at least three weeks before the meeting date to ensure inclusion on the agenda.

Justification and performance (quantitative estimation of the number of RUs per run times; the number of runs for the estimated duration of the request; performance reports of the code to be used or explanations about performance; report of any optimization performed on the code

How much will my account be charged for supercomputer usage?

Disk Storage Questions

What is my disk quota?

Each user has a quota of 500 gigabytes (GB) of storage and 1,000,000 files. You may also have access to a project directory with a separate quota. See Available File Systems for more information.

How can I determine the total disk space used by my account?

Your quota and disk usage are displayed every time you log in. You have limits on both the amount of space you use and the number of files you have. There are separate quotas for your home directory and any project directories you have access to.

Note: The quota information displayed at login is updated once a day, so the usage is for the previous day.

How can I find my largest directories?

To reveal the directories in your account that are taking up the most disk space you can use the du, sort and tail commands. For example, to display the ten largest directories, change to your home directory and then run the command:

du . | sort -n | tail -n 10

How can I use tar and gzip to aggregate and compress my files?

The commands tar and gzip can be used together to produce compressed file archives representing entire directory structures. These allow convenient packaging of entire directory contents. For example, to package a directory structure rooted at src/ use

tar -czvf src.tar.gz src/

This archive can then be unpackaged using

tar -xzvf src.tar.gz

where the resulting directory/file structure is identical to what it was initially.

The programs zip, bzip2 and compress can also be used to create compressed file archives. See the man pages on these programs for more details.

Tar is taking too long. Is there a way to compress quicker?

If using tar with the options zcvf is taking too long you can instead use pigz in conjunction with tar. pigz does gzip compression while taking advantage of multiple cores.

tar cvf - paths-to-archive | pigz > archive.tgz

pigz defaults to using eight cores, but you can have it use more or less with the -p argument.

tar cvf - paths-to-archive | pigz -n 4 > archive.tgz

Due to the parallel nature of pigz, if you are using it on a login node you should limit it to using 2 cores. If you would like to use more cores you need to submit either an interactive or batch job to the queue and do the compression from within the job.

pigz does not significantly improve decompression time.

Email Questions

How do I change the email address OSC uses to contact me?

Send your new contact information to oschelp@osc.edu. Note that changing your email address on ARMSTRONG does not change it for the HPC systems.

I got an automated email from OSC. Where can I get more information about it?

Linux Questions

What is Linux?

Linux is an open-source operating system that is similar to UNIX. It is widely used in High Performance Computing.

How can I get started using Linux?

See the Unix Basics tutorial for more information. There are also many tutorials available on the web.

SSH Questions

What is SSH?

Secure Shell (SSH) is a program to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. SSH provides secure X connections and secure forwarding of arbitrary TCP connections.

How does SSH work?

SSH works by the exchange and verification of information, using public and private keys, to identify hosts and users. The ssh-keygen command creates a directory ~/.ssh and files that contain your authentication information, The public key is stored in ~/.ssh/identity.pub and the private key is stored in ~/.ssh/identity. Share only your public key. Never share your private key. To further protect your private key you should enter a passphrase to encrypt the key when it is stored in the file system. This will prevent people from using it even if they gain access to your files.

One other important file is ~/.ssh/authorized_keys. Append your public keys to the authorized_keys file and keep the same copy of it on each system where you will make ssh connections.

Can I connect without using an SSH client?

The OSC OnDemand portal allows you to connect to Oakley and Glenn using your web browser, without having to install any software. You get a login shell and also the ability to transfer files.

How can I upload or download files?

Most file transfers are done using sftp (SSH File Transfer Protocol) or scp (Secure CoPy). These utilities are usually provided on Linux/UNIX and Mac platforms. Windows users should read the next section, "Where can I find SSH and SFTP clients".

Where can I find SSH and SFTP clients?

There are many SSH and SFTP clients available, both commercial and free. See Getting Connected for some suggestions.

How do I run a graphical application in an SSH session?

Graphics are handled using the X11 protocol. You’ll need to run an X display server on your local system and also set your SSH client to forward (or "tunnel") X11 connections. On most UNIX and Linux systems, the X server will probably be running already. On a Mac or Windows system, there are several choices available, both commercial and free. See our guide to Getting Connected for some suggestions.

Batch Processing Questions

What is a batch request?

On all OSC systems, batch processing is managed by the Portable Batch System (PBS). PBS batch requests (jobs) are shell scripts that contain the same set of commands that you enter interactively. These requests may also include options for the batch system that provide timing, memory, and processor information. For more information, see our guide to Batch Processing at OSC.

How do I submit, check the status, and/or delete a batch job?

PBS uses qsub to submit, qstat -a to check the status, and qdel to delete a batch request. For more information, see our Batch-Related Command Summary.

Can I be notified by email when my batch job starts or ends?

Yes. See the -m option in our PBS Directives Summary. If you are submitting a large number of jobs, this may not be a good idea.

Why won't my job run?

There are numerous reasons why a job might not run even though there appear to be processors and/or memory available. These include:

Your account may be at or near the job count or processor count limit for an individual user.

Your group/project may be at or near the job count or processor count limit for a group.

The scheduler may be trying to free enough processors to run a large parallel job.

Your job may need to run longer than the time left until the start of a scheduled downtime.

You may have requested a scarce resource or node type, either inadvertently or by design.

Why is my job being rejected with the error "Group: is not valid"?

If you are a part of a annual allocations project, usually designated by the prefix PAA-, in addition to another project you must explicitly specify which project you want to charge a jobs usage to. This can be done with the -A PBS directive. For example if a user was a part of both projects PAA0999 and PAQ0343 and wanted to charge a job to PAA0999 they would need to add the following line to their script:

#PBS -A PAA0999

How can I retrieve files from unexpectedly terminated jobs?

A batch job that terminates before the script is completed can still copy files from $TMPDIR to the user's home directory via the trap command. In the batch script, the trap command needs to precede the command causing the TERMination. It could be placed immediately after the PBS header lines. Here is a generic form:

How can I delete all of my jobs on a cluster?

To delete all your jobs on one of the clusters, including those currently running, queued, and in hold, login to the cluster and run the command:

qselect -u <username> | xargs qdel

Compiling System Questions

What languages are available?

Fortran, C, and C++ are available on all OSC systems. The commands used to invoke the compilers and/or loaders vary from system to system. For more information, see our Compilation Guide.

What compiler (vendor) do you recommend?

We have Intel, PGI, and gnu compilers available on both systems. We typically recommend the Intel compilers on Oakley and either PGI or Intel on Glenn. Each compiler vendor supports some options that the other doesn’t, so the choice depends on your individual needs.

Will software built for Glenn run on Oakley?

Most serial code built on one system will run on the other, although it may run more efficiently if it is built and run on the same system. Parallel (MPI) code typically must be built on the system where it will run.

Parallel Processing Questions

What is parallel processing?

Parallel processing is the simultaneous use of more than one computer (or processor) to solve a problem. There are many different kinds of parallel computers. They are distinguished by the kind of interconnection between processors or nodes (groups of processors) and between processors and memory.

What parallel processing environments are available?

On most systems, both shared-memory and distributed-memory parallel programming models can be used. Versions of OpenMP (for multithreading or shared-memory usage) and MPI (for message-passing or distributed-memory usage) are available. A summary of parallel environments will be coming soon.

What is a core?

A core is a processor. When a single chip contains multiple processors, they are called cores.

I'm not seeing the performance I expected. How can I be sure my code is running in parallel?

We are currently working on a guide for this. Please contact OSC Help for assistance.

Libraries/Software Questions

What software applications are available?

Do you have a newer version of (name your favorite software)?

Check the Software section to see what versions are installed. You can also check the installed modules using the module spider command on Oakley or the module avail command on Glenn.

How do I get authorized to use a particular software application?

Access Agreement forms are available on the Web in pdf format. Where appropriate, they may be found for a given application in the Software section.

What math routines are available? Do you have ATLAS and LAPACK?

See the Software section for information on third-party math libraries (e.g., MKL, ACML, fftw, scalapack, etc). MKL and ACML are highly optimized libraries that include the BLAS and LAPACK plus some other math routines. We recommend them instead of ATLAS. MKL is recommended on Oakley; ACML is recommended on Glenn, although MKL is available with the Intel compilers.

You may install open source software yourself in your home directory. If you have your own license for commercial software, contact the OSC Help desk.

I have a software package that must be installed as root. What should I do?

Most packages have a (poorly documented) option to install under a normal user account. Contact the OSC Help desk if you need assistance. We generally do not install user software as root.

What are modules?

Modules are used to manage the environment variable settings associated with software packages in a shell-independent way. On OSC's systems, you will by default have modules in your environment for PBS, MPI, compilers, and a few other pieces of software. For information on using the module system, see our guide to Batch Processing at OSC.

Performance Analysis Questions

What are MFLOPS/GFLOPS/TFLOPS/PFLOPS?

MegaFLOPS/GigaFLOPS/TeraFLOPS/PetaFLOPS are millions/billions/trillions/quadrillions of FLoating-point Operations (calculations) Per Second.

How do I find out about my code's performance?

A number of performance analysis tools are available on OSC systems. Some are general to all systems and others are specific to a particular system. See our performance analysis guide for more info.

How can I optimize my code?

There are several ways to optimize code. Key areas to consider are CPU optimization, I/O optimization, memory optimization, and parallel optimization. See our optimization strategy guide for more info.

Other Common Problems

What does "CPU time limit exceeded" mean?

Programs run on the login nodes are subject to strict CPU time limits. To run an application that takes more time, you need to create a batch request. Your batch request should include an appropriate estimate for the amount of time that your application will need. See our guide to Batch Processing at OSC for more information.

My program or file transfer died for no reason after 20 minutes. What happened?

Why did my program die with a segmentation fault, address error, or signal 11?

This is most commonly caused by trying to access an array beyond its bounds -- for example, trying to access element 15 of an array with only 10 elements. Unallocated arrays and invalid pointers are other causes. You may wish to debug your program using one of the available tools such as the TotalView Debugger.

I created a batch script in a text editor on a Windows or Mac system, but when I submit it on an OSC system, almost every line in the script gives an error. Why is that?

Windows and Mac have different end-of-line conventions for text files than UNIX and Linux systems do, and most UNIX shells (including the ones interpreting your batch script) don't like seeing the extra character that Windows appends to each line or the alternate character used by Mac. You can use the following commands on the Linux system to convert a text file from Windows or Mac format to UNIX format:

dos2unix myfile.txt

mac2unix myfile.txt

I copied my output file to a Windows system, but it doesn't display correctly. How can I fix it?

A text file created on Linux/UNIX will usually display correctly in Wordpad but not in Notepad. You can use the following command on the Linux system to convert a text file from UNIX format to Windows format: