The need for interactive access to compute nodes: While large HPC systems are excellent resources for running production work, they are not configured for development. Often developers use departmental systems, or patiently submit a sequence of jobs to validate code changes. This can be particularly frustrating to new users, who want to "kick the tires", port applications and debug codes on supercomputers.

What most users need for development is interactive access to a set of compute nodes, in order to quickly compile, run and validate MPI or other applications multiple times in rapid succession. To help users with their code development, TACC has created an app that will do that, by allowing users to acquire a set of compute nodes for interactive access. The app is called idev, for Interactive DEVelopment.

The idev application creates an interactive development environment from the user's login window. In the idev window the user is connected directly to a compute node from which the user can launch MPI-compiled executables directly (with the ibrun command).

The idev command submits a batch job that creates a copy of the batch environment and then goes to sleep. After the job begins, idev acquires a copy of the batch environment, SSH's to the master node, and then re-creates the batch environment. The SSH command allows X11 tunneling for setting up a display back to the user's laptop for debugging.

If this is your first time launching idev and you have multiple accounts, idev will prompt you for the account and then save your selection as the default interactive account; otherwise, it will automatically use your only account. It will then begin to initiate your first idev session. Enjoy.

It is necessary for idev to gain access to compute nodes through the batch system; that is the normal mode for acquiring resources on a supercomputer (and at TACC). Hence the user must wait for the idev request to be accepted through the batch system. Fortunately, at TACC there is a "development" queue on each TACC system; idev's default setting uses this queue, and it is often a short wait for idev to acquire the nodes and allow interactive input. idev reports its progress every 4 seconds. Below is an example of idev's progress in setting up a session:

Note the prompt, "c301-202%", in the above session. It is your interactive compute-node prompt. You can test the "ibrun" command by executing "ibrun date" which will return date's output from each core of your session. You can begin immediately launching MPI applications with "ibrun myapp". On the compute nodes in some queues, compiler access is not available. In this case compile on the login node in another window.

By default only a single node is requested for 30 minutes. However, you can change the limits with command line options, using syntax similar to the request specifications used in a job script. The syntax is conveniently described in the idev help display:

On the old Stampede1 system, compiling on compute nodes is only supported on the development nodes. When using a queue other than development, we suggest opening two login windows, and using one for compilations and the other for starting an idev interaction session for running executions.