Open OnDemand User Guide

Introduction

OpenOnDemand is a web-based portal that provides access to the cluster’s HPC resources. You can manage files, jobs, and access graphical interfaces all through OpenOnDemand. This document describes how to access and use OpenOnDemand and its related services. Please be aware that the changes to the job submission process on April 15th, 2020 affect OpenOnDemand interactive sessions. For more information, please review the Responsible Node Sharing document and the Running Jobs document.

Access and Login

To access Open OnDemand, open a web browser and type acf-login1.acf.utk.edu in the address bar. This will bring up the login page for Open OnDemand. Use your UT NetID and password to login. Figure 2.1 shows the login page for OpenOnDemand.

Figure 2.1 - OpenOnDemand Login Screen

After you successfully login, Duo TFA will come up. Select Duo Push or Passcode to log in. The authentication method will be sent to your mobile device. Figure 2.2 shows the Duo TFA options that will appear.

Figure 2.2 - Duo Authentication Options

Upon successful authentication, you will see the Open OnDemand dashboard. The top-left of the dashboard contains several menus that grant access to files, jobs, and your current interactive sessions.

File Explorer

The “Files” menu in the top-left of the OpenOnDemand dashboard gives you access to your files on the cluster from within a browser. Initially, the File Explorer shows your home directory. Figure 3.1 shows the File Explorer interface.

Figure 3.1 - File Explorer Interface

File Explorer Navigation

To navigate through directories, double-click on the directory you wish to enter. To return to your former directory, double-click on the two dots. If you are unsure where you are located within the directory tree, look above the directory listing pane. You will see several names and letters separated by forward slashes (/). This indicates the absolute pathname of your current location. In Figure 3.1, the current directory of the user is his home directory located at /nics/b/home/.

In the left pane of the File Explorer, you see the structure of your home directory. If at any time you wish to return to the top-level of your home directory, click on “Home Directory.” You may also navigate to directories within your home directory using this pane.

To quickly navigate to a specific directory, select the option labeled “Go To…” It has a dossier icon to the left of its label. When it opens, type the absolute pathname of the directory to which you wish to go. For example, to navigate to your Lustre directory, type /lustre/haven/user/<username> where <username> is your NICS username. The file explorer will then place you in the specified directory and display its contents. Figure 3.2 highlights the Go To option and the current directory location.

Figure 3.2 - Current Location and Go To Locations in File Explorer

File Explorer Views

OpenOnDemand’s file explorer can show permissions, ownership, and hidden files. In the top-right of the file explorer interface, check the boxes for “Show Dotfiles” and “Show Owner/Mode” to enable these views. Figure 3.2 shows the directory listing pane with the Show Owner/Mode option enabled. Permissions and ownership cannot be modified from within this interface. Hidden files are prefaced with a dot (.) and are usually user configuration files.

To view a specific file, select the file in the directory listing pane. Click on the option labelled “View.” It has the icon of an eye to the left of the label. The File Explorer will show the plaintext contents of the file.

Creating, Modifying, and Deleting Files

To create a new file in the File Explorer, select the option labelled “New File.” Its icon is a sheet of paper with a plus (+) symbol in the bottom-right. When you click on the option, the File Explorer will prompt you to name the file. After you name the file and confirm its creation, it will be placed within your current directory.

To modify the file, select the file in the directory listing pane, then select the option labelled “Edit.” Its icon is a canvas with a paintbrush on top of it. The OpenOnDemand editor will open. You can then type the information you wish to put in the file. The editor features several customization options, such as the editor’s theme and text mode. Figure 3.3 shows the OpenOnDemand file editor. When you finish editing the file, click “Save” in the top-left of the editor. Figure 3.3 highlights the Save option. When you return to the File Explorer and view the file, you can view the changes you made.

Figure 3.3 - OpenOnDemand File Editor

The File Explorer also allows you to rename existing files and directories. To rename a file or directory, select the item you wish to rename in the directory listing pane. Click the option labelled “Rename.” The File Explorer will prompt you for the new name, and once you confirm it, your changes will take immediate effect.

Files and directories can be copied and pasted within the File Explorer. To copy a file or directory, select the item in the directory listing pane, then select the option labelled “Copy.” Once the item has been copied to your clipboard, navigate to the location to which you intend to copy the file, then select the option labelled “Paste.” The copied item will then appear in your current directory.

If you wish to delete a file or directory, select the item in the directory listing pane, then select the “Delete” option on the right-side of the File Explorer. You will be prompted to confirm the deletion. Once confirmed, the item will be permanently deleted.

Downloading and Uploading Files

Files can be both downloaded from and uploaded to the cluster through the File Explorer. Only small files should be transferred between the cluster and your local system through the File Explorer. For larger file transfers, please use Globus. More information on how to configure and use Globus can be found in the Data Transfer document.

To download a file from the cluster to your local system, select the file in the directory listing pane. If you wish to download multiple files, hold down the Ctrl key on Windows and Linux systems or the Command key on MacOS, then select each file in the directory listing pane. The background of the files will become purple to indicate they are selected. Figure 3.4 shows how the files will appear when they are selected.

Figure 3.4 - Multiple Files Selected in File Explorer

Once all the files you wish to download are selected, click the option labelled “Download.” It is indicated with a down arrow. After the download completes, the file(s) will appear in your “Downloads” folder unless you have specified another location for downloads. Be aware that most browsers block attempts to download multiple files unless you explicitly allow the transfer.

To upload a file to the cluster from your local system, select the “Upload” option in the top-right of the File Explorer interface. The Explorer will prompt you to choose a file. Select “Choose Files” when the dialog box appears. You may then select the file(s) you want to upload to the cluster from your local system. In addition to this method, you may also drag and drop files from your local system into the File Explorer. These items will then appear in the directory listing pane.

Job Management

In the “Jobs” menu, you have two options: Active Jobs and Job Composer. Both options provide advanced job management from within a web browser.

Job Monitoring

In the Active Jobs section, you will see an organized list of your queued and running batch jobs on the cluster. Figure 4.1 shows the Active Jobs interface. You can filter through jobs to see particular entries, in addition to sorting by the attributes of your choice. To do this, click on the arrows next to the attributes until the list is sorted as you desire.

Figure 4.1 - Active Jobs Interface

When a job is queued or running on the cluster, you can view detailed information on the job from within the Active Jobs interface. Figure 4.2 shows the information you will see on queued and running jobs.

To access this view on a queued or running job, click the right-facing arrow on the far-left of the job’s row. You will then see statistics and information about the selected job. Of note is the “Output Location” of the job, which Figure 4.2 highlights. Each job you create and start through OpenOnDemand is placed within your /home/ondemand directory, so when you wish to retrieve the job’s data, remember to navigate to your ondemand directory. You can specify another output location when you create the job through OpenOnDemand or edit the job script to supply another location.

Figure 4.2 - Detailed View of a Running Job

Job Creation

In the Job Composer section, you can create new batch jobs to run on the cluster based upon several criteria. Figure 4.3 shows the Job Composer interface. When you initially open the Job Composer, the system will briefly explain each option available to you. Once you click through these prompts, you may then click “New Job,” then select the template, path, or job you wish to add. Note that if you close the initial information menu, you will not need to click through each prompt.

Figure 4.3 - Job Composer Interface

When you add a new job from the default template, the Job Composer will create a Simple Sequential Job. In addition to the job, it will create a job script and a job directory. Figure 4.4 shows the Job Composer interface when a job is created with the default template. To modify the job’s options, select Job Options. You can specify the job’s name, cluster, script, and account. To edit the job script, highlight the job and select the Edit Files option in the Job Composer interface. Select the job script, then select Edit. You may also rename the job script if necessary. Additional files can also be uploaded or created in the job’s directory from the File Explorer.

Figure 4.4 - Selecting the Default Template for a New Job

Jobs may also be created with other templates. These can be the default templates or templates that you build within the Job Composer. Figure 3.5 shows the Templates menu.

Figure 4.5 - Templates Menu within the Job Composer

In addition to template-based job creation, you may also create jobs based upon a specified path or an existing job. With a specified path, you create a job based upon a directory that already exists. The path will either be to your home or project directory. Once you specify the path, you then modify the typical attributes of the job. With a job created based upon an existing one, select an existing job in the queue, navigate to New Job, then select “From Selected Job.”

Beyond job creation, the Job Composer allows you to submit, stop, and delete your existing jobs. Like the Active Jobs menu, you can filter and sort the available jobs.

noVNC

noVNC allows users to interact with the cluster in a graphical desktop environment from within a web browser. noVNC is particularly useful when you require a graphical X11 application. Be aware that while most browsers support noVNC, outdated or uncommon browsers may not function. Please use Chrome, Firefox, Edge, or Safari when using noVNC interactive sessions.

To launch a noVNC session, log in to OpenOnDemand. In the dashboard, select “Interactive Apps” from the top-left of the window. In the dropdown menu, select “ACF Desktop.” That will open the session submission form depicted in Figures 5.1 and 5.2..

Figure 5.1 - ACF Desktop Session Submission Form (Part 1)

Figure 5.2 - ACF Desktop Session Submission Form (Part 2)

In this form, specify the resources your application(s) require. All the available options map to qsub options for interactive jobs. To learn the meaning behind these options, please review the Interactive Jobs section of the Running Jobs document. For most situations, a single Beacon node with one core is sufficient. If your application requires GPUs, please specify “Beacon” as the partition and “Beacon GPU” as the feature.

Please be aware that the number of cores per node you request directly influences the memory your job receives. For example, on Beacon, a single-core noVNC interactive session will receive 16GB of memory. The same session on Rho would only receive 2GB of memory. If your job exceeds these memory limits, it will be terminated. To avoid this problem, verify that the partition, feature, and number of cores accurately reflects the needs of your application. The System Overview document has additional information on the hardware resources available in each partition.

It is also important to note that you can request node-exclusivity for your session. With node-exclusivity, no other session will run on the node your session receives. In cases where the application you will use in the session is resource-intensive, setting this option to “Yes” may be beneficial. Otherwise, leave this option as “No.”

Submit the noVNC session to the scheduler by clicking on the “Launch” button. For most single-node, single-core jobs, the session should begin within five minutes; however, this is dependent on the current resource utilization of the cluster. If you requested multiple nodes or exclusive access to a node, it could take longer for the session to start. When the session is ready, access it by clicking the “Launch noVNC in New Tab” button. Figure 5.3 highlights this button.

Figure 5.3 - Active noVNC Interactive Session

When you access the session, you will see a standard Linux XFCE desktop environment. In the top-left of the GUI, the Applications menu contains links to several useful packages. In the bottom-middle of the GUI, you can open your home directory with the file cabinet icon. The black icon with a greater-than (>) symbol in it opens a terminal. From this terminal you can perform any task you would from a standard terminal such as accessing files, loading modules, and running applications.

If you wish to manually terminate the noVNC session, click on your username in the top-right of the GUI and select “Log Out.” Then, delete the session in OpenOnDemand. You will receive an email confirmation of the job’s deletion.

Jupyter Notebook

Launching Jupyter Notebook

Jupyter Notebook is a web-based application that enables you to intuitively create and manipulate data on the cluster. It features a fully-functional Python 2 and Python 3 IDE (integrated development environment), a text editor, and a file manager. The sections below outline how to access and set up Jupyter Notebooks on the cluster.

To access Jupyter Notebook, navigate to “Interactive Apps” on the OpenOnDemand dashboard. Select “Jupyter Notebook” from the dropdown menu. The Jupyter Notebook session submission form will appear as it does in Figures 6.1 and 6.2.

Figure 6.1 - Jupyter Notebook Session Submission Form (Part 1)

Figure 6.2 - Jupyter Notebook Session Submission Form (Part 2)

In this form, specify the Python version to use with Jupyter Notebook, the account under which the job should run, the QoS that applies to the job, any special features or partitions, the number of hours the job will run, and the number of cores your job requires. In most cases, all you should specify is the Python version, account, number of hours, and number of cores for the job. If you require the use of GPUs in your Jupyter Notebook session, select the "Beacon" partition and the "Beacon GPU" feature. Please review the Running Jobs document for more information on these fields.

When you specify the necessary options in the form, select “Launch.” It will not take the system long to deploy the Jupyter Notebook session, but this is dependent on the resources you requested. When the session is ready, you will see the same interface that appears in Figure 6.3. The “Connect to” option will open a new Jupyter Notebook tab in your browser. Figure 6.2 highlights this option.

Figure 6.3 - Jupyter Notebook Ready to Launch

Working with Jupyter Notebook

The initial Jupyter interface appears as it does in Figure 6.4. Like the OpenOnDemand File Explorer, you can manipulate your files and directories from within the Jupyter Notebook session. Files and directories can be created, edited, deleted, or moved. In addition, you can upload files to the cluster from your local system through the Jupyter Notebook session.

Figure 6.4 - Jupyter File Explorer

Under the “New” menu on the right-side of the interface, the dropdown menu depicted in Figure 6.5 appears. From here, you can create a Python 2 Notebook session, a text file, a folder (directory), or a terminal.

Figure 6.5 - New Item Menu

The first option, the Python 2 Notebook session, creates a Python 2 IDE in which you can write and execute Python 2 code interactively. Figure 6.6 shows what this session looks like. There are multiple options available to use, though they are beyond the scope of this document. Please consult the official Jupyter Notebook documentation for detailed information on all the options available to you within this IDE. You can rename the session in the upper-left of the interface by selecting the default name, Untitled, and then entering the name you wish to assign. In Figure 6.6, the user assigned the name “Py2-Test.”

Figure 6.6 - Jupyter Notebook Python 2 IDE

As you work within the IDE, Jupyter saves your work in a .ipynb file within your home directory. Its name will be the same as the name you assign within the IDE. You can view the file from the “Files” section within Jupyter. Additionally, if the session is running, the icon for the file will be green and will state that it is running next to the file’s timestamp.

The second option, text file, opens the Jupyter text editor. Figure 6.7 shows the initial text editor interface. Within this editor you can write code for multiple languages or simply use plaintext. Several different programming languages are supported by the editor. To see them all, click on the “Language” menu as shown in Figure 6.8. When you select a language, the editor will perform syntax highlighting and other functions to assist you as you code.

Figure 6.7 - Jupyter Text Editor

Figure 6.8 - Jupyter Text Editor Languages

Once you finish working in the text editor, select File, then Save. Be sure to rename the file by clicking on the default title, untitled.txt, and changing it to a more descriptive name. The saved file will appear in your home directory by whatever name you selected.

The third option, folder, creates a new directory in your current one. The fourth option, terminal, opens a ssh session in another browser tab. Select this option to perform certain tasks that the graphical interface does not allow, such as long listings of a directory or file searches.

To see the running Jupyter processes, return to the Jupyter Files menu by clicking on the Jupyter icon in the top-left of the interface. From there, select the “Running” menu. If you have any terminals or notebooks active, they will appear here. You can shut these processes down if you no longer need them or have completed your work. Figure 6.9 shows an example of the Running menu.

Figure 6.9 - Jupyter's Running Menu

At the time of this writing, the “Clusters” menu seen in Figure 6.9 next to the Running menu has no functionality.

When you finish your work within Jupyter, select the “Logout” option in the top-right of the window. Close the Jupyter Notebook tab, then return to your interactive sessions in OpenOnDemand. If you have time remaining on your job, please delete it. Do note that if you exceed your allotted time in the session, it will be terminated. Also be aware that when you manually delete the job for a Jupyter Notebook session, you will receive an email that confirms its deletion.

Using Anaconda Environments in Jupyter Notebook

When an Anaconda environment is loaded into a Jupyter Notebook session, all the packages maintained in that environment are available to the notebook. Minor modifications can enable this functionality. If you do not have an Anaconda environment and wish to create one, please consult our Anaconda User Guide. This section assumes you already have a working Anaconda environment that you wish to use with Jupyter Notebook.

To load an Anaconda enviroment into a Jupyter Notebook session, follow these steps.

If it is not already loaded, load the relevant Anaconda module. If your packages use Python 2, load Anaconda 2; if they use Python 3, load Anaconda 3.

Activate your Anaconda environment.

Install the ipykernel package in the Anaconda environment. Figure 6.10 shows the command to use for the installation. Replace <env-name> with the name of your Anaconda environment.

Verify that ipykernel exists in the environment by executing the conda list command.

Execute the command presented in Figure 6.11 to add the Anaconda environment to Jupyter. If you are using Python 2, replace python3 with pythonReplace <env-name> with the name of your Anaconda environment.

Specify the correct job parameters. Ensure that the correct Python version is selected for your environment. Submit the job to the scheduler.

Launch the job once it is ready. In the top-right corner of Jupyter Notebook, select the "New" dropdown menu. Your Anaconda environment should appear. Figure 6.12 shows where to look for the Anaconda environment. In the figure, the Anaconda environment available to Jupyter Notebook is named "fithic-env."

Figure 6.12 - Locating Anaconda Environments in Jupyter Notebook

Select the environment. You will be placed in a Jupyter Notebook session with your Anaconda environment available.

Two additional commands are helpful when using Anaconda environments with Jupyter Notebook. The first command outputs a list of the Anaconda environments available to Jupyter Notebook. Figure 6.13 shows this command. You can only execute this command after the relevant Anaconda module is loaded.

jupyter kernelspec list

Figure 6.13 - Command to Output a List of Available Environments

The second command enables you to remove an Anaconda environment from the list of environments available to Jupyter Notebook. Figure 6.14 depicts how to use this command. Replace <env-name> with the name of your Anaconda environment.

jupyter kernelspec remove <env-name>

Figure 6.14 - Command to Remove an Environment from Jupyter

RStudio Sessions

RStudio is a web-based IDE (integrated development environment) for the R statistical language on the cluster. Like Jupyter, it provides additional functionality beyond a standard interactive console. The specifics of how to access and manipulate RStudio are the focus of this section. Please note that the cluster's implementation of RStudio runs from within a container, which means it cannot be modified or updated. Be aware that OpenOnDemand runs RStudio version 3.5.3 at the time of this writing.

Launching RStudio Server

To launch an RStudio Server session, log in to OpenOnDemand. In the dashboard, select “interactive Apps” from the top-left of the window. In the dropdown menu, select “RStudio Server.” That will open the session submission form shown in Figures 7.1 and 7.2.

Figure 7.1 - RStudio Session Submission Form (Part 1)

Figure 7.2 - RStudio Session Submission Form (Part 2)

In this form, specify the resources your dataset(s) will require. As with noVNC and Jupyter Notebook sessions, all the available options map to qsub options for interactive jobs. To learn the meaning behind these options, please review the Interactive Jobs section of the Running Jobs document. For most situations, a single Beacon node with one core is sufficient.

Please be aware that the number of cores per node you request directly influences the memory your job receives. For example, on Beacon, a single-core RStudio Server session will receive 16GB of memory. The same session on Rho would only receive 2GB of memory. If your job exceeds these memory limits, it will be terminated. To avoid this problem, verify that the partition, feature, and number of cores accurately reflects the needs of your dataset(s). The System Overview document has additional information on the hardware resources available in each partition.

It is also important to note that you can request node-exclusivity for your session. With node-exclusivity, no other session will run on the node your session receives. In cases where RStudio Server will process large datasets, setting this option to “Yes” is beneficial. Otherwise, leave this option as “No.”

Submit the RStudio Server session to the scheduler by clicking on the “Launch” button. For most single-node, single-core jobs, the session should begin within five minutes; however, this is dependent on the current resource utilization of the cluster. If you requested multiple nodes or exclusive access to a node, it could take longer for the session to start. When the session is ready, access it by clicking the “Connect to RStudio Server” button. Figure 7.3 highlights this button.

Figure 7.3 - RStudio Ready to Launch

Working with RStudio Server

When you launch an RStudio Server session, a new browser tab will open. When it starts, you will see an interface with three panes: a console pane, an environment pane, and a file explorer pane.

The console pane, depicted in Figure 7.4, is where you run R code. At the top of the window, there are several menus with different options. It is not the purpose of this document to describe these menus in detail. Instead, please consult the official RStudio documentation for more information.

Figure 7.4 - RStudio Console Pane

In addition to the menus, there are several buttons that allow you to create, open, and save files and projects, in addition to options for menu layouts and add-ins. Below these buttons is the console itself. You can input R code into the console, press Enter (Return), and see the output of that code.

The environment pane contains information that pertains to the IDE itself, such as what packages are enabled, what dataset is in use, and your command history. Figure 7.5 shows the environment pane with the Environment tab selected.

Figure 7.5 - RStudio Environment Pane

In the file explorer pane, you can view and modify your files on the cluster, view additional packages, read documentation, and more. Figure 7.6 shows the file explorer pane. In the Files tab of the file explorer pane, you can create new folders, upload, delete, and rename files, in addition to several other options. To navigate through directories in the file explorer pane, click on the directory you wish to enter. To return to the previous directory, click the double dots (..). To return to your home directory, select “Home,” which has the icon of a house next to it.

Figure 7.6 - RStudio File Explorer Pane

Lustre scratch and project space is available in RStudio Server. To access these spaces, select the three dots (…) in the top-right of the file explorer interface. Figure 7.6 highlights these dots. When the prompt appears on-screen, type /lustre/haven/user/<username> for the Lustre scratch directory or /lustre/haven/proj/<project-ID> for Lustre project space. Press Enter to change to these directories.

Whenever you type the “help()” command, the RStudio Server documentation will appear in the “Help” tab of the file explorer pane. You can view all relevant documentation for RStudio from this tab, if necessary. To expand the view so it is easier to read, minimize the environment pane by clicking the horizontal bar in the top-right of the pane. This will expand the file explorer pane, making it easier to read documentation.

Installing Packages in RStudio Server

Users may install packages for use in RStudio Server from the CRAN repository or tar archives. These packages will be installed in your NFS home directory and are enabled on a per-session basis. Be aware of the 10GB quota on your home directory when installing packages. More information on quotas is available in the File Systems document.

For a comprehensive list of packages available from the CRAN repository, please visit the CRAN project’s website. To see the package list, select the “Packages” link on the left-side of the CRAN project’s homepage, then select one of the tables at the top of the page. The table will be organized based on your selection. At the time of this writing, packages that require an RStudio version greater than 3.5.3 will not work in RStudio Server. You can determine the version required by the package by clicking on it and checking the “Depends:” field.

To install a package, follow the steps below.

Select the “Packages” tab from the file explorer pane.

Click on the “Install” option. Figure 7.7 shows the dialog that will appear.

Figure 7.7 – Package Installation in RStudio Server

Install from the CRAN repository. If you have the tar archive of the package, select the option from the dropdown and specify the path to the archive.

Provide the names of the packages to install. These names can be found on the CRAN website.

Check the box to install dependencies.

Verify that the packages will be installed in your home directory, then select “Install.” The console pane will report on the progress of the installation.

Activate the package in the “Packages” tab of the file explorer by checking the box next to the package.

After the package is installed and activated, it can be used from within the console pane.

After you finish your RStudio Server session, select the red IO button in the top-right of the interface and confirm the shutdown. You will then be disconnected from the RStudio Server session. Be sure to also delete the job if any time remains on it. You will receive an email that confirms this deletion.

Known Issues

If you encounter a problem with OpenOnDemand and its services, please review the official Known Issues document before submitting a ticket. If the issue is not present in the official documentation or the proposed workarounds fail, then please submit a ticket to help@acf.utk.edu.