Difference between revisions of "PTP/release notes/6.0"

(New page: == Overview == *DRAFT* started as a copy of 5.0 This document describes how to install PTP and its pre-requisites. This is intended for PTP users. If you are a PTP Developer and want to ...)

Overview

DRAFT* started as a copy of 5.0

This document describes how to install PTP and its pre-requisites. This is intended for PTP users. If you are a PTP Developer and want to build PTP from the plug-ins yourself, see PTP 6.0 Development Environment Setup

The installation process for PTP depends on a number of factors, including the versions of software you have installed on your system, your operating system, and the types of target systems you want to use. The basic installation steps are:

Note! The PTP SC11 tutorial contains detailed information about installing PTP on your workstation or laptop, including screenshots. You may wish to use the slides as a guide for installing Eclipse and PTP on your workstation or laptop. Then see below for information about installing components on your remote cluster as needed.

Components and Pre-Requisites

PTP is a set of plugins that extend the functionality of Eclipse. The following are the minimum requirements for PTP 5.0:

Java 1.5 or later (1.6 preferred)

Eclipse 3.7 or later

C/C++ Developer Tools (CDT) 8.0 or later

Remote System Explorer 3.3 or later is required for RDT (although does not need to be used)

The client side of PTP (the system running Eclipse) is supported on the following platforms:

Linux

Mac OS X

Windows

The server-side of PTP (the remote system used to launch/debug and host remote projects) is supported on the following platforms:

Linux

Mac OS X

Unix (e.g. AIX)

Server-side pre-reqs:

git (if Synchronized projects are used)

Java (if Remote projects are used)

Main Features

The PTP plugins are grouped into the following three main components:

Parallel Tools Platform (PTP) End-User Runtime

This feature contains the core PTP components, including all supported resource managers, the Parallel Language Development Tools (PLDT), and the External Tools Framework (ETFw).

PTP Remote Development Tools (RDT)

This feature adds support for the development of projects that are located on remote systems.

Fortran Development Tools (Photran)

This feature adds support for editing, compiling, and debugging programs written using the Fortran language.

The following table summarizes the components and the pre-requisites required for installation of the PTP features.

Component

C/C++ Development Tools (CDT)

Remote System Explorer End-User Runtime (RSE)

Other

PTP End-User Runtime

required

optional

Open MPI 1.2.x, 1.3.x, or 1.4.x, MPICH 2 1.0.x, gdb 6.3 or later

PTP Remote Development Tools (RDT)

required

required

N/A

Fortran Development Tools (Photran)

required

N/A

N/A

Optional Features

The PTP optional features are a set of features that provide additional functionality not included in the core PTP components. Most of these features require the installation of third party software or specialized components of CDT. The PTP Optional Features are summarized in the table below:

Component

Description

PTP External Tools Framework TAU Extension

Adds support for performance analysis using TAU. Requires installation of TAU on the target system.

PTP Graphical Explorer of MPI Programs (GEM)

Adds support for performance analysis using ISP. Requires installation of ISP on the target system.

PTP Parallel Performance Wizard (PPW)

Adds support for performance analysis using PPW. Requires installation of PPW on the target system.

PTP Parallel Language Development Tools UPC Support

Adds PLDT support for UPC. Requires CDT UPC support.

PTP RDT XL C/C++ Compiler Support

Adds support for the IBM XL C/C++ compiler toolchain to RDT. Requires CDT XL C/C++ Compiler Support, and the XL C/C++ compilers to be installed on the target system.

PTP Remote Synchronization Support

Enables support for synchronizing local and remote projects. Requires PTP Remote Development Tools (RDT).

Installation

There are two options for installing PTP: Full install or install into an existing Eclipse. A full install downloads and installs a copy of Eclipse that contains pre-installed PTP plugins. This is the simplest option. Install into an existing Eclipse is useful if you already have Eclipse installed and you just want to add the PTP functionality. Both methods are described in more detail below.

Install or Update Java

Ensure that a Java Runtime Environment (JRE) version 1.5 or higher is installed. This can be done with the command:

After downloading and extracting Eclipse (zip or tar file), there's nothing else to the installation process from the Eclipse side. In the extracted folder, simply run the eclipse executable file. It will ask for a workspace location; the default is fine.

See also "Updating from the Parallel Package to a more recent release" below

Multi-user installs: Note that the default install assumes that the user has write access to the location where Eclipse in installed (unzipped or un-tarred). See multi-user install information for other scenarios.

Option 2: Install into an existing Eclipse

This option assumes that you have already installed Eclipse and are familiar with the Eclipse installation process.

We recommend starting with either:

"Eclipse Classic" - if you also plan to do Java and/or plugin development, or

"Eclipse IDE for C/C++ Developers" - only includes support for C/C++, which is simpler and smaller

Using the table above, choose the components you wish to install, and note the pre-requisites required.

If you are unsure, just choose "Parallel Tools Platform (PTP) End-User Runtime"

If you are unsure, just choose "Parallel Tools Platform (PTP) End-User Runtime"

If your selection requires RSE as a pre-requisite, check "Remote System Explorer End-User Runtime" (Note: this may not be necessary and should be included automatically)

If any of your selections require CDT as a pre-requisite:

Open the "Programming Languages" category

Check "C/C++ Development Tools"

To add Fortran support:

Open the "Programming Languages" category

Check "Fortran Development Tools (Photran)" (See the Photran 6.0 User's Guide for more information on installing and using Photran)

Select the "Finish" button and follow the prompts

You should restart Eclipse when prompted.

Install optional PTP server components

The installation of PTP includes a number of optional server components that may be used to support the following:

IBM LoadLeveler

IBM Parallel Environment

SLURM

Parallel Debugging

As of PTP 5.0.5, some of the resource managers that support the above need the proxy server components installed on the remote machine, and some do not. The "proxy-based" resource managers do, but the JAXB-based resource managers do not.

IBM LoadLeveler

proxy

IBM Parallel Environment (proxy)

proxy

SLURM

proxy

Other resource managers

no proxy

If your environment uses any of the proxy-based resource managers, you will need to follow the steps below to install the additional components. Also, if you wish to use the PTP debugger, you will need to install the Scalable Debug Manager (SDM). If you are using PTP from a laptop or workstation, and plan to run or debug applications on a remote target machine, these components must be installed on the target machine before PTP can be used.

In the following steps, 5.0.0.201106140904 corresponds to the numeric version and build date of the PTP release. Later releases will have a different version number and build date. Use the latest one installed on your system.

Locate the following directory, replacing <os> and <arch> with the target system's operating system and architecture respectively:

org.eclipse.ptp.<os>.<arch>_5.0.0.201106140904

Change to this directory and run the command:

sh BUILD

This should build the executables. Check the bin directory to see if there are executables in it (e.g. ptp_ibmpe_proxy or sdm). If you see any errors during the build, please refer to the Trouble Shooting section below.

If desired, copy server executables to any convenient directory. When you create a resource manager, you will need to navigate to this directory (the bin directory by default) and select the appropriate resource manager proxy agent (if it requires one). When you configure a debug launch configuration, you will need to select the sdm executable in this directory.

Install optional RDT server component

The RDT server component must be installed manually only if *both* of the following are true:

You are using remote projects (not remote synchronized projects)

You wish to use Remote System Explorer (RSE) for remote connections

If you use Remote Tools (supplied as part of the Parallel Tools Platform (PTP) End-User Runtime), the server component is installed and launched automatically. If you plan to use Remote Tools, you can skip this section.

RDT provides advanced parsing and indexing functionality for C/C++ projects in a remote client-server scenario. The RDT server component, which contains the parser and index database, must be located on the same remote machine as the project files.

Note: The RDT server component is based on the RSE dstore server. Therefore setting up the RDT server is very similar to setting up the RSE dstore server. The guide to setting up the RSE dstore server can be found here, however this section contains documentation specific to the RDT server.

Server Installation on Unix, Linux and MacOS X

The following documentation explains how to install the Linux or UNIX server code, start the server daemon, and make a connection to a remote Linux or UNIX server.

Prerequisites

To use the Remote System Explorer communications server daemon you need to install Perl. Using the daemon helps eliminate some of the manual steps when you connect to the server.

Installing the server code

Ensure that Perl is installed.

Ensure that a Java Runtime Environment (JRE) version 1.5 or higher is installed. An IBM, Sun or equivalent JRE is required; The gcj-based Java shipped with most Linux distributions does not work. If in doubt, run the command java -version and check if there is a reference to gcj.

Choose a directory where you want to install the server code. These instructions will use the /home/user/ directory as an example, but you are free to use any directory. When the server archive file is expanded it will create a directory named rdt-server in the location where it is expanded.

Go to the PTP 3.0 Builds page and download the RDT server package that suits your operating system to this directory.

Switch to the /home/user/ directory.

Run the following command in the /home/user directory to extract the server code from the package appropriate to your operating system. For example, for Linux this command is:

tar -xvf rdt-server-linux-1.0.tar

A directory named /home/user/rdt-server will be created that contains the server files.

Starting the server

You can start the RSE communications server with the server daemon, or manually. Before starting the server, make sure the Java command is in your path, you can do this by running the following command:

Note that the server daemon runs on port 4075 by default. You can pass the optional daemonPort argument to force a different port if you want. If your daemon runs behind a firewall, you may want to specify the optional serverPortRange argument to restrict selected server ports to the range given:

perl ./daemon.pl 4075 10000-10010

To start the server manually

Note: In the following discussion we assume that the RSE server has been installed on Linux. If you are running on a UNIX system the script name is server.sh" rather than server.pl.

If you do not have root access on a remote machine, you can start the server manually for your particular user id only. Run the following commands:

cd /home/user/rdt-server
perl ./server.pl [port]

These commands run the server.pl script located in the /opt/rseserver directory. The port parameter to the server.pl script is optional. If you do not specify a port, then the server will pick the first one available and print the port number to standard out. By default, it is usually 4033. If you would like to use a different port, you will then have to enter this port number in port property for the Files subsystem for your connection in the Remote System Explorer (see Connecting to the Remote Server, below). Otherwise, you do not need to change this property.

Note: When you connect RSE to the server, the server will terminate as soon as you disconnect the client. The daemon, however, will not terminate.

Server Installation on Windows

The following documentation explains how to install the Windows server code, start the server daemon, and make a connection to a remote Windows server.

Installing the server code

Choose a directory where you want to install the server code. These instructions will use the C:/ directory as an example, but you are free to use any directory. When the server archive file is expanded it will create a directory named rdt-server in the location where it is expanded.

Copy the file rdt-server-windows-1.0.zip to the <code>C:\ directory (this could be on a different machine).

Use an unzip utility to extract the server code.

A directory named C:\rdt-server will be created that contains the server files.

Starting the server

You can start the RDT communications server manually, or as a daemon.

To start the server as a daemon

Note that the server daemon does not enforce any user authentication. If you run the server daemon, any user can connect to the machine, work with the file system and run commands. Use of the server daemon on Windows systems is not recommended.

Simply double click the daemon.bat program to start a server daemon. You can edit the daemon.bat file to change properties for the daemon, like a specific daemon port to use or to force a port range for the server (in order to comply with firewalls).

The server daemon runs on port 4075 by default. You can pass the optional daemonPort argument to force a different port if you want. If your daemon runs behind a firewall, you may want to specify the optional serverPortRange argument to restrict selected server ports to the range given:

daemon.bat 4075 10000-10010

To start the server manually

Simply double click on the server.bat program to start the RDT server. The server will pick the first port available and print the port number. By default, it is usually 4033. You will then have to enter this port number in port property for the Files subsystem for your connection in the Remote System Explorer.

For security reasons, the server will only wait a limited time until a client connects (12000 seconds by default). In order to start the server with an exactly specified port or timeout, open a Windows command prompt and enter:

c:
cd \rdt-server
server.bat [port] [timeout]

When you connect RDT to the server, the server will terminate as soon as you disconnect the client. The daemon, however, will not terminate.

Installing Open MPI

If you do want to build OpenMPI yourself, here are instructions based on building OpenMPI 1.4.2. See the Open MPI FAQ for more information on building and installing Open MPI.

Download the openmpi-1.4.2.tar.gz file

Untar it using the command tar -xzvf openmpi-1.4.2.tar.gz and cd to the directory it creates

Run ./configure

The default installation location is /usr/local

To install into a different location, e.g. /usr/local/openmpi1.4.2, use the command:

Updating from a previous release

Major releases

If you are updating from a previous major release (e.g. 4.0, 3.0, etc.) it is recommended that you start with a completely new installation of Eclipse and a new workspace. Existing projects can be imported into the workspace using one of the available import wizards (File > Import).

Updating from the Parallel Package to a more recent release

(This description may also apply to updating from a "Eclipse IDE for Parallel Application Developers" (parallel package) to a PTP more recent release, e.g. 6.0.x)

The normal way to update PTP is to use Help > Check for Updates, however there are two reasons this may not report any updates are available:

If you installed the Eclipse IDE for Parallel Application Developers or some other Eclipse build

If the PTP update site is not enabled in the update manager

If "Check for Updates" does not report any updates, follow these steps. Note that these steps only need to be done once, and "Check for Updates" should work for subsequent releases.

Troubleshooting

I have an OTP token that's required for use with the my remote system. Can I use this with RSE or creating remote c/c++ projects and/or the resource manager?

Solution: Use the OTP password but avoid saving the password with any of the dialogs. If you skip the password in dialogs that prompt for both username and password, you'll get a separate password prompt. Use your OTP password there.

I created a resource manager using the "Remote Tools" remote services but I entered the wrong username/password. How do I change these?

Solution: The Remote Environments view should be in your Eclipse workbench. If it's not, open the "Remote Environments" view by selecting "Window > Show View > Other...", open the "Remote Tools" folder, and selecting "Remote Environments". In the view, open "PTP Remote Host" and double click on the name of the connection you used in the resource manager configuration. Note that a connection must be stopped in order to edit it. Of course, if the username/password is wrong, it probably isn't running.

The parallel debugger does not start (may be stuck in "Upload file" or "Waiting for connection...".)

Solution: There are many reasons that this could be happening:

Check that you can run the MPI program successfully (non-debug), both from the command line, and from within Eclipse

Check that you have compiled the application for debugging with -g

If you are tunneling over ssh using "Remote Tools", check that the debugger session address is set to "localhost"

If you are not tunneling, or are using "RSE", check that the debugger session address is set to the hostname of the machine running Eclipse. Try disabling any firewall you have on your local machine.

Check that you have installed the sdm on the remote machine, and the path is correct on the "Debugger" tab in the launch configuration

I've created an Open MPI resource manager, but it fails to start with the message "Unable to determine version of Open MPI. Please check discover command is correct."

Solution: If the resource manager is local, see troubleshooting local resource managers below. If the resource manager is remote, then it is probably an issue with your PATH or LD_LIBRARY_PATH (or both) on the remote machine. Try running the command ssh remote_host ompi_info -a --parseable. If the ompi_info command is not found, then you need to set your PATH correctly on the remote machine. If you see a message like "ompi_info: error while loading shared libraries: libmpi.so.0: cannot open shared object file: No such file or directory" then you also need to set the LD_LIBRARY_PATH on the remote machine.

Eclipse is unable to start a local resource manager.

Solution: This is usually because your PATH is not being picked up by Eclipse. Most window systems do not get their paths from the shell, so when you launch Eclipse it will have the wrong PATH set. Consult the documentation on your window system for details on how to set the path, or alternatively, launch Eclipse directly from a shell.
See also this PTP FAQ item.

Every time I run an application, all the projects in my workspace build.

Solution: Open the Eclipse preferences. Go to "Run/Debug" and click on the "Launching" preference. Uncheck "Build (if required) before launching".

I only see one node when I start an Open MPI resource manager, how do I simulate multiple nodes on a single machine?

Edit /etc/hosts and add a localhost entry for each name you added to the openmpi-default-hostfile:

127.0.0.1 node0
127.0.0.1 node1
...

Edit /usr/local/etc/openmpi-mca-params.conf and add the following to the end:

orte_default_hostfile = /usr/local/etc/openmpi-default-hostfile

I want to alter the hostfile information but don't have root access.

Solution: Create a file, say 'hostfile' in your home directory containing the hostnames of the nodes. Then set the'OMPI_MCA_orte_default_hostfile' environment variable to '$HOME/hostfile'. You'll need to do this in your .bashrc or .tcshrc (the one which is executed when a new shell is created.) Using .bash_profile or .login won't work.

How do I launch eclipse?

Solution: Normally just run the eclipse executable - however, you may want to try this from a command line instead of from a "shortcut" - some OS's or windowing systems do not properly send the environment information. And, some OS's send the wrong info from the command line. Your mileage may vary.

Why are there XML tags in my parallel program output when I run my OpenMPI program in PTP?

Solution: The Remote C/C++ Editor does not currently support CDT breakpoints. Open the C/C++ Editor on the file instead by right-clicking on the file and selecting "Open With > C/C++ Editor. This problem should be addressed in 4.0.1. Update your version of PTP to 4.0.1 or later.

Known Issues

Resource Managers

Resource managers seem to get stuck at "Opening connection" when starting.

Connecting to a remote resource manager on some operating systems (in particular Fedora Core 11) generate a Kerberos authentication request. Check the command line where you started Eclipse for a Kerberos prompt. Enter your login name and password when requested.

Your operating system documentation may provide information on how to disable this request from being generated.

A "Secure Storage" dialog is seen when starting Eclipse on Linux.

Eclipse stores usernames/passwords in secure storage. On some systems, a password (usually your login password) must be entered to unlock this storage before Eclipse can access it. Check your operating system documentation for ways to disable this prompt.

There are synchronization issues when two (or more) JAXB_LML resource managers share the same remote tools connection.

There is some area of either the remote tools code or perhaps the JSch library that is not thread safe. This needs further diagnosing.

Reproduce By: Create a single remote tools remote connection. Add two JAXB_LML resource managers and give them the same connection. Start them both. Usually on the first pass of the LL_da driver, an exception will be thrown. This can occur repeatedly for as long as the resource managers run.

Debugger

The debugger gets stuck at "Upload file"

When the debugger launches onto a remote system, it attempts to make a connection back to your local machine running Eclipse. If you have the "SSH Tunneling" option enabled on the resource manager, then this connection will be made over the main ssh connection using an ssh tunnel. For this to work, your remote target sshd configuration needs to enable the "GatewayPorts" option. See the sshd_config(5) for more information on this option.

Workaround: If you are not able to change the sshd configuration on the remote target, the only option is to not use tunneling. This requires direct access from the remote target to your local Eclipse client machine (i.e. you from the remote target system, you should be able to use "ping <local_client_host>".) Also, you'll need to disable the firewall on the local client machine so that the incoming connection is permitted.

The debugger gets stuck at "Waiting for connection"

The debugger requires a filesystem that is shared across all nodes in the cluster in order to distribute routing information. If this routing file doesn't exist, then one or more of the debugger server processes will appear to hang.

Workaround: In this situation, the routing file must be manually distributed in order for the server processes to proceed.

This limitation will be addressed in a future version of the debugger.

Accelerator keys (F5, F6) operate on the current process selected in Debug view.

There is currently no way to use these keys for the Parallel Debug view.

Remote Development Tools (RDT)

The Remote C/C++ Editor is used by default for local files

RDT currently provides a framework for developing C/C++ programs on remote machines. Although it is our intention that RDT work seamlessly with CDT and other Eclipse plug-ins, there is a problem with the Eclipse platform which hinders interoperabilty between base CDT and RDT, namely that the Remote C/C++ Editor provided by RDT is always used by default for editing C/C++ files.

Workaround: Although this issue can be worked around by manually selecting the CDT editor (Context Menu >Open With >C/C++ Editor) when you are working with a local CDT project, it can be tedious having to do this for every single file you open. As such, we recommend that you install RDT into an Eclipse installation which you do not intend to do local C/C++ development with. Future versions of Eclipse and RDT should better support local development.

Unable to set CDT breakpoints in the Remote C/C++ Editor

This issue is similar to the previous problem. Right-click on the file and select Open With > C/C++ Editor. Note also that debugging remote projects with the CDT debugger has not been tested. It is likely that problems will occur if the debugger assumes files reside locally, but are actually located on a remote machine. Use at your own risk.

Remote projects are always closed when Eclipse restarts

When you exit and restart Eclipse, all your remote projects are closed. This happens because the .project file is on the remote server and when eclipse starts there is no connection yet, so eclipse thinks the .project file is missing and closes the project.

Workaround: To work around this issue, simply reopen your projects after the connection to the remote machine has been re-established.

Remote editors are not properly restored on Eclipse restart

If you leave remote editors open upon closing down the workbench, when you restart, the editors will not be properly initialized, showing a "resource not found" error. Closing and reopening the editor will not alleviate the problem. This problem occurs because on startup of the workbench, the remote connection is not yet established, and hence opening the editors fails.

Workaround: To work around this problem, close all editors opened on remote resources before shutting down the workbench. If the problem occurs, you can alleviate it by closing the affected editors and restarting the workbench.

Cannot cancel builds when using the RSE provider

There is an RSE bug which prevents RSE from forcibly terminating remote processes. As a result, if you use RSE to build, you won't be able to cancel a build while it's in progress.

Workaround: Use the Remote Tools provider if possible. Otherwise there is no workaround.

Quick Type Hierarchy does nothing

Invoking Quick Type Hierarchy from the editor context menu does not currently do anything.

Workaround: Use "Open Type Hierarchy" instead to open the Type Hierarchy View for an alternate method of viewing type hierarchy information.

Parallel Language Development Tools (PLDT)

Analysis of parallel applications located on a remote server doesn't seem to work

Note that not all PLDT features are tested and verified for running with RDT at this time.

Since 4.0.3, finding artifacts (MPI, OpenMP, UPC) does work on remote projects.

Workaround: To use other PLDT analysis features, use a local CDT project, instead of a remote project.

C/C++ Build Issues

I get the error "The Open MPI wrapper compiler was unable to find the specified compiler gcc in your PATH." when I build an MPI program (e.g. created with the Hello MPI World new project wizard).

The PATH variable may not be getting communicated to CDT correctly. Copy the contents of your PATH variable in the clipboard, and open the Project Properties, c/c++ build, Environment, add var: PATH and paste in the PATH from cmd line

The ptp_ibmpe_proxy and ptp_ibmll_proxy proxies for the IBM PE and IBM LoadLeveler resource managers do not work with LoadLeveler. Messages may be displayed stating that libllapi.so could not be loaded on Linux systems.

Depending on the version of LoadLeveler installed, libllapi.so may be a 32 bit shared library or a 64 bit shared library. This can be determined by locating libllapi.so in /opt/ibmll/LoadL. For 32 bit versions, it will be in /opt/ibmll/LoadL/full/lib, /opt/ibmll/LoadL/scheduler/lib, or /opt/ibmll/LoadL/so/lib. For 64 bit versions, libllapi.so will be in the corresponding lib64 directories. You should set the CFLAGS environment variable to contain either the -m32 or -m64 flags before invoking the BUILD script, for instance CFLAGS=" -m64" sh BUILD configure

Feature Descriptions

Feature

Required

Description

Parallel Tools Platform Core

Y

Core components of PTP

Parallel Tools Platform End-User Runtime

Y

Main PTP feature that combines PTP Core, resource managers, and other components

PTP Remote Services

Y (for remote services)

Adds a remote services abstraction that can be used to connect to remote systems using either Remote Tools or RSE. Can be installed as a stand-alone component.

PTP Common External Components

Y (for proxy agents and debugger)

Components that must be installed on a target system to support certain resource managers and the debugger

PTP Common Utilities

Y

Common utility functions. Can be installed as a stand-alone component.

PTP Parallel Language Development Tools

N (recommended)

Adds MPI, OpenMP, LAPI, and UPC analysis and assistance tools

PTP Parallel Language Development Tools LAPI Support

N

Adds LAPI assistance tools

PTP External Tools Framework

N

Adds support for the integration of external tools (formerly Performance Tools Framework)

PTP External Tools Framework TAU Support

N

Adds support for the integration of the Tuning and Analysis Utilities (TAU)

PTP Remote Tools

N (recommended)

Light weight ssh-based remote services. Can be installed as a stand-alone component.

PTP Remote Tools Enabler

N (recommended)

Adds support to allow PTP to use the remote services provided by Remote Tools

PTP RSE Enabler

N (recommended)

Adds support to allow PTP to use the remote services provided by RSE

PTP Scalable Debug Manager

N (recommended)

External component to allow debugging of parallel applications. Can be installed as a stand-alone component.