[lsfadmin@host_machine2 test]$ bhosts
HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV
host_machine1 ok - 20 0 0 0 0 0
host_machine2 ok - 20 0 0 0 0 0
host_machine3 ok - 20 0 0 0 0 0
------------------------------------------------------
Now LSF cluster is ready to submit a job . bsub - Submits a job for execution and assigns it a unique numerical job ID. You can build a job file one line at a time, or create it from another file, by running bsub without specifying a job to submit. When you do this, you start an interactive session in which bsub reads command lines from the standard input and submits them as a single batch job. You are prompted with bsub> for each line.
Example :
[lsfadmin@host_machine2 test]$ bsub
bsub> sleep 100
bsub> Job <1588> is submitted to default queue <normal>.

NOTE: where job <1588> was submitted to the normal queue (default)
--------------------------------------------------
Next, submit a job to another queue (option -q) from the list of bqueues with job name (option -J) as "job_sachin" on host1 (option -m)

NOTE: where job_sachin (JOBID 1590) was submitted to the queue "short" and running on the host2
----------------------------------------------------------
Next , lets create a output file with option "-o"

The output (if any) is above this job summary.PS:Read file <err_file> for stderr output of this job.

NOTE: If you don't mention the error file , you will see both output & error in same output file . If LSF job runs on remote host , then you need to redirect the output from other machine by using the job directive #BSUB -f " outputfile < outputfile"
---------------------------------------------------------
Now , we can write a small script which could be submitted as a LSF job. You can redirect a script to the standard input of the bsub command as shown here: Create a file submit.lsf
-------------------------------------------------------------
[lsfadmin@host_machine2 new]$ cat submit.lsf
#BSUB -n 12
#BSUB -R "span[ptile=4]" # Where X is in the set {1..X}
#BSUB -J job_sachin # Job Name
#BSUB -outdir "outputdir/%J_%I"
#BSUB -o outputfile
#BSUB -f " outputfile < outputfile"
#BSUB -q short # Which queue to use {short, long, parallel, GPU, interactive}
#BSUB -W 0:55 # How much time does your job need (HH:MM)
#BSUB -L /bin/sh # Shell to use
sleep 30
/opt/xyz/spectrum_mpi/bin/mpirun hostname
----------------------------------------------------------------------------------------------
NOTE: after the job submission you will see 4 tasks running on each node due to ptile=4 for total 12 processes. It will create an output directory with outfile

Job <job_sachin> was submitted from host <host_machine2> by user <lsfadmin> in cluster <CI_cluster1>.Job was executed on host(s) <4*host_machine1>, in queue <short>, as user <lsfadmin> in cluster <CI_cluster1>. <4*host_machine2> <4*host_machine3></home/lsfadmin> was used as the home directory.</tmp> was used as the working directory.Started at Results reported onYour job looked like:

tail -f /NFSshare/lsf_logs &
tail_pid=$!
wait $bsub_pid
kill $tail_pid
--------------------------------------------
where: job2.sh
#BSUB -n 10
#BSUB -R "span[ptile=4]"
#BSUB -J job_sachin
#BSUB -o /NFSshare/lsf_logs
#BSUB -R "rusage[mem=6000]"
hostname
----------------------------------------
The “bpeek” command displays the stdout and stderr of a job while it is running. Usually this is only the most recent 10 lines of output. If you use the “-f” option, “bpeek” will continue to show additional lines as they are produced. It uses the “tail –f” command to do this, so you can stop the display of the output at any time by using <Ctrl-C>.

How to submit the interactive jobs ?
You can also submit
an interactive job using a pseudo-terminal with shell mode support. When you specify the -Ip option, bsub submits
a batch interactive job and creates a pseudo-terminal when the job
starts.