In HPX main is used to
initialize the runtime system and pass the command line arguments to the
program. If you wish to add command line options to your program you would
add them here using the instance of the Boost class options_description,
and invoking the public member function .add_options()
(see Boost Documentation
or the Fibonacci Example
for more details). hpx::init() calls hpx_main
after setting up HPX, which is where the logic of our program is encoded.

Here is hpx_main:

inthpx_main(boost::program_options::variables_map&vm){{// Get a list of all available localities.std::vector<hpx::naming::id_type>localities=hpx::find_all_localities();// Reserve storage space for futures, one for each locality.std::vector<hpx::lcos::future<void>>futures;futures.reserve(localities.size());BOOST_FOREACH(hpx::naming::id_typeconst&node,localities){// Asynchronously start a new task. The task is encapsulated in a// future, which we can query to determine if the task has// completed.typedefhello_world_foreman_actionaction_type;futures.push_back(hpx::lcos::async<action_type>(node));}// The non-callback version of hpx::lcos::wait takes a single parameter,// a future of vectors to wait on. hpx::lcos::wait only returns when// all of the futures have finished.hpx::lcos::wait(futures);}// Initiate shutdown of the runtime system.returnhpx::finalize();}

In this excerpt of the code we again see the use of futures. This time
the futures are stored in a vector so that they can easily be accessed.
hpx::lcos::wait()
is a family of functions that wait on for an std::vector<> of futures to become ready. In
this piece of code, we are using the synchronous version of hpx::lcos::wait(),
which takes one argument (the std::vector<> of futures to wait on). This function
will not return until all the futures in the vector have been executed.

In the Fibonacci Example,
we used hpx::find_here()
to specified the target' of our actions. Here, we instead use hpx::find_all_localities(),
which returns an std::vector<>
containing the identifiers of all the machines in the system, including
the one that we are on.

As in the Fibonacci Example
our futures are set using hpx::lcos::async<>(). The hello_world_foreman_action
is declared here:

Another way of thinking about this wrapping technique is as follows: functions
(the work to be done) are wrapped in actions, and actions can be executed
locally or remotely (e.g. on another machine participating in the computation).

Now it is time to look at the hello_world_foreman() function which was wrapped in the action
above:

voidhello_world_foreman(){// Get the number of worker OS-threads in use by this locality.std::size_tconstos_threads=hpx::get_os_thread_count();// Find the global name of the current locality.hpx::naming::id_typeconsthere=hpx::find_here();// Populate a set with the OS-thread numbers of all OS-threads on this// locality. When the hello world message has been printed on a particular// OS-thread, we will remove it from the set.std::set<std::size_t>attendance;for(std::size_tos_thread=0;os_thread<os_threads;++os_thread)attendance.insert(os_thread);// As long as there are still elements in the set, we must keep scheduling// PX-threads. Because HPX features work-stealing task schedulers, we have// no way of enforcing which worker OS-thread will actually execute// each PX-thread.while(!attendance.empty()){// Each iteration, we create a task for each element in the set of// OS-threads that have not said "Hello world". Each of these tasks// is encapsulated in a future.std::vector<hpx::lcos::future<std::size_t>>futures;futures.reserve(attendance.size());BOOST_FOREACH(std::size_tworker,attendance){// Asynchronously start a new task. The task is encapsulated in a// future, which we can query to determine if the task has// completed.typedefhello_world_worker_actionaction_type;futures.push_back(hpx::lcos::async<action_type>(here,worker));}// Wait for all of the futures to finish. The callback version of the// hpx::lcos::wait function takes two arguments: a vector of futures,// and a binary callback. The callback takes two arguments; the first// is the index of the future in the vector, and the second is the// return value of the future. hpx::lcos::wait doesn't return until// all the futures in the vector have returned.hpx::lcos::wait(futures,[&](std::size_t,std::size_tt){if(std::size_t(-1)!=t)attendance.erase(t);});}}

Now, before we discuss hello_world_foreman(), let's talk about the hpx::lcos::wait() function. hpx::lcos::wait() provides a way to make sure that all
of the futures have finished being calculated without having to call get()
for each one. The version of hpx::lcos::wait() used here performs an non-blocking wait,
which acts on an std::vector<>.
It queries the state of the futures, waiting for them to finish. Whenever
a future becomes marked as ready, hpx::lcos::wait() invokes a callback function provided
by the user, supplying the callback function with the index of the future
in the std::vector<>
and the result of the future.

In hello_world_foreman(), an std::set<> called attendance
keeps track of which OS-threads have printed out the hello world message.
When the OS-thread prints out the statement, the future is marked as ready,
and hpx::lcos::wait()
invokes the callback function, in this case a C+11 lambda. This lambda
erases the OS-threads id from the set attendance,
thus letting hello_world_foreman() know which OS-threads still need to print
out hello world. However, if the future returns a value of -1, the future
executed on an OS-thread which has already printed out hello world. In
this case, we have to try again by rescheduling the future in the next
round. We do this by leaving the OS-thread id in attendance.

Finally, let us look at hello_world_worker(). Here, hello_world_worker() checks to see if it is on the target
OS-thread. If it is executing on the correct OS-thread, it prints out the
hello world message and returns the OS-thread id to hpx::lcos::wait() in hello_world_foreman(). If it is not executing on the correct
OS-thread, it returns a value of -1, which causes hello_world_foreman() to leave the OS-thread id in attendance.

std::size_thello_world_worker(std::size_tdesired){// Returns the OS-thread number of the worker that is running this// PX-thread.std::size_tcurrent=hpx::get_worker_thread_num();if(current==desired){// The PX-thread has been run on the desired OS-thread.charconst*msg="hello world from OS-thread %1% on locality %2%\n";hpx::cout<<(boost::format(msg)%desired%hpx::get_locality_id())<<hpx::flush;returndesired;}// This PX-thread has been run by the wrong OS-thread, make the foreman// try again by rescheduling it.returnstd::size_t(-1);}// Define the boilerplate code necessary for the function 'hello_world_worker'// to be invoked as an HPX action (by a HPX future).typedefhpx::actions::plain_result_action1<std::size_t,// return typestd::size_t,// argumenthello_world_worker// wrapped function>hello_world_worker_action;HPX_REGISTER_PLAIN_ACTION(hello_world_worker_action);

Because HPX features work stealing task schedulers, there is no way to
guarantee that an action will be scheduled on a particular OS-thread. This
is why we must use a guess-and-check approach.