Exercise 2b: NBA Play-offs on Iridis

Parallel R workflows on Iridis

Now let’s move on to running our NBA playoffs exercise on Iridis.

This is a more complex example that involves both sequential and parallel elements, the results of which need to be brought back together to get a final result.

We also have nested parallelisation within our workflow.

In all honesty, our small job could easily be run using the code as is on a single node by just requesting an appropriate number of cores (8 in total).

However we will take this opportunity to explore how we can deploy such non-independent jobs across multiple nodes (if necessary) and parallelise across cores within them.

This can be achieved through a job array as we’ve just seen. However that doesn’t work for inter-dependent jobs like this one (however, see the section on Job Dependency in the Iridis docs for how to set up separate jobs that depend on each others outputs).

To do so, we will be using package future.batchtools to schedule new slurm jobs from within our R script, the results of which will be returned to the main calling process when they finish.

`future.batchtools`

The package future.batchtools, provides a type of parallelisation that utilizes the batchtools package, which in turn provides an API for submitting jobs through to a variety of HPC back-ends, including slurm.

future.batchtools allow users to leverage the compute power of high-performance computing (HPC) clusters via a simple switch in settings - without having to change any code at all.

For instance, if batchtools is properly configured, the below two expressions for futures x and y will be processed on two different compute nodes on an HPC cluster:

library(future.batchtools)
plan(batchtools_slurm)

x %<-% { Sys.sleep(5); 3.14 }
y %<-% { Sys.sleep(5); 2.71 }
x + y

[1] 5.85

Implement parallelisation through `future.batchtools`

First, if connected through the RStudio terminal, let’s exit Iridis to work locally:

exit

Now let’s go ahead and set up our code so that the outer parallelisation layer is dispatched via future.batchtools.

Because this will cause our code to not be runnable outside of a slurm scheduling environment, let’s make a copy of

Copy `nba-playoffs-future_apply.R`

Let’s make a copy of nba-playoffs-future_apply.R for us to edit in the same nba/ directory and name it nba-playoffs-slurm.R.

Next, let’s start editing nba-playoffs-slurm.R.

Important

Make sure you are editing nba-playoffs-slurm.R and not nba-playoffs-future_apply.R. To be sure it might be easiest to close nba-playoffs-future_apply.R.

Modify `nba-playoffs-slurm.R`

To utilise future.batchtools for parallelisation, above the code where we iterate over play_conference, replace:

plan(list(tweak(multisession, workers = outer_cores), 
          tweak(multisession, workers = I(inner_cores))))

with:

plan(list(
  tweak(future.batchtools::batchtools_slurm,
    template = here::here(
      "nba", "slurm",
      "batchtools.slurm.tmpl"
    ),
    resources = list(
      ncpus = 4,
      memory = "1GB",
      walltime = 180
    )
  ),
  multisession
))

Let’s examine what this plan is doing:

The first level of parallelisation will be handled through future.batchtools::batchtools_slurm which will submit additional slurm jobs for the outer level of parallelisation (each conference). It will therefore submit 2 additional jobs.

Each job will be submitted using a template slurm script, specified by argument template, in this case it is file nba/slurm/batchtools.slurm.tmpl. The list of values in the resources argument are passed to the template slurm script.

In this case we are asking for 4 CPU cores, 1GB of memory and 180s clock time for each additional job.

Because the next level of parallelisation will be within a separate job (likely on a separate node), we do not need to tweak multisession as it will parallelise across all available cores by default.

`batchools` submission templates

Let’s examine the contents of nba/slurm/batchtools.slurm.tmpl to see what it does:

batchtools.slurm.tmpl

#!/bin/bash

## Job Resource Interface Definition
##
## ntasks [integer(1)]:       Number of required tasks,
##                            Set larger than 1 if you want to further parallelize
##                            with MPI within your job.
## ncpus [integer(1)]:        Number of required cpus per task,
##                            Set larger than 1 if you want to further parallelize
##                            with multicore/parallel within each task.
## walltime [integer(1)]:     Walltime for this job, in seconds.
##                            Must be at least 1 minute.
## memory   [integer(1)]:     Memory in megabytes for each cpu.
##                            Must be at least 100 (when I tried lower values my
##                            jobs did not start at all).
##
## Default resources can be set in your .batchtools.conf.R by defining the variable
## 'default.resources' as a named list.

<%
# relative paths are not handled well by Slurm
log.file = shQuote(fs::path_expand(log.file))
-%>


#SBATCH --job-name=<%= shQuote(job.name) %>
#SBATCH --output=<%= log.file %>
#SBATCH --error=<%= log.file %>
#SBATCH --nodes=1
#SBATCH --time=<%= ceiling(resources$walltime / 60) %>
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=<%= resources$ncpus %>
#SBATCH --mem-per-cpu=<%= resources$memory %>
<%= if (!is.null(resources$partition)) sprintf(paste0("#SBATCH --partition='", resources$partition, "'")) %>
<%= if (!is.null(resources$afterok)) paste0("#SBATCH --depend=afterok:", resources$afterok) %>
<%= if (array.jobs) sprintf("#SBATCH --array=1-%i", nrow(jobs)) else "" %>

## Initialize work environment like
## source /etc/profile
## module add ...

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

## Run R:
## we merge R output with stdout from SLURM, which gets then logged via --output option
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

The top part of the script are just comments about the resources arguments it accepts.

Following that you will see a familiar by now block of #SBATCH comments. What’s different though is that they include <%= ... %> snippets. These execute R code and enable populating the script through providing arguments to resources in R.

Let’s not worry about the rest for now.

The last line calls

Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

which is the batchtools function for collecting information and launching a slurm batch job.

Tip

For more details on how templates can be supplied see the entry for argument template in the following docs.

`nba-playoffs.slurm` submission file

Let’s also examine the contents of nba/slurm/nba-playoffs.slurm which is the file we will use to submit our NBA play-offs job:

nba-playoffs.slurm

#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --mem-per-cpu=500
#SBATCH --job-name=nba-playoffs
#SBATCH --time=00:03:00
#SBATCH --output=logs/R-%x.%j.out
#SBATCH --error=logs/R-%x.%j.err
#SBATCH --export=ALL,TZ=Europe/London
#SBATCH --mail-type=ALL
#SBATCH --reservation=ParallelR_Workshop  # Specify the reservation name

# send mail to this address
#SBATCH --mail-user=youremail@here.com

module load R/4.4.1-gcc
module load openmpi/5.0.0/gcc

cd parallel-r-materials/
Rscript nba/nba-playoffs-slurm.R

This should be pretty familiar to you now.

Note we are only requesting 2 cores for the initial submission node, enough to parallelise the qualifiers across conferences. The rest will be provisioned by future.batchtools.

Another thing to note is that we are not loading all the additional geospatial libraries this time as they are not required.

We are, however, loading openmpi/4.1.4/intel in additional to R.

The Open MPI Project is an open source Message Passing Interface (MPI) implementation which enables information to be passed between compute nodes. While not required for running jobs on individual nodes that are independent of each other, it is required if we want jobs across nodes to communicate with each other.

Before we move on though, let’s edit the file and add a valid email address for notifications.

Run NBA Playoffs job on Iridis

Before we head over to Iridis to submit our job, let’s synch our files already on Iridis with the changes we’ve just made to our local files. As always, switch out userid with your own username.

Run the following command either in Rstudio terminal on Linux/macOS or in your local shell session on mobaXterm.

rsync -hav --exclude={'.*/','*/outputs/*'} --progress ./* userid@iridis5.soton.ac.uk:/home/userid/parallel-r-materials/

Now let’s log into Iridis and submit our NBA Playoffs job!

sbatch parallel-r-materials/nba/slurm/nba-playoffs.slurm

Monitor our job

Let’s use squeue to monitor our job:

squeue -lu userid

First you will notice a single job running:

  JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
3374907    serial nba-play   ak1f23  RUNNING       0:05      3:00      1 gold51

However, eventually this will spawn another two jobs. That’s batchtools in action! The name of the job is the calling function which generated the job, in our case future_lapply.

Note as well that, at least in my case, the two future_lapply jobs are running on two separate nodes. This might not always be the case if slurm can fit both jobs on the same node but it ensures that the jobs won’t stall if they can’t be executed on a single node. They will simply be dispatched to a different node.

  
  JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
3374909    serial future_l   ak1f23  RUNNING       0:04      3:00      1 gold52
3374908    serial future_l   ak1f23  RUNNING       0:07      3:00      1 gold51
3374907    serial nba-play   ak1f23  RUNNING       0:25      3:00      1 gold51

The two future_lapply jobs will eventually complete. Their results are collected by the initial job (nba-play offs), where the overall finals will be played and the overall playoffs winner will be announced!

  JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
3374909    serial future_l   ak1f23 COMPLETI       0:18      3:00      1 gold52
3374908    serial future_l   ak1f23 COMPLETI       0:15      3:00      1 gold51
3374907    serial nba-play   ak1f23  RUNNING       0:33      3:00      1 gold51

  JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
3374909    serial future_l   ak1f23 COMPLETI       0:18      3:00      1 gold52
3374908    serial future_l   ak1f23 COMPLETI       0:15      3:00      1 gold51
3374907    serial nba-play   ak1f23 COMPLETI       0:35      3:00      1 gold51

Check logs

Once our jobs are all complete, we can go ahead and review our log files:

`.err` file

Let’s go ahead and review the results of our playoffs by looking at the .err file.

cat "$(ls -rt logs/*.err| tail -n1)"

Loading compiler version 2021.2.0
Loading mkl version 2021.2.0
Removing compiler version 2021.2.0
Loading compiler version 2021.2.0
Loading required package: future
Rows: 30 Columns: 9
-- Column specification --------------------------------------------------------
Delimiter: ","
chr (6): name_team, slug_team, url_team_season_logo, city_team, colors_team,...
dbl (3): id_team, prop_win, id_conference

i Use `spec()` to retrieve the full column specification for this data.
i Specify the column types or set `show_col_types = FALSE` to quiet this message.

-- Qualifiers have begun! ------------------------------------------------------

-- Playing qualifiers for conference 1 on `66324` --

v Conference 1 qualifying round complete

-- Playing qualifiers for conference 2 on `66323` --

v Conference 2 qualifying round complete

-- ALL Qualifying matches COMPLETE! --

-- Conference Rounds have begun! -----------------------------------------------

-- Conference Round 1 (conf 1) started! --


-- Playing Conference Round 1 (conf 1) game: `WAS` VS `IND` 
i Game location: 66782 (gold51.cluster.local)
i WAS VS IND match complete in 120 minutes
v Winner: IND

-- Playing Conference Round 1 (conf 1) game: `MIA` VS `MIL` 
i Game location: 66780 (gold51.cluster.local)
i MIA VS MIL match complete in 120 minutes
v Winner: MIL

-- Playing Conference Round 1 (conf 1) game: `BKN` VS `PHI` 
i Game location: 66783 (gold51.cluster.local)
i BKN VS PHI match complete in 120 minutes
v Winner: PHI

-- Playing Conference Round 1 (conf 1) game: `ORL` VS `TOR` 
i Game location: 66781 (gold51.cluster.local)
i ORL VS TOR match complete in 120 minutes
v Winner: TOR
-- Conference Round 1 (conf 1) COMPLETE! --

-- Conference Semi finals (conf 1) started! --


-- Playing Conference Semi finals (conf 1) game: `IND` VS `PHI` 
i Game location: 66782 (gold51.cluster.local)
i IND VS PHI match complete in 120 minutes
v Winner: PHI

-- Playing Conference Semi finals (conf 1) game: `MIL` VS `TOR` 
i Game location: 66780 (gold51.cluster.local)
i MIL VS TOR match complete in 120 minutes
v Winner: MIL
-- Conference Semi finals (conf 1) COMPLETE! --

-- Conference finals (conf 1) started! --


-- Playing Conference finals (conf 1) game: `PHI` VS `MIL` 
i Game location: 66782 (gold51.cluster.local)
i PHI VS MIL match complete in 120 minutes
v Winner: MIL
-- Conference finals (conf 1) COMPLETE! --


-- Conference Round 1 (conf 2) started! --


-- Playing Conference Round 1 (conf 2) game: `POR` VS `OKC` 
i Game location: 57374 (gold52.cluster.local)
i POR VS OKC match complete in 120 minutes
v Winner: POR

-- Playing Conference Round 1 (conf 2) game: `NOP` VS `MIN` 
i Game location: 57372 (gold52.cluster.local)
i NOP VS MIN match complete in 120 minutes
v Winner: NOP

-- Playing Conference Round 1 (conf 2) game: `LAC` VS `PHX` 
i Game location: 57375 (gold52.cluster.local)
i LAC VS PHX match complete in 120 minutes
v Winner: LAC

-- Playing Conference Round 1 (conf 2) game: `DEN` VS `UTA` 
i Game location: 57373 (gold52.cluster.local)
i DEN VS UTA match complete in 132.5 minutes
v Winner: DEN
-- Conference Round 1 (conf 2) COMPLETE! --

-- Conference Semi finals (conf 2) started! --


-- Playing Conference Semi finals (conf 2) game: `LAC` VS `DEN` 
i Game location: 57374 (gold52.cluster.local)
i LAC VS DEN match complete in 120 minutes
v Winner: DEN

-- Playing Conference Semi finals (conf 2) game: `POR` VS `NOP` 
i Game location: 57372 (gold52.cluster.local)
i POR VS NOP match complete in 145 minutes
v Winner: POR
-- Conference Semi finals (conf 2) COMPLETE! --

-- Conference finals (conf 2) started! --


-- Playing Conference finals (conf 2) game: `DEN` VS `POR` 
i Game location: 57374 (gold52.cluster.local)
i DEN VS POR match complete in 120 minutes
v Winner: POR
-- Conference finals (conf 2) COMPLETE! --


-- ALL Conference matches COMPLETE! --

-- Overall Playoff final has begun! --------------------------------------------

-- Play off Finals started! --

-- Playing Play off Finals game: `MIL` VS `POR` 
i Game location: 66074 (gold51.cluster.local)
i MIL VS POR match complete in 145 minutes
v Winner: MIL

-- Play off Finals COMPLETE! --

-- Playoffs complete! ----------------------------------------------------------
v Winner: Milwaukee Bucks (MIL)

Note that only a single .err file has been created for our job in which the messages from all jobs have been compiled.

We see the same winner being announced, showing that our seeds have been propagated successfully and effortlessly across jobs!

By reviewing our messages we can also confirm that conference rounds for each conference were indeed run on separate nodes (gold51 and gold52 respectfully) and within each node, each match within a given round was played in parallel.

We can see this more easily by reviewing the playoff_results.csv file.

Let’s use Rscript to print the contents of playoff_results.csv to the command line.

First load R:

module load R

Now we can use Rscript to execute an R expression by including flag -e

Rscript -e "read.csv('parallel-r-materials/nba/outputs/playoff_results.csv')"

Rows: 15 Columns: 9
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr  (5): winner, team_1, team_2, node, round_name
dbl  (3): pid, game_length, conf
dttm (1): date

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

`.out` file

We can also check our .out logs file.

cat "$(ls -rt logs/*.out| tail -n1)"

Running SLURM prolog script on gold51.cluster.local
===============================================================================
Job started on Mon May 29 09:52:58 BST 2023
Job ID          : 3374907
Job name        : nba-playoffs
WorkDir         : /mainfs/home/ak1f23
Command         : /mainfs/home/ak1f23/parallel-r-materials/nba/slurm/nba-playoffs.slurm
Partition       : serial
Num hosts       : 1
Num cores       : 2
Num of tasks    : 2
Hosts allocated : gold51
Job Output Follows ...
===============================================================================
Total Play-offs Duration: 33.721 sec elapsed
==============================================================================
Running epilogue script on gold51.

Submit time  : 2023-05-29T09:52:56
Start time   : 2023-05-29T09:52:56
End time     : 2023-05-29T09:53:38
Elapsed time : 00:00:42 (Timelimit=00:03:00)

Job ID: 3374907
Cluster: i5
User/Group: ak1f23/jf
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 2
CPU Utilized: 00:00:08
CPU Efficiency: 9.52% of 00:01:24 core-walltime
Job Wall-clock time: 00:00:42
Memory Utilized: 119.81 MB
Memory Efficiency: 11.98% of 1000.00 MB

Note that this lists information about the initial job submitted, not the sub-jobs submitted by batch.tools.

Transferring results from Iridis

Finally let’s transfer our Playoffs results from Iridis to our local machine.

First let’s, if working in Rstudio, let’s exit Iridis

exit

If working in mobaXterm, move to your local shell session.

Next we ask rsync to transfer the nba/outputs on Iridis to our local nba directory.

rsync -hav userid@iridis5.soton.ac.uk:/home/userid/parallel-r-materials/nba/outputs  nba

rsync only transfers files that are not in sync, namely the playoff_results.csv file.

receiving file list ... done
outputs/
outputs/playoff_results.csv

sent 44 bytes  received 1672 bytes  1144.00 bytes/sec
total size is 1505  speedup is 0.88

Let’s go ahead and read in the csv and View it in Rstudio.

View(readr::read_csv(here::here("nba", "outputs", "playoff_results.csv")))

Rows: 15 Columns: 9
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr  (5): winner, team_1, team_2, node, round_name
dbl  (3): pid, game_length, conf
dttm (1): date

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

# A tibble: 15 × 9
   winner team_1 team_2   pid node         game_length date                 conf
   <chr>  <chr>  <chr>  <dbl> <chr>              <dbl> <dttm>              <dbl>
 1 IND    WAS    IND    66782 gold51.clus…        120  2023-05-29 09:53:20     1
 2 MIL    MIA    MIL    66780 gold51.clus…        120  2023-05-29 09:53:20     1
 3 PHI    BKN    PHI    66783 gold51.clus…        120  2023-05-29 09:53:20     1
 4 TOR    ORL    TOR    66781 gold51.clus…        120  2023-05-29 09:53:21     1
 5 PHI    IND    PHI    66782 gold51.clus…        120  2023-05-29 09:53:23     1
 6 MIL    MIL    TOR    66780 gold51.clus…        120  2023-05-29 09:53:24     1
 7 POR    POR    OKC    57374 gold52.clus…        120  2023-05-29 09:53:26     2
 8 NOP    NOP    MIN    57372 gold52.clus…        120  2023-05-29 09:53:26     2
 9 MIL    PHI    MIL    66782 gold51.clus…        120  2023-05-29 09:53:26     1
10 LAC    LAC    PHX    57375 gold52.clus…        120  2023-05-29 09:53:26     2
11 DEN    DEN    UTA    57373 gold52.clus…        132. 2023-05-29 09:53:27     2
12 DEN    LAC    DEN    57374 gold52.clus…        120  2023-05-29 09:53:30     2
13 POR    POR    NOP    57372 gold52.clus…        145  2023-05-29 09:53:30     2
14 POR    DEN    POR    57374 gold52.clus…        120  2023-05-29 09:53:33     2
15 MIL    MIL    POR    66074 gold51.clus…        145  2023-05-29 09:53:37    NA
# ℹ 1 more variable: round_name <chr>

Summary

We’ve successfully managed to:

Use future.batchtools to submit slurm jobs from within R scripts.
Run our code across multiple nodes with nested parallelisation and collect all results in our initial session

Reuse

CC BY-SA 4.0

future.batchtools

Implement parallelisation through future.batchtools

Copy nba-playoffs-future_apply.R

Modify nba-playoffs-slurm.R

batchools submission templates

nba-playoffs.slurm submission file

Run NBA Playoffs job on Iridis

Monitor our job

Check logs

.err file

.out file

Transferring results from Iridis

Reuse

`future.batchtools`

Implement parallelisation through `future.batchtools`

Copy `nba-playoffs-future_apply.R`

Modify `nba-playoffs-slurm.R`

`batchools` submission templates

`nba-playoffs.slurm` submission file

`.err` file

`.out` file