StarPU Handbook

Data Structures | Macros | Functions
Initialization and Termination

## Data Structures
struct | starpu_conf

## Macros
#define | STARPU_THREAD_ACTIVE

## Functions
int | starpu_conf_init (struct starpu_conf *conf)
int | starpu_init (struct starpu_conf *conf) STARPU_WARN_UNUSED_RESULT
int | starpu_initialize (struct starpu_conf *user_conf, int *argc, char ***argv)
int | starpu_is_initialized (void)
void | starpu_wait_initialized (void)
void | starpu_shutdown (void)
void | starpu_pause (void)
void | starpu_resume (void)
unsigned | starpu_get_next_bindid (unsigned flags, unsigned *preferred, unsigned npreferred)
int | starpu_bind_thread_on (int cpuid, unsigned flags, const char *name)
void | starpu_topology_print (FILE *f)
int | starpu_asynchronous_copy_disabled (void)
int | starpu_asynchronous_cuda_copy_disabled (void)
int | starpu_asynchronous_opencl_copy_disabled (void)
int | starpu_asynchronous_mic_copy_disabled (void)
int | starpu_asynchronous_mpi_ms_copy_disabled (void)
void | starpu_display_stats (void)

## Detailed Description

## Data Structure Documentation

## ◆ starpu_conf

struct starpu_conf

Structure passed to the starpu_init() function to configure StarPU. It has to be initialized with starpu_conf_init(). When the default value is used, StarPU automatically selects the number of processing units and takes the default scheduling policy. The environment variables overwrite the equivalent parameters.

### Data Fields
const char * | sched_policy_name
struct starpu_sched_policy * | sched_policy
void(* | sched_policy_init )(unsigned)
int | precedence_over_environment_variables
int | ncpus
int | reserve_ncpus
int | ncuda
int | nopencl
int | nmic
int | nmpi_ms
unsigned | use_explicit_workers_bindid
unsigned | workers_bindid [STARPU_NMAXWORKERS]
unsigned | use_explicit_workers_cuda_gpuid
unsigned | workers_cuda_gpuid [STARPU_NMAXWORKERS]
unsigned | use_explicit_workers_opencl_gpuid
unsigned | workers_opencl_gpuid [STARPU_NMAXWORKERS]
unsigned | use_explicit_workers_mic_deviceid
unsigned | workers_mic_deviceid [STARPU_NMAXWORKERS]
unsigned | use_explicit_workers_mpi_ms_deviceid
unsigned | workers_mpi_ms_deviceid [STARPU_NMAXWORKERS]
int | bus_calibrate
int | calibrate
int | single_combined_worker
char * | mic_sink_program_path
int | disable_asynchronous_copy
int | disable_asynchronous_cuda_copy
int | disable_asynchronous_opencl_copy
int | disable_asynchronous_mic_copy
int | disable_asynchronous_mpi_ms_copy
unsigned * | cuda_opengl_interoperability
unsigned | n_cuda_opengl_interoperability
struct starpu_driver * | not_launched_drivers
unsigned | n_not_launched_drivers
uint64_t | trace_buffer_size
int | global_sched_ctx_min_priority
int | global_sched_ctx_max_priority
void(* | callback_worker_going_to_sleep )(unsigned workerid)
void(* | callback_worker_waking_up )(unsigned workerid)
int | catch_signals
unsigned | driver_spinning_backoff_min
unsigned | driver_spinning_backoff_max
### Private Attributes
int | magic
int | will_use_mpi

#### Field Documentation

## ◆ magic

int starpu_conf::magic
| private

Will be initialized by starpu_conf_init(). Should not be set by hand.

## ◆ will_use_mpi

int starpu_conf::will_use_mpi
| private

Tell starpu_init() if MPI will be initialized later.

## ◆ sched_policy_name

const char* starpu_conf::sched_policy_name

Name of the scheduling policy. This can also be specified with the environment variable STARPU_SCHED. (default = `NULL`).

## ◆ sched_policy

struct starpu_sched_policy* starpu_conf::sched_policy

Definition of the scheduling policy. This field is ignored if starpu_conf::sched_policy_name is set. (default = `NULL`)

## ◆ precedence_over_environment_variables

int starpu_conf::precedence_over_environment_variables

For all parameters specified in this structure that can also be set with environment variables, by default, StarPU chooses the value of the environment variable against the value set in starpu_conf. Setting the parameter starpu_conf::precedence_over_environment_variables to 1 allows to give precedence to the value set in the structure over the environment variable.

## ◆ ncpus

int starpu_conf::ncpus

Number of CPU cores that StarPU can use. This can also be specified with the environment variable STARPU_NCPU. (default = -1)

## ◆ reserve_ncpus

int starpu_conf::reserve_ncpus

Number of CPU cores to that StarPU should leave aside. They can then be used by application threads, by calling starpu_get_next_bindid() to get their ID, and starpu_bind_thread_on() to bind the current thread to them.

## ◆ ncuda

int starpu_conf::ncuda

Number of CUDA devices that StarPU can use. This can also be specified with the environment variable STARPU_NCUDA. (default = -1)

## ◆ nopencl

int starpu_conf::nopencl

Number of OpenCL devices that StarPU can use. This can also be specified with the environment variable STARPU_NOPENCL. (default = -1)

## ◆ nmic

int starpu_conf::nmic

Number of MIC devices that StarPU can use. This can also be specified with the environment variable STARPU_NMIC. (default = -1)

## ◆ nmpi_ms

int starpu_conf::nmpi_ms

Number of MPI Master Slave devices that StarPU can use. This can also be specified with the environment variable STARPU_NMPI_MS. (default = -1)

## ◆ use_explicit_workers_bindid

unsigned starpu_conf::use_explicit_workers_bindid

If this flag is set, the starpu_conf::workers_bindid array indicates where the different workers are bound, otherwise StarPU automatically selects where to bind the different workers. This can also be specified with the environment variable STARPU_WORKERS_CPUID. (default = 0)

## ◆ workers_bindid

unsigned starpu_conf::workers_bindid[STARPU_NMAXWORKERS]

If the starpu_conf::use_explicit_workers_bindid flag is set, this array indicates where to bind the different workers. The i-th entry of the starpu_conf::workers_bindid indicates the logical identifier of the processor which should execute the i-th worker. Note that the logical ordering of the CPUs is either determined by the OS, or provided by the hwloc library in case it is available.

## ◆ use_explicit_workers_cuda_gpuid

unsigned starpu_conf::use_explicit_workers_cuda_gpuid

If this flag is set, the CUDA workers will be attached to the CUDA devices specified in the starpu_conf::workers_cuda_gpuid array. Otherwise, StarPU affects the CUDA devices in a round-robin fashion. This can also be specified with the environment variable STARPU_WORKERS_CUDAID. (default = 0)

## ◆ workers_cuda_gpuid

unsigned starpu_conf::workers_cuda_gpuid[STARPU_NMAXWORKERS]

If the starpu_conf::use_explicit_workers_cuda_gpuid flag is set, this array contains the logical identifiers of the CUDA devices (as used by `cudaGetDevice()`).

## ◆ use_explicit_workers_opencl_gpuid

unsigned starpu_conf::use_explicit_workers_opencl_gpuid

If this flag is set, the OpenCL workers will be attached to the OpenCL devices specified in the starpu_conf::workers_opencl_gpuid array. Otherwise, StarPU affects the OpenCL devices in a round-robin fashion. This can also be specified with the environment variable STARPU_WORKERS_OPENCLID. (default = 0)

## ◆ workers_opencl_gpuid

unsigned starpu_conf::workers_opencl_gpuid[STARPU_NMAXWORKERS]

If the starpu_conf::use_explicit_workers_opencl_gpuid flag is set, this array contains the logical identifiers of the OpenCL devices to be used.

## ◆ use_explicit_workers_mic_deviceid

unsigned starpu_conf::use_explicit_workers_mic_deviceid

If this flag is set, the MIC workers will be attached to the MIC devices specified in the array starpu_conf::workers_mic_deviceid. Otherwise, StarPU affects the MIC devices in a round-robin fashion. This can also be specified with the environment variable STARPU_WORKERS_MICID. (default = 0)

## ◆ workers_mic_deviceid

unsigned starpu_conf::workers_mic_deviceid[STARPU_NMAXWORKERS]

If the flag starpu_conf::use_explicit_workers_mic_deviceid is set, the array contains the logical identifiers of the MIC devices to be used.

## ◆ use_explicit_workers_mpi_ms_deviceid

unsigned starpu_conf::use_explicit_workers_mpi_ms_deviceid

If this flag is set, the MPI Master Slave workers will be attached to the MPI Master Slave devices specified in the array starpu_conf::workers_mpi_ms_deviceid. Otherwise, StarPU affects the MPI Master Slave devices in a round-robin fashion. (default = 0)

## ◆ workers_mpi_ms_deviceid

unsigned starpu_conf::workers_mpi_ms_deviceid[STARPU_NMAXWORKERS]

If the flag starpu_conf::use_explicit_workers_mpi_ms_deviceid is set, the array contains the logical identifiers of the MPI Master Slave devices to be used.

## ◆ bus_calibrate

int starpu_conf::bus_calibrate

If this flag is set, StarPU will recalibrate the bus. If this value is equal to -1, the default value is used. This can also be specified with the environment variable STARPU_BUS_CALIBRATE. (default = 0)

## ◆ calibrate

int starpu_conf::calibrate

If this flag is set, StarPU will calibrate the performance models when executing tasks. If this value is equal to -1, the default value is used. If the value is equal to 1, it will force continuing calibration. If the value is equal to 2, the existing performance models will be overwritten. This can also be specified with the environment variable STARPU_CALIBRATE. (default = 0)

## ◆ single_combined_worker

int starpu_conf::single_combined_worker

By default, StarPU executes parallel tasks concurrently. Some parallel libraries (e.g. most OpenMP implementations) however do not support concurrent calls to parallel code. In such case, setting this flag makes StarPU only start one parallel task at a time (but other CPU and GPU tasks are not affected and can be run concurrently). The parallel task scheduler will however still try varying combined worker sizes to look for the most efficient ones. This can also be specified with the environment variable STARPU_SINGLE_COMBINED_WORKER. (default = 0)

## ◆ mic_sink_program_path

char* starpu_conf::mic_sink_program_path

Path to the kernel to execute on the MIC device, compiled for MIC architecture. When set to `NULL`, StarPU automatically looks next to the host program location. (default = `NULL`)

## ◆ disable_asynchronous_copy

int starpu_conf::disable_asynchronous_copy

This flag should be set to 1 to disable asynchronous copies between CPUs and all accelerators. The AMD implementation of OpenCL is known to fail when copying data asynchronously. When using this implementation, it is therefore necessary to disable asynchronous data transfers. This can also be specified with the environment variable STARPU_DISABLE_ASYNCHRONOUS_COPY. This can also be specified at compilation time by giving to the configure script the option --disable-asynchronous-copy. (default = 0)

## ◆ disable_asynchronous_cuda_copy

int starpu_conf::disable_asynchronous_cuda_copy

This flag should be set to 1 to disable asynchronous copies between CPUs and CUDA accelerators. This can also be specified with the environment variable STARPU_DISABLE_ASYNCHRONOUS_CUDA_COPY. This can also be specified at compilation time by giving to the configure script the option --disable-asynchronous-cuda-copy. (default = 0)

## ◆ disable_asynchronous_opencl_copy

int starpu_conf::disable_asynchronous_opencl_copy

This flag should be set to 1 to disable asynchronous copies between CPUs and OpenCL accelerators. The AMD implementation of OpenCL is known to fail when copying data asynchronously. When using this implementation, it is therefore necessary to disable asynchronous data transfers. This can also be specified with the environment variable STARPU_DISABLE_ASYNCHRONOUS_OPENCL_COPY. This can also be specified at compilation time by giving to the configure script the option --disable-asynchronous-opencl-copy. (default = 0)

## ◆ disable_asynchronous_mic_copy

int starpu_conf::disable_asynchronous_mic_copy

This flag should be set to 1 to disable asynchronous copies between CPUs and MIC accelerators. This can also be specified with the environment variable STARPU_DISABLE_ASYNCHRONOUS_MIC_COPY. This can also be specified at compilation time by giving to the configure script the option --disable-asynchronous-mic-copy. (default = 0).

## ◆ disable_asynchronous_mpi_ms_copy

int starpu_conf::disable_asynchronous_mpi_ms_copy

This flag should be set to 1 to disable asynchronous copies between CPUs and MPI Master Slave devices. This can also be specified with the environment variable STARPU_DISABLE_ASYNCHRONOUS_MPI_MS_COPY. This can also be specified at compilation time by giving to the configure script the option --disable-asynchronous-mpi-master-slave-copy. (default = 0).

## ◆ cuda_opengl_interoperability

unsigned* starpu_conf::cuda_opengl_interoperability

Enable CUDA/OpenGL interoperation on these CUDA devices. This can be set to an array of CUDA device identifiers for which `cudaGLSetGLDevice()` should be called instead of `cudaSetDevice()`. Its size is specified by the starpu_conf::n_cuda_opengl_interoperability field below (default = `NULL`)

## ◆ n_cuda_opengl_interoperability

unsigned starpu_conf::n_cuda_opengl_interoperability

Size of the array starpu_conf::cuda_opengl_interoperability

## ◆ not_launched_drivers

struct starpu_driver* starpu_conf::not_launched_drivers

Array of drivers that should not be launched by StarPU. The application will run in one of its own threads. (default = `NULL`)

## ◆ n_not_launched_drivers

unsigned starpu_conf::n_not_launched_drivers

The number of StarPU drivers that should not be launched by StarPU, i.e number of elements of the array starpu_conf::not_launched_drivers. (default = 0)

## ◆ trace_buffer_size

uint64_t starpu_conf::trace_buffer_size

Specify the buffer size used for FxT tracing. Starting from FxT version 0.2.12, the buffer will automatically be flushed when it fills in, but it may still be interesting to specify a bigger value to avoid any flushing (which would disturb the trace).

## ◆ catch_signals

int starpu_conf::catch_signals

Specify if StarPU should catch SIGINT, SIGSEGV and SIGTRAP signals to make sure final actions (e.g dumping FxT trace files) are done even though the application has crashed. By default (value = `1`), signals are catched. It should be disabled on systems which already catch these signals for their own needs (e.g JVM) This can also be specified with the environment variable STARPU_CATCH_SIGNALS

## ◆ driver_spinning_backoff_min

unsigned starpu_conf::driver_spinning_backoff_min

Minimum spinning backoff of drivers. Default value: `1`

## ◆ driver_spinning_backoff_max

unsigned starpu_conf::driver_spinning_backoff_max

Maximum spinning backoff of drivers. Default value: `32`

## Macro Definition Documentation

## ◆ STARPU_THREAD_ACTIVE

#define STARPU_THREAD_ACTIVE

Value to be passed to starpu_get_next_bindid() and starpu_bind_thread_on() when binding a thread which will significantly eat CPU time, and should thus have its own dedicated CPU.

## Function Documentation

## ◆ starpu_conf_init()

int starpu_conf_init | ( | struct starpu_conf * | conf | ) |

Initialize the `conf` structure with the default values. In case some configuration parameters are already specified through environment variables, starpu_conf_init() initializes the fields of `conf` according to the environment variables. For instance if STARPU_CALIBRATE is set, its value is put in the field starpu_conf::calibrate of `conf`. Upon successful completion, this function returns 0. Otherwise, `-EINVAL` indicates that the argument was `NULL`.

## ◆ starpu_init()

int starpu_init | ( | struct starpu_conf * | conf | ) |

StarPU initialization method, must be called prior to any other StarPU call. It is possible to specify StarPU’s configuration (e.g. scheduling policy, number of cores, ...) by passing a non-`NULL` `conf`. Default configuration is used if `conf` is `NULL`. Upon successful completion, this function returns 0. Otherwise, `-ENODEV` indicates that no worker was available (and thus StarPU was not initialized).

## ◆ starpu_initialize()

int starpu_initialize | ( | struct starpu_conf * | user_conf,
|  | int * | argc,
|  | char *** | argv
| ) |  |

Similar to starpu_init(), but also take the `argc` and `argv` as defined by the application. Do not call starpu_init() and starpu_initialize() in the same program.

## ◆ starpu_is_initialized()

int starpu_is_initialized | ( | void |  | ) |

Return 1 if StarPU is already initialized.

## ◆ starpu_wait_initialized()

void starpu_wait_initialized | ( | void |  | ) |

Wait for starpu_init() call to finish.

## ◆ starpu_shutdown()

void starpu_shutdown | ( | void |  | ) |

StarPU termination method, must be called at the end of the application: statistics and other post-mortem debugging information are not guaranteed to be available until this method has been called.

## ◆ starpu_pause()

void starpu_pause | ( | void |  | ) |

Suspend the processing of new tasks by workers. It can be used in a program where StarPU is used during only a part of the execution. Without this call, the workers continue to poll for new tasks in a tight loop, wasting CPU time. The symmetric call to starpu_resume() should be used to unfreeze the workers.

## ◆ starpu_resume()

void starpu_resume | ( | void |  | ) |

Symmetrical call to starpu_pause(), used to resume the workers polling for new tasks. This would be typically called only once having submitted all tasks.

## ◆ starpu_get_next_bindid()

unsigned starpu_get_next_bindid | ( | unsigned | flags,
|  | unsigned * | preferred,
|  | unsigned | npreferred
| ) |  |

Return a PU binding ID which can be used to bind threads with starpu_bind_thread_on(). `flags` can be set to STARPU_THREAD_ACTIVE or 0. When `npreferred` is set to non-zero, `preferred` is an array of size `npreferred` in which a preference of PU binding IDs can be set. By default StarPU will return the first PU available for binding.

## ◆ starpu_bind_thread_on()

int starpu_bind_thread_on | ( | int | cpuid,
|  | unsigned | flags,
|  | const char * | name
| ) |  |

Bind the calling thread on the given `cpuid` (which should have been obtained with starpu_get_next_bindid()).

Return -1 if a thread was already bound to this PU (but binding will still have been done, and a warning will have been printed), so the caller can tell the user how to avoid the issue.

`name` should be set to a unique string so that different calls with the same name for the same cpuid does not produce a warning.

## ◆ starpu_topology_print()

void starpu_topology_print | ( | FILE * | f | ) |

Print a description of the topology on `f`.

## ◆ starpu_asynchronous_copy_disabled()

int starpu_asynchronous_copy_disabled | ( | void |  | ) |

Return 1 if asynchronous data transfers between CPU and accelerators are disabled.

## ◆ starpu_asynchronous_cuda_copy_disabled()

int starpu_asynchronous_cuda_copy_disabled | ( | void |  | ) |

Return 1 if asynchronous data transfers between CPU and CUDA accelerators are disabled.

## ◆ starpu_asynchronous_opencl_copy_disabled()

int starpu_asynchronous_opencl_copy_disabled | ( | void |  | ) |

Return 1 if asynchronous data transfers between CPU and OpenCL accelerators are disabled.

## ◆ starpu_asynchronous_mic_copy_disabled()

int starpu_asynchronous_mic_copy_disabled | ( | void |  | ) |

Return 1 if asynchronous data transfers between CPU and MIC devices are disabled.

## ◆ starpu_asynchronous_mpi_ms_copy_disabled()

int starpu_asynchronous_mpi_ms_copy_disabled | ( | void |  | ) |

Return 1 if asynchronous data transfers between CPU and MPI Slave devices are disabled.