StarPU Handbook

Frequently Asked Questions

# How To Initialize A Computation Library Once For Each Worker?

Some libraries need to be initialized once for each concurrent instance that may run on the machine. For instance, a C++ computation class which is not thread-safe by itself, but for which several instanciated objects of that class can be used concurrently. This can be used in StarPU by initializing one such object per worker. For instance, the `libstarpufft` example does the following to be able to use FFTW on CPUs.

Some global array stores the instanciated objects:

fftw_plan plan_cpu[STARPU_NMAXWORKERS];
STARPU_NMAXWORKERS
#define STARPU_NMAXWORKERS
Definition: starpu_config.h:236

At initialisation time of libstarpu, the objects are initialized:

int workerid;
for (workerid = 0; workerid < starpu_worker_get_count(); workerid++)
{
switch (starpu_worker_get_type(workerid))
{
case STARPU_CPU_WORKER:
plan_cpu[workerid] = fftw_plan(...);
break;
}
}
starpu_worker_get_type
enum starpu_worker_archtype starpu_worker_get_type(int id)
starpu_worker_get_count
unsigned starpu_worker_get_count(void)
STARPU_CPU_WORKER
@ STARPU_CPU_WORKER
Definition: starpu_worker.h:64

And in the codelet body, they are used:

static void fft(void *descr[], void *_args)
{
int workerid = starpu_worker_get_id();
fftw_plan plan = plan_cpu[workerid];
...
fftw_execute(plan, ...);
}
starpu_worker_get_id
int starpu_worker_get_id(void)

This however is not sufficient for FFT on CUDA: initialization has to be done from the workers themselves. This can be done thanks to starpu_execute_on_each_worker(). For instance `libstarpufft` does the following.

static void fft_plan_gpu(void *args)
{
plan plan = args;
int n2 = plan->n2[0];
int workerid = starpu_worker_get_id();
cufftPlan1d(&plan->plans[workerid].plan_cuda, n, _CUFFT_C2C, 1);
cufftSetStream(plan->plans[workerid].plan_cuda, starpu_cuda_get_local_stream());
}
void starpufft_plan(void)
{
starpu_execute_on_each_worker(fft_plan_gpu, plan, STARPU_CUDA);
}
starpu_cuda_get_local_stream
cudaStream_t starpu_cuda_get_local_stream(void)
STARPU_CUDA
#define STARPU_CUDA
Definition: starpu_task.h:62
starpu_execute_on_each_worker
void starpu_execute_on_each_worker(void(*func)(void *), void *arg, uint32_t where)

# Using The Driver API

Running Drivers

int ret;
struct starpu_driver =
{
.type = STARPU_CUDA_WORKER,
.id.cuda_id = 0
};
ret = starpu_driver_init(&d);
if (ret != 0)
error();
while (some_condition)
{
ret = starpu_driver_run_once(&d);
if (ret != 0)
error();
}
ret = starpu_driver_deinit(&d);
if (ret != 0)
error();
starpu_driver::type
enum starpu_worker_archtype type
Definition: starpu_driver.h:44
starpu_driver_init
int starpu_driver_init(struct starpu_driver *d)
starpu_driver_deinit
int starpu_driver_deinit(struct starpu_driver *d)
starpu_driver_run_once
int starpu_driver_run_once(struct starpu_driver *d)
starpu_driver
Definition: starpu_driver.h:39
STARPU_CUDA_WORKER
@ STARPU_CUDA_WORKER
Definition: starpu_worker.h:65

To add a new kind of device to the structure starpu_driver, one needs to:

1. Add a member to the union starpu_driver::id
2. Modify the internal function `_starpu_launch_drivers()` to make sure the driver is not always launched.
3. Modify the function starpu_driver_run() so that it can handle another kind of architecture.
4. Write the new function `_starpu_run_foobar()` in the corresponding driver.

# On-GPU Rendering

Graphical-oriented applications need to draw the result of their computations, typically on the very GPU where these happened. Technologies such as OpenGL/CUDA interoperability permit to let CUDA directly work on the OpenGL buffers, making them thus immediately ready for drawing, by mapping OpenGL buffer, textures or renderbuffer objects into CUDA. CUDA however imposes some technical constraints: peer memcpy has to be disabled, and the thread that runs OpenGL has to be the one that runs CUDA computations for that GPU.

To achieve this with StarPU, pass the option --disable-cuda-memcpy-peer to `configure` (TODO: make it dynamic), OpenGL/GLUT has to be initialized first, and the interoperability mode has to be enabled by using the field starpu_conf::cuda_opengl_interoperability, and the driver loop has to be run by the application, by using the field starpu_conf::not_launched_drivers to prevent StarPU from running it in a separate thread, and by using starpu_driver_run() to run the loop. The examples `gl_interop` and `gl_interop_idle` show how it articulates in a simple case, where rendering is done in task callbacks. The former uses `glutMainLoopEvent` to make GLUT progress from the StarPU driver loop, while the latter uses `glutIdleFunc` to make StarPU progress from the GLUT main loop.

Then, to use an OpenGL buffer as a CUDA data, StarPU simply needs to be given the CUDA pointer at registration, for instance:

/* Get the CUDA worker id */
for (workerid = 0; workerid < starpu_worker_get_count(); workerid++)
if (starpu_worker_get_type(workerid) == STARPU_CUDA_WORKER)
break;
/* Build a CUDA pointer pointing at the OpenGL buffer */
cudaGraphicsResourceGetMappedPointer((void**)&output, &num_bytes, resource);
/* And register it to StarPU */
starpu_vector_data_register(&handle, starpu_worker_get_memory_node(workerid), output, num_bytes / sizeof(float4), sizeof(float4));
/* The handle can now be used as usual */
starpu_task_insert(&cl, STARPU_RW, handle, 0);
/* ... */
/* This gets back data into the OpenGL buffer */
starpu_data_unregister(handle);
starpu_vector_data_register
void starpu_vector_data_register(starpu_data_handle_t *handle, int home_node, uintptr_t ptr, uint32_t nx, size_t elemsize)
starpu_data_unregister
void starpu_data_unregister(starpu_data_handle_t handle)
STARPU_RW
@ STARPU_RW
Definition: starpu_data.h:59
starpu_task_insert
int starpu_task_insert(struct starpu_codelet *cl,...)
starpu_worker_get_memory_node
unsigned starpu_worker_get_memory_node(unsigned workerid)

and display it e.g. in the callback function.

# Using StarPU With MKL 11 (Intel Composer XE 2013)

Some users had issues with MKL 11 and StarPU (versions 1.1rc1 and 1.0.5) on Linux with MKL, using 1 thread for MKL and doing all the parallelism using StarPU (no multithreaded tasks), setting the environment variable `MKL_NUM_THREADS` to `1`, and using the threaded MKL library, with `iomp5`.

Using this configuration, StarPU only uses 1 core, no matter the value of STARPU_NCPU. The problem is actually a thread pinning issue with MKL.

The solution is to set the environment variable KMP_AFFINITY to `disabled` (http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm).

# Thread Binding on NetBSD

When using StarPU on a NetBSD machine, if the topology discovery library `hwloc` is used, thread binding will fail. To prevent the problem, you should at least use the version 1.7 of `hwloc`, and also issue the following call:

    $ sysctl -w security.models.extensions.user_set_cpu_affinity=1

Or add the following line in the file `/etc/sysctl.conf`

    security.models.extensions.user_set_cpu_affinity=1

# StarPU permanently eats 100% of all CPUs

Yes, this is on purpose.

By default, StarPU uses active polling on task queues, so as to minimize wake-up latency for better overall performance.

If eating CPU time is a problem (e.g. application running on a desktop), pass option --enable-blocking-drivers to `configure`. This will add some overhead when putting CPU workers to sleep or waking them, but avoid eating 100% CPU permanently.

# Interleaving StarPU and non-StarPU code

If your application only partially uses StarPU, and you do not want to call starpu_init() / starpu_shutdown() at the beginning/end of each section, StarPU workers will poll for work between the sections. To avoid this behavior, you can "pause" StarPU with the starpu_pause() function. This will prevent the StarPU workers from accepting new work (tasks that are already in progress will not be frozen), and stop them from polling for more work.

Note that this does not prevent you from submitting new tasks, but they won't execute until starpu_resume() is called. Also note that StarPU must not be paused when you call starpu_shutdown(), and that this function pair works in a push/pull manner, i.e you need to match the number of calls to these functions to clear their effect.

One way to use these functions could be:

starpu_init(NULL);
starpu_pause(); // To submit all the tasks without a single one executing
submit_some_tasks();
starpu_resume(); // The tasks start executing
starpu_task_wait_for_all();
starpu_pause(); // Stop the workers from polling
starpu_resume();
starpu_shutdown();
starpu_task_wait_for_all
int starpu_task_wait_for_all(void)
starpu_shutdown
void starpu_shutdown(void)
starpu_pause
void starpu_pause(void)
starpu_init
int starpu_init(struct starpu_conf *conf) STARPU_WARN_UNUSED_RESULT
starpu_resume
void starpu_resume(void)

# When running with CUDA or OpenCL devices, I am seeing less CPU cores

Yes, this is on purpose.

Since GPU devices are way faster than CPUs, StarPU needs to react quickly when a task is finished, to feed the GPU with another task (StarPU actually submits a couple of tasks in advance so as to pipeline this, but filling the pipeline still has to be happening often enough), and thus it has to dedicate threads for this, and this is a very CPU-consuming duty. StarPU thus dedicates one CPU core for driving each GPU by default.

Such dedication is also useful when a codelet is hybrid, i.e. while kernels are running on the GPU, the codelet can run some computation, which thus be run by the CPU core instead of driving the GPU.

One can choose to dedicate only one thread for all the CUDA devices by setting the STARPU_CUDA_THREAD_PER_DEV environment variable to `1`. The application however should use STARPU_CUDA_ASYNC on its CUDA codelets (asynchronous execution), otherwise the execution of a synchronous CUDA codelet will monopolize the thread, and other CUDA devices will thus starve while it is executing.

# StarPU does not see my CUDA device

First make sure that CUDA is properly running outside StarPU: build and run the following program with `-lcudart` :

#include <stdio.h>
#include <cuda.h>
#include <cuda_runtime.h>
int main(void)
{
int n, i, version;
cudaError_t err;
err = cudaGetDeviceCount(&n);
if (err)
{
fprintf(stderr,"cuda error %d\n", err);
exit(1);
}
cudaDriverGetVersion(&version);
printf("driver version %d\n", version);
cudaRuntimeGetVersion(&version);
printf("runtime version %d\n", version);
printf("\n");
for (i = 0; i < n; i++)
{
struct cudaDeviceProp props;
printf("CUDA%d\n", i);
err = cudaGetDeviceProperties(&props, i);
if (err)
{
fprintf(stderr,"cuda error %d\n", err);
continue;
}
printf("%s\n", props.name);
printf("%0.3f GB\n", (float) props.totalGlobalMem / (1<<30));
printf("%u MP\n", props.multiProcessorCount);
printf("\n");
}
return 0;
}

If that program does not find your device, the problem is not at the StarPU level, but the CUDA drivers, check the documentation of your CUDA setup.

# StarPU does not see my OpenCL device

First make sure that OpenCL is properly running outside StarPU: build and run the following program with `-lOpenCL` :

#include <CL/cl.h>
#include <stdio.h>
#include <assert.h>
int main(void)
{
cl_device_id did[16];
cl_int err;
cl_platform_id pid, pids[16];
cl_uint nbplat, nb;
char buf[128];
size_t size;
int i, j;
err = clGetPlatformIDs(sizeof(pids)/sizeof(pids[0]), pids, &nbplat);
assert(err == CL_SUCCESS);
printf("%u platforms\n", nbplat);
for (j = 0; j < nbplat; j++)
{
pid = pids[j];
printf(" platform %d\n", j);
err = clGetPlatformInfo(pid, CL_PLATFORM_VERSION, sizeof(buf)-1, buf, &size);
assert(err == CL_SUCCESS);
buf[size] = 0;
printf(" platform version %s\n", buf);
err = clGetDeviceIDs(pid, CL_DEVICE_TYPE_ALL, sizeof(did)/sizeof(did[0]), did, &nb);
assert(err == CL_SUCCESS);
printf("%d devices\n", nb);
for (i = 0; i < nb; i++)
{
err = clGetDeviceInfo(did[i], CL_DEVICE_VERSION, sizeof(buf)-1, buf, &size);
buf[size] = 0;
printf(" device %d version %s\n", i, buf);
}
}
return 0;
}

If that program does not find your device, the problem is not at the StarPU level, but the OpenCL drivers, check the documentation of your OpenCL implementation.

# I keep getting a "Incorrect performance model file" error

The performance model file, used by StarPU to record the performance of codelets, seem to have been corrupted. Perhaps a previous run of StarPU stopped abruptly, and thus could not save it properly. You can have a look at the file if you can fix it, but the simplest way is to just remove the file and run again, StarPU will just have to re-perform calibration for the corresponding codelet.