DPLACE(5) DPLACE(5) NAME dplace - NUMA placement specification DESCRIPTION Placement specifications are described in placement files or invoked from the command language library. A valid placement specification contains descriptions for memory and thread placement. Memory topologies as well as affinities to devices may also be specified. Shared virtual address ranges can be mapped to a specified memory. Page size and page migration thresholds can also be controlled. EXAMPLE An example placement file describing two memories and two threads might look like: # placement specification file # set up 2 memories which are close memories 2 in topology cube # set up 2 threads threads 2 # run the first thread on the 2nd memory run thread 0 on memory 1 # run the 2nd thread on the first memory run thread 1 on memory 0 This specification, when used for initial placement, would request 2 nearby memories from the operating system. At creation, the threads are requested to run on an available cpu which is local to the specified memory. As data and stack space is touched or faulted in, physical memory is allocated from the memory which is local to the thread which initiated the fault. SUMMARY Commands are newline terminated. Characters following the comment delimiter '#' are ignored. Tokens are separated by optional whitespace which is ignored. Line continuation is a \, and must occur between tokens. In what follows k,l,m,n,t0,t1,dt,m0,m1 and dm are arithmetic expressions that can contain environment variables preceded by a '$'. A legal statement could be: memories ( $MP_SET_NUMTHREADS + 1)/2 in cube The first example can be written in a scalable fashion as follows: # scalable placement specification file # set up memories which are close memories $MP_SET_NUMTHREADS in topology cube # set up threads threads $MP_SET_NUMTHREADS # run reversed threads across the memories distribute threads $MP_SET_NUMTHREADS-1:0:-1 across memories # set the page placement policy to RoundRobin for data pages policy data placement "PlacementRoundRobin" Static specifications may occur in a placement file or be called later from dplace(3) library routines. Dynamic specifications may only be called from dplace(3) library routines. Here is a summary of the grammar. Static specifications: memories m [[in] [topology] cube|none|physical|cpucluster] [near [/hw/*]+] threads n run thread n on memory m [using cpu k] distribute threads [t0:t1[:dt]] across memories [m0:m1[:dm]] [block [m]]|[cyclic [n]] place range k to l on memory m [[with] pagesize n [k|K|m|M] ] policy stack|data|text pagesize n [k|K|m|M] policy stack|data|text pagewait [off|on] policy stack|data|text placement s policy migration n [%] mode verbose [on|off|toggle] mode signal expr Dynamic specifications: migrate range k to l to memory m move thread|pid n to memory m In the above, the specification: threads t0:t1[:dt] means to use threads numbered t0 through t1 with an optional stride of dt. The default stride value is 1. Similarly, the specification: memories [m0:m1[:dm]] means to use memories m0 through m1 with an optional stride of dm. The default stride value is 1. The qualifier block m implies a block distribution of the threads with at most m threads per memory. If m is omitted, its default value is: the integer ceiling of the number of threads being distributed divided by the number of memories being distributed across. The qualifier cyclic n implies a cyclic distribution of the selected threads across the selected memories. The threads are chosen in groups of n and dealt out to the appropriate memory until all threads are used. If n is omitted, its default value is one and the threads are dealt out to the memories like a deck of cards. ERROR HANDLING If errors are encountered in the placement_file, the default procedure for dplace is to print a diagnostic message to standard error specifying where the error occurred in the placement_file and abort execution. If errors are encountered in the libdplace.so library during the run-time execution of program, then a diagnostic message is sent to standard error, a default signal of SIGKILL is sent to all members of the process group, and execution is aborted. The mode signal expr statement allows a selection of a specific signal number to be generated upon error. If the mode signal expr is specified, the action taken when libdplace.so detects a run-time error is to send the signal number derived from expr to the program invoked by dplace. Under this condition the control is returned to the caller, which is the program. The signal number can range from 1 to 32. An example of how to set the signal number: mode signal 16 Upon detecting an error in libdplace.so during run-time, signal 16 (defined as SIGUSR1) is sent to the calling process (in this case the program and control is returned to the caller. COMMON PLACEMENT FILE ERRORS The most common placement file error is the use of either the run or distribute directive without previously declaring both the number of threads, and the number of memories using the memories and threads directives. TERMINOLOGY In the above, a thread is an IRIX process or any of its descendants which were created using sproc(2) or fork(2). Thread numbers are ordered (from 0 to the number of threads minus 1 ) in the same way as the pid's for each process. A memory is an instantiation of a physical memory. A range is a virtual address range such as 0x100000 to 0x200000. PLACEMENT POLICIES The placement policies are specified according to mmci(5). The current valid policies are PlacementDefault, PlacementFixed, PlacementFirstTouch, PlacementRoundRobin, PlacementThreadLocal, PlacementCacheColor. SEE ALSO Page 4