Threads em C - Precisa Rodar no Linux

IMPLEMENTAÇÃO Parte 1 Faça um programa que crie N threads N é um dado de entrada lido do teclado usando a biblioteca pthreads Cada thread quando executar deverá imprimir na tela Eu sou a thread com TID YY e ZZ onde YY é o TID obtido usando a função pthreadself e ZZ é o TID obtido com a função gettid Imprima TID usando lu Explique as saídas do programa quando este é executado gettid e pthreadself retornam o mesmo valor Se não pesquise qual a diferença entre elas Para chamar gettid use instrução syscallSYSgettid e faça include da biblioteca syssyscallh Parte 2 Faça um programa que some duas matrizes quadradas A e B de dimensões 500x500 e que armazenem números inteiros Faça a soma sem usar threads e com 4 threads uma thread para cada parte da matriz Compare o tempo de execução das duas versões Meça o tempo com precisão de nanossegundos Parte 3 Implementar o seguinte programa Criar um vetor de inteiros de tamanho 100000 Preencher o vetor com números aleatórios entre 1 e 100 Mude o seed quando achar que terminou e rode algumas vezes para ver se não tem erros Criar 2 threads T1 remove números pares do vetor T2 remove números múltiplos de 5 do vetor Remova do fim para o começo do vetor e mantenha a ordem original dos elementos Teste o programa com e sem semáforos ambos com threads observe o resultado e explique o que está acontecendo Compare o tempo de execução sem threads e com threads semáforos Use alguma biblioteca para pegar o tempo atual do sistema Qual o mais rápido OBRIGATÓRIO Construa uma função que avalia se o resultado final está correto ou seja faça remoção de múltiplos de 2 e 5 sequencialmente ou seja sem usar threads compare resultado com saída da versão com threads REGRAS DO TRABALHO 1 Individual ou dupla 2 Os códigos fontes e um vídeo demonstrando a correta execução do programa e explicando o seu código deverão ser enviados pelo classroom a Não enviem arquivo compactado b Não aceito links Anexe cada arquivo SEPARADAMENTE no momento da entrega c Novamente não aceito links nem arquivos compactados d Mostrem apenas a execução correta com alguns testes e Você poderá ser contatado para sanar eventuais dúvidas f Estejam preparados pois a nota dependerá da qualidade da apresentação 3 O trabalho vale 1 ponto na média Parte 1 vale 02 Parte 2 e Parte 3 valem 04 cada Trabalhos parciais receberão notas parciais Tutorials Exercises Abstracts LC Workshops Comments Search Privacy Legal Notice POSIX Threads Programming Author Blaise Barney Lawrence Livermore National Laboratory UCRLMI133316 Table of Contents Abstract 1 Pthreads Overview What is a Thread 1 What are Pthreads 2 Why Pthreads 3 Designing Threaded Programs 4 2 The Pthreads API 3 Compiling Threaded Programs 4 Thread Management Creating and Terminating Threads 1 Passing Arguments to Threads 2 Joining and Detaching Threads 3 Stack Management 4 Miscellaneous Routines 5 5 Exercise 1 6 Mutex Variables Mutex Variables Overview 1 Creating and Destroying Mutexes 2 Locking and Unlocking Mutexes 3 7 Condition Variables Condition Variables Overview 1 Creating and Destroying Condition Variables 2 Waiting and Signaling on Condition Variables 3 8 Monitoring Debugging and Performance Analysis Tools for Pthreads 9 LLNL Specific Information and Recommendations 10 Topics Not Covered 11 Exercise 2 12 References and More Information 13 Appendix A Pthread Library Routines Reference 14 Abstract In shared memory multiprocessor architectures threads can be used to implement parallelism Historically hardware vendors have implemented their own proprietary versions of threads making portability a concern for software developers For UNIX systems a standardized C language threads programming interface has been specified by the IEEE POSIX 10031c standard Implementations that adhere to this standard are referred to as POSIX threads or Pthreads The tutorial begins with an introduction to concepts motivations and design considerations for using Pthreads Each of the three major classes of routines in the Pthreads API are then covered Thread Management Mutex Variables and Condition Variables Example codes are used throughout to demonstrate how to use most of the Pthreads routines needed by a new Pthreads programmer The tutorial concludes with a discussion of LLNL specifics and how to mix MPI with pthreads A lab exercise with numerous example codes C Language is also included LevelPrerequisites This tutorial is one of the eight tutorials in the 4 day Using LLNLs Supercomputers workshop It is deal for those who are new to parallel programming with threads A basic understanding of parallel programming in C is required For those who are unfamiliar with Parallel Programming in general the material covered in EC3500 Introduction To Parallel Computing would be helpful POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 1 of 31 07302014 0900 PM Pthreads Overview What is a Thread Technically a thread is defined as an independent stream of instructions that can be scheduled to run as such by the operating system But what does this mean To the software developer the concept of a procedure that runs independently from its main program may best describe a thread To go one step further imagine a main program aout that contains a number of procedures Then imagine all of these procedures being able to be scheduled to run simultaneously andor independently by the operating system That would describe a multithreaded program How is this accomplished Before understanding a thread one first needs to understand a UNIX process A process is created by the operating system and requires a fair amount of overhead Processes contain information about program resources and program execution state including Process ID process group ID user ID and group ID Environment Working directory Program instructions Registers Stack Heap File descriptors Signal actions Shared libraries Interprocess communication tools such as message queues pipes semaphores or shared memory UNIX PROCESS THREADS WITHIN A UNIX PROCESS Threads use and exist within these process resources yet are able to be scheduled by the operating system and run as independent entities largely because they duplicate only the bare essential resources that enable them to exist as executable code This independent flow of control is accomplished because a thread maintains its own Stack pointer Registers Scheduling properties such as policy or priority Set of pending and blocked signals Thread specific data So in summary in the UNIX environment a thread Exists within a process and uses the process resources POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 2 of 31 07302014 0900 PM Has its own independent flow of control as long as its parent process exists and the OS supports it Duplicates only the essential resources it needs to be independently schedulable May share the process resources with other threads that act equally independently and dependently Dies if the parent process dies or something similar Is lightweight because most of the overhead has already been accomplished through the creation of its process Because threads within the same process share resources Changes made by one thread to shared system resources such as closing a file will be seen by all other threads Two pointers having the same value point to the same data Reading and writing to the same memory locations is possible and therefore requires explicit synchronization by the programmer Pthreads Overview What are Pthreads Historically hardware vendors have implemented their own proprietary versions of threads These implementations differed substantially from each other making it difficult for programmers to develop portable threaded applications In order to take full advantage of the capabilities provided by threads a standardized programming interface was required For UNIX systems this interface has been specified by the IEEE POSIX 10031c standard 1995 Implementations adhering to this standard are referred to as POSIX threads or Pthreads Most hardware vendors now offer Pthreads in addition to their proprietary APIs The POSIX standard has continued to evolve and undergo revisions including the Pthreads specification Some useful links standardsieeeorgfindstdsstandard100312008html wwwopengrouporgaustinpapersposixfaqhtml wwwunixorgversion3ieeestdhtml Pthreads are defined as a set of C language programming types and procedure calls implemented with a pthreadh headerinclude file and a thread library though this library may be part of another library such as libc in some implementations Pthreads Overview Why Pthreads Light Weight When compared to the cost of creating and managing a process a thread can be created with much less operating system overhead Managing threads requires fewer system resources than managing processes For example the following table compares timing results for the fork subroutine and the pthreadcreate subroutine Timings reflect 50000 processthread creations were performed with the time utility and units are in seconds no optimization flags Note dont expect the sytem and user times to add up to real time because these are SMP systems with multiple CPUscores working on the problem at the same time At best these are approximations run on local machines past and present Platform fork pthreadcreate real user sys real user sys Intel 26 GHz Xeon E52670 16 coresnode 81 01 29 09 02 03 Intel 28 GHz Xeon 5660 12 coresnode 44 04 43 07 02 05 AMD 23 GHz Opteron 16 coresnode 125 10 125 12 02 13 AMD 24 GHz Opteron 8 coresnode 176 22 157 14 03 13 POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 3 of 31 07302014 0900 PM IBM 40 GHz POWER6 8 cpusnode 95 06 88 16 01 04 IBM 19 GHz POWER5 p5575 8 cpusnode 642 307 276 17 06 11 IBM 15 GHz POWER4 8 cpusnode 1045 486 472 21 10 15 INTEL 24 GHz Xeon 2 cpusnode 549 15 208 16 07 09 INTEL 14 GHz Itanium2 4 cpusnode 545 11 222 20 12 06 forkvsthreadtxt Efficient CommunicationsData Exchange The primary motivation for considering the use of Pthreads on a multiprocessor architecture is to achieve optimum performance In particular if an application is using MPI for onnode communications there is a potential that performance could be improved by using Pthreads instead MPI libraries usually implement onnode task communication via shared memory which involves at least one memory copy operation process to process For Pthreads there is no intermediate memory copy required because threads share the same address space within a single process There is no data transfer per se It can be as efficient as simply passing a pointer In the worst case scenario Pthread communications become more of a cachetoCPU or memorytoCPU bandwidth issue These speeds are much higher than MPI shared memory communications For example some local comparisons past and present are shown below Platform MPI Shared Memory Bandwidth GBsec Pthreads Worst Case MemorytoCPU Bandwidth GBsec Intel 26 GHz Xeon E52670 45 512 Intel 28 GHz Xeon 5660 56 32 AMD 23 GHz Opteron 18 53 AMD 24 GHz Opteron 12 53 IBM 19 GHz POWER5 p5575 41 16 IBM 15 GHz POWER4 21 4 Intel 24 GHz Xeon 03 43 Intel 14 GHz Itanium 2 18 64 Other Common Reasons Threaded applications offer potential performance gains and practical advantages over nonthreaded applications in several other ways Overlapping CPU work with IO For example a program may have sections where it is performing a long IO operation While one thread is waiting for an IO system call to complete CPU intensive work can be performed by other threads Priorityrealtime scheduling tasks which are more important can be scheduled to supersede or interrupt lower priority tasks Asynchronous event handling tasks which service events of indeterminate frequency and duration can be interleaved For example a web server can both transfer data from previous requests and manage the arrival of new requests A perfect example is the typical web browser where many interleaved tasks can be happening at the same time and where tasks can vary in priority Another good example is a modern operating system which makes extensive use of threads A screenshot of the MS Windows OS and applications using threads is shown below POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 4 of 31 07302014 0900 PM Click on image for a larger version Pthreads Overview Designing Threaded Programs Parallel Programming On modern multicore machines pthreads are ideally suited for parallel programming and whatever applies to parallel programming in general applies to parallel pthreads programs There are many considerations for designing parallel programs such as What type of parallel programming model to use Problem partitioning Load balancing Communications Data dependencies Synchronization and race conditions Memory issues IO issues Program complexity Programmer effortcoststime Covering these topics is beyond the scope of this tutorial however interested readers can obtain a quick overview in the Introduction to Parallel Computing tutorial In general though in order for a program to take advantage of Pthreads it must be able to be organized into discrete independent tasks which can execute concurrently For example if routine1 and routine2 can be interchanged interleaved andor overlapped in real time they are candidates for threading Programs having the following characteristics may be well suited for pthreads Work that can be executed or data that can be operated on by multiple tasks simultaneously POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 5 of 31 07302014 0900 PM Block for potentially long IO waits Use many CPU cycles in some places but not others Must respond to asynchronous events Some work is more important than other work priority interrupts Several common models for threaded programs exist Managerworker a single thread the manager assigns work to other threads the workers Typically the manager handles all input and parcels out work to the other tasks At least two forms of the managerworker model are common static worker pool and dynamic worker pool Pipeline a task is broken into a series of suboperations each of which is handled in series but concurrently by a different thread An automobile assembly line best describes this model Peer similar to the managerworker model but after the main thread creates other threads it participates in the work Shared Memory Model All threads have access to the same global shared memory Threads also have their own private data Programmers are responsible for synchronizing access protecting globally shared data Threadsafeness Threadsafeness in a nutshell refers an applications ability to execute multiple threads simultaneously without clobbering shared data or creating race conditions For example suppose that your application creates several threads each of which makes a call to the same library routine This library routine accessesmodifies a global structure or location in memory As each thread calls this routine it is possible that they may try to modify this global structurememory location at the same time If the routine does not employ some sort of synchronization constructs to prevent data corruption then it is not threadsafe POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 6 of 31 07302014 0900 PM The implication to users of external library routines is that if you arent 100 certain the routine is threadsafe then you take your chances with problems that could arise Recommendation Be careful if your application uses libraries or other objects that dont explicitly guarantee thread safeness When in doubt assume that they are not threadsafe until proven otherwise This can be done by serializing the calls to the uncertain routine etc Thread Limits Although the Pthreads API is an ANSIIEEE standard implementations can and usually do vary in ways not specified by the standard Because of this a program that runs fine on one platform may fail or produce wrong results on another platform For example the maximum number of threads permitted and the default thread stack size are two important limits to consider when designing your program Several thread limits are discussed in more detail later in this tutorial The Pthreads API The original Pthreads API was defined in the ANSIIEEE POSIX 10031 1995 standard The POSIX standard has continued to evolve and undergo revisions including the Pthreads specification Copies of the standard can be purchased from IEEE or downloaded for free from other sites online The subroutines which comprise the Pthreads API can be informally grouped into four major groups Thread management Routines that work directly on threads creating detaching joining etc They also include functions to setquery thread attributes joinable scheduling etc 1 Mutexes Routines that deal with synchronization called a mutex which is an abbreviation for mutual exclusion Mutex functions provide for creating destroying locking and unlocking mutexes These are supplemented by mutex attribute functions that set or modify attributes associated with mutexes 2 Condition variables Routines that address communications between threads that share a mutex Based upon programmer specified conditions This group includes functions to create destroy wait and signal based upon specified variable values Functions to setquery condition variable attributes are also included 3 Synchronization Routines that manage readwrite locks and barriers 4 Naming conventions All identifiers in the threads library begin with pthread Some examples are shown below Routine Prefix Functional Group pthread Threads themselves and miscellaneous subroutines pthreadattr Thread attributes objects POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 7 of 31 07302014 0900 PM pthreadmutex Mutexes pthreadmutexattr Mutex attributes objects pthreadcond Condition variables pthreadcondattr Condition attributes objects pthreadkey Threadspecific data keys pthreadrwlock Readwrite locks pthreadbarrier Synchronization barriers The concept of opaque objects pervades the design of the API The basic calls work to create or modify opaque objects the opaque objects can be modified by calls to attribute functions which deal with opaque attributes The Pthreads API contains around 100 subroutines This tutorial will focus on a subset of these specifically those which are most likely to be immediately useful to the beginning Pthreads programmer For portability the pthreadh header file should be included in each source file using the Pthreads library The current POSIX standard is defined only for the C language Fortran programmers can use wrappers around C function calls Some Fortran compilers may provide a Fortram pthreads API A number of excellent books about Pthreads are available Several of these are listed in the References section of this tutorial Compiling Threaded Programs Several examples of compile commands used for pthreads codes are listed in the table below Compiler Platform Compiler Command Description INTEL Linux icc pthread C icpc pthread C PGI Linux pgcc lpthread C pgCC lpthread C GNU Linux Blue Gene gcc pthread GNU C g pthread GNU C IBM Blue Gene bgxlcr bgccr C ANSI nonANSI bgxlCr bgxlcr C Thread Management Creating and Terminating Threads Routines pthreadcreate threadattrstartroutinearg pthreadexit status pthreadcancel thread pthreadattrinit attr pthreadattrdestroy attr POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 8 of 31 07302014 0900 PM Creating Threads Initially your main program comprises a single default thread All other threads must be explicitly created by the programmer pthreadcreate creates a new thread and makes it executable This routine can be called any number of times from anywhere within your code pthreadcreate arguments thread An opaque unique identifier for the new thread returned by the subroutine attr An opaque attribute object that may be used to set thread attributes You can specify a thread attributes object or NULL for the default values startroutine the C routine that the thread will execute once it is created arg A single argument that may be passed to startroutine It must be passed by reference as a pointer cast of type void NULL may be used if no argument is to be passed The maximum number of threads that may be created by a process is implementation dependent Programs that attempt to exceed the limit can fail or produce wrong results Querying and setting your implementations thread limit Linux example shown Demonstrates querying the default soft limits and then setting the maximum number of processes including threads to the hard limit Then verifying that the limit has been overridden bash ksh sh tcsh csh ulimit a core file size blocks c 16 data seg size kbytes d unlimited scheduling priority e 0 file size blocks f unlimited pending signals i 255956 max locked memory kbytes l 64 max memory size kbytes m unlimited open files n 1024 pipe size 512 bytes p 8 POSIX message queues bytes q 819200 realtime priority r 0 stack size kbytes s unlimited cpu time seconds t unlimited max user processes u 1024 virtual memory kbytes v unlimited file locks x unlimited ulimit Hu 7168 ulimit u 7168 ulimit a core file size blocks c 16 data seg size kbytes d unlimited scheduling priority e 0 file size blocks f unlimited pending signals i 255956 max locked memory kbytes l 64 max memory size kbytes m unlimited open files n 1024 pipe size 512 bytes p 8 POSIX message queues bytes q 819200 realtime priority r 0 stack size kbytes s unlimited cpu time seconds t unlimited max user processes u 7168 virtual memory kbytes v unlimited file locks x unlimited limit cputime unlimited filesize unlimited datasize unlimited stacksize unlimited coredumpsize 16 kbytes memoryuse unlimited vmemoryuse unlimited descriptors 1024 memorylocked 64 kbytes maxproc 1024 limit maxproc unlimited limit cputime unlimited filesize unlimited datasize unlimited stacksize unlimited coredumpsize 16 kbytes memoryuse unlimited vmemoryuse unlimited descriptors 1024 memorylocked 64 kbytes maxproc 7168 Once created threads are peers and may create other threads There is no implied hierarchy or dependency between threads POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 9 of 31 07302014 0900 PM Thread Attributes By default a thread is created with certain attributes Some of these attributes can be changed by the programmer via the thread attribute object pthreadattrinit and pthreadattrdestroy are used to initializedestroy the thread attribute object Other routines are then used to queryset specific attributes in the thread attribute object Attributes include Detached or joinable state Scheduling inheritance Scheduling policy Scheduling parameters Scheduling contention scope Stack size Stack address Stack guard overflow size Some of these attributes will be discussed later Thread Binding and Scheduling Question After a thread has been created how do you know awhen it will be scheduled to run by the operating system and bwhich processorcore it will run on Answer The Pthreads API provides several routines that may be used to specify how threads are scheduled for execution For example threads can be scheduled to run FIFO firstin firstout RR roundrobin or OTHER operating system determines It also provides the ability to set a threads scheduling priority value These topics are not covered here however a good overview of how things work under Linux can be found in the schedsetscheduler man page The Pthreads API does not provide routines for binding threads to specific cpuscores However local implementations may include this functionality such as providing the nonstandard pthreadsetaffinitynp routine Note that np in the name stands for nonportable Also the local operating system may provide a way to do this For example Linux provides the schedsetaffinity routine Terminating Threads pthreadexit There are several ways in which a thread may be terminated The thread returns normally from its starting routine Its work is done The thread makes a call to the pthreadexit subroutine whether its work is done or not The thread is canceled by another thread via the pthreadcancel routine The entire process is terminated due to making a call to either the exec or exit If main finishes first without calling pthreadexit explicitly itself POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 10 of 31 07302014 0900 PM The pthreadexit routine allows the programmer to specify an optional termination status parameter This optional parameter is typically returned to threads joining the terminated thread covered later In subroutines that execute to completion normally you can often dispense with calling pthreadexit unless of course you want to pass the optional status code back Cleanup the pthreadexit routine does not close files any files opened inside the thread will remain open after the thread is terminated Discussion on calling pthreadexit from main There is a definite problem if main finishes before the threads it spawned if you dont call pthreadexit explicitly All of the threads it created will terminate because main is done and no longer exists to support the threads By having main explicitly call pthreadexit as the last thing it does main will block and be kept alive to support the threads it created until they are done Example Pthread Creation and Termination This simple example code creates 5 threads with the pthreadcreate routine Each thread prints a Hello World message and then terminates with a call to pthreadexit Example Code Pthread Creation and Termination include pthreadh include stdioh define NUMTHREADS 5 void PrintHellovoid threadid long tid tid longthreadid printfHello World Its me thread ld tid pthreadexitNULL int main int argc char argv pthreadt threadsNUMTHREADS int rc long t fort0 tNUMTHREADS t printfIn main creating thread ld t rc pthreadcreatethreadst NULL PrintHello void t if rc printfERROR return code from pthreadcreate is d rc exit1 Last thing that main should do pthreadexitNULL Thread Management Passing Arguments to Threads The pthreadcreate routine permits the programmer to pass one argument to the thread start routine For cases where multiple arguments must be passed this limitation is easily overcome by creating a structure which contains all of the arguments and then passing a pointer to that structure in the pthreadcreate routine All arguments must be passed by reference and cast to void Question How can you safely pass data to newly created threads given their nondeterministic startup and scheduling Answer POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 11 of 31 07302014 0900 PM Example 1 Thread Argument Passing This code fragment demonstrates how to pass a simple integer to each thread The calling thread uses a unique data structure for each thread insuring that each threads argument remains intact throughout the program long taskidsNUMTHREADS fort0 tNUMTHREADS t taskidst long mallocsizeoflong taskidst t printfCreating thread ld t rc pthreadcreatethreadst NULL PrintHello void taskidst Example 2 Thread Argument Passing This example shows how to setuppass multiple arguments via a structure Each thread receives a unique instance of the structure struct threaddata int threadid int sum char message struct threaddata threaddataarrayNUMTHREADS void PrintHellovoid threadarg struct threaddata mydata mydata struct threaddata threadarg taskid mydatathreadid sum mydatasum hellomsg mydatamessage int main int argc char argv threaddataarraytthreadid t threaddataarraytsum sum threaddataarraytmessage messagest rc pthreadcreatethreadst NULL PrintHello void threaddataarrayt Example 3 Thread Argument Passing Incorrect This example performs argument passing incorrectly It passes the address of variable t which is shared memory space and visible to all threads As the loop iterates the value of this memory location changes possibly before the created threads can access it int rc long t fort0 tNUMTHREADS t printfCreating thread ld t rc pthreadcreatethreadst NULL PrintHello void t POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 12 of 31 07302014 0900 PM Thread Management Joining and Detaching Threads Routines pthreadjoin threadidstatus pthreaddetach threadid pthreadattrsetdetachstate attrdetachstate pthreadattrgetdetachstate attrdetachstate Joining Joining is one way to accomplish synchronization between threads For example The pthreadjoin subroutine blocks the calling thread until the specified threadid thread terminates The programmer is able to obtain the target threads termination return status if it was specified in the target threads call to pthreadexit A joining thread can match one pthreadjoin call It is a logical error to attempt multiple joins on the same thread Two other synchronization methods mutexes and condition variables will be discussed later Joinable or Not When a thread is created one of its attributes defines whether it is joinable or detached Only threads that are created as joinable can be joined If a thread is created as detached it can never be joined The final draft of the POSIX standard specifies that threads should be created as joinable To explicitly create a thread as joinable or detached the attr argument in the pthreadcreate routine is used The typical 4 step process is 1 Declare a pthread attribute variable of the pthreadattrt data type 2 Initialize the attribute variable with pthreadattrinit 3 Set the attribute detached status with pthreadattrsetdetachstate When done free library resources used by the attribute with pthreadattrdestroy 4 Detaching The pthreaddetach routine can be used to explicitly detach a thread even though it was created as joinable There is no converse routine Recommendations If a thread requires joining consider explicitly creating it as joinable This provides portability as not all implementations may create threads as joinable by default If you know in advance that a thread will never need to join with another thread consider creating it in a detached state Some system resources may be able to be freed POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 13 of 31 07302014 0900 PM Example Pthread Joining Example Code Pthread Joining This example demonstrates how to wait for thread completions by using the Pthread join routine Since some implementations of Pthreads may not create threads in a joinable state the threads in this example are explicitly created in a joinable state so that they can be joined later include pthreadh include stdioh include stdlibh include mathh define NUMTHREADS 4 void BusyWorkvoid t int i long tid double result00 tid longt printfThread ld starting tid for i0 i1000000 i result result sini tani printfThread ld done Result e tid result pthreadexitvoid t int main int argc char argv pthreadt threadNUMTHREADS pthreadattrt attr int rc long t void status Initialize and set thread detached attribute pthreadattrinitattr pthreadattrsetdetachstateattr PTHREADCREATEJOINABLE fort0 tNUMTHREADS t printfMain creating thread ld t rc pthreadcreatethreadt attr BusyWork void t if rc printfERROR return code from pthreadcreate is d rc exit1 Free attribute and wait for the other threads pthreadattrdestroyattr fort0 tNUMTHREADS t rc pthreadjointhreadt status if rc printfERROR return code from pthreadjoin is d rc exit1 printfMain completed join with thread ld having a status of ld tlongstatus printfMain program completed Exiting pthreadexitNULL Thread Management Stack Management Routines POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 14 of 31 07302014 0900 PM pthreadattrgetstacksize attr stacksize pthreadattrsetstacksize attr stacksize pthreadattrgetstackaddr attr stackaddr pthreadattrsetstackaddr attr stackaddr Preventing Stack Problems The POSIX standard does not dictate the size of a threads stack This is implementation dependent and varies Exceeding the default stack limit is often very easy to do with the usual results program termination andor corrupted data Safe and portable programs do not depend upon the default stack limit but instead explicitly allocate enough stack for each thread by using the pthreadattrsetstacksize routine The pthreadattrgetstackaddr and pthreadattrsetstackaddr routines can be used by applications in an environment where the stack for a thread must be placed in some particular region of memory Some Practical Examples at LC Default thread stack size varies greatly The maximum size that can be obtained also varies greatly and may depend upon the number of threads per node Both past and present architectures are shown to demonstrate the wide variation in default thread stack size Node Architecture CPUs Memory GB Default Size bytes Intel Xeon E52670 16 32 2097152 Intel Xeon 5660 12 24 2097152 AMD Opteron 8 16 2097152 Intel IA64 4 8 33554432 Intel IA32 2 4 2097152 IBM Power5 8 32 196608 IBM Power4 8 16 196608 IBM Power3 16 16 98304 Example Stack Management Example Code Stack Management This example demonstrates how to query and set a threads stack size include pthreadh include stdioh define NTHREADS 4 define N 1000 define MEGEXTRA 1000000 pthreadattrt attr void doworkvoid threadid double ANN int ij long tid sizet mystacksize tid longthreadid pthreadattrgetstacksize attr mystacksize printfThread ld stack size li bytes tid mystacksize for i0 iN i for j0 jN j Aij ij3452 Ni pthreadexitNULL POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 15 of 31 07302014 0900 PM int mainint argc char argv pthreadt threadsNTHREADS sizet stacksize int rc long t pthreadattrinitattr pthreadattrgetstacksize attr stacksize printfDefault stack size li stacksize stacksize sizeofdoubleNNMEGEXTRA printfAmount of stack needed per thread li stacksize pthreadattrsetstacksize attr stacksize printfCreating threads with stack size li bytes stacksize fort0 tNTHREADS t rc pthreadcreatethreadst attr dowork void t if rc printfERROR return code from pthreadcreate is d rc exit1 printfCreated ld threads t pthreadexitNULL Thread Management Miscellaneous Routines pthreadself pthreadequal thread1thread2 pthreadself returns the unique system assigned thread ID of the calling thread pthreadequal compares two thread IDs If the two IDs are different 0 is returned otherwise a nonzero value is returned Note that for both of these routines the thread identifier objects are opaque and can not be easily inspected Because thread IDs are opaque objects the C language equivalence operator should not be used to compare two thread IDs against each other or to compare a single thread ID against another value pthreadonce oncecontrol initroutine pthreadonce executes the initroutine exactly once in a process The first call to this routine by any thread in the process executes the given initroutine without parameters Any subsequent call will have no effect The initroutine routine is typically an initialization routine The oncecontrol parameter is a synchronization control structure that requires initialization prior to calling pthreadonce For example pthreadoncet oncecontrol PTHREADONCEINIT Pthread Exercise 1 Getting Started and Thread Management Routines Overview Login to an LC cluster using your workshop username and OTP token Copy the exercise files to your home directory Familiarize yourself with LCs Pthreads environment POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 16 of 31 07302014 0900 PM Write a simple Hello World Pthreads program Successfully compile your program Successfully run your program several different ways Review compile run andor debug some related Pthreads programs provided GO TO THE EXERCISE HERE Mutex Variables Overview Mutex is an abbreviation for mutual exclusion Mutex variables are one of the primary means of implementing thread synchronization and for protecting shared data when multiple writes occur A mutex variable acts like a lock protecting access to a shared data resource The basic concept of a mutex as used in Pthreads is that only one thread can lock or own a mutex variable at any given time Thus even if several threads try to lock a mutex only one thread will be successful No other thread can own that mutex until the owning thread unlocks that mutex Threads must take turns accessing protected data Mutexes can be used to prevent race conditions An example of a race condition involving a bank transaction is shown below Thread 1 Thread 2 Balance Read balance 1000 1000 Read balance 1000 1000 Deposit 200 1000 Deposit 200 1000 Update balance 1000200 1200 Update balance 1000200 1200 In the above example a mutex should be used to lock the Balance while a thread is using this shared data resource Very often the action performed by a thread owning a mutex is the updating of global variables This is a safe way to ensure that when several threads update the same variable the final value is the same as what it would be if only one thread performed the update The variables being updated belong to a critical section A typical sequence in the use of a mutex is as follows Create and initialize a mutex variable Several threads attempt to lock the mutex Only one succeeds and that thread owns the mutex The owner thread performs some set of actions The owner unlocks the mutex Another thread acquires the mutex and repeats the process Finally the mutex is destroyed When several threads compete for a mutex the losers block at that call an unblocking call is available with trylock instead of the lock call When protecting shared data it is the programmers responsibility to make sure every thread that needs to use a mutex does so For example if 4 threads are updating the same data but only one uses a mutex the data can still be corrupted POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 17 of 31 07302014 0900 PM Mutex Variables Creating and Destroying Mutexes Routines pthreadmutexinit mutexattr pthreadmutexdestroy mutex pthreadmutexattrinit attr pthreadmutexattrdestroy attr Usage Mutex variables must be declared with type pthreadmutext and must be initialized before they can be used There are two ways to initialize a mutex variable Statically when it is declared For example pthreadmutext mymutex PTHREADMUTEXINITIALIZER 1 Dynamically with the pthreadmutexinit routine This method permits setting mutex object attributes attr 2 The mutex is initially unlocked The attr object is used to establish properties for the mutex object and must be of type pthreadmutexattrt if used may be specified as NULL to accept defaults The Pthreads standard defines three optional mutex attributes Protocol Specifies the protocol used to prevent priority inversions for a mutex Prioceiling Specifies the priority ceiling of a mutex Processshared Specifies the process sharing of a mutex Note that not all implementations may provide the three optional mutex attributes The pthreadmutexattrinit and pthreadmutexattrdestroy routines are used to create and destroy mutex attribute objects respectively pthreadmutexdestroy should be used to free a mutex object which is no longer needed Mutex Variables Locking and Unlocking Mutexes Routines pthreadmutexlock mutex pthreadmutextrylock mutex pthreadmutexunlock mutex Usage The pthreadmutexlock routine is used by a thread to acquire a lock on the specified mutex variable If the mutex is already locked by another thread this call will block the calling thread until the mutex is unlocked pthreadmutextrylock will attempt to lock a mutex However if the mutex is already locked the routine will return immediately with a busy error code This routine may be useful in preventing deadlock conditions as in a priority inversion situation pthreadmutexunlock will unlock a mutex if called by the owning thread Calling this routine is required after a thread has completed its use of protected data if other threads are to acquire the mutex for their work with the protected data An error will be returned if If the mutex was already unlocked If the mutex is owned by another thread There is nothing magical about mutexesin fact they are akin to a gentlemens agreement between participating POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 18 of 31 07302014 0900 PM threads It is up to the code writer to insure that the necessary threads all make the the mutex lock and unlock calls correctly The following scenario demonstrates a logical error Thread 1 Thread 2 Thread 3 Lock Lock A 2 A A1 A AB Unlock Unlock Question When more than one thread is waiting for a locked mutex which thread will be granted the lock first after it is released Answer Example Using Mutexes Example Code Using Mutexes This example program illustrates the use of mutex variables in a threads program that performs a dot product The main data is made available to all threads through a globally accessible structure Each thread works on a different part of the data The main thread waits for all the threads to complete their computations and then it prints the resulting sum include pthreadh include stdioh include stdlibh The following structure contains the necessary information to allow the function dotprod to access its input data and place its output into the structure typedef struct double a double b double sum int veclen DOTDATA Define globally accessible variables and a mutex define NUMTHRDS 4 define VECLEN 100 DOTDATA dotstr pthreadt callThdNUMTHRDS pthreadmutext mutexsum The function dotprod is activated when the thread is created All input to this routine is obtained from a structure of type DOTDATA and all output from this function is written into this structure The benefit of this approach is apparent for the multithreaded program when a thread is created we pass a single argument to the activated function typically this argument is a thread number All the other information required by the function is accessed from the globally accessible structure void dotprodvoid arg Define and use local variables for convenience int i start end len long offset double mysum x y offset longarg len dotstrveclen start offsetlen end start len x dotstra y dotstrb Perform the dot product and assign result POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 19 of 31 07302014 0900 PM to the appropriate variable in the structure mysum 0 for istart iend i mysum xi yi Lock a mutex prior to updating the value in the shared structure and unlock it upon updating pthreadmutexlock mutexsum dotstrsum mysum pthreadmutexunlock mutexsum pthreadexitvoid 0 The main program creates threads which do all the work and then print out result upon completion Before creating the threads the input data is created Since all threads update a shared structure we need a mutex for mutual exclusion The main thread needs to wait for all threads to complete it waits for each one of the threads We specify a thread attribute value that allow the main thread to join with the threads it creates Note also that we free up handles when they are no longer needed int main int argc char argv long i double a b void status pthreadattrt attr Assign storage and initialize values a double malloc NUMTHRDSVECLENsizeofdouble b double malloc NUMTHRDSVECLENsizeofdouble for i0 iVECLENNUMTHRDS i ai10 biai dotstrveclen VECLEN dotstra a dotstrb b dotstrsum0 pthreadmutexinitmutexsum NULL Create threads to perform the dotproduct pthreadattrinitattr pthreadattrsetdetachstateattr PTHREADCREATEJOINABLE fori0 iNUMTHRDS i Each thread works on a different set of data The offset is specified by i The size of the data for each thread is indicated by VECLEN pthreadcreatecallThdi attr dotprod void i pthreadattrdestroyattr Wait on the other threads fori0 iNUMTHRDS i pthreadjoincallThdi status After joining print out the results and cleanup printf Sum f dotstrsum free a POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 20 of 31 07302014 0900 PM free b pthreadmutexdestroymutexsum pthreadexitNULL Serial version Pthreads version Condition Variables Overview Condition variables provide yet another way for threads to synchronize While mutexes implement synchronization by controlling thread access to data condition variables allow threads to synchronize based upon the actual value of data Without condition variables the programmer would need to have threads continually polling possibly in a critical section to check if the condition is met This can be very resource consuming since the thread would be continuously busy in this activity A condition variable is a way to achieve the same goal without polling A condition variable is always used in conjunction with a mutex lock A representative sequence for using condition variables is shown below Main Thread Declare and initialize global datavariables which require synchronization such as count Declare and initialize a condition variable object Declare and initialize an associated mutex Create threads A and B to do work Thread A Do work up to the point where a certain condition must occur such as count must reach a specified value Lock associated mutex and check value of a global variable Call pthreadcondwait to perform a blocking wait for signal from ThreadB Note that a call to pthreadcondwait automatically and atomically unlocks the associated mutex variable so that it can be used by ThreadB When signalled wake up Mutex is automatically and atomically locked Explicitly unlock mutex Continue Thread B Do work Lock associated mutex Change the value of the global variable that ThreadA is waiting upon Check value of the global ThreadA wait variable If it fulfills the desired condition signal ThreadA Unlock mutex Continue Main Thread Join Continue Condition Variables Creating and Destroying Condition Variables Routines pthreadcondinit conditionattr pthreadconddestroy condition pthreadcondattrinit attr pthreadcondattrdestroy attr POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 21 of 31 07302014 0900 PM Usage Condition variables must be declared with type pthreadcondt and must be initialized before they can be used There are two ways to initialize a condition variable Statically when it is declared For example pthreadcondt myconvar PTHREADCONDINITIALIZER 1 Dynamically with the pthreadcondinit routine The ID of the created condition variable is returned to the calling thread through the condition parameter This method permits setting condition variable object attributes attr 2 The optional attr object is used to set condition variable attributes There is only one attribute defined for condition variables processshared which allows the condition variable to be seen by threads in other processes The attribute object if used must be of type pthreadcondattrt may be specified as NULL to accept defaults Note that not all implementations may provide the processshared attribute The pthreadcondattrinit and pthreadcondattrdestroy routines are used to create and destroy condition variable attribute objects pthreadconddestroy should be used to free a condition variable that is no longer needed Condition Variables Waiting and Signaling on Condition Variables Routines pthreadcondwait conditionmutex pthreadcondsignal condition pthreadcondbroadcast condition Usage pthreadcondwait blocks the calling thread until the specified condition is signalled This routine should be called while mutex is locked and it will automatically release the mutex while it waits After signal is received and thread is awakened mutex will be automatically locked for use by the thread The programmer is then responsible for unlocking mutex when the thread is finished with it Recommendation Using a WHILE loop instead of an IF statement see watchcount routine in example below to check the waited for condition can help deal with several potential problems such as If several threads are waiting for the same wake up signal they will take turns acquiring the mutex and any one of them can then modify the condition they all waited for If the thread received the signal in error due to a program bug The Pthreads library is permitted to issue spurious wake ups to a waiting thread without violating the standard The pthreadcondsignal routine is used to signal or wake up another thread which is waiting on the condition variable It should be called after mutex is locked and must unlock mutex in order for pthreadcondwait routine to complete The pthreadcondbroadcast routine should be used instead of pthreadcondsignal if more than one thread is in a blocking wait state It is a logical error to call pthreadcondsignal before calling pthreadcondwait Proper locking and unlocking of the associated mutex variable is essential when using these routines For example Failing to lock the mutex before calling pthreadcondwait may cause it NOT to block Failing to unlock the mutex after calling pthreadcondsignal may not allow a matching pthreadcondwait routine to complete it will remain blocked Example Using Condition Variables POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 22 of 31 07302014 0900 PM Example Code Using Condition Variables This simple example code demonstrates the use of several Pthread condition variable routines The main routine creates three threads Two of the threads perform work and update a count variable The third thread waits until the count variable reaches a specified value include pthreadh include stdioh include stdlibh define NUMTHREADS 3 define TCOUNT 10 define COUNTLIMIT 12 int count 0 int threadids3 012 pthreadmutext countmutex pthreadcondt countthresholdcv void inccountvoid t int i long myid longt for i0 iTCOUNT i pthreadmutexlockcountmutex count Check the value of count and signal waiting thread when condition is reached Note that this occurs while mutex is locked if count COUNTLIMIT pthreadcondsignalcountthresholdcv printfinccount thread ld count d Threshold reached myid count printfinccount thread ld count d unlocking mutex myid count pthreadmutexunlockcountmutex Do some work so threads can alternate on mutex lock sleep1 pthreadexitNULL void watchcountvoid t long myid longt printfStarting watchcount thread ld myid Lock mutex and wait for signal Note that the pthreadcondwait routine will automatically and atomically unlock mutex while it waits Also note that if COUNTLIMIT is reached before this routine is run by the waiting thread the loop will be skipped to prevent pthreadcondwait from never returning pthreadmutexlockcountmutex while countCOUNTLIMIT pthreadcondwaitcountthresholdcv countmutex printfwatchcount thread ld Condition signal received myid count 125 printfwatchcount thread ld count now d myid count pthreadmutexunlockcountmutex pthreadexitNULL int main int argc char argv int i rc long t11 t22 t33 pthreadt threads3 pthreadattrt attr Initialize mutex and condition variable objects pthreadmutexinitcountmutex NULL POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 23 of 31 07302014 0900 PM pthreadcondinit countthresholdcv NULL For portability explicitly create threads in a joinable state pthreadattrinitattr pthreadattrsetdetachstateattr PTHREADCREATEJOINABLE pthreadcreatethreads0 attr watchcount void t1 pthreadcreatethreads1 attr inccount void t2 pthreadcreatethreads2 attr inccount void t3 Wait for all threads to complete for i0 iNUMTHREADS i pthreadjointhreadsi NULL printf Main Waited on d threads Done NUMTHREADS Clean up and exit pthreadattrdestroyattr pthreadmutexdestroycountmutex pthreadconddestroycountthresholdcv pthreadexitNULL Monitoring Debugging and Performance Analysis Tools for Pthreads Monitoring and Debugging Pthreads Debuggers vary in their ability to handle Pthreads The TotalView debugger is LCs recommended debugger for parallel programs It is well suited for both monitoring and debugging threaded programs An example screenshot from a TotalView session using a threaded code is shown below 1 Stack Trace Pane Displays the call stack of routines that the selected thread is executing 2 Status Bars Show status information for the selected thread and its associated process 3 Stack Frame Pane Shows a selected threads stack variables registers etc 4 Source Pane Shows the source code for the selected thread 5 Root Window showing all threads Threads Pane Shows threads associated with the selected process 6 POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 24 of 31 07302014 0900 PM See the TotalView Debugger tutorial for details The Linux ps command provides several flags for viewing thread information Some examples are shown below See the man page for details ps Lf UID PID PPID LWP C NLWP STIME TTY TIME CMD blaise 22529 28240 22529 0 5 1131 pts53 000000 aout blaise 22529 28240 22530 99 5 1131 pts53 000124 aout blaise 22529 28240 22531 99 5 1131 pts53 000124 aout blaise 22529 28240 22532 99 5 1131 pts53 000124 aout blaise 22529 28240 22533 99 5 1131 pts53 000124 aout ps T PID SPID TTY TIME CMD 22529 22529 pts53 000000 aout 22529 22530 pts53 000149 aout 22529 22531 pts53 000149 aout 22529 22532 pts53 000149 aout 22529 22533 pts53 000149 aout ps Lm PID LWP TTY TIME CMD 22529 pts53 001856 aout 22529 000000 22530 000444 22531 000444 22532 000444 POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 25 of 31 07302014 0900 PM 22533 000444 LCs Linux clusters also provide the top command to monitor processes on a node If used with the H flag the threads contained within a process will be visible An example of the top H command is shown below The parent process is PID 18010 which spawned three threads shown as PIDs 18012 18013 and 18014 Performance Analysis Tools There are a variety of performance analysis tools that can be used with threaded programs Searching the web will turn up a wealth of information At LC the list of supported computing tools can be found at computingllnlgovcodecontentsoftwaretoolsphp These tools vary significantly in their complexity functionality and learning curve Covering them in detail is beyond the scope of this tutorial Some tools worth investigating specifically for threaded codes include OpenSpeedShop TAU PAPI Intel VTune Amplifier ThreadSpotter LLNL Specific Information and Recommendations This section describes details specific to Livermore Computings systems Implementations All LC production systems include a Pthreads implementation that follows draft 10 final of the POSIX standard This is the preferred implementation Implementations differ in the maximum number of threads that a process may create They also differ in the default amount of thread stack space Compiling LC maintains a number of compilers and usually several different versions of each see the LCs Supported Compilers web page The compiler commands described in the Compiling Threaded Programs section apply to LC systems Mixing MPI with Pthreads This is the primary motivation for using Pthreads at LC POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 26 of 31 07302014 0900 PM Design Each MPI process typically creates and then manages N threads where N makes the best use of the available coresnode Finding the best value for N will vary with the platform and your applications characteristics In general there may be problems if multiple threads make MPI calls The program may fail or behave unexpectedly If MPI calls must be made from within a thread they should be made only by one thread Compiling Use the appropriate MPI compile command for the platform and language of choice Be sure to include the required Pthreads flag as shown in the Compiling Threaded Programs section An example code that uses both MPI and Pthreads is available below The serial threadsonly MPIonly and MPIwith threads versions demonstrate one possible progression Serial Pthreads only MPI only MPI with pthreads makefile Topics Not Covered Several features of the Pthreads API are not covered in this tutorial These are listed below See the Pthread Library Routines Reference section for more information Thread Scheduling Implementations will differ on how threads are scheduled to run In most cases the default mechanism is adequate The Pthreads API provides routines to explicitly set thread scheduling policies and priorities which may override the default mechanisms The API does not require implementations to support these features Keys ThreadSpecific Data As threads call and return from different routines the local data on a threads stack comes and goes To preserve stack data you can usually pass it as an argument from one routine to the next or else store the data in a global variable associated with a thread Pthreads provides another possibly more convenient and versatile way of accomplishing this through keys Mutex Protocol Attributes and Mutex Priority Management for the handling of priority inversion problems Condition Variable Sharing across processes Thread Cancellation Threads and Signals Synchronization constructs barriers and locks Pthread Exercise 2 Mutexes Condition Variables and Hybrid MPI with Pthreads Overview Login to the LC workshop cluster if you are not already logged in Mutexes review and run the provided example codes Condition variables review and run the provided example codes Hybrid MPI with Pthreads review and run the provided example codes GO TO THE EXERCISE HERE POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 27 of 31 07302014 0900 PM This completes the tutorial Please complete the online evaluation form unless you are doing the exercise in which case please complete it at the end of the exercise Where would you like to go now Exercise Agenda Back to the top References and More Information Author Blaise Barney Livermore Computing POSIX Standard wwwunixorgversion3ieeestdhtml Pthreads Programming B Nichols et al OReilly and Associates Threads Primer B Lewis and D Berg Prentice Hall Programming With POSIX Threads D Butenhof Addison Wesley Programming With Threads S Kleiman et al Prentice Hall Appendix A Pthread Library Routines Reference For convenience an alphabetical list of Pthread routines linked to their corresponding man page is provided below pthreadatfork pthreadattrdestroy pthreadattrgetdetachstate pthreadattrgetguardsize pthreadattrgetinheritsched pthreadattrgetschedparam pthreadattrgetschedpolicy pthreadattrgetscope pthreadattrgetstack pthreadattrgetstackaddr pthreadattrgetstacksize pthreadattrinit pthreadattrsetdetachstate pthreadattrsetguardsize pthreadattrsetinheritsched pthreadattrsetschedparam POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 28 of 31 07302014 0900 PM pthreadattrsetschedpolicy pthreadattrsetscope pthreadattrsetstack pthreadattrsetstackaddr pthreadattrsetstacksize pthreadbarrierdestroy pthreadbarrierinit pthreadbarrierwait pthreadbarrierattrdestroy pthreadbarrierattrgetpshared pthreadbarrierattrinit pthreadbarrierattrsetpshared pthreadcancel pthreadcleanuppop pthreadcleanuppush pthreadcondbroadcast pthreadconddestroy pthreadcondinit pthreadcondsignal pthreadcondtimedwait pthreadcondwait pthreadcondattrdestroy pthreadcondattrgetclock pthreadcondattrgetpshared pthreadcondattrinit pthreadcondattrsetclock pthreadcondattrsetpshared pthreadcreate pthreaddetach pthreadequal pthreadexit pthreadgetconcurrency pthreadgetcpuclockid pthreadgetschedparam pthreadgetspecific pthreadjoin pthreadkeycreate pthreadkeydelete pthreadkill pthreadmutexdestroy pthreadmutexgetprioceiling pthreadmutexinit pthreadmutexlock pthreadmutexsetprioceiling pthreadmutextimedlock pthreadmutextrylock pthreadmutexunlock pthreadmutexattrdestroy pthreadmutexattrgetprioceiling pthreadmutexattrgetprotocol pthreadmutexattrgetpshared pthreadmutexattrgettype pthreadmutexattrinit pthreadmutexattrsetprioceiling pthreadmutexattrsetprotocol pthreadmutexattrsetpshared pthreadmutexattrsettype pthreadonce pthreadrwlockdestroy pthreadrwlockinit pthreadrwlockrdlock pthreadrwlocktimedrdlock pthreadrwlocktimedwrlock pthreadrwlocktryrdlock pthreadrwlocktrywrlock POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 29 of 31 07302014 0900 PM pthreadrwlockunlock pthreadrwlockwrlock pthreadrwlockattrdestroy pthreadrwlockattrgetpshared pthreadrwlockattrinit pthreadrwlockattrsetpshared pthreadself pthreadsetcancelstate pthreadsetcanceltype pthreadsetconcurrency pthreadsetschedparam pthreadsetschedprio pthreadsetspecific pthreadsigmask pthreadspindestroy pthreadspininit pthreadspinlock pthreadspintrylock pthreadspinunlock pthreadtestcancel httpscomputingllnlgovtutorialspthreads Last Modified 07242014 165640 blaisebllnlgov UCRLMI133316 POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 30 of 31 07302014 0900 PM POSIX Threads Programming httpscomputingllnlgovtutorialspthreads 31 of 31 07302014 0900 PM