Ebook: Parallel Programming: for Multicore and Cluster Systems, 3rd Edition

This textbook covers the new development in processor architecture and parallel hardware. It provides detailed descriptions of parallel programming techniques that are necessary for developing efficient programs for multicore processors as well as for parallel cluster systems and supercomputers.

The content of the book consists of three main parts, covering all areas of parallel computing: the architecture of parallel systems, parallel programming models and environments, and the implementation of efficient application algorithms. The emphasis lies on parallel programming techniques needed for different architectures. The first part contains an overview of the architecture of parallel systems, including cache and memory organization, interconnection networks, routing and switching techniques as well as technologies that are relevant for modern and future multicore processors. Issues of power and energy consumption are also covered.

The second part presents parallel programming models, performance models, and parallel programming environments for message passing and shared memory models, including the message passing interface (MPI), Pthreads, Java threads, and OpenMP. For each of these parallel programming environments, the book introduces basic concepts as well as more advanced programming methods and enables the reader to write and run semantically correct and computationally efficient parallel programs. Parallel design patterns, such as pipelining, client-server, or task pools are presented for different environments to illustrate parallel programming techniques and to facilitate the implementation of efficient parallel programs for a wide variety of application areas. Performance models and techniques for runtime analysis are described in detail, as they are a prerequisite for achieving efficiency and high performance. A chapter gives a detailed description of the architecture of GPUs and also contains an introduction into programming approaches for general purpose GPUs concentrating on CUDA and OpenCL. Programming examples are provided to demonstrate the use of the specific programming techniques introduced.

The third part applies the parallel programming techniques from the second part to representative algorithms from scientific computing. The emphasis lies on basic methods for solving linear equation systems, which play an important role for many scientific simulations. The focus of the presentation is the analysis of the algorithmic structure of the different algorithms, which is the basis for a parallelization, and not so much on mathematical properties of the solution methods. For each algorithm, the book discusses different parallelization variants, using different methods and strategies.

The main goal of this book is to present parallel programming techniques that can be used in many situations for many application areas and to enable the reader to develop correct and efficient parallel programs. Many example programs and exercises are provided to support this goal and to show how the techniques can be applied to further applications. The book can be used as a textbook for students as well as a reference book for professionals. The material of the book has been used for courses in parallel programming at different universities for many years.