Outline of Material for Final Exam

Parallel Programming Platforms

• Flynn's Taxonomy (SISD,SIMD,MIMD)
• Understand discussion in section 2.3.1
• Also understand SPMD (Single Program Multiple Data)
• Memory Access Architectures
Understand:
• Shared Memory Multiprocessor System [Figure 2.5]
• single address space
• uniform memory access (UMA)/ nonuniform memory access (NUMA)
• distributed shared memory system
• Message Passing Multicomputer (Distributed Memory)
• Network Models
• Be able to discuss the implementation, cost, and expected execution times of the following switching networks
• cross bar switch
• bus
• Omega network
• Be prepared to identify and discuss blocking in an omega network (figure 2.13)
• Be prepared to draw a small omega network configuration
• Review homework!
• Be able to discuss static network topologies
• Star Connected
• Completely Connected
• Linear Arrays, Meshes, and k-d Meshes
• Hypercube
• Tree and Fat Tree Networks
• Be able to compute Diameter, Bisection Width, and Cost for Linear, Ring, Mesh, Torus, Tree, and Hypercube networks.
• Understand how to construct a d-dimensional hypercube recursively.
• Be able to define and compute Diameter, Connectivity, Bisection Width, Bisection Bandwidth, and Cost for any arbitrary network.
• Communication Cost Models for Static Interconnection Networks
Know and Understand:
• communication latency
• Communication times for
• Store and Forward Routing
• Cut-Through Routing
• Using Simplified Cost model (and rational for simplification)
• startup time, ts
• Per-word transfer time tw
• hop time th
• Understand Mapping and its impact on performance
• Be able to define and discuss congestion, dilation, and expansion. How does each of these measures of graph mappings impact communication performance.
• PRAM Models
• Be able to discuss the PRAM machine architecture and assumptions
• Understand Types of PRAM models:
Combining CRCW, Priority CRCW, Arbitrary CRCW, Common CRCW, CREW, and CREW.
• Review Homework problems!

MPI and Message Passing

• Be prepared to describe or discuss the roll or function of the following MPI primitives
• MPI_Init()
• What Arguments are passed to this routine?
• What is the purpose of this call?
• MPI_Finalize()
• MPI_COMM_WORLD
• What are communicators and what role does this identifier play in the creation and use of communicators in MPI?
• What calls are related to communicators in the MPI library? What do these calls do?
• MPI_Comm_size()
• MPI_Comm_rank()
• MPI_Comm_dup()
• MPI_Comm_split()
• MPI_Send()
• What is passed to this call and what purpose do these arguments serve?
• MPI_Recv()
• What arguments are passed to this call? What arguments are different from MPI_Send()? Why are there differences?
• MPI_Sendrecv()
• What purpose does this call serve?
• What are the arguments that this call takes?
• MPI non-blocking operations
• Be able to describe how to perform non-blocking operations in MPI.
• Why do we need non-blocking communication operations?
• MPI_Isend()
• How is the Isend different from send? What does it do that is different? How are the arguments different?
• MPI_Irecv()
• How is the Irecv different from the recv? What does it do that is different? How are the arguments different?
• MPI_Test()
• What does this call do? Why would it be useful?
• MPI_Wait()
• What does this call do? How is it different from a MPI_Test()?
• MPI_Request_free()
• What is a MPI_Request? Why would you want to free it?
• MPI collective Communications
• What are collective communications in MPI?
• Can we perform a collective communication of a subset of processors? If so, how do we do that?
• Do collective communications use tags? Why or why not?
• Know the following Collective Calls: What they do and what arguments they have.
• MPI_Barrier()
• MPI_Bcast()
• MPI_Reduce()
• MPI_Allreduce()
How is this different from MPI_Reduce()?
• MPI_Gather()
• MPI_Scatter()
What chapter 4 communication algorithm is this implementing?
• MPI_Allgather()
What chapter 4 communication algorithm is this implementing?
• MPI_Alltoall
What chapter 4 communication algorithm is this implementing?

Parallel Collective Communication Primitives

• Be able to define One-to-All, All-to-All, scatter, gather and All-to-All Personalized communication primitives. (In other words, be able to reconstruct Figures 4.1, 4.8, 4.14, and 4.16 from the text)
• One-to-All Broadcast
• Be able to describe and analyze the Hypercube One-to-All broadcast algorithm.
• Be prepared to answer questions about the algorithms as listed in program form in Algorithm 4.1, and 4.2 from the text.
• Understand how the One-to-All broadcast algorithm can be used as a basis for performing the Single-Node Accumulation.
• Understand and be prepared to discuss the use of relabeling processor ID's using XOR to achieve the generalized algorithm
• All-to-All Broadcast
• Understand the All-to-All broadcast algorithms for the ring and Hypercube topology. In particular, pay attention to message sizes at each step of the algorithm.
• Be prepared to answer questions about the algorithms as listed in program form in Algorithm 4.4, 4.5, and 4.6 from handout.
• Be able to describe how the All-to-All broadcast can be used to as a basis for performing the Reduction or Multinode Accumulation operations.
• Know the difference between an All-to-All reduction and an All-Reduce
• One-to-All Personalized Communication
• Understand how this algorithm is a variation of the One-to-All broadcast except with shrinking message sizes.
• Be able to derive and analyze this algorithm on a Hypercube network by generalizing the One-to-All (scatter,gather) algorithm results.
• All-to-All Personalized Communications
• Be able to describe and analyze the the Hypercube algorithm.
• Be able to describe and analyze the E-cube routing algorithm.
• Be able to generalize the ts + m tw performance model to take in account network bisection width.
• Be able to describe how the all-to-all personalized algorithm relates to matrix transpose.
• Prefix-Sum algorithm
• Know what a prefix-sum is and how it is used
• Be able to describe the hypercube algorithm
• Be able to estimate running time using the ts + m tw communication cost model
• Communication optimizations
• Be able to determine if an argument is optimal based on bisection bandwidth arguments.
• Be able to discuss and analyze the optimizations of communication algorithms as described in section 4.7
• Know table 4.1 and 4.2 for the algorithms we have covered.
• Be able to derive timing measurements for any of the above algorithms!
• Study the Homework Assignments!

Principles of Parallel Algorithm Design

• Algorithm Decomposition
• Be able to define Tasks and Task Dependency Graphs
• Be able to discuss Task Interactions
• granularity (fine or coarse grained)
• degree of concurrency (what does it measure)
• maximum degree of concurrency
• average degree of concurrency
• What is a Critical Path? Why is it important to the analysis of parallel algorithm performance?
• What is a task interaction graph? How is it different from a task dependency graph? What do we use a task interaction graph for?
• Processes, Mapping, and Processors: Understand the roll each of these play in the design of parallel algorithms.
• Decomposition Techniques
• Recursive Decomposition
• What is recursive decomposition
• Well suited to divide and conquer algorithms (such as quick-sort)
• Data Decomposition
• Be able to define data decomposition.
• Be able to discuss: Mapping data partitions to task partitions, partitioning input data v.s. Output Data, and owner-computes rule.
• Exploratory Decomposition
• What is Exploratory Decomposition?
• When would you apply Exploratory Decomposition?
• What types of problems are well suited for exploratory decomposition?
• Speculative Decomposition
• What is Speculative Decomposition?
• How is Speculative Decomposition different from Exploratory Decomposition?
• When would you you apply speculative decomposition?
• Hybrid Decompositions
• What are hybrid decompositions?
• Be able to discuss an example of hybrid decompositions used to design a parallel algorithm.
• Characteristics of Tasks and Interactions
• Be able to define and discuss dynamic task generation. What problems does this present for parallel programs? How do we deal with these problems?
• How does the size of data associated with tasks effect our decisions regarding parallel algorithm implementation?
• How can we take advantage of static task interactions to improve performance? How is this question related to one-way v.s. two-way (one-sided, v.s. two-sided) communication?
• What is the difference between regular and irregular interactions? What is the difference between irregular and dynamic interactions?
• What is the impact of read-only access to data v.s. Read-Write?
• What do we mean when we say an interaction is two-way or two-sided?
• What are some static mapping techniques? (e.g. Array Distribution schemes, block, cyclic, block-cyclic) What are there so many mappings?
• What is graph partitioning? When is it important for parallel algorithm design?
• What are hierarchical Mappings?
• Schemes for Dynamic Mapping
• What is the tension between centralized and distributed schemes?
• What is self scheduling and chunk-scheduling?
• Methods for Containing Interaction Overheads
Be able to define and discuss the eight methods for containing interaction overheads in parallel programs.
• Maximizing Data Locality
• Minimizing Volume of Data Exchange
• Minimize Frequency of Interactions
• Minimizing Contention and Hot Spots
• Overlapping Computations with Interactions
• Replicating Data or Computations
• Using Optimized Collective Interaction Operations
• Overlapping Interactions with Other Interactions
• Parallel Algorithm Models
Be able to define and compare/contrast the different parallel algorithm models ads discussed in section 3.6.
• The Data-Parallel model
• The Task-Graph model
• The Work Pool Model
• The Master-Slave Model
• The Pipeline or Producer-Consumer model
• Review Homework!

Performance and Scalability of Parallel Systems

• General Principles
• Speedup
• Parallel Speedup, S = t1/tp
• Ideal Speedup, S_ideal = p
• Actual Speedup, S_actual = t_best/tp = S*E_a = S_ideal*E_p*E_a
• Efficiency
• Algorithmic Efficiency, E_a = T_best/t1
• Parallel Efficiency, E_p = S/S_ideal = S/p
• System Efficiency, E = E_a*E_p
• Amdahl's Law
If a parallel algorithm consists of a fully parallel part and a fully serial part such that the fraction of instructions that are fully serial is given by f , then the speedup for this parallel algorithm can be given by:

tp = ((1-f)/p + f)*t1

In other words speedup is given by

S = 1/((1-f)/p + f).

Taking the limit as p increases to infinity gives an upper bound on speedup such that

S <= 1/f

• Be able to define Speedup, Efficiency, Cost, Cost Optimal, Problem Size, Overhead Function, and Isoefficiency Function.
• Be able to determine if an algorithm is Cost Optimal.
• Be able to determine if an algorithm has an Isoefficiency function
• Be able to derive the overhead function and use this function to determine the existence of an Isoefficiency function. Be able to show that the overhead function is related to the isoefficiency function when one exists.
• Understand why there is a lower bound on the isoefficiency function as described in section 5.4.4 from text.
• What is the degree of concurrency of a parallel algorithm?
• What is the relationship between degree of concurrency and the isoefficiency function? See section 5.4.5 from text.
• What are the sources of parallel overhead? ANS: Communication time, Load Imbalance, and additional computations performed by the parallel algorithm.
• What is scaled speedup? How do we measure it?
• What is memory and time-constrained scaled speedup? How do we measure these?
• Study the Homework Assignments!

Parallel Sorting Algorithms

• Fundamentals
• Be able to define internal and external sorting algorithms.
• What is the compare-exchange operation? How is it implemented in parallel
• What is the compare-split operation? How much time does it require?
• What is a comparator? How do you draw one? What is a increasing or decreasing comparator and how do you represent it (draw it).
• Sorting Networks
• Be able to define a bitonic sequence, bitonic split and bitonic merge.
• What is a bitonic merging network? What does it do? What is its size? What is its depth? What does it look like?
• What is a bitonic sorting network? What does it do? What is its size? What is its depth? What does it look like?
• How do we map a bitonic sorting network onto a hypercube? What does the communication patterns look like?
• Be able to derive execution times for the bitonic sorting algorithm on a hypercube architecture. Be able to discuss both mapping one element per processor and many elements per processor.
• Be able to perform a scalability analysis of the bitonic sorting algorithm on the hypercube architecture.
• Quick-sort
• Why is it difficult to develop an efficient parallel quick-sort algorithm by simply running independent subproblems in parallel?
• For the shared memory quicksort algorithm, how is the array partitioned?
• For the shared memory quicksort algorithm, how is load balanced?
• What is the prefix sum operation used for in the shared memory quicksort algorithm?
• Be able to derive the running time of the quicksort algorithm.
• Be able to derive the iso-efficiency function of the quicksort algorithm.
• How do we modify the shared memory quicksort algorithm to generate a message-passing formulation?
• How does pivot selection affect the algorithm performance? Is pivot selection more or less important for parallel quicksort? Why?
• Sample Sort
• What are splitters?
• How large a sample is selected to find the splitters?
• What communication operation is used after the splitters have been found?
• What are the steps in the sample sort algorithm?
• What is the serial and parallel run time of this algorithm
• As described in the text
• Using a bitonic sort to sort the splitters
• What is the isoefficiency function of this sorting algorithm?

General Algorithm Analysis Knowledge

• Be able to compute timings for algorithms using the network model.
• Be prepared to use standard communication algorithms (broadcast, reduction, etc) when describing and analyzing parallel algorithms
• Be prepared to perform isoefficiency and cost optimality analysis on any of the algorithms covered in the sorting chapter.
• In short, you are responsible for the basic analysis knowledge that we have covered in Test#1

Parallel Matrix Algorithms

• What is one dimensional and two dimensional partitioning?
• Be able to describe and analyze one dimensional partitioned dense matrix vector multiplication for both row and column decomposition.
• Review the concurrency constraints on the isoefficiency function. Be able to derive the isoefficiency bounds due to concurrency for all of the parallel matrix algorithms.
• Be able to describe and analyze the two dimensional partitioned dense matrix vector multiplication algorithm.
• Be able to describe and analyze the simple (naive) dense matrix-matrix multiplication algorithm.
• Be able to describe and analyze Cannon's algorithm for dense matrix-matrix multiplication.
• Be able to describe and analyze the DNS algorithm for matrix-matrix multiplication.
• Be able to compare and contrast various algorithms (from the set above) for dense matrix-matrix multiply.

Study Homework Assignments!