Outline of Material for Final

General Principles

• Speedup
• Parallel Speedup, S = t1/tp
• Ideal Speedup, S_ideal = p
• Actual Speedup, S_actual = t_best/tp = S*E_a = S_ideal*E_p*E_a
• Efficiency
• Algorithmic Efficiency, E_a = T_best/t1
• Parallel Efficiency, E_p = S/S_ideal = S/p
• System Efficiency, E = E_a*E_p
• Amdahl's Law
If a parallel algorithm consists of a fully parallel part and a fully serial part such that the fraction of instructions that are fully serial is given by f , then the speedup for this parallel algorithm can be given by:

tp = ((1-f)/p + f)*t1

In other words speedup is given by

S = 1/((1-f)/p + f).

Taking the limit as p increases to infinity gives an upper bound on speedup such that

S <= 1/f

Models of Parallel Computers

• Flynn's Taxonomy (SISD,SIMD,MIMD)
• Understand discussion on page 10 in text.
• Also understand SPMD (Single Program Multiple Data)
• Memory Access Architectures
Understand:
• Shared Memory Multiprocessor System [Figure 1.3]
• single address space
• uniform memory access (UMA)/ nonuniform memory access (NUMA)
• distributed shared memory system
• Message Passing Multicomputer (Distributed Memory [Figure 1.4]
• Network Models
• Be able to compute Diameter, Bisection Width, and Cost for Linear, Ring, Mesh, Torus, Tree, and Hypercube networks.
• Understand how to construct a d-dimensional hypercube recursively.
• Be able to define and compute Diameter, Connectivity, Bisection Width, Bisection Bandwidth, and Cost for any arbitrary network.
• Communication Cost Models for Static Interconnection Networks
Know and Understand:
• communication latency ts+m tw
• startup time, ts
• Per-word transfer time tw
• PRAM Models
• Be able to discuss the PRAM machine architecture and assumptions (See Handout from Algorithms text)
• Understand Types of PRAM models:
Combining CRCW, Priority CRCW, Arbitrary CRCW, Common CRCW, CREW, and CREW.
• Review Homework problem!
• LogP Model
• Be able to define the Model Parameters, L, o, g, and P.
• Understand the discussion of estimating running time of the broadcast under the LogP model as discussed in class and the LogP paper.
• Be able to use the LogP model to compute running times of programs.
• Review Homework problem!
• BSP Model
• Be able to discuss the BSP execution model. (What is the BSP execution model and what does it consist of?)
• Be able to use the cost model for BSP [w + hg + l] to estimate running times of BSP algorithms.
• What is a h-relation? A 1-relation? A p-relation?
• What is a superstep in the BSP model?
• What is parallel slackness?
• What role does the barrier operation play in the BSP model?
• Review Homework problem!

MPI and Message Passing

• Be able to discuss how deadlock is possible when using MPI_Send() and MPI_Recv() as discussed in class.
• Be able to discuss how to call and use MPI_Alltoallv() as discussed in class.

Programming Patterns

• Be able to discuss the bag-of-tasks parallel programming pattern. Be prepared to estimate performance of various scenarios in the bag-of-tasks pattern.
• Be able to discuss how to add N numbers on p processors. Also be able to describe how to extend this to more general operators to form reduction operations. What properties are required of an operator for it to be used in a parallel reduction operation?

• Be able to define One-to-All, All-to-All, scatter, gather and All-to-All Personalized communication primitives. (In other words, be able to reconstruct Figure's 4.1, 4.8, 4.14, and 4.16 from handout)
• One-to-All Broadcast
• Be able to describe and analyze the Hypercube One-to-All broadcast algorithm.
• Be prepared to answer questions about the algorithms as listed in program form in Algorithm 4.1, and 4.2 from handout.
• Understand how the One-to-All broadcast algorithm can be used as a basis for performing the Single-Node Accumulation.
• All-to-All Broadcast
• Understand the All-to-All broadcast algorithms for the ring and Hypercube topology. In particular, pay attention to message sizes at each step of the algorithm.
• Be prepared to answer questions about the algorithms as listed in program form in Algorithm 4.4, 4.5, and 4.6 from handout.
• Be able to describe how the All-to-All broadcast can be used to as a basis for performing the Reduction or Multinode Accumulation operations.
• One-to-All Personalized Communication
• Understand how this algorithm is a variation of the One-to-All broadcast except with shrinking message sizes.
• Be able to derive and analyze this algorithm on a Hypercube network by generalizing the One-to-All algorithm results.
• All-to-All Personalized Communications
• Be able to describe and analyze the the Hypercube algorithm.
• Know table 4.1 and 4.2 for the algorithms we have covered.
• Be able to derive timing measurements for any of the above algorithms!
• Study the Homework Assignments!

Performance and Scalability of Parallel Systems

• Be able to define Speedup, Efficiency, Cost, Cost Optimal, Problem Size, Overhead Function, and Isoefficiency Function.
• Be able to determine if an algorithm is Cost Optimal.
• Be able to determine if an algorithm has an Isoefficiency function
• Be able to derive the overhead function and use this function to determine the existence of an Isoefficiency function. Be able to show that the overhead function is related to the isoefficiency function when one exists.
• Understand why there is a lower bound on the isoefficiency function as described in section 5.4.4 from handout
• What is the degree of concurrency of a parallel algorithm?
• What is the relationship between degree of concurrency and the isoefficiency function? See section 5.4.5 from handout.
• What are the sources of parallel overhead? ANS: Communication time, Load Imbalance, and additional computations performed by the parallel algorithm.
• What is scaled speedup? How do we measure it?
• What is memory and time-constrained scaled speedup? How do we measure these?
• Study the Homework Assignments!

General Algorithm Analysis Knowledge

• Be able to compute timings for algorithms using the network model.
• Be prepared to use standard communication algorithms (broadcast, reduction, etc) when describing and analyzing parallel algorithms
• Be prepared to perform isoefficiency and cost optimality analysis on any of the algorithms covered in the following sections.
• In short, you are responsible for the basic analysis knowledge that we have covered in Test#1

Parallel Sorting Algorithms

• Fundamentals
• Be able to define internal and external sorting algorithms.
• What is the compare-exchange operation? How is it implemented in parallel
• What is the compare-split operation? How much time does it require?
• What is a comparator? How do you draw one? What is a increasing or decreasing comparator and how do you represent it (draw it).
• What does it mean to enumerate processors with respect to sorting algorithms? Why is it important for parallel sorting algorithms?
• Sorting Networks
• Be able to define a bitonic sequence, bitonic split and bitonic merge.
• What is a bitonic merging network? What does it do? What is its size? What is its depth? What does it look like?
• What is a bitonic sorting network? What does it do? What is its size? What is its depth? What does it look like?
• How do we map a bitonic sorting network onto a hypercube? What does the communication patterns look like?
• Be able to derive execution times for the bitonic sorting algorithm on a hypercube architecture. Be able to discuss both mapping one element per processor and many elements per processor.
• Be able to perform a scalability analysis of the bitonic sorting algorithm on the hypercube architecture.
• Bubble Sort and Its Variants
• Be able to discuss the problem of parallelizing the bubble sort algorithm.
• Be able to discuss the odd-even transposition algorithm. How many steps does it have? What is the computation time? Is the algorithm scalable?
• Quick-sort
• Why is it difficult to develop an efficient parallel quick-sort algorithm by simply running independent subproblems in parallel?
• What steps are involved in the hypercube quick-sort algorithm?
• Why is pivot selection critical for an efficient hypercube sorting algorithm?
• What is the scalability of the quick-sort algorithm?
• Be prepared to estimate parallel running times for the hypercube based quick-sort algorithm.
• Bucket Sort and Sample Sort
• What is the bucket sort? What assumptions does it make? What is its serial run time?
• What steps are involved in the parallel bucket sort algorithm?
• What is a Sample Sort, how does it differ from Bucket Sort?
• How do we form a sample set in the Sample Sort? How big is the sample?
• How do we sort the Sample in Sample Sort?
• What is the performance bottleneck for the Sample Sort?
• Be able to estimate running time for these algorithms.
• Be able to perform a scalability analysis of these algorithms.

Dense Matrix Algorithms

• What is one dimensional and two dimensional partitioning?
• Be able to describe and analyze one dimensional partitioned dense matrix vector multiplication for both row and column decomposition.
• Review the concurrency constraints on the isoefficiency function. Be able to derive the isoefficiency bounds due to concurrency for all of the parallel matrix algorithms.
• Be able to describe and analyze the two dimensional partitioned dense matrix vector multiplication algorithm.
• Be able to describe and analyze the simple dense matrix-matrix multiplication algorithm.
• Be able to describe and analyze Cannon's algorithm for dense matrix-matrix multiplication.
• Be able to describe and analyze the DNS algorithm for matrix-matrix multiplication.
• Be able to compare and contrast various algorithms (from the set above) for dense matrix-matrix multiply.

Study Homework Assignments!