Outline of Material for Final Exam

Parallel Programming Platforms

Flynn's Taxonomy (SISD,SIMD,MIMD)

Understand discussion in section 2.3.1
Also understand SPMD (Single Program Multiple Data)

Memory Access Architectures

Shared Memory Multiprocessor System [Figure 2.5]

single address space
uniform memory access (UMA)/ nonuniform memory access (NUMA)
distributed shared memory system

Message Passing Multicomputer (Distributed Memory

Network Models

Be able to discuss the implementation, cost, and expected execution times of the following switching networks

cross bar switch
bus
Omega network

Be prepared to identify and discuss blocking in an omega network (figure 2.13)
Be prepared to draw a small omega network configuration
Review homework!

Be able to discuss static network topologies

Star Connected
Completely Connected
Linear Arrays, Meshes, and k-d Meshes
Hypercube
Tree and Fat Tree Networks

Be able to compute Diameter, Bisection Width, and Cost for Linear, Ring, Mesh, Torus, Tree, and Hypercube networks.
Understand how to construct a d-dimensional hypercube recursively.
Be able to define and compute Diameter, Connectivity, Bisection Width, Bisection Bandwidth, and Cost for any arbitrary network.
Communication Cost Models for Static Interconnection Networks

communication latency
Communication times for

Store and Forward Routing
Cut-Through Routing
Using Simplified Cost model (and rational for simplification)

startup time, ts
Per-word transfer time tw
hop time th

Routing Mechanisms for Interconnection Networks

Be able to define:

minimal/nonmiminal routing
deterministic routing
adaptive routing
dimension-ordered routing
E-cube routing

Understand Mapping and its impact on performance

Be able to define and discuss congestion, dilation, and expansion. How does each of these measures of graph mappings impact communication performance.

PRAM Models

Be able to discuss the PRAM machine architecture and assumptions
Understand Types of PRAM models:

Combining CRCW, Priority CRCW, Arbitrary CRCW, Common CRCW, CREW, and CREW.

Review Homework problems!

MPI and Message Passing

Be prepared to describe or discuss the roll or function of the following MPI primitives

MPI_Init()
MPI_Finalize()
MPI_COMM_WORLD
MPI_Comm_size()
MPI_Comm_rank()
MPI_Send()
MPI_Recv()
MPI_Sendrecv()
MPI_Isend()
MPI_Irecv()
MPI_Test()
MPI_Wait()
MPI_Request_free()
MPI_Barrier()
MPI_Bcast()
MPI_Reduce()

Be able to discuss how deadlock is possible when using MPI_Send() and MPI_Recv() as discussed in class.
Be able to discuss buffering in message passing protocols.< /LI>
Review Homework!

Principles of Parallel Algorithm Design

Algorithm Decomposition

Be able to define Tasks and Task Dependency Graphs
Be able to discuss Task Interactions

granularity (fine or coarse grained)
degree of concurrency (what does it measure)

maximum degree of concurrency
average degree of concurrency

What is a Critical Path? Why is it important to the analysis of parallel algorithm performance?
What is a task interaction graph? How is it different from a task dependency graph? What do we use a task interaction graph for?
Processes, Mapping, and Processors: Understand the roll each of these play in the design of parallel algorithms.
Decomposition Techniques

Recursive Decomposition

What is recursive decomposition
Well suited to divide and conquer algorithms (such as quick-sort)

Data Decomposition

Be able to define data decomposition.
Be able to discuss: Mapping data partitions to task partitions, partitioning input data v.s. Output Data, and owner-computes rule.

Exploratory Decomposition

What is Exploratory Decomposition?
When would you apply Exploratory Decomposition?
What types of problems are well suited for exploratory decomposition?

Speculative Decomposition

What is Speculative Decomposition?
How is Speculative Decomposition different from Exploratory Decomposition?
When would you you apply speculative decomposition?

Hybrid Decompositions

What are hybrid decompositions?
Be able to discuss an example of hybrid decompositions used to design a parallel algorithm.

Characteristics of Tasks and Interactions

Be able to define and discuss dynamic task generation. What problems does this present for parallel programs? How do we deal with these problems?
How does the size of data associated with tasks effect our decisions regarding parallel algorithm implementation?
How can we take advantage of static task interactions to improve performance? How is this question related to one-way v.s. two-way (one-sided, v.s. two-sided) communication?
What is the difference between regular and irregular interactions? What is the difference between irregular and dynamic interactions?
What is the impact of read-only access to data v.s. Read-Write?
What do we mean when we say an interaction is two-way or two-sided?
What are some static mapping techniques? (e.g. Array Distribution schemes, block, cyclic, block-cyclic) What are there so many mappings?
What is graph partitioning? When is it important for parallel algorithm design?
What are hierarchical Mappings?
Schemes for Dynamic Mapping

What is the tension between centralized and distributed schemes?
What is self scheduling and chunk-scheduling?

Methods for Containing Interaction Overheads

Be able to define and discuss the eight methods for containing interaction overheads in parallel programs.

Maximizing Data Locality
Minimizing Volume of Data Exchange
Minimize Frequency of Interactions
Minimizing Contention and Hot Spots
Overlapping Computations with Interactions
Replicating Data or Computations
Using Optimized Collective Interaction Operations
Overlapping Interactions with Other Interactions

Parallel Algorithm Models

Be able to define and compare/contrast the different parallel algorithm models ads discussed in section 3.6.

The Data-Parallel model
The Task-Graph model
The Work Pool Model
The Master-Slave Model
The Pipeline or Producer-Consumer model

Review Homework!

Parallel Collective Communication Primitives

Be able to define One-to-All, All-to-All, scatter, gather and All-to-All Personalized communication primitives. (In other words, be able to reconstruct Figure's 4.1, 4.8, 4.14, and 4.16 from the text)
One-to-All Broadcast

Be able to describe and analyze the Hypercube One-to-All broadcast algorithm.
Be prepared to answer questions about the algorithms as listed in program form in Algorithm 4.1, and 4.2 from the text.
Understand how the One-to-All broadcast algorithm can be used as a basis for performing the Single-Node Accumulation.
Understand and be prepared to discuss the use of relabeling processor ID's using XOR to achieve the generalized algorithm

All-to-All Broadcast

Understand the All-to-All broadcast algorithms for the ring and Hypercube topology. In particular, pay attention to message sizes at each step of the algorithm.
Be prepared to answer questions about the algorithms as listed in program form in Algorithm 4.4, 4.5, and 4.6 from handout.
Be able to describe how the All-to-All broadcast can be used to as a basis for performing the Reduction or Multinode Accumulation operations.
Know the difference between an All-to-All reduction and an All-Reduce

One-to-All Personalized Communication

Understand how this algorithm is a variation of the One-to-All broadcast except with shrinking message sizes.
Be able to derive and analyze this algorithm on a Hypercube network by generalizing the One-to-All (scatter,gather) algorithm results.

All-to-All Personalized Communications

Be able to describe and analyze the the Hypercube algorithm.
Be able to describe and analyze the E-cube routing algorithm.
Be able to generalize the ts + m tw performance model to take in account network bisection width.
Be able to describe how the all-to-all personalized algorithm relates to matrix transpose.

Prefix-Sum algorithm

Know what a prefix-sum is and how it is used
Be able to describe the hypercube algorithm
Be able to estimate running time using the ts + m tw communication cost model

Communication optimizations

Be able to determine if an argument is optimal based on bisection bandwidth arguments.
Be able to discuss and analyze the optimizations of communication algorithms as described in section 4.7

Know table 4.1 and 4.2 for the algorithms we have covered.
Be able to derive timing measurements for any of the above algorithms!
Study the Homework Assignments!

Performance and Scalability of Parallel Systems

General Principles

Speedup

Parallel Speedup, S = t1/tp
Ideal Speedup, S_ideal = p
Actual Speedup, S_actual = t_best/tp = S*E_a = S_ideal*E_p*E_a

Efficiency

Algorithmic Efficiency, E_a = T_best/t1
Parallel Efficiency, E_p = S/S_ideal = S/p
System Efficiency, E = E_a*E_p

Amdahl's Law

tp = ((1-f)/p + f)*t1

In other words speedup is given by

S = 1/((1-f)/p + f).

Taking the limit as p increases to infinity gives an upper bound on speedup such that

S <= 1/f

Be able to define Speedup, Efficiency, Cost, Cost Optimal, Problem Size, Overhead Function, and Isoefficiency Function.
Be able to determine if an algorithm is Cost Optimal.
Be able to determine if an algorithm has an Isoefficiency function
Be able to derive the overhead function and use this function to determine the existence of an Isoefficiency function. Be able to show that the overhead function is related to the isoefficiency function when one exists.
Understand why there is a lower bound on the isoefficiency function as described in section 5.4.4 from text.
What is the degree of concurrency of a parallel algorithm?
What is the relationship between degree of concurrency and the isoefficiency function? See section 5.4.5 from text.
What are the sources of parallel overhead? ANS: Communication time, Load Imbalance, and additional computations performed by the parallel algorithm.
What is scaled speedup? How do we measure it?
What is memory and time-constrained scaled speedup? How do we measure these?
Study the Homework Assignments!

General Algorithm Analysis Knowledge

Be able to compute timings for algorithms using the network model.
Be prepared to use standard communication algorithms (broadcast, reduction, etc) when describing and analyzing parallel algorithms
Be prepared to perform isoefficiency and cost optimality analysis on any of the algorithms covered in the following sections.
In short, you are responsible for the basic analysis knowledge that we have covered in Test#2

Parallel Sorting Algorithms

Fundamentals

Be able to define internal and external sorting algorithms.
What is the compare-exchange operation? How is it implemented in parallel
What is the compare-split operation? How much time does it require?
What is a comparator? How do you draw one? What is a increasing or decreasing comparator and how do you represent it (draw it).

Sorting Networks

Be able to define a bitonic sequence, bitonic split and bitonic merge.
What is a bitonic merging network? What does it do? What is its size? What is its depth? What does it look like?
What is a bitonic sorting network? What does it do? What is its size? What is its depth? What does it look like?
How do we map a bitonic sorting network onto a hypercube? What does the communication patterns look like?
Be able to derive execution times for the bitonic sorting algorithm on a hypercube architecture. Be able to discuss both mapping one element per processor and many elements per processor.
Be able to perform a scalability analysis of the bitonic sorting algorithm on the hypercube architecture.

Bubble Sort and Its Variants

Be able to discuss the problem of parallelizing the bubble sort algorithm.
Be able to discuss the odd-even transposition algorithm. How many steps does it have? What is the computation time? Is the algorithm scalable?

Quick-sort

Why is it difficult to develop an efficient parallel quick-sort algorithm by simply running independent subproblems in parallel?
What does the tree represent in the CRCW PRAM quicksort algorithm?
What is the running time of the CRCW PRAM algorithm? Is it cost optimal? Is this algorithm practical?
For the shared memory quicksort algorithm, how is the array partitioned?
For the shared memory quicksort algorithm, how is load balanced?
What is the prefix sum operation used for in the shared memory quicksort algorithm?
Be able to derive the running time of the quicksort algorithm.
Be able to derive the iso-efficiency function of the quicksort algorithm.
How do we modify the shared memory quicksort algorithm to generate a message-passing formulation?
How does pivot selection affect the algorithm performance? Is pivot selection more or less important for parallel quicksort? Why?

Bucket Sort

What is the bucket sort? What assumptions does it make? What is its serial run time?
What steps are involved in the parallel bucket sort algorithm?
Be able to estimate running time for this algorithm.
Be able to perform a scalability analysis on this algorithm.

Sample Sort

What are splitters?
How large a sample is selected to find the splitters?
What communication operation is used after the splitters have been found?
What are the steps in the sample sort algorithm?
What is the serial and parallel run time of this algorithm

As described in the text
Using a bitonic sort to sort the splitters

What is the isoefficiency function of this sorting algorithm?

Parallel Matrix Algorithms

What is one dimensional and two dimensional partitioning?
Be able to describe and analyze one dimensional partitioned dense matrix vector multiplication for both row and column decomposition.
Review the concurrency constraints on the isoefficiency function. Be able to derive the isoefficiency bounds due to concurrency for all of the parallel matrix algorithms.
Be able to describe and analyze the two dimensional partitioned dense matrix vector multiplication algorithm.
Be able to describe and analyze the simple (naive) dense matrix-matrix multiplication algorithm.
Be able to describe and analyze Cannon's algorithm for dense matrix-matrix multiplication.
Be able to describe and analyze the DNS algorithm for matrix-matrix multiplication.
Be able to compare and contrast various algorithms (from the set above) for dense matrix-matrix multiply.

Study Homework Assignments!

Last modified: Wed Apr 27 10:49:27 CDT 2005