Outline of Material for Test #2

Principles of Parallel Algorithm Design

• Algorithm Decomposition
• Be able to define Tasks and Task Dependency Graphs
• Be able to discuss Task Interactions
• granularity (fine or coarse grained)
• degree of concurrency (what does it measure)
• maximum degree of concurrency
• average degree of concurrency
• What is a Critical Path? Why is it important to the analysis of parallel algorithm performance?
• What is a task interaction graph? How is it different from a task dependency graph? What do we use a task interaction graph for?
• Processes, Mapping, and Processors: Understand the roll each of these play in the design of parallel algorithms.
• Decomposition Techniques
• Recursive Decomposition
• What is recursive decomposition
• Well suited to divide and conquer algorithms (such as quick-sort)
• Data Decomposition
• Be able to define data decomposition.
• Be able to discuss: Mapping data partitions to task partitions, partitioning input data v.s. Output Data, and owner-computes rule.
• Exploratory Decomposition
• What is Exploratory Decomposition?
• When would you apply Exploratory Decomposition?
• What types of problems are well suited for exploratory decomposition?
• Speculative Decomposition
• What is Speculative Decomposition?
• How is Speculative Decomposition different from Exploratory Decomposition?
• When would you you apply speculative decomposition?
• Hybrid Decompositions
• What are hybrid decompositions?
• Be able to discuss an example of hybrid decompositions used to design a parallel algorithm.
• Characteristics of Tasks and Interactions
• Be able to define and discuss dynamic task generation. What problems does this present for parallel programs? How do we deal with these problems?
• How does the size of data associated with tasks effect our decisions regarding parallel algorithm implementation?
• How can we take advantage of static task interactions to improve performance? How is this question related to one-way v.s. two-way (one-sided, v.s. two-sided) communication?
• What is the difference between regular and irregular interactions? What is the difference between irregular and dynamic interactions?
• What is the impact of read-only access to data v.s. Read-Write?
• What do we mean when we say an interaction is two-way or two-sided?
• What are some static mapping techniques? (e.g. Array Distribution schemes, block, cyclic, block-cyclic) What are there so many mappings?
• What is graph partitioning? When is it important for parallel algorithm design?
• What are hierarchical Mappings?
• Schemes for Dynamic Mapping
• What is the tension between centralized and distributed schemes?
• What is self scheduling and chunk-scheduling?
• Methods for Containing Interaction Overheads
Be able to define and discuss the eight methods for containing interaction overheads in parallel programs.
• Maximizing Data Locality
• Minimizing Volume of Data Exchange
• Minimize Frequency of Interactions
• Minimizing Contention and Hot Spots
• Overlapping Computations with Interactions
• Replicating Data or Computations
• Using Optimized Collective Interaction Operations
• Overlapping Interactions with Other Interactions
• Parallel Algorithm Models
Be able to define and compare/contrast the different parallel algorithm models ads discussed in section 3.6.
• The Data-Parallel model
• The Task-Graph model
• The Work Pool Model
• The Master-Slave Model
• The Pipeline or Producer-Consumer model
• Review Homework!

Performance and Scalability of Parallel Systems

• General Principles
• Speedup
• Parallel Speedup, S = t1/tp
• Ideal Speedup, S_ideal = p
• Actual Speedup, S_actual = t_best/tp = S*E_a = S_ideal*E_p*E_a
• Efficiency
• Algorithmic Efficiency, E_a = T_best/t1
• Parallel Efficiency, E_p = S/S_ideal = S/p
• System Efficiency, E = E_a*E_p
• Amdahl's Law
If a parallel algorithm consists of a fully parallel part and a fully serial part such that the fraction of instructions that are fully serial is given by f , then the speedup for this parallel algorithm can be given by:

tp = ((1-f)/p + f)*t1

In other words speedup is given by

S = 1/((1-f)/p + f).

Taking the limit as p increases to infinity gives an upper bound on speedup such that

S <= 1/f

• BSP Models
• Be able to describe the Bulk Synchronous Parallel Model
• Be able to define the roll of parameters l and g and h in the BSP nodel.
• Be able to discuss how the BSP model suggests that a binary tree is not optimal for broadcasting. What conditions (e.g. values for l and g ) make the binary tree suboptimal.
• Be able to estimate running time of a a simple algorithm using the BSP model.
• Be able to define Speedup, Efficiency, Cost, Cost Optimal, Problem Size, Overhead Function, and Isoefficiency Function.
• Be able to determine if an algorithm is Cost Optimal.
• Be able to determine if an algorithm has an Isoefficiency function
• Be able to derive the overhead function and use this function to determine the existence of an Isoefficiency function. Be able to show that the overhead function is related to the isoefficiency function when one exists.
• Understand why there is a lower bound on the isoefficiency function as described in section 5.4.4 from text.
• What is the degree of concurrency of a parallel algorithm?
• What is the relationship between degree of concurrency and the isoefficiency function? See section 5.4.5 from text.
• What are the sources of parallel overhead? ANS: Communication time, Load Imbalance, and additional computations performed by the parallel algorithm.
• What is scaled speedup? How do we measure it?
• What is memory and time-constrained scaled speedup? How do we measure these?
• Bulk Synchronous Parallel (BSP) Model
• Be able to describe computations in the BSP model (algorithm described a series of super-steps. What does the super-step comprise?
• What is an h relation in the BSP model?
• How do we model cost in the BSP model?
• What does l, g, and h describe in the cost model? How do these parameters relate to machine performance characteristics?
• In what way is the BSP model more realistic than the PRAM model?
• Study the Homework Assignments!