Outline of Material for Test #1

General Principles

• Speedup
• Parallel Speedup, S = t1/tp
• Ideal Speedup, S_ideal = p
• Actual Speedup, S_actual = t_best/tp = S*E_a = S_ideal*E_p*E_a
• Efficiency
• Algorithmic Efficiency, E_a = T_best/t1
• Parallel Efficiency, E_p = S/S_ideal = S/p
• System Efficiency, E = E_a*E_p
• Amdahl's Law
If a parallel algorithm consists of a fully parallel part and a fully serial part such that the fraction of instructions that are fully serial is given by f , then the speedup for this parallel algorithm can be given by:

tp = ((1-f)/p + f)*t1

In other words speedup is given by

S = 1/((1-f)/p + f).

Taking the limit as p increases to infinity gives an upper bound on speedup such that

S <= 1/f

Models of Parallel Computers

• Flynn's Taxonomy (SISD,SIMD,MIMD)
• Understand discussion on page 10 in text.
• Also understand SPMD (Single Program Multiple Data)
• Memory Access Architectures
Understand:
• Shared Memory Multiprocessor System [Figure 1.3]
• single address space
• uniform memory access (UMA)/ nonuniform memory access (NUMA)
• distributed shared memory system
• Message Passing Multicomputer (Distributed Memory [Figure 1.4]
• Network Models
• Be able to compute Diameter, Bisection Width, and Cost for Linear, Ring, Mesh, Torus, Tree, and Hypercube networks.
• Understand how to construct a d-dimensional hypercube recursively.
• Be able to define and compute Diameter, Connectivity, Bisection Width, Bisection Bandwidth, and Cost for any arbitrary network.
• Communication Cost Models for Static Interconnection Networks
Know and Understand:
• communication latency ts+m tw
• startup time, ts
• Per-word transfer time tw
• PRAM Models
• Be able to discuss the PRAM machine architecture and assumptions (See Handout from Algorithms text)
• Understand Types of PRAM models:
Combining CRCW, Priority CRCW, Arbitrary CRCW, Common CRCW, CREW, and CREW.
• Review Homework problem!
• LogP Model
• Be able to define the Model Parameters, L, o, g, and P.
• Understand the discussion of estimating running time of the broadcast under the LogP model as discussed in class and the LogP paper.
• Be able to use the LogP model to compute running times of programs.
• Review Homework problem!
• BSP Model
• Be able to discuss the BSP execution model. (What is the BSP execution model and what does it consist of?)
• Be able to use the cost model for BSP [w + hg + l] to estimate running times of BSP algorithms.
• What is a h-relation? A 1-relation? A p-relation?
• What is a superstep in the BSP model?
• What is parallel slackness?
• What role does the barrier operation play in the BSP model?
• Review Homework problem!

MPI and Message Passing

• Be able to discuss how deadlock is possible when using MPI_Send() and MPI_Recv() as discussed in class.

Programming Patterns

• Be able to discuss the bag-of-tasks parallel programming pattern. Be prepared to estimate performance of various scenarios in the bag-of-tasks pattern.
• Be able to discuss how to add N numbers on p processors. Also be able to describe how to extend this to more general operators to form reduction operations. What properties are required of an operator for it to be used in a parallel reduction operation?

• Be able to define One-to-All, All-to-All, scatter, gather and All-to-All Personalized communication primitives. (In other words, be able to reconstruct Figure's 4.1, 4.8, 4.14, and 4.16 from handout)
• One-to-All Broadcast
• Be able to describe and analyze the Hypercube One-to-All broadcast algorithm.
• Be prepared to answer questions about the algorithms as listed in program form in Algorithm 4.1, and 4.2 from handout.
• Understand how the One-to-All broadcast algorithm can be used as a basis for performing the Single-Node Accumulation.
• All-to-All Broadcast
• Understand the All-to-All broadcast algorithms for the ring and Hypercube topology. In particular, pay attention to message sizes at each step of the algorithm.
• Be prepared to answer questions about the algorithms as listed in program form in Algorithm 4.4, 4.5, and 4.6 from handout.
• Be able to describe how the All-to-All broadcast can be used to as a basis for performing the Reduction or Multinode Accumulation operations.
• One-to-All Personalized Communication
• Understand how this algorithm is a variation of the One-to-All broadcast except with shrinking message sizes.
• Be able to derive and analyze this algorithm on a Hypercube network by generalizing the One-to-All algorithm results.
• All-to-All Personalized Communications
• Be able to describe and analyze the the Hypercube algorithm.
• Know table 4.1 and 4.2 for the algorithms we have covered.
• Be able to derive timing measurements for any of the above algorithms!
• Study the Homework Assignments!

Performance and Scalability of Parallel Systems

• Be able to define Speedup, Efficiency, Cost, Cost Optimal, Problem Size, Overhead Function, and Isoefficiency Function.
• Be able to determine if an algorithm is Cost Optimal.
• Be able to determine if an algorithm has an Isoefficiency function
• Be able to derive the overhead function and use this function to determine the existence of an Isoefficiency function. Be able to show that the overhead function is related to the isoefficiency function when one exists.
• Understand why there is a lower bound on the isoefficiency function as described in section 5.4.4 from handout
• What is the degree of concurrency of a parallel algorithm?
• What is the relationship between degree of concurrency and the isoefficiency function? See section 5.4.5 from handout.
• What are the sources of parallel overhead? ANS: Communication time, Load Imbalance, and additional computations performed by the parallel algorithm.
• What is scaled speedup? How do we measure it?
• What is memory and time-constrained scaled speedup? How do we measure these?
• Study the Homework Assignments!