School of Computer Science Wiki

• CIS*2030 or ENGG*3640: Successful parallel programming relies on an understanding of the underlying CPU hardware, including how assembly language instructions are executed, and how RAM is organized and referenced.
• CIS*3110: Certain OS topics are highly relevant to parallel programming, including processes and threads, concurrency, virtual memory, memory hierarchy (caching), scheduling and context switching, deadlocks, race conditions, mutexes, and more. Pthreads programming is normally introduced in an OS course, and will be used for one programming assignment.

Today's computer science students are entering a new era in parallel computing, featuring cheap multicores and high-performance clusters, but have received traditional largely-sequential training. This paradigm shift has been called “the end of the lazy programmer era.” This course is aimed at helping soon-to-graduate students (1) move into jobs using current tools for parallel programming, and (2) acquire the theoretical background needed to keep abreast with rapid industry developments and to evolve with them. The textbook will provide foundational knowledge about modern parallel processor architectures and algorithms for organizing concurrent computations. Since parallel programming is all about speed, we will learn ways to measure execution performance and speedup through parallelization.

In terms of practical skills, high-performance (non-shared memory) cluster programming will be introduced via the University of Guelph Pilot library, based on MPI and utilizing message-passing. Programming for multicore shared memory processors will utilize the popular existing parallel programming technique of POSIX threads, and compiler-based OpenMP, supported by the latest suite of Intel tools, as well as Java threads. Heterogeneous architectures–GPUs (graphics processing units) and the Intel Xeon Phi–will be introduced.

1. Historical perspective on concurrency in computing
2. Parallel computer system architectures
   • Abstractions for modeling sequential and parallel computation
   • Homogeneous vs. heterogeneous multiprocessors
   • Shared vs. non-shared memory systems
3. Performance: speedup, Amdahl's Law, measurement, performance losses, difficulties in scaling
4. Abstractions and algorithms for parallel program design: task vs. data parallelism, scalable parallelism, reduce and scan, work assignment
5. Practical programming: message passing, Pilot library, POSIX threads, OpenMP, Java threads
6. Overview of CUDA and OpenACC for GPU programming
7. Overview of Intel Xeon Phi coprocessor
8. Future directions in hardware and software

• Lecture format: YES
• Online materials location and format: CourseLink and instructor's website
• Lab or tutorial format and expectations: Weekly lab sessions are primarily used to introduce parallel programming tools and work through hands-on exercises.

• Number of Assignments: 4 (C and Java), or 3 (C) + project (your choice of language)
• Number of Graded Labs: none
• Number of Quizzes: 7 (each one open for a week in CourseLink)
• Formal Midterm: YES in 4-assignments format; NO with project
• Course project: The course has been offered with and without a project, depending on the number of students. The project requires a class presentation and report (see below).
• Final Exam: YES
• Group work: Assignments are strictly individual work. Groups of 2 are permitted for projects.
• Mostly programming assignments? YES
• Written documents? YES, each assignment includes a report that (a) shows graphs of the program's parallel performance, and (b) discussion of the program's design, its performance, and use of the language/tools. The project requires a similar report.