Managing large batches of jobs [online, CIP202]

Need to run hundreds of variations of your analysis? Do you need to repeat the same computation with different datasets? This workshop shows you how to structure and submit numerous jobs without overwhelming the scheduler. You'll learn grouping and parallelization techniques to launch your analyses.

Prerequisites

• Know the basics of the Linux command line or have completed the workshop “Introduction to the Linux Command Line (LNX101)”
• Know the basics of submitting jobs to a compute cluster or have completed the workshop “First steps on supercomputers (CIP101)”

Agenda

• Introduction to data parallelism
• Job arrays
• Parameter sweeps with GNU Parallel
• Other tools - META-Farm and GLOST

Registration

• Academic: $10 (anyone who studies, teaches, or works at a university, CEGEP, CCTT, or university-affiliated research institute)
• NPO: $10 (anyone who works for a non-profit organization)
• Other: $250 (any other profile)

Instructor

Olivier Fisette, analyst in advanced research computing at Calcul Québec.

Language

English

Technical prerequisites

We will use the Zoom platform. Because this event is a practical workshop, it is very useful having a secondary screen where you would get the instructor window on one screen and your own window on your main screen.

We will use the Jupyter Lab interface. Make sure you have a modern Web browser like Google Chrome, Firefox, Edge or Safari.

Notes

• A certificate of participation will be send to each participant who attends at least 60% of the workshop.
• The workshop is not recorded.
• The workshop could be canceled if the number of registrations is too low.

Contact

For any question, please write us to training@calculquebec.ca

To stay up to date on our upcoming events, subscribe to our Eventbrite page!