Part 2: Scaling Workflows Efficiently

Workshop Details

As you scale up your research, data and computational workflows can become complex, but there are strategies to help you simplify repetitive tasks.  Learn how to handle large data transfers efficiently, automate data transfer tasks, submit multiple similar jobs to Quest efficiently, and use advanced features of Quest’s Slurm job scheduler.   

Prerequisites: This workshop is designed for those who have attended Part 1 of Research Computing Basics and Beyond, or who already know how to transfer data to and from Quest and submit simple batch jobs.   

Attendees will learn how to: 

• Automate data transfers using Globus, including to and from the Genomics Compute Cluster 

• Manage moving active and archival data across storage platforms 

• Use job arrays to submit similar jobs with different input parameters or data sets, for example running the same script on many samples 

• Manage dependencies between multiple jobs so that the next step of your workflow starts as soon as the previous is finished 

• Convert a pipeline of multiple scripts into a single script 

• Monitor computational job efficiency and use that information to reduce wait times  

Event Details

All Research Computing Basics and Beyond workshops are held in person; there is no hybrid option or recording available. These are interactive, hands-on workshops; participants are required to bring their laptop and charger.

Lunch breaks will be provided; participants are responsible for their own lunch.

Attendees

Members of the Northwestern community with an active Northwestern NetID as of the date of the workshop are eligible to attend. Registrations made by those without an active NetID may be cancelled. If you will be at Northwestern this summer, but do not yet have a NetID, please register but email nuit_rcds@northwestern.edu to let us know of the situation.

Refunds and Cancellations

Research Computing Basics and Beyond refunds (minus Eventbrite and credit card fees) will be given to cancellations seven (7) days before the event.

Northwestern registrants needing to cancel with fewer than seven (7) days notice can contact nuit_rcds@northwestern.edu for possible refunds.