Duration: 4 days
Format: Lecture 30%, Lab Work 70%
This course teaches the tools, techniques, and strategies that can be used to debug and analyze system crashes, hangs, and performance issues on the SmartOS Operating System.The course is heavily concentrated on lab work. Students will be given various problems, and asked to use the techniques covered in lecture to solve the problems. Tools include DTrace, mdb, and kmdb. The class will examine problems related to logic errors, memory corruption, synchronization problems, stack corruption, memory leaks, boot problems, and others. The course will also cover architecture specific details needed to do debugging on X86/X64.
- Use DTrace, mdb, and kmdb to examine system and application bugs, including crashes and hangs.
- Determine which tools to use for specific types of problems.
- Describe what happens on a system panic.
- Describe how threads can hang, and how to determine the root cause of this.
- Describe stack frames and stack operation on 32-bit and 64-bit X86.
- Use mdb to examine crashed (and working) systems.
- Use kmdb to determine causes of system hangs.
- Use DTrace to see the flow of control leading up to a problem, as well as for general debugging.
- Use DTrace to examine performance issues, and to drill-down to find the root cause.
This course is designed for system engineers/analyists and kernel programmers. Students may include developers and support engineers. While the course concentrates on kernel issues, application developers can also benefit.
A SmartOS/Illumos/Solaris Internals course should be taken prior to this course. To do the labwork, students will need to know the C programming language.
Module 1: Introduction to Panics and Hangs
- What is a Panic?
- Causes of Panics
- What happens on Panic?
- Panic Dump Actions
- Panic Dump Contents
- The dumpadm Command
- The savecore Command
Module 2: Preliminary Analysis Information Gathering
Module 3: Analysis and Debugging Tools
- Which tool to use
Module 4: mdb
- Running the mdb command
- Examples of the “/”, “?”, and “=” Commands dcmds and walkers
- Lots of examples
- Kernel Panic Example
Module 5: DTrace
- How DTrace works
- Debugging Example
- The D Language Actions
- Built-in Variables
- Anonymous Tracing, Speculative Tracing, and Destructive Actions
- Examining post-mortem DTrace records using mdb
- Tunable variables and troubleshooting DTrace problems
Module 6: kmdb
- Differences between kmdb and mdb
- Examining system hangs with kmdb
- Booting with kmdb
Module 7: zdb
- Examining on-disk ZFS format with zdb
Module 8: X86 Architecture Overview
- Function call mechanism
- Stack Frames
- Description of Assembler Language Problems that can occur with stacks
Module 9: Implementing dcmds and walkers
- Adding functionality to mdb
- Lab Work
Cancellation and refunds policy
Have questions about Kernel Crash Analysis and Debugging?