Aligning LLMs with Direct Preference Optimization

DeepLearning.AI was founded in 2017 by machine learning and education pioneer Andrew Ng to fill a need for world-class AI education.

DeepLearning.AI

We will discuss a powerful alignment technique called Direct Preference Optimisation (DPO) which was used to train Zephyr.

<a href="https://www.youtube.com/watch?v=QXVCqtAZAn4" title="https://www.youtube.com/watch?v=QXVCqtAZAn4" target="_blank" data-msys-clicktrack="0" rel="nofollow noopener noreferrer">Link to watch the event here! </a> Just one year ago, chatbots were out of fashion and most people hadn't heard about large language models and techniques to align them with human preferences. Then, OpenAI broke the internet with ChatGPT and Meta followed suit by releasing the Llama series of language models which enabled the open-source community to build their very own capable chatbots.This has led to a rich ecosystem of datasets and models that have mostly focused on teaching language models to follow instructions through supervised fine-tuning (SFT). However, we know from the work of OpenAI and Meta that significant gains in helpfulness and safety can be had by augmenting SFT with human (or AI) preferences. In this workshop, Lewis Tunstall and Edward Beeching from Hugging Face will discuss a powerful alignment technique called Direct Preference Optimisation (DPO) which was used to train Zephyr (https://arxiv.org/abs/2310.16944) and is rapidly becoming the de facto method to boost the performance of open chat models. By the end of this workshop, attendees will:<ul><li>Understand the steps involved in fine-tuning LLMs for chat applications.</li><li>Learn the theory behind Direct Preference Optimisation and how to apply it in practice with the Hugging Face TRL library.</li><li>Know what metrics to consider when evaluating chat models.</li></ul> About DeepLearning.AI:DeepLearning.AI is an education technology company that is empowering the global workforce to build an AI-powered future through world-class education, hands-on training, and a collaborative community. Take your generative AI skills to the next level with short courses help you learn new skills, tools, and concepts efficiently. About Hugging Face:Hugging Face is an AI company specializing in natural language processing (NLP) and machine learning, and is known for its open-source contributions and collaborative approach to AI research and development. The company is famous for developing the Transformers library, which offers a wide range of pretrained models and tools for a variety of NLP tasks, making it easier for researchers and developers to implement state-of-the-art AI solutions. Hugging Face also fosters a vibrant community for AI enthusiasts and professionals, providing a platform for sharing models, datasets, and research, which significantly contributes to the advancement of AI technology. Speakers:Lewis Tunstall, Machine Learning EngineerLewis Tunstall is a Machine Learning Engineer at Hugging Face whose work lies at the intersection of the open-source and research teams, He is the co-author of the bestseller “NLP with Transformers” book and has previously built machine learning-powered applications for start-ups and enterprises in the domains of natural language processing, topological data analysis, and time series. He holds a PhD in Theoretical Physics, was a 2010 Fulbright Scholar and has held research positions in Australia, the USA, and Switzerland. His current work focuses on building tools and recipes to align language models with human and AI preferences through techniques like reinforcement learning.<a href="https://www.linkedin.com/in/lewis-tunstall/" title="https://www.linkedin.com/in/lewis-tunstall/" target="_blank" data-msys-clicktrack="0" rel="nofollow noopener noreferrer">https://www.linkedin.com/in/lewis-tunstall/</a> Edward Beeching, Research ScientistEd Beeching has been a Research Scientist at Hugging Face for the last two years, focusing on building open tooling for LLM fine tuning, implementing state of the art LLM alignment algorithms and building preference datasets. Ed is the creator of the Open LLM leaderboard, co-author of Zephyr and a maintainer of numerous open-source ML libraries.His PhD was focused on Deep Reinforcement Learning approaches for planning and navigation in Robotics. His current research is predominantly Embodied Learning and LLM alignment.<a href="https://www.linkedin.com/in/ed-beeching-3553b468/" title="https://www.linkedin.com/in/ed-beeching-3553b468/" target="_blank" data-msys-clicktrack="0" rel="nofollow noopener noreferrer">https://www.linkedin.com/in/ed-beeching-3553b468/</a>

Aligning LLMs with Direct Preference Optimization

More events from DeepLearning.AI

Discover more events from DeepLearning.AI, from Science & Tech to other experiences you might love.

Still looking for the right event?

Explore all online events to browse and filter by date, category, and more.

Aligning LLMs with Direct Preference Optimization

More events from DeepLearning.AI

Discover more events from DeepLearning.AI, from Science & Tech to other experiences you might love.

You might also like...

Browse more events with different dates, prices, and formats to find your next great experience.

Still looking for the right event?

Explore all online events to browse and filter by date, category, and more.