Answers
“We’re releasing Spinning Up in Deep RL, an educational resource designed to let anyone learn to become a skilled practitioner in deep reinforcement learning. Spinning Up consists of crystal-clear examples of RL code, educational exercises, documentation, and tutorials.”
"The first non-book, non-video, concise entry point I refer people to is OpenAI’s Spinning Up to Deep RL: [https://spinningup.openai.com/en/latest/](https://spinningup.openai.com/en/latest/)"
“Spinning Up in Deep RL consists of the following core components: A short introduction to RL terminology, kinds of algorithms, and basic theory. An essay about how to grow into an RL research role. A curated list of important key papers organized by topic. A well-documented code repo of short, standalone implementations of: Vanilla Policy Gradient (VPG), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor-Critic (SAC). And a few exercises to serve as warm-ups.”
"There's a Deep Leaning and RL series from DeepMind on youtube. If you just watch the RL videos, it's really cohesive. They follow Sutton and Barto, which is the classic text, but with some additional content."
"Quick one for anyone else interested, DeepMind updated the Reinforcement Learning lecture series in 2018 - see [here](https://youtube.com/playlist?list=PLqYmG7hTraZBKeNJ-JE_eyJHZ7XgBoAyb)."
"I come up with some courses:"
"For coding assignments check this repo from udacity : [https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893)"
"Since I know about ML/DL, I also know about Prob/Stats/Optimization, but only as a CS student."
"Both are good but if you need the lecture material then I recommend the deep rl bootcamp lectures."
"Sutton and Barto's [Reinforcement Learning: An Introduction book](https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf)."
"I think his book is one the most valuable books written in ML,..ever."
"If Sutton himself is using clickbait titles on reddit you can be sure it's the optimal strategy"
All answers
Considerations
Course Source
Prerequisite Knowledge
Course Content
Lecture Material
Math Prerequisites
Sources
About this article
Composed Feb 24, 2023 by
J. Q.
from discussion by
156 contributors around the web
735
words
10.2k
words read
1m 41s
research time
1h 14m
effective research time
Version history
If this article is generated multiple times, old versions will appear here.
J. Q., 24d ago 3
Research
Notes
-
Open.ai’s Spinning Up
- Good for implementing classic papers and reading SOTA literature
- Prerequisite ML knowledge
-
DeepMind’s Deep Learning and RL series
- Cohesive and follows Sutton and Barto
-
Stanford CS234
- Also based on Sutton and Barto and covers more content than David Silver course
"There's a Deep Leaning and RL series from DeepMind on youtube. If you just watch the RL videos, it's really cohesive. They follow Sutton and Barto, which is the classic text, but with some additional content."
"Open.ai’s will get you implementing classic papers and capable of reading SOTA literature (relatively) fast."
"Stanford CS234 on youtube, its also based on Sutton and Barto and covers more things than David Silver course. I liked both"
-
David Silver’s course on YouTube
- It is free
- Released just around the same time as Sutton & Barto’s book
- Lectures only
-
Sutton & Barto’s book
- One of the most valuable books written in ML
- Coauthoring a book on ML is difficult to compare to Sutton & Barto
-
Coursera course
- 7 day free trial, then charged $72/month
- People need to be aware of the charges after the 7 day trial
-
Vishal Garg’s course
- Free content on RL
- Hands on content to relate theory to practice
"Well, I also started publishing some content on RL, have a look, might be of use."
"Reinforcement Learning — What, Why, and How.” by Vishal Garg https://link.medium.com/G8N5TgBYk7"
"He also mentions that the Coursera course 'begins today' as though it is special somehow."
-
Reinforcement Learning Specialization from Coursera and the University of Alberta
- Instructed by Martha and Adam White
- Based on the second edition of Andy Barto’s and Richard Sutton’s textbook Reinforcement Learning: An Introduction
- Can earn credit for the course or audit it for free
-
David Silver’s course on RL
- Lectures are great and have been the standard for a long time
- Very different from the Coursera course
-
Udacity course
- Earlier exercises are very limited
- Later exercises are more open-ended
-
Richard Sutton’s book
- One of the most interesting reads on a topic ever
- Best way to get started with reinforcement learning
-
Math prerequisites
- Basic understanding of probability
- Some linear algebra and calculus helps to understand some of the more complex topics
"I thought it was a newb asking questions, and I was about to provide the same answer as the OP. The only difference is I was going to recommend David Silver's course on RL, not the Coursera listing."
"But Dave's lectures are also great, and have been the standard for a long time."
"Perhaps Martha or Adam can comment on the relationship to David Silver's course. They had the luxury of knowing about Dave's when they made their's."
-
David Silver’s RL Course
- Recommended by a Reddit user
- YouTube video series
-
Sutton and Barto’s Reinforcement Learning: An Introduction book
- Recommended by a Reddit user
- Solutions to problems available on GitHub
-
Reinforcement Learning book by Phil Winder
- Recommended by a Reddit user
- Published by O’Reilly
-
OpenAI’s Spinning Up to Deep RL
- Recommended by a Reddit user
-
Coursera course by the University of Alberta
- Recommended by a Reddit user
-
NPTEL IIT Reinforcement Learning (Barto’s student)
- Recommended by a Reddit user
-
CS 294 Berkeley (Deep RL course from Sergey)
- Recommended by a Reddit user
-
CS 285 Berkeley (Deep RL course from Sergey)
- Recommended by a Reddit user
-
Hugging Face Deep Reinforcement Learning Course
- Recommended by a Reddit user
- Includes unique environments, such as SnowballFight, Huggy the Doggo 🐶, MineRL (Minecraft ⛏️), VizDoom (Doom) and classical ones such as Space Invaders and PyBullet
"It's CS 285 nowadays ([https://rail.eecs.berkeley.edu/deeprlcourse/](https://rail.eecs.berkeley.edu/deeprlcourse/))."
"CS 294 Berkeley (Deep RL course from Sergey)"
"The Coursera one was pretty good imo"
-
David Silver’s course
- It is the best to follow according to one Reddit user
- Course link is provided: https://www.davidsilver.uk/teaching/
"Personally I think David Silver's course is the best to follow."
"I taught myself RL from his course here: https://www.davidsilver.uk/teaching/"
-
CS234: Stanford Reinforcement Learning Winter 2021
- Offered by Stanford
- Recommended by a graduate student at Georgia Tech
-
DeepMind (Hado Van Hasselt): Reinforcement Learning 1: Introduction to Reinforcement Learning
- Offered by DeepMind
- Recommended by a graduate student at Georgia Tech
-
Another DeepMind (David Silver): RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning
- Offered by DeepMind
- Recommended by a graduate student at Georgia Tech
-
UofA Coursera: Practical Reinforcement Learning
- Offered by Coursera
- Recommended by a Reddit user
-
CS285: Berkeley Deep Reinforcement Learning Course
- Offered by Berkeley
- Recommended by a Reddit user
-
SpinningUp: Introduction to the fundamentals of Deep RL
- Offered by OpenAI
- Recommended by a Reddit user
-
Dimitri Bertsekas 2021 video lectures
- Offered by Dimitri Bertsekas
- Recommended by a Reddit user
"I am a graduate student at Georgia tech, currently taking RL, I can certainly say that the first book u mentioned ( Sutton and Barto) is like the Bible for RL, and Deepmind’s course by David silver is the best course out there and it follows the book closely, so I recommend doing both together."
"I am a graduate student at Georgia tech, currently taking RL, I can certainly say that the first book u mentioned ( Sutton and Barto) is like the Bible for RL, and Deepmind’s course by David silver is the best course out there and it follows the book closely, so I recommend doing both together."
"For coding assignments check this repo from udacity : [https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893)"
-
CS234: CS234: Reinforcement Learning Winter 2021 (stanford.edu)
- Course offered by Stanford University
- Course content includes fundamentals of reinforcement learning, exploration/exploitation trade-offs, dynamic programming, Monte Carlo methods, temporal-difference learning, and deep reinforcement learning
-
DeepMind (Hado Van Hasselt): Reinforcement Learning 1: Introduction to Reinforcement Learning - YouTube
- Course offered by DeepMind
- Course content includes introduction to reinforcement learning and related concepts
-
Another DeepMind (David Silver): RL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning - YouTube
- Course offered by DeepMind
- Course content includes introduction to reinforcement learning and related concepts
-
UofA Coursera: https://www.coursera.org/specializations/reinforcement-learning
- Course offered by University of Alberta
- Course content includes fundamentals of reinforcement learning and deep reinforcement learning, exploration/exploitation trade-offs, dynamic programming, Monte Carlo methods, temporal-difference learning, and deep reinforcement learning
-
CS285: http://rail.eecs.berkeley.edu/deeprlcourse/
- Course offered by UC Berkeley
- Course content includes fundamentals of reinforcement learning and deep reinforcement learning, exploration/exploitation trade-offs, dynamic programming, Monte Carlo methods, temporal-difference learning, and deep reinforcement learning
-
HSE Coursera: Practical Reinforcement Learning | Coursera
- Course offered by Higher School of Economics
- Course content includes fundamentals of reinforcement learning and deep reinforcement learning, exploration/exploitation trade-offs, dynamic programming, Monte Carlo methods, temporal-difference learning, and deep reinforcement learning
-
Reinforcement Learning, second edition: An Introduction (Adaptive Computation and Machine Learning series): Sutton, Richard S., Barto, Andrew G.: 9780262039246: Amazon.com: Books
- Book offered by Sutton and Barto
- Book content includes introduction to reinforcement learning, Markov Decision Processes, exploration/exploitation trade-offs, dynamic programming, Monte Carlo methods, temporal-difference learning, and deep reinforcement learning
- Reinforcement Learning: Industrial Applications of Intelligent Agents: D., Phil Winder
"I come up with some courses:"
"I want to start learning RL. I have good knowledge about ML/DL, but RL is completely new to me. I want to build a RL model for an application"
"Since I know about ML/DL, I also know about Prob/Stats/Optimization, but only as a CS student."
-
Spinning Up in Deep RL (OpenAI)
- Short introduction to RL terminology, kinds of algorithms, and basic theory
- An essay about how to grow into an RL research role
- A curated list of important key papers organized by topic
- A well-documented code repo of short, standalone implementations of: Vanilla Policy Gradient (VPG), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor-Critic (SAC)
- And a few exercises to serve as warm-ups
-
PyBullet Gym
- Similar to Roboschool, but not abandoned
-
OpenAI Atari environments
- Free enviroment
-
Roboschool or PyBullet
- Free environments that are pretty much the same as mujoco
“You could use the OpenAI Atari environments, they are free!”
“Try roboschool or pybullet as well for free environments that are pretty much the same as mujoco”
“We’re releasing Spinning Up in Deep RL, an educational resource designed to let anyone learn to become a skilled practitioner in deep reinforcement learning. Spinning Up consists of crystal-clear examples of RL code, educational exercises, documentation, and tutorials.”
-
David Silver’s introductory lecture series
- Based on the Sutton and Barto book
- Available on YouTube
-
OpenAI gym
- Code up the algorithms you learn about
-
Berkeley’s RL course
- Available on YouTube
-
Coursera RL specialization
- Offered by people in Sutton lab
- Sutton and Barto’s book
- OpenAI’s SpinningUp
- Implements algorithms in Atari games, Q-learning, and actor-critic
- Use whatever programming language is most natural
"There should be a coursera RL specialization offered by people in Sutton lab. It closely follows the book. I took this a year ago and found it to be really well taught."
"[David Silver presents an introductory lecture series](https://www.youtube.com/watch?v=2pWv7GOvuf0&list=PLqYmG7hTraZBiG_XpjnPrSNw-1XQaM_gB&index=1) - based on the Sutton and Barto book"
"Just go build it! Start with Atari games, q-learning, and actor-critic. Implement the algorithms yourself and then move on to problems you have a heard of people trying before. Berkeley’s RL course is on YouTube, I believe."
-
Stanford CS234
- Lecture material is recommended
- Practitioners need to write as much code as possible to understand key concepts and the little things that make deep RL work in practice
-
Berkeley Deep RL course
- Lecture material is recommended
- Practitioners need to write as much code as possible to understand key concepts and the little things that make deep RL work in practice
-
Deep RL Bootcamp lectures
- Practitioners need to write as much code as possible to understand key concepts and the little things that make deep RL work in practice
-
Spinning up in Deep RL
- Recommended for coding first approach
-
David Silver’s Reinforcement Learning course
- Recommended as the first step
-
Sutton and Barto’s book
- Recommended as the first step
-
Youtube lecture series
- Famous among upperclassmen, but found to be difficult due to excessive math
"But most practitioners really need to write as much code as possible to understand key concepts and the little things that make deep RL work in practice, and for that I’d recommend spinning up in deep rl."
"Yup but spinning up has code exercises that are more up to date, easier to access, etc. I'd recommend only using deep RL bootcamp as a lecture resource."
"The best possible first step is to see David Silver’s lectures and read wherever you need the book of Sutton and Barto"