Latest videos about Reinforcement Learning
Jev Course – Prefill-Only Inference & Decision Models from Scratch
- **Jev Decision-Making Models**: Jev is a decision-making model built on PyTorch and Hugging Face frameworks that bypasses token-by-token text generation. It uses a single direct pre-fill pass to provide fast, low-latency typed solutions like choices, ordinal evaluations, and boolean values. - **Inference Paradigms**: The video explores two main inference paradigms for Jev: encoder-only and decoder-only models. Encoder models (like BERT) are bidirectional and require re-calculating attention weights for each new question, making them less scalable for multiple questions on the same context. Decoder models (like Qwen) use a causal mask, allowing the state to be pre-cached and reused for multiple questions, significantly improving efficiency. - **RLCD Training Workflows**: The training process for Jev models often involves RLCD (Reinforcement Learning for Calibrated Solutions), which uses a teacher-student distillation approach. It leverages concepts like GRPO (Group Relative Policy Optimization) to generate trajectories, calculate advantages, and build a loss function to align the student model's predictions with the teacher's calibrated probabilities. - **Jev Interfaces**: Jev provides three main interfaces for decision-making: `choice` (nominal options), `evaluation` (ordinal options), and `null` (binary yes/no questions). The training and implementation strategies need to respect the nuances of these different data types. - **Key Takeaways for Implementation**: For practical implementation, understanding attention concepts, KV cache, logits, and softmax is crucial. The choice between encoder and decoder depends on factors like the need to cache state, context length, and the number of questions. Decoder-only models are generally preferred for scenarios with many questions on a fixed context due to their caching capabilities.
Code a Reinforcement Learning Library in C from Scratch (Full Course)
- The video provides a deep dive into building a custom reinforcement learning (RL) library from scratch in C, without relying on external machine learning frameworks. - Key components developed include an autograd engine for automatic differentiation and an implementation of the policy gradient REINFORCE algorithm. - To test the custom library, a complete Snake game environment is coded, allowing an AI agent to be trained to navigate the grid. - The tutorial covers the fundamental concepts of RL, including agent-environment interaction, reward feedback, and the importance of an autograd engine for neural network training. - The speaker acknowledges the complexity and potential for instability with the REINFORCE algorithm, noting that it can still learn effectively with sufficient training time, as demonstrated by the eventual positive rewards in the Snake game.