Lesson 3 of 4Module 1 · 0% of course
‹ PrevCourse ›
Explore › Math › Adjacent: Math for Thinking › Adjacent
Adjacent: Math for Thinking · Adjacent

Information Theory

Claude Shannon's 1948 theory: information as surprise, bits, entropy, compression limits, channel capacity and error-correcting codes, and why cross-entropy is the loss function behind today's language models.

30 min · 6 steps · videoJump to quiz ↓
The Biggest Ideas in the Universe | 20. Entropy and Information Sean Carroll · 99 min
Open in YouTube
Step 1 of 6 · Overview30 min read ·
Key idea
Prereqs: Probability; logarithms. Next: Optimization for ML; LLM basics. Sources: Claude Shannon, 'A Mathematical Theory of Communication' (1948); James Gleick, *The Information* (2011); Cover & Thomas, *Elements of Information Theory*.
Next: 1. Information as surprise

Watch next

The Great Courses · 31 min · works as audio
Ben Syversen · 29 min · works as audio
Oxford Mathematics · 52 min · works as audio
Ben Syversen · 41 min · works as audio
Harvard University · 50 min · works as audio
Harvard University · 46 min · works as audio
Harvard University · 46 min · works as audio
MIT OpenCourseWare · 30 min · works as audio
MIT OpenCourseWare · 37 min · works as audio
Harvard University · 50 min · works as audio
Adams State University · 43 min · works as audio
‹ PrevCourse ›