> For the complete documentation index, see [llms.txt](https://bbb-fullstack-aier.gitbook.io/all-about-llm/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://bbb-fullstack-aier.gitbook.io/all-about-llm/tototolearn/lossless-compression/compressing-for-agi.md).

# Compressing for AGI

Jack Rae

相关解读：<https://zhuanlan.zhihu.com/p/619511222>

原视频：<https://www.youtube.com/watch?v=dO4TPJkeaaU&t=161s>

## Theme of talk

* Think deeply about the <mark style="background-color:red;">training objective of foundation models</mark>
* <mark style="background-color:red;">What are we doing , why dose it make sense, and what are the limitations?</mark>

## Takeaways

* seek the minimum description length to solve perception
* Generative models are lossless compressors
* <mark style="background-color:green;">Large language models are state-of-the-art lossless text compressors(?!)</mark>
* Current limitation of thr apporach

## Minimum Description Length&#x20;

### ...and why it relates to perception

We want to deepest understanding of our observations

ones that generalize

<figure><img src="https://1543406670-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FdojwBqJX2Sm7CWXVxVhG%2Fuploads%2FzytC0j7IjK10F9ISfsR0%2Fimg_v2_51408b25-2ef4-4a83-b03b-77137b87b63g.jpg?alt=media&amp;token=b73b789d-ab07-4f18-94e6-44871c7756b4" alt="" width="563"><figcaption></figcaption></figure>