Cosmin Negruseri’s Post

Curious how the LLM/AGI race will turn out, one winner or multiple winners with very similar model quality. A year and a half after the chatGPT drop the LLM race looks very similar to a Kaggle competition. In Kaggle the top teams usually are neck and neck in terms of quality. Even the ideas in the top solutions have a large overlap. It's surprising how many parallels there are between the LLM race and Kaggle: - teams with more compute can iterate on more ideas - teams experiment fast with lots of things, there's no one thing that makes or breaks the final solution, it's a set of ideas that works well together - there's a dynamic where one sees that some team made a break through and soon other teams just by knowing there's a better solution find it as well - smart ideas "leak" (or rather are shared by people on the forum :) which reminds me of meta releasing llama or of people transferring between frontier labs) - the benchmark/dataset saturates by the end of the competition

Looks like at Google AI writes 50% of the new code. Extrapolating the trend says we won't need coders somewhat soon :D. https://lnkd.in/g5XWRipc

Hao Liu (ring attention author) gives a talk at Berkeley last month covering llms, skimmed the slides, they look very good if one needs to catch up with the current state of LLMs. Some things that are discussed: - scaling laws - long context - ring attention - flash attention - mixture of experts - mixtral model - inference bottlenecks - kvcache - chain of thought

Skimmed the slides, they look great! Good way to catch up on what are the important pieces of training large models. All slides are insightful. I loved the slides on data preparation the most. Other things discussed: - rlhf: ppo vs dpo - flash attention - mixture of experts - parallel training - muP for hpt transfer - speculative decoding Thomas Wolf you also need to cover 1M context size and multimodal models it's 2024 :D

Dataset release from romanian students (feb 20). They share problems from infoarena.ro the most popular coding contest platform in romania, a website that's very dear to a lot of us who got started in the romanian programming contest community. https://lnkd.in/gZ9RgpYP

Here's a good term glossary if you wanna catch up with the cool kids who are prompt engineering, finetuning, merging models, doing rag etc via Omar Sanseviero https://lnkd.in/gtumCyy2

Google Deepmind is getting close to IMO gold medalists on geometry problems. Exciting times. https://lnkd.in/grDEyYtX

ML checklist from Aurélien Géron

It's a year since ChatGPT rolled out. Established companies and startups jumped in, eager to leverage LLMs for a myriad of use cases from code assistants to customer service. GenAI is democratizing AI abilities for beginners and advanced practitioners. One thing that you see again and again is RAG (Retrieval Augmented Generation). demos The idea is simple: mix a database with LLMs, and you've got a system that can update, remember and think. It's a neat trick and makes for impressive demos. But when it's time to scale? That's a whole different ball game. A related point is made on a twitter thread by Omar K. and Richard Socher on twitter: chatGPT with GPT4 is hard to match or beat because it is not just a LLM but it is a full fledge system with many moving pieces. The RAG problem and stack is very similar to the classical search and retrieval stacks or recommender system stacks. At Pinterest, we built recommendation systems that were like finely tuned engines, running on a mix of data types, with tens of indexes, multiple retrival algorithms, and multitask models. All that, plus a bunch of filters to keep things safe. The diagram here shows the basics of a RAG app versus the complex machinery of a full-blown rec system. It's clear there's a lot under the hood. Applications like ChatGPT, Bard, Perplexity, You.com arguably have similar complexity. I'm curious, what examples of RAG systems do you know of that work well in production?