Shuai Tang

If only you could have fixed hallucinations of a pre-trained LLM through contextually relevant prompts. It would have been a solved problem by now! Contextual relevance (RAGs) improve domain specificity, not hallucination! Nice article. #llms #chatgpt4 #aiml #mlops #llmops #ml #rag https://lnkd.in/d44hdifr

11

How do language models comprehend gibberish inputs? Our recent work with James Zou focuses on understanding the mechanisms by which LLMs can be manipulated into responding with coherent target text to seemingly gibberish inputs. Paper: https://lnkd.in/gA9Mjqc4 A few takeaways: In this work we show the prevalence of nonsensical prompts that induce LLMs to generate specific and coherent responses, which we call LM Babel. We examine the structure of Babel prompts and find that despite their high perplexity, these prompts often contain nontrivial trigger tokens, maintain lower entropy compared to random token strings, and cluster together in the model representation space. We find that the efficiency of these prompts largely depends on the prompt length as well as target text’s length and perplexity. We show that reproducing harmful texts with aligned models is not only feasible but, in some cases, even easier compared to benign texts, while fine-tuning language models to forget specific information complicates directing them towards unlearned content.

31

🔥 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐚𝐧𝐝 𝐋𝐋𝐌 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 - 𝐓𝐨𝐩 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 🔥 🤖 I've conducted a mock interview with my student, Bheesetti Anand, focused on Generative AI and LLM (Large Language Models) interview questions. This session is designed to benefit other students preparing for similar interviews, helping them secure their dream roles in the field of Generative AI. 📔 Here’s why watching the video can help you make a 𝐬𝐮𝐜𝐜𝐞𝐬𝐬𝐟𝐮𝐥 ��𝐚𝐫𝐞𝐞𝐫 𝐬𝐰𝐢𝐭𝐜𝐡 𝐭𝐨 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈: ✅ 𝐂𝐨𝐦𝐩𝐫𝐞𝐡𝐞𝐧𝐬𝐢𝐯𝐞 𝐂𝐨𝐯𝐞𝐫𝐚𝐠𝐞: Detailed exploration of top interview questions related to Generative AI and Large Language Models. ✅ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬: Real-world examples and practical applications to understand complex concepts better. ✅ 𝐄𝐱𝐩𝐞𝐫𝐭 𝐆𝐮𝐢𝐝𝐚𝐧𝐜𝐞: Tips and strategies to approach and answer interview questions effectively. ✅ 𝐂𝐚𝐫𝐞𝐞𝐫 𝐏𝐫𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧: Essential knowledge and skills to prepare for interviews and stand out to potential employers. 📽 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐥𝐢𝐧𝐤: https://lnkd.in/g_nkP7fs #generativeai #llm #interview #jobpreparation #mockinterview

14

2 Comments

Here's the technical breakdown of the AI Penalty Challenge authored by Alok Pattani (check out this video if you haven't seen the demo yet: https://lnkd.in/gsD3xjJ2) Key Takeaways: ☁️ Firestore for triggering slow-mo video capture on Android phones & uploading to Google Cloud via Firebase Storage. 🧠 Cloud AutoML for training custom object detectors (ball & goal tracking) to calculate speed & accuracy. 🗣️ Gemini Pro 1.5 on Vertex AI for personalized coaching feedback based on multiple data points. 🎨 Imagen on Vertex AI for instant, prompt-driven image editing for cool player pics. More technical insights from the project to come... #AIChallenge #MachineLearning #SportsTech #GoogleCloud #footballNOTsoccer

90

6 Comments

More opportunities

13

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

15

I spent a couple months at the beginning of this year learning about GPU programming through trying to optimize inference for Cheng Chi’s awesome Diffusion Policy paper. I was able to improve inference time for the convolutional U-Net by ~3.4x over Pytorch eager mode and ~2.65x over Pytorch compile mode! For anyone interested in GPU optimizations for deep learning, I wrote a 9-part blog post that builds up from the physical structure of DRAM/SRAM cells all the way up to integrating custom CUDA kernels in Pytorch: https://lnkd.in/dBMSqh4g I also have a Twitter thread of the most interesting tid-bits here: https://lnkd.in/db_hEqbD This video (requires audio) is unrelated to the Diffusion inference stuff but imo, more amusing… I was able to get my Nvidia RTX 3090 inductor coils to play ‘Twinkle Twinkle Little Star’ using kernels (GPU programs) that modulate power draw at the right frequencies! What’s happening here is each kernel launch triggers a surge of in-rush current in the GPU’s inductor coils. The Lorentz force due to the change in current (proportional to change in current divided by the change in time) causes the coil to move slightly. If we play with the kernel launch frequencies we can vibrate the coils and get noises in the audible range. Unfortunately we can’t make sounds lower than 2000Hz because the ‘change in time’ part of the equation becomes too large, and the resulting vibration is too weak to make audible noise. So we end up with Twinkle Twinkle shifted up many octaves 😀

224

15 Comments

Is In-Context Learning (ICL) equivalent to Gradient Descent (GD)? There is a common belief that applying ICL in #LLM functions like GD-based fine-tuning. But does this hold in real-world LLMs? I'm super excited that our work got picked for an Oral presentation at #ICML this year! This would be impossible without an amazing collaboration with Aayush M. and Daniel Khashabi at The Johns Hopkins University. Find out in our paper: https://lnkd.in/eXb_sqmM.

81

4 Comments

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

Learn how an image-text multi-modality model can perform image classification, image retrieval, and image captioning #llms #llmops #image #opencv #chatgpt4 #gemini #aiml https://lnkd.in/gJFh9Ruf

1

Check out our efficient decoding Grouped-Query Attention (GQA) with low-precision KV cache for LLM inference! Read more on the PyTorch blog: https://hubs.la/Q02zSbSc0

210

Super exited to see "LongBet: Heterogeneous Treatment Effect Estimation in Panel Data" on arxiv. Thanks Meijia Wang and P. Richard Hahn! This paper introduces a novel approach for estimating heterogeneous treatment effects of binary treatment in panel data, particularly focusing on short panel data with large cross-sectional data and observed confoundings. In contrast to traditional literature in difference-in-differences method that often relies on the parallel trend assumption, our proposed model does not necessitate such an assumption. Instead, it leverages observed confoundings to impute potential outcomes and identify treatment effects. The method presented is a Bayesian semi-parametric approach based on the Bayesian causal forest model, which is extended here to suit panel data settings. The approach offers the advantage of the Bayesian approach to provides uncertainty quantification on the estimates. Simulation studies demonstrate its performance with and without the presence of parallel trend. Additionally, our proposed model enables the estimation of conditional average treatment effects, a capability that is rarely available in panel data settings.

104

4 Comments

Ensemble of smaller LLMs (PoLL) is less biased, faster, 7x cheaper when evaluating LLMs performance. Works great on QA & Arena-hard evals arxiv.org/abs/2404.18796

9

Large time-series models (LTMs) enjoy similar power-law scaling behaviour to LLMs. We just put out a paper (https://lnkd.in/givY528D) establishing power-law like scaling-laws for large time-series models as a function of data, compute, and model size. Similar scaling-laws for LLMs (from the landmark Kaplan et al. paper https://lnkd.in/g9KHYN9u) have provided key guidance in allocating enormous resources for predictable - and eventually breakthrough - performance gains. The demonstration of similarly favourable scaling behaviour for large time-series models provides both a motivation and guide in the pursuit of foundation models for time-series forecasting. Foundation models for time-series are coming (with enough data and compute). Thanks to Thomas Edwards, James Alvey, Benjamin Wandelt and Nam Nguyen for the hard work!

47

1 Comment

RULER : Benchmark to evaluate long-context modeling capabilities of language models In a recent paper from Nvidia researchers, authors propose 𝗥𝗨𝗟𝗘𝗥 as a new synthetic benchmark to evaluate long-context modeling capabilities for language model.  It contains 𝗳𝗼𝘂𝗿 𝘁𝗮𝘀𝗸 𝗰𝗮𝘁𝗲𝗴𝗼𝗿𝗶𝗲𝘀 to test behaviors beyond simple retrieval from context, which are as follows: 𝗶) 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 : extends the needle-in-a-haystack test to evaluate retrieval capability with diverse types and quantities of needles. 𝗶𝗶) 𝗠𝘂𝗹𝘁𝗶-𝗵𝗼𝗽 𝗧𝗿𝗮𝗰𝗶𝗻𝗴 :  proposes novel technique of variable tracking, a minimal proxy task for coreference chain resolution to check the behavior of tracing entities with multi-hop connections. 𝗶𝗶𝗶) 𝗔𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗶𝗼𝗻 : proposes common/frequent words extraction, proxy tasks for summarization to test the ability to aggregate relevant information that spans long-range context. 𝗶𝘃) 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻 𝗔𝗻𝘀𝘄𝗲𝗿𝗶𝗻𝗴 : add distracting information to the input of existing short context QA datasets to evaluate question answering capability at various context sizes. 🔒 𝗠𝗼𝘁𝗶𝘃𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗟𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 𝗼𝗳 𝗘𝘅𝗶𝘀𝘁𝗶𝗻𝗴 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝘀 - Simple retrieval-based test like Needle-in-a-Haystack is indicative of only a superficial form of long-context understanding. 🔬 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁 𝗦𝗲𝘁𝘂𝗽 𝗳𝗼𝗿 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 - 10 long-context LLMs were selected, including 9 open-source models and one closed-source model (GPT-4), covering diverse model sizes (6B to 8x7B with MoE architecture) - weighted average score was used to aggregate model performance across various context sizes. ⚖️ 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - all models exhibited large degradation in RULER as sequence length increases. - best performant model on RULER is GPT-4, which has the highest performance at length of 4k and demonstrates the least but non-marginal degradation (15.4) when extending the context to 128K. - top three ranked open-source models, Command-R, Yi-34B and Mixtral, all use a large base frequency in RoPE and are larger in parameter size than other models. 🕵️ 𝗠𝗼𝗱𝗲𝗹 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀 - context sizes overall lead to better performance, but the ranking can be inconsistent for long sequences - scaling model sizes has positive effects on better long-context modeling. - Non-transformer model architectures  demonstrated significant degradation when extending context size to 8K, and both underperform the Transformer baseline Llama2–7B by large margins up till the length of 4K. 🏆 For my detailed analysis of this paper, please refer to my blogpost on it at: https://lnkd.in/e5R9B7SC #largelanguagemodels #generativeai #modelevaluation

6

AI at Meta has released the most powerful open-source model yet as of today: 𝑳𝑳𝑨𝑴𝑨3 - 8B and 70B models at 8k context length! They highlight improvements in each of the following aspects as the key differentiator: a) Model Architecture b) Pretraining data c) scaling up pre-training d) Instruction fine-tuning 💥 𝐏𝐫𝐞𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 & 𝐒𝐜𝐚𝐥𝐢𝐧𝐠 𝐮𝐩: ✈️ Vocab of 128k tokens & GQA across both 8B and 70B, 8k native context length ✈️ Pretrained over 15T tokens, 4 times more code than LLAMA2 ✈️ Multi-lingual : over 5% of pretraining data consists of high-quality non English data that covers over 30 languages: don't expect same performance as English ✈️ LLAMA2 generates training data (like in Self-Instruct) to generate text classifiers that filter data powering LLAMA3 ✈️ Both 8B and 70B improve even after the model is trained on 2 orders of magnitude more data i.e log-linear improvement after 15T tokens training ✈️ Training runs on 2 custom-built 24k GPU clusters with an overall training time efficiency of 95% ✈️ 𝘖𝘷𝘦𝘳𝘢𝘭𝘭, 3 𝘵𝘪𝘮𝘦𝘴 𝘮𝘰𝘳𝘦 𝘦𝘧𝘧𝘪𝘤𝘪𝘦𝘯𝘵 𝘵𝘩𝘢𝘯 ��𝘓𝘈𝘔𝘈2 💥 𝑰𝒏𝒔𝒕𝒓𝒖𝒄𝒕𝒊𝒐𝒏 𝑭𝒊𝒏𝒆𝒕𝒖𝒏𝒊𝒏𝒈: ✈️ Combination of SFT + PPO + DPO + Rejection Sampling ✈️ 𝑸𝒖𝒂𝒍𝒊𝒕𝒚 𝒐𝒇 𝒑𝒓𝒐𝒎𝒑𝒕𝒔 𝒖𝒔𝒆𝒅 𝒊𝒏 𝑺𝑭𝑻, 𝒑𝒓𝒆𝒇𝒆𝒓𝒆𝒏𝒄𝒆 𝒓𝒂𝒏𝒌𝒊𝒏𝒈𝒔 𝒖𝒔𝒆𝒅 𝒊𝒏 𝑫𝑷𝑶, 𝑷𝑷𝑶 𝒉𝒂𝒔 𝒂𝒏 𝒐𝒖𝒕𝒔𝒊𝒛𝒆𝒅 𝒊𝒎𝒑𝒂𝒄𝒕 𝒐𝒏 𝒎𝒐𝒅𝒆𝒍 𝒑𝒆𝒓𝒇𝒐𝒓𝒎𝒂𝒏𝒄𝒆 ✈️ Careful data curation and multiple rounds of QA on annotations from human annotators ✈️Training on preference ranking improved the model performance greatly on coding and reasoning tasks 💥 𝑳𝑳𝑨𝑴𝑨3 𝑮𝒂𝒖𝒓𝒅𝒓𝒂𝒊𝒍𝒔: ✈️ Updated and new safety tools : LLAMA Gaurd2 & cybersecEval 2 ✈️ CodeShield - inference time gaurdrail for filtering insecure code ✈️ Instruction Fine-tuned model redteamed for safety : By generating adverserial prompts that try to elicit problematic responses. ✈️ LLAMAGaurd - foundational for prompt and response safety, can be easily finetuned to create a new taxonomy 💥 𝑰𝒏𝒇𝒆𝒓𝒆𝒏𝒄𝒆 & 𝑭𝒖𝒕𝒖𝒓𝒆 ✈️ Despite LLAMA3 having 1B more params than LLAMA2 7B, the improved tokenizer efficiency and GQA contribute to maintaining the inference efficiency on par with Llama 2 7B. ✈️ 400B+ parameters, Multilingual, Multimodal, and longer context window LLAMA3 would be available on AWS soon. Try it out! #generativeAI #llm #llama3 #aws #bedrock #sota

33

1 Comment

Meta presents Layer Skip Enabling Early Exit Inference and Self-Speculative Decoding https://lnkd.in/eBFpbV4r We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task.

41

1 Comment