Haifeng Geng

Recently, I was re-introduced to the concept of catastrophic forgetting while reading about NVIDIA's ViLa model, which BTW offers pretty cool capabilities like reasoning across multiple images and learning in context and is capable of understanding videos, offering a more comprehensive understanding of visual content and its interplay with language. Overcoming catastrophic forgetting is crucial because it allows AI models to retain valuable knowledge while continuously learning and adapting, enhancing their overall performance and flexibility. Read article here: https://lnkd.in/e8gimTk4

43

1 Comment

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

29

Using generative AI, NVIDIA operations built an AI Planner agent, developed on NVIDIA Inference Microservices (NIM). The agent leverages LLM, NeMo Retriever and CuOpt NIM to reduce re-planning time from hours to just seconds. https://nvda.ws/4bIfS2T

2

👉🏼 Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models 🤓 Honghao Lai https://lnkd.in/eBZ2cduS 🔍 Focus on data insights: - Large language models (LLMs) facilitate systematic reviews. - Methods and reliability of LLMs for assessing risk of bias (ROB) in randomized clinical trials (RCTs) are explored. 💡 Main outcomes and implications: - LLMs demonstrated high correct assessment rates for ROB in RCTs. - LLMs showed substantial accuracy and consistency in evaluating RCTs. - Potential of LLMs as supportive tools in systematic review processes. 📚 Field significance: - Enhancing efficiency and accuracy in assessing ROB in RCTs. - Advancing the use of LLMs in systematic review methodologies. 🗄️: #datainsights #riskofbias #RCTs #systematicreviews

1

As I’ve been reflecting on the novelty of LLMs and agents is actually not that novel in operationalizing it. These systems share a lot in common with distributed systems and micro service architecture, and event driven systems. It seems to me that the recommendation is to have small and well tested components. Also, to restrict the chance of the wrong kind of hallucinations by restricting the data and by comparing it to things we know. What they do sure may be novel. Also, the recommendation seems also to be chaining these together in a workflow. I mean so far many of these ideas are not new at all. They are very old. It’s more important as ever to be a student of history so we aren’t doomed to repeat the same lessons. #mlops #llm

36

6 Comments

Today, we’re launching Substrate. We’re also announcing our $8M Series Seed led by Lightspeed. We believe the most robust, reliable, and productive uses of AI are when many models are used in coordination with each other, with regular programming mixed in. Most people building with AI already know this. But the available tools are suboptimal, unless you work for Google. For the rest of us, we’re left creating an unwieldy mess of chained API calls to multiple providers, with many roundtrips in between. We took a hard look at this state of affairs, and recognized how much it is stifling progress. Building large multi-step AI workloads requires sophisticated high-performance tooling and infrastructure. Nobody wants to deal with more tooling and infrastructure… but everyone would benefit from simple, intuitive interfaces that abstract away a powerful system underneath. No tooling, no infrastructure – just elegant abstractions.

25

1 Comment

The best part of investing at the earliest stages of company building is every once in a while you meet founders who can see the future. Ben Guo & Rob Cheung are two of those founders and after a year of building in stealth they're launching ꩜Substrate, the first inference API optimized for multi-step AI workloads. 👉 https://substrate.run With ꩜Substrate, you connect nodes from a curated library that includes optimized ML models, built-in file and vector storage, a code interpreter, and logical control flow. By simply connecting nodes, you describe a graph workflow, which Substrate then analyzes and runs as fast as possible. Entire graphs of many nodes will often run on a single machine, with microsecond communication between tasks. We are thrilled to participate in their 💵 $8M Seed led by Lightspeed (Guru Chahal/Nnamdi Iregbulem) alongside our friends at South Park Commons (Arian Agrawal), Craft Ventures, Red Swan Ventures (Eric Rubin/Sandy Cass), joined by amazing 🧠 founders and 👨‍🏭 operators like Guillermo Rauch, Immad Akhund, William Gaybrick, Chris Best, Gokul Rajaram, Shreyas Doshi, Jana Messerschmidt, Michael Manapat, and Matthew Hartman.

23

1 Comment

Accelerating ViTs 1.46x in PyTorch using our new Block Sparse Kernels 🔥 Check it out: https://hubs.la/Q02x72-c0

215

💥 Exciting update on accelerating vision transformers! See how the PyTorch team, Meta's FAIR team, and Quansight sped up ViTs using Block Sparsity. They demonstrated up to 1.46x speedup on ViT-B-16 using block sparsity and pruning on module weights with <2% accuracy drop. Key enabler: efficient sparse kernels in @PyTorch. Shout out to Andrew M. James, Pearu Peterson, Nikita (Nik) Vedeneev

7

2 Comments

Two new papers this week with my colleagues at Visa: 1. Measuring machine learning model robustness in production using properties of harmonic functions (https://lnkd.in/gnhbseY6) 2. Measuring LLM hallucination based on same, unsupervised and automated! (https://lnkd.in/geMGX7HV)

33

6 Comments

Last weekend, I read "Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought" by Lee et al, 2024. In this paper propose to use small (<1B params) language model (SLM) to provide a sequence of rationale steps to a LLM to predict the output of a task. The authors argue that this is an efficient way to use LLMs to solve a task by only training the SLM that provides the rationale to the LLM. To me, this approach is interesting because they use reinforcement learning by assigning reward signals for a SLM from a rationale-oriented and task-oriented perspectives to improve the outputs of a LLM. Enjoy! Paper: https://lnkd.in/gcQwW-ty

3

Great work complementing benchmarks which focus on output quality. Speed and cost effectiveness are equally crucial in productization to deliver customer value and business value!

7

Large language models (LLMs) are known to generate plausible sounding factually incorrect responses, especially when faced with unfamiliar concepts, queries. Recent research suggests that strategic fine-tuning and in-context learning, retrieval augmentation can significantly reduce these hallucinations. #llms #chatgpt4 #llmops #aiml #gemini #phi #mlops https://lnkd.in/duRrC3yg

4

🚀 We are thrilled to announce the release of our new paper: "The Evolution of Multimodal Model Architectures." In this paper, we provide a bird's-eye view of all the state-of-the-art multimodal (large) models that have emerged since the inception of Transformers. It highlights the pros and cons of each model architecture, grouping them into four broad categories. These categories can serve as a blueprint to help you determine which architecture is best suited for your specific scenario. A huge thank you to all the co-authors and especially to Shakti Wadekar for putting this paper together during his PhD at Purdue University. Paper: 🔗 [https://lnkd.in/gZHQUS6c] Authors: Shakti Wadekar, Abhishek Chaurasia, Aman Chadha, and Eugenio Culurciello #AI #MachineLearning #MultimodalModels #Transformers #Research #PurdueUniversity

75

4 Comments

Do you want to build and deploy custom LLM agents from scratch? Check out my blog post with Databricks Mosaic Research to read about how JetBlue is shipping robust GenAI solutions with DSPy and Databricks! https://lnkd.in/gKcVKQ_B

82

4 Comments

I think AI agentic machine translation has huge potential for improving over traditional neural machine translation, and am releasing as open-source a demonstration I'd been playing with as a fun weekend project. Using an agentic workflow, this demonstration (i) Prompts an LLM to translate from one language to another, (ii) Reflects on the translation to come up with constructive suggestions, (iii) Uses the suggestions to refine the translation. In our limited testing, this is sometimes competitive with, and sometimes worse than, leading commercial providers. But it gives a highly steerable translation system where by simply changing the prompt, you can specify the tone (formal/informal), regional variation (do you want Spanish as spoken in Spain or as spoken in Latin America?), and ensure consistent translation of terms (by providing a glossary). This is not mature software. But I hope the open-source community can make agentic translation work much better. Given how a simple reflection workflow already gives decent results, I think there's significant headroom to make agentic translation much better. Releasing an early software prototype like this is something new I decided to try to see if it is helpful to the developer community. I'd love any feedback on this. Thanks to Joaquin Dominguez, Nedelina Teneva, PhD and John Santerre PhD for help with this. https://lnkd.in/gjGANH6H

5,275

164 Comments

In his latest newsletter, Andrew Ng gives a great explanation of why evaluation of applications built using LLMs is so incredibly difficult. I recommend reading the post, but at a high level: It is impractical to build ground truth datasets to evaluate LLM systems that generate a free text response since there are usually many different ways to answer a question. And human evaluations are too slow to support development. Many teams therefore use LLM-based evals where advanced LLMs evaluate the generate responses using a provided scoring rubric. But there are many issues with this approach too, including noisiness of the results, cost, and speed. Improving evaluations is therefore an important research area (and one interesting direction Andrew mentions is agentic workflows). #LLM #AI #ArtificialIntelligence

9

1 Comment