Farrukh Rahman

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

29

Last weekend, I read "Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought" by Lee et al, 2024. In this paper propose to use small (<1B params) language model (SLM) to provide a sequence of rationale steps to a LLM to predict the output of a task. The authors argue that this is an efficient way to use LLMs to solve a task by only training the SLM that provides the rationale to the LLM. To me, this approach is interesting because they use reinforcement learning by assigning reward signals for a SLM from a rationale-oriented and task-oriented perspectives to improve the outputs of a LLM. Enjoy! Paper: https://lnkd.in/gcQwW-ty

3

This approach could turn the Generative AI industry upside down. Most everyone knows that AI requires computers to do a lot of calculations and especially for Large Language Models (LLMs) like ChatGPT. The majority of these calculations are actually “Matrix Multiplications”. (You remember, a 3x3 matrix times another 3x3 matrix). If the researchers in the article are proven correct, they have demonstrated a way that Matrix Multiplication can be replaced with a more efficient alternative inside the LLM getting nearly the same results. The article positions this savings as a win for the environment (yeah environment), but I think they are missing the boat. More efficient means less electricity and potentially a smaller need for GPU resources, the bread and butter of stock darling Nvidia. When you combine this cost saving with reduced privacy concerns of hosted LLMs (Open AI) and innovative LLM licensing schemes (LibreChat - https://lnkd.in/gKQjMfz5) you can see the cracks in the current direction of the LLM industry. Q: If you could run your own LLM on your own private environment with MIT open-source licensing and without expensive GPUs, why exactly do you need Nvidia, Open AI and alike? Bold prediction: if the researchers get this working at scale, we could see a rapid shift and commoditization in the LLM and generative AI space. All the pieces are in place. TLDR details from the article, describing the innovation: “The researchers' approach involves two main innovations: first, they created a custom LLM and constrained it to use only ternary values (-1, 0, 1) instead of traditional floating-point numbers, which allows for simpler computations. Second, the researchers redesigned the computationally expensive self-attention mechanism in traditional language models with a simpler, more efficient unit (that they called a MatMul-free Linear Gated Recurrent Unit—or MLGRU) that processes words sequentially using basic arithmetic operations instead of matrix multiplications.” #AI #GenerativeAI #Innovation

22

Conditional Computation in LLMs (MoEs): Understanding the Concept of MoEs The Mixture of Experts (MoE) works on the idea of conditional computation, allowing the model to only run through some parts (an Expert) for a given input compared to a dense model where all parameters are used. It consists of a gating mechanism that selects the experts for the given input, enabling the model to scale without increasing the required computation. Popular LLM architectures like Mixtral and Grok use the same concept. What is MoEs? The Mixture of Experts (MoE) makes a simple modification to the usual architecture by replacing all or some of the Feed Forward layers with an MoE layer. The MoE layer consists of: 1. Router: A parameter gating mechanism that selects the experts used to process the given input by producing a probability distribution over experts. 2. Experts: Standalone NN modules with independent sets of parameters trained jointly with routers. It can use more complex architecture to create hierarchical MoE modules by implementing each expert as another MoE. - Implementation in Switch Transformers : Training was found to be 7x faster than training the base T5, keeping computation and training time constant. - Gshard and MeshTransformer : These frameworks propose efficient methods to train the MoE architectures using the parallelisation of models across GPUs/TPUs. Challenges with MoEs and Solutions 1. Inconsistent Batch Size for Experts: Can be overcome with the use of "soft" constraints in training loss. 2. Load Balancing Across Experts: The issue of repeatedly using the same few experts while training can be overcome by including better balancing mechanisms. 3. Model Instability Issues: Use of router-z Loss as an auxiliary loss to encourage smaller values within the router mechanism. 4. Model Overfitting: Requires different hyperparameter settings for effective fine-tuning. It was found that models benefit from high learning rates and smaller batch sizes. Recommended Reading/References 1. Hugging Face Blog on MoE - https://lnkd.in/g36gQJTx 2. Cameron Wolfe's Substack on Conditional Computation - https://lnkd.in/gtpjMHnJ

60

1 Comment

👉🏼 Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models 🤓 Honghao Lai https://lnkd.in/eBZ2cduS 🔍 Focus on data insights: - Large language models (LLMs) facilitate systematic reviews. - Methods and reliability of LLMs for assessing risk of bias (ROB) in randomized clinical trials (RCTs) are explored. 💡 Main outcomes and implications: - LLMs demonstrated high correct assessment rates for ROB in RCTs. - LLMs showed substantial accuracy and consistency in evaluating RCTs. - Potential of LLMs as supportive tools in systematic review processes. 📚 Field significance: - Enhancing efficiency and accuracy in assessing ROB in RCTs. - Advancing the use of LLMs in systematic review methodologies. 🗄️: #datainsights #riskofbias #RCTs #systematicreviews

1

I think AI agentic machine translation has huge potential for improving over traditional neural machine translation, and am releasing as open-source a demonstration I'd been playing with as a fun weekend project. Using an agentic workflow, this demonstration (i) Prompts an LLM to translate from one language to another, (ii) Reflects on the translation to come up with constructive suggestions, (iii) Uses the suggestions to refine the translation. In our limited testing, this is sometimes competitive with, and sometimes worse than, leading commercial providers. But it gives a highly steerable translation system where by simply changing the prompt, you can specify the tone (formal/informal), regional variation (do you want Spanish as spoken in Spain or as spoken in Latin America?), and ensure consistent translation of terms (by providing a glossary). This is not mature software. But I hope the open-source community can make agentic translation work much better. Given how a simple reflection workflow already gives decent results, I think there's significant headroom to make agentic translation much better. Releasing an early software prototype like this is something new I decided to try to see if it is helpful to the developer community. I'd love any feedback on this. Thanks to Joaquin Dominguez, Nedelina Teneva, PhD and John Santerre PhD for help with this. https://lnkd.in/gjGANH6H

5,276

164 Comments

I recently read “Towards Monosemanticity: Decomposing Language Models with Dictionary Learning” by Bricken et al 2023. Published at transformer-circuits.pub In this paper the authors discuss the use of autoencoders to extract interpretable features from a one-layer transformer model. The approach aims to improve the understanding and transparency of artificial neural networks. To do this they use the concept of dictionary learning which is a method used to decompose the activations of the transformer model into a set of basis features that are sparse and interpretable. This involves training sparse autoencoders to identify a dictionary of monosemantic features (or features with a single meaning). These monosemantic features in single units (artificial neurons) reminds me of the “grandmother neuron” theory in neuroscience which proposes that specific neurons respond to specific highly complex stimuli. However, in real brains, it is widely believed that complex representations are encoded by the collective activity of groups of neurons (and not monosemantic). But going back to the paper, I really like the visualization tools and illustration to make such complex networks more transparent (as opposed to black boxes). Paper: https://lnkd.in/gR87pJcs

3

the opposite argument here is with the design of the interface of these models. ex. ChatGPT, Gemini, and other leading models have a chat interface. this is a design flaw. use-cases like even basic research involve providing proper context for the model to produce better results. a chat interface doesn't influence or command that behavior. therefore we are practicing prompting techniques now. imo this can be easily solved by an appropriate design for the interface.

3

👉🏼 Expansive data, extensive model: Investigating discussion topics around LLM through unsupervised machine learning in academic papers and news 🤓 Hae Sun Jung 👇🏻 https://lnkd.in/ef_MF54v 🔍 Focus on data insights: - Utilized data from Web of Science and LexisNexis to explore topic modeling methods for Large Language Models (LLMs). - Evaluated various topic modeling approaches such as LDA, NMF, CTM, and BERTopic based on diversity and coherence metrics. - BERTopic demonstrated superior performance in diversity and coherence across both platforms. 💡 Main outcomes and implications: - News articles covered a wide range of topics related to LLM applications, focusing on specialized domains. - Research papers concentrated on technical aspects of LLM technology. - Insights gained can help understand future challenges and opportunities for LLMs in different sectors. 📚 Field significance: - Enhances understanding of how LLMs are utilized in academic and news contexts. - Provides insights for enterprises leveraging LLMs for service delivery. 🗄️: #topicmodeling #unsupervisedlearning #machinelearning #datainsights

1

Using a machine learning model and EEGs, the ECE Capstone team of Jioh Kim, Nick Short, Paula Perdomo, and Steven Zhang reconstructed the mental image formed when a subject was shown a picture — tech with the potential to help those who are conscious but unable to communicate. Perhaps even a newborn? That would be cool! You can read more about the ECE Capstone Design Fair here: uoft.me/avG. #UofT #UofTEngineering

10

1 Comment

Recently, I was re-introduced to the concept of catastrophic forgetting while reading about NVIDIA's ViLa model, which BTW offers pretty cool capabilities like reasoning across multiple images and learning in context and is capable of understanding videos, offering a more comprehensive understanding of visual content and its interplay with language. Overcoming catastrophic forgetting is crucial because it allows AI models to retain valuable knowledge while continuously learning and adapting, enhancing their overall performance and flexibility. Read article here: https://lnkd.in/e8gimTk4

43

1 Comment

Large language models (LLMs) are known to generate plausible sounding factually incorrect responses, especially when faced with unfamiliar concepts, queries. Recent research suggests that strategic fine-tuning and in-context learning, retrieval augmentation can significantly reduce these hallucinations. #llms #chatgpt4 #llmops #aiml #gemini #phi #mlops https://lnkd.in/duRrC3yg

4