Angan M.

🌟 Exciting News in AI Forecasting! 🌟 Google Research has unveiled a groundbreaking approach to time series forecasting. Their new decoder-only foundation model is setting a new standard in accuracy and flexibility, capable of leveraging vast datasets across multiple domains seamlessly. This innovative model bypasses the need for domain-specific adjustments, offering a robust solution that adapts to various applications from finance to weather forecasting. Imagine the possibilities when these predictions become more precise and reliable! For professionals involved in data science, AI, and predictive analytics, this could mean more accurate forecasts, efficient resource management, and ultimately, better decision-making. Eager to get my hands dirty with it. Do you think that the future of forecasting will be in tied to the evolution of foundation models vs current approaches? #ArtificialIntelligence #Innovation #GoogleResearch #Forecasting #FoudnationModels

76

5 Kommentare

Hello people if anyone wants to write there code and try to comment here's the info about Latex and it's usage. https://lnkd.in/gKeJcvsT. #jupyter notebook #ML #Latex #datascience

3

From D Hwang [Google LLC] (2024) FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information https://lnkd.in/ecCv_Ccs - Adam optimizer can be considered as an approximation of natural gradient descent through the use of diagonal empirical Fisher information matrix (FIM). - The diagonal FIM results in loss of covariance information but retains variance. This approximation seems to work well in practice. - The second momentum term in Adam is actually the diagonal empirical FIM. However, Adam uses the square root of FIM instead of the direct inverse. This is to get a more stable estimate. - The FIM relies on empirical approximations using training data rather than model distribution. This works better for discrete distributions (like text) than continuous (like images). - Some flaws in original Adam algorithm are uncovered based on the analysis. Proposed fixes are: use invariant natural gradient for momentum, adjust bias corrections, clip gradients. - Weight decay should be applied as a natural gradient too based on FIM. This explains the success of AdamW which decouples weight decay. - Overall modified algorithm is named Fisher Adam (FAdam) which shows strong performance on text, speech and image tasks. abs: This paper establishes a mathematical foundation for the Adam optimizer, elucidating its connection to natural gradient descent through Riemannian and information geometry. We rigorously analyze the diagonal empirical Fisher information matrix (FIM) in Adam, clarifying all detailed approximations and advocating for the use of log probability functions as loss, which should be based on discrete distributions, due to the limitations of empirical FIM. Our analysis uncovers flaws in the original Adam algorithm, leading to proposed corrections such as enhanced momentum calculations, adjusted bias corrections, and gradient clipping. We refine the weight decay term based on our theoretical framework. Our modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR. #AI #LLM

24

Besides supporting your coding as an assistant, function calling using LLMs is an incredibly powerful and fun tool to use in your software projects. My first mini-project to try this out was to create a 'textual calculator'. All it did was convert textual equations into a JSON format that contains the detected operands and operators. This allowed my software to call a math function based on the JSON data. This was run locally with a last generation GPU on a relatively tiny model. Although LLMs can do easy math without function calling, this is not advisable for anything reliable. Can't wait to take the course below!

3

🥁 Liu et al. (2024) have proposed M3Net [https://lnkd.in/dbf6QGcV], a framework designed to address the challenges in LiDAR segmentation for autonomous driving. It stands out by managing tasks across different datasets and sensor modalities using a single parameter set, aligning data, features, and labels across these diverse sources. Its capability is highlighted through its impressive performance across several benchmark datasets, demonstrating its utility in practical scenarios. 🤨 However, I believe there are a few areas where the approach might face challenges. 1️⃣ The method aligns data from multiple sources, which, while beneficial for model robustness and generalizability, significantly increases the complexity of the model training process. This can lead to higher computational overhead and potentially longer training times, which might be a drawback in resource-constrained scenarios. 2️⃣ Although M3Net is designed to handle multiple datasets, the scalability of the approach when continuously adding new data sources or modalities isn't deeply explored. As the number of datasets increases, the alignment process across them could become increasingly cumbersome and might require additional fine-tuning or architectural adjustments. 3️⃣ The success of the model heavily depends on the effectiveness of the multi-space alignment mechanisms. If these alignments are off due to errors in sensor calibration or inconsistencies in data annotations across datasets, the model’s performance could be adversely affected. 4️⃣ By training a single model across multiple domains, there's a risk of diluting specific domain knowledge, potentially leading to a "jack of all trades, master of none" scenario. This could affect performance in specific applications where highly specialized models might perform better. 5️⃣ While the model shows impressive performance on various benchmarks, the robustness of the evaluation itself in terms of real-world applicability isn't deeply discussed. More diverse and challenging real-world scenarios could provide a better testbed for the model's practical effectiveness. ✅ Addressing these challenges in future iterations could help in further enhancing the utility and applicability of the M3Net framework in autonomous driving and other related fields.

13

Mixture of A Million Experts X O He [Google DeepMind] (2024) https://lnkd.in/exxiKu9J - The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows. - Sparse mixture-of-experts (MoE) architectures have emerged to address this issue by decoupling model size from computational cost, but are limited to a small number of experts due to computational and optimization challenges. - This paper introduces PEER (parameter efficient expert retrieval), a novel layer design that utilizes the product key technique for sparse retrieval from a vast pool of tiny experts (over a million). - Experiments demonstrate PEER layers outperform dense FFWs and coarse-grained MoEs in terms of performance-compute trade-off. - By enabling efficient utilization of massive number of experts, PEER unlocks potential for further scaling of transformer models while maintaining computational efficiency. abs: The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows. Sparse mixture-of-experts (MoE) architectures have emerged as a viable approach to address this issue by decoupling model size from computational cost. The recent discovery of the fine-grained MoE scaling law shows that higher granularity leads to better performance. However, existing MoE models are limited to a small number of experts due to computational and optimization challenges. This paper introduces PEER (parameter efficient expert retrieval), a novel layer design that utilizes the product key technique for sparse retrieval from a vast pool of tiny experts (over a million). Experiments on language modeling tasks demonstrate that PEER layers outperform dense FFWs and coarse-grained MoEs in terms of performance-compute trade-off. By enabling efficient utilization of a massive number of experts, PEER unlocks the potential for further scaling of transformer models while maintaining computational efficiency. #AI #MachineLearning

87

Exploring Electricity Price Forecasting in the Netherlands: From Baseline Models to N-BEATS and Exogenous Variables. Discover how we utilized Dart and Statsforecast libraries to establish baseline models, then refined with N-BEATS Fine Tuning using Optuna, and finally incorporated exogenous variables for accurate predictions.

13

💸 Revenue Team Wants AI Costs, But Your MVP's Still Loading... 🤯 A Founding MLE Guide to Pre-MVP Cost Estimation 🔮 Initial Cost Estimation 1. Select Representative Models 📊:    Choose models at various scales. For NLP, consider BERT, GPT-J, (FLAN-)T5-XXL, and Falcon-40B. 2. Match Models to Hardware 🖥️:   Pair each model with appropriate GPU hardware. Example: GPT-J on A10G, Falcon-40B on A100 40GB. 3. Estimate Request Completion Time ⏱️:   Approximate time for each model to complete a request. Example:   - GPT-J: 1 second on A10G (made up number!)   - Falcon-40B: 10 seconds on A100 40GB (made up number!) 4. Calculate Hourly Costs 💸:   Research current GPU pricing. Example:   - A10G: ~$2 per hour (Modal Labs pricing)   - A100 40GB: ~$5 per hour (Modal Labs pricing) 5. Compute Cost per 1000 Requests 🧮:   Use the formula: (Seconds per request * Cost per hour) / (3600 seconds) * 1000   Examples:   - GPT-J: (1 * $2) / 3600 * 1000 ≈ $0.60 to serve 1000 requests   - Falcon-40B: (10 * $5) / 3600 * 1000 ≈ $3.00 to serve 1000 requests 6. Provide Order of Magnitude (OOM) Estimates 📏:   Present a range of costs based on different models. In this case, $0.60 to $3.00 per 1000 requests. 7. Factor in SLAs and Latency Requirements ⚡:   SLAs affect costs and can help constriant the solution space. For example, achieving a p99 latency of Xms might be 10x more expensive due to keeping a machine warm 🔧 Ongoing Cost Optimization (Thanks for the beautiful post Outerbounds) - Analyze Top-line Costs 📈:    Regularly review cloud bills to focus optimization efforts. - Identify Cost-driving Instances and Workloads 🔍:    Use tools to pinpoint expensive instances and tasks. - Monitor Resource Utilization 📊:    Avoid over-provisioning; pay attention to actual usage. - Optimize Workloads 🎛️:    Right-size resource requests based on real usage patterns. - Choose Optimal Execution Environments 🌐:    Leverage multi-cloud strategies for cost advantages. - Refine Based on Specific Needs 🎯:   Narrow estimates by understanding customer problems and required model scales. - Explore Serverless Options ☁️: - Stay Informed on Pricing 📚: Valuable resources: https://lnkd.in/gwragJBw https://lnkd.in/gVkGenhi https://lnkd.in/gq8pi4qd This framework is inspired by countless interactions with the ML/AI community #MachineLearning #CostEstimation #DataScience #AI #CloudOptimization

8

1 Kommentar

Data Scientists, This is BIG! Snowflake's Arctic LLM is Here – and Open-Source! 🤯 Get your hands on it now: Interactive Demo: Explore Arctic's capabilities with this Streamlit app → https://lnkd.in/gJ7y38Ng Replicate: Ready to experiment? Find it on Replicate → https://lnkd.in/gyDXiMju Why This Matters: Arctic lets you quickly build demos of your models. This is a game-changer for showcasing your work and making your models more accessible! Huge thanks to Snowflake for making this happen! Are you excited to try Arctic? Let me know in the comments! 👇 #LLM #OpenSource #DataScience #Streamlit #Snowflake

8

Interested in Orchestra but not sure how it compares to platforms like Airflow? Check out this video below! Video 🔗 https://lnkd.in/e_DwRAQy cc Hugo Lu #dataengineering #airflow #datapipelines

10

3 Kommentare

My take on Yann LeCun vs. Elon Musk: I have more respect for Yann LeCun for his contribution to AI research, especially for making this knowledge accessible and available to everyone. We all should appreciate this part. Yann LeCun said that he published more than 80 papers in 2022 alone while Elon Musk arrogantly said "That is nothing". I think both are wrong here. In my opinion, the number of papers does not matter although this is how acamedia usually works. I rather have one paper that is groundbreaking for many generations to come over 1000s of papers. So the number of papers should not be an indication here of anything. In contrast, ignoring the contribution of top-class researchers who helped to accelerate the development in this field and more importantly sharing their knowledge publically is unjust. Both have different focuses, Elon focuses on the business value which is also fine, while LeCun focuses on the scientific value. By the end of the day, both learn by doing and this is what makes them pioneers in their field. But if one of them were unhealthy, no one of them would be able to do anything. It is always better to be grateful for what we were given such as health, intelligence, and chances that without no one successful would be at where he\she is today.

3

Today's data landscape is marked by its complexity, with a significant portion being temporospatial and governed by Partial Differential Equations (PDEs). However, solving such problems traditionally incurs high computational costs. While large language models aren't suited for this task, alternative approaches like Variational Autoencoders (VAEs) show promise. Considering the intricate nature of the data, a Multi-Modal approach seems apt. Currently, we're addressing complex temporospatial data in the capital market domain, involving both numeric and textual information. Our solution entails building a multi-modal neural network to predict 1D time-dependent PDE systems, crucially requiring high accuracy. The approach comprises two main steps: Equation Builder: The crux lies in determining the appropriate PDE based on the actual data, considering the stochastic variations where no single equation suffices. To address this, we propose constructing a multimodal transformer capable of deducing the equation from the data. This transformer encodes both numerical and symbolic data types, facilitating multi-operator learning. Moreover, the model is designed to generalize to unseen physical parameters and timestamps, without necessitating fine-tuning. Model Generation and Execution: Once the equation is derived, we translate it into a corresponding multilayer perceptron (MLP) architecture via code. This architecture serves as the foundation for further modeling and predictive tasks. This approach confronts the challenge of handling two types of information: data and symbolic. Importantly, our system isn't constrained by a specific symbolic expression, affording flexibility to utilize various PDEs for financial data. Additionally, it accommodates variations in data distribution and input length over time. In essence, our strategy combines sophisticated techniques to tackle the complexities of temporospatial data in capital markets, ensuring accuracy and adaptability. Through this approach, we aim to deliver effective solutions that meet the industry's demanding requirements. For comments please reach out to ajay@innovation-hacks.ai #generativeai #llm #capitalmarkets Innovation Hacks AI

3

Depth Anything V2, a more powerful foundation model for monocular depth estimation. It is capable of 1) providing robust and fine-grained depth prediction, 2) supporting extensive applications with varied model sizes (from 25M to 1.3B parameters), and 3) being easily fine-tuned to downstream tasks as a promising model initialization. https://lnkd.in/gKTRjB4N

3

1 Kommentar

Kolmogorov-Arnold Networks look interesting and explainable. But weren’t we doing this already in engineering school? I mean curve matching and fitting. The innovation here is that they are doing it on a interrelated multiple equation basis, where the “constants” (activation functions) are being changed dynamically, which is pretty neat. Definitely will be more explainable than multilayer perceptrons or simple neural networks. KANs “explainable” nature is equivalent to explain why the constants in the equations are what they are. In other words, this is not the path to AGI, or discovering news laws of physics either, although the method is elegant. In my opinion, where the “gold” resides in Kolmogorov-Arnold Networks is this: Being trained with reasonable good data and then use the trained network to (i) predict a more accurate output when any of inputs is out of whack; (ii) discard inputs that are out of range; (iii) qualify the inputs before a prediction is applied; (iv) get a notion of the extend at which the inputs are diverging from the ground truth One case that comes to my mind is Virtual Flow Meters where data (inputs) coming from well tests is very inconsistent due to uncertainties ocurring during measurement. In the abstract the authors got carried away a bit: " KANs are shown to be useful collaborators helping scientists re-discover mathematical and physical laws." It seems incorrect to me. You don’t rediscover the laws of physics with this ML algorithm. You get a hint of what external factors could be affecting the input data that make the output deviate from the laws of physics. #machineLearning #physics #artificialIntelligence #AI #AGI #spe #petroleumEngineering #VirtualFlowMeters #dataScience

47

10 Kommentare

Back in December last year, I wrote a two-part article on why deep learning-based algorithms are better suited for predictive maintenance analytics. (https://lnkd.in/eGbJhgP3) In the second part of this article series, I mentioned some toying around that I was doing in this area. This two-part article is a result of that. We will essentially build an LSTM-based predictive analytics model. The topic of LSTMs has been extensively covered in many of my video series, like Edge AI Bytes, GenAI Bytes and Deep Learning with Python. #data #analytics #ai #ml #datascience #designedanalytics #deeplearning #generativeai #genai #lstm #rnn https://lnkd.in/gbekW3DV

1

1 Kommentar

𝐒𝐮𝐩𝐩𝐨𝐬𝐞 𝐑𝐞𝐦𝐨𝐭𝐞 𝐒𝐞𝐧𝐬𝐢𝐧𝐠 𝐃𝐚𝐭𝐚 𝐰𝐚𝐬 𝐚𝐬 𝐄𝐚𝐬𝐲 𝐭𝐨 𝐅𝐢𝐧𝐝 𝐚𝐧𝐝 𝐔𝐬𝐞 𝐚𝐬 𝐒𝐩𝐨𝐭𝐢𝐟𝐲 Imagine a world where accessing remote sensing data is as effortless as streaming your favorite song on Spotify? No more hunting down data owners, negotiating terms, or struggling with incompatible formats. Forget about digital rights management complexities. Just a simple search, find, and use the right data for your needs. How transformative would that be? Would that take your breath away? 𝐓𝐞𝐥𝐥 𝐘𝐨𝐮𝐫 𝐆𝐞𝐨𝐬𝐩𝐚𝐭𝐢𝐚𝐥 𝐒𝐭𝐨𝐫𝐲 We are experts at telling your geospatial story. Working with sales and marketing teams, we help share your value proposition in a clear, understandable, compelling way. Want to know more .. sign up for a free consultation: https://lnkd.in/gYhrCF-f Join us for a webinar discussing how to drive geospatial sales and marketing success with AI: https://lnkd.in/gui7u9aV #spatial #geospatial #sales #marketing #gis

6

2 Kommentare

Is Your RAG a Step Backwards? Excellent tutorial (below) from Haziqa Sajid in the all-important field of hybrid search—combining dense vectors with other field data types. Anyone with existing search configurations will know that hybrid approaches are a necessity and can be very challenging compared to merely building some greenfield RAG demo. In my experience, the real challenge is in ranking, which is the core of search. Many folks implementing RAG are entering the world of search, albeit via generative approaches. The uninitiated are quickly finding out that relevance is everything. Moreover, they’re discovering an even more pressing issue: user intolerance of irrelevant or noisy results, which can easily kill the entire effort—users simply stop using the shiny new toys and revert back to legacy systems, many of which are sufficiently well optimized. Compared to such systems, naive implementations of LLM-backed solutions can seem like a backward step.

1

1 Kommentar