Hugging Face

Shape of Motion 4D Reconstruction from a Single Video paper page: https://buff.ly/3S9Zroj Monocular dynamic reconstruction is a challenging and long-standing vision problem due to the highly ill-posed nature of the task. Existing approaches are limited in that they either depend on templates, are effective only in quasi-static scenes, or fail to model 3D motion explicitly. In this work, we introduce a method capable of reconstructing generic dynamic scenes, featuring explicit, full-sequence-long 3D motion, from casually captured monocular videos. We tackle the under-constrained nature of the problem with two key insights: First, we exploit the low-dimensional structure of 3D motion by representing scene motion with a compact set of SE3 motion bases. Each point's motion is expressed as a linear combination of these bases, facilitating soft decomposition of the scene into multiple rigidly-moving groups. Second, we utilize a comprehensive set of data-driven priors, including monocular depth maps and long-range 2D tracks, and devise a method to effectively consolidate these noisy supervisory signals, resulting in a globally consistent representation of the dynamic scene. Experiments show that our method achieves state-of-the-art performance for both long-range 3D/2D motion estimation and novel view synthesis on dynamic scenes.

🔥Mistral-NeMo, a 12B LLM, launched a few hours ago has the community abuzz! 👇Understand nuances about model's capabilities that were not covered in the release blog. We also cover the community's initial reactions. - Trained jointly by Mistral AI and NVIDIA. Mistral NeMo has 50% more parameters (12B) compared to Llama 3 (8B). - Open-source model with Apache 2.0 license 🎉 - 12B parameter count puts it between smaller models (7-8B) and larger ones (30-70B), potentially offering a good balance of performance and resource requirements. - Trained with "quantization awareness," allowing for FP8 inference without performance loss. This approach appears forward-thinking, potentially allowing for better performance when quantized compared to models without quantization-aware training. - VRAM required to run: Model would need about 12GB of VRAM at 8-bit precision, or 6GB at 4-bit precision (not counting context). - Can potentially run on consumer GPUs with 16GB VRAM, and possibly on 12GB cards with quantization. The model seems to be designed to fit on NVIDIA L40S, GeForce RTX 4090, or RTX 4500 GPUs.🤔 - 128k Context Window available - However, using full context size could significantly increase memory requirements, might not be practical for all usecases. - This large context window (128K) is rare among models of this size. This makes Mistral-Nemo potentially valuable for tasks requiring long-range understanding. - Mistral Nemo is a joint release with Nvidia: Model was trained using 3,072 H100 (80GB) -- You can see significant computational resources have been used. - Multilingual: Trained on multiple languages. Benchmarks for non-English languages look particularly strong. - Tokenizer: New Tekken tokenizer based on tiktoken (by OpenAI), which uses byte-pair encoding - llama.cpp compatibility: Not yet out-of-the-box; however, a PR is in motion, might take couple days. This might potentially delay a widespread adoption for the model. - Released same day as GPT-4o Mini 😉-- We are excited to see how these two would compete in Lmsys leaderboard (a @Gradio-built leaderboard and Arena)! - Fine-tuning: As per the community, Mistral-nemo seems to be more suitable for fine-tuning compared to Llama 3. - Temperature used: The model reportedly (HN/Reddit comments) requires lower temperature settings (around 0.3) compared to previous Mistral models, which might affect its behavior in various applications. Useful to know if you were planning for a drop-in replacement for mistral models. - Potential: Could be particularly useful for tasks like coding assistance, creative writing, and role-playing. - Base and Instruct Models on Hugging Face: 1. Mistral-Nemo-Instruct-2407: https://lnkd.in/ek6DHuZD  2. Mistral-Nemo-Base-2407: https://lnkd.in/gRdzezbr Gradio chatbot demo on 🤗Spaces: https://lnkd.in/g_fUTTF6

Chameleon 🦎 by Meta is now available in Hugging Face transformers 😍 A vision language model that comes in 7B and 34B sizes 🤩 But what makes this model so special?  Demo and more in comments, keep reading ⥥ Chameleon is a unique model: it attempts to scale early fusion 🤨 But what is early fusion? Modern vision language models use a vision encoder with a projection layer to project image embeddings so it can be promptable to text decoder (LLM) Early fusion on the other hand attempts to fuse all features together (image patches and text) by using an image tokenizer and all tokens are projected into a shared space, which enables seamless generation 😏 Authors have also introduced different architectural improvements (QK norm and revise placement of layer norms) for scalable and stable training and they were able to increase the token count (5x tokens compared to Llama 3 which is a must with early-fusion IMO) This model is an any-to-any model thanks to early fusion: it can take image and text input and output image and text, but image generation are disabled to prevent malicious use. One can also do text-only prompting, authors noted the model catches up with larger LLMs (like Mixtral 8x7B or larger Llama-2 70B) and also image-pair prompting with larger VLMs like IDEFICS2-80B (see paper for the benchmarks) Thanks for reading!

Hugging Face has made it seriously easy to deploy Gradio apps through the use of Spaces, would recommend everyone to give it a shot!

I'm thrilled to share Multi-LoRA Serving on 🤗Text Generation Inference. Are you navigating the complexities of managing multiple AI models? Say goodbye to the hassle and high costs! With Multi-LoRA, you deploy once and dynamically serve multiple specialized models using the same base model. This means significant cost savings and reduced operational complexity. 📉💡 In this blog you will learn: 🚀 How to deploy a single base model and use it to serve up 30+ specialized models 🧠 Build deep intuition on how Multi-LoRA serving works 💡 Gain insights on the cost efficiency and operational benefits of using Multi-LoRA serving co authors: David Holtz Diego Maniloff

You can now embed a Hugging Face datasets viewer preview as an iframe. I used this to create a Space for quickly viewing all the datasets inside a Hugging Face collection. For example, this Space https://lnkd.in/epyq_T6P allows you to explore all the datasets inside the Alpaca Style Datasets collection: https://lnkd.in/edWHzAFT. This can be very helpful for quickly exploring which datasets might be useful for a project. You can duplicate this Space to explore datasets in any HF collection!

HOT & NEW - NVIDIA's BigVGAN! In essence it is an AI-powered sound synthesizer! - It extracts the mel-spectrogram from the input audio and uses it to produce the actual matching sound waves! - Its power lies in its ability to do this for a wide range of sounds with high quality and flexibility. - BigVGAN is a universal neural vocoder model that generates audio waveforms using mel spectrogram as inputs for a wide variety of inputs like music, voice, sounds, and even those which are outside its training data. - BigVGAN Gradio demo is live on Hugging Face Spaces: https://lnkd.in/g4aBv6iP - Model collection Hugging Face Hub: https://lnkd.in/grvmb26z

We just integrated dataset viewer in AutoTrain 💥 So, now you can look into your dataset, identify correct splits and columns before training the model, without leaving the page 🚀

Google presents YouTube-SL-25 A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus paper page: https://lnkd.in/eUAHknyb Even for better-studied sign languages like American Sign Language (ASL), data is the bottleneck for machine learning research. The situation is worse yet for the many other sign languages used by Deaf/Hard of Hearing communities around the world. In this paper, we present YouTube-SL-25, a large-scale, open-domain multilingual corpus of sign language videos with seemingly well-aligned captions drawn from YouTube. With >3000 hours of videos across >25 sign languages, YouTube-SL-25 is a) >3x the size of YouTube-ASL, b) the largest parallel sign language dataset to date, and c) the first or largest parallel dataset for many of its component languages. We provide baselines for sign-to-text tasks using a unified multilingual multitask model based on T5 and report scores on benchmarks across 4 sign languages. The results demonstrate that multilingual transfer benefits both higher- and lower-resource sign languages within YouTube-SL-25.

State of Open AI - June Edition! https://lnkd.in/db6HTysc June is a great month for the open ML space, but it's hard to keep up! I curated a list of new model releases and grouped them into categories. Let me know if this is useful :)