Neural Magic’s Post

🗓 Happening today at 2PM EST! Learn why vLLM is the leading open-source inference server and how Neural Magic works with enterprises to build and scale vLLM-based model services. https://hubs.li/Q02DVnBd0

FP8 quantization is now available in vLLM - check it out! Quantized inference is one of the best ways to reduce the costs of LLM deployments.

Our bi-weekly vLLM Office Hours continue tomorrow. We are excited to bring Philipp Moritz and Cody Yu from Anyscale for a deep dive into FP8 quantization in vLLM. This is an exciting opportunity to give feedback and get your questions answered. Join us: https://lnkd.in/euF8m73q

Are you looking to optimize your #LLM inference for more performance and lower costs? Tune in to hear Eldar Kurtić, our Sr. ML Researcher, break down how quantization can optimize LLM inference and reduce memory footprint without compromising model accuracy.

The ecosystem of open-source LLMs has exploded over the past year. A new model tops the leaderboard almost every week. Enterprises can now deploy state-of-the-art, open-source LLMs like Llama 3 securely on their infrastructure of choice, fine-tuned with their data for domain-specific use cases, at a significantly lower cost than proprietary APIs. vLLM has emerged as the most popular inference server to deploy open-source LLMs with leading performance, ease of use, broad model support, and heterogeneous hardware backends. Neural Magic is a leading contributor to the vLLM project and offers nm-vllm, an enterprise-ready vLLM distribution. nm-vllm includes stable builds of vLLM with long-term support, tools for optimizing LLMs for inference with techniques like quantization and sparsity, reference architectures for scalable deployments with Kubernetes, integration of telemetry and key monitoring systems, and more. Join us on July 11, 2024, at 2:00 PM EST (11:00 AM PST) to learn why vLLM is the leading open-source inference server and how Neural Magic works with enterprises to build and scale vLLM-based model services.

Neural Magic's CEO, Brian Stevens, recently spent some time with host Heather Haskin from The Catalyst by Softchoice podcast to talk about the intersection of AI, open source, and the future of responsible development. Listen in on "The case for open-source AI" to learn more about the vital role of open-source models and why the democratization of AI is important for the success of today's enterprise. https://lnkd.in/eumGUGBH

Optimizing your AI models with techniques like sparsity and quantization increases production performance while decreasing your total infrastructure spend. Eldar Kurtić, our expert in AI model optimization, shares more details in this podcast. Check it out 👇

Are you using, or considering, vLLM for your LLM inference serving? Join us this Wednesday to ask all your questions and learn more about accelerated #LLM inference with vLLM and Neural Magic. vLLM project maintainer Simon Mo and vLLM project committer Michael Goin will spend one hour with the community answering questions and sharing recent vLLM project updates. Register and ask your questions here: https://lnkd.in/euF8m73q If you can't make it this Wednesday, you can register for the June 20th session.

We were at Nutanix #NEXTConf in Barcelona last week and it was such an exciting event! From the main stage announcements to the enthusiastic conversations with attendees at the AI Pavilion, congrats to Nutanix for creating a dynamic environment where innovation thrived. Thank you to the Nutanix team, including Gregory Lehrer, Gali Ross-Hasson, Tarkan Maner, Luke Congdon, Wolfgang Huse, and others for partnering with Neural Magic. We appreciate the opportunity to be a part of the Nutanix #AI strategy. 🙏