Clem Delangue 🤗’s Post

If AI is the new electricity, data is the power grid ⚡ Hugging Face has been the largest repository of public datasets for training LLMs used by enterprises. Discovering relevant, aligned data faster and better is a significant value add. However, in my experience working with some of the biggest enterprises, performance on public data often doesn't align well with their custom use cases. Creating a 𝐆𝐨𝐥𝐝𝐞𝐧 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤 𝐃𝐚𝐭𝐚𝐬𝐞𝐭 tailored to one's use case is crucial for accurate evaluation. Here's a strategy I recommend for training new models. This gets simpler if your enterprise uses tools like Snowflake, SuperAnnotate, or Lilac (acquired by Databricks). 1. 𝐂𝐨𝐥𝐥𝐞𝐜𝐭 𝐚 𝐆𝐨𝐥𝐝𝐞𝐧 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤 𝐃𝐚𝐭𝐚𝐬𝐞: Ensure it's human-reviewed and thoroughly covers all result possibilities in a balanced manner. 2. 𝐋𝐞𝐯𝐞𝐫𝐚𝐠𝐞 𝐔𝐬𝐞 𝐂𝐚𝐬𝐞-𝐒𝐩𝐞𝐜𝐢𝐟𝐢𝐜 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐃𝐚𝐭𝐚: If you don't have your own, use Hugging Face with filtration techniques to find relevant datasets. Filter based on your use case, used by public models, popularity, example size, etc. 3. 𝐑𝐮𝐧 𝐃𝐢𝐬𝐭𝐫𝐢𝐛𝐮𝐭𝐢𝐨𝐧 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬: Compare the Golden Benchmark dataset with candidate public datasets to identify the closest match to your use case. While ML-driven clustering and analysis techniques are helpful, manual review is essential. 4. 𝐓𝐫𝐚𝐢𝐧 𝐘𝐨𝐮𝐫 𝐌𝐨𝐝𝐞𝐥: If you find a good match, train your LLM/model on that data. If not, invest time and resources in collecting your own data. Period. 5. 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐞 𝐂𝐨𝐧𝐬𝐢𝐬𝐭𝐞𝐧𝐭𝐥𝐲: Always evaluate on the Golden Benchmark dataset, prioritizing it over public/academic benchmarks. Data is crucial. I've personally seen performance jumps of over 10% with data refinement. Let me know your thoughts. #ArtificialIntelligence #MachineLearning #DataScience #HuggingFace #ModelTraining #DataQuality #AI #ML #Benchmarking

🎉 Just shipped! Big update of the Hugging Face Datasets page. You can now filter datasets: 1. By Modalities (🖼️ Image, 🔊 Audio, 📝 Text, ...) 2. By Dataset Size (from 1k to ∞ samples) 3. By Format (JSON, CSV, Parquet, ...) Should be easier to find the perfect dataset(s) for your next project(s)

Day 7 : COMPLETED : ✓ Binary search Algorithm in 2D Array ✓ Matrix is sorted in a row wise and column wise manner ✓ Just cracked a binary search algorithm problem! 🕵️♂️ This time, it was all about transforming a 2D array – turning rows into columns to create a matrix. 💡 The problem pushed me to think creatively and optimize my solution. #100daysofcodechallenge #dsawithkunal #dsachallenge #100daysofcode

Day 3: Multidimensional Arrays 📏 Explore multidimensional arrays. Create a 2D array with: import numpy as np a = np.array([[1, 2], [3, 4]]) Use .shape to see its dimensions: print(a.shape) Great for representing matrices or grids! 🌐 #DataScienceJourney #30DaysOfNumPyDay03 ✨ Now, it's your turn! Create your own 2D arrays, try different shapes, and experiment with accessing elements. Share your discoveries in the comments or on social media using #NumPyMasteryChallenge and #30DaysOfNumPy. Let's dive into the world of multidimensional arrays together! 💡💬

Greetings LinkedIn family, Code 1: Edge Detection and Contour Extraction: It loads an image and converts it to grayscale. Using the Canny edge detection algorithm, it identifies edges in the grayscale image. Contours are then extracted from the detected edges. Vectorization and Tokenization: The contour points are extracted and stored as vectors. Each contour point serves as a token. These tokens (contour points) are vectorized, meaning they are represented as numerical arrays. Finally, the original image, its grayscale version, and multi-color text overlaid on the original image are displayed using Matplotlib. Code 2: Import Libraries: Import necessary libraries - Librosa for audio processing and NumPy for numerical operations. Load Audio: Load an audio file and its sample rate. Extract MFCCs: Use Librosa to compute MFCC features from the audio signal. Tokenize: Treat each MFCC frame as a token, organizing them into a matrix for further processing. Vectorize: Convert each MFCC token into a numerical vector using NumPy arrays. Print: Display the vectorized form of the audio.