Daniel Lee’s Post

Should data scientists learn data engineering? Yes and no. Start by asking yourself, “What is your career focus?” (A) Do you want to stay specialized in statistical analysis? Then the answer is: NO (B) Do you want more ownership and marketability? Then the answer is: YES If you choose (B), kudos to you 👏 It helps. I reaped benefits when I built an entire data/model pipelines end-to-end for a project at Google. Doing so, gave me the benefit of ↓ 1. Increased ownership 2. More flexibility in customizing solutions 3. Quicker fixes when data quality goes haywire So, if you are up for it, learn these 7 ↓ 1. SQL 2. ETL Process 2. Data Modeling 3. Schema Designs 4. Data Warehousing (e.g. OLAP) 5. Big Data Technologies (e.g. Spark) 6. Database Scalability (e.g. Sharding) 7. Workflow Orchestration (e.g. Airflow) ► Curious, do you think data scientist should learn data engineering? Drop your opinion ↓

Logistic regression is a must-know model for data scientists. Here's what you should know in 7 steps ↓ 1️⃣ 𝗢𝘃𝗲𝗿𝘃𝗶𝗲𝘄 It's a go-to model for predicting binary outcomes (e.g. yes/no, spam/ham) 2️⃣ 𝗠𝗮𝘁𝗵 It uses a sigmoid function to fit an S-shaped curve on binary outcomes. This transformation expresses the outcomes in probability form. 3️⃣ 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 Maximum likelihood estimation is used to fit the model on data. The performance is evaluated using metrics like F1-score and ROC-AUC. 4️⃣ 𝗜𝗻𝘁𝗲𝗿𝗽𝗿𝗲𝘁𝗮𝘁𝗶𝗼𝗻 Interpret predictors in odds-ratio (OR). ↳ OR > 1: One unit increase in X increases the odds of an event ↳ OR < 1: One unit increase in X decreases the odds of an event ↳ OR = 1: Change in X has no effect on the odds of an event Interpret significance and uncertainty using P-value and CI of the predictor. 5️⃣ 𝗔𝘀𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻𝘀 → Independent observations → No multicollinearity among predictors → Linear relationship between the logit of the outcome and predictors 6️⃣ 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 ↳ Marketing: Predict customer churn ↳ Finance: Predict credit risk ↳ Healthcare: Predict disease 7️⃣ 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 Questions I've encountered in interviews and also from my client's interview experiences: 1. What is the logistic regression model? 2. How do you interpret the logistic regression model? 3. What's the difference between the linear and logistic models? 4. How do you interpret the confidence interval of a logistic regression model? Ace upcoming interviews with these👇 📕 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗲𝗽 𝗖𝗼𝘂𝗿𝘀𝗲𝘀: https://lnkd.in/gzgB-dHT 📘 𝗝𝗼𝗶𝗻 𝗗𝗦 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/eiA5Ntdp 📙 𝗝𝗼𝗶𝗻 𝗠𝗟𝗘 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/e6HbN6dy 📗 𝗔𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗖𝗼𝘂𝗿𝘀𝗲: https://lnkd.in/g82dMJ77 👉 Smash 👍 and follow Daniel Lee to land dream data & AI jobs 🚀

Offend a data scientist in one line ↓ — The first to get 100 likes or funny reactions wins brownie points!

Anyone claiming “RAG is dead” doesn’t seem to understand building a practical AI solution. Is it really? Let’s put this into perspective. GPT4o has a context limit of 128K input tokens, about 17K words (or 4 chapters of a typical fiction novel). Do you really need 4 chapters of data to answer a simple customer service question? Most likely, it’s an overkill. The cost also is not worth it. 128K tokens cost about $0.64 per input (based on the 4o’s rate of $5 per 1M input tokens). Factor in scale: 10 questions per customer x 10K daily active users x 30 days… That’s a whopping… ~$1.9 million OpenAPI bill per month. This doesn’t include the output token, hosting cost, additional API calls if the AI service involves multiple agents, and so on. Unless your client is ready to splurge, good luck getting the project signed off. Instead, use RAG and save money. Get the context you really need to answer the customer’s question. Sure, there are costs involved in operating RAG (e.g. embedding, Vectorstore - write, store, read). But, the total cost is nowhere near the $1.9M fine your client could be paying. ↳ What’s your thought? RAG or no RAG? Let’s hash it out ↓ -- 👉 Ace upcoming interviews with 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗲𝗽 𝗖𝗼𝘂𝗿𝘀𝗲𝘀: https://lnkd.in/gzgB-dHT 👉 Smash 👍 and follow Daniel Lee to land dream data & AI jobs

Got a data scientist interview this month? Here are 10 resources for prep👇 1. AB testing by Googler: https://lnkd.in/e-TS4mMf 2. Meta technical screen: https://lnkd.in/e2_indTt 3. Applied statistics question: https://lnkd.in/eWFYqC_k 4. Product sense question: https://lnkd.in/eeb8Pexd 5. 100 Core concepts in data science interview: https://lnkd.in/e__t2biu 6. 100 FAANG-style SQL questions: https://lnkd.in/drpiU88m 7. AB testing course with Python: https://lnkd.in/dZ3wcnti 8. Interview classes with Google, Tesla, Meta DS: https://lnkd.in/eBbgMcwF 9. Opportunity sizing case: https://lnkd.in/eibpBKBD 10. Data science coding question: https://lnkd.in/evRXznnM Ace upcoming interviews with these👇 📕 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗲𝗽 𝗖𝗼𝘂𝗿𝘀𝗲𝘀: https://lnkd.in/gzgB-dHT 📘 𝗝𝗼𝗶𝗻 𝗗𝗦 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/eiA5Ntdp 📙 𝗝𝗼𝗶𝗻 𝗠𝗟𝗘 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/e6HbN6dy 📗 𝗔𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗖𝗼𝘂𝗿𝘀𝗲: https://lnkd.in/g82dMJ77 👉 Smash 👍 and follow Daniel Lee to land dream data & AI jobs 🚀

When a friend asks me how machine learning works, I say: — 👉 Your turn. What do you say when your friend asks how machine learning works?👇 -- Ace upcoming interviews with these👇 📕 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗲𝗽 𝗖𝗼𝘂𝗿𝘀𝗲𝘀: https://lnkd.in/gzgB-dHT 📘 𝗝𝗼𝗶𝗻 𝗗𝗦 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/eiA5Ntdp 📙 𝗝𝗼𝗶𝗻 𝗠𝗟𝗘 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/e6HbN6dy 📗 𝗔𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗖𝗼𝘂𝗿𝘀𝗲: https://lnkd.in/g82dMJ77 👉 Smash 👍 and follow Daniel Lee to land dream data & AI jobs 🚀

Transitioning from ML to LLM is not hard when you learn these 5 patterns. ↓ 𝟭. 𝗟𝗮𝗯𝗲𝗹𝘀 → 𝗧𝗲𝘅𝘁𝘀 In ML, you need a dataset with input signals and output labels. In LLM, your dataset consists of prompts as the input and example texts (sometimes written by humans) as the output. 𝟮. 𝗠𝗼𝗱𝗲𝗹 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 → 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 In ML, you often train a model from scratch. In LLM, you start with pre-trained models like GPT, Claude, LLAMA, or Gemini. You may use it out of the box or fine-tune it on your dataset. 𝟯. 𝗙𝗲𝗮𝘁𝘂𝗿𝗲 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 → 𝗣𝗿𝗼𝗺𝗽𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 In ML, you largely improve model accuracy with features like one-hot encoding. In LLM, you largely enhance text output quality with prompt techniques like K-shot learning. 𝟰. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 → 𝗘𝘃𝗮𝗹𝘀 In ML, you evaluate using cross-validation on metrics like AUC or MSE. In LLM, you evaluate using custom evals like Semantic Similarity, LLM as the judge, or humans-in-the-loop. 𝟱. 𝗠𝗟 𝗙𝗹𝗼𝘄 → 𝗟𝗮𝗻𝗴𝗦𝗺𝗶𝘁𝗵 You can operationalize ML using MLFlow. In LLM, you can operationalize using LangSmith (or LangFuse). -- Ace upcoming interviews with these👇 📕 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗲𝗽 𝗖𝗼𝘂𝗿𝘀𝗲𝘀: https://lnkd.in/gzgB-dHT 📘 𝗝𝗼𝗶𝗻 𝗗𝗦 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/eiA5Ntdp 📙 𝗝𝗼𝗶𝗻 𝗠𝗟𝗘 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/e6HbN6dy 📗 𝗔𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗖𝗼𝘂𝗿𝘀𝗲: https://lnkd.in/g82dMJ77 👉 Smash 👍 and follow Daniel Lee to land dream data & AI jobs 🚀

Your project will never succeed if you tell stakeholders that your model achieved "0.94 AUC". Here's why 👇 As an ML engineer, data scientist, or product analyst, if you tell your business stakeholder that your project is successful because: → "NDCG is 0.94" → "AUC is 0.94" → "P-value is 0.03" ... it won't convince them that your deliverable provides ACTUAL IMPACT. You must translate this jargon into metric to convey BUSINESS VALUE. This varies by domain—ads, marketing, etc. 👉 Last chance to 𝗝𝗼𝗶𝗻 the 𝗠𝗟𝗘 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽 (https://lnkd.in/e6HbN6dy). The next cohort led by MLEs at FAANGs starts next week! Ace upcoming interviews with these👇 📕 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗲𝗽 𝗖𝗼𝘂𝗿𝘀𝗲𝘀: https://lnkd.in/gzgB-dHT 📘 𝗗𝗦 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/eiA5Ntdp 📗 𝗔𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗖𝗼𝘂𝗿𝘀𝗲: https://lnkd.in/g82dMJ77 👉 Smash 👍 and follow Daniel Lee to land dream data & AI jobs 🚀

How Zillow lost $500 million because of machine learning ↓ Machine learning is not a magic bullet. Zillow's blunder with their iBuying business is a classic case study. From 2018 to 2021, Zillow aimed to use machine learning to predict home prices, buy properties directly, and sell them for profit. At its peak, 9,680 homes were in their inventory. The result? Zillow closed the program with a huge loss of $500 million. The costs involved in acquisition and renovation, coupled with market volatility, were much higher than anticipated. Not only that, Zillow's ML model was too optimistic, leading to overpayment for properties. A story like this teaches us the following: ⤷ If the best case is a gain, and the worst case is also a "gain." That's a big red flag. ⤷ ML is not the magic bullet when the underlying business model is flawed. This is particularly true today with the "AI" hype. Does a toilet really need to be powered by AI? ⤷ Curious, what's a takeaway for you? ↓ — Ace interviews with these👇 📕 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗲𝗽 𝗖𝗼𝘂𝗿𝘀𝗲𝘀: https://lnkd.in/gzgB-dHT 📘 𝗝𝗼𝗶𝗻 𝗗𝗦 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/eiA5Ntdp 📙 𝗝𝗼𝗶𝗻 𝗠𝗟𝗘 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/e6HbN6dy 📗 𝗔𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗖𝗼𝘂𝗿𝘀𝗲: https://lnkd.in/g82dMJ77 👉 Smash 👍 and follow Daniel Lee to land dream data & AI jobs 🚀

Become 10% better at machine learning by learning to write project code like a software engineer ↓ I stopped jump-starting an ML project on a Jupyter when I realized: ❌ It discourages code reusability. ❌ It is difficult to do version control properly. ❌ Variables are cached globally, leading to data leakage. Now, I start an ML project by first setting up a project directory like this! ↓ ✅ 𝗖𝗿𝗲𝗮𝘁𝗲 𝘀𝘂𝗯-𝗱𝗶𝗿𝗲𝗰𝘁𝗼𝗿𝗶𝗲𝘀: ↳ data/ # Your CSV and JSON files reside here ↳ notebook/ # Your ipynb files reside here ↳ src/ # Place your ML functions/objects in src ↳ tests/ # Unit tests your code in src ↳ main.py # Glues all the operations of training a model ✅ 𝗗𝗼𝗻'𝘁 𝗹𝗲𝘁 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀 𝗹𝗶𝗻𝗴𝗲𝗿 𝗶𝗻 𝗻𝗼𝘁𝗲𝗯𝗼𝗼𝗸𝘀. 𝗠𝗼𝗱𝘂𝗹𝗮𝗿𝗶𝘇𝗲 𝗶𝘁 𝗶𝗻 𝘀𝗿𝗰! Remember the key steps to modeling - extracting and preprocessing data, performing EDA, building features, training the model, and evaluating. Each step often involves custom code or wrappers on top of existing libs like Sklearn. Modularize it in src/, then import it in notebooks and the main file. ✅ 𝗗𝗼𝗻'𝘁 𝗽𝘂𝘀𝗵 𝗰𝗼𝗱𝗲 𝘁𝗼 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝘂𝗻𝗶𝘁 𝘁𝗲𝘀𝘁𝗶𝗻𝗴. Unit testing ensures that your code is reliable. It outputs what it's supposed to do. Whenever I submit a code change to production, I run Pytest to ensure nothing breaks! 👉 What else should ML practitioners know to become 10% better? Drop one👇 Ace upcoming interviews with these👇 📕 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗲𝗽 𝗖𝗼𝘂𝗿𝘀𝗲𝘀: https://lnkd.in/gzgB-dHT 📘 𝗝𝗼𝗶𝗻 𝗗𝗦 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/eiA5Ntdp 📙 𝗝𝗼𝗶𝗻 𝗠𝗟𝗘 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽: https://lnkd.in/e6HbN6dy 📗 𝗔𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗖𝗼𝘂𝗿𝘀𝗲: https://lnkd.in/g82dMJ77 👉 Smash 👍 and follow Daniel Lee to land dream data & AI jobs 🚀