Myra Deng

2 things: - become FAANG engineer - make 3,251xxx+ per year Both goals are worth admiration but not tied to each other. Your income not always tied to your qualifications and you may be prone to survivor bias. We may never know the stories of those other amazing engineers with same traits that did not get there.

14

1 Comment

What society values may not be the best thing for you. When I applied for grad school, I got admits from MIT, Stanford, Purdue, CMU and UIUC. I knew I wanted to do a PhD. I chose MIT immediately, because it was ranked number 1. I never thought whether it was a good decision or not. I just assumed that it’s a good decision because it’s rated higher than the other universities. When I joined MIT, all that rank 1 noise disappeared. All I was surrounded by was my lab, my lab group members and my advisor. I had never carefully considered whether this research group was a good fit for me. I just joined the first MIT research group which came my way. This led to a miserable first 2 years of PhD for me. The group was not a good fit and the research topic was also not a good fit. These lost 2 years were a result of choosing what society values more (university ranking) rather than what is a better thing for me. When I look back, I should have done the following: (1) I should have made a list of all research groups I was interested in, from all universities I was selected to. (2) I should have spoken with the lab advisor from all these research groups and understood their vision. (3) I should have learnt more about the group culture by talking to past members of the group. Graduating members are usually very honest. They would tell you if the culture is toxic, if the advisor is micromanaging etc. (4) I should have looked at the group website to look at the publication rate, what do the alumni end up doing etc? If the average graduation time is 8 years, it’s a red flag. After doing the above 4 things, I should have selected the best research group for me. I should have selected the research group whose culture aligns well with my personality. Society values ranking. However, when you enter the university for a PhD, the ranking does not matter. Your research group matters. What you do every day after waking up matters. And if you blindly select universities based on the ranking, the research group won’t be a good fit and it would lead to a terrible PhD experience. It turns out that society misguides people in many other professions including: - Choosing to prepare for IIT-JEE just because society prefers that you are an IITian - Choosing a job just because society prefers higher salary - Choosing to not do a startup because the society (your parents) don’t approve Always question yourself before blindly making decisions based on what the society values. It might not be the best thing for you. Going against traditional societal values is hard in the short term, but it pays off in the long run. P.S: In the photo, you can see me during my first day at MIT!

310

6 Comments

Do large language models have an in-built notion of "common sense"? 👀 This is the driving question behind my research with the USC Information Sciences Institute this summer as part of the "AI & Complex Systems" group. Using methods grounded in cognitive science and computational linguistics, we’re aiming to conduct a holistic analysis of LLMs that bridges the gap between theoretical computer science and the social sciences. Our preliminary investigations indicate that LLMs have a strong, but not infallible, grasp of "topical regularity"—that is, understanding how different concepts are related to one another. This is probably unsurprising if you’ve played around with #ChatGPT; the majority of what it outputs usually seems to make sense, regardless of whether it’s correct. This understanding of relations between topics could serve as a basis for general reasoning abilities, much like how humans are able to reason using their general understanding of the world—in other words, "common sense". 🧠 Further investigations will lead us to narrow in on whether these LLMs have genuine abstract reasoning capabilities, or if they're using semantic markers (like "IsA", "HasA", "PartOf") as "hints" that enable them to perform well. Stay tuned for further updates on this project and for more #cognitivescience and #artificialintelligence content! . . . #AI #NLP #LLM #research #innovation #computerscience #linguistics #psychology #USC #academia

77

8 Comments

Today, I found a fascinating way to think about “Tokens” in LLMs. It came from a conversation between Jensen Huang and Patrick Collison at a recent Stripe event. When asked about the future of compute capacity in five years, Jensen smartly dodged that question a bit, but then gave an excellent explanation and analogy about why tokens are the new forces that will power the next decades of humanity. He explained that we're now producing something unprecedented (and at scale): floating point numbers that possess “value”, which we now call tokens. These tokens are valuable because they encapsulate “intelligence”. People are now taking these tokens and transforming them into English, French, images, videos, chemicals, proteins, robotic movements, etc. And many are working hard to expand the range of concepts and ideas we can create with these tokens. He then goes on to made a compelling comparison between tokens and electricity. In the past industrial revolution, we successfully found a way to convert “atoms” into “electrons” (by boiling water to power electricity turbines). And now, we've discovered a way to convert "electrons" into "tokens" (by using energy to power data centers that train and run LLMs). When electricity was first introduced, few people understood its value. Today, paying for kilowatts is routine. The same will happen with tokens. Right now, only early adopters and builders are paying tokens. Soon, everyone will be paying for tokens on a daily basis to supercharge productivity and power new products and services. Many new industries will be born from and built on top of tokens. When I first heard about this way of thinking, I got goosebumps. Perhaps it’s because Jensen has a good storytelling skill that helps him sell what he builds, but I have to admit this way of thinking gave me a brief surge of pride. I'm proud of humanity's collective effort and creativity, which have taken us from living under rocks to building machines that can "think." That's nothing short but a miracle.

18

5 Comments

I’m thrilled to announce the launch of my new Substack, Silicon Sand Stack, with my very first article: "Today's Large Language Models Will Never Be Responsible, Safe, or Green: The Need for Investment in Revolutionary New AI Paradigms" In this debut piece, I dig into the critical flaws of current large language models (LLMs) and why they fall short of being responsible, safe, or environmentally friendly. I examine the inherent issues like hallucinations, lack of real-world grounding, unreliable reasoning, and potential biases that plague these models. 🔍 Key insights include: - The technical and ethical limitations of today's LLMs. - The necessity for radically new AI paradigms. - The role of venture capital in fostering innovative AI approaches that prioritize safety, responsibility, and sustainability. Join us as we embark on a journey to rethink AI and advocate for groundbreaking advancements that align with human values and societal needs. Read the full article and subscribe here: https://lnkd.in/eDcTniTg #ArtificialIntelligence #AI #GenerativeAI #TechInnovation #ResponsibleAI #GreenAI #FutureOfAI #SiliconSandStudio

46

1 Comment

A start in evaluating Responsible AI - when LLMs are used for real life applications such as generating first drafts of student testimonials (Appraiser, one of my products!), how much bias is there? Thanks to this team and Resaro for pioneering such efforts within the Singapore government!

17

I've been in my Gemini Era! #Geminibaby But I'm not just talking about being born in the most incredible month ever #June. I'm referring to Google's Gemini AI product. I wanted to share with my network the top 3 ways I use Gemini to help save me time, make life easier and even get discounts or gift cards when I've had a good or bad customer service experience. 1. Did you know Gemini can help draft customer service emails? Have you ever wanted to contact corporate, but felt you didn't have the time nor energy to articulate your frustration? In less than 3 mins. you can craft an email to offer feedback on if your food was cold, a flight was delayed, or even if a waiter spilled your espresso martini. 🍸 (true story) 2. Had an amazing hotel 🏨 experience? Recently I visited Nobu Los Cabos, and everything from the staff, service, food and coffee were amazing. They reached out several times asking for a review. I input everything I loved about my experience and voila! I had 3 paragraphs of goodness to share on hotels .com and a Google review. I was able to help amazing staff get recognized for my positive experience. 3. Need help with a quick day trip itinerary? While traveling, I like to see as much as possible. One of my personality traits is #maximizer. On a recent trip to Europe, I wanted to do a day trip to 🇨🇭 Switzerland. I input top things to see in 24 hours into the #Gemini search bar and it was able to flush me out a full itinerary. Mapped all of the places in Google maps - and started knocking things off my To Do list. I had fun, my plan was efficient, and even got to see the amazing views of the Grossmünster church. So go on and try it out. I know some folks are weary of how AI will affect our futures, but I wanted to share my experience on how it very much helps me in my regular day to day life. I hope these tips help someone! P.S. Be sure to catch the Google I/O in 10 mins. recap. https://lnkd.in/d4GB6UtH

13

1 Comment

Whose Weights and Biases? up on my blog: https://lnkd.in/gUU_t8_u Gwendolyn Gillingham Thank you for your feedback and help with editing this blog. On May 13, Open AI launched GPT-4o, their new flagship model that can reason across audio, vision, and text in real time and the next day on May 15 the scientists leading superalignment team at OpenAI, Ilya Sutskever and Jan Leike, departed from the company. While I am impressed by the future of human-tech interactions, I can’t help but wonder what influence Open AI and enterprises alike will have over the information the modern society consumes. In this opinion piece, I discuss the bias AI models encode and how it impacts society. I also briefly talk about how firms like Open AI, Meta, Google and Anthropic are taking ethics and safety into considerations when training and deploying AI models. Drawing from the recent research and news, I evaluate how the narrow lens of global tech might still fall short in developing AI for all. Finally, I discuss AI regulations around the world and touch a little upon how the global south is behind in terms of making the best use of AI systems. Read more at #icodeformyभाषा and feel free to leave your feedback! #bias #aigorvernance

28

2 Comments

I just published my bi-monthly update on what I've been working on for the last couple months. Over the course of the last 2 months I wrapped up ML research at the University of Toronto, started my summer internship at Immigration, Refugees and Citizenship Canada / Immigration, Réfugiés et Citoyenneté Canada, working with transformers, building a neural network for chess, wrapping up my sophomore year at UofT, & more! I go over everything that I've learnt along the way and some important reflections / lessons I took away from the entire process. Here's a quick tl;dr: 1. Wrapped up ML research with PointNet model 2. Built a neural network that can analyze the next best chess move 3. Deep dive into Transformers 3. Started my summer internship @ IRCC 4. Wrapped up my sophomore year @ UofT 5. Working with Aercoustics to develop ML models for sound classification If you want to check out what I've been up to, click the link below! 😁

31

AI at Meta has released the most powerful open-source model yet as of today: 𝑳𝑳𝑨𝑴𝑨3 - 8B and 70B models at 8k context length! They highlight improvements in each of the following aspects as the key differentiator: a) Model Architecture b) Pretraining data c) scaling up pre-training d) Instruction fine-tuning 💥 𝐏𝐫𝐞𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 & 𝐒𝐜𝐚𝐥𝐢𝐧𝐠 𝐮𝐩: ✈️ Vocab of 128k tokens & GQA across both 8B and 70B, 8k native context length ✈️ Pretrained over 15T tokens, 4 times more code than LLAMA2 ✈️ Multi-lingual : over 5% of pretraining data consists of high-quality non English data that covers over 30 languages: don't expect same performance as English ✈️ LLAMA2 generates training data (like in Self-Instruct) to generate text classifiers that filter data powering LLAMA3 ✈️ Both 8B and 70B improve even after the model is trained on 2 orders of magnitude more data i.e log-linear improvement after 15T tokens training ✈️ Training runs on 2 custom-built 24k GPU clusters with an overall training time efficiency of 95% ✈️ 𝘖𝘷𝘦𝘳𝘢𝘭𝘭, 3 𝘵𝘪𝘮𝘦𝘴 𝘮𝘰𝘳𝘦 𝘦𝘧𝘧𝘪𝘤𝘪𝘦𝘯𝘵 𝘵𝘩𝘢𝘯 𝘓𝘓𝘈𝘔𝘈2 💥 𝑰𝒏𝒔𝒕𝒓𝒖𝒄𝒕𝒊𝒐𝒏 𝑭𝒊𝒏𝒆𝒕𝒖𝒏𝒊𝒏𝒈: ✈️ Combination of SFT + PPO + DPO + Rejection Sampling ✈️ 𝑸𝒖𝒂𝒍𝒊𝒕𝒚 𝒐𝒇 𝒑𝒓𝒐𝒎𝒑𝒕𝒔 𝒖𝒔𝒆𝒅 𝒊𝒏 𝑺𝑭𝑻, 𝒑𝒓𝒆𝒇𝒆𝒓𝒆𝒏𝒄𝒆 𝒓𝒂𝒏𝒌𝒊𝒏𝒈𝒔 𝒖𝒔𝒆𝒅 𝒊𝒏 𝑫𝑷𝑶, 𝑷𝑷𝑶 𝒉𝒂𝒔 𝒂𝒏 𝒐𝒖𝒕𝒔𝒊𝒛𝒆𝒅 𝒊𝒎𝒑𝒂𝒄𝒕 𝒐𝒏 𝒎𝒐𝒅𝒆𝒍 𝒑𝒆𝒓𝒇𝒐𝒓𝒎𝒂𝒏𝒄𝒆 ✈️ Careful data curation and multiple rounds of QA on annotations from human annotators ✈️Training on preference ranking improved the model performance greatly on coding and reasoning tasks 💥 𝑳𝑳𝑨𝑴𝑨3 𝑮𝒂𝒖𝒓𝒅𝒓𝒂𝒊𝒍𝒔: ✈️ Updated and new safety tools : LLAMA Gaurd2 & cybersecEval 2 ✈️ CodeShield - inference time gaurdrail for filtering insecure code ✈️ Instruction Fine-tuned model redteamed for safety : By generating adverserial prompts that try to elicit problematic responses. ✈️ LLAMAGaurd - foundational for prompt and response safety, can be easily finetuned to create a new taxonomy 💥 𝑰𝒏𝒇𝒆𝒓𝒆𝒏𝒄𝒆 & 𝑭𝒖𝒕𝒖𝒓𝒆 ✈️ Despite LLAMA3 having 1B more params than LLAMA2 7B, the improved tokenizer efficiency and GQA contribute to maintaining the inference efficiency on par with Llama 2 7B. ✈️ 400B+ parameters, Multilingual, Multimodal, and longer context window LLAMA3 would be available on AWS soon. Try it out! #generativeAI #llm #llama3 #aws #bedrock #sota

33

1 Comment

𝕎𝕙𝕪 #𝔸𝕀 𝕗𝕠𝕝𝕜𝕤 𝕟𝕖𝕖𝕕 𝕒 𝕓𝕣𝕠𝕒𝕕 𝕓𝕒𝕤𝕖𝕕 𝕀𝕟𝕥𝕣𝕠 𝕥𝕠 #𝔸𝕀 👉 As I go around giving talks/tutorials on the planning and reasoning abilities of LLMs, I am constantly surprised at the rather narrow ML-centric background of grad students/young researchers have about #AI. This seems to be especially the case with those who think LLMs are already doing planning and reasoning etc. Most of them don't seem to know much about the many topics that are taught in a broad-based Intro to #AI course--such as combinatorial search, logic, CSP, difference between inductive vs. deductive reasoning (aka learning vs. inference), soundness vs. completeness of inference/reasoning etc. I can understand why a strong background in ML and DL is sine qua non these days in using/applying the current #AI technology. That doesn't however mean that other things, that are typically not covered in ML courses, but are covered in Intro #AI courses, are expendable. If you don't know those concepts, you are more likely than not to re-invent crooked wheels (see this for examples of how people get tripped up: https://lnkd.in/gUPPb7s4) All this is particularly relevant for those busy building empirical scaffolds over LLMs (the "LLMs are Zero-shot <XXX>" variety). Most often, these young researchers are coming from NLP. At one point, NLP used to be NLU and students had quite a firm grasp of logic (e..g Montague Semantics!). But over the years, NLU became NLP which in turn has become Applied Machine Learning, and students don't quite have the background in logic/reasoning etc. Now that LLMs have basically "solved" the "processing" tasks--such as information extraction, format conversion etc., NLP folks are trying to turn to reasoning tasks--but often lack the necessary background. (See this unsolicited advice to NLP students: https://lnkd.in/gKTdsH2P) Background in the standard Intro AI topics like search/CSP/logic are useful even if you don't plan on directly using those techniques (e.g. because you want differentiable everything to make use of your SGD hammer). Like MDPs, they provide a normative basis for many deeper reasoning tasks AI systems would have to carry out when they broaden their scope beyond statistical learning. Without that background, you will likely try to pigeon hole everything into "in/out of distribution" framework, when what you need to think of is "in/out of deductive/knowledge closure; see https://lnkd.in/gTWVibdt ) One of the other things that you get exposed to in the standard Intro #AI is computational complexity of the various reasoning tasks. People who jumped in directly via applied ML might understand a bit of sample complexity (maybe?), but are not that attuned to reasoning complexity. (Contd. in the comment below)

374

46 Comments

Here's the technical breakdown of the AI Penalty Challenge authored by Alok Pattani (check out this video if you haven't seen the demo yet: https://lnkd.in/gsD3xjJ2) Key Takeaways: ☁️ Firestore for triggering slow-mo video capture on Android phones & uploading to Google Cloud via Firebase Storage. 🧠 Cloud AutoML for training custom object detectors (ball & goal tracking) to calculate speed & accuracy. 🗣️ Gemini Pro 1.5 on Vertex AI for personalized coaching feedback based on multiple data points. 🎨 Imagen on Vertex AI for instant, prompt-driven image editing for cool player pics. More technical insights from the project to come... #AIChallenge #MachineLearning #SportsTech #GoogleCloud #footballNOTsoccer

90

6 Comments

O-1 (and EB-2 NIW) for Software Engineers. Regarding GitHub contributions. The temporary visa category of O-1 is rightfully getting more attention given the challenges of the H-1B cap. More F-1 students are looking at an O-1 as a possible alternative to H-1B. This is particularly important for Indian and Chinese born immigrants who cannot realistically self-petition for a green card and get any work authorization given the long waits of the Visa Bulletin cut-off dates. An O-1 requires employer sponsorship and employers sometimes dissemble in their consideration of O-1. This is particularly true at large employers who use certain law firms that are really efficient at pushing everyone into H-1B and PERM. One mistake I see is that people wait until AFTER losing the H-1B cap lottery their last time and then scrambling to try for O-1. In my view this is 'too late' - and what you want to be doing is preparing your O-1 profile (and maybe explicitly your O-1 case) during your F-1 OPT and STEM OPT period so if you are unlucky in the H-1B cap lottery, you are ready to go for O-1. A lot of F-1 students study software engineering and data science, which were probably the right fields to choose 10 years ago and maybe 5 years ago but maybe not the right fields for the 2024 job market. Besides the challenging job market, software engineers and data scientists often face headwinds in trying to get approved for O-1 as their fields are not necessarily well-suited for the O-1 criteria. [A link to the official USCIS policy manual for O-1 is in the first comment below]. Contributions to GitHub can possibly be one way to demonstrate subject matter expertise, of relevance to O-1 but also EB-2 NIW and potentially even EB-1A. In general the challenge faced by software engineers and data scientists and data analysts is that they are hyper-focused on making their employers more efficient and profitable. Those are very useful activities for any organization but those things may not necessarily be useful in meeting the O-1 criteria, or showcasing an endeavor of national importance for EB-2 NIW. Besides 'waiting too long' on F-1 OPT and STEM OPT, and besides 'studying the wrong thing' (which no one really wants to hear) -- the other mistake I am seeing is people moving into Day 1 CPT and thinking this solves a problem. Day 1 CPT is a band-aid. It buys you another chance at the lottery but next year's odds in the lottery will also be bad. What people need to be doing is using their time to get out of the H-1B cap lottery nonsense and the only way to do that (other than moving abroad and returning on L-1, or maybe marrying a US citizen) is trying to build a profile for O-1. I do think Day 1 CPT is a possible short-term solution for people but way too many people are pursuing it now and the odds of winning the H-1B cap lottery next year will be as bad or worse than this year. You need to do something much more substantial. ###

120

11 Comments

Let's take a moment to celebrate failures and near-death experiences. Paul Graham (Co-Founder of Y Combinator) best describes startups as a mosquito - all speed and attack but 1 swat away from death Recontact has had 6 near-death experiences before the current moment, where we now have early customers and revenue. Paul Graham also says - be like a "cockroach" - just staying annoyingly persistent and hard-to-kill So that's what I did and here's what I learnt: 1. Be selling your story to everyone, ALL the time 2. Pick cofounders from smart friends from your past 3. Ask for money UPFRONT, before building, based on basic mockups 4. Solving your own problem is the fastest way to get good feedback 5. Grit is rare, just survive 6. Ask for help shamelessly. Be desperate with risks. If you want to read all the stories of how we got to these learnings, with each failed launch attempt - checkout - https://lnkd.in/ggs4bbSt Couldn't have done all this without my co-conspirators Jayanth Krishnaprakash and Abhishek Gupta as well as our early supporters Shantha Mohan Ph.D. Neel Balar Prajwal Prakash Akash Anand Saksham K. Guillermo Gutierrez Ananya Madhav Nancy Zheng David A. Thomas Safwaan Khan Rohan Mukherji Sheryl Root Stuart Evans Gladys Mercier Still lots to come and learn but hope that our story urges others like us, to just hang in there and endure till the promised land. Have a near death story you wanna share with us? Drop it in the comments and what you learnt from it. #entrepreneurship #startups #failure

73

11 Comments

Today's "Interesting person I discovered via RSS and decided to follow on LinkedIn" is William Carter. 'Carter, a UC Berkeley Ph.D. candidate in geography, was mining these centuries-old slave ship logs in 2020 as part of his research into the transatlantic slave trade and what lessons from then might apply to our own understandings about race, literacy and power today. But there was a problem: He couldn’t read a single sentence. For even the most skilled readers, the faded, swooping lettering would be nearly indecipherable. But it was especially inaccessible for Carter, who has dyslexia and whose screen reader tools for written notes and published books didn’t work on 400-year-old slave ship logs. So in collaboration with UC Berkeley’s Disabled Students’ Program (DSP), Carter created a first-of-its-kind research solution to allow him to read these centuries-old materials. Carter finds logbooks and takes digital photos of the pages, which he then shares with DSP’s accessibility experts. That team works with a company to digitize the files in a layout mirroring the original documents, almost as if a deckhand in 1691 pulled out a typewriter or opened a Google Sheet. The result is a file that Carter’s screen reader can interpret.' https://lnkd.in/eHXZAVcm

1

Longtermism, often via AI safety, has a big voice in the AI community—I spoke with David Thorstad, Professor at Vanderbilt, who interrogates what longtermists can really claim to know about the future and their impact on it. We also spoke about his important bounded rationality program—much of this from his upcoming book, Inquiry Under Bounds.

4

1 Comment

𝒀𝒐𝒈𝒂 𝑨𝒔𝒔𝒊𝒔𝒕𝒂𝒏𝒕 𝑪𝒉𝒂𝒕𝒃𝒐𝒕 𝑷𝒂𝒓𝒕 2: 𝑭𝒊𝒏𝒆-𝒕𝒖𝒏𝒊𝒏𝒈 𝑳𝑳𝑴𝒔 Although I initially planned to update this series weekly, it seems more realistic to do so every 2-3 weeks 😬. In Part 1, I introduced how to fine-tune image models effectively in 2024. Now, in Part 2, I'm excited to cover the fine-tuning of LLMs, using the recent release of Llama v3 with the torchtune framework as an example. We've got a fun mix of coding tutorials, data prep tips, and some in-depth explanations. Grab a cup of coffee, get comfy, and check it out! Github: https://lnkd.in/g5Wg3zXr 瑜伽聊天機器人系列第二篇，介紹如何準備自己的資料集來微調LLMs，同時整理了常見的微調方法、原理與好用的Libraries，完整中文內容可以參考： Medium: https://lnkd.in/gn9tRmdg

11

4 Comments