Dr. Rumman Chowdhury

I think we’re going to start seeing a lot more “less-is-more” approaches to models - for this reason, as well as energy consumption, computational centralization, and general cost-efficiency. At some point, in any analysis, data and parameters introduce noise to computation.

5

A new study, titled "Risk Scores, Label Bias, and Everything But the Kitchen Sink," by Julian Nyarko, Professor at Stanford Law School, an Associate Director and Center Fellow at Stanford Institute for Human-Centered Artificial Intelligence (HAI) together with colleagues from Stanford University and Harvard University, explores the challenges in risk prediction models, where using too much data can lead to inaccuracies due to “label bias” where the models use inapt proxies instead of measuring the real outcomes of interest. Risk prediction models are used in various sectors like lending, college admissions, and healthcare, to forecast potential outcomes based on data. These models often follow a "kitchen sink" approach, where the assumption is that more data will yield better predictions. The study shows that this approach might not be ideal because it can lead to what's called "label bias." Label bias happens when models use proxies—a substitute measurement used when direct measurements are unavailable. The study suggests a simpler, less-is-more approach with models using fewer data points could improve both the accuracy and fairness of these predictions. This approach prioritizes direct and relevant data over a broad array of less pertinent information, leading to models that are not only easier to understand but potentially more just in their outcomes. For example, arrest rates might be used as a proxy for criminal behavior. However, these proxies might not accurately represent the true outcome they are supposed to measure. This can lead to inaccuracies; for instance, if arrest rates are higher in a particular area, it might not truly reflect the individuals' likelihood to commit crimes but rather the intensity of police activity in that area. In the medical field, a similar issue arises when predicted future medical costs are used to determine which patients should enter high-need care programs. This can disadvantage certain groups, like Black patients, who might have equal medical needs but lower predicted costs because they seek less medical treatment. Applying this insight to college admissions, the findings suggest that admissions models should avoid overly complex algorithms that incorporate a wide range of data points, many of which may not be directly relevant to predicting student success. E.g., instead of considering a vast array of factors like family background, high school attended, and various extracurricular activities, colleges might focus on a smaller set of predictors more directly linked to academic success, such as GPA, specific aptitude test scores, and relevant academic achievements. Article: https://lnkd.in/gpykKPA7 Study: https://lnkd.in/gPsXwaJu

226

19 Comments

Investigative work done by The Markup in partnership with a few other civil society organizations unearthed some egregious errors from a chatbot deployed by the local NYC government to aid citizens in finding answers to FAQs. An example of a clearly problematic outcome was answering questions like “Can I take a cut of my workers’ tips?“ with: “Yes, you can take a cut of your worker's tips. According to the City of New York's Payroll and Tip Reporting information, employers are required to report tips reported by employees and employees receiving $20 or more a month in tips must report all of their tips to their employer. Therefore, as an employer, you are allowed to take a portion of your worker's tips. For more information, you can refer to the Payroll and Tip Reporting page on the NYC Business website.” (You can read the full report from The Markup for more examples.) There are a whole slew of ethical issues that arise from such an incident such as: 1. Erosion of trust in the reliability of information provided in other instances by the chatbot, even when it might be right - also called algorithmic aversion. 2. Legal consequences due to wrong information whereby the person might act on the provided information that opens them up to lawsuits, e.g., in the workers’ tips case. 3. Harm to individuals and businesses, e.g., refusing to accept Section 8 vouchers or accepting tenants that receive rental income, i.e., discriminating against them, which is illegal for a landlord to do and would make finding housing difficult in an already problematic rental market such as NYC. Our quick take on small measures that would have helped to avoid this mishap proactively would be to: 1. Involve domain experts in the testing and validation phases of the chatbot development who could have spot-tested / smoke-tested responses to see if they went against housing regulations, workers’ rights, etc. 2. Rigorous testing and validation, especially constraining the system to rely on techniques like retrieval augmented generation (RAG) to sharpen answers and ground them in actual city laws that could be fed as a fine-tuning step into the chatbot’s AI system. 3. Transparency and user education could alert users more proactively that there is always an inherent risk in the chatbot providing responses that might not be accurate, or in this case, wrong to the point of being illegal, so users should proceed with caution and verify with second and third sources to confirm the validity of the response. While each of the steps require an investment upfront before the launch of the system for public consumption, the cost, we believe strongly, is well worth it to avoid harm and build trust capital with citizens, especially for essential services. #ethicalai #ai #ethics #aiethics #responsibleai https://lnkd.in/dAXzGqvP

9

Amazing when a community can convene, collaborate and move work forward on a topic as complex as open source & AI. I know this has been a community effort, but huge kudos to Camille François and Ayah Bdeir for the vision and drive to bring this group together to get things done. Read the paper! https://lnkd.in/ei3ecrXw

20

1 Comment

This week at #ICLR2024, #Microsoft #Research present #MathVista, an open-source #benchmark for measuring foundation models’ #mathematical #reasoning capabilities in #multimodal scenarios. Learn about the work now from coauthor Michel Galley in the “Abstracts” podcast. https://msft.it/6042YpuIk #AI #GenerativeAI #LLM #Mathematics

9

Are Chatbots Biased? Unpacking the Impact of Training on AI Political Preferences David Rozado's research paper, "The Political Preferences of LLMs," offers a revealing look at how large language models (LLMs) like ChatGPT, Google’s Gemini, and Twitter’s Grok manifest political biases. The study, encompassing 24 LLMs and employing 11 political orientation tests, found these AI systems predominantly lean towards left-of-center views. Key Findings - LLMs mostly fell into left-leaning quadrants across various political tests, including the Political Compass Test and Eysenck Political Test. - The research highlights a significant influence of post-training interactions (Supervised Fine Tuning and Reinforcement Learning) in shaping these political biases. Implications for Business Leaders - As AI becomes integral to business and consumer interactions, understanding and managing inherent biases in these systems is crucial. - The findings stress the need for a balanced approach in training AI models, to mitigate unintended skew in automated decision-making processes. Action Points 1. Review the AI systems your company employs for inherent biases. 2. Consider implementing diverse data sets and training scenarios to counteract bias. 3. Engage with AI ethics specialists to ensure your AI deployments are as neutral as possible. For CXOs and CDOs, the key takeaway is the profound influence of training data on AI behavior, emphasizing the necessity of strategic oversight in AI implementations to foster trust and fairness in automated systems. #AIethics #DigitalTransformation #LeadershipInAI #biases #responsibleai https://lnkd.in/gmfyqrWJ

3

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3

You're Invited to join our LLM Reading Group on May 28: Efficient RLHF for Large Language Models with Hakim Sidahmed of Google Research. Paper: Parameter Efficient Reinforcement Learning (PERL) with Low-Rank Adaptation (LoRA). Why it matters: Reduced costs of training large language models make Reinforcement Learning with Human Feedback more accessible, enhancing model alignment with human preferences. You will hear from the lead author, Hakim Sidahmed of Google Research. RSVP here: https://lnkd.in/gwzYG5p2 #ai #ml #rlhf #humanfeedback #opensource #opensourceAI #data

9

There are many LLM security papers out there that do something odd and get potentially harmful content out. But rarely is there any clue if this is against policy - ie., is unintended. Reviewers are left to judge in the dark what policy a given LLM was built on. It's a mess. Values and norms come from everywhere and vary everywhere, and without explicit policy known, it's impossible to judge whether that model's policy has been circumvented, or the difficulty/significance of doing so.  For example, if an LLM has no policy, all outputs are intended. It might be unsafe and a generally poor move to build and distribute it, but being able to get bad things out of a model with no policy is NOT a security problem.

97

13 Comments

In a new paper titled "Will we run out of data? Limits of LLM scaling based on human-generated data" researchers investigate the potential constraints on LLM scaling posed by the availability of public humangenerated text data. They forecast the growing demand for training data based on current trends and estimate the total stock of public human text data. Their findings indicate that if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032, or slightly earlier if models are overtrained. They explore how progress in language modeling can continue when human-generated text datasets cannot be scaled any further. They argue that synthetic data generation, transfer learning from data rich domains, and data efficiency improvements might support further progress. Paper: https://lnkd.in/gF-eH6Tg #llms #genai #generatieveai #capgemini #llm #opensourceai #capgeminiindia #ai #artificialintelligence #software #data #scalinglaws

12

2 Comments

Excellent report from General Dynamics Information Technology John P. Curran regarding adoption and trends of AI in the Govt. I used Meta to generate summary below and included link to original article. Main AI use cases are chat and creating/updating computer code. Note also the 23% failure rate of AI projects due to user expectation misalignment. This can be overcome with thorough protoyping and establishing use case metrics. - Successful AI project completion timelines are narrowing, with an average of 14 months from pilot to production. - Future demand for AI projects is building around chat and coding-related functions. - 54% of surveyed AI experts said projects currently under development involve improving computer code. - 59% of projects involve "research and understanding", which may relate to creating chat functions. - 23% of AI projects fail due to misalignment between AI solutions and user expectations. - Partnership efforts are required for successful AI projects. Data diversity is crucial, and incorporating diverse data sources can lead to better AI outcomes. Trends: - AI adoption is increasing across the Federal government, with a growing focus on chat and coding-related functions. - Agencies are leveraging cloud-native services, tooling, and other capabilities to accelerate AI project development. - There is a growing interest in translating older code to newer languages, such as Python or cloud-based analytic tools. - Domain-specific chat functionality is becoming a priority, with many agencies planning to implement chat services in the next two years. Statistics: - 325 AI experts surveyed across the Federal government - 14 months: average time from pilot to production for AI projects - 54%: percentage of projects currently under development that involve improving computer code - 59%: percentage of projects that involve "research and understanding" - 23%: percentage of AI projects that fail due to misalignment between AI solutions and user expectations. Bill Casola John Grim Leslie Anne McIndoe Kim Giffin Michael S. Freeman III Tobias Quaranta https://lnkd.in/eqvmAVFG

4

I remember learning category theory in undergrad and wondering: when would this ever be useful? Turns out it's very useful for program synthesis and software modeling -- thanks Kristopher for the great talk!

9

https://lnkd.in/eSrt4ey7 "Llama or Large Language Model Meta AI is a family of LLMs introduced by Meta AI in February 2023. The first version of the model was released in four sizes — 7B, 13B, 33B, and 65 billion parameters. Reportedly, the 13B model of Llama outperformed OpenAI’s GPT-3 which had 135 billion parameters. To simplify, parameters here are a measure of the size and complexity of an AI model and generally, a larger number of parameters means an AI model is more complex and powerful." Meta AI at Meta #Llama #AI #LLM #LargeLanguageModel

1

SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, Jing Gao Abstract Large language models (LLMs) often generate inaccurate or fabricated information and generally fail to indicate their confidence, which limits their broader applications. Previous work elicits confidence from LLMs by direct or self-consistency prompting, or constructing specific datasets for supervised finetuning. The prompting-based approaches have inferior performance, and the training-based approaches are limited to binary or inaccurate group-level confidence estimates. In this work, we present the advanced SaySelf, a training framework that teaches LLMs to express more accurate fine-grained confidence estimates. In addition, beyond the confidence scores, SaySelf initiates the process of directing LLMs to produce self-reflective rationales that clearly identify gaps in their parametric knowledge and explain their uncertainty. This is achieved by using an LLM to automatically summarize the uncertainties in specific knowledge via natural language. The summarization is based on the analysis of the inconsistency in multiple sampled reasoning chains, and the resulting data is utilized for supervised fine-tuning. Moreover, we utilize reinforcement learning with a meticulously crafted reward function to calibrate the confidence estimates, motivating LLMs to deliver accurate, high-confidence predictions and to penalize overconfidence in erroneous outputs. Experimental results in both in-distribution and out-of-distribution datasets demonstrate the effectiveness of SaySelf in reducing the confidence calibration error and maintaining the task performance. We show that the generated self-reflective rationales are reasonable and can further contribute to the calibration. The code is made public 👉 https://lnkd.in/djpaHfpP #machinelearning

2

Can LLMs be manipulated? A new ground breaking study by Caltech explores how careful design of input prompts can significantly influence LLM outputs—transforming unlikely predictions into likely ones! This study, in my opinion, also shines a light on our limited knowledge in the fundamental workings of LLMs and the critical need for greater transparency in LLM outputs to encourage enterprise adoption. #ai #genai #ai_transparency

33

An interesting paper by EpochAI.org on the constraints on Large Language Model (LLM) scaling based on the availability of public human-generated text data. The question is: where is the next bottleneck? GPUs or data? [direct arXiv link below in the comments] https://lnkd.in/eXXxQKv3

9

2 Comments

Harnessing AI for Humanity: Strategic Priorities for AI Research, Policy, and Practice This CACM article outlines ten critical priorities for AI research, policy, and practice, emphasizing AI's transformative potential and risks. These priorities aim to guide the development and deployment of AI in ways that maximize benefits while mitigating harm. Key recommendations include: 1. Scientific Foundations: Invest in research to deepen understanding of AI capabilities and limitations, ensuring robust human oversight. 2. Safety, Reliability, Equity: Enhance R&D to address safety, fairness, accuracy, and transparency in AI systems. 3. Regulation: Develop regulations and best practices to ensure responsible AI deployment. 4. Disinformation: Address uses of AI by malicious actors for disinformation, manipulation, and impersonation, which pose a risk to an informed citizenry and to the fabric of our democracies. 5. Resources: To foster collaborative AI research and bridge the resource gap between academia and industry. 6.      Vibrancy of Academia: Prevent a brain drain to industry by supporting academic institutions and retaining top talent. Cultivating and retaining top computer-science talent at universities will be critical for research advancement and teaching and mentoring the next generation of AI leaders. 7.      Jobs and Economy: Monitor AI’s impact on jobs and promote policies encouraging shared economic prosperity. These efforts can foster social and economic inclusion and empowerment in the face of AI-driven automation and transformation. 8. Diplomacy: Engage in international diplomacy to promote AI safety and human rights. Foster global cooperation and coordination on AI issues that transcend national boundaries and interests. 9. Deep Currents: Research long-term psychological, social, and cultural impacts of AI. Engage in focused interdisciplinary research efforts to anticipate and address human and societal implications of AI that may not be immediately apparent or measurable. 10. Catastrophic Outcomes: Proactively address potential catastrophic risks associated with AI. Introduce laws and regulations that preclude deploying powerful AI systems that do not meet due diligence standards. Proactive efforts on catastrophic risks can help to prevent and mitigate worst-case scenarios of AI The authors stress the need for interdisciplinary collaboration and proactive policy-making to shape AI’s future for the common good, ensuring that AI technologies contribute positively to human flourishing while guarding against their potential to cause harm. #AI #GENAI #AIREGULATION #AIRESEARCH https://lnkd.in/g3vrkpqA

4