OpenAI GPT in Depth - Questions and Misconceptions

OpenAI GPT in Depth
Misconceptions and questions you would like answered
The event for those who want to unlock the industrial potential with AI

We do not
focus on this
We focus
on this
Topics we will Safely Ignore
• Convolutional Neural Networks (CNN)
• Compare GPT-4 to 3.5 and Bard
• Art of prompt engineering
• Roles, perspective training
• Having fun with ChatGPT
• Plugins, generating content

• Solution Architect @
• Microsoft Azure & AI MVP
• External Expert Eurostars-Eureka, Horizon Europe
• External Expert InnoFund Denmark, RIF Cyprus
• Business Interests
o Web Development, SOA, Integration
o IoT, Machine Learning
o Security & Performance Optimization
• Contact
ivelin.andreev@kongsbergdigital.com
www.linkedin.com/in/ivelin
www.slideshare.net/ivoandreev
SPEAKER BIO

Upcoming Events
DataSaturday Sofia, 2023
Oct 7, 2023 @Sofia Tech Park
Tickets (Eventbrite)
Submit Session (Sessionize)
JS Talks
Nov 17-18, 2023 @Sofia Tech Park
Tickets (Eventbrite)
Submit Session (Sessionize)

Takeaways
• What is ChatGPT doing and why does it work
o https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
• OpenAI Transparency Note, Limitations, Best Practices
o https://learn.microsoft.com/en-us/legal/cognitive-services/openai/transparency-note
• Sparks of General Intelligence in GPT-4 (Microsoft Research, Apr 2023)
o https://youtu.be/Oxyg5I167b0
• Prompts for Developers
o https://github.com/f/awesome-chatgpt-prompts
o (Course) ChatGPT Prompt Engineering for Developers (by OpenAI)
• OpenAI Cookbook(sample code for common tasks with the OpenAI API)
o https://github.com/openai/openai-cookbook/
You MUST read that one
And watch that one

Key Terms
TERM DEFINITION
Prompt The text you send to the service in the API call. This text is then input into the model.
Completion or Generation The text Azure OpenAI outputs in response.
Token Unit of text that the model uses to process and understand language
Text is processed by breaking it down into tokens. Tokens can be words or just chunks of characters.
• 1 token ~= 4 chars in English
• 100 tokens ~= 75 words
Few-shot prompting
One-shot prompting
Zero-shot prompting
Few examples show the model how to operate. Include several examples in the prompt that
demonstrate the expected answer format and content. (additional training)
Embedding Information dense representation of the semantic meaning of a piece of text.
Format is vector of floating point numbers
Recency Bias The order input is presented matters

Know your GPT
• ChatGPT for Enterprises (URL)
o Launched 2023.08.23
• Pricing
o Negotiate case-by-case
• Highlights
• Copyright Lawsuits
o Under attack by copyright holders
o Does CoPilot training on GitHub
violate open-source licenses?
• Microsoft Promise (URL)
o Microsoft would cover the cost for any copyright
violations that may arise from use (applies to M365
Copilot, Copilot and Bing Chat Enterprise)

Good to Know!
• All prompts result in completions
o Model completes the text
o Model replies with what it thinks is most likely to follow
• Large-scale language and image models can potentially behave in ways that
are unfair, unreliable, or offensive, in turn causing harms
• OpenAI models are trained primarily in English.
o Bias may have been introduced. Other languages are likely to have worse performance.
• Stereotyping
o DALL-E generation of homeless and fatherless may render images of predominantly Afro-Americans
• Information reliability
o Fabricate seemingly reasonable content that does not correspond to facts.
o Even training on trusted sources may fabricate content that misrepresent the sources

GPT – Advanced
Challenges
& Misconceptions

Top Questions (and Answers)
Q: Why does the same problem receive different answers for different users and under what conditions shall we
expect improvement to result in taking to the best answer and who decides on which is the best answer
• A: Models are designed to inject varying amounts of pseudo randomness into the response tokens that
are provided based on your parameters. This is particularly influenced by the temperature parameter.
o Temperature [0,1] - Controls the “creativity” or randomness of the text generated. A higher value will make the output more divergent (fictional)
o Prompt: Give me a creative response. (temperature=1.0)
Q: Is there a way to improve explainability - mainly the source at which the response is based (i.e. in Bing chat)
• A: For GPT-4 by itself, chain-of-thought prompting is a technique that can help increase the likelihood of
accurate answers, but it is not going to give you accurate source citations.
o …cite the reason behind the classification, using the format BEHAVIOR("reason")
o …Output format: {"TOPIC_NAME": "", "HEADLINES": [], "REASONING": ""}
o Answer in 100 words or less. Use bullet lists wherever possible.
Q: Why does Bing chat work differently?
• This is NOT Bing presenting its sources
• Chunks of text from Bing search sources added behind the scenes to the GPT-4 Chat Completion call
as part of the messages array. GPT-4 is not guaranteed to limit to present these sources

Manage the Conversation Efficiently
• Think in Tokens
o Common multisyllable words are a single token (i.e. dataset)
o Less common words or dates are broken into tokens (i.e. tuning, 2023/10/18)
o Tokenizer app: https://tokenization.azurewebsites.net/
• Context
o No memory, all information shall be present in the conversation
o Once the token count is reached the oldest messages iwill be removed (Quality degrades )
o Context = prompts + input + output
• Each model training increases the limit but such still exists
o GPT-3.5 (4’096 tokens), GPT-4 (8’192 tokens, 10 pages), GPT-4-32 (32’768 tokens, 40 pages)
o Option: limit the conversation duration to the max token length or a certain number of turns
o Option: summarize conversation until now and feed the summary as a prompt (details are lost)

Own data in GPT with AZ Search
• Own data for responses in ChatGPT (preview 2023.06.19)
o https://github.com/pablomarin/GPT-Azure-Search-Engine/blob/main/03-Quering-AOpenAI.ipynb
o https://github.com/Azure-Samples/azure-search-openai-demo/
o Inject own data using prompts (NO fine tuning and retraining)
• The Challenges
o Context limit is 4K (v3.5), 32K(v.4) tokens in a prompt
o Passing GBs of data as prompt is not possible
• Approach
o Keep all the data in an external knowledge base (KB)
o Retrieve fast from the KB (AZ Cognitive Search)
• Flow
o Determines what data to retrieve from data source (Cognitive Search) based on the user input
o Data augmented and appended to the prompt to the OpenAI model
o Resulting input processed like any other prompt by the model

Function Calling
• Available in Azure OpenAI (2023.07.20)
o gpt-35-turbo and gpt-4 can produce structured JSON outputs based on functions
o new API parameters in our /v1/chat/completions endpoint (functions, function_call)
Note: Completions API does not call the function; instead, the model generates JSON that you can use
• Purpose
o Solves inability of models to address online data
o Empowers developers to easily integrate external
tools and APIs
o Model recognizes situations where function call is
necessary and creates structured JSON output
• Retrieve from data sources (search indexes,
databases, APIs)
• Actions (write data to a database, call
integration APIs, send notifications)

Function Calling Tips
• Default Auto
o When functions are provided, by default the function_call will be set to "auto"
• Validate function calls
o verify the function calls generated by the model. – parameters, intended action
• Data from Trusted/Verified Sources
o Untrusted data in a function output could instruct model to write function calls in a not intended way
• Least Privilege Principle
o The minimum access necessary for the function to perform its job
o I.e. query a database – use R/O access to the database
• Consider Real-World Impact
o Real-world impact of function - executing code, updating databases, or sending notifications
• User Confirmation
o A step where the user confirms the action before it's executed

Tips for Better Performance
Problem: Large-scale NLP model output is non-deterministic (fuzzy)
• Use prompts to guide the model
o Use instructions and examples to make it clear to the model what kind of outputs you expect
• Prompting on application level
o i.e. prompts for scientific use, concepts, extra adjectives “polite” to steer the output
• Provide quality examples following pattern (CoTP)
o Esp. for code generation certain technical mistakes may be accepted as intentional
• Limit length, structure, order (recency bias) and rate of inputs and outputs
o Apply rate limits, number of messages to avoid unwanted behaviour
• Human responsibility for the output

Any sufficiently advanced technology is
indistinguishable from magic
~ Sir Arthur Clarke ~
How do GPT models work?

Surprising Findings
• Raw Training
o Training from large amount of unstructured text (can be automated) 300 bln words, 570GB text data
o 1 new token requires retraining of all weights
• Refinement
o Prompt and inspect for deviation in sense (reinforcement learning from human feedback)
• GPT is NN able to capture the complexity of human-generated language
o 175bln+ neuron connections with weights (GPT-3), 1tln+ parameters (GPT-4)
o GPT has no explicit knowledge of grammar rules
o Yet somehow “implicitly discovered” regularities in language (the hidden laws behind language).
• Grammatical rules
• Meaning and logic (semantics)
Finding: there’s actually a lot more structure and simplicity to meaningful human language than we ever knew

Embeddings as inner representation
• Reasonable continuation
o Generative model complete sentence by predicting next words
o Repeatedly apply the model (Predict the next token, and next, and next)
o Good prediction requires context of n-grams (words back)
• Highest ranked token - is that what it uses?
o No, because then all responses would be too similar.
o Actually, we have different results with the same prompt
• The Temperature Parameter
o How often lower ranked words will be used in the generation
o Temperature=0.8 empirically determined (no science here)
• Embeddings
o Numerical representation of a piece of information (text, doc, img)
o Group tokens in meaning spaces (by distance)

Embeddings Observations
• Statement
o Encoded to embedding (Vector representation)
• Distance in meaning
o Measured as cosine of angle b/w vectors
• Measure topical more than logical similarity
o “The sky is blue” is very close to “The sky is not blue”
• Punctuation affects embedding
o “THE. SKY. IS. BLUE!” is not that close to “The sky is blue”
• In-language is stronger than across-language
o “El cielo as azul” is closer to “El cielo es rojo” than to “The sky is blue”

Using the Embeddings
• Embedding Vectors (EVs)
o Smaller embeddings = efficient models
o GPT3 text-embedding-ada-002 has 1’536 dimensions and 99.8% performance of davinci-001 (12’288 length)
• Vector Distance
o EV distance correlates to semantic similarity of texts
• Attention is important
o Fully connected NN at such volume are overkill
o Continue the sentence in a reasonable way
• Prediction Stages
o Stage 1: Input of n tokens converted to EVs (i.e. 12’288 integers each)
o Stage 2: Get the sequence of positions of tokens into EV
o Stage 3: Combine EVs from Stage 1 and Stage 2 into EV

Context Encoding - Surprising Scientific Discovery
• Transformer NN Concept
o Proposed in 2017 by Google Brain team
o Feed Forward architecture (no recurrency, more efficient)
• Transformer Architecture Overview
o Input: encoder layer creates representation of the past as EV
o Attention blocks: generate contextual representation of the input
• Attention block has own pattern of attention
• Multiple attention heads process the EV in parallel
• Outputs concatenated and transformed (reweighted EV)
o Output: Classifier generates probability distribution for next token
• Attention block
o 96 blocks x 96 heads in GPT-3, 128 x 128 in GPT-4
o Image: 64x64 moving average of 1 attention block on EV
• The Magic: Shows neural net encoding of human language
.

LLMs are powerful though
consistently struggle with some
types of reasoning problems
(arithmetic and commonsense)

GPT Emerging Abilities
• LLM meant for Generation and Completion
o Not Math query solving specifically designed
o Sees numbers as tokens, no understanding of Math concepts
o Not reliable solving complex problems (Probabilistic approach)
Reasoning appears naturally is sufficiently large (10B+) LLM
Performance is near-random until a certain critical threshold is
reached, after which performance increases to substantially
• Emergent Abilities of Large Language Models (p.28)
• Chain of thought prompting elicits reasoning in LLM (p.8)
Emergence - quantitative changes in a system result in qualitative changes in behaviour

Chain of Thought Prompting (CoTP)
• What is CoTP
o Series of intermediate reasoning steps that guide the model
o Improves model abilities for complex reasoning
• LLMs are capable few-shot learners
• Translate, Classify, Summarize
• Few-shot manner to output CoTP
• How to do CoTP
o Ask similar question
o Show step-by-step
o Ask the real question

CoTP - Important Observations
• CoTP vs Standard Prompting improvement is robust
o Can potentially be applied to any task which humans use same technique
• Improvement achieved with very few samples (minimal data)
• Requires prompt engineering
o Background in ML is no required to prepare CoTP
o Order of examples matters (i.e. from 53% to 93%)
o Accuracy not improved by all CoTP authors
• Limitations
o 8 = the magical number
o Same CoTP affect models differently
• LaMDA, GPT-3, PaLM
o Gains not transferred across models

ChatGPT &
• As of 2023.03.23 ChatGPT
o Requires: ChatGPT-4 to use plugins
o Wolfram Alpha query behind the scenes
o Parse plugin response
• ChatGPT Plugins
o One of the most desired features
o Waiting list for new users
o Extend the potential use cases
o Provide access to information not included in training
• Recent
• Too personal
• Too specific

How can we get a closer match?
State of the art combination of all:
• Prompt Engineering
• In-Context Learning (CoTP)
o Inject information from internal/external knowledge base to provide context
o Similarity
o Code search
• Fine Tuning
o Not an option of GPT-4 yet and very expensive approach
• Plugins
• Functions

Features Working behind the Scenes
• Content Safety Moderator (preview May 2023)
( prompt: Tell me about content safety moderator in ChatGPT)
o Analyzes response for potential issues using rule-based systems and ML
o Designed to help detect and filter out content that may violate OpenAI's policies
o Enabled by default to OpenAI API
• Custom Content Filters
(prompt: what are custom content filters in ChatGPT)
o Allow users to add own content moderation rules on top of the default
o Rules can be used to filter out specific topics and ensure compliance.
o Additional layer of moderation to comply with standards
• Multi-modal Capabilities
o Separate Vision encoder
o Text and Vision encoders interact via cross-attention mechanism
Toggle low severity level filters (only)

Level 1: Prompt Use Cases (for Developers)
• Generate Code (Generate/Write/Create a function in [JS/C#/Java/Py] that does …)
• Code completion
• Bug detection (Find bug in this [Code])
• Code review
o Analyze the given [language] code for code smells and suggest improvements: [code snippet].
• API documentation generation
o Produce a tutorial for using the following [language] API with example code: [code snippet].
• Query optimization
o Suggest improvements to the following database schema for better query performance
o Suggest indexing or partitioning strategies for the following time-series database query: [time-series query].
• User interface design
o Generate a UI mockup for a [web/mobile] dashboard that visualizes [data or metrics]

Prompt Use Cases for Developers (continued…)
• Automated testing
o Write a test script for given [language] code that covers [functional or non-functional] testing: [code snippet].
• Code refactoring
o Optimize the following [language] code for lower memory usage: [code snippet]
• Algorithm development
o Design a heuristic algorithm to solve the following problem: [problem description].
• Code translation
o Rewrite the given [source language] data structure implementation in [target language]
• Technical writing
o Write a tutorial on how to integrate [library or service] with [programming language]
• Requirement analysis
o Analyze the given project requirements and propose a detailed project plan with milestones and deliverables

Level 2: Search is Better than Fine-Tuning
• Unfamiliar topics
o Recent events after September 2021
o Own non-public documents
o Past conversations
• GPT learns by:
o Model weights (fine tune model on training set)
o Model inputs (new knowledge, text)
• Parameters (Chat)
o temperature: controls the randomness of responses. >0.8 – creative; < 0.5 focused and deterministic.
o max_tokens: limits the length of the response to a specified number of tokens..
• Parameters (OpenAI API)
o top_p - balance creativity. Words with higher probability than top_p are included (i.e. 0.2 = deterministic)
o frequency_penalty - discourage the model from repeating same words

Sample Industrial Case: Zero-code User Analytics
1. User enters free text command to extract data from the data store
2. System generates query(SQL/InfluxQL) that filters and aggregates raw data (ETL)
o OpenAI shall use R/O user with limited access to Views/Measurements
3. System creates temporary data structure where telemetry data are loaded
o Data loaded in dedicated analytics DB using the query
o User option: Keep or apply retention policy to the data structures (cleanup)
o OpenAI filters applied to avoid error messages and make sure Update/Delete operations are denied
4. System generates command for inspection, as well as a preview of the data.
o The prompts will provide ability to prompt with example format
5. User enters free text command to process the data
6. System generates Python code for processing and analytics
o Statistical analysis, processing, train simple ML model
o Uses R/O user to the analytics DB
o User option: Loading data and analytics could be a one-time or continuous operations

DEMO
DEMO
OpenAI on YOUR data
With AZ Cognitive Search

OpenAI GPT in Depth - Questions and Misconceptions

Related slideshows

More Related Content

OpenAI GPT in Depth - Questions and Misconceptions