4

We have a certain fixed vocabulary or ontology and would like to match the output of a LLM to this vocabulary.

The LLM output could be either a single term, a list of terms, or free form text. For example, the vocabulary is a list of occupations, and we would like to ask "provide a list of occupations for which this skill is useful".

The solution could either be a post-processing step (which would make it useful for non-LLM-texts, too), or somehow integrated in the generation.

  • For a small vocabulary, I could include it in the prompt. However the vocabulary is likely very large. Going by the identifiers alone it could also be ambiguous, like "Doctor" (of medicine or of physics).
  • One thing I tried was to calculate the embedding of each term in the ontology, then the embedding of the term I would like to match, and then pick the term that is closest to that. This works for simple words or lists, but it doesn't work very well in free-form texts. I did manage to work around it, by asking the LLM to mark occupation terms with "[]". It also doesn't work well when the word alone doesn't completely define the term.

Is there any standard way to restrict or match the output of an LLM to a fixed vocabulary?

Improve this question
2
  • I don't know of a standard way, but there is an additional way you didn't mention. You could custom train your model on the ontology and give it prompts that heavily reward it for picking from that list. You could also add a bunch of those free form text responses that are supposed to map to particular items in your ontology. It will never be perfect, but you could also combine that with a post-processing step.
    – JP Alioto
    Commented Jan 10 at 18:30
  • That's a good addition! As a first step we only considered techniques which dont require retraining, but I'll check that out, too.
    – jdm
    Commented Jan 11 at 11:22

2 Answers 2

Reset to default
2

Use an LLM Completion Generation Framework

Take a look at Completion Generation Frameworks such as Guidance (created by Microsoft).

Here's how you would force the LLM to choose one of the selected options. The framework guarantees you get your desired output 100% of the time.

from guidance import user, assistant

# load a chat model
chat_lm = models.LlamaCppChat(path)

# wrap with chat block contexts
with user():
    lm = chat_lm + 'Do you want a joke or a poem?'

with assistant():
    lm += f"A {select(['joke', 'poem'])}."`

A side bonus is that the framework will generate the least amount of tokens needed to select an option out of the available options, this saves both generation time and compute cost. Generated tokens are marked in green.

enter image description here

Frameworks available

All of these should support both fixed and open-ended ontologies.

Improve this answer
0

To restrict or match the output of a Large Language Model (LLM) to a fixed vocabulary, several strategies can be employed. One effective method is to calculate embeddings for each term in your vocabulary and compare them to embeddings of the LLM's outputs using similarity measures like cosine similarity. This approach works well for straightforward terms and lists but may struggle with more complex or ambiguous outputs. For free-form text, consider incorporating keyword extraction or named entity recognition (NER) techniques to identify relevant terms from the LLM's output that align with your vocabulary. Additionally, crafting specific prompts or fine-tuning the LLM on your vocabulary can steer its outputs towards desired terms. Hybrid approaches, combining embedding similarity with pattern matching or context-specific cues, can provide flexibility and accuracy in matching LLM outputs to a diverse vocabulary.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.