Menu

Nucleus/ Top_p Sampling 

Nucleus/ Top_p Sampling

Nucleus sampling is also known as Top_p Sampling.  In Nucleus sampling, the model only considers the tokens with the highest probability mass, which is determined by the top_p parameter. OpenAI recommends to use either one but not both:

  • nucleus sampling or
  • temperature sampling

Top_p Sampling

The Top_P parameter controls the diversity of the text generated by the GPT model. This parameter sets a threshold such that only words with probabilities greater than or equal to the threshold will be included in the response. To understand Top_p sampling, we need to understand two things.

  • Probability Distribution
  • Cumulative Distribution

Probability Distribution

After processing input tokens, the GPT model predicts the next token by assigning probabilities to all possible next tokens in its vocabulary. This results in a probability distribution over the vocabulary.

Cumulative Distribution

The probabilities are sorted in descending order, and a cumulative sum is calculated. This process identifies the smallest set of tokens whose cumulative probability exceeds a threshold p. The top_p gets its name from the threshold p, which represents the percentage of the total population or sample being considered.

Example

For example, a value of 0.9 means that only the tokens with the top 90% probability mass are considered. i.e. the set includes the smallest number of most probable tokens that together account for 90% of the total probability.

The next token is randomly sampled from this set rather than from the entire vocabulary or the top-k most likely words. This method ensures that the tokens chosen are both likely and diverse.
Likely: Since they’re part of the cumulative probability exceeding the threshold p
Diverse: Since even less likely words have a chance to be selected, as long as they’re within the top-p set.

Adjust the value

We can adjust the top_p value in API requests or the Playground UI
while interacting with the model.
Top_p Sampling GPT

Related Posts

GPT Temperature Setting

GPT Temperature Setting

The GPT Temperature setting is an important setting that significantly impacts the model’s response. It controls the amount of randomness in the output

Presence penalty OpenAI

Presence Penalty

In this tutorial, we will learn how to set the Presence Penalty setting using OpenAI Playground or API requests. This setting increases the model’s probability

Frequency Penalty OpenAI

Frequency Penalty

The frequency penalty setting suppresses or allows the model to repeat while generating the response. This tutorial will teach us how to set this setting using OpenAI