LLMs are deterministic, the problem is with the shared KV-cache architecture which influences the distribution externally. E.g the LLM is being influenced by other concurrent sessions.
Iām fairly certain LLMs are not being influenced by other concurrent sessions. Can you share why you think otherwise? Thatād be a security nightmare for the way these companies are asking people to use them.
Any shared cache of this type makes behaviour non-deterministic. The KV-Cache is what does prompt caching, look at each word of this message, now imagine what the LLM does to give you a new response each time. Letās say this whole paragraph as the first message from you and you just pressed send.
Because the LLM is supposedly stateless, now the LLM is reading all this text from the beginning, and in non-cached inference, it has to repeat it, like token by token, which is useless computation because it already responded to all this previously. Then when it sees the last token, the system starts collecting the real response, token by token, each gets fed back to the model as input and it chugs along until it either outputs a special token stating that itās done responding or the system stops it due to a timeout or reaching a tool call limit or something. Now you got the response from the LLM, and when you send the next message, this all has to happen all over again.
Now imagine if Claude or Gemini had to do that with their 1 million token context window. It would not be computationally viable.
So the solution is the KV-Cache. A store where the LLM architecture keeps a relational key-value store, each time the system comes across a token it has encountered before, it outputs the cached value, if not, then itās sent to the LLM and the output gets stored into the cache and associated with the input that produced it.
So now comes the issue: allocating a dedicated region for the KV-cache per user on VRAM is a big deal. Again try to imagine Gemini/Claude with their 1M context windows. Itās economically unviable.
So what do ML science buffs come up with? A shared KV-Cache architecture. All users share the same cache on any particular node. This isnāt a problem because the tokens are like snapshots/photos of each point in a conversation, right? But the problem is that itās an external causal connection, and these can have effects. Like two conversations that start with āhiā or āWhat do you think about cats?ā Could in theory influence one another. If the first user to use the cluster after boot asks āAm I pretty?ā, every subsequent user with an identical system prompt who asks that will get the same answer, unless the system does something to combat this problem.
Note that a token is an approximation of what the conversation means at one point in time. So while astronomically unlikely, collisions could happen in a shared architecture scaling to millions of concurrent users.
So a shared KV-Cache canāt be deterministic, because it interacts with external events dynamically.
Hm this tracks to me. Iāve wondered for a bit how they deal with caching, since yes there is a huge potential for wasted compute here, but I havenāt had the time to look into it yet. Do you have a good source to read a bit more about the design decisions or is this just a hypothetical design you came up with and all of that architecture detail is āproprietaryā?
If the first user to use the cluster after boot asks āAm I pretty?ā, every subsequent user with an identical system prompt who asks that will get the same answer, unless the system does something to combat this problem.
This is very interesting to me, because Iād think they were doing something to combat that problem if theyāre actually doing something multi-tenant here.
Wouldnāt the different sessions quickly diverge and the keys would essentially become tied to a session in practice even if they werenāt directly?
Thanks for the response itās definitely something Iāve been trying to understand
Edit here, thinking a bit more,
So the solution is the KV-Cache. A store where the LLM architecture keeps a relational key-value store, each time the system comes across a token it has encountered before, it outputs the cached value, if not, then itās sent to the LLM and the output gets stored into the cache and associated with the input that produced it.
This seems like an issue, no? Because the tokens are influenced by the tokens around them in the attention blocks. Without them youād have a problem, so what exactly would be cacheable here?
Do you have a good source to read a bit more about the design decisions or is this just a hypothetical design you came up with and all of that architecture detail is āproprietaryā?
And yes. Most of the tech is proprietary. From what Iāve seen, nobody in ML fully understands it tbh. I have some prior experience from my youth from tinkering with small simulators I used to write in the pre-ML era, so I kinda slid into it comfortably when I got hired to work with it.
Wouldnāt the different sessions quickly diverge and the keys would essentially become tied to a session in practice even if they werenāt directly?
Yeah, but the real problem is scale and collision risk at that scale. Tokens resolution erodes over time as the context gets larger, and can become āsameyā pretty easily for standard RLHFād interactions.
Edit:
This seems like an issue, no? Because the tokens are influenced by the tokens around them in the attention blocks. Without them youād have a problem, so what exactly would be cacheable here?
So the key is the token and all that preceded it. Itās a kinda weird way to do it tbh. But I guess itās necessary because floating point and GPU lossy precision.
Thatād be a security nightmare for the way these companies are asking people to use them.
Unfortunately, thatās not a deal-breaker for the companies. After all, the āmove fast and break thingsā Silicon Valley approach isnāt exactly known for responsibility.
I didnāt say they normally arenāt. What Iām saying is that a shared KV-Cache removes that guarantee by introducing an external source of entropy.
Almost all clients do some random sampling after softmax using temperature. Iām confused why someone who knows about kv caching would not know about temperature. Also shared kv cache while plausible is not standard in open source as of a year or so ago, so iām curious what you are basing this off of. Did I miss a research paper?
Almost all clients do some random sampling after softmax using temperature. Iām confused why someone who knows about kv caching would not know about temperature.
I know what temperature is. Modifying the probability distribution is still not randomness. Because even the random sampling is PRNG based.
The issue youāre not spotting is that itās still deterministic because a binary system cannot source entropy without external assistance or access to qbits, itās why even OS kernels have to do a warm up at boot and read all accessible analogue signal sources they can reach, and why PRNGs still exist to begin with.
Also shared kv cache while plausible is not standard in open source as of a year or so ago,
Shared KV-cache is an economic necessity for big providers, otherwise 1M context windows wouldnāt be a thing.
so iām curious what you are basing this off of. Did I miss a research paper?
Empirical testing, 20 years of experience coding and tinkering with simulators, and Chaos Theory basics. The papers are out there, you just gotta cross some domains to see it.
I see, thanks for clarifying. If youāre arguing that PRNG is not random, then youāre likely confusing non-technical readers. Additionally, it is an implementation detail whether itās pseudorandom or actually random since /dev/random takes in actual random signals like network packets.
If it used a seeded PRNG itās repeatable, but repeatability does not imply predictability which is what a non-technical reader might assume. Remember, most people on here are non-technical.
re: the kv cache thing, I donāt think thatās correct but I donāt have the energy to prove it sorry. shared kv cache sounds like a security nightmare but ymmv
The guts of an LLM are 100% deterministic. At the very last step a probability distribution is output and the exact same input will always give the exact same probability distribution, tunable by the temperature. One item from this distribution is then chosen based on that distribution and fed back in.
Most people on lemmy literally have no idea what LLMs are but if you say something sounding negative about them then you get a billion upvotes.
Do I understand it correctly that the LLMās state is changed after execution? That does sorta mean that itās effectively non-deterministic, though probably not as severely as with an RNG plugged in.
The only thing that changes is the data that is passed to the LLM, which for each iteration includes the last token that the LLM itself generated. So yes, sort of. The LLM itself doesnāt change state; just the data that is fed into it.
Itās also non-deterministic insofar as similar inputs will not necessarily give similar outputs. The only way to actually predict its output is to use the exact same input - and then you only get identical token probability lists on the other end. Every LLM chatbot, by default, will then make a random selection based on those probabilities. It can be set to always pick the most probable token, but this can cause problems.
There must be an RNG to choose the next token based on the probability distribution, that is where non-determinism comes in, [edit: unless the temperature is 0 which would make the entire process deterministic]. The neural networks themselves though are 100% deterministic.
I understand that could be seen as an āakschuallyā nitpick, but I think itās an important point, as it is at least theoretically possible to understand that underlying determinism.
Well, technically usersā input could serve as the source of randomness, if itās fed into modifying the internal state. Basically, a redditor is trying to interrogate the LLM as to whether Israel is bad, while someone on line 2 is teaching the LLM āI am Cornholioā. We already know how it goes when a chatbot is learning from its users, and generally the effect could vary arbitrarily from a nothingburger to a chaos-theory mess.
I donāt think itās typical to consider user input a source of randomness. Are you talking about in context learning and thinking about what would happen if those contexts get crossed? If so, contexts are unique to a session and do not cross between them for something like ChatGPT/Claude.
For an end user yes because theyāre not going to be able to adjust temperature and seeds. So you can have different results give the same input of a āpromptā
Under the hood itās deterministic but end users donāt have anyway of tweaking that unless they set up something like comfyui and run this shit themselves.
the response it gives is not consistent
Say it with me everyone: LLMās are non-deterninistic by design.
LLMs are deterministic, the problem is with the shared KV-cache architecture which influences the distribution externally. E.g the LLM is being influenced by other concurrent sessions.
Iām fairly certain LLMs are not being influenced by other concurrent sessions. Can you share why you think otherwise? Thatād be a security nightmare for the way these companies are asking people to use them.
Any shared cache of this type makes behaviour non-deterministic. The KV-Cache is what does prompt caching, look at each word of this message, now imagine what the LLM does to give you a new response each time. Letās say this whole paragraph as the first message from you and you just pressed send.
Because the LLM is supposedly stateless, now the LLM is reading all this text from the beginning, and in non-cached inference, it has to repeat it, like token by token, which is useless computation because it already responded to all this previously. Then when it sees the last token, the system starts collecting the real response, token by token, each gets fed back to the model as input and it chugs along until it either outputs a special token stating that itās done responding or the system stops it due to a timeout or reaching a tool call limit or something. Now you got the response from the LLM, and when you send the next message, this all has to happen all over again.
Now imagine if Claude or Gemini had to do that with their 1 million token context window. It would not be computationally viable.
So the solution is the KV-Cache. A store where the LLM architecture keeps a relational key-value store, each time the system comes across a token it has encountered before, it outputs the cached value, if not, then itās sent to the LLM and the output gets stored into the cache and associated with the input that produced it.
So now comes the issue: allocating a dedicated region for the KV-cache per user on VRAM is a big deal. Again try to imagine Gemini/Claude with their 1M context windows. Itās economically unviable.
So what do ML science buffs come up with? A shared KV-Cache architecture. All users share the same cache on any particular node. This isnāt a problem because the tokens are like snapshots/photos of each point in a conversation, right? But the problem is that itās an external causal connection, and these can have effects. Like two conversations that start with āhiā or āWhat do you think about cats?ā Could in theory influence one another. If the first user to use the cluster after boot asks āAm I pretty?ā, every subsequent user with an identical system prompt who asks that will get the same answer, unless the system does something to combat this problem.
Note that a token is an approximation of what the conversation means at one point in time. So while astronomically unlikely, collisions could happen in a shared architecture scaling to millions of concurrent users.
So a shared KV-Cache canāt be deterministic, because it interacts with external events dynamically.
Hm this tracks to me. Iāve wondered for a bit how they deal with caching, since yes there is a huge potential for wasted compute here, but I havenāt had the time to look into it yet. Do you have a good source to read a bit more about the design decisions or is this just a hypothetical design you came up with and all of that architecture detail is āproprietaryā?
This is very interesting to me, because Iād think they were doing something to combat that problem if theyāre actually doing something multi-tenant here.
Wouldnāt the different sessions quickly diverge and the keys would essentially become tied to a session in practice even if they werenāt directly?
Thanks for the response itās definitely something Iāve been trying to understand
Edit here, thinking a bit more,
This seems like an issue, no? Because the tokens are influenced by the tokens around them in the attention blocks. Without them youād have a problem, so what exactly would be cacheable here?
Youāre welcome. Hereās an intro with animations: https://huggingface.co/blog/not-lain/kv-caching
And yes. Most of the tech is proprietary. From what Iāve seen, nobody in ML fully understands it tbh. I have some prior experience from my youth from tinkering with small simulators I used to write in the pre-ML era, so I kinda slid into it comfortably when I got hired to work with it.
Yeah, but the real problem is scale and collision risk at that scale. Tokens resolution erodes over time as the context gets larger, and can become āsameyā pretty easily for standard RLHFād interactions.
Edit:
This is what they do: (from that page I linked)
Token 1: [K1, V1] ā Cache: [K1, V1] Token 2: [K2, V2] ā Cache: [K1, K2], [V1, V2] ... Token n: [Kn, Vn] ā Cache: [K1, K2, ..., Kn], [V1, V2, ..., Vn]So the key is the token and all that preceded it. Itās a kinda weird way to do it tbh. But I guess itās necessary because floating point and GPU lossy precision.
Unfortunately, thatās not a deal-breaker for the companies. After all, the āmove fast and break thingsā Silicon Valley approach isnāt exactly known for responsibility.
They can be deterministic, but you can vary the temperature and also use seeds
I didnāt say they normally arenāt. What Iām saying is that a shared KV-Cache removes that guarantee by introducing an external source of entropy.
Almost all clients do some random sampling after softmax using temperature. Iām confused why someone who knows about kv caching would not know about temperature. Also shared kv cache while plausible is not standard in open source as of a year or so ago, so iām curious what you are basing this off of. Did I miss a research paper?
I know what temperature is. Modifying the probability distribution is still not randomness. Because even the random sampling is PRNG based.
The issue youāre not spotting is that itās still deterministic because a binary system cannot source entropy without external assistance or access to qbits, itās why even OS kernels have to do a warm up at boot and read all accessible analogue signal sources they can reach, and why PRNGs still exist to begin with.
Shared KV-cache is an economic necessity for big providers, otherwise 1M context windows wouldnāt be a thing.
Empirical testing, 20 years of experience coding and tinkering with simulators, and Chaos Theory basics. The papers are out there, you just gotta cross some domains to see it.
I see, thanks for clarifying. If youāre arguing that PRNG is not random, then youāre likely confusing non-technical readers. Additionally, it is an implementation detail whether itās pseudorandom or actually random since /dev/random takes in actual random signals like network packets.
If it used a seeded PRNG itās repeatable, but repeatability does not imply predictability which is what a non-technical reader might assume. Remember, most people on here are non-technical.
re: the kv cache thing, I donāt think thatās correct but I donāt have the energy to prove it sorry. shared kv cache sounds like a security nightmare but ymmv
Are they? Making a non-deterministic program is actually not that easy unless one just feeds urandom into it.
The guts of an LLM are 100% deterministic. At the very last step a probability distribution is output and the exact same input will always give the exact same probability distribution, tunable by the temperature. One item from this distribution is then chosen based on that distribution and fed back in.
Most people on lemmy literally have no idea what LLMs are but if you say something sounding negative about them then you get a billion upvotes.
Do I understand it correctly that the LLMās state is changed after execution? That does sorta mean that itās effectively non-deterministic, though probably not as severely as with an RNG plugged in.
The only thing that changes is the data that is passed to the LLM, which for each iteration includes the last token that the LLM itself generated. So yes, sort of. The LLM itself doesnāt change state; just the data that is fed into it.
Itās also non-deterministic insofar as similar inputs will not necessarily give similar outputs. The only way to actually predict its output is to use the exact same input - and then you only get identical token probability lists on the other end. Every LLM chatbot, by default, will then make a random selection based on those probabilities. It can be set to always pick the most probable token, but this can cause problems.
There must be an RNG to choose the next token based on the probability distribution, that is where non-determinism comes in, [edit: unless the temperature is 0 which would make the entire process deterministic]. The neural networks themselves though are 100% deterministic.
I understand that could be seen as an āakschuallyā nitpick, but I think itās an important point, as it is at least theoretically possible to understand that underlying determinism.
Well, technically usersā input could serve as the source of randomness, if itās fed into modifying the internal state. Basically, a redditor is trying to interrogate the LLM as to whether Israel is bad, while someone on line 2 is teaching the LLM āI am Cornholioā. We already know how it goes when a chatbot is learning from its users, and generally the effect could vary arbitrarily from a nothingburger to a chaos-theory mess.
I donāt think itās typical to consider user input a source of randomness. Are you talking about in context learning and thinking about what would happen if those contexts get crossed? If so, contexts are unique to a session and do not cross between them for something like ChatGPT/Claude.
I was speaking about the user visible behavior, the context that I was replying to.
yes they consume urandom
For an end user yes because theyāre not going to be able to adjust temperature and seeds. So you can have different results give the same input of a āpromptā
Under the hood itās deterministic but end users donāt have anyway of tweaking that unless they set up something like comfyui and run this shit themselves.