Thin slice : the paper's paradoxes seem to come from a confusion definitions of "information". Perhaps, there is a confusion between : the accuracy of information transfer ("surprise") and the accuracy of representing originating knowledge ("semantics").
Information theory is concerned with "already quantified" data, and not about "quantifying unquantified states". So entropy is meaningful as a descriptor of quantified data, and not meaningful as a descriptor of how accurate a quantification operation was upon an unquantified substrate.
Vis-a-vis LLMs, whether the input data is the traditional "verbal / just text" or multimodal "multi/sensory" ... these are "already quantified data" at the point of ingestion, and information is preserved as the LLMs compute any egestions.
For those of US OBSERVING the LLMs, we may find that the LLMs produced "new information", but that does not refer to insight about the ingested data (stateT=0), but rather about the the UNQUANTIFIED WORLD STATE prior to the creation of the ingested data (stateT=-N).
Please forgive my unfamiliarity with the underlying material, and my faint disgust at seeing 50+ comments applauding without adding much to conversation ( I think I saw few dissenters haha ).
No comments :
Post a Comment