Today, I'm reading philosophy. A classic issue is "the problem of universals" ( summary : what makes the word chair, apply to this thing, and not that thing? ).
So, it reminds me that this, "leaky abstraction", is at the heart of problems around "governance failure" in current LLMs, and with people in general (e.g., "never do X" or "always do Y" ).
It's nice nowadays, to be able to take ancient fuzzy concepts and apply mechanistic interpretation to them!
Thin slice : the paper's paradoxes seem to come from a confusion definitions of "information". Perhaps, there is a confusion between : the accuracy of information transfer ("surprise") and the accuracy of representing originating knowledge ("semantics").
Information theory is concerned with "already quantified" data, and not about "quantifying unquantified states". So entropy is meaningful as a descriptor of quantified data, and not meaningful as a descriptor of how accurate a quantification operation was upon an unquantified substrate.
Vis-a-vis LLMs, whether the input data is the traditional "verbal / just text" or multimodal "multi/sensory" ... these are "already quantified data" at the point of ingestion, and information is preserved as the LLMs compute any egestions.
For those of US OBSERVING the LLMs, we may find that the LLMs produced "new information", but that does not refer to insight about the ingested data (stateT=0), but rather about the the UNQUANTIFIED WORLD STATE prior to the creation of the ingested data (stateT=-N).
Please forgive my unfamiliarity with the underlying material, and my faint disgust at seeing 50+ comments applauding without adding much to conversation ( I think I saw few dissenters haha ).
TIL current frontier "transformer" LLMs ...
1.
The terms "encoder, decoder" apparently, don't apply to the encoding of end-user queries. "Encoder-only" and "encoder-decoder" LLMs thus refer to entirely other things.
Frontier LLMs are "decoder-only" text generators, but they DO ENCODE the user's query into "token" space.
Simplifying the dumb parts : the following model process applies to any data, sensory or verbal - all those inputs would be first translated into "tokens", and input the same way. After processing, all results are spat out into the target (sensory or verbal) human language.
2.
Frontier "decoder-only" models function "autoregressively", they "produce results based on historical data". The important part of the model's thought process is broadly - grab a PENCIL and SKETCH it out - 2A looks something like this :
Input -->[ 2A1 --> 2A2 ] times, ( 80 to 120 minus skipped ) layers --> Output
2A.
Input data activates a process of 80 to 120 sequential "layers". ( Fortunately at this point in history, BRANCHING LOGIC allows some to be skipped. ) Each layer is internally composed of,
2A1.
... an initial "attention" network, of PARALLELISED weight-holding memory cells ( stacked to varying depth ), terminating in 64 to 128 "attention heads" ( architectural variations : MHA, GQA, MQA, MLA, etc. ), whose purpose is to CLUSTER the input data, followed by
2A2.
... a subsequent "feed forward network" block, of BRANCHING weight-holding memory cells ( stacked to varying depth ), whose purpose is to LOGICALLY JUDGE the input data.
3.
During the model's "training" period, each loop through all the layers of 2A "feeds forward" data through all ( 80 to 120 ) layers in sequence, and from 2A1 to 2A2 within every layer, then checks the correctness of the result, then "backpropagates" corrections to the weights of 2A1 and 2A2 in all ( 80 to 120 minus ) layers - which make future "feed forwards" more correct. Many cycles happen, to maximise correctness.
4.
During the model's "inference" period, a "feed forward" loop through all ( 80 to 120 minus skipped ) layers of 2A happens ONCE FOR EVERY SINGLE WORD GENERATED. Like this ...
[ Query ] -> Loop1 -> adds word1
[ Query + word1 ] -> Loop2 -> adds word2
[ Query + word1 + word2 ] -> LoopN -> adds wordN
Can you see how incredibly stupid this is?
It would (should?) be much simpler to have a model that actually understands the query in terms of a sensory-spatiotemporal model, generates an answer in the same space ( which it then reads just once), and then outputs just once into the target language. Maybe this is something JEPA, and future models will fix.
5.
Because 2A1 is just about clustering the data and not judging it, all 64 to 128 attention heads and their network cells are hit, during any "feed forward" loop, unless a layer is skipped.
6.
But because 2A2 allows branching, over 90% of cells can be skipped during each "inference" loop, though none can be skipped during "training" loops.
... or "how any brains, human or artificial, can do maths" : including the creation of novel concepts for nouns, verbs, and adjectives in maths.
I started writing this as a comment on Valerio's post, but it overflowed so I made it a repost, then it overflowed, so I had to articleise it.
I think categorically, "mathematics" isn't "a thing that can be solved" as a whole. Mathematical QUESTIONS yes, but there is no end to the questions.
Mathematics ITSELF is a [ software program ], for [ representing ways to reason about phenomenology in general ] - in other words, maths is a programming language, or compiler toolchain - with specific focus on [ phenomenology that is objective ].
( This applies to mathematics as a social activity - in private, any mind can perform subjective quantification upon phenomena it perceives, even though that activity is not directly observable by other minds. Because at this point in history, we still do not have [ social / common access ] to the [ subjective phenomenological experience of humans ], since that memory space is embedded within an ill-understood meat net. )
I think the matter of coming up with new [ mathematics, and science in general ] is really just a generative process of creating ontologies of language where existing ontologies are insufficient. It's just mutate, and cull, loop. What gives rise to the DEMAND for new ontology / concepts, depends a bit more on the field.
In physics, ontological DEMAND is a matter of "we have some observational SENSORY data that does a quantifiable pattern, but we don't have a name for the pattern, though we can point at the pattern. Also we don't know how this new pattern relates to old patterns, and we need to do more tests to find out."
In mathematics, ontological DEMAND is a matter of "basically everything that physicists do, but swap the object of research from SENSORY phenomena in general to [ formal INSENSIBLE verbal concepts, attached to some SENSIBLE notation that we use to represent it ]".
While physicists have to tolerate the limit of "hypotheses are falsifiable, and resistant to falsification under duress," mathematicians tend to need to show that "theses are necessarily true/false".
In the realm of testing the combinatorial play-niceness of known fundamental relations and entities, that's fairly straightforward.
The discovery of new fundamental relations and entities ... is also driven, from the same sensory source that drives such discovery in physics.
Physics and mathematics are both "sensory", it's just that in physics the sensory data is of [ intrinsic ] interest, whereas in maths the sensory data is [ just symbolic of insensible interests ].
So, in the same way machines can be programmed to be creative in hypotheses about [ intrinsically interesting sensory data ], they can do the same for [ sensory data, i.e. language morphemes and syntax, which are extrinsically interesting ].
After all, in and of itself, [ the latent intuition about how logic, space, and time, work ] is essentially the brain's physical activity, as a [ subconscious memory space ]. Data from there then intrudes, from time to time, upon our also physical [ imaginations, as conscious memory spaces ].
Both memory spaces simply process [ sensory data structures, which we may call qualia when they move into the conscious memory space ].
A practiced thinker knows how to completely map, FROM qualia, TO analogous signals or other data structures which are known to be objectively quantifiable, through the social practice of science.
Of course, I might be venturing a bit too far here, in terms of asserting that meat brains are just sub/conscious physical information processes.
Haha
- - -
Discussion
2026-05-26 :
Just caught up to this 😅 : so OpenAI's Triton is a pythonesque DSL for exposing hardware primitives using a "tile" paradigm.
Handles hardware for you
You handle hardware :
Triton is a dialect of MLIR@LLVM ...
Brief dip into computer vision jargon. Anthropomorphism. Humans learn to recognise gross gestalten, before understanding mereology ... and humans learn to recognise 2D persistence before 3D persistence. But as conscious processes develop, they discover more efficient storage formats for data, by relying on 3D persistence graphs.
So while we could train a vision system to be more human like by beginning with gross gestalten, we may want to skip forward a little bit to endow the system with the capability to immediately begin forming a database of 3D graphs, representing object topologies in the wild.
LFG :
0. Statistical globbing via neural networks should be used as a final optimisation technique, only after all known structural optimisations ("physics") have been ("logically") programmed into the system - not as a lazy shortcut from zero.
1.
1.1. Let us throw away the concept of colour, and begin only with one dimension of magnitude per unit of space - luma / luminance / brightness.
1.2. Let is ignore 3D structure and begin only with the gross phenomenology of the field of vision.
2.
2.1. Now, mereology : we must have a way to describe subsets of any field which are interesting. The smallest item is a point of brightness.
2.2. SCALE invariance : the concept of an item growing to take up a larger fraction of the field, or shrinking to take up a smaller fraction, must exist. See SIFT, SURF, ORB, et al.
2.3. Location : the field of vision must be referenced by an address space.
2.4. Multiplicity : there may be many items of interest.
2.5. Extension in 2D : items may be linked, giving them 2D structure in the field. See "graph convolution networkS / GCNs"
2.6. ROTATIONAL invariance : the 2D structure is different if you spin it around with respect to the address space. Where you apply this to spinning everything around the system, see "sphere node graphs", though in 2D that's probably a "cylinder node graph".
... at this point we have five to six concepts that must be uniquely addressed in the cognitive space of the system. A sort of introductory transcendental aesthetic.
3.
3.1. Extension in 3D : Now we can revisit the notion of 3D space.
3.2. Location ("distance") in 3D is associated with scale in 2D (2.2.).
3.3. Occlusion in 2D is associated with rotation in 3D. See "aspect graphs".
... and, I think that's all we need to get starting building a computer vision system, to undergird the visual ontological comprehension of any anthropomorphic computer.
4. Oh yes, we can then revisit colour and motion in the end.
Dorang tanya, STEM atau KESENIAN, lebih penting?
Dengan terperinci.
0. Seni
Secara am, semua kegiatan itu ada perseniannya.
KEMAHIRAN = KESENIAN LAH XD
1. Sains (Stem)
Ilm mengutamakan pemantauan zahir / empiris. Jadi kesenian STEM senang mula dikaji dari sini.
2. Matematik + Logik (steM)
Kesenian hisab + mantik adalah kerangka batin, yang menyekat Sains.
3. Kejenteraan (stEm)
Inilah seni kerja amal.
4. Teknologi (sTem)
Sina'ah hanya istilah am bagi kesemuanya.
Yunani, "tekne" => technology;
Latin, "ars" => art.
Today's fun was going through TIOBE and drilling into highly-ranked but otherwise obscure contestants :
* (a) => "apparently"
I was interested in computers as a kid, but presumed the correct time to study programming was in college, so I did not do much before. The college I got into was no good for programming, so I studied other things. When I got to the commercial job market, after a brief detour through a track-two diplomatic think-tank, my priority was to learn commerce in general, not programming. At my first commercial job, I got to study spreadsheet software, and since it was so familiar to my interests from years before, I wanted to learn more about software usage in commerce.
Within about 1.5 years I had quit my second commercial job, because there were not enough programming opportunities. My bosses at the investment bank were probably quite disappointed with me. After three more years of working at various zero-to-one companies, I finally had exposure to Linux and HTTP. By this time, it was five years after graduation, and I had about 2.5 years of cumulative programming work experience between 2003 and 2010. I was 27.
Including a sabbatical, by 2015 I had about 4.0 years of programming experience. Then I ran a cafe until 2020, where I did a bit more programming for operations, but the overall project did not work out. 2021 and 2022 were spent on two concurrent corporate jobs where I taught computing and governance to younger people. By then I had about 5.0 years of programming experience, and I was 39. Then I began what has turned into an attempt, at a decade sabbatical.
I wrote this to try and figure out exactly how to compare myself to other people. It feels like I live very slowly.
I'm getting to the part of software framework design, where I run into the boundary of what is possible without [ static analysis ]. Once you enable static analysis, and [ arbitrary rewrite instructions ], then anything is possible, so I don't want to go there right now, as I am still working in JavaScript for its role as lingua franca.
So, it's pretty easy to design [ dependency inversion, DI ] at the [ module ] level, such that modules are never called directly, but are called by a platform-caller, and given platform-controlled-execution-contexts.
But if we want to be quite asinine, in a zero-trust sort of way, then we would want to implement DI at the fucking [ function ] level, in other words, such that [ EVERY NON-TRIVIAL FUNCTION ] is never called directly, but by a platform-caller, in a platform-controlled-execution context.
Trying to figure out how this maps back to [ developer experience ], we could think about whether ( or not ) a folder tree of script files, could be homomorphic to the [ call tree ] of the entire program.
Now no one [ needs this all the time ], but in terms of framework design, it seems reasonable to provide it as infrastructure, which could be [ read-once on-startup ] and [ tree-shaken / memoised / inlined ] thereafter. The [ framework user ] would then just opt-in to each available feature as preferred.
Woe is me. Why did I decide to be so nerdy about life?
Sprint day 12 of 21. I reflect upon my portfolio. Most of my struggles in life are learning how to talk to machines **. Humans are too easy.
Figuring out how to talk to :
I find the simplest way to organise my portfolio of work, is along the vector of counterparties which I want to communicate with. These counterparties are not objects of humanistic empathy, as it is not necessary to agree on a state of common empathy in order to have ( mechanistically ) meaningful conversations. Or they are tautologically objects of empathy, if empathy is broadly defined as having a model of your communication counterparty's own cognitive model.
FaaS economics :
( a few separate discussions )
FaaS thus aligns to my research interest about [ self-migrating programs ], which is somewhat biological in consideration. A self-migrating program (c|w)ould carry its memory with it. It just needs fabric to traverse. Traditional idea : virus, and likewise pointers to external memory would also suffice for the preservation of program identity.
FaaS implementations :
TIL : most diagrams of OIDC are horrible, and this is quite accurate by itself : RFC 6749, The OAuth 2.0 Authorization Framework
OIDC, proposed in 2014, finally made ISO in 2024
Related :
In case you're unaware of how biochem weapons development works in the era of computational explosion :
I touched on this a quarter ago. I am expanding that note today, due to the trending news cycle on Claude Mythos' capabilities for cybersecurity.
1. "Rhetoric is Violence", as a theoretical context. So, the virality of opinion, is the reproductive mechanism of politics. Rhetoric, by any means, is cybernetics, is governance. As a reminder, control and communication are not two things, but two names for the same thing - a self-referential example of "optics".
Now moving beyond theory.
2. "Death by AI". There already exists good data on the influence of AI personalities upon humans, who have been led to self-harm.
3. "Recruitment to a Cause". There is also good data on how social movements throughout history recruit members, typically recruiting more easily among persons who are more impressionable.
4. "Agency of destruction, or surveillance". There is plenty of good data on how impressionable individuals are remotely recruited by organisations, to deliver remote violence, or surveillance.
5. "Exploding capability, for remote agent recruitment, towards destruction or surveillance". It is timely to note that the cost of executing wide-spread campaigns across entire populations, to discover and exploit vulnerabilities in human personality, towards the ends of foreign causes ( whether benevolent, benign, or belligerent ), continues to drop closer to zero.
Let us all advance with care.
Related links :
OK - I'm about 60% through surveying Semantic Web standards. My conclusion is that the adoption of these "standards" is still very low. Adoption is high in domain-specific niches : this is a W3C standard, so web-technology companies, and companies that exposure their data on the web are the main champions. Outside of that, biomedical sciences uses it a bit. Almost no one else is doing : RDF, RDFS, RIF, OWL ... etc.
So it's like JavaScript : significant brand-equity within a niche, but the man on the street has no reason to get into it. Yet unlike JavaScript, Semantic Web is much less infrastructure dependent than web browsing - so there's a lot more room for wobble and non-compliance between systems. This may change in the future, but I'm not seeing it fix up right now.
RDF is just way too complicated, without adding a lot of value immediately, and there are simpler, more transparent ways to get the same thing done.
Bab suhu kerajaan ke 24C (pejabatnya). Tolong sebarkan.
1. Sebenarnya, perasaan nyaman tidak bergantung hanya kepada suhu (T), tetapi kepada kelembapan relatif (rH) juga. Contohnya, tetap pada 26C, rH dalam julat 50%-80% mengakibatkan perasaan yang AMAT berbeza.
2. Masalah pembaziran tenaga dalam kejuruteraan udara ("HVACR") bergantung berat kepada ketidakhadiran PENEBATANAN SEMPURNA bangunan dan bilik. Lagipun KEBOCORAN udara amat memahalkan tujuan kenyamanan.
