Well, there are only two things in consciousness. Consciousness is information processes + qualia, and for this kind of discussion we can ignore qualia.
If mathematicians had a robust, quantified, understanding of how creativity works, from a phenomenological point of view, I am sure they might disagree that LLMs are not creative. Arguably, creativity is all LLMs do. ( Set union operations, associated with the Default Mode Network, synthetic thought. ) LLMs are simply bad at logic. ( Set intersection operations, associated with the Central Executive Network, analytical thought. ) But we have ATPs for that, so the parts are on the table, but they are not always put together nicely.
But what is the data? Well the data operated on by pure mathematicians is basically language ... the ability to discuss properties of language, and to identify unnamed behaviours of language, and to create new names and new grammar to discuss those.
If we go to applied mathematics, then we are simply back in the realm of physics, and we need to talk about machines have sensors, sensory data types that match human sensations, and then the ability to put that data in planes of space-time, and then to form and test hypotheses about that data.
