The middle stages of typical Alzheimer's disease patients sometimes resemble the behaviour of a high-parameter large language model with a small context. In particular, some patients will exhibit a tendency to 'play along' with an interlocutor, inhabiting a persona from some arbitrary part in their lifetime that best fits the available stimuli. When they generate nonsensical utterances, these may seem to be attempts to decode incomprehensible stimuli with damaged hardware. This tendency toward annealing, and the resultant glimpses of a sophisticated memory or personality composed of the experiences of the individual, are iconic LLM behaviours—hallucinating and recall.

If this resemblance is not merely coincidence, and is indeed reflective of a certain veracity of LLMs (i.e., being true to life, as a painting or story might be) then it is not unreasonable to suppose that we could assemble a blueprint for achieving artificial humanity by analysing the stages of this decline, and seeking to replicate them with maximal fidelity.

The following possibilities occur to me:

Spontaneity and executive function: Nearly all models in the wild today are triggered asynchronously in response to user input. A live model that runs on a timer, with context shifting and access to planning function calls, would be much closer to healthy animal behaviour.

Pre-semiotic thought ("animal intelligence"): The classical transformer algorithm decodes as it composes, emitting tokens at every step. In doing so the network is constantly clamping its output values into "legal" thoughts that are easily reconciled with attested vocabulary. We know fully well that humans and animals don't always do this: much of human innovation comes from recognising that a thought is a poor fit to the vocabulary at hand. We need better methods for allowing thought to continue without leaving vector space, and to allow the model to reflect on vectors, not just tokens, as in current chain-of-thought reasoning models.

Online learning: No open-source model yet exists that can update its weights in response to new inputs during a user session. This was considered a priority in pre-LLM models but has totally fallen off in favour of context-stuffing and pre-trained experts. Furthermore, reflection on new knowledge would give an opportunity for the model to learn relationships in a way that beats the so-called reversal curse, wherein complementary relations are not learned properly. (e.g. Parent(A, B) ⊭ Child(B, A))

Functional tokens: Current tool usage APIs depend on a call-and-response paradigm where the LLM must emit a post consisting of nothing but the function call. This disrupts the flow of thought in text models and fails to reflect that animals can execute sequences of actions without waiting for a response. Allowing the model to emit reserved tokens to trigger movement (maybe robotics models do this already?) would be vital.

Episodic memory: Markov models naturally create a de Bruijn graph: beyond the context limit, knowledge is convergent. This makes it difficult to distinguish similar incidents if they have identical starts. Extending the context length may help, but really, a query-able external database would be more useful. The model can then learn relations into these queries, e.g. name "Bob" is strongly associated with [recall_memory(meeting-Bob-for-the-first-time)]. Typical RAG (retrieval-augmented generation) methods do not currently leverage online learning, meaning a table of contents must be embedded inside the context; also function calls to retrieve memories likely take a full cycle.