The boundary between probability and hard logic

Image by SURESH KUMAR SAINI EAR via Wikimedia Commons

The problem of symbolic grounding endlessly fascinates me. It is one of the core problems in AI. If we are to faithfully reason about information, the reasoning process must correctly understand the meaning of the information.

LLMs have shown that they have mastered the task of parsing and, for the most part, understanding the intention of what was said. In addition, a significant level of emergent reasoning capability has also been demonstrated by the most advanced models. How exactly this property emerges is not entirely clear, but what is clear is that formal reasoning is not part of a transformer or similar model's function. It is in fact entirely probabilistic reasoning since the model itself operates via statistical inference. 

While the level of reasoning which can be achieved in this way is truly remarkable, we should remember that even humans perform probabilistic reasoning, both by nature and by necessity. 

We are not born as logic machines. Evolution so far has taught us to work with what we have in the time we have. We induct (generalise from observation) and abduct (infer the most likely conclusion) quite naturally and to great utility in daily life. See Lenat and Marcus 2023 for an overview of this. Could it be that LLMs have picked up some wisdom of the crowd here? It could be that a lot of what is not written can be inferred from enough examples of similar text.

After the discovery of formal logic (I am with Philip Wadler on discovery vs. invention), we have endeavored to apply it as much as possible. But even at our best, humans are fallible and hence formal theorem proving systems were made to assist us. This can sometimes beg the question that hard logic is required all of the time. We do not NEED to formally prove quite a lot of things. Yet many important things certainly ought to be.

Formal logic underpins the very computing we rely on every day. So much so that all of general purpose computing is based around it. Some might argue that it's bound by it - which is where statistical machine learning and later deep learning came about with spectacular results. In fact, according to Jensen Huang of Nvidia, "AI is not a chip problem, it is a computing problem". Which is to say that the future of generative AI, most of which is currently based on probabilistic methods, requires a change in computing paradigm altogether. 

So, where exactly is the boundary between the sea of "good enough" reasoning and hard formal proof? I get the mental image of where two large bodies of water meet yet cannot mix.

Recent attempts to ground reasoning with LLMs involve the use of some domain-specific knowledge graph. The problem is posed to the LLM which posits a solution, perhaps as a turn-based chain-of-thoughts, which are each verified against the knowledge graph. This solution can reduce the overall effort required vs. sheer exhaustive traversal of the graph. AlphaGeometry appears to use a similar approach of using the LLM in tandem with a formal mathematical engine to reduce the overall search space while applying a powerful backend. This is so far my closest image of what a near term general reasoning engine might look like. (Note that PALM-E SayCan also does something similar by using embodied and situation aware robotic affordances).

However, these approaches are still, currently, restricted by the knowledge and reasoning of the LLM driving the process. As a future step, one could imagine a mixture of experts (MoE) model, yet this only eases the constraint - admittedly, possibly by a wide margin.

Perhaps a generalized reasoning engine might take the form of some Large GNN (LGM, anyone?), running over some enormous conceptual graph, interposed between an initial LLM and domain-specific formalised engines. Natural language capability and semantic reasoning by the GNN could be fused, something like SayCan, to work around confounding from the interpretation of any relationships within the context of the query or task.

However, the knowledge problem remains. How large must an ontologically aligned total graph of human knowledge be? How could if be made? Eventually a system must be able to keep up with changing times, acquire new knowledge from sources of varying reliability and then verify that information for itself. What critical mass of such well-encoded knowledge is required to enable that?

There are other issues too. If we would like to individual such generalized AIs to reason together, is natural language enough to allow this? How would they handle or resolve differences of interpretation? Is some modified language, perhaps unintelligible to humans in real-time, required?

While I end with only speculation and questions, I feel that we are slowly and steadily encroaching on this problem and am thoroughly excited by all the progress.