Assessment of expertise through neural activation pattern analysis in LLMs

Abstract

The performance benchmarks of Natural Language Processing (NLP) tasks have been overwhelmed by Large Language Models (LLMs), with their capabilities outshining many previous approaches to language modeling. Despite the success in these tasks and the more widespread use of these models in many fields of application, the understanding of how these models reach the outputs they achieve is lacking. The objective of this research is to study attributes of activation patterns of the neuron layers of these models that are related to how these models capture the underlying expertise in human written text. The research uses the LLM-MRI library to generate Neuron Region Activation Graphs (NRAGs) to represent the activation patterns across an LLM’s neural layers, and analyses them through various complex network metrics and a newly developed “center of mass” metric. Our results show that texts demonstrating different levels of expertise will generate measurably distinct activation patterns in LLMs, especially with greater variance of node strength. The proposed “center of mass” metric proved to be useful for locating the layers containing the most relevant expertise for the processed text. This research contributes to the field of Explainable Artificial Intelligence (ExAI) by offering new perspectives and tools to understand the internal workings of LLMs and improve their interpretability and reliability.