On Expertise and Expert Performance

Expert (Chambers Dictionary of Etymology)

What do expertise and expert performance have to do with this digital curation of knowledge creation project?
Well, if I can understand better what expertise is, where it comes from, and how it is constituted, I will have armed myself with a very valuable tool to identify how experts in the reading of ancient documents can be digitally supported in their task.
The Cambridge Handbook of Expertise and Expert Performance [1] is a very informative and rich book, which gathers findings on and around expertise as studied from various points of view: from Psychology, from Artificial Intelligence, from the Cognitive Sciences. Here are the highlights of my readings, presented, of course, within the framework of my own interests, that is, my aim to develop a piece of software which experts working with inscribed artefacts (including papyrologists and ancient near-east scholars) will find useful to conduct their research.

How to define Expertise?
Naturally, expertise is defined with respect to a specific domain; experts are specialists, they excel in a given domain. Yet regardless of the domain, there are two stances to define expertise and thereby experts. The first one states that expertise is a talent; this is the absolute approach, where experts are identified as those who produce exceptional results. The other one states that expertise is characterized by a high level of proficiency; this is the relative approach, where experts are those whose achievements and experience are greater than that of novices [2]. In that scope, Hoffman [3] defined the following proficiency scale (analogical to the craftsmanship stages as established in medieval times):

More generally, anyone in the range from 1 to 4 in this table is considered a novice (0 corresponds to the person completely ignorant in the studied domain), and 5 and 6 are experts. Those two stances are in my opinion complementary. The second one is pragmatic, and does by no means exclude the presence of talent.
In terms of what expertise intrinsically is and requires, the understanding of expertise is rather fluid, not only evolving with time and new research results, but also dependent on the domain, on the context, and on the intention behind the efforts to define it.

How to study Expertise?
First, why would one want to study expertise? Provided that expertise is a mixture of talent and ability, then understanding what the talent and/or the ability are made of is one way of informing the process of how to teach novices to become experts. One other objective in the study of expertise is to build computational models that can either emulate or support experts.
The trickiness in studying expertise as a general concept resides in that expertise cannot be dissociated from the domain in which it exists. I will present in detail in the next section the general traits that scholars have however been able to identify in experts; in this section, I will restrict myself to presenting the strategies that have been developed to study and identify those traits. These methods can be organized on an axis ranging from unstructured methods to structured methods, where I’m using the term structure as referring to the presence (or not) of a predefined workflow designed by the investigator for the experts being studied, rather than as a qualitative appraisal of the method itself; so that at the far end of the “unstructured” range of methods, one would find ethnographic studies of expertise, and at the far end of the “structured” range of methods, one would find a set of specific tasks defined and completed in lab settings and viewed as characteristic/representative of the tasks experts undertake in their “natural settings”. Many of those methods can be combined, and most of them set their focus on what is widely called Cognitive Task Analysis (CTA) [4] – by contrast with Behavioural Task Analysis (BTA).

• Ethnographic study (BTA – CTA); based mostly on observation of experts in their “natural settings”; uses contrast frameworks (practice vs process – behaviour vs function – activity vs task – invisible vs overt – documentation vs literal account – knowledge application vs tension resolution) and Multiple Perspective Analysis (from the point of view of persons, objects, settings, tasks, communities, temporality, networks of all that precede) [5]
• Unstructured interview (CTA)
• Think Aloud Protocol (CTA); experts explain what they do and why while they do it
• Retrospective task analysis (CTA); experts recounts one or several specific tasks that they have conducted in the past and how/why
• structured interview (CTA)
• constrained-processing tasks, limited information tasks (CTA)
• predetermined characteristic tasks (CTA)

Depending on how those methods are applied or combined, they can yield:

• critical decision maps (e.g. decision trees, coded transcripts using annotations such as: appraisal, cue, action, deliberation, contingency, meta-cognition)
• work domain analyses (e.g. abstraction-decomposition matrix)
• concept maps (e.g. relational diagrams between concepts and propositions)
• psychometric scores

So, in some sense, the study of expertise is the search for a model of expertise suited to the intended application, a model that can guide whoever uses it in their endeavour, be it   teaching or the building of expert systems. Further possible outcomes, based on the list above are: knowledge bases and domain ontologies (as understood by computer scientists).
Knowledge bases and domain ontologies are often part of expert systems. When it comes to building expert systems that imitate human thought or at least that have some cognitive ability, the general view is to adopt a model of expertise where expertise is made of two main components [6]:

• a knowledge base, where knowledge can be categorized as follows:
• factual knowledge – composed of textbook knowledge and “common” knowledge in the specific domain
• heuristic knowledge – made of more person-specific experience-based knowledge
• a knowledge representation and reasoning (KRR) framework, (e.g. an ontology) which can be  made of (possibly a selection amongst):
• a set of production rules – similar to First-Oder Logic rules, but that would accommodate uncertainty
• a structured object or schema that defines a taxonomy of the domain
• a problem solving model, such as inference engines
• an analogical reasoning scheme, like those developed in Machine Learning

In keeping with the scientific tradition, the breaking up into smaller components aims to facilitate their computational handling. I will comment in a future post on my take on KRR and on how the separation between knowledge bases and KRR might sometimes be difficult to make, as well as on the problems that might occur when it comes to encoding and codifying knowledge and reasoning.
Despite the variety of ways in which expert knowledge can be studied, it is essential to remember that:

“Expertise is not just about inference applied to facts and heuristics, but about being a social actor.” [5, p127]

Expertise is a situated activity. So that to understand expertise, and when undertaking knowledge elicitation in order to build expert systems, it is essential for knowledge engineers to keep in mind that they are undertaking “an epistemological enterprise” [6, p91] as much as an ethnographic study [5], where cultural and social contexts participate in the building of expertise.

What characterizes  Experts?
Following the exposition of the strategies developed to study expertise, here is now an attempt at summarizing the characteristics that experts exhibit, according to the Handbook [1]. I’m not claiming it’s an impartial view, as I’ve tried to categorize the series of characteristics evoked and described in each of the chapters in such a way that they can be seen as abstract expressions of specific traits [2,6-10].

• Experience. Experts have accumulated years of practice, which enable them to be efficient, to automate some tasks. It also entails that some shortcuts have been devised and that some of the processes have been internalized and have become implicit.
• Acquired knowledge. Experts know a lot in their domain. This can be assimilated to the cognitive ability known as crystallized intelligence (aka Gc in the psychometrics framework), it is a capacity to store data and facts relevant to their domain related to Long-Term Working Memory (LTWM).
• Knowledge organization and retrieval. Experts structure their knowledge. They are able to identify salient features and organize their knowledge into meaningful cognitive units, facilitating the dialogical relationship of LTWM and information retrieval (also considered part of Gc). The very definition of what saliency is for a feature depends on the domain and on what the best “handle” on the data is – where I’m using the term “handle” to express both reach-ability and representativeness of the data based on a feature.
• Modelling. Experts spend a lot of time assessing qualitatively the problem at hand, making it into an abstract and conceptual problem that can share properties with other already encountered problems and thus enable them to recall strategies to solve the current problem.
• Reasoning. Reasoning involves juggling with the knowledge and models at hand, possibly at a symbolic level. It involves strategies such as data-driven reasoning and hypothesis-driven reasoning (which seems to be used more by novices than by experts). It calls upon inferences (induction, abduction, deduction), analogy, consistency checking, counter-factual reasoning, and the handling of constraints, uncertainty and ambiguity. This can be assimilated to what is called fluid intelligence (Gf) in the psychometrics framework.
• Meta-cognition. Experts constantly and accurately self-monitor; they keep track of what they’re doing, check for errors in their reasoning, and are more resistant  to interruptions than novices.
• Opportunism and creativity. Experts are opportunistic with their sources of information, they adapt to the present problem. Regarding creativity and imagination, there are views according to which experts can be affected by functional fixedness, and thus lack in creativity. Yet, other views claim that creative thinking advances knowledge and thus enhances expertise (and vice versa expertise and skill promote creativity!).

Attempts at being objective in this categorization are rather futile in my opinion, provided that I am immersed in a specific culture and society and have my own specific intentions when drawing up this list, although, of course, one of my intentions was to summarize the findings exposed in the Handbook [1]. For those very same reasons of cultural bias all of these traits are affected by the context in which experts accomplish their tasks (despite the claim that Gc is affected by cultural bias but not Gf – a claim that doesn’t entirely convince me, possibly because the evaluation of Gf cannot be unbiased?). 

Around expertise: epistemology, knowledge, imagination, and embodied cognition.

To conclude this post, I would like to narrow down the subject, and refocus on my research endeavour, that is on the processes at play in the interpretation of ancient documents. In particular, I was intrigued by Voss’s mention, as in passing [9,p579] of three factors to evaluate the quality of an argument in History:

• acceptability of the evidence
• supportive-ness towards a certain claim
• consideration of opposing evidence

These three factors could be compared to the three characteristics that enable to evaluate the goodness of an argument according to Haack [11], in a more general epistemological framework:

• favourableness towards a claim: ranging from preclusive to conclusive (via supportive)
• independent security: ensuring that the full argument doesn’t entirely collapse if one piece of evidence is removed
• comprehensiveness: ensuring that all relevant evidence (supportive or otherwise) has been taken into account

And although these two characterizations cannot be mapped bijectively (i.e. with a one to one correspondence), they seem to be covering the same kind of grounds. The main reservation I have about those otherwise seductive theories of justifications is that none of them seems to be taking into account that the justification is produced by an expert, a person, a person with a body in a cultural context, and with their specific intentions and expectations. 

Do you think that one or the other characterization could accommodate the injection of the cultural and/or personal bias of an expert? How do you think that might be performed? Would that even be possible?

References:
[1]  K. A. Ericsson, N. Charness, P. J. Feltovich, and R. R. Hoffman, eds., The Cambridge Handbook of Expertise and Expert Performance. New York: Cambridge University Press, 2006. (ToC available here)
[2]  M. T. H. Chi, “Two approaches to the study of experts’ characteristics,” in  [1], ch. 2, pp. 21–30.
[3]  R. R. Hoffman, “How can expertise be defined?  Implications of research from cognitive psychology,” in Exploring expertise (R. Willimas, W. Faulkner, and J. Fleck, eds.), New York: Macmillan, 1998.
[4]  R. R. Hoffman and G. Lintern, “Eliciting and representing the knowldege of experts,” in [1], ch. 12, pp. 203–22.
[5]  W. J. Clancey, “Observation of work practices in natural settings,” in [1], ch. 8, pp. 127–45.
[6]  B. G. Buchanan, R. Davis, and E. A. Feigenbaum, “Expert systems: A perspective from computer science,” in [1], ch. 6, pp. 87–103.
[7]  E. Hunt, “Expertise, talent and social encouragement,” in [1], ch. 3, pp. 31–38.
[8]  P. J. Feltovich, M. J. Prietula, and K. A. Ericsson, “Studies of expertise from psychological perspectives,” in[1], ch. 4, pp. 41–67.
[9]  J. F. Voss and J. Wiley, “Expertise in history,” in [1], ch. 33, pp. 539–84.
[10] R. W. Weisberg, “Modes of expertise in creative thinking: Evidence from case studies,” in [1], ch. 42, pp. 761–87.
[11] S. Haack, Evidence and Inquiry: towards reconstruction in epistemology, ch. Foundherentism articulated, pp. 73–94. Oxford: Blackwell, 1993.