Relational data
In AI, we often see knowledge and the sensed world (e.g. from vision, touch, hearing) represented in relational form. Sets of binary relationships between two things (e.g. Mary likes apples; John likes apples) can be represented as a labelled, directed graph:
Figure 1. A graph of three nodes representing two binary relationships.
Large bodies of knowledge have been codified as labelled graph structures known as knowledge graphs and exemplified by the Google Knowledge Graph, which contains a vast amount of factual information about the world in which we live. A fragment from a knowledge graph might look like this:
Figure 2. A fragment of knowledge graph representing a collection of facts.
The nodes of the graph represent various kinds of entity, including: object categories, like 'fruit'; place names, like 'Paris'; images and dates. The edges represent relationships such as 'likes', and 'isa', meaning that one entity is an instance of another.
A knowledge graph can be used to infer new information. From the knowledge graph above, we could infer that Peter Brown likes russet apples.
Graphs can also be used to represent the relationships between the objects in a scene, as a precursor to recognising activities involving those objects. In the video below, both hands and a variety of solid objects on a table top have been tracked by computer vision:
The centroid of each object is represented by a white dot. We can see these dots as the nodes of a graph. The spatial relationships between objects are classified according to the distance apart and the rate of change of this distance (i.e. the rate at which two objects come together or move apart). The class label associated with each pair of objects is represented by the colour of the connecting edges.
Using machine learning on the dynamic graphs that result from lots of examples of people manipulating objects, a set of characteristic primitive activities can be discovered. One such example is shown below:
In an experiment, the activities emergent from videos such as the one above were as follows:
Figure 3. Emergent activity classes from machine learning.
This kind of learning based on graphs could be used for more abstract kinds of activities. The following video was used in experiments carried out in 1944 by Heider and Simmel [1] to demonstrate that people observe emotional behaviours even when the objects involved are simple geometric shapes.
Mathematical logic and automated reasoning
Mathematical logic provides formal mechanisms for performing inference on the kinds of relational data considered above. It enables us to reason automatically that some hypothesis about the world follows from a set of things that we assume to be true. There are extensions to mathematical logic that deal with probabilistic reasoning, computing the probability that some hypothesis is true from a set of things which may themselves be uncertain. The area of automated reasoning of this kind has a long tradition in AI.
Graph neural networks provide a complementary mechanism for performing inference on knowledge represented by graphs.