Data, Metadata, and the Ship of Theseus: Ontology Modeling Concepts to Help You Diagnose IA Issues

I’m working on a big software program right now. It’s got at least seven engineering teams working on it in parallel.

They try to coordinate, but inevitably teams develop hyperlocal understanding of what the product is and how it works based on their view into it. And this creates messes.

For example: location. It’s an important concept in the product, but there’s disagreement about how to model it. One team sees all the ways location connects to other things. They consider location primary to the product and treat it as data.

A second team working on another part of the product sees location almost as an annotation on another object that’s more important to them. They consider it secondary and treat it as metadata.

This means that in one part of the product, users encounter location as data – there’s a screen to manage it and you can assign attributes to it. And in other places they see it as tags in a corner of the interface. Which is confusing – are these the same things? Are they different?

Conversations about this always come back to usage, which hasn’t gotten us to resolution. So, how can we shift that conversation and get to a clear answer of whether location should be considered data or metadata?

Today we’ll talk about how to use ontology modeling methods and perspective to give us a framework for unraveling thorny IA messes like this one. We’ll even discover an ancient Greek paradox in the depths of our IA along the way.

Let’s start with a definition of what an ontology is. An ontology is a model of the entities in a domain, their attributes, and the relationships between them.

In this example, our domain is mapping, for a mapping app. We’ve got some entities, Location and Building, arranged in a hierarchy, and we’ve got some attributes on each of those entities, and some relationships between them.

We also have instances, or data, where we assert values for those attributes and relationships.

Ontologies are formal. They’re mathematical or logical structures, strictly defined so that a machine can interpret the model. This makes them precise in a way that other models we create don’t have to be.

And this makes them interesting. And useful. Ontology principles give us precision tools we can apply to any system to understand and unravel semantic confusion. To look at this question of data vs. metadata from an ontology lens, let’s start by defining metadata.

Metadata is data about data.

This is maybe the most common definition of metadata. It’s true. But it sweeps a lot of complexity under the rug. It implies that data and metadata are objective things that are easy to tell apart.

But one system’s data is another system’s metadata. Context is everything. So, I like this definition better: Metadata is data about data within the context of a specific system.

To get a sense of what “context of a specific system” means, let’s look at how location information is modeled in some other systems.

Location is the primary concern of the mapping domain. It’s impossible to even imagine talking about a map without talking about locations. So, in a mapping app, location is likely treated as data.

Beyond that, looking at the experience Google Maps provides via its UI, you see the model needs to be able to capture attributes of a location to display to users like name, a description, whether you can bring your dog.

We’ll need to put locations in relationship with other entities in our model such as other locations or buildings. We know we’ll also need to show a rating and probably also need to track administrative information like when the location information was last updated. Things like this are typically captured as metadata, so the things they are about need to be data.

By contrast, in a photo app, while typically we want to know the location a photo was taken at, our app doesn’t have much to say about the location beyond its coordinates. We’re not saying anything about its attributes or relationships with other locations. So, it makes a lot of sense to model the location information as metadata.

We can still do things like search, sort, filter our photos by that metadata. What we can’t do, however, is attach metadata to that metadata. That would be metametadata, and that would be ridiculous.

We also can’t assert relationships between two bits of metadata, by the rules of our ontology language. So, we have some tradeoffs to make, depending on our context of use.

To determine what’s data and what’s metadata, we need clarity about which concepts our model needs to express and which concepts are outside of its boundaries. In ontology, we call this ontological commitment.

In practice, you can think of ontological commitment as the same thing as model scope. But the term does have another level of meaning and weight that comes from ontology’s roots in first-order logic. Ontological commitment basically says: these entities exist in the world of our model. These entities are necessary for our model to cohere and make sense. And anything excluded from our model isn’t necessary to create the meaning of our system.

Returning to our example apps, the ontological commitment of Google Maps model says: locations exist in this domain. Routes, buildings, events; these are required to make our model make sense.

And the ontological commitment of the photo app is: photos exist in this domain. That’s it. If you take a photo of a cat, the photo is in the domain, but the cat is not.

You might be able to query and get all photos with cats in them, but you wouldn’t expect your photo app to be able to tell you facts about a cat. By the same token, the location you took the photo from is not part of the ontological commitment.

Ontologists often use competency questions to determine which concepts are needed to make sense for a given context.

Competency here refers to the system’s ability to answer its users’ information needs. Basically, you start from the outside in: you identify the information you’ll need to retrieve from the system and then work backwards to create a model that supports that retrieval.

This tells you what concepts your model needs to commit to. It’s a method that works really well with UX. You can explore questions for each persona during user research, and then use them to make sure your model supports the information needs of all your users.

I like competency questions because they shift the discussion out of the realm of usage or perspective to the more fundamental question about whether or not a concept exists in our domain. And that is much harder for anyone to shoot down based on opinion.

Here are some competency questions for our product, along with whether or not we can answer them if we model location as data or as metadata.

Even with some generous scoring of the first two questions*, you can see that we can’t answer all of them if we model location as metadata.

`* While technically you could answer the first two questions with location modeled as metadata, this would potentially involve parsing all of the metadata in the system to find the answer which is a slow and inefficient way of doing things.

So, the competency questions give us a clear answer: For our context of use, we need to make location data. Let’s look at one more.

Same product. It’s got an object called a deployment: a template that defines a software installation: which servers the software is distributed to, how to configure it, stuff like that.

It’s also got an object that is basically the set of software apps that were installed by a specific template. Which is also called a deployment. And then — unfortunately — these things are both the input and the output of a process called: a deployment process.

So, maybe we should just change one of the labels. And, yes! We should. But there’s something else going on underneath and it’s a good thing to understand because it’s a pattern that pops up a lot and it can wreak havoc.

Deep in the information architecture of our modern software product is an ancient Greek paradox called The Ship of Theseus. You might know this one. Basically, Theseus’ ship was important to the Athenians, so they preserved it. As boards decayed or broke, they replaced them until nothing original remained. The paradox asks: is the rebuilt ship with no original parts in it the same ship as Theseus’?

This paradox asks something about where the identity of a thing comes from: does it come from the parts or the whole?

A strange thing about it: the parts have a different relationship with time than the whole. The parts wear out and are replaced. The whole is almost outside of time, or at least operates on a longer time scale.

Frank Duffy coined the term shearing layers to describe how different systems of a building operate on different timescales. As quoted by Stewart Brand in How Buildings Learn, Duffy said: “Our basic argument is that there isn’t any such thing as a building. A building properly conceived is several layers of longevity of built components.” https://en.wikipedia.org/wiki/Shearing_layers

Let’s look a little more closely at why our deployment mess is an example of a Ship of Theseus pattern. But “Deployment” is a abstract and confusing concept. So, let’s get more concrete and take the example of a switch in a company’s network.

We’ll call it NS1, network switch 1, because the spec that laid out the network and specified its components calls it that.

As with any network, a lot of people work on it and interact with it:

• Our network engineer handles day today operation of the whole network. She often refers to the original spec and calls this switch NS1. Just like in the plans.
• Our hardware tech installs and updates the network hardware. When he originally installed a switch at NS1, according to the spec, he recorded its make, model, and serial number in a database of assets. He refers to it by its serial number: A123.

If we ask the network engineer and the hardware tech to walk us over to the thing those labels – NS1 and A123 – describe, they would take us to the same piece of hardware. They would agree on its operating properties: it’s a 25G switch. It’s plugged in to Ethernet. It would seem like they were using different names to describe the same thing.

But say right after we walk away, the switch fails. Another hardware tech comes by and swaps out the switch for a new one that meets the same requirements for NS1 in the spec.

Now, if we ask the engineer and the tech whether the switch is the same as what they showed us previously, they’ll give us different answers. The engineer will say: sure, it’s still NS1. And the tech will have to say: no. The serial number has changed.

So maybe NS1 and A123 or B123 are two different things? But that seems wrong: two things can’t occupy the same space at the same time. If you touch the switch, you’re definitely touching one thing.

What’s happening is that we have a conceptual thing from our blueprint, NS1, and a physical thing, A123 or B123. And the paradox is: sometimes we need to treat them as the same thing. But other times, we need to treat them as different things.

My network switch example is a simplified transcription of this case study from the BORO ontology: https://borosolu tions.net/sites/d efault/files/LOA 2002 - What is Pump Facility PF101 (paper).pdf

From an ontology perspective, sometimes we need to treat the conceptual thing as an instance that we can assign values to. And other times we need to treat it like we normally would in our model: as a class that we create instances of.

Instances of a type 25GSwitch versus the class of all 25GSwitches as an instance itself.

Humans do this all the time; it’s fundamental to the way we think. We flip between these representations with no loss of meaning. If I tell you: all 25G switches transfer data at 25Gb/second; and I also tell you: this thing we’re looking at is a 25G network switch with a specific serial number. It’s not a problem.

You might wish we could stop talking about networking equipment. But there’s no logical fallacy here.

But in general, it’s hard to represent this in the systems we build. Our systems don’t handle that switch back and forth between representations well, or that ambiguity of one label that refers to two types of things. In ontology, making statements about the class of all switches moves us out of the model into the metamodel.

The metamodel is where we make statements about our model. Much as we saw with metadata, where we couldn’t attach more metadata or assign relationships, there are limits to what we can say here.

Partly because the ontological commitment of the metamodel is our model, not our domain.

And while different ontology languages allow different amounts of expressivity in the metamodel, ultimately we run into a fundamental limitation of modeling: if we could say anything about anything in our model, it wouldn’t be a model. It would be the world. Our models are always bound by ontological commitment.

Lewis Carroll and Jorge Luis Borges have both riffed on the idea of a map that’s as big as the world: https://en.wikipedia.org/wiki/On_Exactitude_in_Science

So, this paradox represents something that’s fundamental to human thought. We can’t “fix” it. It’s so fundamental we don’t realize we’re doing it. And we routinely encode in our systems.

But what we can do is recognize it so we can handle it in our systems, so it doesn’t create chaos. So, a couple of things to watch out for:

This pattern crops up whenever we’re modeling templates or blueprints and the things they instantiate. Or, as we say in object oriented UX: the cookie cutter and the cookie.

The part-whole pattern can be subtle. It might not be an obvious component, like our example of a switch in a network. The whole can take the form of a concept of a thing, and the parts can be the instances. Like the concept and instances of a deployment.

In this pattern, the parts and the wholes, or the concepts and instances, will have different relationships to time. Paying attention to that can help you recognize when you’re actually talking about two different things with one label.

Keep an eye out for transitions. That concept has to turn into an instance somewhere! In our case, the process diagram with the same label for the input and the output was a clear sign of the pattern. We’re not transitioning between different entities in the model here as much as moving from concept to actual.

A practice from object-oriented UX is to create instances as you create your model. If you find yourself with an instance that only uses a subset of the attributes you’ve defined, that’s a good hint that you might be talking about two different things with one label.

An even better approach, also from OOUX: instead of defining attributes and creating instances on your own, do it as a collaborative exercise with stakeholders to surface teams’ local definitions.

Object Oriented UX has a lot of ideas about how to recognize and model conceptual vs. actual things: http://www.ooux.com/

Once you’ve recognized the pattern, there’s no one right answer about what to do.

I want to give you three very different examples of how I’ve handled this in different contexts:

1. In the product I’m working on now, we wound up changing the name of one of the objects to deployment instance. We’ve just talked about how changing the name doesn’t solve the underlying problem, and that’s true. For various reasons, we can’t make any structural changes. But we can generate less confusion for users by changing the name. Remember, people mush this stuff up in their heads all the time – that’s how we got here! In this case, we just need to resolve things to a point where it’s clear enough for people to make meaning out of what we’re showing them in the UI.

2. We found a second instance of the pattern in our product. And we just left it. For now, we only reference it as a concept in one place, and it seemed like it would introduce more complexity to separate them out. But it feels like a huge win just to identify it and be able to track it going forward.

3. Because while this pattern has been pretty easy to manage in this project, that’s not always the case.

In a previous life, I worked on a product built on top of an ontology. We ran into this pattern, and it stopped us dead in our tracks. We couldn’t ship on time, and it delayed our release several weeks while we tried to figure out what to do. And we never really figured out what to do.

The difference there was that that system was highly concerned with the identity of a thing, and we couldn’t tell when we needed to be using the class itself as data. This can get very messy to implement. There’s an entire ISO standard about how to resolve this in systems that describe things like factories and oil rigs.

ISO 15926 describes an ontology designed to handle the challenges of managing data about a process plant and its components over time: https://readi-jip.org/wpcontent/uploads/2020/10/ISO_1592614_2020-09-READI-Deliverable.pdf

So, what about the robots? What does this mean for us in the era of large language models?

LLMs are predictive models about word frequency and co-occurrence. They don’t have a model of the semantics underlying the words.

What is the ontological commitment of an LLM hallucination? Or any statement? The machine isn’t producing a system of meaning that requires certain entities to cohere. It can be hard for our minds to remember that when we’re reading their outputs.

And LLMs can’t handle statements where a single label is used to implicitly refer to more than one type of thing. [1] Which is basically what we’re doing when we treat our classes as data.

LLMs can’t handle the flexibility of representation human brains take for granted. Just like any other digital system we make. So, we need to be clear that LLMs are not interpreting language. And we need to be careful about what kind of tasks we ask them to do for us. Because wherever human cognition and human language goes, we are bringing the Ship of Theseus paradox along with us.

Thank you, IAC!

Thank you. And thanks to the generosity of the many colleagues who helped me create this talk.