Root Causes of the Data Integration Problem

The Fundamental Phenomenon – Human Behavior

4/24/2005

Writing over a century ago, Emile Durkheim and Marcel Mauss recognized and documented the true root cause of today’s data integration woes. (Primitive Classification, 1903, page 5-6 as quoted by Mary Douglas in Natural Symbols, page 61-62)

At the bottom of our conception of class there is the idea of circumscription with fixed and definite outlines. 

Given that this concept of classification is the basis of logic, social discourse, religion and ritual, it should not be a surprise that it also comes into play when software developers write software. They make assumptions and assertions in the design, data and code of their systems that rely on a fixed vision of the problem. Applications may be written for maximum flexibility in some ways, and still there is an intent on the part of the developers to define the breadth and width of the system,  in other words, to bound and fix in place the concepts and relations supportable by the application.

The highly successful ERP products like SAP, JD Edwards, and ORACLE Financials allow tremendous flexibility to configure for different business practices. The breadth of businesses that can make these products work for them is very large. However, it is a common understanding in the ERP professional community (of installers) that there are some things in each product that just can’t be changed or accomplished. In these areas, the business is said to have to change to accommodate the tool. The whole industry of “change management” was born from the need to change the PRACTICE of business due to the ultimate limitations of these systems which were imposed by the conceptual boundaries their authors had to place upon them. (This is a different subject which should be pressed and researched). No matter how flexible the business system is, it is ultimately, and fundamentally, a fixed and bounded symbolic system.

 So how does this relate to my claim that Durkheim and Mauss have unwittingly predicted the current crisis of data integration? Because they go on to point out that: 

It would be impossible to exaggerate, in fact, that state of indistinction from which the human mind developed. Even today a considerable part of our popular literature, our myths, and our religions is based on a fundamental confusion of all images and ideas. They are not separated from each other, as it were, with any clarity. 

This “conceptual stew” is present in every aspect of life. The individual human mind is particularly adept at working within this broad confusion, picking and choosing what to believe is true based on internal processes. Groups of individuals, in order to communicate, will add structure and formality to certain portions thru discussion and negotiation. But this “social” activity is not always accompanied by strong enforcement by the community.

 As Mary Douglas (Natural Symbols, page 62) continues from Durkheim and Mauss, individuals in modern society (and increasingly this encompasses the global community) are presented with many different conceptual mileaus during the course of a single day. Within each person, she indicates,

 A classification system can be coherently organized for a small part of experience, and for the rest it can leave the discrete items jangling in disorder. Or it can be highly coherent in the ordering it offers for the whole of experience, but the individuals for whom it is available may enjoy access to another competing and different system, equally coherent in itself, from which they feel free to select segments here and there eclectically, not worrying about the overall lack of coherence. Then there will be conflicts, contradictions and uncoordinated areas of classification for these people.

 This not only describes a few individuals, but it is my contention that this describes the whole of human experience. Nowhere in the modern world especially, except perhaps when alone with oneself, will the individual find a single, coherent, non-contradictory and comprehensive classification of the world. Instead, the individual is faced with dozens or hundreds of partial, conflicting conceptions of the world. Being the adaptable human being her ancestors evolved her to be, however, this utter muddle is rarely a problem in a healthy person. The brain is a reasoning engine built especially to handle this confusion, in fact it thrives on it – the source of much that we call “creative” or “humorous” or “brilliant” is derived from this ever-changing juxtaposition and jostling of different, partial conceptions. Human society expands from the breadth and complexity created by these different classification systems. Communication between strangers depends on the human capacity to process and understand commonalities and fill in the blanks in the signal.

The very thing which defines us as human, our ability to communicate across fuzzy boundaries, is also that thing that creates and exacerbates the Data Integration Problem in our software. Our software “circumscribes with fixed and definite outlines” some small aspect of our experience. In doing so, it denies the fuzziness of our larger reality, and imposes barriers between systems.

Q&A: Meaning Symbol Sign and Mind (Part 1)

On one of my recent posts, a commentor named “psycho” asked me some very good questions. I decided I needed to respond in more detail than just a single comment reply. I respond in pieces below, so just for context, here is psycho’s entire original comment.

But if you take more meanings, and put them together to get yet another meaning. Don’t you feel like those meanings were again like symbols creating a new meaning?

In my understanding, every bit of information is a symbol – what is represented by the invididual neurons in the brain. And if you take all related bits (that is neurons, symbols), and look at it as a whole, what you get is meaning.

The sentence is a symbol, and it is made of word-symbols. And the list of word-symbols makes a meaning. Which, when given a name (or feeling), becomes a symbol, that can be further involved in other meanings.

I’ll respond to each paragraph in a separate post, in order to get all of my thoughts down in a reasonably readable fashion. Here is part one.

Construction of Symbols

But if you take more meanings, and put them together to get yet another meaning. Don’t you feel like those meanings were again like symbols creating a new meaning?

I try to make a very strong statement of the difference between symbols, signs and their “meanings”. Perhaps I’m being too analytical, but it allows my to think about certain types of information events in a way I find useful in my profession as a data modeller. So let me try to summarize here the distinctions I make, then I’ll try to answer this question.

First, in my writings, I separate the thing represented by a symbol from the thing used as the representation. The thing represented I call the “concept” or “meaning”. The thing which is used to represent the concept I have termed “the sign”.  A symbol is the combination of the two. In fact, a specific symbol is a discrete object (or other physical manifestation) built for the express purpose of representing something else. That specific symbol has a specific meaning to someone who acts as the interpreter of that symbol.

As I have come to learn as I continue reading in this subject area, this is a somewhat ideosyncratic terminology compared to the formal terms that have grown out of semiology and linguistics. To that I say, “so be it!” as I would have  a lot of re-writing to do to make my notions conform. I think my notions are comparable, in any case, and don’t feel I need to be bogged down by the earlier vocabulary, if I can make myself clear. You can get a feel for some of my basic premises by poking around some of my permanent pages, such as the one on Syntactic Media and the Structure of Meaning.

There is obviously a lot of nuance to describing a specific symbol, and divining its specific meaning can be a difficult thing, as my recurring theme concerning “context” should indicate. However, within my descriptive scheme, whatever the meaning is, it is not a symbol. Can a symbol have several meanings? Certainly. But within a specific context at a specific time, a specific symbol will tend to have a single specific meaning, and the meaning is not so fluid.

How do you express a more complex or different idea, then? It is through the combination of SIGNS which each may represent individual POTENTIAL concepts that I am able to express my thoughts to you. By agreement (and education) we are both aware of the potential meanings that a specific word might carry. Take for example this word (sign):

blue

When I show you that word in this context, what I want you to recognize is that by itself, I am merely describing its “sign”-ness. Those four letters in that combination form a word. That word when placed into context with other words may represent several different and distinct ideas. But by itself, it is all just potential. When you read that word above, you cannot tell if I’m going to mean one of the colors we both might be able to see, or if I might be about to tell you about an emotional state, or if I might describe the nature of the content of a comedian’s act I just saw…

While I can use that sign when I describe to you any of those specific meanings, in and of itself, absent of other symbols or context, it is just a sign with all of those ambiguous, potential meanings, but in the context of our discussion, it has no specific meaning.

It has a form, obviously, and it has been constructed following rules which

Photo of an Actual Stop Sign In Its Normal Context

you and I now tacitly understand. Just as a stop sign has been constructed following rules we have been trained to recognize.

Imagine now a warehouse at the Department of Transportation where a pile of new stop signs has been delivered. Imagine they are laid flat and stacked on a pallet, just waiting to be installed on a corner near you.

While they lay in that stack, they certainly have substance, and they each have the potential to mean something, but until they are placed into a proper context (at a corner by a road) their meaning is just as ambiguous as the word sign above. If you were driving a fork lift through the warehouse and came upon the pallet, would you interpret the sign right then as applying to you? Probably not! Could you say, just be looking at an individual instance of a sign, exactly which cars on which road it is intended to stop? No, of course not.

So this is the distinction between the sign and the meaning of a symbol. The sign is a physical construct. When placed into a recognized context, it represents a specific meaning. In that context, the sign will only carry that one specific meaning. If I make another instance of the sign and put it in a different context, while the signs may look the same, they will not mean the same, and hence I will have made two different symbols.

Just to be perfectly clear on the metaphor I’m presenting, here is a “pile” of signs (words) which I could use in a context to express meaning:

blue

blue blue

blue blue blue blue

Now let me use some of them and you will see that given a context (which in this case consists of other word signs and some typcal interpretations) I express different meanings (the thoughts in your head when you read them together):

once in a blue moon

blue mood

blue sky project

blue eyes crying in the rain

But make no mistake, while i have now expressed several different ideas to you using the same sign in different contexts, they are each, technically, NOT THE SAME SIGN AT ALL! Rather they are four examples of a type of sign, just as each of the stop signs on that pallet at the DoT are examples of a type of sign, but each is uniquely, physically its own sign! This subtlety is I think where a lot of people’s thinking goes awry, leading to conflation and confusion of the set of all instances of a sign with all of the concepts which the SET of signs represents.

To make this easier to see, consider the instance of the word (sign) “blue” above which I have colored red. That is a specific example of the “blue” sign, and it has a specific, concrete meaning which is entirely different from the word (sign) “blue” above which I have colored green.  The fact that both phrases have included a word (sign) of “blue” is almost coincidental, and does not actually change or alter the individual meanings of the two phrases on their own.

Finally, since I have belabored my nit-picking a bit, if I were to re-word your initial statement slightly to use the terminology I prefer on this site, It would change to:

But if you take more [signs], and put them together to get yet another meaning. Don’t you feel like those [signs] were again like symbols creating a new meaning?

And to this question, it should be clear, that my answer is “Yes, precisely: when you put other signs together, you create new meaning”.

Example of How Meaning Is Attached to Structure

What follows is a detailed example of the thought process followed by a software developer to create a class of data structures and how meaning is attached to those structures.

Consider that the meaning of one data structure may be composed of the collection of meanings of a set of smaller structures which themselves have meaning. Take the following description as the meaning to be represented by a structure:

An employee is a human being or person. Each employee has a unique identity of their own. Each employee has a name, which may be the same as the name of a different person or employee. Being human, each employee has an age, calculated by counting the number of years since they were born up to some other point in time (such as present day).Each person of a certain age may enter into a marriage with another human being, who in turn also has their own identity and other attributes of a person.

To represent this information using data structures (i.e., to project the meaning of this information onto a data structure), we might tie the various concepts about a human being/employee to a computer-based data structure. Recognizing that a human being is an object with many additional characteristics of which we might want to know about, we might choose to project the concept of “human beings” or “people” onto a relational table and the concept of a particular individual onto one of that table’s rows (or a similar record structure).

This table would represent a set of individual human beings, and onto each row of the table would be projected the meaning of a particular human being. Saying this again in a more conventional manner, we would say that each row of the table will reference a singular and particular human being, the all of the rows will represent the set of all human beings we’ve observed in the context of our usage of the computer system.

In a more mathematical vein, we would define a projection Þ from the set of actual human beings Α onto Š, (Þ(Α) |–> Š), the set of data structures such that for any α in Α where α is a human being, there is a record or row σ in Š that represents that human being.

A record data structure being a conglomeration of fields, each of which can symbolically represent some attribute of a larger whole, then we might project additional attributes of the human being, such as their name and identifier, to particular fields within the record. If σ is the particular record structure representing a particular human being, α, then the meaning (values) of the attributes of that person could be associated with the fields, f1..fn, of that record through attribute-level projections, ψ1..ψn for attributes 1 .. n.

To represent a particular person, first we would project the reference to the person to a particular row, Þ(α) |–> σ, then we would also project the attribute facts about that person onto the individual fields of that row:

ψ1(α.1) |–> σ.f1

ψn(α.n) |–> σ.fn

Projection onto Relational Structure

When modeling a domain for incorporation into computer software, the modeler’s task is to define a set of structures which software can be written to manipulate. When that software is to use relational database management systems, then the modeler will first project the domain concepts onto abstract relational structures defined over “tuples”. These abstract structures have a well-defined mathematical nature which if followed provides very powerful manipulations. The developer projects meaning onto relations in a conventional way, such as by defining a relation of attributes to represent “PERSON” – or the set of persons, and another relation of attributes to represent “EMPLOYEE” – or the set of persons who are also employees. Having defined these relational sets, the relational algebra permits various mathematical operations/functions to be applied, such as “JOIN” and “INTERSECTION”. These functions have strictly defined properties and well-defined results over arbitrary tuples. The software developer having projected meaning onto the individual relations, he is also therefore able to project meaning on the outcomes of these operations which can then be used to manipulate large sets of data in an efficient, and semantically correct way.

As the developer creates the software however, they must keep in mind what these functions are doing on two levels, at the level of the set content and at the level of the represented domain (the referent of the sets and manipulations). Thus the intersection of the PERSON and EMPLOYEE relations should produce the subset of tuples (records, etc.) which has its own meaning derived from the initial projected meaning of the original sets. Namely, this intersection represents the set of PERSONS who are also EMPLOYEES, (which is the same, alternatively, as the set of EMPLOYEES who are also PERSONS). This is an important point about software: the meaning is not simply recorded in the data structure but the manipulations of the data by the computer themselves have specific connotations and implications on the meaning of data as it is processed.

Representational Redundancy

As a typical practice in the projection of information onto data structures within the relational model, there will usually be a repetition of the information projected onto more than one symbol. In particular, the reference to the identity of a single person will be represented both by the mere existence of a single row in the table, and also by a subset of fields on the row which the software developers have chosen (and which the software enforces) for this purpose. In other words, under common software development practices, each record/row as a conglomerate entity will represent a single person. In addition, there will be k attributes (1 <= k <= n) on that record structure whose values in combination also represent that same individual. These k attributes make up the “primary key” of the data structure. The software developer will use and repeat these columns on multiple data structures to permit additional concepts regarding the relationship between that person and other ideas also being recorded. For example, a copy of one person record’s primary key could be placed on another person record and be labelled “spouse”. The attributes which make up the primary key often have less mechanical meanings as well (for example, perhaps the primary key for our person includes the name attribute. As part of the primary key, the name value of the person merely helps to reference that person. It also in its own right represents the name of the person.

How Meaning Attaches to Data Structures: A Summary

What follows is a high level summary of how humans attach meaning to various kinds of data structures within a computer. It will serve as a good baseline account, though certainly not an exhaustive one, providing a model upon which more detailed dicussion can begin. 

 Background Terminology

Computer systems provide functionality to support the performance and record of business processes. They do that through three inter-related features: DATA, LOGIC, and PRESENTATION. The presentation consists of information displays permitting both an information visualization aspect and an information capture aspect. The logic consists of several aspects, much of it having to do with support of the presentation and manipulation of displays, but also a lot of it having to do with creation, transformation and storage of data. Data consists of sets of symbols constructed in a systematic, regular fashion using a set of data structures. Different data structures are constructed to represent different aspects of the recorded activity. It is in the relationships between the macro and micro structures where the specific detailed information captured.generated by the business process resides. By following a codified, rigid construction of its data structures, the computer system is able to record multiple recurring instances of similar events. Through the development of fixed transformations using program logic, the computer system is able to make routine, conventional conclusions about those events or observations, and it is able to maintain and retain those observations virtually indefinitely.
Data is maintained and stored in DATA STRUCTURES. The more regular these data structures are, the more easily they are interpreted by a broad audience of software developers. In most situations, the PRESENTATION of the data captured by a system to the end user of that system is in a more directly understandable form than the way that information is stored in the computer.  (This statement is not only trivially true, but in a very deep sense too, since the computer actually stores everything using more and more complex sequences of binary digits. That’s a different subject than our current presentation.)  The data structures within the computer system typically exist in two, simultaneous forms, one intended to support human reasoning (through what is often called a “logical”, “abstract” or “conceptual” model) and one supporting manipulations by the computer. Most software developers today strictly deal with the abstract model of the data for design, coding, and discussion. (There are still some developers working in assembly level code, but even that is at a more abstract level than the actual electro-mechanical machinations of the actual hardware!)
An obvious observation, at least on its face, is that different computer systems will store data representing similar ideas using different structures. We need to keep this in the back of our minds as we progress through the rest of this discussion, but it will be more directly adressed in other entries.
 A final thought concerns sets of data of similar structure, called a POPULATION. A population of data consists of some set of data symbols, all constructed using the same data structure pattern which represents a set of similar ideas. The classification of populations of data structures applies to the DATA portion of systems, represents an analogous classification of sets of observed events external to the computer system, and is affected by and affecting the LOGIC and PRESENTATION portions of the computer system. A more detailed definition of the notion of a “population” will also be treated in separate sections.

Commonalities of Structure

Many computer systems, especially those built in support of business (or other human activity) processes, are constructed using a conventional system of abstract data structures. (When I say they are “conventional” what I mean is that the majority of software developers follow conventional patterns for the construction of data structures to represent their idiosynchratic subject areas.) Whether these structures are called “objects”, “tables”, “records”, or something else, they typically take the form of a heterogenous collection of smaller structures grouped together into regular conglomerations. Instances or examples of the larger collections of data structures will each be said to “represent” individual intances of some real-world conglomerate. Each of the individual component element structures of these conglomerations will each be said to represent the individual attributes or characteristics of the real-world conglomerate object. In order to permit efficient processing by the computer,   instances of similar phenomenon will be represented by the same kind of conglomeration.
Typically, business systems will be based on a data structure called a RECORD.  Records consist of a series of “attribute data structures” all related in some fashion to each other. (A more complex structure called an “object” still has record-like attributes combined together to represent a larger whole, the nuances and variation of object-based representation is a subject for later.)  Each RECORD will stereotypically symbolize one instance of a particular concept. This could be a reference to and certain observed details of a real-world object, or it could be something more ephemereal like observations of an event. For example, one “PERSON” record would represent a single individual person.
RECORDS themselves consist of individually defined data elements or FIELDS. Each RECORD of a particular type will share the same set of FIELDS. Each FIELD will symbolize one kind of fact about the thing symbolized by the RECORD. For example, a NAME field on a PERSON record will record what the represented individual’s name is, at least as it was at the time the record was created. 
The set of all records within a system having the same structure will typically be collected and stored together, often in a data structure called a TABLE. Each TABLE will symbolize the set of KNOWN INSTANCES of whatever type of thing each record represents. TABLES are also described as having ROWS and COLUMNS. Each row of a table is one RECORD. The set of shared element-attribute structures across the set of  rows can be described as the “columns” of the table. Each column represents the set of all instances of a FIELD in the table, in other words, the same field across all records. Tables are a commonly used data structure because they readily support interpretation using relational algebra and set theoretic operations, as well as being easily presented and understood both by human and computer.  

Basic Data Structures and Their Relationships

The nomenclature of “record”, ” table”, “row”, “column” and “fields” describes the construction building blocks of an abstract syntactic medium whose usage permits humans to represent complex concepts within the computer system. By assigning names to various collections and combinations of these generic structures, humans project meaning onto them. Using diagrams called “data models”, a short hand of sorts allows the modeler to describe how the generic tables and fields relate to each other and what these relationships signify in the external world. These models also, by virtue of the typified short hand they use, allows for the generation of computer logic that can be applied to a database to support certain standard operations and manipulations of the data generated by a computer system.

Traditional data modeling results in the creation of a data dictionary which relates each structural element to a particular kind of concept. Every structure will be given a name, and if the developers are diligent, these can be associated with more fully realized text descriptions as well. Some aspects of the data structures are not described, at least typically, within a data model, such as populations or subsets of records with similar structures.

Traditional data dictionary entries record name and description of the set of all structures contained in a table. Using a set of structures to represent a set or collection of similar objects is itself a symbolic action. So not only does each row in a table represent one instance of some type of thing, and each column represents one observed (or derived) fact or attribute of that instance, but the collection of all instances of these row data structures also represents the logical set or population of these things.

The strategy for applying meaning to these data structures begins when the decision is made to treat the entirety of each record as the representation of a member of a population of like things. Being similar, then, a set of fields is conceived to capture various detailed observations regarding the things. These fields are intended to capture details about both how each thing is different from the other things in the collection, but also how different things may share similarities. Much of the business logic of the application system will be consumed by the comparisons between individual things, and the mathematical derived counts (and other metrics) of those sets of things (and of subsets within). Using the computer to compare the bit sequences contained in each field, the computer will indicate whether these contents are the same or different between different instances. Humans will then interpret the results of these comparisons by projecting the conclusion out of the computer and into the conceptual world.

For example, let’s say that we have defined the computer sequence “10101010” to represent a reference to a specific person, “Julie Smith”. If we take two different instances of bit sequences and compare them in the computer, the computer will tell us if they are the same or not. As humans, we would then interpret the purely electro-mechanical result which the computer calculated that “10101010” and “10101010” are the same as an indication that the two instances of these sequences represent the same specific person. Likewise, we would interpret a computer result indicating that two bit sequences were not the same as an indication that different people were being referred to.  This type of projection of meaning from mechanical result to logical inference is fundamental to the way humans use computers.

The specific number of fields and their bit sequence representations (data types)  that are developed within a computer application is entirely dependent on the complexity of the problem domain and the attributes of the objects required to reason over that domain. However, no matter how simple or complex, it is the projection of meaning onto the representation of these attributes in the computer and the projection of an interpretation onto the results of the computer comparisons of the physical representations which makes the computer the powerful engine that it is in our society.

How Row Subsets Represent Subpopulations
How Row Subsets Represent Subpopulations

 

How Community Changes The Artist’s Conception

The Artist and the Standard Interpretation

The Artist and the Standard Interpretation

  • The Artist creates her artwork, with a particular symbolic meaning in mind.
  • The Art Dealer/Gallery Owner tries to explain what the artist had in mind.
  • The Art Critic sees something somewhat different by projecting his own notions on the work.
  • The Art Historian synthesizes what she’s heard, and unwittingly, and unbenownst guesses some of the original intent.
  • Ultimate truth is the one written by History, so over time, this final interpretation becomes the accepted meaning.

 

Context Shifting Is Easy

Today’s discussion asks that you perform a thought experiment.

Imagine that you are sitting in a room with a bunch of other people. All of your chairs face to the front of the room where there is a large desk. A young woman walks in with a stack of papers and places them on the desk. She picks up a piece of chalk from the desk, then, still standing, she turns to face all of you, smiles and begins to speak.

Right here I’m going to pause the narrative and ask that you consider the situation. Imagine it in your head for a moment. What is the context Ive described?

So what do I mean by context? Well if I were to say that our story so far is a very familiar context for most of us, one we all remember from childhood: an elementary school classroom, then here are some of the things you might expect to happen.

Having now stated a context, you, dear reader, should have images of yourselves sitting quietly in your desks while your teacher imparts some lesson. You also already know many of the basic ground rules of being in a classroom:

  • Pay attention to the teacher
  • Take notes
  • Don’t speak unless the teacher calls on you
  • Raise your hand if you have a question or comment and the teacher will call on you

Do you recognize this context? Feels familiar and confortable, right? Great! Let’s hold this thought now and count slowly to twenty while we let the memories of this context play about in our heads.

Really, start counting, or you won’t get the total effect:

1, 2, 3, 4, 5

6, 7, 8, 9, 10

11, 12, 13, 14, 15

16, 17, 18, 19, 20

Now let me throw you a little curve ball and tell you that you’ve been thinking about this in the wrong way. The situation I described is not really a classroom and that woman is not a teacher. She’s an actress, presenting a one-woman show about a famous teacher. The desk is a set, the papers just props. You are not in a classroom, you are in a theater made to appear as a classroom. This is just a play and you are a member of the audience. In fact, so there’s no doubt in your mind about this, you suddenly remember you put your ticket stub in your front pocket.

Did you feel that grinding sensation in your head as you read these last few sentences? That shifting from the classroom to the theater context – you should actually be able to feel it happen in your mind. The fact that even this little bit of information has allowed you to sense a shift in context is not a trivial matter. Usually, when you switch contexts like this, it is never so palpable or apparent. We humans are switching contexts all of the time, sometimes in the same sentence. It is one of our particular talents to recognize and adjust our conceptualizations at will when the context changes.

We have just completely switched contexts and you didn’t even need to lift a finger, did you? Just by my saying “this is a play” your expectations have completely changed. Now that we’re in the “performance context” what has happened to our mutual expectations. First of all, the roles have shifted, instead of a teacher, our woman is an actress, you, dear reader, are not students you are an audience. As a member of the audience (especially an audience witnessing a play about a teacher) here are some of the different expectations you may now have:

  • If you raise your hand, you may get an usher, but the actress will not respond to you
  • While you will still sit quietly and listen, the expectation is that at the end of the performance, you will clap your hands
  • The actress will provide the audience (hopefully) with an entertainment

So, shifting contexts is easy. And thus, I end this little monologue by pointing out that really, dear reader, we aren’t in a theater either. Instead, we’re sharing a context called “reading a blog entry”. I hope you enjoyed this little exercise!

Bridging Contexts

If it’s true that every human grouping can form its own context, how can communication occur between different groups? If one group defines a set of symbols using some set of concepts and a syntactic media that is different from those of another group, as a practical matter, how can the chasm be spanned? The answer is through the development of bridging contexts.

The following figure depicts several common strategies, each with its particular benefits and drawbacks.

Three Types of Bridging Contexts

Three Types of Bridging Contexts Within One Corporate Organization

There are three basic forms of bridging contexts. First and perhaps the most common in the real world is the creation of a specific, point-to-point bridging context through discussions/negotiations between the representatives of the two specific contexts. Most organizations take this approach because it simplifies, focuses and shortens the discussion, leading to faster turn-around. All application and data interfaces that are custom-built as point-to-point connections, no matter what the actual transmission protocol or language used, fall into this category.

The second form of bridging context occurs when two groups rely on a pre-existing, parent context to act as the bridge. The parent context may push a common context down onto the previously individual contexts, or the two contexts may appeal to the parent to resolve the conflict. In either case, the result can be that the child contexts become absorbed by the parent context, thus eventually what began as a bridging context becomes the entire context. These forms of bridging contexts are often common in such situations as corporate mergers, enterprise architecture initiatives, and business process reengineering projects.

The third form of bridging context is found whenever an organization selects a third-party standard as a communications protocol. In these cases, the organization creates a bridging context between itself and the external standard, including mapping its symbols into those of the standard. Theoretically, once completed, the organization can use such a bridging context to communicate with other organizations that have likewise built bridges to the standard. In practicality, however, it is not uncommon that organizations will bias their bridging context to their own point of view. When this happens, the external standard devolves into mere syntax, and other organizations must create new, subtle bridging contexts (a la form number one) in order to communicate successfully with this organization. This was a common occurrence in the heyday of Electronic Data Interchange (EDI), and still occurs today even with more modern, XML-based standards.

While proponents of standards bodies decry other approaches, it must be stated that the third form of bridging context is also the most complicated to develop, as well as requiring the longest amount of time to establish, and is often the hardest to maintain. The reason for this is that it requires so many more people to define, and for most situations, the key to its success is also its biggest drawback, namely that the context is defined externally to the organization. Thus, the interplay among the membership of the standards body creates the external context. The organization has a business activity establishes the local context. The humans involved in establishing the bridging context must be able to translate from the local context to the external standard. There is always a risk that these individuals will misunderstand the external standard and translate their local context to it incorrectly. In addition, the bridging context must be maintained constantly as changes occur both in the standard and in the internal organization. At least within the local context, it is more likely that a change will be noticed.

In addition to EDI and XML protocols, other examples of the third form of bridging context would include Semantic Web approaches, but also such mundane approaches as the use of ERP systems, or any other packaged application where a fixed syntactic media is presented.

Overlapping Context and Fuzzy Edges

Parent-Child Context Relationships: Intersection/Union

3/1/2005

The following figures depict some notional ideas for how to graphically describe some of the interesting relationships among contexts as they occur in a large, formal organization. The idea occurred to me that there must be some way of describing the similarities and differences in the concepts and discourse of the various subgroups of an organization (any organization). In the diagram, each oval represents a defined organizational group established by the business to allocate and accomplish all of the work necessary for the business to function. Each oval within another oval represents a specific group of individuals working in that business, until we reach the largest oval representing all employees in all groups. Even this largest oval exists in a larger context, that of the culture at large.

The discussion which follows touches on some incomplete ideas about how the concepts, signs and symbols within a given context relate to those of both smaller child and larger parent contexts.

Graphical depiction of Parent Child Contexts

Above: A Bird's Eye View of Nested Contexts; Below: Cross Section View of Nested Contexts

“Inheritance” of concept flows down from the broadest context down to the lowest context. This is not like the inheritance of properties in an object oriented paradigm, so the term may need to be changed. The idea really is that in the absence of an explicit statement of a concept in a lower level context, the members of the community may defer to the definition of that concept from one of the broader contexts that exist above them. In other words, the larger community of humans may have defined the concept and the more detailed context may neglect to reiterate the concept, preferring instead to use the larger context’s definition.

On the other hand, any concept defined in a broader context may be re-defined at a more detailed level. This may or may not be intentional, or even noticed by either members of the larger context or the more insular context. When noticed, it still doesn’t typically cause a problem in normal human discourse, as the humans are able to translate between each context, and hold in their minds each definition.

Contexts at different levels that do not share the same lineage may define a concept in different ways. If their members do not interact under normal circumstances, then there is still not a problem of communication or data integration. However, problems arise out of this layering and locality-driven conceptualization when the information must be shared, either tete-a-tete through direct interface (as happens in workflow integration problems) or through some roll-up to a common conceptual, parent context (as happens in reporting and business intelligence problems). This is the origin of the “single version of the truth” goal that many organizations now take as a given, best practice.

“Inheritance” of concepts flows down. What this means is that concepts defined in the parent’s broader context may still hold meaning in the more narrow child context. Exceptions/replacements are not limited to replacing concepts from the immediate parent, but can happen with any concept above. Each context layer, almost by definition, will define concepts that are uniquely their own, as well. This is one of the sources of intra-organization argument and confusion, as the same terms (syntactic medium) may be used to refer to two slightly (or even grossly) divergent ideas within the same corporate context.

Not every symbol will be meaningful in every child context, the process of transference of concepts can filter out concepts as well as borrow them. At each contextual layer, shared structure may be given different meanings. Lack of specificity/explicitness of definition at a layer does not imply automatic inheritance from above, as it can also reflect a vagueness of thought or lack of agreement about a fringe aspect.

The vacuum created, however, tends to favor the wholesale borrowing of the concept from the parent context.

Each context layer is complete in its own right. The sizes shown in the diagram suggest a size of content but this is just an artifact of the notation. A child context may define an infinite number of concepts over time, just as its parent context does. Theoretically, each context could be depicted or described in full without reference to the broader parent contexts.

Not every concept defined within any particular layer will wind up represented within some application software used by the humans participating in that context. However, if the humans in that context have acquired software to support their activities, the concepts within that system will naturally conform to the context, although they may force the context to be changed to reflect limitations and capabilities that the software imposes.

The reality is of course much more complicated than the diagram suggests. Since the context at each level is defined by the humans who inhabit and communicate within it, new members may introduce or adapt concepts from other contexts that are unrelated to the hierarchy of autonomy and control. Rather than attempt to trace the origin point of concepts across all contexts, it is recommended that these few concepts be considered  either of local origin, or as part of a bridging context between the context and the context of origin. This will have to be chosen only based on the value to be gained from either point of view.

Bridging contexts are new contexts established to bridge between some subset of concepts from each of two different contexts. These are established when new information communication between the two contexts is required. The bridging context can be recognized by the relative sparseness of the conceptual inventory, and by the fact that the lineage of the concepts is limited to two (or perhaps a handful at most) otherwise disjoint contexts.

Most transaction oriented interfaces, as well as any data interface between two functionally disparate systems (of any type) are defined within a bridging context limited to just the mediating symbols.

Brass Tacks and Comparability

So I thought I should try to explain “comparability” very simply. Reading my previous posts, which were derived from larger texts, I spend a lot of time saying a lot of generalities, and I think the main point is getting missed. So here’s me getting down to brass tacks on the subject.

A computer CPU is a very basic electrical device. Send it a stream of electrons and a command to “add”, and it returns another stream of electrons representing a purely “mechanical” (i.e., unintelligent) electrical result. That CPU doesn’t know anything about semantics, or whether the switches and gates it opens and closes should appropriately be applied to those particular data streams. It just does what it was designed to do given that particular sequence of electron streams. If the streams are comparable before they get to the CPU, then the output will be meaningful. If they are not comparable, then the output (and being a CPU, there will be some output) will not be meaningful.

So the job of the software is to manipulate each symbol before presenting it to the CPU. In particular, the software needs to take each symbol and replace it with one that MEANS the same as the original symbol, but which will present itself to the CPU as COMPARABLE to the other symbols.

Comparability has to be put into the computer, through the software, by a human being. In particular, it is the human who understands when one data stream is not comparable to another, and it is the human being who writes the code to change one stream so that it becomes comparable to the other.

So what really are we talking about? Let me make a non-computer example to show the point.

2 + 00000010 = IV

If I take a pencil and write the above string of characters on a piece of paper, and show it to another computer programmer, after a few moments, I would expect that person to agree that this is a correct mathematical statement

 two plus two equals four

Part of the success of the person in understanding the original statement is that they are able to parse each symbol in the string, interpret the MEANING of each symbol, then translate each into COMPARABLE numeric ideas.

If the computer CPU could experience each symbol as I’ve written it (let’s agree that each of the symbols depicted here would have similar diversity of structure in the computer as they do here on the page), then we can immediately grasp what comparability is. The CPU does not know what the symbols mean, it cannot make the interpretation just by looking at the symbols as they are presented and come to the same conclusion as the human. 

If we look at what I, the human did, to provide you, the reader, with a more readable version of the equation, I replaced each symbol with another one that meant the same, but which appeared as mutually comparable symbols:

  • 2   –>  two
  • +  –>  plus
  • 00000010  –>  two
  • =  –>  equals
  • IV  –>  four

Before the CPU can compare the symbol “2” to the symbol “00000010”, they must both be replaced with two other symbols, each with the standard interpretation of “two”. These new symbols must be structured to flow through the CPU in such a way that their very structure is modified by the CPU to create a third symbol whose standard interpretation has the meaning “four”. The “plus” symbol must be translated into the CPU’s “ADD” instruction, and the “equals” symbol is represented by the stream of electricity leaving the CPU with the resulting symbol.

Comparability: How Software Works

Back in 1990, I was working on a contract with NASA building a prototype database integration application. This was the dawn of the Microsoft Windows era, as Windows 3.0 had just been released (or was about to be). Oracle was still basically a start-up relational database vendor trying to reach critical mindshare. The following things did not yet exist which we take for granted today (and even think of as kind of out dated):

  • ODBC – allowing standardized access to databases from the desktop
  • Microsoft Access and similar personal data management utilities
  • Java (in fact most of the current web software stack was still just the twinkles in the eyes of their subsequent inventors)
  • Message-based engines, although EDI techniques existed
  • SOA and XML data formats
  • Screen-scrapers, user simulators, ETL utilities…

The point is, it was still largely a research project just to connect different databases that an enterprise might be using. Not only did the data representational difficulties that we face today exist back in 1990, but there was also a complete lack of infrastructure to support remote connection to databases: from network communication protocols, to query interfaces, to security and session continuity functions, even to standardized query languages (SQL was not the dominant language for accessing data back then), and more.

In this environment, NASA had asked us to prototype a generic capability that would permit them to take user search criteria, and to query three different database applications. Then, using the returned results from the three databases, our tool was to generate a single, unified query result.

While generally a successful prototype, during a critical review, it became clear to NASA and to us that maintaining such an application would be horribly expensive, so the research effort was ended, and the final report I wrote was delivered, then put into the NASA archives. It is just as well too, because within five years, much of the functional capabilities we’d prototyped had started to become available in more robust, standards-based commercial products.

What follows is a handful of excerpts from the final report, which while now out of context, still expresses some important ideas about how software symbols actually work. The gist of the excerpt describes how software establishes the comparability and sometimes the equivalence of meaning of the symbols it manipulates.

In a nutshell, software works with memory addresses with particular patterns of voltage (or magnetic field direction) representing various concepts from the human world. Software is constantly having to compare such “structures” together in order to establish either equivalence of meaning, or to alter meaning through the alteration of the pattern through heavily constrained manipulations. The key operation for the computer, therefore, is to establish whether or not two symbols are “comparable“. If they are not comparability, quite literally, then the computer cannot reliably compare them and produce a meaningful result.

Without further ado, here are the important excerpts from the research study’s final report, which I wrote and delivered to NASA in November 1990.

“Database Integration Graphical Interface Tools, Future Directions and Development Plan”, Geoff Howe, November 1990

2.2 The Comparability of Fields

There are many kinds of comparisons that can be made among fields. In databases, the simplest level of comparability is at the data type level. If two fields have the same simple data type (e.g., integer, character, fixed string, real number), then they can be compared to each other by a computer. This level of comparability is called “basal comparability”. Thus, if fields A and B are both integers, they can be combined, compared and related in any way appropriate for two integers.

However, two elements meeting the qualification for basal comparability may still be incomparable at the next level, that of the syntactic level. The syntactic level of comparability is that level in which the internal structure of a field becomes important. Examples of internal formats which might matter and might be important at this level include date formats, identification code formats, and string formats. In order to compare two fields in different formats, one or the other of these fields would have to be converted into the other format, or else both would have to be converted into a third format. The only meaningful comparisons that can be made among the fields of a database or databases must be made at the syntactic level.

As an example, suppose A is a field representing a date in Julian format, and suppose B is a field representing a date in Gregorian format. Assuming that both fields are stored as integers, comparing these dates would be meaningless because they lack the same syntactic structure. In order to compare these dates one or the other of these dates would have to be converted into the other format, or else both would have to be converted into a third format.

Unfortunately, having the same syntactic structure is not a guarantee that two fields can be compared meaningfully by a computer process. Rather, syntactic comparability is the minimum requirement for meaningful comparison by computer process. Another form of comparability must be incorporated as well, that of semantic comparability. Semantic comparability is based on the equivalence of the meanings attached to the contents of some pair of data items. The semantics of data items are not readily available to computer processes directly; a separate description in some form must be used to allow the computer to understand the semantic equivalence of concepts. Once such representation is in place, the computer should be able to reason over the semantic equivalence of concepts.

As an example of semantic comparability consider the PCASS fields, ITEM PART NUMBER from the FMEA PARTS table of the PCASFME subsystem, and CRIT_LRU_PART_# from the CRITICAI LRU table of the PCASCLRU subsystem. Under certain circumstances, both of these fields will hold the part numbers of “line replaceable units” or LRUs. Hence, these fields are semantically comparable. Given a list of the contents of ITEM PART NUMBER, and a similar list for CRIT LRU PART #, the assumption can be made that some of the same “line replaceable units” will be referenced in both lists.

Semantic comparability is useful when integrating data from different databases because it can be used to indicate the equivalence of concepts. Yet, semantic comparability does not imply syntactic comparability, and thus both must be present in order to satisfactorily integrate the values of fields from different databases. A definition of the equivalence of fields across databases can now be offered. Two fields are equivalent if they share the same base type; if their internal syntactic structure is the same; if their representational domains are the same; and if they represent the same concept in all contexts.

2.3 Heterogeneous Data Dictionary Architecture

 The approach which seems to have the most documentary support in the research for solving the integration of heterogeneous distributed databases uses a two-tiered data dictionary to support the construction of location-independent queries. The single data dictionary, used by both the single-site database management system, and the homogenous distributed environment, is split in two across the physical-conceptual boundary. This results in a two-level dictionary where one level describes in detail the physical fields of each integrated database, and the second level describes the general concepts stored across systems. For each unique concept represented by the physical level., there would be an entry in the conceptual level data dictionary describing that concept. Figure 2 shows the basic architecture of the two level data dictionary.

As an example of the difference between the conceptual and physical data dictionary levels, consider again the field PCASFME.FMEA PARTS.ITEM PART NUMBER. This is the full name of the actual field in the PCASS database. The physical level of the data dictionary would have this full name, plus the details of how this field is represented (character string, twelve places long). The conceptual level of the data dictionary would contain a description of the contents of the field, and a conceptual field name, “line replaceable unit part number”. Other fields in other tables of PCASS or in other databases may also have the same meaning. This fact poses the problem of mapping the concept to the physical field, which will be described below. Notice, however, how much easier it would be for a user to be able to recall the concept “line replaceable unit part number”, as opposed to the formal field name. This ease of recall is one of the major benefits of the two-level data dictionary being proposed. Two important relationships exist between the conceptual and physical data dictionaries. One of the relationships between fields of the conceptual level data dictionary and fields of the physical level data dictionary can be characterized as one-to-many. That is, one concept in the conceptual data dictionary could have many physical implementations. Identification of this type of relationship would be a matter of identifying and recording the semantic equivalences across system boundaries among fields at the physical level. All physical fields sharing the same meaning are examples of this one-to-many relationship.

Within the PCASS system, the concept of a line replaceable unit part number” occurs in a number of places. It has already been mentioned that both the ITEM PART NUMBER field of the FMEA_PARTS table, and the CRIT LRU PART # field of the CRITICAI_LRU table, represent this concept. The relationship between the concept and these two fields is, therefore, one-to-many.

The second type of relationship which may also be present, depending on the nature of the existing databases, relates several different concepts to a single field. This relationship is characterized as “many-to-one”. Systems which have followed strict database design rules should result in a situation where every field of the database represents one and only one concept. In practical implementations, however, it is often the case that this rule has not been thoroughly implemented, for a variety of reasons. Thus it is more than likely, especially in large database systems, that some field or set of fields may have more than one meaning under various circumstances. Often, these differences in meaning will be indicated by the values of other associated fields.

As an example of this type of relationship, consider the case of the ITEM PART NUMBER field of the PCASS table FMEA PARTS in the FMEA dataset one-more time. This field can have many meanings depending on the value of the PART TYPE field in the same table. If PART TYPE is set to “LRU”, the ITEM PART NUMBER field contains a line replaceable unit part number. If PART TYFE is set to “SRU”, the ITEM PART NUMBER field actually contains a shop replaceable unit part number. Storing both kinds of part numbers in the same structure is convenient. However, in order to use the ITEM PART NUMBER field properly, the user must know how to read and set the PART TYPE field to disambiguate the meaning of any particular instance of the record. Thus, the PART TYPE field in the physical database must hold either an “SRU” or “LRU” flag to indicate the particular meaning desired at any one time.

In the heterogeneous environment, it may be possible to find a different database in which the same two concepts which have been stored in one filed in one database, are stored in separate fields. It may in fact be possible that in one or more databases, only one of the two concepts has been stored. This is certainly the case among the separate data sets which make up the PCASS system. For example, in the PCASCLRU data set, only the “line replaceable unit part number” concept is stored (in the field, CRIT_LRU_PART_#). For this reason, the conceptual level of the data dictionary must include both concepts. Then there must be some appropriate construct within the data definition language of the data dictionary system which could express the constraints under which any particular field had any particular meaning. In order to be useful in raising the level of data location transparency, these conditional semantics must be entered into the data dictionary using this construct.

It is obvious now that the relationship between entries in the conceptual data dictionary and the physical data dictionary is truly many to many (see Figure 3). To implement such a relationship, using relational techniques, a third major structure (in addition to the set of tables supporting the conceptual data dictionary and the set of tables supporting the physical data dictionary) must be developed to mediate this relationship. This structure is described in the next section.

2.3.1 Conceptual – Physical Data Mapping

As an approach to implement this mapping from conceptual to physical structures, a table must be developed which relates every concept with the fields which represent it, and every field with the concepts it represents. This table will consist of tautological statements of the semantic equivalence of physical fields to concepts. A tautology is a logical statement that is true in all contexts and at all times. In thiis approach, the tautologies take the following form (please note that the “==” operator means “is semantically equivalent to”, not “is equal to”):

 normalized field f == field a from location A

 The normalized field f of the above example corresponds directly to an entry in the conceptual data dictionary. We call the field, f, normalized to indicate that it is a standard form. As will be described later, the comparison of values from different databases will be supported by normalizing these values into the representation described in the conceptual data dictionary for the normalized field.

Conditional semantics must now be added to the structure to support discussion. Given a general representation for a tautology, conditional semantics may be represented by adding logical operations to the right side of the equivalence. Assume that a new database, D, has a field, d1, which is equivalent to the normalized field, f, but only when certain other fields have specific values. Logically, we could represent this in the following manner:

normalized field f == field d1 from location D iff
field d2 from location D = VALUE1 AND
field d3 from location D = VALUE2 AND …
field dn from location D opn VALUEn

 In more general terms, the logical statement of the tautology would be as follows:

 R == P iff  E

where R is the normalized field representation, P is the physical field, and E is the set of equivalence constraints which apply to the relation. In our part number example, the following tautologies would be stored in the mapping:

Line Replaceable Unit Part Number == PCASFME.FMEA.PARTS.ITEM_PART_NUMBER iff PCASFME.FMEA.PARTS.PART_TYPE = “LRU”

Shop Replaceable Unit Part Number == PCASFME.FMEA.PARTS.ITEM_PART_NUMBER iff PCASFME.FMEA.PARTS.PART_TYPE = “SRU”

Line Replaceable Unit Part Number == PCASCLRU.CRITICAL_LRU_CRIT_LRU_PART_#

The condition statements are similar to condition statements in the SQL query language. In fact, this similarity is no accident, since these conditions wilt be added to any physical query in which ITEM PART NUMBER is included.

From a user’s point of view, implementing this feature allows the user to create a query over the concept of a line replaceable unit part number without having to know the conditions under which any particular field represents that concept. In addition, by representing the general – concept of a line replaceable unit part number, something the user would be very familiar with, this conceptual mapping technique has also hidden the details of the naming conventions used in each of the physical databases.

2.4.2 Integrating Data Translation Functions Into the Data Dictionary

In the simplest case, the integration of data translation functions into the data dictionary would be a matter of attaching to the data mapping tautologies described above a field which would store an indication of the type of translation which must occur to transform a result from its Location-specific form into the normalized form. This approach can be simplified further by allowing translations at the basal level to be identified by the source and target data types involved, and not recording any further information about the translation. It may not be unreasonable to assume that in certain well-defined domains, most of the translation functions required would be either identity functions or simple basal translation functions.

It is now possible to define completely the data structure required to store any arbitrary physical-conceptual field mapping tautology. The data structure would consist of the following parts:

  • concept field – a single, unique concept which the physical projection represents
  • normalized – a reference to the conceptual data dictionary entry used to represent the concept
  • physical projection – the field or set of fields from the physical data dictionary which under the conditions specified in the equivalence constraints represent the concept
  • equivalence constraints – the conditions under which the physical projection can be said to represent the concept
  • translation function – the function which must be performed on the physical projection in order to transform it into the normalized format of the normalized field

The logical statement of the tautology would be as follows:

R = Ft (P) iff E

where R is the normalized field representation, Ft is the translation function over the physical projection, P, and E is the set of equivalence constraints which apply to the relation. The exact implementation of this data structure would depend on the environment in which the system were to be developed, and would have to be specified in a physical design document. Note that instead of the “==” sign, which was defined above as “is semantically equivalent to”, has been replaced by “=” which means “is equivalent to”, and is a stronger statement. The “=” implies that not only is the left side semantically equivalent to the right, but it is also syntactically equivalent.

%d bloggers like this: