JSON Collections: Self-referring Documents (aka Recursive Collection(s))

Recursive collections of documents might be necessary to model a specific domain data set. How can this be done?

Example: Parent-Child Relationship

One of the standard examples of self-referring data structures is the parent-children relationship. People have parents and might have children. Parents are always children themselves. Children might have children also. This example is recursive, and it has the interesting property that the knowledge about people is not complete.

Normalized Collection: References Only

The most normalized representation in a document-oriented world is one document for each person. So if a person is known or needs to be represented, a document is added to the system for that person with the data of that person.

This leaves the discussion of how to model the relationships between people. There are several alternatives (and this is not an exhaustive list):

  1. Has-parents relationships as separate document. In this case the relationship has-parents can be implemented as a document that has three properties in it: person, mother, father. (Possible implementations of references are discussed here: https://realprogrammer.wordpress.com/2012/08/17/json-graph-structures/). So every person has possibly a corresponding document representing its parents. An alternative design can be that the relationship is split into has-mother and has-father (represented as two different documents with two properties each).
  2. Has-children relationship as separate documents. The has-children relationship is more complicated. If a person has children, then the question arises, is there one document per child or one document that contains the reference to all children? One complication is that a child has two parents and not all children of a person have to be with the same partner. A document per child is easier to structure then a document for all children if the partner has to be referenced also. If that information is not necessary, having all children referenced in one document is reasonable as well.

The alternative representations contain the same semantics from a data modeling viewpoint, however, their computational impact might be different. For example, if the 80% case is finding the children of a person, having all children as one document is more efficient in general.

What would be even more efficient in this case would be a non-normalized approach where e.g. the references to the children of a person are within the document of that person.

Non-Normalized Collection: Combination of References and Embedding

To move from a normalized representation to a non-normalized one is in many cases related to the computational effort upon retrieval of the documents. There might be other reasons, too. What are the alternatives?

In principle, non-normalization in a document world means to represent relationships not as documents themselves, but as sub-collections inside documents. This is moving from a by-reference representation to a by-value representation. In the above case, the references to children of a person are added as a sub-collection “children” inside the document of that person. The upside is that accessing the children of a person is quick, the downside is that finding the parents of a child means to search through sub-collections.

This leads to an interesting consequence. If relationships have to be traversed in both directions, explicit representations as documents make that easy as each document representing a relationship can be interpreted in both directions. In the non-normalized case, this is not given any more as the entries in sub-collections are uni-directional. Therefore, in order to make the traversal in both directions easier, it might help if each child has a property “parent” added for the opposite direction.

Furthermore, if not only the children have to be found in the 80% case, but also their complete data set has to be fetched every time, it might be even more efficient to embed the whole documents of the children in the sub-collection “children”. This makes read-access fast, but duplicates the document of a child. So the penalty is the increased effort necessary for updates of child data in at least two places.

What follows from here is more of the same. If always grand-children have to be determined, but not children, the transitive closure for that level can be explicitly stored. With all the same pros and cons.

And, of course, the same discussion applies to the parent and grand-parent relationship.

Summary

It is possible, of course, to store and manage recursive relationships in document-oriented databases. The alternatives of normalized and non-normalized representation need to be carefully explored in terms of read-access and update effort and efficiency. All alternatives are valid under the set of access patterns that are expected by the information system.

In JSON land, recursive collections are a special case of graph structures, as discussed here: https://realprogrammer.wordpress.com/2012/08/17/json-graph-structures/.

Advertisements

JSON is strictly By-Value

JSON is an externalization format, not a programming language data type implementation. Why is this relevant?

JSON

JSON (http://www.json.org) is an ASCII representation of data. It provides base data type representations and a grammar about how to structure JSON structures properly.

As for terminology, here are some synonyms often used:

  • JSON “object”: document
  • JSON “members”: properties
  • JSON “pair”: property
  • JSON “string” in “pair”: property name
  • JSON “value” in “pair”: property value

JSON is Syntax

It is syntax only. There is not data type semantics attached to it; all programming languages that process JSON define their own interpretation of the meaning of the syntax JSON defines.

JSON Semantics as such is Undefined

The semantics of JSON is not defined by JSON itself. It is undefined by the standard. Programming languages and databases have to define their (!) semantics of JSON.

For example, the JSON standard does not make a statement about unique property names in a document. In JSON terminology, an object (aka document) can have several pairs (aka properties), each having a string and a value (aka property name and property value). JSON does not constrain the property names to be unique. According to JSON it would be valid for a document to have several properties with the same name.

JSON Semantics Implementation

How is this implemented? Do systems actually allow several properties with the same name in a document? Let’s look at two of those.

MongoDB (version 2.0.2): it is possible to save a document that has the same property twice:

MongoDB shell version: 2.0.2
connecting to: test
> use json
switched to db json
> db.test.save({"a":1, "a":2})
> db.test.findOne()
{ "_id" : ObjectId("5006d58e92cca1a32772df6a"), "a" : 2 }
>

So, clearly MongoDB does not complain about the fact that a property name is stated twice. However, it opts to make a selection itself and uses the second of the two and actually stores only one. Of course, this bears the question if MongoDB on input applies other modifications as well.

JsonLint (http://jsonlint.org/) exposes the same behavior. When validating {“a”:1, “a”:2} then two things are happening: it deems the input as ‘Valid JSON’, but then it modifies the input to only {“a”:2}.  This is bad in two ways. First, it is not clear which version is ‘valid’, and second, it is not a read-only lint! This means that if you paste in a JSON document, always check if JsonLint modified it; your document might not be valid, but JsonLint’s modification of it.

So it seems like that some systems actually do not support having two properties with the same name. But then I would suggest they flag this as error and not modify the document.

JSON and References

How does a JSON document refer to a second one? For example, an object representing a user referring to an object representing an address?

JSON does not have the notion of reference or pointer nor the notion of address or unique identifier (both necessary to make referencing work). In order to uniquely identify a document, the author of the document has to add a property that by convention is deemed to be a unique identifier (see e.g. MongoDB above, the database adds a ‘_id’ property). Secondly, a reference to such a unique identifier is a regular property that by convention has to be understood as being a reference. For example, the value 5006d58e92cca1a32772df6a above could be a value. By convention the interpretation of this property would know that the number is actually a unique identifier.

Side note: this is the same situation relational databases are in, they don’t have references either. But, in their case there is the concept of key and foreign key supervised by the database, both relying on data values. Except for the supervision functionality, JSON could be interpreted this way, too.

Coming back to the semantics for a moment. Assume a property “ref” is used to indicate that its value is a reference to another document. How is the absence of link established, meaning, the notion that a given object does not refer to another one? On possible way is to set “ref” to the value null. Alternatively “ref” could be left out. Either (or both ways) are possible, but that needs to be established.

JSON and Linked Structures

In summary, in order to establish linked structures using JSON, special properties have to be called out in order to establish addressing and referencing. JSON itself does not provide concepts for this. In addition, the JSON semantics and interpretation has to be established (either explicit by rules or implicit by the programming code).

Engineers and architectus be aware: JSON is an ASCII syntax for externalization, not a programming language data type system.