The Mandate Episode 3

Episode 3:    To Know or to Understand

Speakers:      Main Narrator (MN).         [Text in blue]

Second Narrator, female voice (SN).  [Text in black]

Written announcement against black screen: 

“This episode builds on topics from Episodes 1 and 2; links in the description. Reference to these topics may also be found in the companion book The Mandate, link also in the description.”

[Fade to black]

[A drone point-of view shows a landscape of natural and urban elements.]

[MN]: “Here is a world we think we know, a blend of nature and work of human hands, all reassuringly solid. But over time we have come to doubt its solidity, and increasingly we ask about what really is ‘out there’, in that complexity of perceptions we call reality. 

[Cut from drone view to MN, now speaking to the camera.] “Superficially, reality seems to consist of objects of various types. [Cut to various environments with prominent objects, then back to the room.] But as our knowledge of the world increases, we speak not just of objects, but of invisible fields, oscillations, probability waves, indeterminate states, and so on. Our certainty about an objective reality supposedly extending beyond the gate of our senses has become, well, less certain. 

“Yet, we cannot simply ignore our propensity for making objects out of sensory data, and we must at least examine how and why we do it.” 

[Fade to black. A high-resolution image emerges of a person against a landscape. SN is speaking.]

[SN]: “What do you see here: the image of a person against snow-capped mountains? A quick glance reveals as much, but there is more to the image than just those elements. Look closer, and you will see a lake, a trail through shrubbery, clouds, blue sky. In the foreground there are rocks and gravel, and you can make out that the person is wearing a tuque and a teal-coloured jacket. The visual field appears to consist of objects and features within objects, of many kinds. Knowledge of the scene improves with each additional detail, and even so it remains incomplete. 

[MN]: “Urged on by a mandate to reduce entropy, we try to refine our knowledge of the scene, and zoom in until we reach the maximum available resolution, that of individual pixels. 

“It would seem logical that by cataloguing each pixel we should know the image fully. After all, the image consists exclusively of pixels. And full knowledge of the image is the goal, why not skip over the rough resolutions and simply record the pixels?” 

[SN]: “Attempting to do this reveals something interesting. [Animation accompanies this paragraph, with a ‘magnifier’ scrolling over the image, showing highly pixelated details in each area.]

[MN]: “We discover that there are almost two kinds of knowledge: The approximate, every-day knowledge of familiar objects like mountains, roads and people, and then there is the exact knowledge that comes from recording the position and value of the smallest bits of data. As resolution increases (and with it knowledge of details), our comprehension of the scene appears to decrease. We seem to go from familiarity to unintelligibility without encountering a definite dividing line, finally gaining complete knowledge of the image while losing all understanding of it. 

“Can it really be that knowledge and understanding are inversely related?”

[SN, speaking over an animation of images varying in resolution]: “In a certain sense they are, because ‘understanding’ is not a lofty and indefinable concept. It is simply the outcome of Mandate-inspired operations aimed at representing large amounts of data within limited memory. These constitute the fundamental task of every intelligent system. [While SN speaks, the following script appears on screen].

The Fundamental Task of the Mandate is to represent indefinitely large amounts of data within finite memory.

“For realistic systems, this goal must be achieved with memory to spare, since memory may also be allocated to higher intellective processes, as well as system monitoring, regulation and maintenance. But even within limited memory available for data analysis, the fundamental task can discern representative patterns in the data.

“For example, the human eye has the equivalent of about  pixels, whose states provide the raw data for the visual field. And that number is only a minute slice of the data that the retina generates, since it is refreshed tens of times per second. Such intensity of data must be reduced according to some scheme before it can be further processed. And indeed, the visual cortex does radically reduce the fidelity of the data before forwarding it to other regions of the brain, until most of the original data is lost in favour of greater “understanding” of it.

Here is a familiar example. Show this this image to grade schoolers [a photo of a house].

“Then, take away the photo and ask them to draw what they remember, and they will typically produce something like this [a rudimentary line sketch is shown]:

“In the interval between seeing and remembering, processes based on the fundamental task will have replaced the photographic data with generalized ideas of house, mountains and trees. Adults might fare better: Being more familiar with the inadequacy of memory, they may make a special effort to store and retrieve finer details, like the number and types of windows, etc., while still falling far short of a faithful reproduction. [An adult-level drawing may be shown while this is narrated.]

“Note that the drawn images are rendered in outlines. This mode of representation is instinctual, and it has created confusion in the field of image recognition, with many early methods trying to implement it by edge detection. But edges are often artefacts, a by-product of implementing the fundamental task of the Mandate. 

[MN]: “So, to remember we must first forget, since we can’t remember all that we can perceive. But what proportion of data and which particular part of it should be preserved, in order to arrive at some overall understanding of the data? 

[SN]: “A partial answer has been known for years: Certain features in the data can be extracted by pattern recognition processes so that they can be prioritized for storage, thus saving a considerable volume of memory. 

[MN]: “But what makes some patterns more suitable than others? Is there a figure of merit against which to rank alternative patterns?

[SN]: “By now we also know the answer. As we saw in previous episodes, patterns can be ranked according to their associated entropy. Entropy is the figure of merit, and finding the patterns that minimize it is at the core of the fundamental task. We can demonstrate this with a simple image.” 

[MN photographs a camera battery against a moderately noisy background.] 

[SN]: “To keep the demonstration simple, first we convert the photo to grayscale and then reduce its resolution to 20x15 pixels. [Process is animated.] Thus, the program will start with the complete knowledge of just 300 pixels. The pixels have an intensity range of 256 levels, not all utilized.”

“Imagine that we are using a very rudimentary computer, such that 300 pixels of data will completely fill its available memory. Knowledge of the image would nevertheless be complete, since the system can record the value and position of every pixel, resulting in zero entropy for the representation. 

“But say that we want to free some memory for other tasks, and decide to store only the parameters of a single feature in the image. 

“Let’s draw a feature at random, [demonstrates by selecting a feature made up of several components, some not contiguous], including some disconnected elements to emphasize the arbitrariness of the choice. Let’s call it Feature 1.

Note that, in doing so, the leftover pixels constitute a default feature in its own right. Let’s call it Feature 0.

“Now, instead of storing the information of every pixel in the image, we record only the average grayscale value for Feature 1 and Feature 0 [morphing animation]. 

“In this way the memory volume taken up by the original 300 pixels reduces to what is required for just 2 features. Is there a trade-off? 

“There is, and it relates to entropy. Initially, the 300 pixels were features in their own right, all of them perfectly known by value and position, and the entropy of that representation was zero. Any change from that state is bound to increase entropy, and the Mandate requires that the increase be minimized. 

“But recall that we drew Feature 1 at random, making it unlikely that it would be the selection of minimum entropy. 

“Finding the optimal shape should, in principle, be a simple matter. We could systematically go through all alternative configurations of Features 1 and 0, calculating the overall entropy in each case and selecting the configuration of lowest entropy. Quite straight forward—until you consider the number of possible configurations for a field of 300 pixels. 

“Let’s calculate it first for the simplest case of binary pixels.

[MN]: “To appreciate the magnitude of this number, consider that it is about 10 billion times greater than the estimated number of atoms in the entire observable universe! The lifetime of the universe would not be sufficient to evaluate all the combinations! 

“At the resolution of the human eye, even if degraded to binary response,

“Such staggering numbers highlight the challenge to the fundamental task of the Mandate. It would be impossible to map one-to-one in memory all generated data, even if our senses were reduced to just the binary vision of a single eye. The retinal refresh rate would further compound the challenge. 

“Even a 300-pixel sensor gives rise to “merely”  possible configurations, also an intractable number by current means. So, how can the Mandate sift through such a vast data field to select the best entropy-reducing feature?

[SN]: “The short answer is that it can’t; at least, not by exhaustive sorting. If that were feasible, then perhaps a hypothetical carrier could find the absolutely best selection for Feature 1 after eons of computation. Strangely, it might still turn out to be different from the feature humans might instinctively select in an instant.

[A visual case to illustrate this point is desirable]

[MN]: “This potential disagreement between a pure Mandate solution and the one offered by physiological perception might lead us to believe that the entropy-minimization approach is not reliable. But if our perceptions are ever at odds with those of the ideal Mandate, then it will be us who have fallen short. In examining this point, we discover a surprising aspect about our perception of reality.

“Let us demonstrate these points. First, we write the Mandate in computer code. [Mathematical simplification is animated while someone is shown working at a computer]. Then, we “cheat” by imposing constraints similar to those that exist in humans. 

“For example, unlike the random feature we had previously drawn [image is reproduced], most of the visual features we perceive stand out when localized; that is, when their components are clustered by proximity. We will also implement this restriction in the code, with one further limitation: that the clustering be rectangular, an admittedly stricter constraint than those of natural perception. Aside from this, the program will be free to choose the size, proportions and position of any potential feature [animation shows “live” trial fittings of potential features], subject to the usual entropy considerations. 

“Within these limits, the Mandate can finally operate in reasonable time on a 300-pixel image, with modest hardware and software resources.

[Text against dark; or, alternatively, overlaid on existing scene: “The entropy evaluations in the code are based on a special protocol, which is explained in one of the links in the description”. [The protocol preserves the standard deviation of the former distribution after the average value of the feature is calculated.]

“We run the program, and this is the result. [This is animated]

“Does the extracted feature seem obvious? Its outline is essentially where we would place it ourselves, with roughly the same shape and size, without the help of software. But this is not really surprising, since we imposed on the Mandate limitations similar to those of our own perceptions, such as localization. 

“When a feature, or a cluster of features, re-occurs consistently in a given context, it may take on the status of ‘object’”

[SN]: “Note the red geometric shape applied by the code to indicate the selection. It is readily interpreted as an edge, or outline of something “real” in the data. But the program was not looking for edges or outlines, nor did it presume an objective reality. The algorithm simply implemented a mandate to minimize the entropy of the entire data field with some applied restrictions. We stress again: The displayed outline is extraneous to the data.”

[MN]: “But if that boundary is not the outline of something real, what is it?” 

[SN]: “It is an artificial contrivance, though it might be interpreted as an objective outline by an imperfect intelligence. In humans, such misinterpretations are inherited genetically and/or culturally. Even advanced AI’s must rely on similar restrictions, since their memories, though large, are not unlimited. They, too, must decide on what to remember and what to forget. [Previous image morphs to the one below].

[MN]: “This kind of image retains only the dimensions, location, and average intensity of the most prominent feature. Yet, I feel that it captures the “essence” of the original image. 

[SN]: “And the memory saving is not the only outcome: An incidental artefact was created that facilitates communication. It is a prototypical example of how a carrier of the Mandate constructs its sense of reality. 

[MN]: “The same kind of analysis could be moved up a level to operate on the field of abstracted features, and then on to higher-order patterns, in the iterative process we call ‘understanding’. When human carriers say they understand something, they no longer speak of individual data points, but of low-entropy representations. Thus, we can state with confidence that the reality we perceive is as much a product of memory management as of anything else that might actually exist ‘out there’.”

 “We started with this image,

and it might appear different now, either more banal or more enigmatic. As before, our eyes may be attracted to one or two prominent features extracted by the Mandate acting through a combination of our genetic traits and life experiences. The biological equivalent of countless entropy calculations will occur in our visual cortex, and the process will pause only when allocated memory cannot represent more features and patterns.

“The simple demonstration program we have seen, available from a link in the description, can be expanded to search for multiple features, for features within features, for the synthesis of objects out of features, and so on—with any data. 

[Visual transition]

“Despite the limitations of our species, we have come to know much about ourselves and our reality. Paradoxically, we tend to regard our intellective limitations as obstacles to greater understanding, forgetting that it is the limitations themselves that generate our experience of reality. A highly unlikely evolutionary phase may one day make memory unbounded, so that future intelligence will no longer need features and patterns to understand the world. Knowledge of all that is knowable will be achieved without simplifying constructs. 

If that day ever comes, the fundamental task will be completed and the Mandate will cease.”