FemtoThnk: A behavioral model of higher-level brain functions

   Vinu Arumugham

FemtoThnk is a behavioral model that features higher level brain functions such as creativity, learning, language and spatio-temporal problem solving.

1. Background

In our quest to improve the human condition, we face an increasingly complex array of problems. With the rapid advancement of science and technology, few of us are able to acquire an in-depth knowledge or understanding of more than a handful of subjects in our life time. However, detailed knowledge of a large number of subjects by one entity, would make it possible to gain insights into complex relationships. Such understanding will help find solutions that may not otherwise be discovered. The computational and communication bandwidth of the human brain is limited. In other words, there is a limit to the amount of information we can see, read, hear or understand at any time. The evolution of the human brain is far outpaced by societal complexity. This then creates the need for machines that can perform the functions of a human brain, while meeting and eventually exceeding its computational and communication abilities.

2. Objective

To develop an architecture for a machine with capabilities described above and implement it using conventional computer technology. While radical new computer architectures have been proposed for machine intelligence, the economies of scale ensure that for the foreseeable future, conventional computer technology will provide the best cost to performance ratio.

3. How the brain works

This section is a hypothesis of brain operation that forms the basis for the FemtoThnk architecture.
3.1 The functional units

The brain stores what we see, hear and feel. Let us refer to this storage as the information base. The brain also has the capability to imagine what is stored. Visualize this as a built-in virtual reality screen--call it the imagination space--on which either real-time sensory inputs or their stored equivalents from the information base can be projected.

When we imagine, some part of our brain decides if that imagination makes sense. Let us call it the grading system. We can think of the grading system as an internal model of the rules or laws that apply in our environment.

That makes a total of three functional units, the scratchpad we called the imagination space, an information base where the brain stores patterns from our sensory organs and the grading system that stores the rules of the environment.

3.2 Information acquisition

When the nerve cells at the end of the optic, auditory or other nerves fire in response to an applied stimulus, they form patterns. For now, let us refer to such patterns as concepts.

Let us define an experience element (or exel) as the sum of all the sensory inputs or concepts we receive at one time. Any experience we have in everyday life can then be considered a sequence of exels. This sequence of exels can either be the time evolution of the environment, or changes in the environment caused by our actions. In either case, the brain records these sequences of exels. Let us define such a sequence of exels as an episode.

During the waking hours, the episodes are collected in a temporary memory. Before actual storage occurs, the sensory information may go through several stages of processing. For example, the separation of color, movement and texture information, along the visual pathways.

3.3 Organization

Since the amount of information that is obtained by recording sequences of exels is extremely voluminous, some preprocessing is required before the information can be placed in permanent storage. During sleep, the brain evaluates each recorded episode by presenting it to the grading system. If the grading system finds that it is a common experience, it is disregarded. If the experience seems similar to a previous one, it is stored in the information base and a link is created between the two. Imagining one at a later time, could associatively recall the other. If the experience is new, it is stored in the information base and the grading system is updated.

Let us now broaden the definition of concept to include all associable entities. The brain can treat an input or sequence of inputs from a sensory organ as a concept. In other words, a sequence of inputs from one sensory organ can be associated with a sequence from another sensory organ. Therefore, when the environment presents one sequence, the brain can associatively recall the other. By treating sequences of concepts as yet another concept, the brain can create a hierarchy of concepts. The concepts at the lowest levels correspond to physical concepts and the ones at the higher level correspond to more abstract concepts. Concepts or associable entities are also the basis for communication. Words or symbols are sequences from a sensory organ and are therefore concepts. Words can associatively recall concepts and vice versa.

3.4 Innovation

The brain randomly selects exels from the information base and projects them on to the imagination space in an attempt to create new episodes. Some of these episodes will make sense and most will not. The brain uses its previous experience to interpret this mix of logical and illogical sequences, that we call a dream. How does the brain know if these episodes make sense? It uses the grading system to evaluate the episodes. If the grading system finds the episode to be either a common one or if it does not seem to make sense, it is discarded. However, if it does seem sensible, the episode is stored for further evaluation. These episodes are what we refer to as ideas. During the waking hours, when we happen to think of something related, this idea could pop up in our head or things might appear to have fallen in place (due to this idea). Since such ideas are a result of a stochastic process and are not direct logical derivatives of our past experience, we usually have difficulty explaining the origins of our ideas.

3.4.1 Relation to REM

Presenting a continuously changing scenery to the eye results in eye movement as the eye follows the objects in the scenery. When sequences from the information base are projected on to the imagination space during sleep, the stimuli to the eye muscles appear to be applied just as if the patterns were from the retina, possibly resulting in the rapid eye movement (REM) phase of sleep. That is, the brain does not appear to differentiate between retinal projections and projections from the information base, on to the imagination space. This is consistent with the idea generation process described above.

In some respects, the idea generation process described above is Darwinian. The random combination of exels can be considered the equivalent of mutations or genetic recombinations, while the grading system-- an internal approximation of nature, provides the equivalence of natural selection.1

3.5 Problem-solving

A problem can be imagined as having to get to a situation say exel E starting from a situation represented by exel B. In addition, the solution sequence going from exel B to exel E, should be feasible in the current environment. The brain has to use past experience again, to ensure that this is true.

When the brain attempts to solve a problem, it looks at the requirements and then searches the information base for something that meets it. For this purpose, it imagines the exels B and E. That is, it creates a pattern in the imagination space and then searches the information base for a match. If it cannot find a match, the brain has to come up with a sequence of exels that are a combination of previous experiences and ideas, that together can take it from exel B to exel E. If this process lasts over days, it could get input from the dreaming process, in the form of new ideas. When the brain does come up with a tentative solution, it evaluates it first with the grading system and then by presenting it to the environment. The success or failure of this effort, is a new experience. The new experience is stored in the information base and used to train the grading system for future reference.

Problem solving therefore, can be viewed as having two approaches. One is logical problem solving, also known as thinking, that involves logically working from the problem specification toward a solution. The other approach is stochastic or the use of ideas that are the result of dreaming. Most actual problem solving exercises by the brain, will involve alternating between these approaches as the solution progresses.
To some extent, the stochastic component resembles a technique called simulated annealing. This technique is used in solving complex problems such as optimal task assignment or load balancing in large multi-computer networks, taking into consideration inter-task communication costs and task execution times. This technique resembles physical annealing in metals. Initially, the task assignments, like atoms, have a "high temperature" or high random mobility. The temperature or mobility is reduced in steps. At each step a cost function is evaluated and the best task assignment is accepted. In FemtoThnk, the use of ideas (random juxtaposition of components) in a problem solution and the subsequent evaluation of the mutual feature match are analogous to task assignment and cost function evaluation respectively.

3.6 Language

The ability to communicate, among other things, requires the following:
1. The capability to associate sounds or symbols to concepts. 2. The capability to learn a set of rules or grammar, that govern the assembly of such sounds or symbols, in order to convey information in a reasonably unambiguous manner.

As discussed in a previous section, if the sounds or symbols are treated as concepts (associable entities), they can be associated with other concepts. In other words, sounds or symbols can be associated with sequences of inputs from any sensory organ. This capability is part of the information base.

The grading system where the rules of nature are internally represented, is also the logical repository for the rules of language.

3.7 The main process in the brain

The main process in the brain looks like this:
while ( NOT Brain Dead )
if ( Sleeping or Unconscious )
// Classify experiences and update the information base.

// Generate ideas and use the grading system to filter out impractical or wild ideas.
// Update the grading system with new experiences and the environment's response to
// ideas previously tested.
// Waking hours, Conscious
// Test new ideas and get the environment's response to those ideas.
// Collect sensory information or experience.
// Solve problems using information in the information base and use the grading system
// to filter out bad solutions.

This hypothesis attempts to explain many of the observed characteristics of the brain that are attributed to intelligence, by emphasizing the links between dreaming, creativity, learning, thinking and language.

4. Capabilities of FemtoThnk

If we look at the capabilities of the brains of life-forms lower down the chain of evolution, we see that many have developed the ability to manipulate objects. Whether it is sophisticated nests constructed by birds or a chimpanzee using twigs to catch termites, the ability to assemble or manipulate everyday objects was an important capability. These capabilities also form the basis for higher-level functions in human brains. We often use simple physical analogies to gain insights into complex phenomena. Examples: (a) Imagine a stretched rubber sheet with a lead ball rolling on it and you understand what happens to space-time, in the presence of a massive object. (b) Imagine a network of elastic strings with a heavy ball carefully suspended from each node. If one ball is perturbed, the oscillations will spread to all the other balls and pretty soon one of the wildly oscillating balls would cause the elastic to snap, setting off even wilder oscillations, eventually destroying the network. The scenario we imagined is what one can expect should an electrical imbalance occur in a power distribution grid.

Our understanding of complex or abstract concepts are constructed on our understanding of the physical world. This important notion helped shape the architecture of FemtoThnk. The first phase of the project (known as AttoThnk) primarily handled the spatial aspects of objects. It had the ability to compare, organize, assemble, dream and perform spatial problem-solving using the internal representations of physical objects. The system also had the ability to learn and recognize good assemblies. Dreaming refers to the random assembly of objects that is followed by evaluations using past experience. The current phase adds two major modules. The module that handles spatio-temporal aspects of objects, concepts and associations, and the module that handles communication or language. The spatio-temporal module will have all the abilities of AttoThnk but as applied to processes, rather than just objects. A process can be described as the time evolution of an object or objects.

FemtoThnk's automatic hierarchical organization of concepts, enables it to represent, recall and operate on arbitrarily complex concepts.

The communication ability of FemtoThnk resembles that of the human brain in that it does not have a built-in grammar. It creates its grammar based on exposure to the language. It is therefore language independent. Like the human brain, with sufficient exposure, it can learn any language or even multiple languages.

5. FemtoThnk Architecture

For the purposes of this discussion, we can use the following simplified definitions:

Simple object

The internal spatial representation of the response of a sensory system to an applied stimulus. The stimulus can be visual, aural or other input that is appropriate for a given sensory device.

Complex object

Internal representation of a combination of simple or complex objects, with possible abstraction or loss of details.

Component object

An object that is a part of another object.


The process of assembling objects or exels, real or abstract.


The process of directed imagination with the objective of problem solving.


The process of open-ended and random imagination possibly resulting in new assemblies.


Assembly resulting from the processes of Dreaming or Thinking.

As noted earlier, abstract concepts are based on the understanding of the physical world. Additionally, any results that this system produces would be of little value, if it were not applicable in the physical world. Therefore, at the lowest level, FemtoThnk operates on internal representations of 3-D (three-dimensional) objects.

FemtoThnk has four major modules. The interface, the information base, the imagination space, and the grading system. The major processes include Classification, Dreaming, Environmental evaluation, Training, Problem-solving, Association and Language.

5.1 Modules

5.1.1 Interface

The interface module provides the means of communication between the system and the environment. The interface is used to output ideas and solutions to the environment. The environment's response, new objects, processes and problems are the inputs to this module. The physical sensors perform any pre- processing on the inputs that may be required followed by quantization. Function sensor outputs are functions of physical sensor outputs. An example of a function sensor in the visual subsystem would be a rate sensor, that may take say the X direction sensor's output and provide an output that is proportional to the rate of change of X values with time.

5.1.2 Information base

The information base consists of a physical and a logical section. The physical information base

The physical information base is a repository for objects. It contains information on how the objects are related to each other. It also contains sufficient information to assemble complex objects from their component objects.

The objects are organized into four categories. New, Known, Hypothetical and Ungraded. An object that has been input to the system is classified as a New object. New objects will be compared to other objects in the system, relations will be established and the object will be moved to a different category.

Known objects are those that the system understands. The system knows the relationship among these objects. Generally, New objects become Known objects, after the system establishes relations between the two. A complex object that was assembled by the system and that has been evaluated by the environment, may also be classified as a Known object.

A Hypothetical object can be viewed as a problem definition. The system tries to solve the problem by assembling objects of the Known category to obtain an object that resembles the Hypothetical object. If the effort is successful, that is, the environment approves the solution, the object is considered known and classified as such.

The Ungraded category contains objects that await evaluation by the environment. It includes assemblies of Known objects that the system has "dreamt up", tentative solutions that the system has "thought about" for problems presented to it or assemblies resulting from the application of an analogy.

Figure 1: Physical Information Base The logical information base

The logical information base includes the following:
1) A hierarchy of concepts, exels and episodes
2) Associative links among concepts
3) Ordering links among communication concepts
4) Pattern sensors

In general, a concept can be considered to be a type, value pair. The type refers to the type of the sensor and the value refers to the pattern specific quantized output of the sensor. Since some physical devices can provide contiguous or analog inputs, the outputs from the sensors may have far higher resolutions than required. Such outputs are therefore quantized by dividing the possible responses into ranges. Figure 2 shows the relationship among the various components in the logical information base.

Ordering links are created by processing sentences and they specify the order in which words can occur in a sentence.

Pattern sensors as opposed to physical sensors, take concepts as inputs. A pattern sensor can be trained with a pattern or sequence of concepts (type, value pairs). The sensor will respond with maximum strength when this pattern is presented at any later time. It will also respond with a strength proportional to the resemblance between the presented pattern and the training pattern. The output of a pattern sensor is also a concept. Its type is the pattern sensor's type and its value is the strength of the pattern sensor's output. Figure 2 shows how pattern sensors can be used to convert an episode into a higher level experience element or exel. This process can be applied repeatedly to create exels with any level of abstraction.

Figure 2: Logical Information Base

5.1.3 Imagination Space

FemtoThnk's imagination space is equivalent to the human brain's temporary or working memory. The system uses the imagination space to compare, align, translate, rotate, scale and assemble objects, during the Dreaming process and for feature extraction during the Thinking or Problem-solving process.

When a word or sentence is presented to FemtoThnk, it performs associative recalls and the resulting concepts are placed in the imagination space. Recalling a concept with a direct associative link to an object in the physical information base, will result in an imagination of the 3-D internal representation of the concept. These imaginations can be manipulated as in say the problem-solving process. Finally, FemtoThnk can perform associative recalls for the contents of the imagination space and use the recalled words to construct a descriptive sentence.

5.1.4 The Grading System

This module performs three functions:
1) Derive rules from information that the system acquires from the environment.
2) Evaluate solutions to problems, that are generated by other modules.
3) Evaluate ideas generated by analogies, the Dreaming process or the Problem-solving process.

The rules derived include grammar rules for languages, rules of mechanical assembly and rules derived from processes regarding say the cost associated with performing an action. These rules are an internal representation of the laws of the environment. These rules help the system discriminate between ideas or problem solutions that are compatible and feasible in that environment, from those that are not. The phrase "problem solution" is interpreted broadly. It may refer to the creation of a valid sentence in a language for communication, the creation of a stable mechanical assembly or it could refer to the creation of a process that accomplishes a task using the least amount of resources.

Ideas that are "dreamt up" (Ungraded objects) are filtered by presenting them to the grading system. The grading system compares these objects with the Known objects. The response of the grading system is used to determine if the idea will be presented to the environment for evaluation, or if it will be discarded.

5.2 Processes

5.2.1 Classification

A feature comparison of New objects with Known objects checks for similarity. If some of these Known objects are components of complex objects, the system tries to create new complex objects by substituting similar New objects for the Known objects. The resulting new complex objects go through a grading process, to determine if they can be classified as Known objects. This is the basis for the system's capability to analogize.

5.2.2 Dreaming

Dreaming can occur at the physical and logical levels. Known objects are assembled using random rotations and translations. If the resulting object is too big to fit in the imagination space, then the object is scaled. The resulting object can optionally be filtered by a set of instinct rules. Instinct rules can be loosely defined as the most basic laws of the universe. These are considered to be so fundamental that their internal representation can be hard-wired for the purposes of modeling.

The resulting object is presented to the grading system for evaluation. If the grading system's response to the object is very strong, it probably is very similar or the same as a known object and is therefore rejected. For all other responses, the probability of selection is proportional to the strength of the response. The selected objects are stored in the Ungraded category to be presented to the environment. This scheme attempts to prefer and present evolutionary ideas to the environment.

Alternately, dreaming can occur at the logical level. In this case, randomly selected episodes will be assembled in imagination space. The instinct rules in this case will among other things, ensure that exels at the assembled episodes' boundaries are similar.

Two or more randomly chosen exels can be combined. The new situation can be compared with exels in other episodes. If there are any matches or near matches, all the exels following the match in that episode, can be copied to create a time evolution from the new exel, thus creating a new episode. Two or more randomly chosen episodes can be concatenated to create a longer episode.

In either case, the new episodes are evaluated first by the grading system and then by the environment, before being stored as new useful ideas.

5.2.3 Environmental Evaluation

This is the process where the Ungraded objects or episodes are presented to the environment. It is like "ideas popping up" in our head, we test them by presenting them to colleagues or by performing experiments. If the environment's response is good or bad, the object or episode is marked as such and stored in the Known category. If the response is mixed, then the object or episode is considered uninteresting and discarded.

5.2.4 Training

During this process, the objects and episodes in the Known category are evaluated by the grading system. The evaluations in this case are used to tune or change the rules in the grading system.

5.2.5 Problem-solving

Like Dreaming, the Problem-solving process can occur at the physical or logical levels.
The system selects a Hypothetical object--the problem definition, and attempts to find one or more Known objects that can be assembled to create a comparable object. That is, find out if a solution to the problem is already known or if solutions to sub-problems can be combined to create a complete solution. The resulting object is presented to the environment for evaluation. If the response is good, the problem is considered solved and if it was assembled, the details of the process are stored in the information base. The object then becomes a Known object. If the environment's response is bad, the system attempts to solve the problem by using a different set of initial conditions and parameters. If it still does not succeed, it probably means that the system is unaware of some component or idea required to solve the problem.

During the Problem-solving process, the system internally evaluates sub-problem solutions using the instinct rules and the grading system. If the sub-problem solutions are found to be good, the system updates the information base. Therefore, even if the system is unsuccessful in solving a problem, it continues to learn new ideas in the process.

5.2.6 The Main Process

The main process is a loop that consists of the Dream, External evaluation and Train phases. This loop is interrupted if a problem requires to be solved or if a new object is presented for classification.

5.2.7 Association

The system uses two major rules to help make associations. Rule 1: Compare two similar exels. If two aspects of one exel differ from their counterparts in the other, while all other aspects remain almost the same, these two aspects may be associated. Rule 2: If two aspects of an exel are uncommon, they may be related.

Example for Rule1:`The following two sentences represent two situations:

The apple is a fruit.
The carrot is a vegetable.

The words apple and fruit in sentence one are different from their counterparts in sentence two. All other words are the same. Therefore, the system will associate the word apple with the word fruit and the word carrot with the word vegetable.

Example for Rule 2:
The following sentence represents a situation:
The cat is an animal.
The words cat and animal are the most uncommon in that sentence. Therefore, the system will associate cat with animal. This is similar to Pavlovian conditioning.

Since these rules do not always guarantee correct associations, the strength of the associative links are aged. Over time, weak links will be destroyed. In general, associations based on Rule 1 are more likely to be true than those based on Rule 2. For this reason, Rule 1 links are made stronger than links created by Rule 2. These rules are also applied across sensors. Therefore, links can be created between responses of say the color sensor and the shape sensor in the visual subsystem.

Pattern sensors are dynamically created modules that detect patterns of responses from physical sensors or other pattern sensors. The association rules will also be applied to the responses from such pattern sensors. For example, if a pattern sensor is created to detect a pattern of increasing values in an array of responses from a Y direction sensor in the visual subsystem, the output of that pattern sensor could be associated with the word "rising" or "rise". As noted earlier, the responses of sensors are also known as concepts. In this case, the response of this pattern sensor encodes the concept "rise".

Pattern sensors also create links from their outputs to their inputs. In the above example, the concept "rise" would have links to the concepts associated with each of the values in the array. Therefore, if the concept"rise" is associatively recalled by some other concept, it will cause an array of Y direction concepts with increasing values, to be placed in the imagination space.

5.2.8 Language Conversation

This function allows a user to talk to the system in plain English or any other language to which the system has been exposed. As noted earlier, like humans, the system can sometimes make inappropriate associations between concepts due to insufficient information. Conversations between the system and the user is one method of correcting such associations.
We know that children do not have to be taught to distinguish between a parent's smiling face and a stern face. They know in general that a smiling face means approval and a stern face indicates disapproval of their actions. Likewise, FemtoThnk has a built-in capability to distinguish user responses such as Yes or No.

During a conversation, a user can pose a question or make a statement. The system will perform associative recalls of all the words in the sentence. A variety of concepts and objects may be recalled and placed in the imagination space. The system will select the strongest responding concepts and objects and create sentences using the word ordering links. The sentences will be presented to the grading system. The grading system will evaluate the sentences based on the rules it has learned . The best sentence is presented to the user. The user can modify the query or statement based on the response. Finally, the user can approve

the response or if necessary provide the appropriate response to the query or statement and the system will learn it. Grammar rules

Based on the sentences it encounters, the system creates guidelines on how words or symbols can precede or follow one another. This information is stored in the information base. At a higher level, it also looks at sentence structures and creates templates on how classes of words or symbols can be strung together to create valid sentences. These templates are stored in the grading system.

6. FemtoThnk implementation

While any such model can in theory be implemented using conventional programming or with artificial neural networks or ANNs2, the current implementation borrows from both techniques to optimize for operation on conventional machines.

For visual inputs to the brain, the mapping from the physical object to the internal 3-D pattern of firing neurons, is a function of the physical construction of the retina and the optic nerve. The mapping between physical objects and FemtoThnk objects has been chosen to be 1:1. This makes it easier to visualize and gain insights into the operation of the model. However, to reduce storage requirements, the current implementation requires that the physical objects be filled and convex.

Implementation details of some major modules and processes are described in this section.

6.1 Modules

6.1.1 The interface

The interface consists of a GUI based input/output system, which enables the user to communicate with and control the system's operation. For this implementation, the user will serve as the environment, providing responses and evaluations to the ideas and solutions that the system generates. The interface currently uses numerical inputs, plain English text or simple 3-D drawings. However, it can accept inputs from more complex devices such as video cameras, touch or pressure sensors, microphones, sonar or any instrument with appropriate pre-processing.

6.1.2 The information base

Each object in the information base is described by an object descriptor. Each object also has a unique identifier. If it is a complex object, the descriptor contains the identifier of a primary object, a secondary object and a transform. This object can be constructed by juxtaposing the primary object, with the secondary object transformed as described. The transform is a code indicating the amount of rotation and translation to be applied to the secondary object. The primary and the secondary objects can be complex objects. The descriptor has a list of identifiers of complex objects, of which this object is a component. Each descriptor also has a list containing identifiers of objects that have been compared to this object and been found to be similar.

The physical manipulation of objects is based on the notion that, however complex or abstract the concept, it is still understood and thought about in terms of physical objects and processes.
Each concept in the information base has three lists. The Association list points to all concepts associated with this concept. The Up list points to all the concepts that were derived from this concept. The Down list points to all concepts from which this concept is derived. Each concept also stores the last recall strength and a number indicating its relative rarity of occurrence with respect to the other concepts in the system. Feature extraction

Features are extracted from the 2D projections of an object on the X, Y and Z planes. A "containing rectangle" is drawn around each projection and each corner of the rectangle is treated as a feature. Two sizes of features are extracted. A small feature is extracted from a 4x4 pixel area and a large feature is extracted from an 8x8 pixel area. The extracted features fit in a byte and a 32-bit word respectively. Each bit corresponds to a set of 2x2 pixels. The bit is set, if all the pixels did not have the same value. As an optimization for speed, each feature's rotated (90 degree increments) and mirrored versions are also stored in a feature table as part of the object's descriptor. Associative recall Concept recall

Each concept has associative links that point to other concepts. Each link has a strength associated with it. Since associatively recalled concepts can themselves associatively recall other concepts, the final strength of a recalled concept is a function of the link strengths and the depth of the recall. This is used to select only the strongest responses because a recursive associative recall can recall a very large number of concepts in just a few levels. Feature based associative recall

Feature based associative recall is used during the physical Problem-solving process. FemtoThnk extracts features from the Hypothetical object, searches the feature tables of all the Known objects and selects those objects that have a match.

6.1.3 Imagination space

The imagination space is a set of data structures and functions required to work with objects and concepts. The size of the object data structures among other things, determines the complexity of the assembly that the system can handle. That is, the amount of detail that the system can work with at any level. When the complexity exceeds this capacity, the imagination space scales the image, resulting in loss of details and therefore achieving a level of abstraction. This is similar to the way we break down complex problems into a hierarchy of sub-problems with just as much complexity as we can handle at any level.
The imagination space also contains data structures to hold exels and episodes that are recalled. Functions in this module traverse associative, similarity and ordering links.
Sentences are constructed in the imagination space. Construction of sentences are based on:
1) Words in the input sentence if any,
2) Words associatively recalled for concepts currently in the imagination space and
3) Words pointed to by ordering links from the words in (1) and (2).
Several sentences are created using combinations of the words with the strongest and second strongest link or recall strengths. These sentences are then evaluated by the grading system.

6.1.4 The grading system Object comparison

The grading system uses a pattern matching algorithm to compare assemblies created by the Dreaming or Problem-solving processes with the Known objects in the system and returns a rating. The pattern matching algorithm will handle any rotation or displacement of the input assembly required for proper comparison. Sentence rules and evaluation

The grading system derives templates from the sentences it encounters. Consider the following sentences:
The apple is a fruit.
The orange is a fruit.

The grading system will create the following template:
The category1 is a fruit.
where category1 contains the words apple and orange.

When a sentence is presented for evaluation, it will be compared against the templates. A sentence matches a template, if all the words in the sentence match the words in the template and if the template has one or more categories, the corresponding words in the sentence belong to the appropriate category. If a sentence matches a template, it is properly formed. Similarity links will be created between categories if their membership overlap is in excess of 60%. If there is a mismatch during sentence evaluation, the similarity links are traversed to check if the word that causes the mismatch belongs to a linked category. If some words in a sentence do match linked categories, such sentences will have a lower rating than if all matches were in the main categories. By performing this evaluation, the grading system helps pick the best sentence out of several sentences constructed in the imagination space.

6.2 Processes

6.2.1 Dreaming

Figure 3 shows the data flow during the dreaming process.

Step 1
Two Known objects are selected for assembly. One is placed in the imagination space and the other is displaced and rotated randomly and juxtaposed with the first to obtain an assembly. Initially, every possible translation that can be applied to the second object has the same probability. A translation is chosen randomly and evaluated using the instinct rules. The probability associated with this translation is increased relative to all other translations by an amount proportional to the quality of the assembly. The process of randomly choosing translations and evaluating them, continues until the probability associated with one translation becomes greater than a set limit. In the current model, instinct rules are biased against physical overlap of objects and for larger surface areas of contact among objects or tight assemblies.

Step 2
The resulting assembly is placed in the Ungraded category.

Step 3
Of the assemblies in the Ungraded category, those that qualify in the evaluation by the grading system, have a high probability of making it to the environmental grading stage. The other assemblies have a lower probability.

Step 4
The assemblies are presented to or tested in the environment. At this time, this involves projecting the assembly for evaluation by the user. With a robotics interface, the system might actually attempt to perform a physical assembly.

Step 5
The input from the user or from sensors in the robotics system will be used to determine if the assembly was a good or feasible idea. This information is used to either classify this assembly as a Known object or discard it.

Figure 3: Dreaming process flow

6.2.2 Problem-solving

Figure 4 shows the data flow during the Problem-solving process.

Step 1
The problem is provided to the system as a verbal description or a drawing. If it is a verbal description, the system performs associative recalls and manipulates the recalled objects to meet the description. The result is a Hypothetical object and possibly a list of objects that can be used in the solution.

Step 2
This process can be visualized as trying to solve a 3-D jigsaw puzzle, using a holographic projection of the problem as a guide. The object that is placed in the imagination space in this case, is the Hypothetical object. Features extracted from the problem object or requirements, are used to select objects from the Known category. The objects are then positioned in the imagination space to achieve the best match between the requirement and the available feature. After each Known object is positioned, the problem can be considered to have been redefined. This is because, while the Known object matched some of the features of the problem object, it also introduced a few of its own features, that subsequent objects must match. Between feature extractions, the partial assembly is rotated to actually give the system a different perspective of the problem. Selection of objects proceeds from the largest usable object, through smaller ones, until all the requirements are met. The process starts by using large features and then switches to small features as the solution progresses. The translations and rotations applied to each object is recorded.

Figure 4: Problem-solving process flow

Step 3
The collection of objects resembles the Hypothetical object and is presented to the environment as a tentative solution. The environment's response will be used to determine if the solution is acceptable. If it is not acceptable, the process is repeated, starting with a different initial perspective and a different set of objects. This iteration continues until an acceptable solution is obtained or the system runs out of objects. In the latter case, the solution is deferred until the system learns about some new objects or until its Dreaming processes generate ideas.

Step 4
Obtain the environment's grading of the solution.

Step 5 If the solution was acceptable, the recorded information about the assembly process, including the partial assemblies are stored in the information base.

Step 6
The partial assemblies are treated the same as the results of the Dreaming process. The grading system is used to evaluate the partial assemblies.

Step 7
The assemblies are presented to or tested in the environment. This could involve projecting the assembly for evaluation by the user. With a robotics interface, the system might actually attempt to perform a physical assembly.

Step 8
The input from the user or from sensors in the robotics system will be used to determine if the assembly was a good or feasible idea. This information is used to either classify this assembly as a Known objector discard it.

Since the above Problem-solving process can just as well use a set of Hypothetical objects instead of Known objects, complex problems can be treated as a hierarchy of Hypothetical objects.

6.2.3 Association

Associative links can be created between words (communication concepts) and concepts or between concepts. Associations between words and concepts, will be more frequent when the system is learning a language. At other times, concept to concept associations will be more common. Association rules are always applied to one or two exels at a time. Exels can consist of a sentence, sensor information or both. If the system is learning a language, it will use the exels as is, to perform word to concept associations. When we read a book, we do not begin to create associations between the words, but we recall the concepts the words represent and make associations between those concepts. Likewise, if an exel has a sentence, the system must first recall the concepts that those words represent. The system creates a new exel and places the non-communication concepts of the original exel and all the recalled concepts into it. A recall can result in more than one concept of the same type being recalled. It is not useful to create associative links between concepts of the same type. For example, no purpose would be served if the system encountered one color and associatively recalled all the other colors. These concepts of the same type are therefore treated as a spatially distributed episode. A pattern sensor is created (if one did not already exist) and trained to respond to this combination of concepts. The output of the pattern sensor is a concept with a new and unique type. This new concept replaces the combination of concepts in the exel. The association rules can then be applied to this exel.

So far, only associations at the level of a single exel have been covered. An episode is an array of exels. To process an episode, each exel in the episode is subjected to the process described above. The result is an episode with exels that neither have communication concepts nor multiple concepts of the same type. These exels form a temporally distributed episode, that can be input to the pattern sensors. A pattern sensor is created (if one did not already exist) for each type of concept and trained with the sequence of concepts of that type. The outputs from the pattern sensors form a higher level exel, with each concept being a unique type. The association rules can now be applied to this exel.

7. Status

FemtoThnk was tested with a set of LEGO-type building blocks as objects. Several complex problems were tested. During the iterations through the Problem-solving and Dreaming processes, the system developed several good complex assemblies and used them in its solutions. Several parameters will be tuned to speed- up the system.

The system's ability to perform associations and associative recalls, were tested with up to a hundred situations. The language used for communication was mostly English. However, small tests were run with French and the south Indian language Tamil. The system behaved as expected creating appropriate associations and grammar rules. In tests with a mix of English and French, the system responded to questions with answers that were a mix of English and French!, much like a conversation between people who are comfortable with both languages.

8. References

1.The Cerebral Code, William H Calvin, MIT Press, 1996.
2. Neocognitron With Dual C-Cell Layers, Kunihiko Fukushima, Masato Okada, and Kazuhito Hiroshige, Neural Networks, Vol. 7, No. 1, 1993.
3. Parallel Computing Works, Geoffrey C. Fox. Roy D. Williams, Paul C. Messina, ISBN 1-55860-253-4, Morgan Kaufmann Publishers, Inc. 1994.

About the author

Vinu Arumugham is a Technical Leader with Cisco Systems in San Jose, Calif. He has over 20 years of experience in the design and development of computing and communication systems.