Biological Basis, and Theory, Underly...

One of the primary problems that traditional connectionists, and their neural networks, have been unable to solve can be stated like this:

How do biological learners deal with the dichotomy of needing to provide extremely detailed responses to any given situation, while possessing non-infinite resources with which to hold all those details?

Netlab's take on this problem has been quite different than the traditionally espoused theories and solutions[Note].

In a nutshell, Netlab recognizes that natural neural networks don't try to hold every single detail ever experienced. Instead, like all biological solutions, they act as the ultimate realists, and make the best of their real circumstances. Since they can't keep every single detail they've ever learned about how to respond to each new situation, they, instead, adapt to each new situation. How they do that can be summed up in three steps:

1. Present-moment requires a blank-slate every time we encounter it. That is, it requires memories (connection-strengths) that form very quickly in response, and then decay very quickly when we no longer need them. The only alternative is to keep every tiny detail, of every response to past "present-moments" we've ever experienced.
   
   
2. In order for blank short-term weights to adapt and respond quickly to new experiences, we must draw upon longer-term learning. We have found, however, that a new response can't simply be inserted into existing responses represented in the strengths of connections between neurons. Each learned response maintained in long-term connections must be taught by being interleaved with other responses, where each response-presentation can only have a slight effect on long-term connections.
   
   
3. This problem is solved by breaking up the speed of learning —and decay— into two (or, likely, more) different temporal spaces. First, a short term weight-space, which decays very quickly, and uses responses begun by long term responses to promote very fast learning (think hand-over-hand prompting). — The other side of this coin is a set of long term connection-strengths —at respective connections— which learn a little bit from each short-term response re-learned, as needed, in the short-term weights.

. . . . . . .
Note

Though the worst of the church have been trying (perhaps a little too hard?) to re-write the recent history (past 20 years, or so), a cursory reading of the non-back-filled literature easily demonstrates that the field had been mostly oblivious about this solution, up until after multitemporal synapses were introduced. [Plagiarism Index], e.g.,

• Complimentary Learning Systems — CLS
  What else can one do, but document the behavior, and hope the truth eventually prevails?

• How to Build a Mind? This Learning Theory May Hold the Answer

• What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated (DOI: 10.1016/j.tics.2016.05.004)

• Complementary learning systems within the hippocampus : A neural network modeling approach to reconciling episodic memory with statistical learning