-
Notifications
You must be signed in to change notification settings - Fork 14
Deep Temporal Memory #13
-
Just posted a first implementation of a Deep Temporal Memory -- a predictive engine with eight levels of hierarchy.
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 2 comments 1 reply
-
Hi,
I think a big issue currently with Triadic Memory is the memory usage. I am guessing that the deep temporal memory example used over 8GB RAM - which is a lot to memorize ~2KB worth of data. The problem is essentially equivalent to storing large amounts of 3D voxels, so I would suggest trying to store the memories in a sparse voxel octree.
It is also important to emphasize the generalization aspects I think - to differentiate Triadic Memory from an expensive lookup table. An obvious way of doing this would be to predict noisy results, e.g. predicting the evolution of a noisy Lorenz attractor perhaps.
Anyways, just some ideas!
Beta Was this translation helpful? Give feedback.
All reactions
-
A triadic memory instance with n=1000 and population p=10 can store 1M triples. That translates to 31001000000 bits (38MB) assuming a sparse SDR representation. The memory usage for this configuration is 954MB, so we're looking at an efficiency of 4 percent. Assuming a dense SDR representation (as realized by the brain), the efficiency would be 40 percent.
Consider that this is a redundant storage which does not require indexing...
At full capacity, one third of memory locations is 0, one third is 1, and one third has values from 2 to about 11. Would a sparse voxel octree improve memory usage when 2/3 of possible values are populated?
Great idea to use the memory as a fast lookup mechanism for solutions of differential equations -- your brain must be doing something similar when catching a ball or riding a bike. What would be a simple proof of concept?
Beta Was this translation helpful? Give feedback.
All reactions
-
The capacity may indeed be high (although I cannot verify this in practice yet), but it also needs to be able to generalize. That's why I suggest adding some generalization tests, this can be as simple as just adding some noise to a stream of inputs and seeing if it still works.
The differential equation task was just a suggestion of a task that is difficult to do with pure memorization, as it is a chaotic system.
A test on MNIST might also be interesting (using an image encoder and diadic memory). I have already written one in Odin, still playing around with some stuff though.
Regarding the sparse voxel octree, I think it may be possible to store groups of values that are the same. Not sure though, I would have to look into it more.
Beta Was this translation helpful? Give feedback.