Practical playing around - and some other papers I've found interesting.

I always lean best by doing, and at this point I've been reading and reading for weeks, so I decided to make some of my own neural memories, just for kicks. I went back to Kosko's 1987 Adaptive bidirectional associative memories paper, and have just about gotten one working in python.

An ABAM is essentially a neural network with full feedback, where the weights (which can be thought of as long-term memory) are manipulated to store an association between two pieces of information as a stable state. That is, when information A (which in the network can be thought of as a short-term memory of A) exists on one side of the network, then it produces B, which in turn produces A. The great bit is that these memories can stabilize on known states if the input starts out as being somewhat close to one or the other. The cleverly named bidirectional model can recall an association from either direction, so I could start it out with something like B, which would then produce something like A, which produces something a bit closer to B, which produces something a bit closer to A, until the stable state is reached and no more change occurs in the two short-term memory sections. Hopefully my explanation made enough sense to convince you that these things are pretty awesome.

I unfortunately had issues getting numpy set up on my Zaurus, so I had to write the matrix math functions myself, which slowed things down. (If you don't know what a zaurus is, it's my little Japanese palmtop. If you at all like to tinker with computers, I highly suggest you check it out) I do a lot of my reading on the train and bus during my rather long daily commute, and so it's been particularly handy to be able to program readily while I'm in transit as well.

Anyway, I figure an ABAM is a nice starting point, and from there I can begin to play more with input filtering and try linking them together in different ways. I think the first thing I'm going to do is try to get one that can learn up simple melodies. Pretty much all my practical experience up to this point has been with feed-forward networks, because they're faster and make it easier to get immediately applicable results, and the business world likes that.

Anyway, while the programming has been fun, my literature search continues. I'm hopefully meeting with a professor at Johns Hopkins sometime over the coming week to talk about their program, which has me very excited! On top of that, here are a couple of the niftier papers I've read recently:

Bidirectional Associative Memories, by Yeou-Fang Wang, Jose B. Cruz, Jr., and James H. Mulligan, Jr.
(this one is a good overview of BAMs, as far as I can tell)

Biological Plausibility of Artificial Neural Networks: Learning by Non-Hebbian Synapses, by Daniel L. Alkon, Kim T. Blackwell, Thomas P. Vogl, and Susan A. Werness.
(I'm finally starting to find more biologically-based papers, and even though this one is old it was a fascinating one, where they studied the central nervous system of snails and rabits in an attempt to craft a more accurate model of synaptic learning than the standard Hebbian model - which has been in use since we knew next to nothing about how nerves worked.)