A conversation on the train

This past Saturday I got to use the absolute nerdiest opening to a conversation that has ever crossed my lips. I was on the train, which had been stuck the station before so it was packed sardine-like with all the people who had built up waiting on the platform. Standing next to me were a group of people who I overheard talking, and then I suddenly turned and asked, "Excuse me, but I couldn't help overhearing. Were you talking about modeling the thalamo-cortical loop?"

It seriously made my day. Particularly because I had a paper about that in my hands at the moment. I mean really, what are the odds?

What was particularly encouraging is that I generally held my own in the ensuing conversation, despite the fact that they were studying it actively, and I'm just trying to pick it up on my own enough to study it. We started talking about neuron and cortical column models, and before they got off a couple stops later, one guy suggested I look into something called the max model, which apparently compares the maximum input to the neuron's threshold level, rather than summing them. The obvious question in my mind is how such a system could handle inhibition, other than resorting to summation or perhaps taking the input of the greatest magnitude, which would allow a high-magnitude negative input to prevent firing. Anyway, I'm going to look into it.

Recently I've been reading a lot from UT Austin, trying to size up their program there. Some of my favorites have been:

Yoonsuck Choe's Analogical Cascade: A Theory on the Role of the Thalamo-Cortical Loop in Brain Function, from 2002

Srinivasa V. Chakravarthy and Joydeep Gosh's Studies on a network of complex neurons, a paper on the role of chaos in a more complex variant on a hopfield model

And I've also been enjoying Eytan Ruppin's Neural Modeling of Psychiatric Disorders, from 1995. It's probably a bit out of date, but it provides a nice overview of attempts to model various disorders such as alzheimers and schizophrenia, which is a very interesting area to me, because it should provide a good test in the future to see if we're close to the way human brains work. (Damage them in the same way, and see if the same effects occur)

Practical playing around - and some other papers I've found interesting.

I always lean best by doing, and at this point I've been reading and reading for weeks, so I decided to make some of my own neural memories, just for kicks. I went back to Kosko's 1987 Adaptive bidirectional associative memories paper, and have just about gotten one working in python.

An ABAM is essentially a neural network with full feedback, where the weights (which can be thought of as long-term memory) are manipulated to store an association between two pieces of information as a stable state. That is, when information A (which in the network can be thought of as a short-term memory of A) exists on one side of the network, then it produces B, which in turn produces A. The great bit is that these memories can stabilize on known states if the input starts out as being somewhat close to one or the other. The cleverly named bidirectional model can recall an association from either direction, so I could start it out with something like B, which would then produce something like A, which produces something a bit closer to B, which produces something a bit closer to A, until the stable state is reached and no more change occurs in the two short-term memory sections. Hopefully my explanation made enough sense to convince you that these things are pretty awesome.

I unfortunately had issues getting numpy set up on my Zaurus, so I had to write the matrix math functions myself, which slowed things down. (If you don't know what a zaurus is, it's my little Japanese palmtop. If you at all like to tinker with computers, I highly suggest you check it out) I do a lot of my reading on the train and bus during my rather long daily commute, and so it's been particularly handy to be able to program readily while I'm in transit as well.

Anyway, I figure an ABAM is a nice starting point, and from there I can begin to play more with input filtering and try linking them together in different ways. I think the first thing I'm going to do is try to get one that can learn up simple melodies. Pretty much all my practical experience up to this point has been with feed-forward networks, because they're faster and make it easier to get immediately applicable results, and the business world likes that.

Anyway, while the programming has been fun, my literature search continues. I'm hopefully meeting with a professor at Johns Hopkins sometime over the coming week to talk about their program, which has me very excited! On top of that, here are a couple of the niftier papers I've read recently:

Bidirectional Associative Memories, by Yeou-Fang Wang, Jose B. Cruz, Jr., and James H. Mulligan, Jr.
(this one is a good overview of BAMs, as far as I can tell)

Biological Plausibility of Artificial Neural Networks: Learning by Non-Hebbian Synapses, by Daniel L. Alkon, Kim T. Blackwell, Thomas P. Vogl, and Susan A. Werness.
(I'm finally starting to find more biologically-based papers, and even though this one is old it was a fascinating one, where they studied the central nervous system of snails and rabits in an attempt to craft a more accurate model of synaptic learning than the standard Hebbian model - which has been in use since we knew next to nothing about how nerves worked.)