Learning AI Poorly: Using Neuroimaging Techniques to Analyze Deep Neural Networks on Your Kid's Gaming Rig

(Originally posted on LinkedIn)

I’ve spent most of my career working in medical imaging and I got a little nostalgic for it after running into an old boss this week. It got me thinking, can we look at a neural network using methods neuroscientists use to learn about human brains? Better yet, can I use my kid’s gaming rig to do it? Let’s find out!

In the few years that I worked in neuroinformatics, I learned that neuroscientists basically just put people in MRI machines so they can see which parts of their brains light up when they show them pictures.* For example, here’s what happens when a neuroscientist puts someone in an MRI and shows them pictures of people vs. houses:

Why can’t we do the same thing with a neural network? You know, watch a neural network light up when it does inference on an image. I googled “visualize neural network like a neuroscientist” and of course people have already done this. Funny enough, the second hit was from MIT and it said, “Study urges caution when comparing neural networks to the brain.” I did not click on that link. Instead, I clicked on the third hit which was a paper from 2022 called “Visualizing Deep Neural Networks with Topographic Activation Maps”

The abstract includes the line, “Here, we draw inspiration from how neuroscience uses topographic maps to visualize brain activity” Perfect! I encourage you to click on that link and read the paper. Reading papers is fun, but, do you know what is more fun? The source code! You can find it here - (https://github.com/valeriekrug/ANN-topomaps)

If you turned your kid’s gaming rig into a ML/AI workstation last week, boot that thing up, pull down that code, and run it. The code takes a neural that was trained on a set of handwritten digits, keeps track of which neurons activate when you show it a new handwritten digit, and then displays those activations in a neat topographical map that mimics what we might see in the fMRI data of a real brain.

To run it, I installed Jupyter Lab which is an easy to use notebook environment. Then I created a python virtual environment. In the github project’s README you’ll see they provide a google colab version in a google drive. Since we are using our kid’s machine instead of colab, go ahead and pull down ANN-topomaps.ipynb and the models and output folders. Put them into your notebook and start running code.

Sadly, it isn’t going to work without a few changes. You can’t (or at least I couldn’t) “pip install” the dependencies in Jupyter b/c it doesn’t play nicely with the virtual environment and the dependencies listed there aren’t quite all you need. Open a command line and run, /the/path/to/your/venv/bin/python -m pip install scipy matplotlib minisom umap-learn pdf2image networkx ipywidgets I already had tensorflow, so if you don’t you’ll need that, too. Secondly, skip the part about attaching your google drive. just change shared_drive_path = '/your/local/path/to/ANN-topomaps'

Once you do that, you should be able to run everything and see a “brain map.”

This is neat b/c you can see different parts of the neural network being activated by different images. Isn’t that neat? It sort of looks like that brain picture way up there at the beginning. Neat! Is this really a good way to understand how a neural network… works? I don’t know, not really. It seems really subjective. There are a lot of different, possibly more objective, methods out there for helping us understand what is going on inside a neural network while it is doing its thing. They might be better than this… I wouldn’t know, but if you want to learn more some things to google include:

• Model Interpretability
• Model Introspection
• Shap Methods (Gradient Shap, DeepLiftShap)
• Saliency
• Feature Ablation and Permutation
• GradCAM
• LayerActivation

Some frameworks that help you with this include Captum and tf-keras-vis.

Ok, my kid wants his computer back so… until next week!

• This is… a joke. Apologies to all the neuroscientists…. but, isn’t it obvious I’m just jealous? i know.