Lec 10 Multi-brain Areas Coordinated Brain-inspired Decision-Making SNNs Based on Braincog

Hello， in this issue, I’ll share how to build a multi-brain areas coordinated decision-making spiking neural network using Braincog

Research on brain-inspired intelligence takes inspiration from the brain’s information processing mechanisms, building a Spiking Neural Network model with brain-inspired structure and mechanism, achieving human-like cognitive levels. The Brain-inspired Decision-Making Spiking Neural Network (BDM-SNN) is a multi-brain areas coordinated spiking neural network based on the neural circuits of autonomous decision-making in the mammalian brain. Based on the findings from brain science and neuroscience, we first summarize the brain regions and connections between brain regions involved in the decision-making circuits of the mammalian brain (as shown in Figure 1). Then, we adopt SNNs to construct excitatory and inhibitory connections between brain regions. Besides, a reward-modulated synaptic plasticity rule is used to optimize network weights to achieve reinforcement learning ability.

We implemented this model using the basic components in BrainCog and verified the learning ability based on LIF neurons and simplified H-H neurons, respectively. The framework of BrainCog implementation for BDM-SNN is shown in Fig 2, the details are as follows:

Basal ganglia modeling (base/brainarea/basalganglia): we invoke node, learningrule, and connection under “base” to construct basal ganglia brain area. By assigning neuronal nodes (IFNode() or SimHHNode()) to different sub-brain areas, designing the connection matrix between different sub-brain regions, and assigning learning rules to the connections between sub-brain regions, a multiple sub-brain areas coordinated basal ganglia SNN could be achieved.

class basalganglia(nn.Module):
def __init__(self,ns,na,we,wi,node_type):
    #define connections between multi-brain areas
    bg_connection.append(CustomLinear(weight_exc*con_matrix1,con_matrix1))
    bg_connection.append(CustomLinear(weight_inh*con_matrix4,con_matrix4))
    self.node = [IFNode() for i in range(self.num_subBG)]
 self.learning_rule.append(MutliInputSTDP(self.node[2], [self.connection[2],self.connection[8]]))

BDM-SNN structure and mechanism modeling (model_zoo/bdmsnn): with the basal ganglia as the core, we further construct a prefrontal cortex-basal ganglia-thalamus-motor cortex loop to realize a whole autonomous decision-making networks from environmental state to behavioral choice and finally to execution of actions. The modeling between multiple brain areas is similar to that in step 1, which realizes the interactive connection between the basal ganglia brain area and several other brain areas. In the reinforcement learning task, the state of the environmental perception is transmitted to the prefrontal cortex and the output behavior is obtained via the BDM-SNN model. After obtaining the feedback from the environment, a reward-modulated STDP plasticity is calculated for updating the weights of the network, thus realizing the learning and optimization of the BDM-SNN.

 class basalganglia(nn.Module):
    def __init__(self,ns,na,we,wi,node_type):
    #define the architectur of BDM-SNN
    def forward(self, input):
    #Choose action based on input
    self.out[6],self.dw[6] = self.learning_rule[6](self.out[5], self.out[6])
    def UpdateWeight(self,i,dw):
    #Update weight 
    self.connection[i].update(dw)

Applying BDM-SNN on reinforcement learning task (examples/decision_making/BDM-SNN): we applied the BDM-SNN on the Flappy Bird game by invoking the BDM-SNN to select the behavior and optimizing the BDM-SNN for reinforcement learning. Experimental result of the BDM-SNN implemented by BrainCog on the Flappy Bird game is shown in Figure 3.

def chooseAct(Net,input,weight_trace_d1,weight_trace_d2):
    for i_train in range(500):
        out, dw = Net(input)
        # rstdp
        weight_trace_d1 *= trace_decay
        weight_trace_d1 += dw[0][0]
        weight_trace_d2 *= trace_decay
        weight_trace_d2 += dw[1][0]
        if torch.max(out) > 0:
            return torch.argmax(out),weight_trace_d1,weight_trace_d2,Net
def updateNet(Net,reward, action, state,weight_trace_d1,weight_trace_d2):
    r = torch.ones((num_state, num_state * num_action), dtype=torch.float)
    r[state, state * num_action + action] = reward
    dw_d1 = r * weight_trace_d1
    dw_d2 = -1 * r * weight_trace_d2
    Net.UpdateWeight(0, state,num_action,dw_d1)
    Net.UpdateWeight(1, state,num_action,dw_d2)
return Net

The specific code details can be found at

https://github.com/BrainCog-X/Brain-Cog/tree/main/examples/decision_making/BDM-SNN
https://github.com/BrainCog-X/Brain-Cog/blob/main/braincog/model_zoo/bdmsnn.py

If you are interested in our work, please feel free to cite our articles and contact us by email if you have any questions.

@article{zhao2018a,
  title={ A Brain-inspired Decision-Making Spiking Neural Network and Its Application in Unmanned Aerial Vehicle},
  author={ Feifei Zhao, Yi Zeng, Bo Xu},
  journal={ Frontiers in Neurorobotics},
  volume={12},
  year={2018},
  publisher={Frontiers Media SA}
}

@misc{https://doi.org/10.48550/arxiv.2207.08533,
  doi = {10.48550/ARXIV.2207.08533},
  url = {https://arxiv.org/abs/2207.08533},
  author = {Zeng, Yi and Zhao, Dongcheng and Zhao, Feifei and Shen, Guobin and Dong, Yiting and Lu, Enmeng and Zhang, Qian and Sun, Yinqian and Liang, Qian and Zhao, Yuxuan and Zhao, Zhuoya and Fang, Hongjian and Wang, Yuwei and Li, Yang and Liu, Xin and Du, Chengcheng and Kong, Qingqun and Ruan, Zizhe and Bi, Weida},
  title = {BrainCog: A Spiking Neural Network based Brain-inspired Cognitive Intelligence Engine for Brain-inspired AI and Brain Simulation},
  publisher = {arXiv},
  year = {2022},
}