Lec 15 SNN-based Music Memory and Generation Based on BrainCog

Introduction

Music comes from our heart and is full of emotion and imagination, and it is closely related to our brain activities. Here, we’re inspired by the neural mechanisms of the brain at multiple scales to study these problems about music.

1. Music Memory Based On SNN

Music memory is a basic and important issue to be studied first. To research the problem of music learning and memory, we refer to the function of the hippocampus and primary auditory cortex, and build a spiking neural network in which multiple brain areas work cooperatively.

• On the micro level, the neural network was constructed by the LIF neuron model, which is provided by the BrainCog. Besides, the transmission delay of action potential and the STDP learning rule are also adopted.

• On the mesoscopic level, neural circuits are established within and between brain areas, and these circuits are updated by the STDP learning rule. These brain areas collaborate with each other to complete the musical learning and memory tasks.

• On the macro level, the model can memorize and recall the music like a human.

Inspired by the mechanisms of neurons with different selectivities for pitches and time intervals in the primary auditory cortex, we build different sub-networks to encode the pitch and duration of notes. Besides, the network can also encode different tracks of music. In the learning process, neurons receive stimulus and emit spikes with the input of the note sequence, and the synaptic connections are generated dynamically. Meanwhile, the synaptic weights can be updated by the STDP learning rule.

We use 331 classical piano pieces from 25 classical musicians as a dataset to verify the encoding, memory and recall abilities of the model. Experimental results show that the recalling accuracy based on the music name can reach 99.9%.

Our code has been published on the Braincog platform and details of the code can be found here

https://github.com/BrainCog-X/Brain-Cog/tree/main/examples/Knowledge_Representation_and_Reasoning/musicMemory/task/musicMemory.py

2. Stylistic Composition Based On SNN

Based on music memory, we propose a brain-inspired spiking neural network to generate music melodies with different styles, that is, the model can generate melodies with different genre styles or different composer styles. In this model, we also investigate the function of the prefrontal cortex of the brain and construct a network consisting of two subsystems.

The first subsystem is a hierarchical knowledge network, which is responsible for encoding and storing the basic information of music, including the information of composer, genre and music name. The other one is a sequential memory subsystem, which focuses on learning the note sequences. In the learning process, with the input of music, neurons in these two subsystems fire after they receive the stimulus, and the synapses between these active neurons are dynamically updated through the STDP learning rule. After the learning process, the model is able to generate melodies with different styles of genres or composers.

We still use 331 classical music works as a dataset to evaluate the pieces generated by our model through a user questionnaire. Experiments show that the model can well generate melodies with different styles of genres and composers, as shown in the figure below.

We have put some samples generated by our model on our website, http://www.braincog.ai/ai-music/. The code has also been published on the Braincog platform,

https://github.com/BrainCog-X/Brain-Cog/tree/main/examples/Knowledge_Representation_and_Reasoning/musicMemory/task/musicGeneration.py

If you’re interested in our research, you can download the Braincog and run it, for more information about music learning algorithms, you can refer to our published papers. If you have any questions, please email us.

@article{LQ2020,
    author  = {Liang, Qian and Zeng, Yi and Xu, Bo},
    year    = {2020},
    month   = {07},
    pages   = {51},
    title   = {Temporal-Sequential Learning With a Brain-Inspired Spiking Neural Network and Its Application to Musical Memory},
    volume  = {14},
    journal = {Frontiers in Computational Neuroscience}
}


@article{LQ2021,
    title     = {Stylistic composition of melodies based on a brain-inspired spiking neural network},
    author    = {Liang, Qian and Zeng, Yi},
    journal   = {Frontiers in systems neuroscience},
    volume    = {15},
    pages     = {21},
    year      = {2021},
    publisher = {Frontiers}
}