# Multisensory Integration DEMO

In `MultisensoryIntegrationDEMO_AM.py` and `MultisensoryIntegrationDEMO_AM.py`, we implement the SNNs based multisensory integration framework. To load the dataset, preprocess it and get the weights with the function `get_concept_datase_dic_and_initial_weights_lst()`​. We use `IMNet`​ or ​`AMNet`​ to describe the structure of the IM/AM model. For presynaptic neuron, we use the function `convert_vec_into_spike_trains()​` to generate the spike trains.

While for postsynaptic neuron, we use the function `reducing_tol_binarycode()` to get the multisensory integrated output for each concept. And  *tol* is the only parameter.

In `measure_and_visualization.py`, we will measure and visualize the results.

## Multisensory Dataset

When implement the model in braincog, we use the famous multisensory dataset--BBSR.

Some examples are as follows:

| Concept   | Visual      | Somatic   | Audiation   | Taste    | Smell    |
| --------- | ----------- | --------- | ----------- | -------- | -------- |
| advantage | 0.213333333 | 0.032     | 0           | 0        | 0        |
| arm       | 2.5111112   | 2.2733334 | 0.133333286 | 0.233333 | 0.4      |
| ball      | 1.9580246   | 2.3111112 | 0.523809429 | 0.185185 | 0.111111 |
| baseball  | 2.2714286   | 2.6071428 | 0.352040714 | 0.071429 | 0.392857 |
| bee       | 2.795698933 | 2.4129034 | 2.096774286 | 0.290323 | 0.419355 |
| beer      | 1.4866666   | 2.2533334 | 0.190476286 | 5.8      | 4.6      |
| bird      | 2.7632184   | 2.027586  | 3.064039286 | 1.068966 | 0.517241 |
| car       | 2.521839133 | 2.9517244 | 2.216748857 | 0        | 2.206897 |
| foot      | 2.664444533 | 2.58      | 0.380952429 | 0.433333 | 3        |
| honey     | 1.757142867 | 2.3214286 | 0.015306143 | 5.642857 | 4.535714 |

## How to Run

To get the multisensory integrated vectors:

```
cd examples/MultisensoryIntegration/code
python MultisensoryIntegrationDEMO_AM.py
python MultisensoryIntegrationDEMO_IM.py
```

To measure and analysis the vectors:

```
cd examples/MultisensoryIntegration/code
python measure_and_visualization.py
```