Multi-Channel Image Data Analysis using Sonification

In biomedicine as well as in many other areas experimental data consists of topographically ordered multidimensional data arrays or images.

In our collaboration, multi parameter flourescence microscopy data of immunoflourescently labeled lymphocytes has to be analysed. One experimental data set consists of n intensity images of the sample. As a result of a specific immunolabeling technique in each image different subsets of the lymphocytes appear with high intensity values, expressing the existence of a specific cell surface protein. Because the positions of the cells are not affected by the labeling process, the n flourescence signals of a cell can be traced through the image stack at constant coordinates.

The analysis of such stacks of images by an expert user is limited to two strategies in most laboratories: the images are analyzed one after the other or up to three images are written into the RGB channels of a color map. Obviously, these techniques are not suitable for the analysis of higher dimensional data.

Here, Sonification of the stack of images allows to perceive the complete pattern of all markers.
The biomedical expert may probe specific cells on an auditory map and listen to their flourescence patterns.
The sonification was designed to satisfy specific requirements:

* **Identification** - Cells with identical patterns should very easily be perceived as identical sounds
* **Similarity** - Similar cell flourescence patterns should lead to sonifications that sound similar
* **Extensibility** - the sonification should be extensible, so that the future addition of markers does not change the sound characteristic, driven by the other markers
* **Short Duration** - the whole sonification should last only a short time of about 1 sec, to allow a fast browsing of the image.

Such sonifications can be derived using several strategies. One is to play a tone for each marker if the corresponding flourescence intensity is more than a threshold. Thus a **rythmic pattern** emerges for each cell.
Another strategy is to use frequency to distinct markers. Thus each cell is a superposition of tones with different pitch and a chord or tone-cluster is the result. This leads to a **harmonic presentation** of each cell.
However, using both time and pitch, the result is a rythmical sequence of tones and thus a specific melody for a cell.
As our abbilities to memorize and recognice melodies or musical structures is better than recognizing visual presented histograms, this yields a promising approach for the inspection of such data by an expert. Now, an example sonification is presented using only five dimensional data images. However, the results are even good with much higher dimensionality - we tested the method with a stack of 12 images.
The following demonstration uses only 5 markers. A map is rendered to show all cells for browsing (shown right)

<table COLS=4 WIDTH="85%">
<tr ALIGN=LEFT VALIGN=TOP>
<td NOSAVE><img SRC="https://pub.uni-bielefeld.de/download/2763993/2763999" height=100 width=120 align=TEXTTOP>
<br><b>cd-02</b></td>

<td><img SRC="https://pub.uni-bielefeld.de/download/2763993/2764001" height=100 width=120>
<br><b>cd-08</b></td>

<td></td>

<td ><b>Identical patterns:</b>
<br>&nbsp;<a href="https://pub.uni-bielefeld.de/download/2763993/2763994">Cell 1</a>
<br>&nbsp;<a href="https://pub.uni-bielefeld.de/download/2763993/2763995">Cell 2</a></td>
</tr>

<tr ALIGN=LEFT VALIGN=TOP>
<td><img SRC="https://pub.uni-bielefeld.de/download/2763993/2763998" height=100 width=120>
<br><b>cd-03</b></td>

<td>
<br>&nbsp;
<br>&nbsp;
<br>&nbsp;
<br>&nbsp;
<p><b>------</b></td>

<td><img SRC="https://pub.uni-bielefeld.de/download/2763993/2764003" height=100 width=120 align=TEXTTOP>
<br><b>superposition</b></td>

<td><b>Similar pattern:</b>
<br>&nbsp;<a href="https://pub.uni-bielefeld.de/download/2763993/2763996">Cell 3</a></td>
</tr>

<tr ALIGN=LEFT VALIGN=TOP>
<td><img SRC="https://pub.uni-bielefeld.de/download/2763993/2764000" height=100 width=120 align=TEXTTOP>
<br><b>cd-04</b></td>

<td><img SRC="https://pub.uni-bielefeld.de/download/2763993/2764002" height=100 width=120 align=TEXTTOP>
<br><b>hla-dr</b></td>

<td></td>

<td><b>Very different pattern:</b>
<br>&nbsp;<a href="https://pub.uni-bielefeld.de/download/2763993/2763997">Cell 4</a></td>
</tr>
</table>

A specific advantage of this method is, that it allows to examine the high-dimensional data vectors without the need to change the viewing direction. However, there are many other methods to present such data acoustically, e.g. by using different timbre classes for the markers, like percussive instruments, fluid sounds, musical instruments or the human voice. These alternatives and their applicability are currently investigated.

在生物医学以及其他众多领域，实验数据通常由按地形顺序排列的多维数据阵列或图像组成。在我们的合作项目中，必须分析免疫荧光标记的淋巴细胞的多参数荧光显微镜数据。一个实验数据集包含n个样本强度图像。由于特定的免疫标记技术，每张图像中都会出现具有高强度的不同淋巴细胞子集，这表明存在特定的细胞表面蛋白。由于细胞的位姿不受标记过程的影响，因此可以通过图像堆栈在恒定坐标中追踪单个细胞的n个荧光信号。在大多数实验室中，专家用户分析此类图像堆栈的策略有限，通常只有两种：依次分析图像，或将多达三个图像写入彩色地图的RGB通道。显然，这些技术不适用于高维数据的分析。在此，图像堆栈的声学转化使得可以感知所有标记的全局模式。生物医学专家可以在听觉地图上探测特定细胞并聆听其荧光模式。该声学转化旨在满足特定要求：

* **识别** - 拥有相同模式的细胞应能非常容易地被感知为相同的声波。
* **相似性** - 相似细胞的荧光模式应导致相似的声波转化。
* **扩展性** - 声学转化应具有可扩展性，以便未来标记的添加不会改变由其他标记驱动的声音特征。
* **短时性** - 整个声学转化应仅持续约1秒，以便快速浏览图像。

通过多种策略可以获得此类声学转化。其中一种策略是，如果对应的荧光强度超过阈值，则为每个标记播放一个音调。因此，每个细胞都会出现一种**韵律模式**。另一种策略是使用频率来区分标记。因此，每个细胞都是不同音高的音调的叠加，结果是**和声呈现**。然而，使用时间和音高，结果是一种音调的韵律序列，因此是一个细胞的特定旋律。由于我们记忆和识别旋律或音乐结构的能力优于识别视觉呈现的直方图，这为专家检查此类数据提供了一种有前景的方法。现在，使用仅包含五维数据图像的示例声学转化进行展示。然而，对于更高的维度，结果同样良好——我们已测试了12张图像的堆栈。以下演示仅使用5个标记。一个地图被渲染出来以显示所有细胞以供浏览（如图所示）

<table COLS=4 WIDTH="85%"><tr ALIGN=LEFT VALIGN=TOP><td NOSAVE><img SRC="https://pub.uni-bielefeld.de/download/2763993/2763999" height=100 width=120 align=TEXTTOP><br><b>cd-02</b></td><td><img SRC="https://pub.uni-bielefeld.de/download/2763993/2764001" height=100 width=120><br><b>cd-08</b></td><td></td><td ><b>相同模式：</b><br>&nbsp;<a href="https://pub.uni-bielefeld.de/download/2763993/2763994">细胞1</a><br>&nbsp;<a href="https://pub.uni-bielefeld.de/download/2763993/2763995">细胞2</a></td></tr><tr ALIGN=LEFT VALIGN=TOP><td><img SRC="https://pub.uni-bielefeld.de/download/2763993/2763998" height=100 width=120><br><b>cd-03</b></td><td><br>&nbsp;<br>&nbsp;<br>&nbsp;<br>&nbsp;<p><b>------</b></td><td><img SRC="https://pub.uni-bielefeld.de/download/2763993/2764003" height=100 width=120 align=TEXTTOP><br><b>叠加</b></td><td><b>相似模式：</b><br>&nbsp;<a href="https://pub.uni-bielefeld.de/download/2763993/2763996">细胞3</a></td></tr><tr ALIGN=LEFT VALIGN=TOP><td><img SRC="https://pub.uni-bielefeld.de/download/2763993/2764000" height=100 width=120 align=TEXTTOP><br><b>cd-04</b></td><td><img SRC="https://pub.uni-bielefeld.de/download/2763993/2764002" height=100 width=120 align=TEXTTOP><br><b>hla-dr</b></td><td></td><td><b>差异很大的模式：</b><br>&nbsp;<a href="https://pub.uni-bielefeld.de/download/2763993/2763997">细胞4</a></td></tr></table>

该方法的特定优势在于，它允许在无需更改观察方向的情况下检查高维数据向量。然而，还有许多其他方法可以以声学方式呈现此类数据，例如，通过使用不同音色的标记，如打击乐器、流体声音、乐器或人声。这些替代方案及其适用性目前正在研究中。