Synthetic Multimodal Graph Suite

# Synthetic Multimodal Graph Suite Four multimodal graphs whose partial information decomposition is known by construction. Each node carries one rendered image and one text caption, and the four regimes differ in where the label information sits: entirely in redundancy, entirely on the image side uniqueness, entirely in synergy, or spread across three components in the pattern measured on real multimodal benchmarks. | Regime | Label | Classes | Designed decomposition (nats) | |---|---|---|---| | `redundancy` | the node's own shape | 10 | R = 2.30 | | `unique` | mode of neighbors' texture | 10 | U_image = 2.22 | | `synergy` | a balanced binary map of two neighborhood modes, texture and size | 2 | S = 0.69 | | `mixed` | (shape group, size, texture group) | 30 | R = 1.61, U_text = 1.10, U_image = 0.69 | Components not listed are zero by construction, synergy included. The `mixed` values are those of its informative nodes; since 10% of its nodes carry no information (see Notes), measured components there scale by about 0.9. All four graphs have 14,096 nodes split into 10,000 train and 4,096 test, whose images come from disjoint pools of attribute combinations. ## Layout ``` train/ test/ rendered images, attributes encoded in the filename as shape_texture_color[_i].png graph_{regime}.pt edge_index, train_mask, test_mask, labels captions_{regime}.txt one caption per node, in node order text_embeddings_{regime}_llama-3-2-1b-instruct.pt [14096, 2048] float32 image_embeddings_{regime}_clip-vit-base-patch32.pt [14096, 512] float32 nodes_indexed_graph_{regime}.npy per-node records: image file, attributes, caption, degree, split graph_labels_{regime}.npz ground-truth factor arrays, not model input (redundancy, unique and synergy only; mixed keeps its factors in the graph file) synergy_map.npz, synergy_inputs.npz synergy only: the label map and the neighborhood modes it was applied to mixed_meta.json mixed only: config and designed components generation_{regime}.json exact command and parameters that built it embeddings_meta_{regime}.json encoders used, shapes ``` ## Loading ```python import torch g = torch.load("graph_mixed.pt") text = torch.load("text_embeddings_mixed_llama-3-2-1b-instruct.pt") image = torch.load("image_embeddings_mixed_clip-vit-base-patch32.pt") y = g["y"] # regime-correct label, see the table below edge_index = g["edge_index"] # undirected, stored in both directions train, test = g["train_mask"], g["test_mask"] ``` The per-node records are pickled object arrays, so they need `allow_pickle`: ```python import numpy as np records = np.load("nodes_indexed_graph_mixed.npy", allow_pickle=True) ``` `y` always holds the regime-correct label. The graph files also keep the other regimes' label fields for convenience, so use the named field if you address them explicitly: | Regime | field | |---|---| | `redundancy` | `share_label` | | `unique` | `graph_label` | | `synergy` | `synergy_label` | | `mixed` | `mixed_label` | `graph_mixed.pt` additionally carries `corrupted_mask` and the ground-truth factors `shape_group`, `texture_group`, `intended_shape`, `intended_size`, `intended_texture`. These describe how the data was built and are meant for auditing, not as model inputs. ## Notes Captions never mention the attribute a regime reserves for the image, and object size exists only in text because every shape is rendered at one fixed scale. In `mixed`, 10% of nodes are deliberately uninformative: their caption reads only "An object." and their image uses one of four textures held out of the label mapping, so those nodes are classifiable only through their neighbors. The embeddings are provided for convenience and can be regenerated from the images and captions with the command in each `generation_{regime}.json`, so the suite does not depend on the encoder checkpoints used here.