Cross-modal analysis of the De la Cruz-Badiano codex

Materials and Methods Data, code, reproducibility, and interactive online resource All data is derived from the 1939 William Gates translation into English (public domain). All data, code, and outputs presented in this manuscript are available for download from zenodo: https://doi.org/10.5281/zenodo.21246324. The raw data includes: original English text and Nahuatl names; Spanish translations; traces of all leaves and the original traced images; a metadata sheet that provides official Nahuatl names and associated synonyms, subchapters names are mentioned, the type of Nahuatl name (plant, stone, bird, animal, or other categories), morphemes and their translations, and pages and locations of illustrations. An interactive online resource was built in markdown using MkDocs. The website is available at https://danchitwood.github.io/DeLaCruzBadiano1552/. Network co-occurrence and structural modeling To model structural relationships within the text, pairwise co-occurrence matrices were calculated at both the individual entry and component morpheme levels. Subchapter networks were constructed by tracking shared entries within each text subchapter. The global network was filtered to restrict visualization to nodes exhibiting a threshold of ten or greater. Morpheme co-occurrence networks were generated by evaluating the intersections of component morphemes parsed across all recorded Nahuatl names. Network topologies for both implementations were computed using the NetworkX framework. Graph representations were rendered using a deterministic Kamada-Kawai force-directed layout algorithm initialized with a fixed random layout seed. Text aggregation, text embedding, and semantic clustering To analyze textual similarities across the codex, English and Spanish subchapter texts were dynamically aggregated by each individual Nahuatl name. Text preprocessing normalized string sequences by mapping historical spelling variants and synonyms to their unified official names, stripping punctuation and numerical characters, and removing standard English and Spanish functional stop words. Document-level feature vectors were extracted by tokenizing the aggregated, preprocessed texts and mapping tokens to dense, 300-dimensional pretrained FastText word embeddings (from the models cc.en.300.bin and cc.es.300.bin), capturing global semantic context by computing the document-wise mean across token sequences. Pairwise document similarities were evaluated via cosine similarity matrices. For lower-dimensional distribution plotting, embeddings were projected into a two-dimensional space via Uniform Manifold Approximation and Projection (UMAP) and partitioned into five semantic clusters using K-means clustering. Distinct group vocabularies were isolated by calculating Term Frequency–Inverse Cluster Frequency (TF-ICF) scores for each partitioned token subset, effectively penalizing ubiquitous vocabulary to optimize and render unique semantic word clouds for each document cluster. Leaf shape and geometric morphometric analysis To evaluate leaf shapes in botanical illustrations across the codex, geometric morphometric analysis was used. Leaf outlines were traced starting and ending at the base of the leaf. Pseudo-landmarks were calculated by sampling 200 equally spaced, interpolated points along the perimeter from start to end. Global geometric differences arising from translation, scale, and rotation were isolated and minimized by calculating an iterative Generalized Procrustes Analysis (GPA) consensus mean shape from the outlines. All raw traces were then globally aligned and scaled against this reference target. Aligned leaf coordinate profiles were then grouped and averaged by their identification number, collapsing 177 individual illustrations to 123 consensus profiles. The reduction in the number of original illustrations to a smaller number of consensus identification numbers reflects orthographic, synonymous, and redundant illustrations of the same plant or illustrations of plants without any other mention in the text. Shape divergence between these consensus groups was evaluated by calculating a complete pairwise Procrustes distance matrix. The resulting distance matrix was projected into a lower-dimensional geometric morphospace using Multidimensional Scaling (MDS). The coordinate space was partitioned into four distinct morphological shape groups using K-means clustering, and average leaf shapes for each unique Nahuatl plant name were visualized. Cross-modal data analysis To evaluate the structural congruence across modalities, a Mantel test was used to assess pairwise distance matrix correlations. For each pairwise combination of modalities (subchapter, morpheme, English text, Spanish text, and leaf shape), matrices were subset to their common intersection of unique Nahuatl names. The Pearson correlation coefficient was calculated on the lower triangular indices of the overlapping matrices. Statistical significance was verified through permutation over 1000 iterations. In each permutation step, rows and columns of the second matrix were symmetrically shuffled to break covariance while preserving internal distance distributions. Two-tailed p-values were calculated as the proportion of permuted coefficients with an absolute value greater than or equal to the observed empirical correlation. To evaluate the predictive signals embedded within text and leaf shape modalities for morphemes, supervised machine learning was used. For English and Spanish text, a 1D Convolutional Neural Network (CNN) was trained on 300-dimensional FastText embeddings. For leaf shape, an identical classification model used a Multi-Layer Perceptron (MLP) trained on 5-dimensional Multidimensional Scaling (MDS) coordinates derived from pairwise Procrustes shape distances. A multi-label approach was used to accommodate the multiple morphemes that can be associated with a single Nahuatl name. Model performance across all three architectures was evaluated using a strict leave-one-out cross-validation (LOOCV) strategy. For each fold, a single sample was completely withheld as a testing instance, while the models were trained from scratch on the remaining training data. The class-wise true positive predictions accumulated across all independent test folds were then compiled to build a row-normalized multi-label prediction matrix. Because multiple morpheme labels can co-occur within a single sample, a post-hoc permutation testing framework was used to evaluate whether the prediction accuracies were driven by true structural signal or random chance. A baseline null distribution was generated by conducting 1000 independent randomized trials. In each trial, the true empirical morpheme label vectors were randomly shuffled across samples—breaking the authentic link between the features (whether text or leaf shape) and their annotations while strictly preserving the exact co-occurrence rates of the classes. Empirical p-values for each morpheme class were determined by calculating the proportion of the 1000 random trials where the shuffled baseline diagonal score met or exceeded the true diagonal classification hit-rate derived from the authentic LOOCV pipeline. For the text models, word clouds were extracted via a Term Frequency–Inverse Class Frequency (TF-ICF) algorithm. Results Interactive online resource Our central hypothesis tests if there are structural relationships within and between different data modalities of the De la Cruz-Badiano codex. The modalities that we test (but which the codex is not limited to) include: the co-occurence of Nahuatl names in subchapters, the co-occurence of morphemes within Nahuatl names, the similarity of text associated with Nahuatl names, and the similarity of leaf shapes in illustrations of plants. The codex is a book with sequential pages. It is extremely difficult to sort through and find specific names, morphemes, or illustrations distributed throughout the work. To aid both our own ability and that of others to analyze the work, and to allow the ability to interpret the results we present, we created an interactive online resource, publicly available at: https://danchitwood.github.io/DeLaCruzBadiano1552/. We use the 1939 William Gate’s translation in English (which is in the public domain) that is also the basis of the analysis in this manuscript. Users can access any subchapter and read the original English or a Spanish translation (Figure 1A). Alternatively, users can browse through illustrations by name (Figure 1B-D). When organized by Nahuatl name, morphemes are indicated. All Nahuatl names are hyperlinked. In the provided example in Figure 1, we start with the first subchapter, “Curation of the head”. We can select the hyperlink to the first Nahuatl name, “eca-patli”, and see its component morphemes and their meaning. We can alternatively select “eca-patli” through the “Illustrations” tab and see all illustrations of this plant, onto which are projected each individual leaf outline used in this study. Also in the tab we can access Nahuatl names that are “Plants”, “Stones”, “Animals”, “Birds”, or “Other”. By digitally curating every Nahuatl name, their synonyms, and component morphemes embedded in the text, and tracing every leaf we found across illustrations, we are able to provide an accessible resource that allows the user to access the codex in an enhanced and directed manner. Co-occurrence of Nahuatl names in subchapters and morphemes in names Nahuatl names are used to describe the principal plants, stones, birds, animals, and other entities used in the medicinal recipes. Each recipe is a short subchapter written in Latin (we are using the 1939 Gates translation in English) with the original Nahuatl names preserved. The Nahuatl names contain both orthographic variants and synonyms. We reviewed all Nahuatl names in the codex and assigned them identification numbers, collapsing related orthographic variants and synonyms and assigning an overall official name. These Nahuatl names are the only Nahuatl in the surviving codex and they provide two important pieces of structural information: the co-occurrence of Nahuatl names in subchapters and the co-occurence of component morphemes within names. We represent the co-occurrence of Nahuatl names in subchapter as a graph (Figure 2). Each node represents a Nahuatl name and each edge the two names that co-occur in the same subchapter. The width of the edge (its “weight”) indicates if the two names co-occur more than once. The overall graph has 227 nodes and 2,050 edges, but only nodes with at least 10 connections are visualized here, resulting in 109 nodes in Figure 2. Plants form a diffuse and highly interconnected network. However, the names of stones (tetl), birds (tototl), and animals are tightly clustered with each other. This reflects the repetitive listing of these names in subchapters. For example in subchapter 8l, we have lists of animals, stones, and birds that give rise to the structure seen in the graph: Also the blood of wild animals, namely the red ocelot, cuetlachtli, miztli, ocotochtli, white ocelot, tlaco-ocelotl, is sought for. With this and the above liquors the body is well anointed. Second, the precious stones quetzal-iztli, eztetl, tlacal-huatzin, tetlahuitl, red earth and the small stones in the stomachs of the birds huexo-canauhtli, hauctli and apopotli are cast into water in which they stay for a night in order that the healthful juice may be drawn out, with which the body is to be frequently bathed. Just as names co-occur within subchapters, component morphemes of names co-occur within names as well. We detected 55 different morphemes in the text. The most abundant morphemes with a count of eight or more include: xihuitl (herbs, green leaves), 37; xochitl (flower), 34; quahuitl (tree or wood) 23; patli (medicine), 18; xocotl (sour fruit), 14; atl (water), 13; tetl (stone), 10; quilitl (edible green) 9; iztac (white), 8; quetzal (precious, feathered, or green), 8. We note that the foreword of the 1939 Gates translation highlights five classes important in Nahuatl plant classification, the first three botanical in nature and the latter two economic: xihuitl, quahuitl, xochitl, patli, and quilitl. We created a co-occurrence graph for morphemes within plant names (Figure 3). The resulting network has 53 nodes and 80 weighted edges. We note the prominent use of the five plant-related morphemes in the network. The morphemes are apparent in their associated names seen in the graph. For example, patli (medicine) is found in the following plant names: quetzal-xoxouca-patli (quetzal, precious, feathered, or green; xocotl, sour fruit), tlal-patli (tlal, earth or ground), tlazol-patli (tlazol, filth), iztac-patli (iztac, white), tzonpilihuiz-patli (tzon, hair). Semantic clustering In order to understand the semantic structure of the codex, we used text embeddings. For each unique Nahuatl name, we aggregate all subchapters that the name is used. We separately analyze the original English translation from Gates (1939) as well as a Spanish translation. We remove punctuation and stop words, or uniformative, common words, in each language. The aggregated text is then vectorized, or converted into a string of numbers, by a pre-trained model, creating the text embedding for each Nahuatl name. The text embeddings reflect semantic context and accommodate out-of-vocabulary words, allowing the Nahuatl to be processed as well. To understand the semantic structure of the codex, we clustered the resulting text embeddings into five clusters in both English (Figure 4A) and Spanish (Figure 4B). Although the overall structure of each language is different, there are detectable correspondences in the world clouds between each. For example, there is a word cloud in English and Spanish with “animals”/”animales”, “ocelot”, “miztli” (puma), “ocotochtli” (bobcat), “cuetlachtli” (wolf), and “brain”/”cerebro”, reflecting the repetitive use of animals in recipes. Other corresponding word clouds between English and Spanish include one with “eyes”/”ojos”, “instilled”/”instilar”, “sauce”/”salsa”, “chilmolli” (chilli sauce), “mizquitl” (mesquite), and “matalxochitl” (green or blue flower) reflecting uses of chili-based salsas; another with “quickly”/”rápidamente”, “fevers”/”fiebres”, “birds”/”aves”, “tlapaltototl” (colorful bird), and “xiuhquecholtototl” (vibrant green bird”) reflecting repetitive uses of birds; and another with “plant”/”planta”, “roots”/”raíces”, “crushed”/”molidas”, “acacapacquilitl” (salty, edible herb), and “tlalecapatli” (plant of the earth and wind used for medicine), reflecting plants ground and used for medicine. Taken together, using an unsupervised approach, we conclude that there are repetitive, semantic structures in the text of the codex. Leaf shape analysis Plant illustrations in the codex contain information as well. In order to extract and analyze this source of embedded data, we turned to morphometrics. From 177 illustrated plants in the codex, we traced all leaves, a total of 4,645 leaves (Figure 5). The number of leaves per plant varied substantially, with a minimum of 1 leaf and maximum of 370 leaves per plant, and an average of 26.2 leaves and median of 14 leaves per illustration. After consolidating synonyms or redundantly illustrated plants, or illustrated plants without any mention in the text, there were 123 unique plants remaining. We calculated the average mean leaf shape and aligned all leaves to the global mean, after which we calculated the average leaf shape for each of 123 plants. We calculated the overall similarity in shape between each pair of means using their Procrustes distance, and visualized shape differences by embedding in two dimensions of a multidimensional scaling (MDS) plot (Figure 6). Using clusters to help highlight shape differences in this morphospace, leaf shape is both continuous but tends to also cluster into a few discrete types. At the extreme are highly elongated shapes, for example the spines of cactuses like teo-nochtli or tlaloc nochtli. We note that this group largely lacks plant names containing the morpheme huitz- (meaning “thorn” or “spine”), and that the illustrations usually depict the spines and thorns of plants distinct from their leaves. Neighboring the elongated shape group is a cluster of elongated leaves that are wider but tend to be uniform width, up to and including their tip. Following and connected to the previous group, the leaf shape becomes more deltoid and pointed at the tip. Finally, scattered and more separated from each other in distance is a group of leaves that are deeply lobed or compound, often with a tri-lobed morphology. Using an unsupervised approach, we conclude that leaf shape in the codex represents a continuous spectrum with distinct types, beginning with highly elongated leaf shapes that form a distinct cluster, followed by wider leaves that are uniform in width that are neighbored by leaves with a distinct, deltoid tip, before leading to deeply lobed or compound leaves that tend to be tri-foliate. Multi-modal data analysis The principal hypothesis we sought to answer was whether or not there is any structure within and between the different data modalities of the codex. In the previous sections we have addressed the four different modalities in turn: co-occurence of Nahuatl names in subchapters; co-occurence of morphemes within Nahuatl names; text associated with Nahuatl names (both English and Spanish); and leaf shapes associated with Nahuatl names. In the analysis of each data modality, we kept pairwise distance matrices of each, structured by Nahuatl name: co-occurrence of names and morphemes, cosine similarity of text embeddings, and Procrustes distance between averaged leaf shapes. To determine if there is a global relationship between these modalities, we turn to the Mantel test. Given two pairwise distance matrices of the same size with corresponding samples, the Mantel test calculates the correlation between corresponding distances. For all pairwise Mantel tests between data modalities, only the following were significant, calculated by permutation (p = 0.001): subchapter and English text; subchapter and Spanish text; and English text and Spanish text. The above significant correlations are expected and serve as a positive control. Because the text embeddings include out-of-vocabulary words and Nahuatl names, we expect subchapter co-occurrence and text to be highly correlated, and similarly we expect translations of the same text in English and Spanish to be highly correlated. Any other pairing with morpheme or leaf shape (or text, except with subchapter or translations) was not significant (Table 1). We conclude that, at a global level, there are no significant associations between data modalities. Although we detected no signal at a global level between modalities, that doesn’t mean that there isn’t local signal or individually strong relationships. Because subchapters and Nahuatl names are confounded as we describe above, we sought to look for relationships between data modes and morphemes, the smallest unit of semantic meaning. The foreword to the Gates translation (1939) suggests that five morphemes—xihuitl, quahuitl, xochitl, patli, and quilitl—have special importance in describing plants, and indeed we find evidence for this (Figure 3). To make the machine learning models unbiased as possible, we both consider all morphemes with instances above five to accommodate a leave-one-out cross-validation (LOOCV) strategy and use multi-label modeling and permutation testing that accounts for multi-label distributions. We model morpheme classes as a function of text embeddings (both English and Spanish) as well as average leaf shapes. We used the previous calculated text embeddings in English and Spanish (Figure 4) to find associations with morphemes. In English, the following morphemes were significantly associated with text (p values in parentheses): tetl (0.001), tototl (0.004), xiuhuitl (0.032). In Spanish, the following morphemes were associated with text: tetl (0.000), tototl (0.005), quahuitl (0.013), tlal (0.045). In both languages, tetl is associated with “groin”/“ingle”, “pain”/”dolores”, “swollen”/”hinchadas”, “teamoxtli” (papery stone plant), and “texiyotl” (the split plant) (Figure 7A). Tototl is associated with “death”/”muerte”, “eyes”/”ojos”, “birds”/”aves”, “tlapaltototl” (colorful bird), and “xiuhquecholtototl” (vibrant green bird”), similar to the “bird” word cloud grouping in Figure 4 (Figure 7B). For the botanical morphemes, in English the text associated with “xihuitl” include “yxiuh” itself, “pepper”, “feet”, “tonatiuh” (the shining or hot plant), and “xiuhtontli” (little plant) (Figure 7C), and in Spanish the text associated with “quahuitl” has animal associations similar to the “animal” grouping seen in Figure 4, including“ocelote”/”ocelot”, “animales”/”animals”, and “miztli” (puma) but also “cansancio”/”fatigue”, “cacloxochitl” (raven flower), and “cedro”/”cedar” (Figure 7D). Lastly, Spanish associations with “tlal” include uses of the morpheme itself in “tlalahuehuetl” (ahuehuete of the earth) and “tlalmizquitl” (mesquite of the earth) as well as “gusanos”/“worms”, as we would associate with “earth” (Figure 7E). Only quahuitl was significantly associated with leaf shapes. The average leaf shapes of quahuitl plants are shown in Figure 7E, ranging from the deeply lobed leaf class seen in ayauh-quahuitl to the more evenly tapered leaves of a-quahuitl, quauh-yyauhtli, and tlanextia-quahuitl. Common features in leaves of quahuitl plants include a relatively distinct leaf base or petiole and leaf tip. Figure Legends Figure 1: Interactive online resource. A) Screenshot of the first subchapter, “Curation of the head”. Original text in English and Spanish translation available in the tabs. The three illustrated plants in the first subchapter, xiuh-ecapatli, itzac ocoxochitl, and te-amoxtli, are shown with each leaf traced in color. B) Alternatively, the hyperlinked name eca-patli itself can be clicked taking the user to a page listing its variant spellings and component morphemes. Scrolling through the eca-patli page the user can find other illustrations, synonyms, and references to the plant across the codex, including C) xiu-eca-patli and D) heca-pahtli. Only leaves for the indicated plant (in this case, eca-patli and synonyms) are traced. Figure 2: Nahuatl name co-occurence in subchapters. Each node is a Nahuatl name and edges indicate co-occurrence between names in subchapters. Edge width (weight) is proportional to the number of co-occurrences. Nodes are limited to names with 10 or more co-occurrences with other names. Nodes are colored by the type of name: plant, teal; stone (tetl), magenta; bird (tototl), purple; animal, orange. Figure 3: Morpheme co-occurrence in Nahuatl names. Each node is a morpheme and edges indicate co-occurrence within Nahuatl names. Node size and color, as indicated in legend, are proportional to the number of times the morpheme co-occurs within names. Figure 4: Semantic clustering of text embeddings. Clustering of A) original English text embeddings from the 1939 Gates translation and B) Spanish translations. Each point represents the aggregated text embedding of a unique Nahuatl name. Five clusters are indicated by color and corresponding frequency-based word clouds for each. The clusters and their colors do not indicate correspondence between English and Spanish. Figure 5: Leaf traces. For the 177 plant illustrations found in the codex (shown in white), the 4,645 traced leaves are shown (depicted as filled shapes in various colors). Figure 6: Morphospace of average leaf shapes. For 123 uniquely identified plants from 177 plant illustrations, average leaf shapes were calculated. A Procrustean pairwise distance matrix was calculated between all average shapes and embedded in two dimensions using multidimensional scaling (MDS). Four distinct clusters are indicated by color ranging from highly elongated leave (green), leaves with uniform width to the tip (magenta), leaf shapes with a pointed, deltoid tip (orange), to deeply lobed or compound leaves that tend to be tri-foliate (lavander). Figure 7: Cross-modal relationships with morphemes. A) Word clouds showing a significant association between text and the morpheme tetl, in English (left) and Spanish (right). B) Word clouds showing a significant association between text and the morpheme tototl, in English (left) and Spanish (right). Word clouds showing significant associations between text and the morphemes C) xihuitl in English, D) quahuitl in Spanish, and E) tlal in Spanish. F) Average leaf shapes for plants with the morpheme quahuitl, which is significantly associated with leaf shape. Table 1: Summary of Mantel test results Mode 1 Mode 2 Corresponding Samples (N) Correlation Coefficient (r) p-value Subchapter Morpheme 228 0.01589 0.162 Subchapter English Text 228 0.35085 0.001 Subchapter Spanish Text 228 0.3519 0.001 Subchapter Leaf shape 123 -0.004 0.903 Morpheme English Text 228 -0.00518 0.811 Morpheme Spanish Text 228 -0.01038 0.631 Morpheme Leaf shape 123 -0.02634 0.337 English Text Spanish Text 228 0.92336 0.001 English Text Leaf shape 123 -0.02921 0.676 Spanish Text Leaf shape 123 -0.03846 0.6