NLP Analysis Reveals Structured Language Patterns in Mysterious Voynich Manuscript

Voynich Manuscript Structural Analysis This project began as a personal challenge to explore what modern Natural Language Processing (NLP) techniques could reveal about the mysterious Voynich Manuscript, without delving into speculative translations or imagined patterns. I am neither a linguist nor a cryptographer, but I was curious to see if a language as enigmatic as Voynichese could exhibit traits consistent with structured, real languages when subjected to advanced modeling methods such as clustering, Part-of-Speech (POS) inference, Markov transitions, and section-specific pattern analysis. Surprisingly, it did. Why This Matters The Voynich Manuscript remains one of the greatest unsolved puzzles in history, with no widely accepted linguistic or cryptographic explanation. Traditional approaches to understanding it have typically split into two categories: statistical entropy checks, which involve measuring randomness, and wild guesswork, which often lacks scientific rigor. This project aims to bridge the gap by applying computational linguistics to determine whether the manuscript displays the structural properties of a real language, regardless of what it actually says. Project Structure Key Contributions Suffix Stripping: A crucial preprocessing step involved removing recurring suffix-like endings from each word, such as "aiin," "dy," and "chy." This allowed for isolating potential root forms and enhancing the clustering behavior. Words with similar stems grouped more tightly, and the transition matrix displayed cleaner and more coherent patterns. Clustering Analysis: Utilizing techniques like SBERT embeddings, I created visual representations of word clusters, revealing a structure that suggests the presence of a syntactic framework. Transition Matrix Heatmap: This visualization highlighted the frequency and sequence patterns of words, further supporting the notion of a structured language. Preprocessing Choices One of the most significant assumptions I made was to strip common suffixes from Voynich words. The rationale was to uncover root forms that repeat with variations, akin to suffix usage in many known languages. This approach markedly improved the clustering and transition matrices, showing more defined and coherent groupings. However, this preprocessing decision is not neutral; it influenced the results. For those interested in comparing outcomes without suffix stripping or by treating suffixes as distinct token classes, feel free to fork this repository and conduct your own analyses. I would be genuinely interested in any insights you may gain. Key Findings Structured Language: The Voynich Manuscript exhibits syntax and a clear distinction between function and content words, which are characteristic of real languages. Section-Specific Linguistic Shifts: Different sections of the manuscript show variations in language patterns, suggesting that the content is organized and purposeful. Syllabic Padding and Positional Repetition: These features indicate that the manuscript might encode a constructed or mnemonic language. Hypothesis The Voynich Manuscript likely contains a structured, constructed language or a mnemonic system characterized by syllabic padding and positional repetition. It displays syntax, a separation between function and content words, and consistent changes in linguistic patterns across different sections, all of which are hallmarks of a well-formed language. How to Reproduce To replicate the results, follow these steps: Clone the repository. Preprocess the text by stripping suffixes or treating them as separate tokens. Perform clustering using SBERT embeddings and reduce dimensions with PCA for visualization. Generate a transition matrix and create a heatmap to analyze word sequences. Example Visualizations Figure 1: SBERT Cluster Embeddings (PCA-reduced): This figure provides a visual representation of word embeddings, showing how similar words group together in multidimensional space. Figure 2: Transition Matrix Heatmap: This heatmap illustrates the frequency and sequence of word transitions, highlighting the structured nature of the manuscript. Limitations While the findings are intriguing, this project has several limitations: Preprocessing Assumptions: The removal of suffixes might introduce biases and affect the results. Lack of Ground Truth: Without a known reference language, it's challenging to validate the structural analysis fully. Generalizability: The models and techniques used may not be applicable to all types of texts or languages. Author's Note This project was designed as an educational journey to explore the capabilities of AI and NLP in analyzing a language that defies conventional understanding. My goal was not to decipher the Voynich Manuscript but to investigate its structure using modern tools and methods. By taking this approach, I hope to contribute to a more nuanced and scientifically grounded debate about the nature of the manuscript. If you're looking for a definitive translation, you'll be disappointed. However, if you're interested in modeling a language that challenges our perceptions, your participation is welcome. Contributions Welcome I encourage extensions, critiques, and collaborations from experts in linguistics, cryptography, constructed languages, and computational language research. Your input can help refine the analysis and deepen our understanding of this fascinating document.