Topic Modeling and Text Network Analysis on Cryptocurrency Tweets

Tags: NLP, Topic Modeling, Text Network Analysis, Python, Data Engineering

The Core Challenge: Making Sense of Unstructured Data

In 2021, my team and I advanced to the Final Round of the Gemastik Data Mining competition. While the finals brought their own unexpected challenges, the core architecture that got us there was a comprehensive Natural Language Processing (NLP) pipeline designed to analyze public sentiment around cryptocurrency in Indonesia.

Social media data—specifically Twitter—is notoriously messy. It is completely unstructured, riddled with slang, abbreviations, and grammatical inconsistencies. The objective of this project was to take months of raw Twitter data (scraped from January to May 2021) and engineer a system that could automatically discover and map the underlying sub-topics being discussed by the public.

Phase 1: Aggressive NLP Preprocessing

You cannot apply machine learning to Indonesian social media data without an aggressive preprocessing strategy. The algorithm is only as good as the vocabulary it processes.

In the Python/Jupyter Notebook pipeline, a significant portion of the script was dedicated to text cleaning. Beyond standard lowercasing and punctuation removal, I implemented a robust slang-translation module. By integrating an open-source Indonesian slang dictionary (kamus-alay), the script mapped and converted highly localized, informal Twitter jargon into standardized Indonesian vocabulary. Coupled with custom stopword removal, this ensured that the subsequent algorithms were analyzing meaningful semantic signals rather than structural noise.

Phase 2: Topic Modeling with Latent Dirichlet Allocation (LDA)

Once the corpus was clean, I applied Latent Dirichlet Allocation (LDA).

If you ask a human to read 10,000 tweets and categorize them, it would take weeks. LDA is a probabilistic model that does this automatically. The script processed the cleaned text and calculated the distribution of words to identify distinct, hidden sub-topics within the cryptocurrency discourse. Instead of just searching for specific keywords, the LDA model grouped words that frequently co-occurred, allowing us to mathematically define what the Indonesian public was actually talking about (e.g., specific coin trends, regulatory fears, or trading strategies) without manually labeling a single tweet.

Phase 3: Text Network Analysis

Identifying the topics was only the first step. To provide real analytical value, we needed to understand the relationships between these concepts.

To achieve this, I implemented Text Network Analysis on each of the resulting sub-topics generated by the LDA. Rather than just presenting a list of frequent words, the script built a relational network. This mapped out how specific terms connected to one another within the conversations, essentially visualizing the "architecture" of the public discourse. This allowed us to see not just what words were being used, but how they were linked in context.

The Business Translation

Looking back at the gemastik21 repository, the technical pipeline is exactly what enterprise clients need today.

As an IT Consultant, I frequently see businesses struggle to extract value from unstructured data (like customer feedback, reviews, or social mentions). This project was a blueprint for solving that problem. By combining robust preprocessing, LDA for topic discovery, and Text Network Analysis for relational mapping, we transformed chaotic social media noise into structured, actionable intelligence.

You can view the Jupyter Notebooks and the NLP pipeline on my GitHub: gemastik21