**Tags:** System Architecture, Data Logic, Scalability, Infrastructure, Post-Mortem

### The MVP: Engineering Database Logic in a Spreadsheet (2019)
Before writing a single line of backend code, an IT system usually starts as a Minimum Viable Product (MVP) to solve an immediate business need. In 2019, my faculty needed a way to conduct student union elections digitally, but we had absolutely zero budget.

My solution was highly manual but functionally robust. I designed a unique, single-use token system using Google Forms to collect the data. To validate the votes, I essentially turned a spreadsheet (`pemilu.xlsx`) into a relational database and validation engine.

In the `master` sheet, I mapped every generated TOKEN and built a logic flow to audit the incoming form submissions. The core mechanism relied on a *Terhitung sebanyak* (Counted as) column. Because Google Forms couldn't prevent a user from submitting twice, the spreadsheet had to handle the data integrity.

* If a student voted, the spreadsheet cross-referenced their token. The very first chronological entry tied to a valid token was successfully mapped to the *Voted for* column and flagged as *Suara Sah* (Valid Vote).
* If a student tried to cheat the system and use the same token again for a different candidate, the logic caught the duplicate usage.
* Any duplicate entries, or any random/unrecognized token strings inputted by users, were automatically filtered out and categorized as *Suara Tidak Sah* (Invalid Vote).

It was a manual, administrative headache to distribute the tokens, but the logic held up perfectly. I had essentially built primary key constraints and data validation using spreadsheet formulas.

### Iteration 1: The Flask Monolith (2020)
By 2020, it was time to automate. I replaced the `pemilu.xlsx` spreadsheet system with a fully functional monolithic web application (`vote-app`) built using Flask and SQL.

This system automated the token validation and securely stored the voting data. We deployed it for the faculty-level election, which consisted of fewer than 500 students spanning four academic years. Because the user base was small and traffic was highly predictable, the monolithic architecture was more than capable. The system ran flawlessly, and I thought I had "solved" digital voting.

### Iteration 2: Decoupled Architecture and The University Challenge (2021)
Success brings new problems. In 2021, the university-level student union approached me to adapt the system for the entire campus. Instead of hundreds of voters, we were now looking at thousands. Furthermore, they came with a very short notice.

Realizing the scale of the project, I teamed up with a colleague to upgrade the architecture. We transitioned from a monolith to a decoupled system, separating the backend API (`vote-app-v2`) from the frontend interface (`fe-voting-apps`). From a pure software engineering perspective, the code was cleaner, more modern, and ready to scale.

However, code does not run in a vacuum.

Anticipating the massive spike in concurrent users, I submitted a request to the university for dedicated server resources. The request was denied. Left with no budget and no official infrastructure, I had to deploy the entire decoupled architecture onto free-tier servers (Heroku).

### The Bottleneck: Surviving a D-Day Crash
When election day arrived, the inevitable happened. As thousands of students flooded the system to cast their votes simultaneously, we slammed directly into the free server's bandwidth and transaction rate limits.

The application choked. The database connections timed out, the API endpoints dropped requests, and the system essentially bottlenecked under the load. Fortunately, because I was skeptical of running enterprise-level traffic on free servers, I had explicitly warned the university committee during the planning phase to prepare a manual backup plan. When the digital system went south, they were able to pivot to the fallback method.

### The Post-Mortem Takeaway
This project remains one of the most formative experiences of my career. It taught me a brutal but necessary lesson: **Software architecture is useless without the infrastructure to support it.**

You can decouple your frontend and backend, write clean API endpoints, and design brilliant database schemas, but if your server cannot handle the concurrent network requests, your product will fail. Today, as an IT Consultant, this experience directly influences how I design enterprise solutions. I never just look at the application layer; I look at the resource allocation, the load testing parameters, and the worst-case fallback scenarios. Sometimes, the best way to learn how to build scalable systems is to watch one crash.

*You can view the evolution of the codebases on my GitHub:*
* [vote-app (2020 Monolith)](https://github.com/SokKanaTorajd/vote-app)
* [vote-app-v2 (2021 Backend API)](https://github.com/SokKanaTorajd/vote-app-v2)
* [fe-voting-apps (2021 Frontend)](https://github.com/akhfzz/fe-voting-apps)

The Architectural Evolution of a Campus E-Voting System

**Tags:** NLP, Topic Modeling, Text Network Analysis, Python, Data Engineering

### The Core Challenge: Making Sense of Unstructured Data
In 2021, my team and I advanced to the Final Round of the Gemastik Data Mining competition. While the finals brought their own unexpected challenges, the core architecture that got us there was a comprehensive Natural Language Processing (NLP) pipeline designed to analyze public sentiment around cryptocurrency in Indonesia.

Social media data—specifically Twitter—is notoriously messy. It is completely unstructured, riddled with slang, abbreviations, and grammatical inconsistencies. The objective of this project was to take months of raw Twitter data (scraped from January to May 2021) and engineer a system that could automatically discover and map the underlying sub-topics being discussed by the public.

### Phase 1: Aggressive NLP Preprocessing
You cannot apply machine learning to Indonesian social media data without an aggressive preprocessing strategy. The algorithm is only as good as the vocabulary it processes.

In the Python/Jupyter Notebook pipeline, a significant portion of the script was dedicated to text cleaning. Beyond standard lowercasing and punctuation removal, I implemented a robust slang-translation module. By integrating an open-source Indonesian slang dictionary (`kamus-alay`), the script mapped and converted highly localized, informal Twitter jargon into standardized Indonesian vocabulary. Coupled with custom stopword removal, this ensured that the subsequent algorithms were analyzing meaningful semantic signals rather than structural noise.

### Phase 2: Topic Modeling with Latent Dirichlet Allocation (LDA)
Once the corpus was clean, I applied Latent Dirichlet Allocation (LDA).

If you ask a human to read 10,000 tweets and categorize them, it would take weeks. LDA is a probabilistic model that does this automatically. The script processed the cleaned text and calculated the distribution of words to identify distinct, hidden sub-topics within the cryptocurrency discourse. Instead of just searching for specific keywords, the LDA model grouped words that frequently co-occurred, allowing us to mathematically define what the Indonesian public was actually talking about (e.g., specific coin trends, regulatory fears, or trading strategies) without manually labeling a single tweet.

### Phase 3: Text Network Analysis
Identifying the topics was only the first step. To provide real analytical value, we needed to understand the relationships between these concepts.

To achieve this, I implemented Text Network Analysis on each of the resulting sub-topics generated by the LDA. Rather than just presenting a list of frequent words, the script built a relational network. This mapped out how specific terms connected to one another within the conversations, essentially visualizing the "architecture" of the public discourse. This allowed us to see not just what words were being used, but how they were linked in context.

### The Business Translation
Looking back at the `gemastik21` repository, the technical pipeline is exactly what enterprise clients need today.

As an IT Consultant, I frequently see businesses struggle to extract value from unstructured data (like customer feedback, reviews, or social mentions). This project was a blueprint for solving that problem. By combining robust preprocessing, LDA for topic discovery, and Text Network Analysis for relational mapping, we transformed chaotic social media noise into structured, actionable intelligence.

*You can view the Jupyter Notebooks and the NLP pipeline on my GitHub: [gemastik21](https://github.com/SokKanaTorajd/gemastik21)*

Topic Modeling and Text Network Analysis on Cryptocurrency Tweets

**Tags:** Web Scraping, API Integration, Python, Flask, Data Engineering

### The Challenge
Social media data is one of the most requested assets by businesses trying to understand market trends. In a past project, the objective was straightforward: build a system to extract data and automate posts across Twitter, Facebook, and Instagram.

Initially, the plan was to play by the book. I engineered integrations using the official Twitter API and the Facebook Graph API. However, in enterprise and high-volume environments, "playing by the book" often means hitting a brick wall.

### The Official APIs
Integrating with the Facebook Graph API was a lesson in navigating strict bureaucratic logic. To pull data from Instagram, the system couldn't just log in; it required a multi-step authentication flow: verifying the Facebook account, identifying the linked Facebook Page, retrieving the specific Instagram ID connected to that page, and finally generating the authorization token. I successfully built this integration, successfully routing through their strict developer environment testing scenarios.

For Twitter, I built a robust module to handle data extraction and automated posting. It worked perfectly in testing, but production revealed a massive bottleneck.

### The Roadblock: Rate Limiting
The client wanted to automate thousands of posts a day. The problem? Twitter's API strictly limited automated posts to 400 per day per app. To temporarily bridge the gap, I utilized a workaround by registering multiple developer applications to rotate tokens, which successfully multiplied our daily limit to 4,000 tweets. However, as the client's volume demands scaled beyond 10,000 actions a day, it became clear that the official endpoints were completely unscalable for their budget and requirements.

### The Pivot: Engineering a Custom Solution
When official APIs bottleneck business operations, you have to pivot. Instead of continuing to fight rate limits and deprecating endpoints, I developed a custom Flask-based scraping architecture (`flask-socmed-scraping`).

By separating the extraction logic from the official API constraints, this backend service allowed us to bypass the artificial rate limits and scrape the sheer volume of data the client actually needed, ensuring the data pipeline remained intact without triggering bot-detection mechanisms.

### The Takeaway
Working with third-party platforms requires a backup plan. Official APIs are great for standard, low-volume tasks, but when business requirements outgrow those constraints, you need the technical flexibility to build custom solutions that actually deliver.

*You can view the source codes for these modular components on my GitHub:*
* [Twitter API Integration](https://github.com/SokKanaTorajd/twitter-app)
* [Facebook Graph API Service](https://github.com/SokKanaTorajd/FacebookGraphAPI)
* [Flask Custom Scraping Solution](https://github.com/SokKanaTorajd/flask-socmed-scraping)

When Official APIs Aren't Enough for Business Needs

**Tags:** API Integration, Data Architecture, Compliance, Python

### The Business Requirement
To generate accurate social media insights, a client required a reliable data pipeline to extract analytics and trends directly from Instagram. Because third-party scraping can be volatile and prone to bot-detection, the strategic choice was to build a compliant, secure integration using the official Facebook Graph API.

### The Authentication Maze
Unlike simpler APIs where a single key grants access, the Facebook Graph API is heavily fortified and requires navigating a strict bureaucratic logic. You cannot simply log in to pull Instagram data.

To make this work, I engineered a multi-step, automated authentication flow:
* Authenticate the base Facebook User account.
* Query the Graph API to retrieve the ID of the specific Facebook Page managed by that user.
* Cross-reference that Page ID to locate the linked Instagram Business Account ID.
* Finally, generate the authorization token to hit the Instagram endpoints.

Building this sequence required deep dives into the Graph API documentation, mapping the exact parameter requirements to ensure the pipeline would not break upon token expiration.

### Adapting to Deprecation and Compliance
In the middle of the testing phase, the backend team reported that the API was returning error messages about missing parameters. I audited the logs and the official documentation, discovering that Facebook had deprecated the specific API version we were utilizing. This is a common hazard in enterprise integrations. I immediately refactored the module to support the newer version, ensuring minimal downtime for the client.

Furthermore, Facebook's developer ecosystem is highly regulated. Before the app could go live, I had to configure comprehensive testing scenarios, document the development environment, and record the exact application flow so the Facebook review team could verify compliance.

### The Takeaway
Building resilient enterprise integrations requires patience and precision. Working with the Graph API was a masterclass in handling strict security compliance, anticipating endpoint deprecation, and translating convoluted authentication requirements into a seamless, automated data pipeline.

*View the integration module on GitHub: [FacebookGraphAPI](https://github.com/SokKanaTorajd/FacebookGraphAPI)*

Navigating the Maze: Building a Resilient Integration with Facebook Graph API

**Tags:** API Integration, Automation, IT Consulting, Problem Solving

### The Business Requirement
During my 2021 internship, I was pulled into a live project. A core part of modern marketing strategy relies on high-volume, automated social media engagement. I was tasked by a client to build an integration that could automate Twitter postings at scale while also extracting specific timeline data for trend analysis. The initial approach was the most standard one: building a Python-based application that hooks directly into the official Twitter API.

### The Technical Roadblock
The integration itself was straightforward. I built the module to handle authentication and post endpoints seamlessly. However, the friction did not come from the code; it came from the platform's infrastructure.

Once deployed, the application hit a hard wall. The client's business requirement demanded thousands of automated posts a day, but Twitter's official API strictly throttles automated posting to 400 tweets per day, per application. If you push past that, or try to tweet too frequently within a single hour, the API outright rejects the payload.

### Engineering the Workaround
When official constraints bottleneck business operations, you have to get creative. To bridge the gap immediately, I utilized a strategic workaround. By registering multiple developer applications under a student tier, I secured 10 distinct API tokens. I restructured the application to rotate these tokens dynamically, effectively multiplying our daily threshold and allowing us to push 4,000 automated tweets a day without triggering spam filters or facing temporary bans.

### The Strategic Pivot
This workaround successfully stabilized operations for a time, but business demands eventually scaled past 10,000 actions a day. At this point, the challenge shifted from a technical problem to a business consultation.

I initiated direct discussions with the Twitter support team to explore enterprise-tier pricing and service benefits. We discovered that while there were numerous paid tiers for pulling data, the options for pushing high-volume tweets were opaque and cost-prohibitive for the client's specific use case. Ultimately, this led us to evaluate custom scraping and automation architectures outside the official API ecosystem.

### The Takeaway
Technology must align with business realities. This project reinforced that integrating a third-party API is rarely just about reading the documentation; it is about anticipating rate limits, engineering fail-safes, and knowing when to pivot to custom solutions when official channels can no longer support business growth.

*View the integration module on GitHub: [twitter-app](https://github.com/SokKanaTorajd/twitter-app)*

Beyond the Rate Limit: Engineering Workarounds for Twitter API Constraints

**Tags:** Web Scraping, Python, Data Engineering, Automation, Cloud Computing

### The Research Objective
During my college years around early 2020, I assisted a lecturer with a research project that required analyzing a massive volume of academic publications. The target data source was Mendeley, a prominent reference manager and academic social network. Manually searching, copying, and structuring this metadata was out of the question—it would have taken hundreds of hours.

To solve this, I was tasked with building a programmatic solution to automate the extraction process. The result was the `MendeleyCrawler` project: a custom-built script designed to systematically navigate the platform and harvest academic metadata at scale.

### How It Was Built: The Technical Architecture
The core of the project was a Python-based web scraper. When building a crawler for a structured academic database, the primary challenge is not just fetching the data, but navigating the platform's architecture and maintaining clean, usable output.

Here is how the crawler was engineered to function:
*   **Target Acquisition & Pagination:** The script was designed to initiate search queries based on the research parameters. Because academic results span thousands of pages, I implemented pagination logic that allowed the crawler to automatically detect the "Next Page" parameters and loop through the result sets without human intervention.
*   **DOM Parsing & Data Extraction:** Once on a target page, the script parsed the HTML structure. I mapped the specific HTML tags and classes corresponding to the required metadata—such as the publication title, list of authors, abstract, publication year, and journal name.
*   **Data Structuring:** Raw scraped data is messy. The crawler was programmed to clean the text strings (removing rogue HTML tags or whitespace) and compile the extracted metadata into a structured format, writing it directly into a clean dataset file that my lecturer could immediately plug into her analysis tools.

### The Execution: An Unexpected Hardware Lesson
Once the code was written and tested successfully on a small batch of pages, it was time for the production run. This is where a side story occurred that taught me a valuable lesson about infrastructure.

Because the script ran flawlessly during testing, I assumed my personal laptop could easily handle the full extraction workload. I launched the script, watched the first few dozen records successfully write to the database, and confidently left the laptop running overnight to finish the job.

When I woke up the next morning, my laptop was completely powered off.

It turned out that crawling at scale is incredibly resource-intensive. The constant barrage of network requests, continuous memory allocation for parsing document objects, and sustained CPU usage had pushed my hardware past its limits. The laptop had overheated in the middle of the night and triggered an emergency thermal shutdown to prevent physical damage.

### The Pivot: Taking It to the Cloud
The code was perfectly fine, but the execution environment was not. Luckily, I had pushed the entire repository to GitHub.

Operating on a student budget, I sourced a free remote server. I connected to the server, cloned my `MendeleyCrawler` repository directly from GitHub, and launched the script there. Freed from the hardware constraints of a standard laptop, the crawler ran smoothly in the cloud and successfully extracted the massive dataset the research required.

### The Takeaway
This project was a fantastic exercise in automated data extraction, but the overnight hardware failure provided the most lasting professional lesson. Today, when I design enterprise data pipelines, I never look at the code in a vacuum. Efficient code is only half the battle; ensuring you have the right infrastructure to support the workload is what actually brings a project across the finish line.

*You can view the source code and extraction logic on my GitHub: [MendeleyCrawler](https://github.com/SokKanaTorajd/MendeleyCrawler)*

Building a Python Web Crawler for Mendeley Data

**Tags:** Machine Learning, Data Processing, R, Academic Research, C5.0

### The Context: Assisting in IT Research
Back in 2019, during my college years, I had the opportunity to step outside the standard academic curriculum and assist my IT lecturer, Ulfi Saidata Aesyi, with her research. Her work explored the application and efficacy of machine learning algorithms on complex, sensitive datasets.

To benchmark and test these algorithms, her research utilized a publicly available orthopedic dataset. The primary objective was to classify the biomechanical features of the vertebral column to identify abnormalities—such as disk hernia or spondylolisthesis—versus a normal, healthy spine. While she designed the overarching IT research methodology and selected the algorithms to be evaluated, my role was strictly technical: I was tasked with building the programmatic script that would process this public data and execute the machine learning model she was studying.

### The Requirement: Why the C5.0 Algorithm?
In academic IT research, you cannot simply feed data into a library and blindly accept the output; the mechanics of the algorithm itself must be heavily scrutinized. For this specific phase of her research, she chose a Decision Tree architecture—the C5.0 algorithm.

The goal was to utilize an explainable model. A decision tree works by splitting data based on a series of clear, logical questions, ultimately generating a literal flowchart of decisions. This transparency allowed the research to trace exactly how the algorithm handled specific biomechanical features (like pelvic tilt or sacral slope) and at what mathematical thresholds it decided to flag an abnormality. My job was to ensure the code accurately captured and outputted these decision pathways so they could be properly documented and analyzed in her research paper.

### The Technical Implementation with R
I was not inventing a new algorithm from scratch. Instead, my responsibility was to build a robust pipeline to execute the C5.0 model exactly to the research specifications. To do this, I utilized R—a programming language heavily favored in academic and statistical research for its powerful data processing and modeling libraries.

The development process required a highly structured approach to data handling. Even though we were using a public dataset, orthopedic data contains sensitive numeric attributes that require careful treatment. The initial phases of my R script were heavily focused on exploratory data analysis (EDA) and data formatting. I had to ensure that the data frames feeding into the algorithm were processed correctly without losing their structural integrity.

Once the data was clean, I implemented the decision tree model using R's specific C5.0 packages. One of the strongest features of the C5.0 algorithm is its native "winnowing" capability—automatically identifying and discarding attributes that do not significantly contribute to the final classification. My script ultimately outputted both the accuracy metrics of the classification and a visual representation (plot) of the decision tree. This gave the lead researcher exactly what she needed: a transparent, human-readable path of logical deductions to analyze and include in her findings.

### The Takeaway: Engineering for Academic Methodologies
Looking back at this 2019 project, it served as a foundational lesson in technical implementation. It taught me that my job as a developer is not always to write the most complex code possible, but to choose the right tools—like leveraging R for statistical transparency—to faithfully execute a specific methodology.

By focusing on clean data processing and utilizing the right libraries for an explainable model, I was able to provide my lecturer with a reliable programmatic tool that advanced her academic work. It reinforced a principle I carry into my enterprise consulting projects today: code must be specifically tailored to serve the exact analytical needs of the stakeholder.

*You can view the R script and the pipeline for this project on my GitHub: [spine-c50-decision-tree](https://github.com/SokKanaTorajd/spine-c50-decision-tree)*

Translating IT Research into Code: Applying a C5.0 Decision Tree to Spine Classification

**Tags:** Digital Marketing, SEO, Web Analytics, WordPress, Business Strategy

### The Challenge: A Push into the Deep End
Every IT professional has that one origin-story project—the first time they had to build something real that existed outside of a textbook. For me, that project was jasasondir.web.id.

During my early college years, the Head of the Information Systems Study Program gave the students a mandate: form a team of three to four people, pick a business case from a given list, and execute it. The ultimate goal was not just to write code, but to learn the mechanics of digital marketing in the real world. Being young and eager, my team and I dove in headfirst. We chose to build a digital presence for a niche engineering service: soil penetration testing (jasa sondir).

### Building the Foundation
Today, setting up web infrastructure feels like second nature, but back then, it was uncharted territory.

We had to learn the absolute basics of establishing a digital footprint. We pooled our resources to purchase a domain name, figured out how to configure web hosting, and set up a WordPress environment. This was my first practical exposure to the fact that building a digital solution requires managing the underlying infrastructure first. But once the site was live, we hit the real challenge: a website is completely useless if nobody visits it.

### The Strategy: Engineering Visibility
This is where the project shifted from an IT task to a business strategy. The service we were representing was highly specific (construction and soil testing), which meant we needed highly targeted traffic.

Instead of guessing what potential clients wanted, we relied on data. We spent hours conducting keyword research utilizing tools like Google Trends, Ubersuggest, and analyzing raw Google Search behaviors. We needed to map exactly what construction managers and contractors were typing into the search bar when they needed soil testing services.

Armed with this data, we began writing targeted, SEO-optimized articles. We engineered the content structure, the meta-tags, and the keyword density with one singular goal: reaching the first page of Google organically.

### The Introduction to Data Analytics
Getting organic traffic is a long game, so we also explored paid acquisition by setting up Google Ads. But the most formative part of this project for my future career was integrating Google Analytics into the WordPress site.

For the first time, I wasn't just looking at a static webpage; I was looking at user behavior. I could see where our visitors were coming from, which keywords brought them in, how long they stayed on our service pages, and where they dropped off. It was my first taste of Business Intelligence—using raw data to measure performance and dictate the next strategic move.

Eventually, our organic SEO strategy paid off. After months of consistent effort, the site successfully ranked on the first page of Google for our targeted keywords.

### The Takeaway: Tech Must Serve the Business
Looking back at jasasondir.web.id, it might seem like a simple WordPress project compared to the complex enterprise data pipelines and SAP integrations I handle today. However, the core lesson remains exactly the same.

This college project taught me that technology is just a tool; the real objective is business impact. You can build the most beautiful website or write the cleanest backend code, but if it doesn't generate visibility, optimize operations, or bring in revenue, it has failed its primary purpose. That realization—that IT must always bridge the gap to business goals—is the foundation of how I operate as an IT Consultant today.

Project