Building a Python Web Crawler for Mendeley Data

Tags: Web Scraping, Python, Data Engineering, Automation, Cloud Computing

The Research Objective

During my college years around early 2020, I assisted a lecturer with a research project that required analyzing a massive volume of academic publications. The target data source was Mendeley, a prominent reference manager and academic social network. Manually searching, copying, and structuring this metadata was out of the question—it would have taken hundreds of hours.

To solve this, I was tasked with building a programmatic solution to automate the extraction process. The result was the MendeleyCrawler project: a custom-built script designed to systematically navigate the platform and harvest academic metadata at scale.

How It Was Built: The Technical Architecture

The core of the project was a Python-based web scraper. When building a crawler for a structured academic database, the primary challenge is not just fetching the data, but navigating the platform's architecture and maintaining clean, usable output.

Here is how the crawler was engineered to function:

• Target Acquisition & Pagination: The script was designed to initiate search queries based on the research parameters. Because academic results span thousands of pages, I implemented pagination logic that allowed the crawler to automatically detect the "Next Page" parameters and loop through the result sets without human intervention.
• DOM Parsing & Data Extraction: Once on a target page, the script parsed the HTML structure. I mapped the specific HTML tags and classes corresponding to the required metadata—such as the publication title, list of authors, abstract, publication year, and journal name.
• Data Structuring: Raw scraped data is messy. The crawler was programmed to clean the text strings (removing rogue HTML tags or whitespace) and compile the extracted metadata into a structured format, writing it directly into a clean dataset file that my lecturer could immediately plug into her analysis tools.

The Execution: An Unexpected Hardware Lesson

Once the code was written and tested successfully on a small batch of pages, it was time for the production run. This is where a side story occurred that taught me a valuable lesson about infrastructure.

Because the script ran flawlessly during testing, I assumed my personal laptop could easily handle the full extraction workload. I launched the script, watched the first few dozen records successfully write to the database, and confidently left the laptop running overnight to finish the job.

When I woke up the next morning, my laptop was completely powered off.

It turned out that crawling at scale is incredibly resource-intensive. The constant barrage of network requests, continuous memory allocation for parsing document objects, and sustained CPU usage had pushed my hardware past its limits. The laptop had overheated in the middle of the night and triggered an emergency thermal shutdown to prevent physical damage.

The Pivot: Taking It to the Cloud

The code was perfectly fine, but the execution environment was not. Luckily, I had pushed the entire repository to GitHub.

Operating on a student budget, I sourced a free remote server. I connected to the server, cloned my MendeleyCrawler repository directly from GitHub, and launched the script there. Freed from the hardware constraints of a standard laptop, the crawler ran smoothly in the cloud and successfully extracted the massive dataset the research required.

The Takeaway

This project was a fantastic exercise in automated data extraction, but the overnight hardware failure provided the most lasting professional lesson. Today, when I design enterprise data pipelines, I never look at the code in a vacuum. Efficient code is only half the battle; ensuring you have the right infrastructure to support the workload is what actually brings a project across the finish line.

You can view the source code and extraction logic on my GitHub: MendeleyCrawler