scrabble tiles in a pile

07 July 2022


Where Text Mining Projects Struggle & How to Overcome Common Challenges

AI has practically limitless applications—especially for organizations struggling with supply chain delays, fraud, and changing customer expectations—but working with textual data is one of the most intriguing.

We encounter textual data every day—our lives are filled with continuous reading, parsing, and understanding text input. But, working with text is also one of the most challenging data science tasks, as texts can have multiple meanings, are prone to interpretation, and are full of contextual and cultural nuances. 

Based on the findings of a recently finished project by the Research Team at RapidMiner, we want to highlight common challenges of working on large NLP (Natural Language Processing) projects and share some best practices. 

A Bit of Background on the TechRad Project 

The goal of the TechRad project was to create an autonomous technology scouting engine that helps engineers and CTOs discover and monitor emerging technology trends within their area of expertise. 

Technology scouting helps businesses stay on the cutting edge and understand what’s going on in their space right now. But it’s also time- and resource-consuming when done internally and expensive when done by professional agencies. 

TechRad’s vision was to collect and analyze a large volume of documents and use that textual data to build an evaluation engine based on state-of-the-art NLP algorithms. This engine would be able to find new technologies, determine their application areas, and evaluate their technology readiness level. The whole project was supervised by legal experts that checked potential repercussions the project might have in practice. 

Overcoming Key NLP Project Challenges 

Text mining projects are typically pretty nuanced (due to the interpretation challenges we mentioned before). Here are the two principal challenges we encountered during the TechRad project and what we did to overcome them. 

Collecting Data 

Good data is the wings all data science projects need to fly. 

When we started the project, we didn’t have an existing dataset, so we first had to collect data. The plan was to have a continuously updated system that could detect emerging trends, as a static document collection would become outdated quite fast. 

We used two methods to execute this: 

For the training and evaluation phase, we used research papers from the repository of over 1.7 million documents and Wikipedia articles about selected technologies. 

Asking the Right Questions and Defining the Correct Scope 

This is one of the greatest challenges in any data science project, but the ambiguous goal of interpreting texts and coming to meaningful conclusions makes it even more difficult. 

To start with, the simple question of “What actually is a technology?,” led to many long discussions within our group. Even without considering any edge cases, this is way harder to define than, for example, to determine if there’s a cat in a picture or not. 

In the end, the definition we settled for was “A technology is a distinguishable item, such as an algorithm, a commercial product, or software library or framework.”  The next step was to create a training dataset so we could train a machine learning algorithm to follow our definition and recognize new technologies based on the textual context. 

For training, we used the open-source tool doccano, which makes it extremely easy to label text snippets. We set up a doccano instance, fed it with several thousands of research papers publicly available on, and in a distributed effort labeled over two thousand text snippets in a few days. The model trained with this data was able to successfully identify new technologies, like specific algorithms or production procedures. 

Marked technologies within the doccano app

With the labeled data, we could complete our pipeline and build our demonstration of the technology scouting engine. It included: 

Wrapping Up 

When working through an NLP project, having properly labeled data and a clear understanding of the problem you’re trying to solve are the keys to success. 

With TechRad in particular, we were interested to see how powerful modern, deep learning models are for extracting knowledge out of large text collections. We’ll take our findings from experience and distill it into our upcoming works on integrating these specialized deep learning models into our product suite. 

For the technically inclined, we also plan to provide you with more deep dives and overviews on the workings of transformer based deep neural networks, such as BERT and its current successors. Stay tuned! 

Want to learn more about what’s new at RapidMiner? Check out the blog post from our founder, Ingo, on our next gen platform launch. 

Related Resources