Enhancing Gutenberg Book Clustering using Advanced NLP Techniques

Text clustering, an unsupervised ML technique in NLP, groups similar texts based on content. Techniques like hierarchical, k-means, or density-based clustering categorize unstructured data, unveiling insights and patterns in diverse datasets. This exploration was part of the NLP course in my University of Ottawa master's program in 2023.

• Required libraries: scikit-learn, pandas, matplotlib.
• Execute cells in a Jupyter Notebook environment.
• The uploaded code has been executed and tested successfully within the Google Colab environment.

Unsupervised Text Clustering problem

Text clustering involves grouping comparable texts based on content similarity, a crucial unsupervised technique.(chose 5 differnet books for 5 differnet author and genre)

selected_books=['austen-emma.txt','whitman-leaves.txt','milton-paradise.txt', 'melville-moby_dick.txt','chesterton-thursday.txt']

Key Tasks Undertaken

1. Data Preparation, Preprocessing and, Cleaning:

   • Listing all the books in Gutenberg’s library.

     {'austen-emma.txt': 'Jane Austen',
     'austen-persuasion.txt': 'Jane Austen',
     'austen-sense.txt': 'Jane Austen',
     'carroll-alice.txt': 'Lewis Carroll',
     'chesterton-ball.txt': 'G.K. Chesterton',
     'chesterton-brown.txt': 'G. K. Chesterton',
     'chesterton-thursday.txt': 'G. K. Chesterton',
     'edgeworth-parents.txt': 'Maria Edgeworth',
     'melville-moby_dick.txt': 'Dick  Herman Melville',
     'shakespeare-caesar.txt': 'William Shakespeare',
     'shakespeare-hamlet.txt': 'William Shakespeare',
     'whitman-leaves.txt': 'Walt Whitman'}

   • Choose five different books by five different authors belong to the same category (History).

   • Data preparation:

     • Removing stop words.
     • Converting all words to the lower case.
     • Tokenize the text.
     • Lemmatization is the next step that reduces a word to its base form.

   • Data Partitioning: partition each book into 200 documents, each document is a 100 word record.

   

   • Data labeling as follows:

     • austen-emma→ a
     • chesterton-thursday→ b
     • shakespeare-hamlet→ c
     • chesterton-ball→ d
     • carroll-alice→ e

   • Word Cloud Generation: Generates word clouds displaying the most frequent 100 words in books for each author.

2. Feature Engineering:

   • Transformation
     • Bag of Word (BOW):It represents the occurrence of words within a document, it involves two things:
       • A vocabulary of known words.
       • A measure of the presence of known words.
     • Term Frequency - Inverse Document Frequency (TF-IDF):a technique to quantify words in a set of documents. We compute a score for each word to signify its importance in the document and corpus.
     • Latent Dirichlet Allocation (LDA): Perform topic modeling to extract latent topics from the text data. Each document is represented as a mixture of topics.
     • Word Embedding (Word2Vec)

• Encoding

3. Modeling: For each technique of the above, these following models are trained and tested.

   • K-Means
   • Expectation Maximization (EM)
   • Hierarchical clustering (Agglomerative)

4. Model Evaluation

   • using Silhouette Score

     

     

   • using Kappa Score

     [!IMPORTANT] The method for calculating the Kappa Score has been uploaded in the document titled "Kappa Score.pdf".

     

     

5. Champion Model

   • on Silhouette Score

     

   • on Kappa Score

     

6. Error Analysis of Champion Model:

• By reducing the number of clusters from 5 to 3

  • on Silhouette Score

  

  • Champion Model

  

  • on Kappa Score

  

  

  • Champion Model