Advanced NLP Applications: Beyond Basic Text Processing

Understanding Sentiment Analysis

Sentiment analysis can be approached at different levels of granularity:

There are several approaches to implementing sentiment analysis:

• Rule-based approaches: Using pre-defined rules to identify sentiment.
• Lexicon-based methods: Using dictionaries of words with associated sentiment scores.
• Machine learning approaches: Training classifiers on labeled data.
• Deep learning approaches: Using neural networks, particularly transformer-based models.

Implementation with Python

Let's implement sentiment analysis using different approaches, from simple to advanced:

1. Lexicon-Based Approach with VADER

VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool specifically attuned to sentiments expressed in social media.

import nltk
from nltk.sentiment import SentimentIntensityAnalyzer

# Download VADER lexicon if not already downloaded
nltk.download('vader_lexicon')

# Initialize the sentiment analyzer
sia = SentimentIntensityAnalyzer()

# Example texts
texts = [
    "I love this product! It's amazing and works perfectly.",
    "This is okay, but not great. It works as expected.",
    "Terrible experience. The product broke after one day and customer service was unhelpful."
]

# Analyze sentiment for each text
for text in texts:
    sentiment_scores = sia.polarity_scores(text)
    print(f"Text: {text}")
    print(f"Sentiment Scores: {sentiment_scores}")
    
    # Determine overall sentiment
    compound_score = sentiment_scores['compound']
    if compound_score >= 0.05:
        sentiment = "Positive"
    elif compound_score <= -0.05:
        sentiment = "Negative"
    else:
        sentiment = "Neutral"
    
    print(f"Overall Sentiment: {sentiment}\n")

VADER provides scores for positive, negative, neutral, and a compound score that represents the overall sentiment. The compound score is normalized between -1 (most negative) and 1 (most positive).

2. Machine Learning Approach with Scikit-learn

For more flexibility, we can train a machine learning model on labeled data:

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

# Sample dataset (in practice, you would load a real dataset)
data = {
    'text': [
        "I absolutely love this product!",
        "This works as expected, nothing special.",
        "Terrible product, complete waste of money.",
        "Great value for the price, highly recommend.",
        "It's okay but not worth the cost.",
        "Disappointed with the quality, would not buy again.",
        "Amazing customer service and fast shipping!",
        "Average performance, nothing to write home about.",
        "Worst purchase I've ever made."
    ],
    'sentiment': ['positive', 'neutral', 'negative', 'positive', 'neutral', 
                 'negative', 'positive', 'neutral', 'negative']
}

# Create DataFrame
df = pd.DataFrame(data)

# Split data into features and target
X = df['text']
y = df['sentiment']

# Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create TF-IDF features
vectorizer = TfidfVectorizer(max_features=1000, stop_words='english')
X_train_tfidf = vectorizer.fit_transform(X_train)
X_test_tfidf = vectorizer.transform(X_test)

# Train a logistic regression model
model = LogisticRegression(max_iter=1000)
model.fit(X_train_tfidf, y_train)

# Make predictions
y_pred = model.predict(X_test_tfidf)

# Evaluate the model
print(classification_report(y_test, y_pred))

# Function to predict sentiment for new text
def predict_sentiment(text):
    # Transform text using the same vectorizer
    text_tfidf = vectorizer.transform([text])
    # Predict sentiment
    prediction = model.predict(text_tfidf)[0]
    # Get probability scores
    probabilities = model.predict_proba(text_tfidf)[0]
    return prediction, probabilities

# Test with a new example
new_text = "I'm really impressed with the features and the build quality."
sentiment, probabilities = predict_sentiment(new_text)
print(f"Text: {new_text}")
print(f"Predicted Sentiment: {sentiment}")
print(f"Confidence: {max(probabilities):.2f}")

This approach allows you to train a model specific to your domain and classification needs. In a real-world scenario, you would use a larger labeled dataset relevant to your domain.

3. Transformer-Based Approach with Hugging Face

For state-of-the-art performance, we can use pre-trained transformer models:

from transformers import pipeline, AutoModelForSequenceClassification, AutoTokenizer

# Option 1: Using the pipeline API (simplest approach)
sentiment_analyzer = pipeline("sentiment-analysis")

# Analyze text
texts = [
    "I love this product! It's amazing and works perfectly.",
    "This is okay, but not great. It works as expected.",
    "Terrible experience. The product broke after one day and customer service was unhelpful."
]

for text in texts:
    result = sentiment_analyzer(text)
    print(f"Text: {text}")
    print(f"Sentiment: {result[0]['label']}, Score: {result[0]['score']:.4f}\n")

# Option 2: More control with explicit model loading
# Load model and tokenizer
model_name = "distilbert-base-uncased-finetuned-sst-2-english"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Function for sentiment analysis with custom threshold
def analyze_sentiment(text, threshold=0.9):
    # Tokenize and convert to model inputs
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
    
    # Get model outputs
    outputs = model(**inputs)
    
    # Get prediction
    import torch
    import torch.nn.functional as F
    
    # Apply softmax to get probabilities
    probs = F.softmax(outputs.logits, dim=-1)
    
    # Get the predicted class and its probability
    predicted_class = torch.argmax(probs, dim=-1).item()
    confidence = probs[0][predicted_class].item()
    
    # Map to label
    label = model.config.id2label[predicted_class]
    
    # Apply confidence threshold
    if confidence < threshold:
        return "Mixed/Uncertain", confidence
    
    return label, confidence

# Test with examples
for text in texts:
    sentiment, confidence = analyze_sentiment(text)
    print(f"Text: {text}")
    print(f"Sentiment: {sentiment}, Confidence: {confidence:.4f}\n")

Transformer-based models like BERT and DistilBERT provide excellent performance for sentiment analysis, especially when fine-tuned on domain-specific data.

Fine-tuning for Your Domain

Pre-trained models work well for general sentiment analysis, but for domain-specific applications, fine-tuning can significantly improve performance.

When to Fine-tune

• When dealing with specialized vocabulary or jargon
• When sentiment expressions are domain-specific
• When you need more granular sentiment categories
• When you have sufficient labeled data for your domain

Fine-tuning a Transformer Model

from transformers import AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments
from datasets import Dataset
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

# Sample dataset (in practice, use a larger domain-specific dataset)
data = {
    'text': [
        "The API documentation is comprehensive and easy to follow.",
        "This framework has too many dependencies and breaks often.",
        "The code runs efficiently with minimal resource usage.",
        "Installation process is straightforward and well-explained.",
        "Frequent updates keep breaking backward compatibility.",
        "The community support for this library is outstanding.",
        "Poor error messages make debugging a nightmare.",
        "Clean architecture and well-organized codebase.",
        "The learning curve is steep but worth it."
    ],
    'label': [1, 0, 1, 1, 0, 1, 0, 1, 1]  # 1 for positive, 0 for negative
}

# Create DataFrame
df = pd.DataFrame(data)

# Split into train and validation sets
train_df, val_df = train_test_split(df, test_size=0.2, random_state=42)

# Convert to Hugging Face datasets
train_dataset = Dataset.from_pandas(train_df)
val_dataset = Dataset.from_pandas(val_df)

# Load pre-trained model and tokenizer
model_name = "distilbert-base-uncased"
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Tokenization function
def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

# Tokenize datasets
train_dataset = train_dataset.map(tokenize_function, batched=True)
val_dataset = val_dataset.map(tokenize_function, batched=True)

# Define training arguments
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir="./logs",
    logging_steps=10,
    evaluation_strategy="epoch"
)

# Define compute_metrics function
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    accuracy = np.mean(predictions == labels)
    return {"accuracy": accuracy}

# Initialize Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    compute_metrics=compute_metrics,
)

# Fine-tune the model
trainer.train()

# Save the fine-tuned model
model_path = "./fine-tuned-sentiment-model"
model.save_pretrained(model_path)
tokenizer.save_pretrained(model_path)

# Test the fine-tuned model
test_texts = [
    "This library has excellent documentation and examples.",
    "The package has too many bugs and isn't maintained."
]

# Load fine-tuned model
fine_tuned_model = AutoModelForSequenceClassification.from_pretrained(model_path)
fine_tuned_tokenizer = AutoTokenizer.from_pretrained(model_path)

# Create a pipeline with the fine-tuned model
from transformers import pipeline
sentiment_pipeline = pipeline("sentiment-analysis", model=fine_tuned_model, tokenizer=fine_tuned_tokenizer)

# Test the model
for text in test_texts:
    result = sentiment_pipeline(text)
    print(f"Text: {text}")
    print(f"Sentiment: {result[0]['label']}, Score: {result[0]['score']:.4f}\n")

Practical Tips for Sentiment Analysis

NER Fundamentals

Named Entity Recognition involves two main tasks:

1. Detection: Identifying which parts of the text are named entities
2. Classification: Determining the category of each identified entity

Common entity types include:

Different NER systems may use different entity types or more fine-grained categories depending on the application domain.

Building a NER System

Let's implement NER using different approaches, from simple to advanced:

1. Rule-Based Approach with Regular Expressions

For simple cases or specific patterns, regular expressions can be effective:

import re

# Sample text
text = """
Apple Inc. is planning to open a new office in New York City by January 2024. 
The company's CEO, Tim Cook, announced this during a press conference on 
May 15th, 2023. The project will cost approximately $50 million and create 
about 200 new jobs. Apple's stock rose 3% following the announcement.
"""

# Define regex patterns for different entity types
patterns = {
    'DATE': r'\b(?:January|February|March|April|May|June|July|August|September|October|November|December)\s+\d{1,2}(?:st|nd|rd|th)?,\s+\d{4}\b|\b\d{1,2}/\d{1,2}/\d{2,4}\b',
    'PERSON': r'\b[A-Z][a-z]+ [A-Z][a-z]+\b',
    'ORG': r'\b[A-Z][a-z]+ (?:Inc\.|Corp\.|LLC|Ltd\.)\b',
    'MONEY': r'\$\d+(?:\.\d+)? (?:million|billion|trillion)\b|\$\d+(?:,\d+)*(?:\.\d+)?\b',
    'PERCENT': r'\b\d+(?:\.\d+)?%\b'
}

# Find entities
for entity_type, pattern in patterns.items():
    print(f"\n{entity_type} entities:")
    for match in re.finditer(pattern, text):
        print(f"  - {match.group(0)}")

# Visualize entities in text
def highlight_entities(text, patterns):
    # Create a copy of the text for highlighting
    highlighted = text
    
    # Dictionary to store entity positions
    entity_positions = []
    
    # Find all entities and their positions
    for entity_type, pattern in patterns.items():
        for match in re.finditer(pattern, text):
            start, end = match.span()
            entity_positions.append((start, end, entity_type, match.group(0)))
    
    # Sort by start position in reverse order (to avoid messing up indices when inserting tags)
    entity_positions.sort(key=lambda x: x[0], reverse=True)
    
    # Insert HTML-like tags for highlighting
    for start, end, entity_type, entity_text in entity_positions:
        highlighted = highlighted[:end] + f"[/{entity_type}]" + highlighted[end:]
        highlighted = highlighted[:start] + f"[{entity_type}]" + highlighted[start:]
    
    return highlighted

# Print highlighted text
print("\nHighlighted text:")
print(highlight_entities(text, patterns))

While simple to implement, rule-based approaches have limitations in handling ambiguity and require manual pattern creation for each entity type.

2. Using spaCy for NER

spaCy provides pre-trained models for NER that work well out of the box:

import spacy
from spacy import displacy

# Load English model
nlp = spacy.load("en_core_web_md")

# Sample text
text = """
Apple Inc. is planning to open a new office in New York City by January 2024. 
The company's CEO, Tim Cook, announced this during a press conference on 
May 15th, 2023. The project will cost approximately $50 million and create 
about 200 new jobs. Apple's stock rose 3% following the announcement.
"""

# Process the text
doc = nlp(text)

# Print entities
print("Named Entities:")
for ent in doc.ents:
    print(f"  - {ent.text} ({ent.label_}: {spacy.explain(ent.label_)})")

# Function to visualize entities in text
def visualize_entities(text):
    doc = nlp(text)
    
    # Display entities in HTML format
    html = displacy.render(doc, style="ent", jupyter=False)
    
    # For non-Jupyter environments, you can save to a file
    with open("ner_visualization.html", "w", encoding="utf-8") as f:
        f.write(html)
    
    print("Visualization saved to ner_visualization.html")
    
    # Return entity counts for analysis
    entity_counts = {}
    for ent in doc.ents:
        if ent.label_ in entity_counts:
            entity_counts[ent.label_] += 1
        else:
            entity_counts[ent.label_] = 1
    
    return entity_counts

# Visualize entities
entity_counts = visualize_entities(text)
print("\nEntity counts:", entity_counts)

# Process a larger text for analysis
larger_text = """
Microsoft Corporation announced a partnership with OpenAI in San Francisco last week.
The deal, worth $1 billion, will focus on developing artificial intelligence technologies.
Satya Nadella, CEO of Microsoft, met with Sam Altman on Thursday to finalize the agreement.
The collaboration will start on October 1st, 2023, and is expected to last for at least 5 years.
Both companies' stocks performed well after the announcement, with Microsoft seeing a 2.3% increase.
"""

# Process and analyze
doc_large = nlp(larger_text)

# Analyze entity relationships
print("\nEntity Relationships:")
for sent in doc_large.sents:
    sent_doc = nlp(sent.text)
    entities = [(e.text, e.label_) for e in sent_doc.ents]
    if len(entities) > 1:
        print(f"Sentence: {sent}")
        print(f"Entities: {entities}")
        print("---")

spaCy provides excellent out-of-the-box performance for NER and includes visualization tools to help understand the results.

3. Transformer-Based Approach with Hugging Face

For state-of-the-art performance, we can use pre-trained transformer models:

from transformers import pipeline, AutoTokenizer, AutoModelForTokenClassification
import pandas as pd

# Option 1: Using the pipeline API (simplest approach)
ner_pipeline = pipeline("ner", aggregation_strategy="simple")

# Sample text
text = """
Apple Inc. is planning to open a new office in New York City by January 2024. 
The company's CEO, Tim Cook, announced this during a press conference on 
May 15th, 2023. The project will cost approximately $50 million and create 
about 200 new jobs. Apple's stock rose 3% following the announcement.
"""

# Get entities
entities = ner_pipeline(text)

# Display results
print("Named Entities:")
for entity in entities:
    print(f"  - {entity['word']} ({entity['entity_group']}, score: {entity['score']:.4f})")

# Option 2: More control with explicit model loading
# Load model and tokenizer
model_name = "dslim/bert-base-NER"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

# Function for NER with custom threshold
def extract_entities(text, threshold=0.9):
    # Tokenize and convert to model inputs
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
    
    # Get model outputs
    import torch
    outputs = model(**inputs)
    
    # Get predictions
    predictions = torch.argmax(outputs.logits, dim=2)
    
    # Get confidence scores
    scores = torch.nn.functional.softmax(outputs.logits, dim=2)
    
    # Extract entities
    tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
    
    # Get label map
    id2label = model.config.id2label
    
    # Extract entities
    entities = []
    current_entity = {"text": "", "type": "", "score": 0, "tokens": []}
    
    for i, (token, pred, score) in enumerate(zip(tokens, predictions[0], scores[0])):
        label = id2label[pred.item()]
        confidence = score[pred.item()].item()
        
        # Skip special tokens
        if token in [tokenizer.cls_token, tokenizer.sep_token, tokenizer.pad_token]:
            continue
            
        # Handle subword tokens
        if token.startswith("##"):
            token = token[2:]  # Remove ## prefix
            if current_entity["text"]:
                current_entity["text"] += token
                current_entity["tokens"].append({"token": token, "label": label, "score": confidence})
                # Update average score
                current_entity["score"] = sum(t["score"] for t in current_entity["tokens"]) / len(current_entity["tokens"])
            continue
            
        # If we're in an entity and the label changes or is O, finish the current entity
        if current_entity["text"] and (label == "O" or not label.endswith(current_entity["type"])):
            if current_entity["score"] >= threshold:
                entities.append({
                    "text": current_entity["text"],
                    "type": current_entity["type"],
                    "score": current_entity["score"]
                })
            current_entity = {"text": "", "type": "", "score": 0, "tokens": []}
            
        # If we have a new entity, start tracking it
        if label != "O":
            # Extract entity type (remove B- or I- prefix)
            entity_type = label[2:] if label.startswith("B-") or label.startswith("I-") else label
            
            if not current_entity["text"]:
                current_entity = {
                    "text": token,
                    "type": entity_type,
                    "score": confidence,
                    "tokens": [{"token": token, "label": label, "score": confidence}]
                }
            else:
                current_entity["text"] += " " + token
                current_entity["tokens"].append({"token": token, "label": label, "score": confidence})
                # Update average score
                current_entity["score"] = sum(t["score"] for t in current_entity["tokens"]) / len(current_entity["tokens"])
    
    # Don't forget the last entity
    if current_entity["text"] and current_entity["score"] >= threshold:
        entities.append({
            "text": current_entity["text"],
            "type": current_entity["type"],
            "score": current_entity["score"]
        })
    
    return entities

# Extract entities with custom function
custom_entities = extract_entities(text)

# Display results
print("\nCustom Entity Extraction:")
for entity in custom_entities:
    print(f"  - {entity['text']} ({entity['type']}, score: {entity['score']:.4f})")

# Convert to DataFrame for analysis
df = pd.DataFrame(custom_entities)
print("\nEntity Summary:")
print(df.groupby("type").agg({"text": "count"}).reset_index().rename(columns={"text": "count"}))

Transformer-based models provide state-of-the-art performance for NER, especially for complex texts and ambiguous entities.

Custom Entity Extraction

For domain-specific applications, you may need to extract custom entity types not covered by pre-trained models. There are several approaches to this:

1. Training a Custom spaCy NER Model

import spacy
from spacy.tokens import DocBin
from spacy.training import Example
import random

# Sample training data with custom entity types
# Format: (text, {"entities": [(start, end, label)]})
TRAIN_DATA = [
    ("Tesla released a new electric vehicle model called Cybertruck", 
     {"entities": [(0, 5, "COMPANY"), (49, 59, "PRODUCT")]}),
    
    ("Apple's latest iPhone 13 Pro has impressive camera capabilities", 
     {"entities": [(0, 5, "COMPANY"), (14, 26, "PRODUCT")]}),
    
    ("Microsoft Azure provides cloud computing services for enterprises", 
     {"entities": [(0, 9, "COMPANY"), (10, 15, "PRODUCT")]}),
    
    ("Google's DeepMind developed AlphaFold for protein structure prediction", 
     {"entities": [(0, 6, "COMPANY"), (8, 16, "DIVISION"), (27, 36, "PRODUCT")]}),
    
    ("Amazon Web Services (AWS) is a leading cloud platform", 
     {"entities": [(0, 6, "COMPANY"), (7, 19, "DIVISION"), (21, 24, "ABBREVIATION")]}),
]

# Function to prepare training data
def prepare_training_data(train_data, model=None):
    # Load or create a blank model
    if model is None:
        nlp = spacy.blank("en")
    else:
        nlp = spacy.load(model)
    
    # Create a DocBin to store training documents
    doc_bin = DocBin()
    
    # Add entity labels to the NER pipe
    if "ner" not in nlp.pipe_names:
        ner = nlp.add_pipe("ner")
    else:
        ner = nlp.get_pipe("ner")
    
    # Add entity labels
    for _, annotations in train_data:
        for _, _, label in annotations["entities"]:
            ner.add_label(label)
    
    # Convert training data to spaCy format
    for text, annotations in train_data:
        doc = nlp.make_doc(text)
        example = Example.from_dict(doc, annotations)
        doc_bin.add(example.reference)
    
    return nlp, doc_bin

# Prepare training data
nlp, doc_bin = prepare_training_data(TRAIN_DATA)

# Save the DocBin to disk
doc_bin.to_disk("./train.spacy")

# Training configuration (in a real scenario, save this to config.cfg)
# spacy train config.cfg --output ./model --paths.train ./train.spacy --paths.dev ./train.spacy

# For demonstration, we'll simulate the training process
def train_model(nlp, train_data, iterations=30):
    # Create training examples
    train_examples = []
    for text, annotations in train_data:
        doc = nlp.make_doc(text)
        example = Example.from_dict(doc, annotations)
        train_examples.append(example)
    
    # Initialize the optimizer
    optimizer = nlp.begin_training()
    
    # Train the model
    for i in range(iterations):
        # Shuffle examples
        random.shuffle(train_examples)
        
        # Update the model
        losses = {}
        for example in train_examples:
            nlp.update([example], drop=0.5, losses=losses)
        
        # Print progress
        if (i + 1) % 10 == 0:
            print(f"Iteration {i + 1}, Losses: {losses}")
    
    return nlp

# Train the model
trained_nlp = train_model(nlp, TRAIN_DATA)

# Test the model
test_text = "NVIDIA released the RTX 4090 GPU for gaming enthusiasts"
doc = trained_nlp(test_text)

print("\nCustom Entity Recognition:")
for ent in doc.ents:
    print(f"  - {ent.text} ({ent.label_})")

# In a real scenario, save the model
# trained_nlp.to_disk("./custom_ner_model")

This approach allows you to train a custom NER model for your specific domain. In a real-world scenario, you would need more training data and proper validation.

2. Pattern-Based Entity Extraction with spaCy's EntityRuler

import spacy
from spacy.pipeline import EntityRuler

# Load model
nlp = spacy.load("en_core_web_md")

# Create an entity ruler
ruler = EntityRuler(nlp)

# Define patterns for custom entities
patterns = [
    {"label": "PROGRAMMING_LANGUAGE", "pattern": "Python"},
    {"label": "PROGRAMMING_LANGUAGE", "pattern": "JavaScript"},
    {"label": "PROGRAMMING_LANGUAGE", "pattern": "Java"},
    {"label": "PROGRAMMING_LANGUAGE", "pattern": "C++"},
    {"label": "PROGRAMMING_LANGUAGE", "pattern": "TypeScript"},
    
    {"label": "FRAMEWORK", "pattern": "React"},
    {"label": "FRAMEWORK", "pattern": "Angular"},
    {"label": "FRAMEWORK", "pattern": "Vue.js"},
    {"label": "FRAMEWORK", "pattern": "Django"},
    {"label": "FRAMEWORK", "pattern": "Flask"},
    
    {"label": "DATABASE", "pattern": "MongoDB"},
    {"label": "DATABASE", "pattern": "PostgreSQL"},
    {"label": "DATABASE", "pattern": "MySQL"},
    {"label": "DATABASE", "pattern": "SQLite"},
    {"label": "DATABASE", "pattern": "Redis"},
]

# Add patterns to the ruler
ruler.add_patterns(patterns)

# Add the ruler to the pipeline
nlp.add_pipe("entity_ruler", before="ner")

# Test text
text = """
The project uses Python for backend development with Django framework.
The frontend is built with JavaScript and React, while data is stored in PostgreSQL.
Some components are written in TypeScript for better type safety.
"""

# Process the text
doc = nlp(text)

# Print entities
print("Custom Technical Entities:")
for ent in doc.ents:
    print(f"  - {ent.text} ({ent.label_})")

# More complex patterns with token attributes
token_patterns = [
    # Pattern for version numbers
    {"label": "VERSION", "pattern": [{"SHAPE": "d.d"}, {"LOWER": "version", "OP": "?"}]},
    {"label": "VERSION", "pattern": [{"SHAPE": "d.d.d"}, {"LOWER": "version", "OP": "?"}]},
    
    # Pattern for file extensions
    {"label": "FILE_TYPE", "pattern": [{"SHAPE": "*.xxx"}]},
    
    # Pattern for URLs
    {"label": "URL", "pattern": [{"LIKE_URL": True}]},
    
    # Pattern for email addresses
    {"label": "EMAIL", "pattern": [{"LIKE_EMAIL": True}]}
]

# Create a new ruler for token patterns
token_ruler = EntityRuler(nlp)
token_ruler.add_patterns(token_patterns)

# Add to pipeline (after the previous ruler)
nlp.add_pipe("entity_ruler", name="token_ruler")

# Test with complex patterns
complex_text = """
The application was updated to version 2.3.1 yesterday.
You can download the file example.pdf from our website https://example.com.
For support, contact support@example.com.
"""

# Process the text
complex_doc = nlp(complex_text)

# Print entities
print("\nComplex Pattern Entities:")
for ent in complex_doc.ents:
    print(f"  - {ent.text} ({ent.label_})")

The EntityRuler approach is useful for rule-based entity extraction when you have well-defined patterns for your custom entities.

Practical Applications of Custom NER

Extractive vs. Abstractive Summarization

There are two main approaches to text summarization:

Let's explore how to implement both approaches using various techniques and libraries.

Building a Summarizer

Let's implement text summarization using different approaches, from simple to advanced:

1. Extractive Summarization with TextRank

TextRank is a graph-based ranking algorithm inspired by Google's PageRank. It can be used for extractive summarization by ranking sentences based on their importance:

import numpy as np
import networkx as nx
import nltk
from nltk.tokenize import sent_tokenize
from nltk.corpus import stopwords
from sklearn.metrics.pairwise import cosine_similarity
import re

# Download required NLTK resources
nltk.download('punkt')
nltk.download('stopwords')

def preprocess_text(text):
    # Clean the text
    text = re.sub(r'\s+', ' ', text)
    # Convert to lowercase
    text = text.lower()
    # Remove special characters
    text = re.sub(r'[^\w\s]', '', text)
    return text

def sentence_similarity(sent1, sent2, stopwords=None):
    if stopwords is None:
        stopwords = []
    
    # Convert sentences to word vectors
    sent1 = [w.lower() for w in sent1 if w.lower() not in stopwords]
    sent2 = [w.lower() for w in sent2 if w.lower() not in stopwords]
    
    # Create a set with all unique words
    all_words = list(set(sent1 + sent2))
    
    # Create word vectors
    vector1 = [0] * len(all_words)
    vector2 = [0] * len(all_words)
    
    # Build the vectors
    for w in sent1:
        vector1[all_words.index(w)] += 1
    
    for w in sent2:
        vector2[all_words.index(w)] += 1
    
    # Calculate cosine similarity
    return cosine_similarity([vector1], [vector2])[0][0]

def build_similarity_matrix(sentences, stop_words):
    # Create an empty similarity matrix
    similarity_matrix = np.zeros((len(sentences), len(sentences)))
    
    for i in range(len(sentences)):
        for j in range(len(sentences)):
            if i != j:
                similarity_matrix[i][j] = sentence_similarity(
                    sentences[i], sentences[j], stop_words)
    
    return similarity_matrix

def textrank_summarize(text, num_sentences=5):
    # Tokenize the text into sentences
    sentences = sent_tokenize(text)
    
    # Preprocess the sentences
    clean_sentences = [preprocess_text(sentence) for sentence in sentences]
    
    # Tokenize the sentences into words
    sentence_tokens = [nltk.word_tokenize(sentence) for sentence in clean_sentences]
    
    # Get stop words
    stop_words = set(stopwords.words('english'))
    
    # Build similarity matrix
    similarity_matrix = build_similarity_matrix(sentence_tokens, stop_words)
    
    # Create a graph from the similarity matrix
    similarity_graph = nx.from_numpy_array(similarity_matrix)
    
    # Apply PageRank algorithm
    scores = nx.pagerank(similarity_graph)
    
    # Sort sentences by score
    ranked_sentences = sorted(((scores[i], i, s) for i, s in enumerate(sentences)), reverse=True)
    
    # Get the top n sentences
    top_sentences = sorted(ranked_sentences[:num_sentences], key=lambda x: x[1])
    
    # Return the summary
    return " ".join([s for _, _, s in top_sentences])

# Example usage
text = """
Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and humans through natural language. The ultimate objective of NLP is to read, decipher, understand, and make sense of human language in a valuable way.

NLP combines computational linguistics—rule-based modeling of human language—with statistical, machine learning, and deep learning models. These technologies enable computers to process human language in the form of text or voice data and to 'understand' its full meaning, complete with the speaker or writer's intent and sentiment.

NLP drives computer programs that translate text from one language to another, respond to spoken commands, and summarize large volumes of text rapidly—even in real time. There's a good chance you've interacted with NLP in the form of voice-operated GPS systems, digital assistants, speech-to-text dictation software, customer service chatbots, and other consumer conveniences.

But NLP also plays a growing role in enterprise solutions that help streamline business operations, increase employee productivity, and simplify mission-critical business processes. NLP can be used to analyze customer feedback, support tickets, online reviews, social media comments, and more to extract insights about customer sentiment, identify emerging issues, and inform product development.
"""

summary = textrank_summarize(text, 2)
print("TextRank Summary:")
print(summary)

TextRank is effective for extractive summarization and doesn't require any training data, making it a good starting point for summarization tasks.

2. Extractive Summarization with NLTK and Frequency Analysis

Another approach to extractive summarization is to use word frequency analysis to identify important sentences:

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.probability import FreqDist
from heapq import nlargest
from string import punctuation

nltk.download('punkt')
nltk.download('stopwords')

def frequency_summarize(text, num_sentences=5):
    # Tokenize the text into sentences and words
    sentences = sent_tokenize(text)
    words = word_tokenize(text.lower())
    
    # Remove stopwords and punctuation
    stop_words = set(stopwords.words('english') + list(punctuation))
    filtered_words = [word for word in words if word not in stop_words]
    
    # Calculate word frequencies
    word_frequencies = FreqDist(filtered_words)
    
    # Normalize frequencies
    max_frequency = max(word_frequencies.values())
    for word in word_frequencies:
        word_frequencies[word] = word_frequencies[word] / max_frequency
    
    # Calculate sentence scores based on word frequencies
    sentence_scores = {}
    for i, sentence in enumerate(sentences):
        for word in word_tokenize(sentence.lower()):
            if word in word_frequencies:
                if i in sentence_scores:
                    sentence_scores[i] += word_frequencies[word]
                else:
                    sentence_scores[i] = word_frequencies[word]
    
    # Get the top n sentences
    summary_sentences = nlargest(num_sentences, sentence_scores, key=sentence_scores.get)
    
    # Sort sentences by their original order
    summary_sentences.sort()
    
    # Return the summary
    return " ".join([sentences[i] for i in summary_sentences])

# Example usage
summary = frequency_summarize(text, 2)
print("\nFrequency-based Summary:")
print(summary)

This approach is simple yet effective for many summarization tasks, especially for news articles and informational content.

3. Abstractive Summarization with Transformers

For state-of-the-art abstractive summarization, we can use pre-trained transformer models:

from transformers import pipeline, AutoModelForSeq2SeqLM, AutoTokenizer

# Option 1: Using the pipeline API (simplest approach)
summarizer = pipeline("summarization")

# Example text (same as above)
# Note: Most models have a maximum input length, so for longer texts, you may need to chunk the input

# Summarize text
summary = summarizer(text, max_length=150, min_length=50, do_sample=False)
print("\nTransformer Pipeline Summary:")
print(summary[0]['summary_text'])

# Option 2: More control with explicit model loading
# Load model and tokenizer
model_name = "facebook/bart-large-cnn"  # A model fine-tuned for summarization
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Function for summarization with custom parameters
def generate_summary(text, max_length=150, min_length=50):
    # Tokenize and convert to model inputs
    inputs = tokenizer(text, return_tensors="pt", max_length=1024, truncation=True)
    
    # Generate summary
    summary_ids = model.generate(
        inputs["input_ids"],
        max_length=max_length,
        min_length=min_length,
        num_beams=4,
        length_penalty=2.0,
        early_stopping=True,
        no_repeat_ngram_size=3
    )
    
    # Decode the summary
    summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
    
    return summary

# Generate summary
custom_summary = generate_summary(text)
print("\nCustom Transformer Summary:")
print(custom_summary)

# Example with a different model (T5)
t5_model_name = "t5-small"
t5_tokenizer = AutoTokenizer.from_pretrained(t5_model_name)
t5_model = AutoModelForSeq2SeqLM.from_pretrained(t5_model_name)

def t5_summarize(text):
    # T5 requires a specific format for summarization
    input_text = "summarize: " + text
    
    # Tokenize and convert to model inputs
    inputs = t5_tokenizer(input_text, return_tensors="pt", max_length=512, truncation=True)
    
    # Generate summary
    summary_ids = t5_model.generate(
        inputs["input_ids"],
        max_length=150,
        min_length=40,
        num_beams=4,
        no_repeat_ngram_size=2,
        early_stopping=True
    )
    
    # Decode the summary
    summary = t5_tokenizer.decode(summary_ids[0], skip_special_tokens=True)
    
    return summary

# Generate T5 summary
t5_summary = t5_summarize(text)
print("\nT5 Summary:")
print(t5_summary)

Transformer-based models like BART and T5 provide state-of-the-art performance for abstractive summarization, generating fluent and coherent summaries that can paraphrase the original text.

4. Hybrid Approach: Combining Extractive and Abstractive Methods

For very long documents, a hybrid approach can be effective:

def hybrid_summarize(text, extractive_ratio=0.5, final_length=150):
    """
    A hybrid summarization approach that first extracts important sentences,
    then applies abstractive summarization to the extracted content.
    
    Args:
        text: The input text to summarize
        extractive_ratio: The proportion of the original text to keep in the extractive step
        final_length: The target length of the final summary
    
    Returns:
        A summary of the text
    """
    # Step 1: Extractive summarization to reduce text length
    sentences = sent_tokenize(text)
    num_sentences = max(2, int(len(sentences) * extractive_ratio))
    
    # Use TextRank for extractive summarization
    extractive_summary = textrank_summarize(text, num_sentences)
    
    # Step 2: Abstractive summarization on the extracted content
    # Use the transformer pipeline for abstractive summarization
    abstractive_summary = summarizer(
        extractive_summary, 
        max_length=final_length, 
        min_length=min(50, final_length-50), 
        do_sample=False
    )
    
    return abstractive_summary[0]['summary_text']

# Example with a longer text
long_text = text * 3  # Just repeating our example text to make it longer

hybrid_summary = hybrid_summarize(long_text)
print("\nHybrid Summary:")
print(hybrid_summary)

This hybrid approach is particularly useful for summarizing very long documents that exceed the context window of transformer models.

Evaluating Summaries

Evaluating the quality of summaries is challenging but essential for improving summarization systems. There are several metrics and approaches:

1. ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

from rouge import Rouge

def evaluate_summary(reference_summary, generated_summary):
    """
    Evaluate a generated summary against a reference summary using ROUGE metrics.
    
    Args:
        reference_summary: The human-written or gold standard summary
        generated_summary: The automatically generated summary to evaluate
    
    Returns:
        ROUGE scores
    """
    rouge = Rouge()
    scores = rouge.get_scores(generated_summary, reference_summary)
    
    return scores

# Example evaluation
reference = "NLP is a field of AI focused on human-computer language interaction. It combines linguistics with machine learning to process and understand human language. NLP powers translation, voice commands, and text summarization. It's used in consumer applications and enterprise solutions to analyze customer feedback and streamline operations."

# Evaluate our summaries
print("\nROUGE Evaluation:")
print("TextRank Summary:")
print(evaluate_summary(reference, summary))
print("\nTransformer Summary:")
print(evaluate_summary(reference, custom_summary))

ROUGE measures the overlap of n-grams, word sequences, and word pairs between the generated summary and reference summaries.

2. Human Evaluation

Despite automated metrics, human evaluation remains the gold standard for assessing summary quality. Key aspects to evaluate include:

3. Practical Tips for Better Summarization

• Preprocessing: Clean and normalize text before summarization
• Domain adaptation: Fine-tune models on domain-specific data
• Length control: Adjust parameters to get appropriate summary length
• Chunking: Break long documents into manageable chunks
• Post-processing: Fix common issues like repetition or incomplete sentences

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, Trainer, TrainingArguments
from datasets import Dataset
import pandas as pd
import torch

# Sample dataset (in practice, use a larger domain-specific dataset)
data = {
    'text': [
        "The patient presented with fever, cough, and shortness of breath for 3 days. Chest X-ray showed bilateral infiltrates consistent with pneumonia. COVID-19 test was positive. The patient was admitted for supportive care and started on remdesivir.",
        "Q1 financial results exceeded expectations with revenue of $2.3B, up 15% YoY. Operating margin improved to 28.5%. The company raised full-year guidance and announced a $500M share repurchase program. New product launches contributed significantly to growth.",
        "The study examined the effects of climate change on coral reef ecosystems. Results showed a 30% decline in coral coverage over 10 years. Ocean acidification and rising temperatures were identified as primary factors. Conservation efforts have shown limited success in affected areas."
    ],
    'summary': [
        "COVID-19 positive patient with pneumonia admitted for supportive care and remdesivir treatment.",
        "Company reported strong Q1 results with 15% revenue growth, improved margins, and raised guidance.",
        "Study found 30% coral reef decline over 10 years due to climate change effects including acidification and temperature rise."
    ]
}

# Create a small dataset
df = pd.DataFrame(data)
dataset = Dataset.from_pandas(df)

# Load a pre-trained summarization model
model_name = "facebook/bart-large-cnn"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Tokenization function
def tokenize_function(examples):
    inputs = tokenizer(examples["text"], padding="max_length", truncation=True, max_length=512)
    outputs = tokenizer(examples["summary"], padding="max_length", truncation=True, max_length=128)
    
    return {
        "input_ids": inputs.input_ids,
        "attention_mask": inputs.attention_mask,
        "labels": outputs.input_ids,
    }

# Tokenize dataset
tokenized_dataset = dataset.map(tokenize_function, batched=True)

# Define training arguments
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir="./logs",
)

# Initialize Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
)

# Fine-tune the model (commented out as this is just an example)
# trainer.train()

# Test with a new example
medical_text = "The patient is a 45-year-old male with a history of hypertension who presented to the emergency department with chest pain radiating to the left arm. ECG showed ST elevation in leads II, III, and aVF. Cardiac enzymes were elevated. The patient was diagnosed with an inferior myocardial infarction and taken for emergency cardiac catheterization."

# In a real scenario, you would use the fine-tuned model
inputs = tokenizer(medical_text, return_tensors="pt", max_length=512, truncation=True)
summary_ids = model.generate(inputs["input_ids"], max_length=100, min_length=30, num_beams=4)
domain_summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)

print("\nDomain-specific summary example:")
print(domain_summary)

Understanding Topic Modeling

Topic modeling algorithms analyze text data to find clusters of words that frequently appear together, and then group documents based on these clusters. The key idea is that:

• Documents are mixtures of topics (e.g., a news article might be 70% politics and 30% economics)
• Topics are probability distributions over words (e.g., a politics topic might have high probabilities for words like "election," "government," and "policy")

Common applications of topic modeling include:

LDA Implementation

Let's implement topic modeling using different approaches, starting with the classic Latent Dirichlet Allocation (LDA):

import pandas as pd
import numpy as np
import gensim
from gensim import corpora
from gensim.models import LdaModel
from gensim.utils import simple_preprocess
from gensim.parsing.preprocessing import STOPWORDS
import nltk
from nltk.stem import WordNetLemmatizer

# Download required NLTK resources
nltk.download('wordnet')

# Sample documents
documents = [
    "Machine learning algorithms build mathematical models based on sample data to make predictions without being explicitly programmed.",
    "Deep learning is a subset of machine learning that uses neural networks with many layers.",
    "Natural language processing helps computers understand, interpret, and generate human language.",
    "Computer vision systems can identify objects, people, and activities in images and videos.",
    "Reinforcement learning is training algorithms to make decisions by rewarding desired behaviors.",
    "Neural networks are computing systems inspired by the biological neural networks in animal brains.",
    "Supervised learning algorithms learn from labeled training data to make predictions.",
    "Unsupervised learning finds patterns in data without pre-existing labels.",
    "Transfer learning reuses a pre-trained model on a new problem with limited data.",
    "Generative AI can create new content like images, text, and music based on training data."
]

# Preprocess the documents
def preprocess(text):
    # Remove stopwords and tokenize
    result = [word for word in simple_preprocess(text) if word not in STOPWORDS]
    
    # Lemmatize
    lemmatizer = WordNetLemmatizer()
    result = [lemmatizer.lemmatize(token) for token in result]
    
    return result

# Process all documents
processed_docs = [preprocess(doc) for doc in documents]

# Create a dictionary
dictionary = corpora.Dictionary(processed_docs)

# Create a document-term matrix
bow_corpus = [dictionary.doc2bow(doc) for doc in processed_docs]

# Train the LDA model
lda_model = LdaModel(
    corpus=bow_corpus,
    id2word=dictionary,
    num_topics=3,  # Number of topics to extract
    random_state=42,
    passes=10  # Number of passes through the corpus during training
)

# Print the topics
print("LDA Topics:")
for idx, topic in lda_model.print_topics(-1):
    print(f"Topic {idx}: {topic}")

# Classify documents
print("\nDocument Classifications:")
for i, doc in enumerate(bow_corpus):
    topic_distribution = lda_model.get_document_topics(doc)
    dominant_topic = sorted(topic_distribution, key=lambda x: x[1], reverse=True)[0]
    print(f"Document {i}: Topic {dominant_topic[0]} (Probability: {dominant_topic[1]:.2f})")
    print(f"  Original text: {documents[i][:50]}...")

LDA is a probabilistic model that assumes each document is a mixture of topics and each topic is a mixture of words. It's a good starting point for topic modeling tasks.

Non-Negative Matrix Factorization (NMF)

NMF is another popular approach that often produces more coherent topics:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF

# Create TF-IDF features
vectorizer = TfidfVectorizer(
    max_features=1000,
    stop_words='english',
    max_df=0.95,  # Ignore terms that appear in more than 95% of documents
    min_df=2  # Ignore terms that appear in fewer than 2 documents
)

# Transform documents to TF-IDF features
tfidf = vectorizer.fit_transform(documents)

# Get feature names
feature_names = vectorizer.get_feature_names_out()

# Train NMF model
nmf_model = NMF(
    n_components=3,  # Number of topics
    random_state=42,
    alpha=0.1,
    l1_ratio=0.5
)

# Apply NMF to the TF-IDF features
nmf = nmf_model.fit_transform(tfidf)

# Print the topics
print("\nNMF Topics:")
for topic_idx, topic in enumerate(nmf_model.components_):
    top_words_idx = topic.argsort()[:-11:-1]  # Get indices of top 10 words
    top_words = [feature_names[i] for i in top_words_idx]
    print(f"Topic {topic_idx}: {' '.join(top_words)}")

# Classify documents
print("\nDocument Classifications:")
for i, doc_topics in enumerate(nmf):
    dominant_topic = doc_topics.argmax()
    print(f"Document {i}: Topic {dominant_topic} (Weight: {doc_topics[dominant_topic]:.2f})")
    print(f"  Original text: {documents[i][:50]}...")

NMF often produces more coherent and interpretable topics than LDA, especially for short texts.

BERTopic: Leveraging Transformers for Topic Modeling

BERTopic combines transformer-based embeddings with traditional clustering techniques for state-of-the-art topic modeling:

from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups

# For demonstration, we'll use a larger dataset
newsgroups = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))
docs = newsgroups.data[:100]  # Using just 100 documents for speed

# Create and fit the BERTopic model
topic_model = BERTopic(verbose=True)
topics, probs = topic_model.fit_transform(docs)

# Print the topics
print("\nBERTopic Topics:")
for topic_id in sorted(set(topics)):
    if topic_id != -1:  # -1 represents outliers
        words = topic_model.get_topic(topic_id)
        print(f"Topic {topic_id}: {[word for word, _ in words[:5]]}")

# Visualize the topics (in a notebook environment)
# topic_model.visualize_topics()

# Visualize the documents with their topics
# topic_model.visualize_documents(docs)

# Get similar topics
print("\nSimilar Topics:")
similar_topics = topic_model.find_topics("computer", top_n=2)
for topic_id, similarity in similar_topics:
    if topic_id != -1:
        words = topic_model.get_topic(topic_id)
        print(f"Topic {topic_id} (Similarity: {similarity:.2f}): {[word for word, _ in words[:5]]}")

# Reduce topics
print("\nReducing Topics:")
topic_model.reduce_topics(docs, nr_topics=5)
for topic_id in range(5):
    words = topic_model.get_topic(topic_id)
    print(f"Reduced Topic {topic_id}: {[word for word, _ in words[:5]]}")

BERTopic leverages the semantic understanding of transformer models to create more meaningful topics, especially for complex or nuanced content.

Visualizing Topics

Visualization is crucial for interpreting and communicating topic modeling results. Let's explore some visualization techniques:

import matplotlib.pyplot as plt
import seaborn as sns
from wordcloud import WordCloud
import pyLDAvis
import pyLDAvis.gensim_models as gensimvis

# 1. Word Clouds for Topics
def visualize_topics_wordcloud(model, feature_names, n_top_words=10):
    """Generate word clouds for each topic"""
    for topic_idx, topic in enumerate(model.components_):
        top_words_idx = topic.argsort()[:-n_top_words-1:-1]
        top_words = {feature_names[i]: topic[i] for i in top_words_idx}
        
        # Generate word cloud
        wordcloud = WordCloud(
            width=800, 
            height=400, 
            background_color='white',
            max_words=50
        ).generate_from_frequencies(top_words)
        
        plt.figure(figsize=(10, 5))
        plt.imshow(wordcloud, interpolation='bilinear')
        plt.axis('off')
        plt.title(f'Topic {topic_idx}')
        plt.tight_layout()
        plt.savefig(f'topic_{topic_idx}_wordcloud.png')
        plt.close()
        
        print(f"Word cloud for Topic {topic_idx} saved as topic_{topic_idx}_wordcloud.png")

# Visualize NMF topics
visualize_topics_wordcloud(nmf_model, feature_names)

# 2. Topic Distribution in Documents
def visualize_document_topics(doc_topic_matrix, n_docs=10):
    """Visualize topic distribution in documents"""
    # Select a subset of documents
    subset = doc_topic_matrix[:n_docs]
    
    # Create a heatmap
    plt.figure(figsize=(10, 8))
    sns.heatmap(
        subset, 
        annot=True, 
        cmap='YlGnBu', 
        fmt='.2f',
        xticklabels=[f'Topic {i}' for i in range(subset.shape[1])],
        yticklabels=[f'Doc {i}' for i in range(subset.shape[0])]
    )
    plt.title('Topic Distribution in Documents')
    plt.tight_layout()
    plt.savefig('document_topic_distribution.png')
    plt.close()
    
    print("Document-topic distribution saved as document_topic_distribution.png")

# Visualize document-topic distribution for NMF
visualize_document_topics(nmf)

# 3. Interactive Visualization with pyLDAvis
def visualize_lda_interactive(lda_model, corpus, dictionary):
    """Create an interactive visualization of LDA topics"""
    # Prepare the visualization
    vis_data = gensimvis.prepare(lda_model, corpus, dictionary)
    
    # Save as HTML
    pyLDAvis.save_html(vis_data, 'lda_visualization.html')
    print("Interactive LDA visualization saved as lda_visualization.html")

# Visualize LDA model
visualize_lda_interactive(lda_model, bow_corpus, dictionary)

# 4. Topic Similarity Network
def visualize_topic_similarity(model, feature_names, threshold=0.2):
    """Visualize topic similarity as a network"""
    import networkx as nx
    
    # Calculate topic similarity using cosine similarity
    from sklearn.metrics.pairwise import cosine_similarity
    similarity = cosine_similarity(model.components_)
    
    # Create a graph
    G = nx.Graph()
    
    # Add nodes (topics)
    for i in range(len(model.components_)):
        # Get top words for the topic
        top_words_idx = model.components_[i].argsort()[:-6:-1]
        top_words = [feature_names[idx] for idx in top_words_idx]
        label = ', '.join(top_words[:3])
        G.add_node(i, label=label)
    
    # Add edges (similarities above threshold)
    for i in range(len(similarity)):
        for j in range(i+1, len(similarity)):
            if similarity[i, j] > threshold:
                G.add_edge(i, j, weight=similarity[i, j])
    
    # Draw the graph
    plt.figure(figsize=(12, 8))
    pos = nx.spring_layout(G, seed=42)
    
    # Draw nodes
    nx.draw_networkx_nodes(G, pos, node_size=500, node_color='lightblue')
    
    # Draw edges with width based on similarity
    edge_widths = [G[u][v]['weight'] * 5 for u, v in G.edges()]
    nx.draw_networkx_edges(G, pos, width=edge_widths, alpha=0.7)
    
    # Draw labels
    labels = {node: G.nodes[node]['label'] for node in G.nodes()}
    nx.draw_networkx_labels(G, pos, labels, font_size=10)
    
    plt.title('Topic Similarity Network')
    plt.axis('off')
    plt.tight_layout()
    plt.savefig('topic_similarity_network.png')
    plt.close()
    
    print("Topic similarity network saved as topic_similarity_network.png")

# Visualize topic similarity for NMF
visualize_topic_similarity(nmf_model, feature_names)

These visualizations help in interpreting the topics and their relationships, making it easier to communicate the results to stakeholders.

Advanced Topic Modeling Techniques

For more advanced applications, consider these techniques:

from gensim.models import LdaSeqModel

# Example usage (conceptual)
# ldaseq = LdaSeqModel(corpus=time_slices, 
#                     id2word=dictionary,
#                     time_slice=document_counts_per_time_slice,
#                     num_topics=5)

from gensim.models.hdpmodel import HdpModel

# Example usage
hdp_model = HdpModel(corpus=bow_corpus, 
                    id2word=dictionary)

from gensim.models import LdaModel

# Define seed topics
seed_topics = {
    0: ['learning', 'model', 'data', 'algorithm'],
    1: ['neural', 'network', 'deep', 'layer'],
    2: ['language', 'processing', 'text', 'nlp']
}

# Example usage (conceptual)
# guided_lda = LdaModel(corpus=bow_corpus,
#                      id2word=dictionary,
#                      num_topics=3,
#                      seed_topics=seed_topics)

# Using multilingual embeddings
from sentence_transformers import SentenceTransformer

# Load multilingual model
model = SentenceTransformer('paraphrase-multilingual-mpnet-base-v2')

# Example usage (conceptual)
# embeddings = model.encode(multilingual_documents)
# Then apply clustering or topic modeling to embeddings

QA System Architecture

A typical QA system consists of three main components:

1. Question Understanding: This involves extracting relevant information from the question.
2. Information Retrieval: This involves finding documents that contain the relevant information.
3. Answer Generation: This involves generating a coherent and accurate answer based on the retrieved information.

Building a QA System

Let's implement a simple rule-based QA system using Python:

import re

# Sample text
text = """
Apple Inc. is planning to open a new office in New York City by January 2024. 
The company's CEO, Tim Cook, announced this during a press conference on 
May 15th, 2023. The project will cost approximately $50 million and create 
about 200 new jobs. Apple's stock rose 3% following the announcement.
"""

# Define regex patterns for different entity types
patterns = {
    'DATE': r'\b(?:January|February|March|April|May|June|July|August|September|October|November|December)\s+\d{1,2}(?:st|nd|rd|th)?,\s+\d{4}\b|\b\d{1,2}/\d{1,2}/\d{2,4}\b',
    'PERSON': r'\b[A-Z][a-z]+ [A-Z][a-z]+\b',
    'ORG': r'\b[A-Z][a-z]+ (?:Inc\.|Corp\.|LLC|Ltd\.)\b',
    'MONEY': r'\$\d+(?:\.\d+)? (?:million|billion|trillion)\b|\$\d+(?:,\d+)*(?:\.\d+)?\b',
    'PERCENT': r'\b\d+(?:\.\d+)?%\b'
}

# Find entities
for entity_type, pattern in patterns.items():
    print(f"\n{entity_type} entities:")
    for match in re.finditer(pattern, text):
        print(f"  - {match.group(0)}")

# Visualize entities in text
def highlight_entities(text, patterns):
    # Create a copy of the text for highlighting
    highlighted = text
    
    # Dictionary to store entity positions
    entity_positions = []
    
    # Find all entities and their positions
    for entity_type, pattern in patterns.items():
        for match in re.finditer(pattern, text):
            start, end = match.span()
            entity_positions.append((start, end, entity_type, match.group(0)))
    
    # Sort by start position in reverse order (to avoid messing up indices when inserting tags)
    entity_positions.sort(key=lambda x: x[0], reverse=True)
    
    # Insert HTML-like tags for highlighting
    for start, end, entity_type, entity_text in entity_positions:
        highlighted = highlighted[:end] + f"[/{entity_type}]" + highlighted[end:]
        highlighted = highlighted[:start] + f"[{entity_type}]" + highlighted[start:]
    
    return highlighted

# Print highlighted text
print("\nHighlighted text:")
print(highlight_entities(text, patterns))

While simple to implement, rule-based approaches have limitations in handling ambiguity and require manual pattern creation for each entity type.

2. Using spaCy for NER

spaCy provides pre-trained models for NER that work well out of the box:

import spacy
from spacy import displacy

# Load English model
nlp = spacy.load("en_core_web_md")

# Sample text
text = """
Apple Inc. is planning to open a new office in New York City by January 2024. 
The company's CEO, Tim Cook, announced this during a press conference on 
May 15th, 2023. The project will cost approximately $50 million and create 
about 200 new jobs. Apple's stock rose 3% following the announcement.
"""

# Process the text
doc = nlp(text)

# Print entities
print("Named Entities:")
for ent in doc.ents:
    print(f"  - {ent.text} ({ent.label_}: {spacy.explain(ent.label_)})")

# Function to visualize entities in text
def visualize_entities(text):
    doc = nlp(text)
    
    # Display entities in HTML format
    html = displacy.render(doc, style="ent", jupyter=False)
    
    # For non-Jupyter environments, you can save to a file
    with open("ner_visualization.html", "w", encoding="utf-8") as f:
        f.write(html)
    
    print("Visualization saved to ner_visualization.html")
    
    # Return entity counts for analysis
    entity_counts = {}
    for ent in doc.ents:
        if ent.label_ in entity_counts:
            entity_counts[ent.label_] += 1
        else:
            entity_counts[ent.label_] = 1
    
    return entity_counts

# Visualize entities
entity_counts = visualize_entities(text)
print("\nEntity counts:", entity_counts)

# Process a larger text for analysis
larger_text = """
Microsoft Corporation announced a partnership with OpenAI in San Francisco last week.
The deal, worth $1 billion, will focus on developing artificial intelligence technologies.
Satya Nadella, CEO of Microsoft, met with Sam Altman on Thursday to finalize the agreement.
The collaboration will start on October 1st, 2023, and is expected to last for at least 5 years.
Both companies' stocks performed well after the announcement, with Microsoft seeing a 2.3% increase.
"""

# Process and analyze
doc_large = nlp(larger_text)

# Analyze entity relationships
print("\nEntity Relationships:")
for sent in doc_large.sents:
    sent_doc = nlp(sent.text)
    entities = [(e.text, e.label_) for e in sent_doc.ents]
    if len(entities) > 1:
        print(f"Sentence: {sent}")
        print(f"Entities: {entities}")
        print("---")

spaCy provides excellent out-of-the-box performance for NER and includes visualization tools to help understand the results.

3. Transformer-Based Approach with Hugging Face

For state-of-the-art performance, we can use pre-trained transformer models:

from transformers import pipeline, AutoTokenizer, AutoModelForTokenClassification
import pandas as pd

# Option 1: Using the pipeline API (simplest approach)
ner_pipeline = pipeline("ner", aggregation_strategy="simple")

# Sample text
text = """
Apple Inc. is planning to open a new office in New York City by January 2024. 
The company's CEO, Tim Cook, announced this during a press conference on 
May 15th, 2023. The project will cost approximately $50 million and create 
about 200 new jobs. Apple's stock rose 3% following the announcement.
"""

# Get entities
entities = ner_pipeline(text)

# Display results
print("Named Entities:")
for entity in entities:
    print(f"  - {entity['word']} ({entity['entity_group']}, score: {entity['score']:.4f})")

# Option 2: More control with explicit model loading
# Load model and tokenizer
model_name = "dslim/bert-base-NER"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

# Function for NER with custom threshold
def extract_entities(text, threshold=0.9):
    # Tokenize and convert to model inputs
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
    
    # Get model outputs
    import torch
    outputs = model(**inputs)
    
    # Get predictions
    predictions = torch.argmax(outputs.logits, dim=2)
    
    # Get confidence scores
    scores = torch.nn.functional.softmax(outputs.logits, dim=2)
    
    # Extract entities
    tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
    
    # Get label map
    id2label = model.config.id2label
    
    # Extract entities
    entities = []
    current_entity = {"text": "", "type": "", "score": 0, "tokens": []}
    
    for i, (token, pred, score) in enumerate(zip(tokens, predictions[0], scores[0])):
        label = id2label[pred.item()]
        confidence = score[pred.item()].item()
        
        # Skip special tokens
        if token in [tokenizer.cls_token, tokenizer.sep_token, tokenizer.pad_token]:
            continue
            
        # Handle subword tokens
        if token.startswith("##"):
            token = token[2:]  # Remove ## prefix
            if current_entity["text"]:
                current_entity["text"] += token
                current_entity["tokens"].append({"token": token, "label": label, "score": confidence})
                # Update average score
                current_entity["score"] = sum(t["score"] for t in current_entity["tokens"]) / len(current_entity["tokens"])
            continue
            
        # If we're in an entity and the label changes or is O, finish the current entity
        if current_entity["text"] and (label == "O" or not label.endswith(current_entity["type"])):
            if current_entity["score"] >= threshold:
                entities.append({
                    "text": current_entity["text"],
                    "type": current_entity["type"],
                    "score": current_entity["score"]
                })
            current_entity = {"text": "", "type": "", "score": 0, "tokens": []}
            
        # If we have a new entity, start tracking it
        if label != "O":
            # Extract entity type (remove B- or I- prefix)
            entity_type = label[2:] if label.startswith("B-") or label.startswith("I-") else label
            
            if not current_entity["text"]:
                current_entity = {
                    "text": token,
                    "type": entity_type,
                    "score": confidence,
                    "tokens": [{"token": token, "label": label, "score": confidence}]
                }
            else:
                current_entity["text"] += " " + token
                current_entity["tokens"].append({"token": token, "label": label, "score": confidence})
                # Update average score
                current_entity["score"] = sum(t["score"] for t in current_entity["tokens"]) / len(current_entity["tokens"])
    
    # Don't forget the last entity
    if current_entity["text"] and current_entity["score"] >= threshold:
        entities.append({
            "text": current_entity["text"],
            "type": current_entity["type"],
            "score": current_entity["score"]
        })
    
    return entities

# Extract entities with custom function
custom_entities = extract_entities(text)

# Display results
print("\nCustom Entity Extraction:")
for entity in custom_entities:
    print(f"  - {entity['text']} ({entity['type']}, score: {entity['score']:.4f})")

# Convert to DataFrame for analysis
df = pd.DataFrame(custom_entities)
print("\nEntity Summary:")
print(df.groupby("type").agg({"text": "count"}).reset_index().rename(columns={"text": "count"}))

Transformer-based models provide state-of-the-art performance for NER, especially for complex texts and ambiguous entities.

Evaluating QA Performance

Evaluating the performance of QA systems is challenging but essential for improving their accuracy and reliability. There are several metrics and approaches:

1. Accuracy

Accuracy measures the percentage of questions that the system correctly answers. It's a straightforward metric but may not capture the nuances of QA systems.

2. F1 Score

F1 Score is the harmonic mean of precision and recall. It's a good metric for imbalanced datasets and provides a balance between precision and recall.

3. BLEU Score

BLEU Score is a popular metric for evaluating the quality of machine translation. It can also be used to evaluate the coherence of answers.

4. ROUGE Score

ROUGE Score is a metric for evaluating summarization systems. It measures the overlap of n-grams between the system's output and the reference summary.

5. Human Evaluation

Despite automated metrics, human evaluation remains the gold standard for assessing QA systems. Key aspects to evaluate include:

6. Practical Tips for Better QA Systems

• Preprocessing: Clean and normalize text before processing
• Domain-specific Training: Fine-tune models on domain-specific data
• Ensemble Methods: Combine multiple models for better performance
• Human-in-the-loop: Use human feedback to improve system performance

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, Trainer, TrainingArguments
from datasets import Dataset
import pandas as pd
import torch

# Sample dataset (in practice, use a larger domain-specific dataset)
data = {
    'question': [
        "What is the capital of France?",
        "What is the population of New York City?",
        "What is the main language spoken in Brazil?"
    ],
    'context': [
        "France is a country in Europe. The capital city is Paris.",
        "New York City is a city in the United States. It has a population of approximately 8.4 million.",
        "Portuguese is the main language spoken in Brazil."
    ],
    'answer': [
        "The capital of France is Paris.",
        "The population of New York City is approximately 8.4 million.",
        "Portuguese is the main language spoken in Brazil."
    ]
}

# Create a small dataset
df = pd.DataFrame(data)
dataset = Dataset.from_pandas(df)

# Load a pre-trained summarization model
model_name = "facebook/bart-large-cnn"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Tokenization function
def tokenize_function(examples):
    inputs = tokenizer(examples["question"], padding="max_length", truncation=True, max_length=512)
    outputs = tokenizer(examples["context"], padding="max_length", truncation=True, max_length=512)
    
    return {
        "input_ids": inputs.input_ids,
        "attention_mask": inputs.attention_mask,
        "labels": outputs.input_ids,
    }

# Tokenize dataset
tokenized_dataset = dataset.map(tokenize_function, batched=True)

# Define training arguments
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir="./logs",
)

# Initialize Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
)

# Fine-tune the model (commented out as this is just an example)
# trainer.train()

# Test with a new example
question = "What is the population of Tokyo?"
context = "Tokyo is the capital city of Japan. It has a population of approximately 37 million."

# In a real scenario, you would use the fine-tuned model
inputs = tokenizer(question, context, return_tensors="pt", max_length=512, truncation=True)
answer_ids = model.generate(inputs["input_ids"], max_length=100, min_length=30, num_beams=4)
answer = tokenizer.decode(answer_ids[0], skip_special_tokens=True)

print("\nDomain-specific QA example:")
print(f"Question: {question}")
print(f"Context: {context}")
print(f"Answer: {answer}")

Deployment Options

There are several options for deploying NLP applications:

• API Deployment: Creating a REST API that can be consumed by other systems.
• Cloud Services: Using cloud platforms like AWS, GCP, or Azure for hosting and scaling.
• Serverless Architectures: Deploying applications without managing servers.
• Containerization: Using containers like Docker for packaging and deployment.
• Server-side Processing: Running NLP tasks on a server or cloud platform.

Creating an NLP API

Let's create a simple NLP API using Flask:

from flask import Flask, request, jsonify
from transformers import pipeline

app = Flask(__name__)

# Load a pre-trained model
model_name = "distilbert-base-uncased-finetuned-sst-2-english"
sentiment_analyzer = pipeline("sentiment-analysis", model=model_name)

@app.route('/analyze', methods=['POST'])
def analyze():
    data = request.get_json()
    text = data['text']
    result = sentiment_analyzer(text)
    return jsonify({'sentiment': result[0]['label']})

if __name__ == '__main__':
    app.run(debug=True)

This API can be easily deployed on cloud platforms or containerized environments.

Scaling Considerations

Scaling an NLP application involves:

• Horizontal Scaling: Adding more instances of the application to handle increased traffic.
• Vertical Scaling: Increasing the resources of the existing instances.
• Load Balancing: Distributing incoming requests evenly across instances.
• Caching: Storing frequently accessed data to reduce response time.
• Database Optimization: Optimizing database queries for better performance.

Further Learning

To deepen your understanding of NLP, consider the following resources:

• Books: "Natural Language Processing with Python" by Steven Bird, Ewan Klein, and Edward Loper.
• Online Courses: Coursera's "Natural Language Processing" specialization by Stanford University.
• Websites: NLP-related blogs and websites like the Natural Language Toolkit (NLTK) and spaCy.
• Research Papers: Explore recent research papers on topics like sentiment analysis, named entity recognition, and text summarization.

Project Ideas

Here are some project ideas to help you apply your new skills:

• Sentiment Analysis Dashboard: Build a dashboard that analyzes customer sentiment on social media platforms.
• Named Entity Recognition System: Develop a system that identifies and categorizes entities in legal documents or news articles.
• Text Summarization Tool: Create a tool that summarizes long documents while preserving key information.
• Topic Modeling Application: Develop an application that discovers topics in a large collection of documents.
• Question Answering Bot: Build a chatbot that answers questions based on a knowledge base.

Recommended Resources

Here are some resources to help you get started:

• GitHub Repositories: Explore NLP-related projects on GitHub for inspiration and code examples.
• Online Communities: Join forums like Stack Overflow or Slack channels for NLP enthusiasts.
• Tutorials and Workshops: Attend workshops or take online courses to learn new techniques and tools.
• Competitions and Challenges: Participate in NLP competitions to test your skills and learn from others.