📝 Intelligent Document Insight System | Technical Documentation:

📝 Technical Documentation: Intelligent Document Insight System

Framework: TAJ (Think → Ask → Judge → Accomplish) Architectural Model
Target Use Case: Automated Resume Screening & Talent Pipeline Filtering
Author/Developer: AI Engineering Student Portfolio

---

🗺️ Project Overview & Architecture Blueprint

The Intelligent Document Insight System shifts away from chaotic, manual "one-shot prompting" toward a reliable, multi-step automated data processing pipeline. The system handles raw, unstructured candidate resumes (PDFs and Text files), segments them to preserve context, applies strict skill-based matching rules, structures unstructured data into predictable JSON profiles, runs safety gates for missing values, and exposes this pipeline to an interactive user workspace.

🏗️ Complete End-to-End System Architecture

    Raw Data Ingestion (PDF/TXT)
               │
               ▼
     [Layer 1: Preprocessing] ──► Strips noise (headers, footers, margins)
               │
               ▼
     [Layer 2: Chunking Engine] ──► 500-character windows with 50-char overlaps
               │
               ▼
     [Layer 5: Query Matcher] ──► Extracts chunks mentioning target keywords
               │
               ▼
     [Layer 3 & 6: Intelligence] ──► GPT-4o-mini parses chunks into strict Pydantic layouts
               │
               ▼
     [Layer 4: Aggregation] ──► Merges and deduplicates chunk-level data arrays
               │
               ▼
     [Layer 7: Validation Gate] ──► Checks data completeness
               ├──► Clear Pass ─────► [Layer 8: Streamlit Dashboard UI]
               └──► Fail Validation ──► Flagged Queue for Human Review

---

🛠️ Phase-by-Phase Execution Log & Decisions Taken

Module 1: Preprocessing & Data Ingestion Layer

• Action Taken: Programmatically isolated raw text streams from file uploads and removed system layout formatting artifacts.
• Tool Chosen: pypdf (PdfReader) for fast native Python data stream extraction.
• Architectural Decision: Implemented a regular expression filter (re.sub(r'\n+', '\n', text_content)) to collapse multiple erratic whitespace gaps into uniform line breaks, preventing token bloat.
• Next-Step Connection: Passed the cleaned, unified text block directly into the split sequencer.

Module 2: Chunking Engine

• Action Taken: Fragmented large document data streams into predictable processing units to prevent LLM context-window degradation.
• Tool Chosen: Pure Python character-based looping indices.
• Architectural Decision: Set a strict window size boundary of 500 characters paired with a trailing 50-character overlap. This 10% overlap acts as a context bridge, ensuring critical candidate terms (like job spans or compound skill sets) are not clipped midway at chunk boundaries.
• Next-Step Connection: Populated a structured dictionary array ([{"chunk_id": X, "text": Y}]) to pass downstream.

Module 3 & 6: Intelligence & Output Structuring Layer

• Action Taken: Applied programmatic models over independent chunks to strip out explicit metadata elements.
• Tool Chosen: OpenAI API (gpt-4o-mini) integrated with Pydantic core typing rules.
• Architectural Decision: Abandoned standard text generation prompts. Adopted OpenAI's Native Structured Outputs API (client.beta.chat.completions.parse) paired with a rigid CandidateProfile Pydantic class. This guarantees that the LLM will never return loose text, forcing it to align with predictable JSON arrays or explicitly fail.
• Next-Step Connection: Dispatched structured chunk-level profile classes into the text matrix synthesiser.

Module 4: Aggregation Layer

• Action Taken: Consolidated isolated micro-insights from disjointed chunks back into a unified document profile.
• Tool Chosen: Native Python Data Structures (set expressions & list comprehensions).
• Architectural Decision: Extracted duplicate profile fields (e.g., if a skill or school was mentioned in three separate chunks) by converting them into Python sets and casting them back to lists. This cleanly prevents data duplication without making unnecessary extra LLM calls.
• Next-Step Connection: Handed the unified, aggregated single record to the system auditor.

Module 5: Query Layer

• Action Taken: Filtered text chunks prior to LLM processing to reduce token consumption and eliminate data noise.
• Tool Chosen: Python case-insensitive string parsing (keyword.lower() in chunk.lower()).
• Architectural Decision: Selected a fast, deterministic keyword-matching query tool instead of a heavy vector database (like FAISS/ChromaDB). This matches the requirements of resume processing perfectly, where recruiters need to verify explicit skill sets (e.g., "Python") rather than general semantic similarities. Included a fallback safety mechanism: if no keywords match, the system passes all chunks to ensure candidate info is never dropped.
• Next-Step Connection: Filtered chunks to reduce LLM workloads in Modules 3 & 6.

Module 7: Validation & Reliability Layer

• Action Taken: Engineered an automated data quality gate to catch incomplete AI extractions.
• Tool Chosen: Conditional logic checking criteria exceptions.
• Architectural Decision: Configured a data isolation rule: if vital identifiers like candidate_name, education, or past_job_titles return as "Unknown" or empty, the record is immediately flagged and routed to a dedicated Human Review Queue. This protects production databases from corrupt data while keeping human workers in the loop.
• Next-Step Connection: Dispatched clear records to the UI dashboard and flagged items to the review holding panel.

Module 8: MVP Build & User Interface Layer

• Action Taken: Wrapped the entire pipeline logic into an intuitive, functional dashboard.
• Tool Chosen: Streamlit interactive Python framework.
• Architectural Decision: Designed a dual-tab configuration split into a Verified Insights Dashboard and a Human Review Queue. Added a manual "Force Approve" feature to the review queue, letting human recruiters override pipeline flags and move patched profiles to production instantly.

---

💻 Complete Monolithic Implementation Source Code

This file integrates all layers of the Intelligent Document Insight System into a single runnable codebase.

import os
import re
import json
import streamlit as st
from pypdf import PdfReader
from pydantic import BaseModel, Field
from typing import List, Optional
from openai import OpenAI

# Initialize OpenAI Client (automatically reads the OPENAI_API_KEY environment variable)
client = OpenAI()

# ==========================================
# LAYER 6: STRUCTURED OUTPUT DATA SCHEMAS
# ==========================================
class CandidateProfile(BaseModel):
    candidate_name: str = Field(description="Full name of the candidate or 'Unknown'")
    past_job_titles: List[str] = Field(default=[], description="List of unique past job positions held")
    education: List[str] = Field(default=[], description="Degrees, diplomas, or schools attended")
    certifications: List[str] = Field(default=[], description="Professional certifications or licenses")
    skills_found: List[str] = Field(default=[], description="Skills matched specifically from the search criteria")
    executive_summary: str = Field(description="A brief sentence synthesizing the profile details")

# Set up Streamlit Page Configuration
st.set_page_config(page_title="Intelligent Document Insight System", layout="wide")
st.title("⚙️ Intelligent Document Insight System")
st.caption("A production-grade resume analysis system built on the TAJ Framework")

# Initialize session states for storing pipeline results across reruns
if "processed_profiles" not in st.session_state:
    st.session_state.processed_profiles = []
if "human_review_queue" not in st.session_state:
    st.session_state.human_review_queue = []

# ==========================================
# SIDEBAR: RECRUITER CONTROLS & INGESTION
# ==========================================
with st.sidebar:
    st.header("📥 Ingestion & Criteria Settings")
    
    # Keyword criteria input for Layer 5 Matcher
    search_keywords_input = st.text_input(
        "Enter Target Skill Keywords (comma-separated)", 
        value="Python, AWS, Project Management"
    )
    keywords = [k.strip() for k in search_keywords_input.split(",") if k.strip()]
    
    st.divider()
    
    # Layer 1 Ingestion UI Component
    uploaded_files = st.file_uploader(
        "Upload Candidate Resumes (PDF or TXT)", 
        type=["pdf", "txt"], 
        accept_multiple_files=True
    )
    
    trigger_process = st.button("🚀 Process Resumes", type="primary")

# ==========================================
# CORE BACKEND PIPELINE FUNCTIONS
# ==========================================

def layer_1_preprocess_data(uploaded_file) -> str:
    """Extracts raw text data and cleans up obvious whitespace structures."""
    text_content = ""
    if uploaded_file.name.endswith(".pdf"):
        reader = PdfReader(uploaded_file)
        for page in reader.pages:
            page_text = page.extract_text()
            if page_text:
                text_content += page_text + "\n"
    else:
        text_content = uploaded_file.read().decode("utf-8")
        
    # Basic data normalizing: clean multiple line breaks and strip leading/trailing spaces
    normalized_text = re.sub(r'\n+', '\n', text_content).strip()
    return normalized_text

def layer_2_chunk_engine(text: str, chunk_size: int = 500, overlap: int = 50) -> List[dict]:
    """Divides long texts into manageable windows with character-based overlap approximation."""
    chunks = []
    start = 0
    chunk_id = 1
    
    if len(text) <= chunk_size:
        return [{"chunk_id": 1, "text": text}]
        
    while start < len(text):
        end = start + chunk_size
        chunk_segment = text[start:end]
        chunks.append({"chunk_id": chunk_id, "text": chunk_segment})
        chunk_id += 1
        start += (chunk_size - overlap)
        
    return chunks

def layer_5_keyword_query_matcher(chunks: List[dict], target_keywords: List[str]) -> List[dict]:
    """Filters chunks that specifically mention target screening keywords (Case Insensitive)."""
    relevant_chunks = []
    for chunk in chunks:
        for keyword in target_keywords:
            if keyword.lower() in chunk["text"].lower():
                relevant_chunks.append(chunk)
                break  # Save chunk once if any keyword hits
    return relevant_chunks

def layer_3_and_6_intelligence_and_structuring(chunk_text: str, query_keywords: List[str]) -> Optional[CandidateProfile]:
    """Invokes LLM reasoning over text units and forces output into structured schemas."""
    keywords_str = ", ".join(query_keywords)
    
    system_prompt = (
        "You are an expert HR data parsing assistant. Extract candidate profile elements accurately.\n"
        f"Analyze the text context provided specifically searching for evidence of these target skills: [{keywords_str}]."
    )
    
    user_prompt = f"Extract details from this resume segment:\n\n{chunk_text}"
    
    try:
        # Utilizing OpenAI's native Structured Outputs parsing capabilities
        completion = client.beta.chat.completions.parse(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_prompt}
            ],
            response_format=CandidateProfile,
        )
        return completion.choices.message.parsed
    except Exception as e:
        st.error(f"LLM Processing Error: {str(e)}")
        return None

def layer_4_aggregation_layer(profiles: List[CandidateProfile]) -> CandidateProfile:
    """Merges all unique values captured across chunk profiles into one global file structure."""
    if not profiles:
        return CandidateProfile(candidate_name="Unknown", executive_summary="No data aggregated.")
        
    # Extract structural items while weeding out generic empty initializations
    names = [p.candidate_name for p in profiles if p.candidate_name.lower() != "unknown"]
    global_name = names[0] if names else "Unknown Name"
    
    # Deduplicate collection categories using sets
    global_jobs = list(set(title for p in profiles for title in p.past_job_titles))
    global_edu = list(set(edu for p in profiles for edu in p.education))
    global_cert = list(set(cert for p in profiles for cert in p.certifications))
    global_skills = list(set(skill for p in profiles for skill in p.skills_found))
    
    # Compile text narrative summary strings together
    summaries = [p.executive_summary for p in profiles if p.executive_summary]
    global_summary = " ".join(summaries[:3]) # Constrain length bounds
    
    return CandidateProfile(
        candidate_name=global_name,
        past_job_titles=global_jobs,
        education=global_edu,
        certifications=global_cert,
        skills_found=global_skills,
        executive_summary=global_summary
    )

# ==========================================
# CORE APP RUNTIME EXECUTION CONTROLLER
# ==========================================
if trigger_process and uploaded_files:
    # Flush existing application cache tracking rows
    st.session_state.processed_profiles = []
    st.session_state.human_review_queue = []
    
    for file in uploaded_files:
        with st.spinner(f"Processing candidate profile: {file.name}..."):
            # 1. Ingestion
            raw_text = layer_1_preprocess_data(file)
            
            # 2. Chunking
            text_chunks = layer_2_chunk_engine(raw_text)
            
            # 5. Query Filtering via simpler Keyword Matching
            matched_chunks = layer_5_keyword_query_matcher(text_chunks, keywords)
            
            # If no keyword conditions hit, fall back to entire text chunks to avoid missing profile info
            chunks_to_analyze = matched_chunks if matched_chunks else text_chunks
            
            chunk_profiles = []
            for chunk in chunks_to_analyze:
                # 3 & 6. Core Intelligence processing to extract parsed profiles
                parsed_chunk = layer_3_and_6_intelligence_and_structuring(chunk["text"], keywords)
                if parsed_chunk:
                    chunk_profiles.append(parsed_chunk)
            
            # 4. Aggregating disparate chunk items back into global view
            final_aggregated_profile = layer_4_aggregation_layer(chunk_profiles)
            
            # ==========================================
            # LAYER 7: VALIDATION & RELIABILITY GATE
            # ==========================================
            # Checking critical structural values for missing components
            is_invalid = (
                final_aggregated_profile.candidate_name == "Unknown Name" or
                final_aggregated_profile.candidate_name == "Unknown" or
                not final_aggregated_profile.education or
                not final_aggregated_profile.past_job_titles
            )
            
            if is_invalid:
                # Flag profile data context for Manual Human Review Bucket
                st.session_state.human_review_queue.append({
                    "filename": file.name,
                    "partial_data": final_aggregated_profile.model_dump()
                })
            else:
                # Certified validation passed successfully
                st.session_state.processed_profiles.append(final_aggregated_profile.model_dump())
                
    st.success("All pipelines executed successfully!")

# ==========================================
# LAYER 8: USER INTERFACE PRESENTATION WORKSPACE
# ==========================================
tab_dashboard, tab_review = st.tabs(["📊 Candidate Insights Dashboard", "⚠️ Human Review Queue"])

with tab_dashboard:
    if st.session_state.processed_profiles:
        st.subheader("Verified Professional Profiles")
        for profile in st.session_state.processed_profiles:
            with st.expander(f"👤 Profile: {profile['candidate_name']}", expanded=True):
                col1, col2 = st.columns()
                
                with col1:

st.write("Matched Skills:")
if profile('skills_found'):
for skill in profile('skills_found'):
st.caption(f"✔️ {skill}")
else:
st.write("None matched.")

with col2:
st.write(f"Executive Summary: {profile('executive_summary')}")
st.write(f"Past Roles: {', '.join(profile('past_job_titles'))}")
st.write(f"Education Details: {', '.join(profile('education'))}")
if profile('certifications'):
st.write(f"Certifications: {', '.join(profile('certifications'))}")
else:
st.info("No verified candidate files parsed yet. Upload documents on the sidebar panel to launch optimization processing.")

with tab_review:
if st.session_state.human_review_queue:
st.warning(f"There are {len(st.session_state.human_review_queue)} records that failed systemic verification gates.")
for index, review_item in enumerate(st.session_state.human_review_queue):
st.error(f"Incomplete Data Payload: Tracking File [{review_item('filename', })")
st.json(review_item('partial_data'))

# Simple override simulation button to demonstrate moving elements manually to production
if st.button(f"Force Approve File Record {index+1}", key=f"btn_{index}"):
# Patch record names if needed
if review_item('partial_data')('candidate_name') in ("Unknown", "Unknown Name"):
review_item('partial_data')('candidate_name') = f"Manual Verification Needed ({review_item('filename')})"
st.session_state.processed_profiles.append(review_item('partial_data'))
st.session_state.human_review_queue.pop(index)
st.rerun()
else:
st.success("Clear Skies! Review desk queue is empty. All processed files cleared the validation safety checks.")

***

### 🎓 Summary of Achieved Transformation
By implementing this project, you have successfully completed the core journey from a standard **prompt user** to a structured **AI systems builder**:
1. You moved away from large, erratic inputs and replaced them with structured, **preprocessed chunk payloads**.
2. You eliminated formatting issues by introducing programmatic data cleaning.
3. You resolved unstructured AI outputs by enforcing structured layouts with **Pydantic schemas**.
4. You removed system dependencies on unstable text generations by writing **deterministic evaluation layers** that flag bad data for human review.

***

Now that your project documentation is fully compiled, what are your next steps? I can help you with:
* Setting up a **GitHub repository** structure to host your code.
* Writing a **deployment guide** to deploy your app live on Streamlit Cloud.
* Drafting a final summary blog post to showcase your entire project on social media.