K12Online Knowledge Toolkit: Data Pipeline & RAG Profiler

"A high-speed data cleaning machine that gathers messy web articles, sweeps away broken HTML code, and turns them into organized files ready for AI to learn from."

1. The Problem (Why I Built This)

Raw help center articles are too messy for AI search engines:

• Noisy Markup: Articles contain dynamic JavaScript buttons, broken tables, and tracking pixels that confuse AI language models.
• Lack of Structure: Missing titles, undefined categories, and inconsistent formatting hurt search accuracy.

2. How I Solved It (Under The Hood)

I engineered an automated data extraction and cleaning pipeline:

• High-Throughput Scraping: Automated extraction across 380+ support articles using asynchronous Python workers.
• Content Sanitization: Cleaned noisy HTML into structured JSON objects containing title, breadcrumbs, tags, and clean markdown text.
• Data Profiler: Checked document completeness, token lengths, and keyword frequency before feeding into AI knowledge bases.

3. Key Takeaway & Results

• Efficiency: Processed 380+ platform articles in less than 3 seconds.
• Data Quality: 100% structured data consistency with zero corrupted entries.