1,060 conversations and 23,465 messages — trapped in JSON exports until I built a semantic search layer over all of them.
If you've been using ChatGPT or Claude seriously for more than six months, you have a knowledge management problem you probably haven't noticed yet.
Every conversation you've had — the debugging sessions, the architecture discussions, the research deep dives, the creative brainstorms — is trapped in a linear chat history. You can scroll through it. You can search by keyword if the platform supports it. But you can't search by meaning. You can't ask "what did I learn about connection pooling?" and get answers from across dozens of conversations spanning months.
I had 1,060 conversations and 23,465 messages across ChatGPT and Claude when I realized: this is one of the most valuable datasets I own, and I have no way to use it.
So I built Discotheque.
What Discotheque Does
Discotheque is a standalone Flask service that imports conversation exports from ChatGPT and Claude, embeds every message with vector embeddings, and provides semantic search across the entire archive.
The core idea is simple: every conversation you've had with an AI assistant is a knowledge artifact. It contains decisions you've made, problems you've solved, patterns you've discovered, and context you've forgotten. Making that knowledge searchable by meaning — not just by keyword — transforms a linear chat history into a queryable knowledge base.
The stack: PostgreSQL with pgvector for vector storage, OpenRouter's text-embedding-3-small model for 1,536-dimensional embeddings, SQLAlchemy for the ORM layer, and an HTMX-powered web UI for search. It runs on port 5016 alongside the other 12 services in my ecosystem.
The Import Pipeline
ChatGPT and Claude both allow you to export your conversation history as JSON. The structure is different between platforms, but the core data is the same: conversations contain messages, messages have roles (user/assistant), and messages have content.
Discotheque's import pipeline handles both formats: parse the JSON export, normalize the structure into a common schema (conversations table, messages table), deduplicate using a composite key of source platform and source ID, and store everything in PostgreSQL.
The deduplication matters more than you'd think. ChatGPT exports include your entire history every time you export. Without dedup on (source, source_id), you'd double your data with every import. The composite unique constraint catches this automatically — if you re-import the same export, zero new records get created.
Semantic Search Over Conversations
Once messages are imported, the embedding pipeline runs across the corpus. Each message gets embedded individually, producing a 1,536-dimensional vector that captures its semantic meaning.
The embedding process for the full corpus — 23,465 messages — takes about 25.8 minutes. The key lesson was input truncation. OpenRouter's text-embedding-3-small has an 8,191-token limit, but tokens aren't characters. Code and JSON content compress at roughly 2-3 characters per token, meaning a ChatGPT message full of code hits the token limit much sooner than plain prose.
My first attempt truncated at 30,000 characters. This worked for prose but failed consistently on technical conversations. Dropping to 8,000 characters eliminated all embedding errors with negligible quality loss.
With embeddings in place, search is a pgvector nearest-neighbor query. Type a question in natural language, and the system returns the most semantically similar messages from across your entire conversation history, ranked by cosine similarity.
Why This Matters More Than You Think
Rediscovering forgotten solutions. I've solved the same problem more than once because I forgot I'd already worked through it. Semantic search catches this.
Cross-pollinating ideas. Conversations exist in silos. Semantic search connects them. A query about "service discovery patterns" might return messages from conversations about Kubernetes, knowledge graph architecture, and microservices from six months ago.
Building on past context. When starting a new AI session, I query Discotheque for relevant past conversations and feed the highlights into the new session's context.
Institutional memory for solo operators. When you're a team of one, there's no colleague to ask "didn't we try this before?" Discotheque is that colleague.
Integration With the Broader Ecosystem
Discotheque doesn't exist in isolation. As part of the 13-service ecosystem connected through Memory Archive's knowledge graph, it enriches the entire system.
Career Bot can search conversation history for context when answering questions about my work. Doc-Steward can reference past discussions when evaluating documentation decisions. The Blog App can surface related conversations when developing new content.
Each conversation I have adds to the searchable context that all services can draw from.
Patterns Anyone Can Adapt
You don't need 13 services or a knowledge graph to get value from this approach:
- Export your conversations. Both ChatGPT and Claude support full history exports.
- Store them in a database with vector support. PostgreSQL with pgvector is free and production-ready.
- Embed every message. Budget roughly $2-5 for a corpus of 20K+ messages.
- Build a simple search UI. An HTMX page with a search box and results list is enough.
- Truncate inputs to 8K characters. Trust me on this one.
The entire system is a single Flask application, a PostgreSQL database you might already have, and an embedding API call. The value is in the data, not the complexity.
What This Means for Knowledge Management
We're generating more knowledge through AI conversations than through any other medium, and almost none of it is being captured systematically.
For individuals, the solution is something like Discotheque: embed your history, make it searchable, start building on what you've already learned.
For organizations, the implications are bigger. Teams using AI assistants daily are generating a corpus of problem-solving knowledge that currently lives in individual chat histories. Making that knowledge searchable, shareable, and persistent is a knowledge management opportunity that most organizations haven't recognized yet.
The AI conversation isn't just the interaction. It's the artifact. Treat it that way.
See what I've been up to: coreyscherrer.com
See what fun I've been up to: coreyiscorey.com
Disclaimer: These articles were drafted with AI assistance (Claude) and reviewed by a human. All projects, systems, and technical details described are real — sourced directly from production sessions captured in a PostgreSQL database. Questions? I'd love to talk shop — reach out anytime.