Enterprise RAG data foundation
2025 · Delivered client engagement
Solidigm AI
Making engineering data trustworthy enough for AI diagnostics
As the contract product manager for a CodeLab client team, I led an eight-week engagement to turn Jira service tickets and unstructured engineering PDFs into a configurable, reviewable data pipeline for Solidigm’s RAG-based diagnostic tool.
Quick scan
TL;DR
The shortest version: what changed, what shipped, what was verified, and where the proof stops.
Overview
Problem
Solidigm’s planned diagnostic assistant depended on years of service tickets and engineering documents that were too inconsistent and unstructured to feed directly into a RAG system.
Move
Scoped separate extraction paths for Jira CSVs and PDFs, then shaped a two-view operator experience for configuration, review, and correction.
Outcome
Delivered a Dataiku-based pipeline and plugin interface; the published project write-up reports more than 99% metadata-extraction accuracy.
Scope
Contract product management across discovery, requirements, technical scoping, product direction, client alignment, and eight-week delivery.
Proof
- Delivery window
- 8 weeksThe published project article dates the engagement from April through June 2025.
- Input surface
- 2 source formatsThe pipeline handled structured Jira CSV exports and unstructured engineering PDFs.
- Extraction result
- >99% reportedThe team’s article reports the result; no public benchmark dataset was included.
- Operator workflow
- 2 core viewsSettings/Dashboard supported configuration; Extraction View supported search, review, and correction.
01 · Context
The diagnostic assistant depended on a data problem no one could skip.
Solidigm wanted an AI-powered tool that could surface quick diagnostics and suggested resolutions when customers encountered product service issues. The intended retrieval layer would draw on past tickets, fixes, and product specifications, but those sources arrived in very different forms.
Jira exports had predictable columns but inconsistent language inside each record. Engineering PDFs mixed tables, free-form technical descriptions, repeated headers, and document-specific formatting. Before retrieval could be useful, the system needed a repeatable way to extract, clean, classify, and vectorize both sources.
02 · The decision
Put human control around the automated extraction.
Configuration before processing. Inspection after processing. Correction when the pipeline is wrong.
The engagement began with ambiguity around the user-facing product. Interviews and working sessions with Solidigm’s lead engineer clarified three operator needs: control over what metadata gets extracted, visibility into extraction quality, and a fast way to review a specific ticket.
That led to a dual-view plugin. A Settings/Dashboard view let engineers manage categories and keyword lists without changing backend code. An Extraction View made past runs searchable, placed source content beside extracted metadata, and allowed manual correction.
The interface was not decoration around a backend workflow. It was the control and audit layer that made an automated pipeline maintainable as terminology, document formats, and extraction needs changed.
Engineering noteEach input format received a purpose-built extraction path.
The pipeline avoided treating every source as generic text. Structured exports and engineering PDFs were processed with different tools and heuristics before their metadata entered the same downstream workflow.
- Pandas normalized columns, dates, terminology, and merged Jira CSV datasets.
- PDFPlumber preserved surrounding free-form text and page position.
- LlamaIndex organized table content while custom heuristics removed noise and isolated relevant sections.
03 · What shipped
A configurable ingestion pipeline with a visible quality-control loop.
The team built inside Solidigm’s existing Dataiku environment, using its visual workflow, storage, inference, and plugin extension points rather than introducing a parallel platform. Custom JavaScript, HTML, and CSS supplied the operator-facing views, while an Azure virtual machine and VPN-supported access fit the work into the client’s internal environment.
The resulting system handled structured Jira service tickets and unstructured engineering documents, exposed category and keyword controls, kept a searchable extraction history, and let engineers compare raw content with extracted fields before correcting misclassifications.
Review and correct
Delivery noteThe existing platform was both an accelerator and a constraint.
Dataiku matched the client’s infrastructure and offered the workflow primitives the project needed, but its plugin model imposed a steep onboarding curve and limited interface flexibility.
- The team committed to the client’s existing Dataiku environment instead of replacing it.
- The interaction model was simplified to fit Dataiku’s plugin structure.
- Technical onboarding and product-definition work ran inside the same eight-week delivery window.
04 · Evidence
The published result is strong, with an important evidence boundary.
The CodeLab project article reports that the tuned pipeline achieved more than 99% accuracy in metadata extraction. It attributes that result to a modular combination of rule-based logic, semantic parsing, model-driven classification, and document-specific heuristics.
The public sources do not include the evaluation dataset, sample size, error taxonomy, or a reproducible benchmark. The portfolio therefore presents 99% as the team’s published project result—not as an independently revalidated claim.
No production adoption, support-resolution, or customer-outcome analytics were published. The defensible evidence is the delivered eight-week workflow, the two source formats it handled, the operator controls it exposed, the client-facing artifacts, and the recorded extraction result.
05 · Reflection
For enterprise AI, inspectability is part of the product.
The pivotal product decision was recognizing that a backend extraction pipeline still had users. Engineers needed to shape the rules, see what happened, find a past result, and correct the system without waiting for a code change.
Building inside the client’s platform also kept the work honest. The best architecture on paper would not have mattered if it ignored Solidigm’s existing tools, access model, and maintainers. Product scope had to join model quality, operator workflow, and infrastructure fit.
A future iteration should make the quality claim more legible by publishing the evaluation method: representative document strata, field-level precision and recall, common error classes, and how manual corrections feed future tuning.