Solidigm AI Engineering Data for AI Diagnostics
Service records engineers can inspect before AI uses them.
- Role
- Contract Product Manager
- Timeline
- Apr–Jun 2025 · 8 weeks
- Delivered
- Dataiku pipeline · review plugin
- Stack
- Dataiku · Pandas · LlamaIndex
I led discovery and technical scoping for a two-source data pipeline, then shaped the controls engineers use to inspect and correct its output.
- Prepare records before retrievalJira tickets and PDFs needed different paths into one diagnostic system.See the problem
- Give engineers control over metadataSettings and review let operators tune fields and correct mistakes.Explore the controls
- Fit delivery to DataikuThe production pipeline and plugin keep processing beside operator review.See what shipped
- Keep the data adjustableDataiku limited interface freedom but kept corrections beside processing.Read the trade-off
01 · Context
Useful diagnostics begin with usable service records.
When a customer encountered a product issue, Solidigm wanted its diagnostic assistant to draw on earlier service tickets, fixes, and specifications. That information was available, but a useful answer depended on preparing the records first. Jira exports supplied regular columns with inconsistent language inside them, while engineering PDFs mixed tables, technical prose, and repeated headers and footers.
Our eight-week engagement focused on turning those sources into consistent metadata for retrieval. The work called for different extraction methods, and it also called for a way to catch mistakes: engineers maintaining the system needed to see which source produced a field and change the result when the automated pass got it wrong.
02 · Feature set
Give operators control over the inputs and the result.
I worked with Solidigm's lead engineer to understand who would maintain the workflow after delivery. Interviews and working sessions narrowed the product to three operator needs: choose which metadata to extract, judge whether extraction worked, and find a particular ticket when a result needed attention.
Those needs led us to two connected views. In Settings, engineers can edit categories and keyword lists without changing backend code. In the Extraction View, they can find an earlier ticket, compare its source with the extracted fields, and correct a misclassification. As source material changes, engineers can update the rules and repair individual results themselves.
Engineering noteKeep source-specific parsing before shared review
A Jira export and an engineering PDF expose different kinds of structure. The team processed each with methods suited to its format before bringing extracted metadata into the common operator workflow.
- Pandas normalized columns, dates, terminology, and merged Jira CSV datasets.
- PDFPlumber extracted surrounding text while retaining page position.
- LlamaIndex organized tables, with custom heuristics isolating relevant sections and removing repeated noise.
03 · What shipped
Deliver the workflow inside Solidigm's existing platform.
The team built the processing workflow in Dataiku, which Solidigm already used for data work. Its workflow, storage, inference, and plugin tools let the project fit the client's environment, while custom HTML, CSS, and JavaScript supplied the two operator views.
That fit came with a real constraint. Dataiku's plugin model took time to learn and limited interface flexibility, so we concentrated the custom UI on the settings and review tasks the platform did not provide out of the box. The team delivered the workflow within eight weeks inside the environment Solidigm already operated.
Solidigm now runs the extraction workflow, review plugin, and downstream diagnostic assistant in its production stack. The published project account reports more than 99% accuracy for metadata extraction after the team combined rules, semantic parsing, and model-driven classification. It does not publish the evaluation dataset or method, so the figure remains a reported team result rather than a reproducible benchmark.
Review and correct
04 · Reflection
Keep the data foundation adjustable in production.
Jira tickets and engineering PDFs needed different parsing, but their extracted fields had to serve the same diagnostic system. Separating those paths from a common review workflow gave maintainers a place to correct the fields that feed retrieval. That division still matters in production as source material and terminology change.
Building inside Dataiku limited how far we could customize the interface, but it kept those controls beside the processing in an environment Solidigm already runs. The team used that constraint to focus on category changes and source-level correction, giving engineers a practical way to keep the diagnostic inputs current as routine categories and keywords evolve.