English原文：本指南暂无您所选语言的译本。
Short answer: Freeze one representative job, with its diagram revision, approved part subset and rule set, and run it through your current process and the new workflow under the same written acceptance criteria. Judge the result on accepted output and whole-task effort: preparation, review, correction and failed attempts. How quickly the software produces a file is close to irrelevant.
What the pilot has to answer
One question: for this equipment family, does the new workflow produce an assembly our design lead accepts, with less total engineering effort, in a way we can operate and support? A pilot that ends in "interesting, let's keep looking" has failed, however good the demo looked. Design it so that it ends in accept, adjust or stop, with stated reasons.
Designing the pilot
Pick the job
Choose a real job from the family you would roll out to. It should be typical, neither the cleanest example nor the strangest one. For MST that means semiconductor gas equipment in SOLIDWORKS, where the workflow is P&ID gas-line diagram + approved PDM parts + equipment rules -> native, editable SOLIDWORKS assembly, starting with semiconductor gas equipment. A gas box with several sticks, purge and vent branches and a normal mix of welded and face-seal joints is a reasonable candidate.
Freeze the inputs
Record the diagram revision, the exact list of approved parts with revisions, and the version of the rule document. Both paths use the same set. If someone fixes a part mid-pilot, note it and decide whether the baseline needs rerunning too. Otherwise you end up comparing two different jobs.
Write acceptance criteria first
Before anyone sees output, the design lead writes down what "accepted" means for this job:
- Every diagram connection exists physically, with the correct flow direction through directional components.
- Joint types follow the rules: welded where specified, face seals at service breaks.
- Tool and wrench access at every VCR nut and fitting that may be serviced.
- No interference, and clearance to enclosure walls and keep-outs.
- Mates are stable on rebuild and the assembly isn't over-defined.
- BOM matches the diagram and uses released part numbers at the correct revisions.
- Assembly configurations, if used, are named and behave as your conventions require.
Name the people
Design lead for acceptance, PDM administrator for the subset and access, IT for the data path, engineering manager for the decision. One person keeps the log.
Log everything
Record preparation, each run, failed runs and why they failed, review, and each correction with its cause. Keep rejected work in the record. MST's product page frames this as comparing the cost of an accepted task, including the work people still have to do ([product page](https://mst-us.ai/product/)).
The metrics, qualitatively
Set no target numbers in advance. Results depend on your scope, library and baseline, and a pre-set target invites people to game it. Compare each of these against your own current process.
Acceptance and quality
- Acceptance outcome: accepted, accepted with edits, or rejected, with the reason.
- Defect classes: wrong part, wrong orientation, rule violation, routing conflict, access problem, mate instability, BOM mismatch.
- Visibility of problems: were rule violations and gaps reported by the workflow, or found by the reviewer?
Whole-task effort
- Preparation: vault cleanup, rule writing and input checks, separating one-time work from per-job work.
- Review: how much effort it took the design lead to reach a confident decision.
- Correction: small edits such as rotating a valve or moving a run, versus rebuilding a section.
- Failed attempts: runs that produced nothing usable, and the cause of each.
Traceability
- Parts: can every component be traced to a part number and approved revision?
- Connections: can every diagram connection be traced to a physical joint and route?
- Unresolved items: were missing parts or ambiguous connections listed clearly?
Usability of the output
- Editability: can engineers keep working on the native assembly with normal SOLIDWORKS edits without mates exploding?
- Downstream fit: do drawings, BOM export and PDM check-in work as usual? A quick check is to export both BOMs to CSV and run them through the free [BOM Compare](https://mst-us.ai/bom-compare-guide/) tool, which lists added, removed and changed lines by part number.
Operability
- Library readiness: how much of the subset was usable as delivered?
- Rule maintenance: who owns rule changes, and how are they controlled?
- Data path: did the deployment model satisfy IT?
Classifying corrections
The most useful pilot artifact is a correction log in which every entry has one cause:
| Cause | Example | What it tells you |
|---|---|---|
| Missing rule | Valve rotated for tubing access | Write the rule down |
| Library data | Wrong mate reference on a fitting | Fix the part in PDM |
| Ambiguous input | Purge tie-in side not shown | Fix diagram conventions |
| Output error | Route meets the rules but is clumsy | A genuine tool limitation |
| Reviewer preference | Style choice, not a rule | Decide: rule or ignore |
If most corrections fall in the first rows (rules, library, inputs), the inputs are the issue and they are fixable. If most fall in the output-error row, that tells you something about fit.
Scenario: reading a pilot result
A team runs a gas-box job both ways. The generated assembly is accepted with edits. The log shows that most corrections came from an unwritten access rule and from a pair of fittings whose mate references were inconsistent in the vault. The workflow flagged one diagram connection as ambiguous, and the reviewer agreed it was. Review concentrated on routing near a crowded stick rather than on re-mating. The manual baseline needed less preparation but much more modeling, and its own review caught a BOM leftover from the copied job.
The decision: adjust. Add the rule, fix the parts, rerun, then run another job from the family. That is a good pilot outcome even before anyone mentions savings, because every open issue has an owner and a fix.
Keep two questions apart
Did the task take less total effort? and Did that change a delivery date, overtime or outsourcing? are different questions. Released capacity only becomes a financial result if the team uses it. MST's [Q32](https://mst-us.ai/questions/engineering-manager-ai-assembly-adoption-roi/) covers this management view.
Pilot checklist
| Item | Done |
|---|---|
| Typical job from the target family | |
| Inputs frozen: diagram rev, parts list, rule version | |
| Acceptance criteria written before output | |
| Owners named | |
| Log covering preparation, runs, failures, review, corrections | |
| Same job run through the current process | |
| Corrections classified by cause | |
| Decision meeting held: accept, adjust or stop |
Tradeoffs
- One job is narrow. It can tell you whether the approach works for this family. It can't tell you everything about rollout.
- Baselines are messy. Engineers remember the manual path as faster than it was. Log it as carefully as the new path.
- First-run preparation inflates effort. Separate one-time cleanup from per-job work, or the pilot will understate the steady state and overstate the first job.
If your team designs semiconductor gas equipment in SOLIDWORKS and wants to scope a pilot job, [contact MST](https://mst-us.ai/contact/) with a non-confidential description.
FAQ
Why no targets?
Because results depend on your scope and baseline. Compare against your own process.
Should we time generation?
Record it, but preparation, review and correction dominate.
How many jobs?
Start with one well-defined job. Add more from the same family once the first result is clear.
What if it needs corrections?
Expect that. Classify each correction by cause and see whether the counts move as inputs improve.
Who decides?
The engineering manager, based on the design lead's acceptance and the effort comparison.