AllSpiceMirrors/OrthoRoute
0
Fork 0
mirror of https://github.com/bbenchoff/OrthoRoute.git synced 2026-08-24 14:47:01 +00:00
Files Issues Projects Releases
Files
main
OrthoRoute/docs/optimization/optimization_baseline_2026-04-05.md
OpenFixture Developer e836eb1eaf Add automated optimization workflow with smoke test validation 
- Create scripts/analyze_log.py: Standalone log parser for routing metrics extraction and golden comparison
- Create scripts/optimize_and_validate.ps1: PowerShell automation wrapper for full optimization cycle
- Create scripts/README.md: Comprehensive documentation for automation scripts
- Create tests/regression/smoke_metrics.json: Golden thresholds for smoke test (100 nets, 4 layers)
- Create docs/optimization/optimization_workflow.md: Complete optimization workflow guide (800+ lines)
- Create docs/optimization/baseline_template.md: Standardized template for future optimization baselines
- Extend launch_kicad_debug.ps1: Add -RunValidation flag for post-KiCad smoke test
- Extend copy_to_kicad.ps1: Add -Validate switch for deployment validation
- Update optimization docs: Add workflow references to QUICK_REF, README, golden_result_2026-04-10
- Update historical baselines: Add workflow tool references to 2026-04-03, 04-05, 04-08

Workflow enables fast iterative optimization: make change → sync → smoke test (30s) → analyze → commit
Exit codes: 0=PASS, 1=FAIL routing, 2=WARN regression, 3=ERROR environment
Smoke test becomes primary validation checkpoint (middle complexity: 100 nets vs 512 backplane)
2026-04-12 20:10:06 +02:00

228 lines
10 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# OrthoRoute Performance Baseline — April 5, 2026
**Date**: April 5, 2026
**Version**: OrthoRoute (post-regression-suite branch `feature/profiler-optimization`)
**Test Board**: TestBackplane.kicad_pcb
**Purpose**: Capture current routing performance after persistent-kernel and algorithm improvements since April 3 baseline
> **Update (April 12, 2026):** Automated optimization workflow now available!
> See [optimization_workflow.md](optimization_workflow.md) for complete guide.
> Use `.\scripts\optimize_and_validate.ps1` for automated testing and validation.
---
## Executive Summary
Successfully routed a **18-layer backplane** with **512 nets**, **1,604 pads** in **11.96 minutes (717.5s)** — a **4× improvement over the April 3 baseline (47.9 min)** and **2× improvement over the April 3 persistent-kernel run (25 min)**.
**Run history (all runs, both dates):**
| Run | Date | Change | Total Time | Iter avg | Iters | Tracks | Vias | Notes |
|-----|------|--------|------------|----------|-------|--------|------|-------|
| 1 | Apr 3 | Baseline — multi-launch kernel | 47.9 min | ~39s | 74 | 4,118 | 2,585 | 512/512 ✓ |
| 2 | Apr 3 | Persistent CUDA kernel enabled | 25 min | ~20s | 70 | 4,118 | 2,585 | **2× gain** |
| 3 | Apr 3 | `cupyx.scatter_add` bitmap fix | ~25 min | ~22s | — | — | — | Bug fix; no perf gain |
| 4 | Apr 3 | GPU-resident `node_owner_gpu` | ~25 min | ~22s | — | — | — | No measurable improvement |
| **5** | **Apr 5** | **Current state** | **11.96 min** | **11.0s** | **65** | **4,290** | **2,754** | **512/512 ✓ — 4× baseline** |
**Primary gain between Apr 3 Run 2 and today**: iteration count reduced (70 → 65) and per-iter time halved (~20s → 11s), together yielding the 2× improvement over the already-optimized Apr 3 run.
---
## Test Board Characteristics
| Property | Value |
|----------|-------|
| Copper layers | 18 |
| Pads | 1,604 |
| Routable nets | 512 |
| Total nets | 1,088 |
| Components | 12 |
| Board size | 73.1 × 97.3 mm |
| Lattice nodes | 446,472 |
| Lattice edges | 14,281,664 |
---
## Routing Run — Measured Results (April 5, 2026)
**Hardware**: NVIDIA T1200 Laptop GPU, 4 GB VRAM (3.0 GB free), Turing architecture (Compute Capability 7.5)
Two routing runs completed April 5 — the second (`run_20260405_223659.log`) is fully retained and used as the primary reference.
| Metric | Run 1 (golden ref) | Run 2 (confirmed log) |
|--------|-------------------|----------------------|
| **Log** | `run_20260405_211914.log` (gone) | `run_20260405_223659.log` ✓ |
| **Kernel mode** | unknown | **PERSISTENT kernel** confirmed |
| **Success rate** | 512/512 (100%) | 512/512 (100%) |
| **Total time** | 717.5s (11.96 min) | 761.2s (12.69 min) |
| **Iterations** | 65 | 67 |
| **Iter 1 time** | 29.2s | 29.9s |
| **Iter avg time** | ~11.0s | ~11.4s |
| **Iter min** | — | 4.0s (iter 67) |
| **Final overuse** | 0 | 0 |
| **Barrel conflicts** | 444 | **379** (`[FINAL]` line) |
| **Tracks written** | 4,290 | 4,287 |
| **Vias written** | 2,754 | 2,751 |
Both runs are within 6% of each other — consistent, reproducible performance.
### Convergence trend — Run 2 (`run_20260405_223659.log`)
| Iter | Nets active | Overuse (edges) | Barrel | Iter time | Total | Notes |
|------|-------------|-----------------|--------|-----------|-------|-------|
| 1 | 512/512 | 3,931 | 2,564 | 29.9s | 29.9s | `pres_fac=1.00` — first full pass; CUDA warm-up |
| 2 | 512/512 | 2,584 | 2,236 | 9.4s | 40.3s | persistent kernel active from here |
| 3 | 512/512 | 2,331 | 2,094 | 4.6s | 45.7s | |
| 4 | 512/512 | 2,221 | 1,937 | 9.2s | 55.8s | |
| 5 | 512/512 | 1,885 | 1,665 | 10.0s | 66.6s | |
| 6 | 512/512 | 2,071 | 1,649 | 8.3s | 81.8s | overuse oscillates — normal |
| 10 | **504/512** | 1,778 | 832 | 8.9s | 121.7s | `pres_fac=5.20` — 8 nets temporarily excluded |
| 20 | **506/512** | 2,083 | 526 | 9.1s | 236.7s | `pres_fac=20.48` |
| 30 | **504/512** | 1,967 | 539 | 8.3s | 353.2s | `pres_fac=64.00` (capped) |
| 40 | **508/512** | 1,450 | 570 | 9.7s | 476.2s | congestion penalty now dominant |
| 50 | **501/512** | 1,866 | 522 | 9.5s | 596.3s | nets oscillate in/out of exclusion |
| 60 | 512/512 | 144 | 379 | 5.6s | 706.1s | all nets active, near zero overuse |
| 61 | 512/512 | 242 | 379 | 4.6s | 719.8s | slight overuse bounce — normal |
| 65 | 512/512 | 107 | 379 | 5.5s | 743.8s | |
| 66 | 512/512 | 69 | 379 | 4.6s | 755.9s | |
| **67** | **512/512** | **0** ✓ | 379 | 4.0s | **761.2s** | `[CLEAN]` + `ROUTING COMPLETE` |
**Key observations:**
- Iter 1 is slow (29.9s) — CUDA kernel JIT compile + first-net graph setup
- Iters 2–59 settle to ~8–10s avg driven by the persistent kernel
- Final 7 iters (61–67) drop to 4–5s as only a handful of overuse edges remain
- Barrel conflicts fall sharply (2,564 → 379 by iter 10) then plateau — 379 is the acceptable floor
- Net exclusion/retry (iters 10/20/30/40/50) is normal PathFinder behaviour: difficult nets are temporarily removed from routing, retried, and all 512 succeed by iter 60
- `pres_fac` caps at 64.0 by iter 30; convergence after that is purely re-routing with maximum congestion penalty
---
### Board state after routing (golden_board.json)
| Field | Post-routing value |
|-------|--------------------|
| `tracks_existing` | 9,605 |
| `vias_existing` | 6,021 |
| `pads` | 1,604 |
| `copper_layers` | 18 |
---
## Performance Comparison vs. April 3 Baseline
| Metric | Apr 3 Run 1 (baseline) | Apr 3 Run 2 (best) | **Apr 5** | vs. baseline | vs. Apr 3 best |
|--------|------------------------|---------------------|-----------|--------------|----------------|
| Total time | 47.9 min | 25 min | **11.96 min** | **−75%** | **−52%** |
| Iterations | 74 | 70 | **65** | −12% | −7% |
| Iter avg | ~39s | ~20s | **11.0s** | −72% | −45% |
| Iter 1 | — | — | 29.2s | — | — |
| Tracks | 4,118 | 4,118 | **4,290** | +4% | +4% |
| Vias | 2,585 | 2,585 | **2,754** | +7% | +7% |
| Success rate | 100% | 100% | **100%** | = | = |
| Overuse | 0 | 0 | **0** | = | = |
---
## Active Thresholds (golden_metrics.json — GPU block)
These are the soft-warning thresholds used by the regression test suite (measured × 1.25 headroom):
| Threshold | Value | Basis |
|-----------|-------|-------|
| `total_time_s_max` | 900s | 717.5s × 1.25 |
| `iter_avg_time_s_max` | 15.0s | 11.0s × 1.36 |
| `iter_1_time_s_max` | 40.0s | 29.2s × 1.37 |
| `iterations_max` | 80 | 65 × 1.23 |
| `tracks_delta_min` | 4,000 | floor below 4,290 |
| `vias_delta_min` | 2,500 | floor below 2,754 |
| `overuse_final_max` | 0 | exact |
| `converged` | true | exact |
---
## Regression Test Status (April 5, 2026)
Suite: `tests/regression/` + `tests/unit/`
Runner: `python -m pytest tests/ -v`
| Category | Passing | Skipping | Failing |
|----------|---------|----------|---------|
| Unit tests | 26 | 0 | 0 |
| Regression — board load | 6 | 0 | 0 |
| Regression — log health | 2 | 0 | 0 |
| Regression — GPU/iter soft checks | 4 | 0 | 0 |
| Regression — lattice size | 4 | 0 | 0 |
| Regression — routing quality | 17 | 0 | 0 |
| Regression — write-back | 2 | 0 | 0 |
| Regression — routing quality | 0 | 23 | 0 |
| Regression — write-back | 0 | 6 | 0 |
| **Total** | **36** | **29** | **0** |
**Why 29 still skip**: Routing quality and write-back tests require `routing_result` or `log_routing_result` populated from a live routing run. The April 5 log was overwritten by subsequent KiCad launches before tests could consume it. Next full routing run with `ORTHO_DEBUG=1` will unlock these.
---
## Known Bottlenecks — Updated Status
| Priority | Target | Apr 3 Status | Apr 5 Status |
|----------|--------|-------------|-------------|
| ✅ DONE | MULTI-LAUNCH Python→CUDA overhead | Fixed (persistent kernel compiled, not triggered) | **Appears resolved** — 11s/iter vs 20s/iter |
| ✅ DONE | `int(seed)` bitmap loop correctness | Bug fixed Apr 3 | Stable |
| ✅ DONE | GPU-resident `node_owner_gpu` | No gain measured Apr 3 | Deployed |
| 🟡 #1 | Unknown per-net gap (~70–100ms) | Not identified | Partially resolved — halved iter time |
| 🟡 #2 | `initialize_graph()` — 20.9s one-time | Identified | Not measured this run |
| 🟢 #3 | `_rebuild_via_usage_from_committed` | ~1% runtime — deferred | Still deferred |
**Observation**: The 11s/iter average (down from 20s in Apr 3 Run 2 and 39s in baseline) strongly suggests the persistent kernel or equivalent optimization is now active. The "unknown 70–100ms gap" from April 3 has been substantially reduced. Exact cause unknown — the routing log was not retained.
---
## Next Optimization Targets
1. **Profile the Apr 5 improvement** — run with `ORTHO_DEBUG=1`, retain the log, identify what changed vs Apr 3. Compare `[GPU-SEEDS]` lines: is it now using PERSISTENT KERNEL?
2. **Unlock routing quality regression tests** — retain the next routing log before launching KiCad again. The 29 skipping tests require a log with `[ITER N]` lines.
3. **`initialize_graph()` profiling** — at 11 min total, the 20.9s init is now ~3% of runtime. Still worth a 30–40% speedup for future larger boards.
4. **CPU-only baseline** — run `python main.py cli TestBoards/TestBackplane.kicad_pcb --cpu-only` to populate the `cpu` block in `golden_metrics.json`.
---
## System Health
| Check | Result |
|-------|--------|
| Errors in log | 0 |
| Exceptions | 0 |
| Keepout violations | 0 |
| Routing success | 512/512 (100%) |
| Final overuse | 0 |
**Conclusion**: System is stable. Routing quality and performance both improved over the April 3 baseline. Ready for further optimization and regression test expansion.
---
## References
### Related Documents
- [optimization_baseline_2026-04-03.md](optimization_baseline_2026-04-03.md) — Previous baseline (identifies MULTI-LAUNCH root cause)
- [OPTIMIZATION_QUICK_REF.md](OPTIMIZATION_QUICK_REF.md) — Priority table
- [CURRENT_LOGGING_REVIEW.md](CURRENT_LOGGING_REVIEW.md) — Logging analysis
### Key Files
- **Golden metrics**: `tests/regression/golden_metrics.json` — GPU thresholds from this run
- **Golden board**: `tests/regression/golden_board.json` — Post-routing board signature
- **Main router**: `orthoroute/algorithms/manhattan/unified_pathfinder.py`
- **GPU kernels**: `orthoroute/algorithms/manhattan/pathfinder/cuda_dijkstra.py`
### Test Board
- `TestBoards/TestBackplane.kicad_pcb` — 18-layer backplane, 1,604 pads, 512 nets
---
**Document Version**: 1.0
**Date**: April 5, 2026
**Status**: Baseline Established — 4× improvement over April 3
Reference in New Issue View Git Blame Copy Permalink