Roadmap¶
What we build toward and what we have decided never to build. Ordered work is in GitHub issues; this is the index and the rationale. Everything is judged against the ceiling in docs/design/ceiling.md: affine ∩ relational ∩ local.
How we measure¶
Not "can we say what other tools say?" — that has no finite answer — but "what
is sayable in a language that must stay streamable?" A primitive is done when
lowering.py accepts it and the differential test passes.
What we publish is coverage against two corpora, generated by a test rather
than by hand (#27): Calliope's base math plus its published custom-math library,
and PyPSA's optimization/constraints.py. Every item scores expressible,
out of scope by design (data prep — SPEC §11), or
genuinely arbitrary Python (an escape island, or stays a callback). No
numbers are published until #27 produces them.
Expressiveness is not the axis we lead on. Cost is, on the one claim a reader can check rather than take: end-to-end cost to a loaded solver, wall and peak, against the eager lane's own best path to the same place (benchmarks). Build memory alone is not that claim — it is a small fraction of what solving costs, so shrinking it further changes nothing a caller feels. Two claims we would rather lead on are not measured yet and are therefore not made: the floor under the LP-file route as a cold cost, and marginal cost per model in a loop. Both are listed in Not measured yet; neither is quoted until a table backs it.
The degree axis¶
Of the three admissibility rules, degree is the only scope choice; relational
and local fall out of the architecture. The plan is affine-by-design, quadratic
out of scope — a decision with a revisit condition, not an axiom. It holds
because demand is almost entirely quadratic objectives, and for convex 1-D
curves we ship the substitutes practitioners prefer (piecewise: with
convex: true, and the epigraph pattern in plain YAML) which keep LP duals,
warm starts and MILP compatibility that a quadratic objective gives up.
Quadratic constraints are a class HiGHS cannot solve at all, and every sink
we ship assumes affine rows.
What would have to hold to move it. The old list said "sinks carry a
quadratic stream", and that condition is already met — HiGHS passHessian and
Gurobi setMObjective both exist, and linopy's QuadraticExpression means the
oracle covers it. What remains is one blocker and one cost:
- MIQP is excluded on the default solver — the blocker. HiGHS returns
kErrorforHessian + integrality, andbinary:/integer:and nonconvexpiecewise:are already in the language, so quadratic would conflict with shipped features. Gurobi has no such exclusion, which makes this a capability question: quadratic is the motivating case for Track 4's conjunction-exclusion axis, not an exception to it. - The Hessian is passed whole — a cost, not a rule-4 violation, since
under the aligned-only scope
Qis diagonal and small beside the solver's own model. The arithmetic and its two caveats are in docs/benchmarks.md.
Plus a data-time convexity guard, a new plan node, a lowering case and a
label-ordering change — a bigger diff than SOS, which is a declaration and needs
none of it. The question that actually decides it is whether the demand is
LP-only: if the fuel-curve users also want unit-commitment binaries, (1) kills
it on the default path however cheap the plumbing is, and
piecewise: {convex: true} stays the answer.
An escape cannot lift degree either way: islands return affine COO rows. Scope if it lands, fixed in advance: degree ≤ 2, factors coordinate-aligned, objective only, convex-guarded — aligned products are a pointwise self-join; general bilinear coupling is a cross join with |terms|² rows and is excluded permanently.
Sequencing, not a verdict. The cost side is small and the oracle is free —
linopy's QuadraticExpression builds the comparison, which is normally the
expensive half of a new primitive. What is unsettled is not whether quadratic is
worth having but where it can land: HiGHS refuses Hessian + integrality,
and binary:, integer: and nonconvex piecewise: all ship today, so on the
default path quadratic conflicts with features already in the language. That
makes two things prerequisites rather than quadratic work — Track 4 declaring
the conjunction exclusion, and a second solver bound to solver_direct that has
no such exclusion (Gurobi, #106).
Landing the primitive first ships math the default solver refuses.
The live argument — in particular whether coordinate-aligned is the right restriction, or whether the real axis is is the coupling declared as data? — is in #261 and #84, which supersede the two paragraphs above. This section states the sequencing; those state the math.
Track 1 — language primitives (taxed)¶
| # | Primitive | Proposed surface | Unlocks | Locality | Status |
|---|---|---|---|---|---|
| 1 | Indexed access, constant | at(x, over=dim, index=value) |
boundary conditions, multi-period linking, carrier ratios | pointwise | planned |
| 2 | Indexed access, lookup | at(x, over=dim, index=map) |
DC power flow, hydro cascades — topology math beyond nodal balance | pointwise | planned |
| 3 | Coordinate helpers in where |
index(dim, i), negative i from the end |
"first/last snapshot" without hardcoding a label | coordinate-space | blocks 4 |
| 4 | Window sum | sum_next_n(x, over=dim, n=N) |
rolling horizons, DSR, reserves, min up/down time | bounded-halo | needs 3 |
| 5 | Component-aware where |
defined(v) over variables |
masks compose across components instead of being re-derived and drifting | pointwise | needs design |
| 6 | all-reduction in where |
all(x, over=dim) |
reducing a mask over an out-of-foreach dim at all — today that is a DimensionError, since an implicit reduction would silently widen the mask |
pointwise | small |
| 7 | Bounds as expressions | — | signs, bidirectional flows, component libraries | pointwise | #31 |
| 8 | Namespacing / qualified names | — | component libraries | n/a | #29 |
| 9 | Compose-then-build (schema merge) | — | one lower/stream pass over merged libraries | n/a | #30 |
| 10 | Cumulative sum, data only | cumsum(p, over=dim) over a variable-free expression |
investment schedules, cumulative budgets and caps — today a data-prep round trip for one column | coordinate-space | small |
| 11 | Semi-continuous variables | semi_continuous: true beside binary/integer |
"off, or between min-stable and max" without the big-M pair it is written as today | n/a | small |
Items 1–4 are one piece of work — indexed access plus the masks that make it
usable — and are the difference between "dispatch and balances" and "network and
multi-period math". Item 2 is the adjoint of group_sum: same mapping table,
join without the aggregate. Calliope spells 1 and 2 as x[dim=value]; we do not,
because a dimension in a kwarg key cannot be parameterised by a macro.
The window family is three questions, not one. They are worth separating because only the third is expensive, and the row in Deliberate non-primitives used to price all three as if they were the third:
diffisx - shift(x, over=d, by=1). It needs nothing — it is a macro today, andedge=already answers what happens at the boundary.- Fixed-width windows (item 4) are bounded-halo, the same locality class as
shift, whose cyclic form already ships. Admissible, and gated only on item 3. cumsumsplits on what it reduces. Over a variable-free expression it is one data column with no term expansion — free, and item 10 above. Over a variable the window is t rather than a constant, so row t carries t terms: an 8760-snapshot horizon emits ~38M nonzeros where the state-variable recurrence emits ~26k.
That last number is the whole argument, and it is not a build cost. Our own
throughput is ours to fix and must never gate what the language can say. But a
construct that inflates the emitted model is paid by the solver, on every
solve, and no engineering on this side reduces it later. So cumsum over a
variable is not refused on locality grounds — it is priced, and the open
question is the mechanism: the label budget of
#38 is a cap on rows and columns
declared before the work runs, which is exactly the shape needed here, and
generalising it from escape islands to any superlinear construct would make the
size visible at load time rather than discovered in the solver. Deciding that
also decides how much job is left for an island at all.
Item 11 is the cheapest thing on this table and the only one blocked by nothing:
HiGHS carries kSemiContinuous and kSemiInteger natively, so unlike SOS
(#23) and quadratic it needs no Track 4 finding on the default sink; linopy has
Model.semi_continuous, so the oracle already covers it; and it adds a
declaration flag, not a plan node — no expression grammar, no lowering case
beyond the column type. The LP-file sink needs its own bound section, which is
where the work actually is.
Tracks 2–3 — untaxed¶
Nothing here touches the build path or either lane's semantics, with one flagged exception noted under 2b.
- 2 — post-solve stage. Declared derived results (LCOE, curtailment, emissions by group) are SQL over solution tables we already materialise. Sequenced after Track 1 items 1–4.
- 2b — the operational surface. Mostly queries, and the work that decides
whether the engine is usable at 3am: algebraic rendering of a row, IIS
read-back where a solver provides one, model statistics and coefficient-range
diagnostics. Dual read-back has shipped —
sol.dual(constraint), the same label join asprimalagainst the row table, refusing rather than zero-filling where a model has no dual solution; reduced costs and slacks ride the same join and have not (#78). The headline item is elastic relaxation, the infeasibility answer — HiGHS has no IIS, sosolver_directreturns a status code for a model too large to open in an editor; slack variables with penalty costs, then a query of nonzero slacks grouped by constraint block, needs no solver feature, works on every sink, and points at which constraints conflict rather than at a minimal set. It is the exception on this track: adding slacks means new variables, so it needs a schema-level expansion pass the waypiecewise:does, and is taxed like a primitive rather than free (#80).
The REPL gap is verbs, not structure. Anyone arriving from linopy loses
the ability to hold a model in a session and poke at it, and that reads as the
larger loss — but the substrate here is better, not worse: a built model is
four polars frames, which answers "what is in row 12" with a filter, where a
labelled Dataset answers it with a scatter. What is missing is that nothing
on the built object reads them back. lps.build() already hands back the live
executor; it just has nothing readable on it. Every verb worth having is a
query over frames that already exist — render a bound row with its labels
and coefficients (typeset/ renders the declaration, which is the other
half), preview which rows survive a where before solving, evaluate an
expression against bound data, coefficient ranges and model statistics. All of
it is read-only: it inspects a built model and never declares one, which
is the line that keeps this from becoming the Python modeling API rule 6
refuses. That line is what makes the whole block cheap, and it should be
stated in the first issue rather than defended in each one.
- 2c — value-only re-solve. In scope, because var_label is the solver
column index with no remapping: changing a bound or an RHS is a label query
plus changeColsBounds/changeRowsBounds, which is what rolling horizon,
sweeps and Benders need. With one precondition: "value-only" has to be a
property of the model, not of the user's intent. var_label is a
ROW_NUMBER() over the rows that survive the where mask, so a parameter read
by a where changes which rows exist and every later label shifts — an
in-place bound update then writes to the wrong columns. A changed group_sum
mapping leaves the labels alone but rewrites A, which a bounds-only update
cannot express at all. Both are decidable off the resolved AST, and that check
ships with the feature
(#82). Structural editing
stays out — it invalidates the label contract. Related gap: warm starts, which
solver_direct would need to set through highspy.
Integrality is label-preserving too, and rides the same contract for free.
Relaxing a MILP is one call in linopy (Variable.relax()); here it is a second
YAML file or an edit, which is why "is my model even feasible as an LP" is
harder than it should be. But var_label is a ROW_NUMBER() over the rows
surviving the where mask, and integrality does not decide which rows exist —
so changeColsIntegrality is exactly as safe as changeColsBounds, with none
of the caveats above about where-read parameters or a changed group_sum.
It belongs on the solver session (#204)
as a sibling of value rebinding, not in the language: relax-and-fix, rounding
heuristics and LP bounds on a MILP are all how you solve a model, not what
the model says.
- 3 — AST consumers (#21): math → LaTeX (a tree walk, no data — the
product is the declarative math, so it should render as typeset
documentation), CLI check/solve/write (#35), observability (#34).
Track 4 — sink capabilities¶
Decision: the ceiling and sink capability are two axes, not one. The closure (affine ∩ relational ∩ local) is about streamability and is solver-independent; what a sink can ingest is separate. Treating them as one let HiGHS's limits read as architectural law — the measured table is in docs/benchmarks.md.
Adopted: a declared capability per sink (not per solver — lp_file is not
a solver but has capabilities), modelled on linopy's SolverFeature /
Solver.features, with two divergences. Entries are three-valued
(native / reformulated / absent) so satisfying a capability by
reformulation later is additive; and the model expresses conjunction
exclusions, because linopy declares HiGHS with INTEGER_VARIABLES and
QUADRATIC_OBJECTIVE in one flat frozenset while HiGHS refuses the pair.
check(model, sink=...) takes the sink optionally: bare check asks
whether the model is sayable and lowerable, the argument asks whether that sink
will take it. Quadratic is in scope as the case that forces the exclusion axis;
SOS goes first on effort, not principle.
Hard rule 3 is untouched — it governs lanes accepting the same language, and naming an alternative sink is not the lane redirection it forbids.
Full design, work breakdown and open questions:
#89. First spike is SOS2 on
lp_file: no new dependency, no license question, no memory risk.
Track 5 — the memory axis¶
The engine holds the model it builds, so peak tracks the model rather than a number the caller sets. That is the right default — it is also what makes the lifetime disappear from the API — but it leaves two things open, and they are worth stating rather than discovering:
- A declared ceiling. There is no way today to say "build this within N gigabytes or fail". The honest version is not a knob bolted onto the engine but partition-wise execution: build the model in slices of the leading foreach dim and sink each slice, which the locality closure (pointwise ∩ bounded-halo) already guarantees is safe. Wanted when a model is written once and run on a machine chosen by someone else.
- The solver is the larger term anyway. Measured at roughly an order of
magnitude above the build at 10⁷ variables
(benchmarks), so a ceiling on the build is worth
having for the write path —
lps.writeto LP or MPS, handed to something else — and worth much less when the same process goes on to solve.
Deliberate non-primitives¶
Billed vs banned: an escape: island (#38) returns affine COO rows, so it
buys back what the relational and local rules exclude, priced against a
label budget enforced before any Python runs. It cannot buy back degree.
Reach for one only after #38's decision tree: different values → a parameter;
different in-language shape → a second constraint with where; only unsayable
shapes escape.
| Request | Why not | Instead | Escape? |
|---|---|---|---|
| Quadratic / bilinear terms | degree pinned at 1 | piecewise: (convex: true) or the epigraph pattern |
banned today, blocked by the QP∧MILP exclusion on the default solver — a Track 4 capability question, and the bigger diff of the two (new plan node, lowering, convexity guard) |
| SOS / indicator (#23) | the default solver has no such concept | piecewise: λ-formulation (SOS2's usual purpose); big-M for indicator, given bounds |
banned today, and the easier one: lp_file carries it as a text section and Gurobi natively — it needs Track 4, not a new invariant |
| Running sum over a variable | not locality — the emitted model is O(T²) rows | state-variable recurrence (a storage SOC balance is the rewrite) | priced, not banned — see the window family |
Normalisations (x / sum(x)) |
degree, not locality — a variable divisor is refused by law 9 | fix the denominator as a parameter, or state the ratio as a constraint | billed |
| Conditionals, iteration, data-dependent structure | destroys the closed AST | where masks + foreach dims; computation → data prep |
billed — inside an island only |
| Resampling, clustering, IO, units | data layer, not math | preprocess; pass a parameter | n/a — free in Python already |
Domain helpers (reduce_carrier_dim) |
encodes one domain into the language | component libraries over generic primitives | n/a |
Arbitrary array ops (merge, reindex) |
unbounded; xarray with extra steps | data prep | billed |
Honest snapshot¶
Cheaper here, because the model is tables: model statistics and coefficient-range diagnostics; IIS read-back (a join, not a scatter — as dual read-back, now shipped, already was); serialization (parquet, more portable than netCDF); elastic relaxation; and dualization — transposing a COO matrix is swapping two column names.
Ahead of comparable declarative layers: sparse-by-construction build with
no dense intermediate, and a hand-off straight to the solver rather than through
a file (see benchmarks; a declared memory ceiling is
Track 5, not something we have);
parameterised macros: (Calliope's sub-expressions take no arguments); binary
and integer variables; piecewise as N links with per-link signs, convex mode and
active gating; non-cyclic shift; load-time validation of every expression,
where string and uncalled macro template.
Behind: Track 1 items 1–6 and 10–11, plus Tracks 2 and 3. Where linopy stays
ahead it is not expressiveness but the post-solve object (labelled DataArrays vs
our tidy tables — to_dataarray bridges), debugging (IIS via Gurobi, print()
of a row, against our status code — until Track 2b exists), lifecycle (mutate,
re-solve, warm start, relax/fix, callbacks vs our build-and-solve), solver
breadth (ten backends and four handoffs vs HiGHS-direct plus LP files, #106),
and one variable type we simply do not have (semi_continuous, item 11). None
are ceiling questions; all are cheaper here once written, which is why they need
scheduling rather than assuming they follow from the architecture.
The ranking that follows from this page, if it is read as a whole: indexed access (items 1–4) blocks whole model classes today; the Track 2b verbs block using the engine at 3am; solver breadth blocks arrival from linopy at all; and items 10–11 are cheap, unblocked and unscheduled.
Solvers we actually speak: HiGHS through solver_direct, Gurobi planned on
the same path, LP files for everything else. Stated because anyone arriving from
linopy will assume solver-agnosticism; the LP route costs the float→text→parse
round trip solver_direct exists to remove, so "fast" is per-solver binding work
rather than a property of the engine.