RFC-036: Absent and Unknown Values

Context

A law reads most of its facts from registers, and a register does not hold a value for everyone. “There is nothing here” has three different meanings, and the law treats each differently. The difference is not in the world but in what the engine knows about it. This RFC exists to keep the three apart, and to give each one an explicit expression everywhere it occurs: in the law, in the data, in a test, in the outcome.

1. The engine knows there is none (absence). The BRP has no partner for this person; the rent register has no contract for a homeowner. The register is authoritative for the non-existence, so “none” is a fact you can decide on, and the legal text usually attaches a consequence to it (“indien de belanghebbende geen partner heeft”). Codd called this the I-mark: inapplicable.
2. The engine does not have the fact (unknown). The wage has not been reported, the applicant has not filled in the rent, the register was not reachable. Whether the fact exists is part of what is not known: an unknown partner is not “no partner”, it is “we cannot tell”. Nothing can be decided about it; data has to be obtained. Administrative law has a procedure for exactly this: the authority gathers the necessary facts (Awb art. 3:2), the applicant supplies what they reasonably can (art. 4:2), and an incomplete application is completed within a term before it may be set aside (art. 4:5). A decision must be reasoned (art. 3:46); “the calculation gave null” reasons nothing. Codd’s A-mark: applicable, value missing.
3. The model is wrong (author error). A misspelled key in parameters:, a required parameter the caller forgot, a field read from a record that does not have it. This is a property of the YAML, not of the world, and it must stay an error: an unknown here would hide a bug behind a plausible non-decision.

The engine can distinguish these only if every party says which one it means. The register (or whoever materialises its data) says “none” or says nothing. The law says where it decides on absence. The caller says what it passes. The test says what it expects. None of this can be inferred; it has to be written down.

Until now the engine had one value, null, for both, and no consistent rule for it. Ordered comparison and arithmetic on null were type errors, so the rent clause failed the whole benefit for a homeowner. EQUALS compared structurally, so EQUALS($x, null) was the idiom for “absent”. And null read as false in AND, OR and IF.

The first version of this RFC gave null one meaning, unknown, and made comparison, arithmetic and property access propagate it, with Kleene logic in AND, OR, NOT and IF. Review showed that this still produces a confident wrong decision on the production corpus. EQUALS kept its structural meaning, so EQUALS(null, true) is a definite false. The zorgtoeslag decides eligibility with

and vermogen_onder_grens is LESS_THAN_OR_EQUAL over a wealth that is built by ADD from box-3 inputs. A blank box-3 cell therefore flowed as null into EQUALS, came out as false, and AND said “geen recht op zorgtoeslag” with full confidence. The Kleene rule in AND never got a chance: it was handed a definite false. The same shape decides the right to vote in the kieswet. One value for two meanings cannot be made safe: any operation that answers on absence (and EQUALS null must answer, the corpus depends on it) also answers on a value that is merely not known.

Decision

The engine has two values for “nothing”, and a third case that stays an error.

Unknown itself has two origins that the provenance names, because the remedy differs: no_data (a register input no source could supply: ask the register or the citizen) and not_passed (an optional parameter the caller left out: ask the applicant, or fix the caller).

On the word null. Readers who know SQL or Kleene’s three-valued logic expect null to mean “unknown”. Here it means the opposite, and the RFC accepts that cost for two reasons. The literal null is what YAML and JSON have, it is already the absence test in the corpus (EQUALS … null, Burgerlijk Wetboek boek 5 art. 42 on a missing municipal ordinance), and it is what a register emits for “there is none”. And the unknown must not have a literal at all: a law that could write “unknown” would invent a missing fact. So the prose says absent and unknown, the Gherkin step is is absent, the editor and the demo say “geen” and “onbekend”, and the bare word null appears only where the file format leaves no choice: in a YAML literal and in a data cell.

null is absence. It is a value, like 0 or false. A law writes it (value: null), a data source delivers it as an explicit null, and the engine produces it where the law itself says nothing: an IF with no matching case and no default, MIN/MAX over an empty collection. Absence can be tested and compared, and that is all: a legal text never treats “geen” as an amount or a verdict without saying so.

Delegation is not absence. An optional open term without an implementing regulation (RFC-003) is a gap the higher law fills with the term’s own default; a law does not test the term for null. Burgerlijk Wetboek boek 5 art. 42 did, and now carries the statutory distance as the default of the open term. An implementing regulation that yields null for a case is silent on that case: no deviation was granted, so the default of the delegating law applies. A term with a default is never absent. The default stands in for an absent deviation only, never for an unknown one: an implementation whose own register value is unknown leaves the term unknown, and the caller asks for the missing fact instead of silently applying the statutory value. The corpus is the closed world for regulations (a verordening that is not in it does not exist for the engine); data is the open world. The consequence is that an implementing regulation cannot express “there is none” for a term that has a default, and that is acceptable: what is delegated is the power to deviate from the rule of the higher law, not the power to declare that no rule applies. Because a typo in the implementation looks the same as silence, the engine logs a warning and marks the trace node distinctly (OPEN_TERM_SILENT) when the default is taken for that reason. With several implementations of one term, one that fills the term wins over one that is silent, whatever their priority order; the default applies only when all are silent. For a required term without a default there is no such net: the implementation’s output is the term’s value, so the type checker (RFC-037, N5) refuses an implementation that declares that output nullable.

Unknown is a fact the engine does not have. Whether it exists is not known either. It cannot be written in a law. It is produced only by resolution, and it carries provenance: which law needed which fact, and why the engine does not have it.

An unknown propagates through every operation and the provenance sets merge. An unknown inside a list or a record counts as much as a bare one: EQUALS, NOT_EQUALS, IN and NOT_IN compare containers element by element, and [unknown] against [650] is as undecidable as unknown against 650. Nothing turns an unknown into a definite answer, except a definitive operand where logic allows it (Kleene):

An untranslatable (RFC-012) beats an unknown: a gap in the model makes the gap in the data irrelevant for that value. The precedence holds per operand and per element: IN without a definite match returns the untranslatable element that could have been the match, before it says unknown.

AND, OR and IF evaluate their operands in order and stop at the first operand that decides: a false in AND, a true in OR, a matching case in IF. An operand after that point is never evaluated, so AND(false, null) is false while AND(null, false) is an error. This is what makes the guard idiom work: AND(NOT(EQUALS $huur null), GREATER_THAN $huur 500) never orders the absent rent. The precedence between untranslatable and unknown is decided among the operands that were evaluated.

The error on an absent operand is deliberate. huur > 500 for a homeowner is not unknown, and it is not false either: the clause does not apply, and the law has to say so (EQUALS($huur, null) first, or an eligibility condition). The error names the operation and points the author at the missing condition. This is the closed-world statement made explicit, at the one place a jurist can check it.

Cross-law calls (RFC-007) pass exactly what the caller writes in parameters:. A required parameter that resolves to null or unknown does not run the target; the input is that same null or unknown. The one exception is a required parameter the target declares nullable: the target has said it can decide on nobody, so a null is passed and the target runs, exactly as at the top level. nullable says nothing about unknown, which skips as before. With several such parameters the outcome does not depend on their order: if any is unknown, the input is unknown for the union of their facts; only when all are null is the input null. The skip applies to a target law the engine has loaded; a call to a law that is not loaded stays the LawNotFound error it always was. An optional parameter that is null or unknown is passed through. A required parameter the caller omits is the error it always was. The engine never fills an omitted parameter from data sources bound to the target: a parameter is what the caller supplies, and a value the target should read from its own register is an input with source: {} in the target, where a reader of that law can see it.

A required parameter names the person. A source: {} input stays unresolved, and a reference to it fails, when a required parameter of the article is not passed at all: the register was never asked about anybody, so a forgotten or misspelled key is an error and not an unknown. A required parameter passed as null or unknown at the top level, by the caller evaluating the law, is the same error under another guise (MissingParameter: “a law cannot be evaluated for nobody”), unless the parameter is declared nullable, in which case the null is a value the law decides on; the cross-law skip above is the only place where a null or unknown required parameter is neither an error nor a value, because there the law itself produced it.

A lookup key that is unknown or null never reaches a register. When a data source keys on a parameter that is unknown (a partner nobody has named yet), the input is that unknown, so the outcome asks for the partner and not for the birth year the register could not deliver. When the key is null (there is no partner), the input is null: asked about nobody, the register answers with an absence, the same rule as the cross-law skip. A record whose key field is literally the string "null" or "unknown" does not match such a criterion.

required defaults to true, on the top-level path and the cross-law path alike. The engine has always read an absent flag that way; the schema text said “defaults to false”. Schema v0.5.8 corrects the wording.

Absence is declared, not discovered. Whether a field may be null is a property of its type: nullable: true on a parameter, input or output (schema v0.5.8, the one construct this RFC adds). Without it a field is never absent, and the engine holds every party to that:

The static type checker of RFC-037 enforces the declaration before a law runs: a comparison with null needs a nullable field (N1), the literal null appears only as the value of a nullable output (N2), an IF without default is allowed only as the value of a nullable output (N3), a nullable variable enters arithmetic, a decision or a non-nullable output only after an absence test on that path (N4), and an input taking a nullable output or a skipped call from another law is nullable itself (N5). just validate and the loader run the same rules, so a law that forgets an absence test is refused where it is written, not three cross-law calls later. Every field in the corpus that a law tests for null is declared nullable; the demo’s data contract (absent: null in bindings.yaml) must coincide with it, and a test says so.

The result carries the non-decision. An unknown output is serialised as {"__unknown": true, "missing": [{"law", "name", "kind"}]}. A portal can ask the applicant for the missing fact and a caseworker’s screen can say “niet beslisbaar; ontbreekt: spaargeld (wet_inkomstenbelasting_2001)” without parsing the trace. The trace records every propagation with the same facts.

In Gherkin, the empty cell means one thing in every table: nothing is stated. In a data table the record has no value for that column (the key is omitted, so the input is unknown); in a the following parameters: table the parameter is not passed (an optional one is then unknown for lack of it, a required one is the caller’s omission). The literal null in a cell means absence. The quoted empty string, parameter "x" is "", is null too, in both runners; the editor never writes it, because a blank field in the form is a parameter that is not filled in and is left out of the scenario. Two assertions pin the outcome: output "x" is unknown and output "x" is unknown for lack of "huur". The editor shows an unknown as “onbekend” with the missing facts, an absence as “geen”, an output the engine never produced as “niet berekend”, and never the word null.

A law cannot write an unknown. The serialized form {"__unknown": true, "missing": [...]} exists so results and traces round-trip, and the deserializer would build the value from a law’s YAML just as well. The loader therefore walks every literal in the definitions and the actions and refuses a document that carries one: an unknown written by hand would put an invented provenance into an outcome. A sentinel without a usable missing list is not an unknown but an ordinary object, for every reader alike.

Why

A confident wrong decision is structurally impossible. No operation maps an unknown to a definite boolean. The zorgtoeslag with a blank box-3 cell now answers “onbekend, ontbreekt: spaargeld”, which is the truth.

The outcome is what the law needs. An unknown fact yields a named non-decision, the input to the Awb art. 4:5 step and to the reasoning duty. An absent fact yields a decision along the condition the legal text itself states. The predecessor of this engine tracked missing_required as a first-class status and its API and explanations depended on it; the demo reconstructed the same thing from the trace. Two consumers rebuilding a state is the proof it belongs in the result.

Absence is in the signature. A jurist reading a law sees on the field whether “none” is a case the law handles, and the validator refuses a law that tests a field for null without saying so, or that lets a null reach a sum.

Author errors stay errors. A misspelled parameter key, a field on a record that exists, a calculation on an absence: each is a property of the model, not of the world, and each is reported as such. Only data can produce an unknown.

Deterministic and auditable. The provenance is a function of the evaluation; trace and result say the same thing. The conformance suite (bucket B, bdd/conformance/null_semantics.feature) pins every row of the two tables above, so another engine cannot drift.

Tradeoffs

• Laws have to model absence. A law that compared $huur without asking whether there is a rent now fails for the homeowner instead of computing something. That is the point, but it is work in the corpus, and the migrated demo laws show how much: they were written against a runtime that skipped None operands silently.
• The data layer has to mean what it says. An explicit null is a claim (“there is none”); omitting the value is honesty (“we do not have it”). A data source that writes null for everything it lacks turns unknowns into false facts. The demo’s bindings state this per input, and the law states with nullable which inputs may carry one; the two are checked against each other.
• No IS_UNKNOWN in the law language. A legal text does not branch on whether a fact is known; where the law decides on incompleteness (reversal of the burden of proof, estimates) that is procedure, not calculation. Should a text be found that does, that is the moment to add the construct, and not before (the same discipline RFC-012 and RFC-016 apply).
• Two things to know instead of one. null and unknown look alike in a data table until you learn that an empty cell and the word null differ. The editor makes the difference visible.

Alternatives Considered

Keep null an error, fix the laws. Honest, but it knows nothing between “decided” and “broken”, while the law has exactly that middle state. It also aborts the whole evaluation, so a caseworker sees one missing fact per run, and what was decidable (is_verzekerde: true) is lost.

One null that means unknown (the first version of this RFC). Rejected for the reason in the context: EQUALS has to answer on absence and therefore answers on unknown too, and the production corpus decides eligibility with EQUALS … true. It also left the outcome a bare null without saying what was missing, and it let the engine fill an omitted optional parameter from the target’s data sources, which made a call’s outcome depend on which sources happened to be loaded, invisible to a reader of the law.

Reuse the untranslatable machinery for unknown. The mechanism is right (a value carrying its origin, propagating through everything) and this RFC uses it. The meaning is wrong: an untranslatable is a gap in the model and counts as such in every dashboard and in the error mode; a missing register value is a gap in this case’s data, and a law may legitimately act on the neighbouring case of absence.

Treat absence as unknown in arithmetic (so huur > 500 for a homeowner is unknown, not an error). This would ask the homeowner for their rent. The register knew there was none; the model was missing a condition. An error tells the author; an unknown tells the citizen something untrue.

Consequences

• Engine: Value::Unknown(Vec<MissingFact>) in packages/law-model; operations.rs, context.rs, service.rs, data_source.rs (no lookup key from a null or unknown criterion), error.rs (AbsentOperand, MissingParameter), load_check.rs (no unknown literal in a law), trace.rs, wasm.rs (an undefined property is dropped before deserialising, in parameters and records alike and at any depth, so JavaScript cannot turn “we do not have it” into null).
• Conformance: bdd/conformance/null_semantics.feature rewritten against the synthetic laws test_null_semantics, test_null_semantics_bron and test_null_semantics_strikt; the nested-FOREACH scenario in collections.feature now expects an error where RFC-016 expected null (RFC-016 carries a note).
• Grammar: assert_unknown, assert_unknown_for; empty table cells omit the key or the parameter, in both the Rust and the JavaScript runner.
• Schema: v0.5.8 adds nullable on parameters, inputs and outputs and corrects the wording of required; the test laws declare it.
• Editor: unknown rendered as “onbekend” with the missing facts; a null cell is offered only for nullable columns; is absent asserts absence instead of being neutral.
• Corpus: laws whose parameters are application-form fields another law never passes declare them required: false; laws that compute over a register value that can be absent test for absence first. Bucket A runs unchanged.
• Demo: the data layer omits what it does not have and writes null only where a register is authoritative for absence; the portal asks for the facts the result names.

References

• Awb art. 3:2, 3:46, 4:2, 4:5 (feiten vergaren, motivering, aanvulling van een onvolledige aanvraag)
• RFC-007: cross-law execution and source: {} inputs
• RFC-012: untranslatables, the neighbouring case
• RFC-016: collections; null in FOREACH
• E. F. Codd, The Relational Model for Database Management, Version 2 (1990), ch. 8: the A-mark (applicable, value missing) and the I-mark (inapplicable), the same distinction under other names
• bdd/conformance/null_semantics.feature