Harvester

The harvester downloads Dutch legislation and converts it to the RegelRecht YAML format. It handles two sources: national law from the BWB (Basiswettenbestand / wetten.nl) and decentralized regulations from the CVDR (Centrale Voorziening Decentrale Regelgeving). The CLI picks the right source from the identifier, so BWBR… IDs are fetched from BWB and CVDR… IDs from the CVDR.

Overview

• Language: Rust
• Location: packages/harvester/
• Sources: BWB / wetten.nl (national law) and CVDR (local/decentralized regulations)
• Output: YAML law files with textual content (no machine_readable yet)

How it works

Pipeline Steps

1. Validate input - BWB ID format (BWBR + 7 digits) and date
2. Download WTI metadata - title, regulatory layer, publication date
3. Resolve consolidation date - from manifest.xml, find version valid for target date
4. Download content XML - the consolidated law text (with size limit check)
5. Parse elements - via extensible registry of element handlers
6. Split articles - hierarchical splitting into artikel → lid → lijst → li with dot-notation numbering (e.g., 1, 1.1, 1.1.a)
7. Normalize text - fix spacing, Unicode NFKD, wrap at 115 chars
8. Generate YAML - schema-compliant output with yamllint compliance
9. Atomic write - temp file → sync → rename

Element Processing

The harvester uses an extensible registry system for XML element handling:

The marker matters more than it looks. The BWB XML states some norms as a picture: artikel 22a Participatiewet puts the whole kostendelersnorm formula in an <illustratie>. Those elements used to be skipped, so the sentence that introduces the formula ended on its colon and the norm vanished with nothing to show it had. An incomplete article then read as a complete one. The harvester still cannot render an image, but it no longer drops it in silence, and a <formule> carrying a readable fallback keeps that text.

Dutch Law Hierarchy

Articles are split following the legal text structure:

Produces dot-notation: 1, 1.1, 1.1.a, 1.1.a.1°

Usage

CLI

As Library

For CVDR regulations use download_cvdr_law; detect_source returns the right source for either kind of identifier.

Output path convention

Example: regulation/nl/wet/wet_op_de_zorgtoeslag/2025-01-01.yaml

The regulatory layer is determined from the WTI metadata (soort-regeling field).

HTTP Resilience

• Timeout: 30 seconds (accommodates large XML files)
• Max response size: 100 MB (configurable via --max-size)
• Retries: 3 attempts with exponential backoff (500ms, 1s, 2s)
• Retry triggers: Connection errors, timeouts, 5xx responses
• No retry on: 4xx client errors

Current limitations

• Text-only extraction - tables and complex formatting simplified to text
• No machine_readable - output contains text only; executable logic added separately
• Reference extraction incomplete - cross-references detected but not fully resolved
• Large laws require --max-size flag (e.g., Wet op het financieel toezicht at 52.6 MB)

Testing

Integration tests use fixtures from tests/fixtures/zorgtoeslag/ (real WTI and content XML) to validate the complete pipeline from XML to valid YAML.

Further reading

• Law Format - the YAML format the harvester produces
• Pipeline - job orchestration for harvesting tasks