Europa/Nuvolari/docs/data-sources.md

# Data sources

Every endpoint below was verified on 2026-09-10. Anything not verified is marked
**TO VERIFY** and must not be relied on until checked.

## 1. Radar — Radar-DPC (the active source)

Public platform of the Italian Dipartimento della Protezione Civile.

- Base URL: `https://radar-api.protezionecivile.it/`
- Transitional test base URL: `https://wagiqofvnk.execute-api.eu-south-1.amazonaws.com/prod`

### `GET /findLastProductByType?type=<PRODUCT>`

Returns the timestamp of the most recent sample of a product.

```json
{
  "total": 1,
  "lastProducts": [
    { "productType": "VMI", "time": 1758706200000, "period": "PT5M" }
  ]
}
```

- `time` is epoch milliseconds, UTC.
- `period` is the ISO-8601 cadence of the product.
- `400` when `type` is missing, `404` when no product is available.

### `POST /downloadProduct`

Returns a pre-signed S3 URL for the raw raster file.

```json
{ "productType": "VMI", "productDate": 1758706200000 }
```

```json
{
  "bucket": "dpc-radar",
  "key": "VMI/22-09-2025-14-20.tif",
  "url": "https://dpc-radar.s3.eu-south-1.amazonaws.com/...",
  "expiresSeconds": 900
}
```

- `productDate` must be **rounded down to the product step**, otherwise `404`.
- Pre-signed URLs expire after 300–900 s: download immediately, never cache the URL.
- `400` when `productDate` is missing or non-numeric.

### Mandatory header

```
origin: https://radar.protezionecivile.it
```

Required on **every** request to the platform for security reasons; requests without it
are rejected. `Content-Type: application/json` is also required on `downloadProduct`.

### Products and cadence

| Cadence | Products |
|---|---|
| 5 min | `VMI`, `SRI`, `SRT1`, `IR_108` |
| 30–60 min | `TEMP`, `CUM3`, `CUM6`, `CUM12`, `CUM24`, `CAPPI_1`…`CAPPI_10`, `VIL`, `ETM`, `POH` |
| — | `SITES` (GeoJSON, fetched directly from S3 with `GET`) |

The MVP uses **`VMI`** (Vertical Maximum Intensity) as the precipitation layer.
History retention is 14 days. Files are GeoTIFF; CRS is read from the file metadata
rather than assumed — the worker reprojects to EPSG:3857 in all cases.

### Raster format (read from a live VMI file, 2026-09-10)

| Property | Value |
|---|---|
| Size | 1200 × 1400 px |
| Pixel size | 1000 × 1000 m — a **1 km grid** |
| Bands / type | 1 band, **Float32**, LZW compressed |
| Tie point | raster (0,0) → model (-600000, 650000) |
| Model type | **projected**, datum WGS84 (GeographicTypeGeoKey 4326) |
| Projection centre | 12.5° E, 42.0° N (`ProjCenterLong`/`ProjCenterLat`) |
| Linear units | metre |
| Nodata tag | absent — expect NaN |

**The rasters are not in EPSG:4326 and not in EPSG:3857.** They are on a custom
projection centred on Italy, so reprojection to Web Mercator is mandatory, not an
optimisation. The extent works out to 1200 km east-west and 1400 km north-south around
that centre, which covers the whole country — the worker crops to the region bounding
box before doing anything else, so it never carries the full mosaic through the
pipeline.

The GeoKeys as written are internally inconsistent (`ProjCoordTransGeoKey` = 1, which
is transverse Mercator, combined with the centre keys an azimuthal projection uses).
**Do not hardcode a source CRS.** The worker reads the CRS from the file with rasterio
and warps from whatever it finds, and logs the detected CRS on every run so a change at
the source is visible instead of silently shifting the imagery.

Values are physical units (dBZ for VMI), not palette indices — the colour mapping is
ours, and the legend that goes with it is published in the manifest.

### Push channel

`wss://radar-wss.protezionecivile.it` pushes a notification whenever a new sample is
published. The worker uses this as its **primary trigger**, with a 5-minute cron as a
fallback, so we never poll DPC aggressively.

### License

**CC BY-SA.** Attribution to "Radar-DPC" is mandatory and derived products must be
released under the same license. The rendered PNG frames we publish are a derived
product and inherit CC BY-SA.

Docs: <https://dpc-radar.readthedocs.io/it/latest/>

## 2. Radar — ARPA Piemonte (future adapter, disabled)

C-band polarimetric radars at Bric della Croce and Monte Settepani, scanning every
5 minutes. Real-time volumes of reflectivity, Doppler velocity and differential
reflectivity are published in **OPERA HDF5 (ODIM)** format, covering the last hour.

Listed as open data at <https://www.arpa.piemonte.it/dato/open-data>, described there as
constantly updated, machine-readable and free of charge.

**Two things block adoption, and both belong to the project owner, not to the code:**

1. **The access link is not public.** The page states that to obtain the real-time link
   for radar and radiosonde data *"è necessario inviare una e-mail all'indirizzo
   info.meteo@arpa.piemonte.it"*. There is no documented endpoint to call without it.
2. **The licence is stated site-wide, not on that page.** The open-data page itself says
   only that the data is *gratuiti*. ARPA's legal notice at
   <https://www.arpa.piemonte.it/note-legali> does state a licence for environmental and
   open data: **CC BY 4.0, commercial use permitted**, crediting
   *"Fonte: Arpa Piemonte - www.arpa.piemonte.it"*. Logos, emblems and third-party
   content are excluded from it.

   Both ARPA REST APIs link that same notice from their OpenAPI `description`, so for
   them the licence is unambiguous. The radar page does not repeat it, so the safe move
   is to ask ARPA to confirm the terms apply to the radar volumes in the same email that
   requests the access link.

Until the link exists, `ArpaRadarSource` stays a stub whose every method throws. It is
named by the region config as unavailable so the seam is visible, not because it is
nearly ready.

## 3. ARPA Piemonte REST APIs — surveyed 2026-09-10, not yet adopted

Two public APIs, **no authentication, no key, no registration**. Both link
<https://www.arpa.piemonte.it/note-legali> from their OpenAPI `description`, so both are
CC BY 4.0 with commercial use permitted.

### `api_realtime` — <https://utility.arpa.piemonte.it/api_realtime>

Spec at `/api_realtime/openapi.json` (the `/docs` page is JS-rendered and cannot be
scraped; fetch the spec instead).

| Endpoint | Content |
|---|---|
| `GET /pie_anag` | Station registry: **374 stations**, `lat`/`lng`, elevation, municipality, province, basin, sensor letters |
| `GET /data_pie` | Hourly observations, last 3 days: `air_temperature`, `humidity`, `cum_rain_1h/3h/6h/12h/24h`, `wind`, `gust_of_wind` + directions, `snow_height`, `hydrometric_level`. Paginated |
| `GET /pie_neve` | Daily validated snow height, 08:00 |
| `GET /ggd` | Heating and cooling degree days |
| `GET /status` | Health check |

Sensor coverage across the 374 stations: **286 with a rain gauge (P)**, 277 temperature,
143 hydrometer (H), 90 wind, 79 snow. Rain gauges span 74 m to 2820 m elevation.

**Measured latency: ~4.5 hours**, uniform across every station sampled at 16:30 on
2026-09-10, on an hourly cadence. This is the single most important property of the feed:
it is an observation archive with a publishing delay, **not** a real-time source, and it
cannot be presented alongside 5-minute radar as if it were current.

**Hydrometric thresholds are effectively absent**: only 4 of 143 hydrometer stations
carry `guard_threshold` / `danger_threshold`, and only 5 carry a river name. A river-level
warning feature built on this would be blank for 97% of stations.

### Meteoweb — <https://utility.arpa.piemonte.it/docs/>

Spec at `/schema/` (YAML). Django REST Framework, hyperlinked, paginated.

Historical archive under `/meteoidro/`: daily and monthly meteo (**3.27 M** daily
records), hydrological (**931 k**), snow, century-long series, annual rainfall maxima,
flow-duration curves and rating curves, plus the station and measurement-point
registries. Latest daily data was one day behind (`2026-09-09`).

Also `/sismica/`: **35 910** seismic events since 2000 with position, depth and
magnitude, and the seismic classification of municipalities.

### Verdict

Only one thing here is on-scope: **rain-gauge accumulations as ground truth for the
radar**. Radar infers rainfall from reflectivity aloft; gauges measure what actually
reached the ground, and the network is dense exactly where Alpine terrain blocks the
radar beam. Worth adding as a clearly-timestamped observations layer once the worker
exists — never as a live reading.

Everything else is out of scope for this app: snow depth, degree days, the historical
archive and the seismic catalogue all belong to a different product. Hydrometric levels
would be valuable during heavy rain but the thresholds that would make them meaningful
are not published.

## 4. Forecast — out of scope

The app shows radar, not forecasts. No forecast provider is integrated, and MET Norway,
ItaliaMeteo ICON-2I and Open-Meteo are all out of scope. See CLAUDE.md for the current
scope boundary.

## 5. Alerts — ARPA Piemonte XML-CAP

- Feed: `https://www.arpa.piemonte.it/export/xmlcap/allerta.xml`
- Landing page: <https://www.arpa.piemonte.it/rischi_naturali/snippets_arpa/allerta/index.html>
- Official PDF bulletin: `https://www.arpa.piemonte.it/rischi_naturali/boll/bollettino_allerta.pdf`

Issued daily by 13:00 by the Centro Funzionale Regionale, 36 hours of validity.

### Alert zones (11)

`Piem-A` … `Piem-M` (A, B, C, D, E, F, G, H, I, L, M), covering: Toce; Val Sesia/Cervo/
Chiusella; Valli Orco/Lanzo/bassa val Susa/Sangone; Alta val Susa/Chisone/Pellice/Po;
Valli Varaita/Maira/Stura; Valle Tanaro; Belbo e Bormida; Scrivia; Pianura
settentrionale; Pianura e colline torinesi; Pianura cuneese.
The exact code-to-name mapping is stored in `app/assets/regions/piemonte.json`.

### CAP document structure (verified against the live feed, 2026-09-10)

One `<alert>` contains one `<info>` block **per zone**; the zone is in
`info/area/areaDesc` and holds the bare code (`Piem-A`), not a name.

```xml
<alert xmlns="urn:oasis:names:tc:emergency:cap:1.2">
  <identifier>261/2026</identifier>          <!-- bulletin number / year -->
  <sender>centro.funzionale@arpa.piemonte.it</sender>
  <sent>2026-09-09T10:00:02+00:00</sent>
  <status>Actual</status><msgType>Alert</msgType><scope>Public</scope>
  <note>AVVISO DI CONDIZIONI METEOROLOGICHE AVVERSE ...</note>
  <info>
    <event>GIALLO</event>                    <!-- overall level for this zone -->
    <onset>...</onset><expires>...</expires>
    <parameter><valueName>EFFETTI SUL TERRITORIO</valueName><value>...</value></parameter>
    <parameter><valueName>TEMPORALI_1224</valueName><value>GIALLO</value></parameter>
    <parameter><valueName>TEMPORALI_2436</valueName><value>VERDE</value></parameter>
    ...
    <area><areaDesc>Piem-A</areaDesc></area>
  </info>
  <!-- 10 more info blocks, one per zone -->
</alert>
```

Parameters are named `<CATEGORY>_<WINDOW>`:

- **Categories** — `IDRAULICO`, `IDROGEOLOGICO`, `TEMPORALI`, `NEVE`, `VALANGHE`.
- **Windows** — `1224` (hours 12–24) and `2436` (hours 24–36), matching the 36 hours
  of validity.

So each zone carries 10 level values plus one overall `<event>` level and a free-text
`EFFETTI SUL TERRITORIO`.

### Levels — six values, not four

| Value | Meaning |
|---|---|
| `VERDE` | no significant phenomena |
| `GIALLO` | localised phenomena |
| `ARANCIONE` | widespread phenomena |
| `ROSSO` | numerous or extensive phenomena |
| `BIANCO` | **avalanche scale only** — outside the season / not assessed |
| `-` | no value published for that category and window |

`BIANCO` and `-` are not "green with a different name" and must not be collapsed into
it. The parser models the level as an enum with explicit `bianco` and `notAvailable`
members, and the UI renders them as their own state rather than as "no alert".

Levels are republished **verbatim, never reinterpreted**, always alongside a link to
the official bulletin. No license is stated on the ARPA page, so we credit ARPA
Piemonte and link the source rather than claiming any reuse right.

Official bulletin PDF, linked from every alert view:
`https://www.arpa.piemonte.it/rischi_naturali/boll/bollettino_allerta.pdf`

### Zone codes

`Piem-A` … `Piem-M`, eleven zones: **the Italian alphabet is used, so J and K do not
exist** — the sequence is A B C D E F G H I L M. The code-to-name mapping lives in
`app/assets/regions/piemonte.json`; the human-readable names are not in the feed and
come from the ARPA zone documentation.

A captured copy of the live feed is kept as a test fixture so the parser is verified
against the real document rather than a hand-written approximation.

## 6. Municipalities — Istat boundaries

The 1180 municipalities of Piedmont with a representative point each, bundled as
`app/assets/regions/piemonte_comuni.json` (98 KB) and generated by
`tool/generate_places.py`.

**Source**: Istat, *Confini delle unita amministrative a fini statistici*,
generalised boundaries at 1 January 2025, **CC BY 4.0**.
`https://www.istat.it/storage/cartografia/confini_amministrativi/generalizzati/2025/Limiti01012025_g.zip`

Bundled rather than geocoded online because the app is region-scoped: the search
is instant, works with no network, needs no API key, has no rate limit to
respect, and cannot return a result somewhere the app has no radar for.

Two properties of the source file are worth knowing before touching the
generator:

- **The shapefile is in WGS84 / UTM zone 32N**, not in degrees. The generator
  runs an inverse transverse Mercator to get latitude and longitude.
- **The DBF is mixed-encoding.** It is UTF-8 — "Aglie" ends in a single U+00E8 —
  but at least one bilingual Friulian record makes strict cp1252 fail, and a
  terminal renders utf-8 and latin-1 output identically, so this cannot be
  settled by looking at printed text. The generator asserts that no output name
  contains the tell-tale mojibake sequences, and a test asserts it again against
  the shipped asset.

Points are the centroid of each municipality's **largest** part, not of the whole
shape: several are multipart, and averaging a main body with its exclaves can
land the point on a neighbour. Expect 1-4 km between the point and the town
centre, which is well inside what framing a map needs.

## 7. Base map — OpenFreeMap

OpenStreetMap vector tiles served by **OpenFreeMap** (<https://openfreemap.org>).

- Style in use: `https://tiles.openfreemap.org/styles/positron`
- Tiles: `https://tiles.openfreemap.org/planet`
- Glyphs: `https://tiles.openfreemap.org/fonts/{fontstack}/{range}.pbf`

**No API key, no registration, no request limits, commercial use permitted.** That is
what makes it the right starting point: every other free tier — MapTiler, Stadia, Jawg,
Thunderforest — requires a key, which means a secret to manage and a quota to outgrow.

Positron rather than Liberty or Bright: a radar overlay has to be the loudest thing on
screen, and Positron is a desaturated grey base designed to sit under data. On Liberty
the precipitation colours compete with road casings and landuse fills.

### Mandatory credits

| Credit | Required? |
|---|---|
| `© OpenStreetMap contributors` (ODbL) | **yes** |
| `© OpenMapTiles` | **yes** |
| `OpenFreeMap` | optional, and appreciated |

The style JSON carries **no `attribution` field**, so MapLibre will not display these on
its own. The app renders them itself from the region config: the two mandatory credits
go in the always-visible attribution bar, and all three are listed on the Sources screen.

### When there is no network

With `MAP_STYLE_URL=offline` the app generates a self-contained style from the region
bounding box — flat background plus the extent outline, no network sources at all. It is
deliberately plain so it is never mistaken for a finished map, and it is what the widget
tests run against.

## 8. Lightning — out of scope

Not part of the app. If it is ever revisited: **Blitzortung prohibits commercial use**,
and the DPC `LTG` product is the source to reach for instead.