German date display + optional street address for precise geocoding
- Date pickers (entry form + edit dialog) now display TT.MM.JJJJ instead of
ISO (DATE_DISPLAY_FORMAT). Storage stays YYYY-MM-DD; parse_date() already
accepted both formats, so existing data and the self-updater are unaffected.
- New optional "Straße" field (street + house number) in the entry form and
edit dialog, backed by a new `street` CSV column. geocode_city() and
GeocoderWorker.enqueue() gained a street parameter: when set, a full-address
query is tried first for a much more precise map point, falling back
automatically to the existing city/PLZ search if it doesn't resolve.
- Tab 2 and the entry queue show a Straße column; Tab 2 search now also
matches on street.
- Fix: pandas turns a blank CSV cell into NaN even for a dtype=str column, so
every existing (blank-street) row would have shown literal "nan" in Tab 2.
DataStore._load now does street.fillna("") after every read.
Verified with a non-GUI test suite (date parsing, query construction, CSV
round-trip incl. the NaN case) and a full GUI build/drive test on Python
3.12/Tk 9. Docs updated (changelog, overview, architecture, data-model,
dev-notes, improvements).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -7,8 +7,8 @@ Everything lives in [`app.py`](../app.py). There is no package structure.
|
||||
| Class / function | Responsibility |
|
||||
|------------------|----------------|
|
||||
| `DataStore` | Owns `orte.csv`. Loads it into a pandas `DataFrame`, exposes read helpers (`get_map_data`, `get_autocomplete_strings`, `total_count`, `find_duplicates`) and write helpers (`append_rows`, `update_row`, `delete_rows`). All writes go through `_save_csv` → rotating backup + atomic write. All columns are read as strings. Falls back to an empty frame (or the newest backup) if `orte.csv` is missing/empty/corrupt. |
|
||||
| `geocode_city()` | Pure function: `(city, plz) -> (lat, lon) | None`. Tries several Nominatim queries with a configurable regional bias (`GEOCODE_REGIONS`). Raises `GeocodingUnavailable` if *every* query failed on a service/network error (vs. simply not finding the place). |
|
||||
| `GeocoderWorker` | A `threading.Thread` (daemon). Holds a FIFO list of `(row_id, city, plz, callback)` items guarded by a lock, woken by an `Event`. Processes one item per second, calls `geocode_city`, then invokes `callback` **from the worker thread**. Each item is wrapped in `try/except` so one failure never kills the thread; after `SERVICE_ERROR_THRESHOLD` consecutive service errors it fires the optional `on_service_error` callback. |
|
||||
| `geocode_city()` | Pure function: `(city, plz, street="") -> (lat, lon) | None`. Tries several Nominatim queries with a configurable regional bias (`GEOCODE_REGIONS`); if `street` is given, a full-address query is tried first and falls back to the city/PLZ-only queries on no match. Raises `GeocodingUnavailable` if *every* query failed on a service/network error (vs. simply not finding the place). |
|
||||
| `GeocoderWorker` | A `threading.Thread` (daemon). Holds a FIFO list of `(row_id, city, plz, street, callback)` items guarded by a lock, woken by an `Event`. Processes one item per second, calls `geocode_city`, then invokes `callback` **from the worker thread**. Each item is wrapped in `try/except` so one failure never kills the thread; after `SERVICE_ERROR_THRESHOLD` consecutive service errors it fires the optional `on_service_error` callback. |
|
||||
| `parse_date()` / `parse_coord()` / helpers | Module-level pure functions for normalizing user input (date → ISO, coordinate strings → float, `"City (PLZ)"` splitting). Shared by the entry tab and `EditDialog`. |
|
||||
| `generate_map()` | Builds `karte.html` from `DataStore.get_map_data()` with Folium. |
|
||||
| `EditDialog` | `tk.Toplevel` modal dialog for editing one row. Returns its result via `self.result`. |
|
||||
@@ -22,7 +22,7 @@ Everything lives in [`app.py`](../app.py). There is no package structure.
|
||||
user types -> _on_add_row()
|
||||
-> append dict to self._queue
|
||||
-> insert row into Tab-1 Treeview (coords = "⏳ wird gesucht…")
|
||||
-> geocoder.enqueue(row_id, city, plz, self._geocode_done)
|
||||
-> geocoder.enqueue(row_id, city, plz, street, self._geocode_done)
|
||||
|
||||
GeocoderWorker thread (1/sec):
|
||||
coords = geocode_city(...)
|
||||
|
||||
@@ -2,6 +2,28 @@
|
||||
|
||||
Notable changes to the app. Newest first.
|
||||
|
||||
## 2026-09-12 — German date display + optional street address for geocoding
|
||||
|
||||
- **Date fields now display `TT.MM.JJJJ`** (e.g. `12.09.2026`) instead of ISO,
|
||||
in both the entry form and the edit dialog (`DATE_DISPLAY_FORMAT`). Storage on
|
||||
disk is unchanged (`YYYY-MM-DD`) — `parse_date()` still accepts either format
|
||||
when typed, so nothing about existing data changes.
|
||||
- **New optional "Straße" field** (street + house number) on the entry form and
|
||||
in the edit dialog, and a new `street` CSV column. When set, geocoding tries
|
||||
the full address first for a much more precise map point, falling back to the
|
||||
existing city/PLZ search if the address doesn't resolve — see
|
||||
[overview.md](overview.md#geocoding-behaviour). `geocode_city()` and
|
||||
`GeocoderWorker.enqueue()` both gained a `street` parameter.
|
||||
- Tab 2 and the entry queue both show a **Straße** column; the Tab 2 search box
|
||||
now also matches on street.
|
||||
- Fixed a bug this surfaced immediately: pandas turns a **blank CSV cell into
|
||||
`NaN`, even for a column forced to `dtype=str`** — every pre-existing row (all
|
||||
blank on `street`) would have displayed the literal text `nan`. `DataStore._load`
|
||||
now does `self.df["street"] = self.df["street"].fillna("")`. See
|
||||
[dev-notes.md](dev-notes.md#pandas-turns-blank-csv-cells-into-nan-even-with-dtypestr).
|
||||
- `find_duplicates` intentionally still ignores `street` (unchanged: date + city
|
||||
+ PLZ only).
|
||||
|
||||
## 2026-09-07 — fix: right-click context menu instantly triggered its first item
|
||||
|
||||
On Linux the Tab 2 (and queue) right-click menu flashed and immediately ran
|
||||
|
||||
+11
-3
@@ -7,14 +7,19 @@ just the header if it does not exist.
|
||||
|
||||
| Column | Type on disk | Meaning | Notes |
|
||||
|--------|--------------|---------|-------|
|
||||
| `date` | string | Assignment date | Always stored as `YYYY-MM-DD`. Input is validated and normalized by `parse_date()` (accepts `YYYY-MM-DD`, `DD.MM.YYYY`, `DD.MM.YY`, `YYYY/MM/DD`); invalid input is rejected before saving. Rows written before this change may still hold non-ISO strings. |
|
||||
| `date` | string | Assignment date | Always **stored as `YYYY-MM-DD`**, regardless of display. Input is validated and normalized by `parse_date()` (accepts `YYYY-MM-DD`, `DD.MM.YYYY`, `DD.MM.YY`, `YYYY/MM/DD`); invalid input is rejected before saving. The date pickers *display* `TT.MM.JJJJ` (`DATE_DISPLAY_FORMAT`) since the 2026-09-12 change, but that's cosmetic — parsing/storage is unchanged, so old and new rows are identical on disk. |
|
||||
| `city` | string | Place name | Free text. Used for autocomplete and duplicate detection (trimmed, case-insensitive). |
|
||||
| `postal_code` | string | German PLZ | Optional. Kept as a string so leading zeros survive. Regex elsewhere accepts 4–5 digits. |
|
||||
| `street` | string | Street + house number | Optional (e.g. `"Hauptstraße 12"`). Added 2026-09-12 for more precise geocoding — see [overview.md](overview.md#geocoding-behaviour). Rows written before that date have it blank. |
|
||||
| `lat` | string | Latitude | Blank until geocoded. Rounded to 5 dp. Parsed with `pd.to_numeric(errors="coerce")` when building the map. |
|
||||
| `lon` | string | Longitude | As above. |
|
||||
|
||||
Every column is read as a string (`dtype={"postal_code": str, "lat": str, "lon": str}`
|
||||
plus `city`/`date` default object). Numeric conversion happens only where needed.
|
||||
plus `city`/`date` default object) — **except this doesn't stop a blank cell from
|
||||
coming back as `NaN`** (a float), even for a column forced to `dtype=str`. This
|
||||
bit `street` immediately (all pre-existing rows have it blank): `_load()` now
|
||||
does `self.df["street"] = self.df["street"].fillna("")` right after reading.
|
||||
See [dev-notes.md](dev-notes.md#pandas-turns-blank-csv-cells-into-nan-even-with-dtypestr).
|
||||
|
||||
### Row identity
|
||||
|
||||
@@ -33,7 +38,10 @@ that ends up wrong.
|
||||
|
||||
`find_duplicates` flags a queued row when an existing row matches on all three
|
||||
of: `date` (string-equal), `city` (trimmed, lower-cased), `postal_code`
|
||||
(trimmed). It only produces a status-bar hint; duplicates are still written.
|
||||
(trimmed). `street` is **not** part of the match (unchanged by the 2026-09-12
|
||||
address feature) — two visits to the same city/PLZ on the same day are flagged
|
||||
as possible duplicates even with different streets. A modal now asks whether to
|
||||
save anyway (see [changelog.md](changelog.md)).
|
||||
|
||||
### Map aggregation
|
||||
|
||||
|
||||
@@ -57,6 +57,32 @@ Lesson: a tile source that works from `curl` or an `http://` page can still fail
|
||||
from `file://`. Test the map by **opening the generated `karte.html` directly**,
|
||||
not just by checking the URL in the HTML.
|
||||
|
||||
## pandas turns blank CSV cells into NaN even with `dtype=str` (2026-09-12)
|
||||
|
||||
Assumption that turned out wrong: forcing a column's dtype (e.g.
|
||||
`dtype={"street": str}`) does **not** stop `pd.read_csv` from parsing a truly
|
||||
empty field as `NaN` (a float) instead of `""`. Verified directly:
|
||||
|
||||
```python
|
||||
>>> pd.read_csv(io.StringIO("a\n\n"), dtype={"a": str})["a"][0]
|
||||
nan # not ''
|
||||
>>> str(pd.read_csv(io.StringIO("a\n\n"), dtype={"a": str})["a"][0])
|
||||
'nan' # !!
|
||||
```
|
||||
|
||||
This is *why* `lat`/`lon` already needed `_is_blank()` / `_coord_str()` helpers
|
||||
— but the new `street` column (added 2026-09-12) hit it immediately and much
|
||||
harder: it's optional, so *every* existing row has it blank, and without a fix
|
||||
every one of them would show the literal text `nan` in Tab 2. Fix: after
|
||||
`pd.read_csv` (all three load paths — normal, empty-file, backup-recovery),
|
||||
`DataStore._load` runs `self.df["street"] = self.df["street"].fillna("")`.
|
||||
|
||||
Rule for this codebase: **any column that can legitimately be blank needs an
|
||||
explicit `.fillna("")` (or the `_is_blank`/`_coord_str` treatment) right after
|
||||
`pd.read_csv`** — `dtype=str` alone is not enough. `city`/`postal_code` haven't
|
||||
needed this only because they're rarely actually blank in practice, not because
|
||||
they're immune.
|
||||
|
||||
## Tk popup menus: bind `<ButtonRelease-3>`, not `<Button-3>` (2026-09-07)
|
||||
|
||||
`widget.bind("<Button-3>", …)` + `menu.tk_popup(x, y)` + `finally:
|
||||
|
||||
@@ -215,6 +215,20 @@ Open follow-ups: no rollback if a pushed update is broken (mitigated by
|
||||
`--ff-only` from a branch Patrick controls + pip errors caught pre-restart);
|
||||
`run.sh` / `.desktop` still not committed (item 5).
|
||||
|
||||
## German date + street address (2026-09-12) — ✅ done, one idea deferred
|
||||
|
||||
Date pickers display `TT.MM.JJJJ`; storage untouched. New optional `street`
|
||||
CSV column feeds a full-address geocode attempt before falling back to
|
||||
city/PLZ. See [changelog.md](changelog.md).
|
||||
|
||||
**Deferred idea, not requested:** since the same person is often logged at the
|
||||
same city repeatedly, the address autocomplete could remember and suggest the
|
||||
last-used street for a selected city (the way selecting a city already
|
||||
auto-fills its PLZ). Would remove re-typing the same street each time, at the
|
||||
cost of a bit more state to reason about. Worth doing if re-typing the address
|
||||
turns out to be annoying in practice — not implemented now to keep the change
|
||||
minimal.
|
||||
|
||||
## Suggested order of remaining work
|
||||
|
||||
1. Deploy to the laptop: `git clone` + `./install.sh` (see [setup.md](setup.md)).
|
||||
|
||||
+36
-22
@@ -16,14 +16,16 @@ Workflow: build up a **queue** of entries, let them geocode in the background,
|
||||
then save the whole batch at once.
|
||||
|
||||
1. **Entry form** – date (a **calendar picker**, `ttkbootstrap.DateEntry`,
|
||||
pre-filled with today; typed input is accepted in `JJJJ-MM-TT` or
|
||||
`TT.MM.JJJJ` and **validated** – an invalid date is rejected with a dialog),
|
||||
city (autocomplete combobox), postal code (optional). Pressing `Return` in any
|
||||
field, or the **+ Hinzufügen** button, adds the row to the queue.
|
||||
The PLZ field only accepts digits.
|
||||
2. **Queue table** – shows date, city, PLZ, and a live coordinate column that
|
||||
updates from `⏳ wird gesucht…` to either `lat / lon` or `⚠ nicht gefunden`
|
||||
as the background geocoder works through the queue.
|
||||
pre-filled with today and **displayed as `TT.MM.JJJJ`**, e.g. `12.09.2026`;
|
||||
typed input also accepts `JJJJ-MM-TT` and is **validated** – an invalid date
|
||||
is rejected with a dialog), city (autocomplete combobox), postal code
|
||||
(optional, digits only), and **Straße** (street + house number, optional,
|
||||
second row of the form) — see [Geocoding behaviour](#geocoding-behaviour) for
|
||||
why it's worth filling in. Pressing `Return` in any field, or the
|
||||
**+ Hinzufügen** button, adds the row to the queue.
|
||||
2. **Queue table** – shows date, city, PLZ, street, and a live coordinate column
|
||||
that updates from `⏳ wird gesucht…` to either `lat / lon` or
|
||||
`⚠ nicht gefunden` as the background geocoder works through the queue.
|
||||
- Select a row and press <kbd>Delete</kbd>, use the **"Auswahl entfernen"**
|
||||
button, or click the `✕` cell.
|
||||
- Right-click a row for *Entfernen* / *Koordinaten erneut suchen*.
|
||||
@@ -38,10 +40,12 @@ then save the whole batch at once.
|
||||
|
||||
A table view of everything in `orte.csv`.
|
||||
|
||||
- **Columns:** Datum, Ort, PLZ, and **Karte** – a status column showing `✓` when
|
||||
the row has coordinates or a red `fehlt` when it doesn't (the whole row is red
|
||||
too). The raw lat/lon numbers live in the edit dialog, not this table.
|
||||
- **Search box** – live filter across date, city, and PLZ (substring match).
|
||||
- **Columns:** Datum, Ort, PLZ, Straße, and **Karte** – a status column showing
|
||||
`✓` when the row has coordinates or a red `fehlt` when it doesn't (the whole
|
||||
row is red too). The raw lat/lon numbers live in the edit dialog, not this
|
||||
table.
|
||||
- **Search box** – live filter across date, city, PLZ, **and street**
|
||||
(substring match).
|
||||
- **Sortable columns** – click a header to sort; clicking again reverses. The
|
||||
active column shows a ▲/▼ arrow. Default sort is by date, newest first.
|
||||
Sorting by **Karte** groups the rows without coordinates together.
|
||||
@@ -49,10 +53,11 @@ A table view of everything in `orte.csv`.
|
||||
coordinates yet. The hint line always shows the total count, plus how many are
|
||||
missing coordinates.
|
||||
- **Edit** – double-click a row (or *Bearbeiten*) opens a modal dialog to change
|
||||
date (calendar picker, validated) / city / PLZ. Two ways to fix coordinates:
|
||||
a "Koordinaten automatisch neu suchen" toggle re-runs geocoding after saving,
|
||||
or the **Breitengrad / Längengrad** fields let you type them in by hand
|
||||
(both empty = no map marker).
|
||||
date (calendar picker, `TT.MM.JJJJ`, validated) / city / PLZ / **Straße**. Two
|
||||
ways to fix coordinates: a "Koordinaten automatisch neu suchen" toggle re-runs
|
||||
geocoding after saving (now using the street if one is set), or the
|
||||
**Breitengrad / Längengrad** fields let you type them in by hand (both empty =
|
||||
no map marker).
|
||||
- **Right-click → Koordinaten suchen** – runs geocoding for that one row
|
||||
(handy for rows that failed the first time).
|
||||
- **Delete** – *Löschen* removes the selected row(s) after a confirmation
|
||||
@@ -87,13 +92,22 @@ Generated by `generate_map()`:
|
||||
|
||||
## Geocoding behaviour
|
||||
|
||||
`geocode_city(city, postal_code)` tries a series of queries in order and returns
|
||||
the first hit:
|
||||
`geocode_city(city, postal_code, street="")` tries a series of queries in order
|
||||
and returns the **first hit**:
|
||||
|
||||
1. `"<PLZ> <city>, Germany"` (only if a PLZ was given)
|
||||
2. `"<city>, Baden-Württemberg, Germany"`
|
||||
3. `"<city>, Hessen, Germany"`
|
||||
4. `"<city>, Germany"`
|
||||
1. `"<street>, <PLZ> <city>, Germany"` — only if a **street** was given (with
|
||||
PLZ if there is one, without it if not). This is what the optional Straße
|
||||
field is for: a full address geocodes far more precisely than a city name
|
||||
alone (a point in the right town vs. the right building).
|
||||
2. `"<PLZ> <city>, Germany"` (only if a PLZ was given)
|
||||
3. `"<city>, Baden-Württemberg, Germany"`
|
||||
4. `"<city>, Hessen, Germany"`
|
||||
5. `"<city>, Germany"`
|
||||
|
||||
If the street-level query (1) doesn't match anything in Nominatim (typo, house
|
||||
number Nominatim doesn't know, etc.), it **falls back automatically** to the
|
||||
same city/PLZ-level searches used when no street is given — a bad street never
|
||||
makes geocoding *worse* than before, only better when it resolves.
|
||||
|
||||
The regional bias is the `GEOCODE_REGIONS` constant (`Baden-Württemberg`,
|
||||
`Hessen`) near the top of `app.py`. Results are rounded to five decimal places.
|
||||
|
||||
Reference in New Issue
Block a user