The server exposes a GET /health endpoint that returns 200 OK when the service is running. Check it from the server or any machine with network access:
curl https://<your-server>/health
For account-scoped stats (number of synced lists and registered devices), use the authenticated status endpoint:
curl -H "Authorization: Bearer <auth-token>" https://<your-server>/dailyroundup/status
The healthcheck.py script in roundup-server/ performs a more thorough post-deployment check: it polls GET /health until the service responds, validates APNs JWT signing, and verifies Trello API connectivity for all configured accounts. It is run automatically by the Deploy code to sync server GitHub Actions workflow after each deployment.
GitHub Actions sends a notification when the Deploy code to sync server or Upload to App Store Connect workflow fails. No automated alerting is configured for runtime server errors; monitor the service by watching logs (see Server Logs) or polling GET /health.
When DD_AGENT_HOST is set, the server emits a focused set of low-cardinality DogStatsD metrics over UDP to a local Datadog Agent so feature/endpoint usage, latency, and outbound calls are visible in Datadog. Counters are incremented by 1 per event; the request-latency metric is a histogram sample per request. When DD_AGENT_HOST is unset the metrics client is a complete no-op and opens no socket, so development, tests, and CI are unaffected. Set DD_DOGSTATSD_PORT to override the default UDP port (8125). Both variables are documented in the Environment Variables table.
| Metric | Type | Tags | Emitted when |
|---|---|---|---|
dailyroundup.requests |
counter | endpoint (matched view name, e.g. api.sync, or unknown), method, status_class (2xx/3xx/4xx/5xx) |
Every HTTP request completes (Flask after_request hook) |
dailyroundup.request.duration |
histogram (ms) | endpoint (matched view name, e.g. api.sync, or unknown) |
Every HTTP request completes; the value is request wall-clock latency in milliseconds (Flask before_request/after_request hooks) |
dailyroundup.trello.calls |
counter | operation (logical Trello call, e.g. GET_cards_:id), status (success/rate_limited/timeout/auth_error/not_found/server_error/error) |
Each outbound Trello API call finishes |
dailyroundup.apns.notifications |
counter | type (conflict/duplicate/list_removed/card_reassigned/calendar_sync_failure/liveactivity/sync_push), status (sent/failed/permanent_error/skipped) |
Each APNs push attempt (per device), or once as skipped when APNs is not configured |
dailyroundup.geocoding |
counter | direction (forward/reverse), status (success/failure/timeout) |
Each Nominatim geocode during location sync |
dailyroundup.calendar_sync.errors |
counter | category (auth/reauthorization/notify) |
A Google access-token refresh fails terminally: reauthorization when the grant is revoked or expired and only the account owner can restore it, auth for every other terminal failure — retries exhausted, a rejection not worth retrying, or a malformed token response. notify is a different shape: the sync itself worked and a Trello card reporting it could not be raised |
The endpoint and operation tag values are deliberately templatized (object IDs are collapsed to :id) to keep tag cardinality a small finite set. No per-user, per-card, or per-token tags are emitted.
dailyroundup.request.duration is sent as a DogStatsD histogram, so Datadog automatically derives the dailyroundup.request.duration.avg, .95percentile, .count, and .max sub-metrics (among others). Chart .avg and .95percentile for typical and tail web-call latency, and use .count to confirm request volume.
The host deployment sets DD_AGENT_HOST=127.0.0.1 and DD_DOGSTATSD_PORT=8125 automatically via the Deploy code to sync server workflow, which writes them (as non-secret values) into roundup-server/.env. The systemd service loads that file, so a Datadog Agent listening for DogStatsD on 127.0.0.1:8125 receives the metrics after the next deploy and restart. If the Agent is not yet installed, the sends fail silently (logged at WARNING) and do not affect request handling.
The auth token in Settings is invalid or missing. Tap Connect to Server… and enter your bootstrap token to create a new account. If you already have a permanent token, expand “I already have a token” and paste it directly.
403 responses or no trello_webhook_secret configured warnings.The Webhook Secret in the iOS app does not match the value Trello is signing requests with. Update the Webhook Secret in Settings → Trello Configuration to match, then save.
Check that all four APNS_* environment variables are set in .env and the server has been restarted. The server logs a DEBUG message on startup if any are missing. Confirm APNS_USE_SANDBOX matches the build type (set to 1 for development builds, omit for App Store/TestFlight).
iCloud Keychain syncs the auth token automatically, but may take a few minutes. If the token does not appear, use “I already have a token” in Settings to paste it from your previous device.
A systemctl restart kills the gunicorn worker without awaiting detached background-job threads, so a sync/merge/import job can be cut off mid-run. The system self-heals (issue #393): the deploy workflow runs drain.py first to let in-flight sync jobs finish (bounded by DAILYROUNDUP_DRAIN_TIMEOUT), startup recovery in create_app flips any job still running at boot to error so clients stop polling and re-push, and the scheduler’s startup deep reconcile re-sends whatever was left partial. To confirm recovery ran after a restart, look for Startup recovery: marked N interrupted background job(s) as error and scheduler: deep reconcile (startup) in the journal. If drift persists, the daily deep reconcile imports untracked Trello cards, re-creates missing Reminders, and corrects completion mismatches with Trello as source of truth; it can be hastened by restarting the service. See Sync Internals — Self-healing and restart recovery.
The reorder is persisted and queued the moment the app returns, but the Trello writes happen afterwards, so a lag of a few seconds is normal (issue #491). If the board never catches up:
list-reorder drain: applied N list position(s) for account <id> in the journal. The variant ending M still queued after failing this pass is logged at ERROR and means those lists will be retried on the next scheduler tick.WARNING as list-reorder drain: failed to apply the queued position for sync list <id>. Trello auth errors and rate limits are the common ones; both clear on retry once the cause is fixed.list-reorder drain: could not read the lists on board <id> means the board’s composition could not be fetched, so no position could be chosen for it. Those lists stay queued and are retried — nothing is written to Trello from a board the server cannot see.list-reorder drain: Trello list <id> not found; dropping the queued position means that list no longer exists on Trello. The queued write is discarded — recreate the synced pair if the list is meant to be there.reorder_lists: another writer holds the sync lease is not a failure. The push deferred to whoever holds the lease, and the scheduler drains it on its next tick.skipping list-order drift check … still queued for Trello. That is expected during a drain; if it persists for more than 15 minutes the stand-down lifts on its own and drift correction resumes, which is the signal that a queued write is genuinely stuck rather than in flight.Running with DAILYROUNDUP_SCHEDULER_ENABLED=0 removes the retry and restart-recovery passes: the background thread started by the request is then the only thing that pushes a reorder, so a failure at that moment is not retried until the next reorder.
A list orders itself only when at least one of its two toggles is on:
move_due_to_top_enabled for due cards, move_pinned_to_top_enabled for 📌
cards. Both live on sync_lists and are set from the app’s per-list options.
list-order: reordered list <id> (N due, M pinned, K moved, reason=…) — at INFO for an event-driven reorder, and at WARNING when a reconcile pass had to fix drift, which means a webhook event was missed.list-order: failed to reposition card <id> on list <id> at WARNING is a Trello write that failed. The rest of the reorder still applies, and the next pass retries the card.list-order: reorder failed for list <id> at WARNING means the list’s cards could not be fetched at all — usually Trello auth or a rate limit.updateCard trigger one; label, attachment, and custom-field events deliberately do not, because nothing they change can affect due or pinned order.Pinned cards are ordered by card_action_index.pinned_at, recorded the first
time the server sees the card pinned and never rewritten. To inspect the block’s
order: SELECT trello_card_id, pinned_at FROM card_action_index WHERE
sync_list_id = '<id>' AND pinned_at IS NOT NULL ORDER BY pinned_at. A card that
should sit lower has an older recorded moment than expected; unpinning and
re-pinning it clears and re-records the moment, sending it to the bottom, as does
archiving and unarchiving it.
Destinations are per list. Check the list’s own row rather than the account:
SELECT name, move_due_target_list_id, move_remind_target_list_id FROM
sync_lists WHERE id = '<id>'. NULL means “keep the card on this list”, and a
list set that way is never moved regardless of the account’s own
move_due_target_list_id / move_remind_target_list_id — those govern only the
board’s unsynced lists. A card being moved from a list whose destinations are
both NULL is therefore on an unsynced Trello list, not on that synced list.
scheduler: failed to move card <id> at WARNING is a Trello write that
failed; the entry stays indexed and is retried on the backoff. A destination
that has since been archived or deleted shows up here, because it is only
validated when it is chosen.
Trello-set due dates stored before the timestamp normalization fix are repaired automatically, with no operator action.
Every process that opens the database runs a repair pass and logs Normalized fractional-second timestamps … for each table it rewrote — migrate.py and drain.py during the deploy, then the server itself at startup. A pass that rewrites nothing logs nothing.
Only the server may mark the backfill finished, and only after a pass that finds nothing left. That is deliberate: the deploy’s migrate.py and drain.py steps both run while the outgoing server is still writing in the old format, so sealing there would strand whatever it writes next. Once sealed, an applied_data_migrations row records it and the scans stop for good.
Tasks whose repair is already carried by a queued change get no extra change; the rest get one update change each, so the affected reminders recover on the next sync.
The migration described below is not implemented yet. It is tracked in issue #546. Until it ships, the safe answer for an account whose destination calendar already holds Sync Assistant events is to leave
Sync calendarsoff. Sources can be configured in the meantime; nothing is copied while the switch is off.
DailyRoundup finds the copies it owns by the dr identity tag and description
marker it stamps on them. Events written by Sync Assistant carry cca tags
instead, and the adoption paths in calendar_sync.py only recognise dr. A
sync that runs against a calendar full of cca-tagged events therefore
recognises none of them, treats every event as new, and writes a second copy of
each, duplicating the calendar.
An account with no existing calendar events has nothing to migrate and can enable syncing now.
Once the migration exists, the safe order will be: configure the sources with syncing off, run the migration, then turn syncing on. This section should be rewritten at that point to describe what the script actually does, rather than what it was expected to do.
This warning previously lived in the app, under the Sync calendars toggle. It
was removed from there because it describes a one-time cutover rather than what
the switch does, and would otherwise be shown permanently to users who never
had anything to migrate.
sync_tasks.url and .url_name are dropped on the first migrate.py run that follows the #482 deploy. They backed a single-URL sync that shipped briefly in March 2026 and was reverted without removing the schema, so a database that ran it in that window may still hold values — and for part of that window the same paths also deleted the URL attachment from the Trello card, so a stored value can be the only surviving copy of that URL.
The migration therefore salvages before it drops. Any row still carrying a value is logged at WARNING before the columns go, one line per row, with the sync task id, the Trello card id, and the card title alongside the URL:
Retiring sync_tasks.url/.url_name (issue #482); 2 row(s) still hold a value, listed below. Recover from this log if the URL exists nowhere else.
retired url for sync_task 4f3a… (card 65b1…, 'Renew passport'): url='https://…' url_name='Form'
This lands in the GitHub Actions deploy log, not journald. deploy.yml runs migrate.py over SSH, so its output is captured by the workflow run, and the drop is one-time — the server’s own startup never re-emits it. Read it from the Actions run for the deploy that shipped #482, and save anything you need before that run’s retention expires. journalctl will not have it.
Two consequences worth knowing:
log_filter’s redaction does not cover it, and it exists because recovery needs the URL verbatim. Treat the deploy log accordingly.The drop is skipped, with the columns left in place, on SQLite older than 3.35.5 — DROP COLUMN arrived in 3.35.0, but 3.35.5 fixed defects in it that could corrupt the database file — and on any OperationalError, which covers a schema left half-dropped by an interrupted run and the gunicorn workers racing the migration. In each case the log says so and the next clean run finishes the job. No salvage report is emitted on those paths, since nothing was destroyed; do not read their silence as “no values were carried”.
sync_tasks carried eight hash columns — title_hash, description_hash, due_date_hash, start_date_hash, completion_hash, position_hash, priority_hash, and location_hash — and sync_locations a ninth copy of location_hash. They were meant for server-side change detection that was never built: no production comparison against any of them has ever existed, and seven were never written at all. Change detection belongs to the client, in SyncTaskHashing (issues #517 and #521).
This was retired over two releases. #593 stopped the server reading and writing them; #598 drops the columns on the first migrate.py run that follows its deploy.
The split was required, not tidiness. deploy.yml runs migrate.py at its “Run database migrations” step, well before “Restart service”, so a migration takes effect while the outgoing process is still serving traffic with the previous release’s code. Before #593 that code’s _row_to_sync_task read title_hash and five siblings without a presence guard, so dropping them in the same release would have made every sync-task read in the outgoing process raise IndexError until the restart — including the jobs the drain step then waits on. The #482 retirement above was safe from this only because its mapper already read row["url"] if "url" in keys else None.
The #598 drop logs one line per table at INFO:
Dropped retired sync_tasks hash column(s): title_hash, description_hash, … (issue #598)
Dropped retired sync_locations.location_hash (issue #598)
These normally land in the GitHub Actions deploy log rather than journald, for the same reason as #482’s: deploy.yml runs migrate.py over SSH, and the drop is one-time.
Two cases where they do not appear there:
OperationalError path — a transient refusal, or a schema left half-dropped by an interrupted run. The migration logs the failure at WARNING and leaves the columns; the next successful run finishes the job, and if that run is a create_app boot rather than migrate.py, the INFO lines land in journald instead.create_app boots against the same library on the same host, so it skips too, on every boot, until SQLite is upgraded. What appears instead is one WARNING per table — the sync_tasks one naming all eight columns, the sync_locations one naming its single column — each giving the SQLite version that caused the skip.A database created after #593 never had the columns and logs nothing at all; silence is the normal case there, not a failure to run.
Three things for operators:
location_hash was a digest of fields the sync_locations row still carries — one that stopped being updated in #593, so on a row whose location changed in between it described the previous value. There is no salvage log because there is nothing to recover.DROP COLUMN arrived in 3.35.0, but 3.35.5 fixed defects that could corrupt the database file — and on any OperationalError, which covers a schema left half-dropped by an interrupted run. In each case the log says so, names only the columns actually still present, and the next clean run finishes the job.priority_hash and location_hash — so against a database this drop has touched it raises IndexError on every sync-task read. That is now every database: the drop removes the columns from long-standing production databases as well as from ones created since #593. Before #598 ran, a database predating #593 was still safe to roll back to; it no longer is. Roll forward, or recreate the database on the older release.One Trello list pairs with exactly one sync list, enforced by
idx_sync_lists_trello_list_id_unique. Every process that opens the database
builds that index if it is missing, and a database written before it existed
may hold duplicates the index cannot be built over.
Those are retired first. The row kept is the one with the strongest claim to
the list, ranked by a paired Reminders calendar, then task count, then age —
deliberately not the most recent insert, since a duplicate arises when a second
import lands on a list that was already set up, so the newer row is usually the
accident. It is a best guess about rows nothing else distinguishes, so check
the log rather than assuming it chose correctly. Every other row is deleted
along with its tasks, locations, and queued changes, and logged at WARNING
before the delete:
Retiring duplicate sync list 74ee… ('Rovo RCA', account 8f2c…) for Trello list 69ae…: it was paired with Reminders calendar BC10… and owned 3 tasks. Keeping c5b7… ('Rovo RCA'), which has the stronger claim to the list. Recover from this log if the pairing exists nowhere else
Where to read it depends on which process got there first. deploy.yml
runs migrate.py over SSH, so on a deploy the lines land in the GitHub Actions
run; a server that starts without one writes them to journald. The index is
built once, so whichever process wins never re-emits them. The device that held
the retired pairing keeps its Reminders calendar, which simply stops updating —
the calendar id in the log line is what identifies it.
A retirement that fails is logged at ERROR on its own and the rest are still
attempted. That line says the rows named in the Retiring … lines above it are
still present, so those are a record of what would have been destroyed rather
than what was:
Could not retire the duplicate sync lists for Trello list 69ae…; the rows named above are still present and were not deleted
If any duplicate remains afterwards the index is left unbuilt and the process still starts, naming the ids that are still duplicated — not the ones found before the pass:
Could not enforce unique Trello list pairings: sync_lists still holds duplicate trello_list_id rows (['69ae…']). Running with application-level checks only
A database this pass cannot read at all — locked, or an IO error — is reported the same way and the process still starts:
Could not enforce unique Trello list pairings; the index was not built. Running with application-level checks only
Sync is unaffected in either case — the list endpoints refuse a duplicate
pairing on their own, and get_sync_list_by_trello_id warns on every read that
finds one — but the guarantee is down to application code until the cause is
cleared and the process restarted.
A database with no duplicates logs nothing. That is the expected case: production had none when this shipped.
The server logs to stdout/stderr, captured by systemd’s journal. Logs are written in this format:
2026-02-24T14:32:01 INFO [dailyroundup.webhook] Recorded 'create' change for Trello card abc123
Fields: YYYY-MM-DDTHH:MM:SS LEVEL [logger.name] message
Reading logs requires SSH access to the server and the journalctl tool (included with systemd). Replace <username> with the system user running the service:
# Stream live logs
journalctl -u dailyroundup@<username> -f
# Last 100 lines
journalctl -u dailyroundup@<username> -n 100
# Logs from the past hour
journalctl -u dailyroundup@<username> --since "1 hour ago"
# Errors only
journalctl -u dailyroundup@<username> -p err
# Save today's logs to a file
journalctl -u dailyroundup@<username> --since today > ~/dailyroundup-$(date +%F).log