Flow Docs

ig-event-quality-and-delivery

5 min read

Summary

The Instagram collector had been running since July — 442 API calls across 275 monitored pages, events found and cached. event_imports contained 175 rows, every one of them from ert_open_data and not one from Instagram.

The pipe was never broken. The content was not worth sending.

What the collector was producing

Real values from the production cache:

title:    "Quando abbiamo detto ci vediamo presto intendevamo presto"
title:    "Un�altra settimana insieme tra gli alberi del nostro"
title:    "Ultimo giro di giostra"
location: ""        image_url: None        category: "musica" (the default)

Those are first lines of captions, not event names. One carries mojibake. The venue is empty on every single one.

Note

Uploading to Flow was a dashboard button, and nobody ever pressed it. That was the correct instinct: pressing it would have filled the app with captions.

The three defects

1. The title was “the first line with two words”

_short_title walked the caption and returned the first line with at least two words, skipping date-only lines. A page that opens with a greeting therefore produced the greeting as the name of an event.

ig_quality.best_title scores candidate lines instead of taking the first one, over the first ten lines only:

SignalWeight
ALL CAPS or Title Case — how flyers write names+3
contains an event marker (night, dj set, aperitivo, w/, b2b…)+3
2–6 words+2
prose opener (quando, grazie, ci vediamo, ultimo…)−5
later in the caption−0.3 per line

Below zero it returns nothing, and the post goes to review rather than being published under an invented name.

It also strips emoji from both ends and a trailing time, so "APERITIVO IN CORTE dalle 18:30" becomes "Aperitivo in Corte" rather than carrying a clock into the title.

Measured on the real captions:

BeforeAfter
Quando abbiamo detto ci vediamo presto…rejected
Grazie a tutti per ieri serarejected
Ultimo giro di giostraOffclub w/ Marco Faraone
🔥 TECHNO BUNKER 🔥Techno Bunker
Un�altra settimana insieme…Aperitivo in Corte

2. The venue was thrown away

location was empty on every collected event, because the rule was “the caption, or nothing”. But a venue’s own page rarely names itself in its own caption — everyone reading it already knows where they are.

The page identity is a fact about the source, so it is now the third fallback after Instagram’s location tag and the caption:

castello_di_carpineti  ->  Castello Di Carpineti
italghisa.re           ->  Italghisa
arci_tunnel            ->  Arci Tunnel

Flagged with venue_from_page so a reviewer can tell a derived venue from a stated one. It lands in a staging row a human confirms — which is what the staging table is for.

Warning

venues in Flow is mostly ERT-derived addresses (“Piazza Garibaldi 9”), not named places. Matching handles against it would fail for almost everything, which is why the handle proposes a venue rather than looking one up.

3. Mojibake

repair_mojibake undoes the UTF-8-read-as-Latin-1 round trip, and only keeps the result when it produces fewer suspicious characters — a blind decode would corrupt captions that were fine. A lone replacement character between two letters becomes an apostrophe (Un�altra → Un'altra); anywhere else it is left alone rather than guessed at.

Delivery, and its gate

event_puller now hands each collection round to flow_uploader.upload_auto(), which sends only what clears AUTO_UPLOAD_MIN_CONFIDENCE (0.65). Everything else stays collected, queued and visible in the dashboard — it simply does not travel on its own.

confidence scores the finished event, not any one field:

title present+0.25
title contains an event marker+0.10
title reads as prose−0.30
date+0.25
time+0.15
venue+0.15
image+0.10

It populates event_imports.confidence, a column that has been in the schema unused, so the review queue can sort by “worth a look” rather than by arrival time. The ingest endpoint already accepted the field (import-events line 182); no backend change was needed.

Note

The gate is deliberately strict because the two mistakes are not symmetric. A missed event waits in the dashboard until someone reviews it. A wrong one is a user opening Flow and finding a caption where an event should be.

End-to-end on real captions: 3 of 4 cleared the gate, the fourth being the “grazie a tutti” recap, held with no title. A cleared payload:

{ "title": "Offclub w/ Marco Faraone", "source": "instagram",
  "source_handle": "@italghisa.re", "date": "2026-08-09T23:00:00",
  "end_date": "2026-08-10T04:00:00", "venue_name": "Italghisa",
  "confidence": 0.8 }

Note the five-hour duration — a real night out, against the ERT imports that run for 11,000 days.

A bug worth recording

The first version of _TRAILING_TIME did not match anything. The compiled pattern contained \x08 — an actual backspace character — where \b should have been: the patch script wrote it inside a non-raw Python string, so \b was interpreted as an escape before it ever reached the file.

The regex worked perfectly when tested in isolation and failed inside the module, which is exactly the shape of bug that costs an hour. Fixed by writing the replacement with chr(92) + "b" and asserting on the compiled pattern afterwards.

Files

File
pipeline/ig_quality.pynew — repair, title scoring, venue from handle, confidence
pipeline/ig_graph_client.py_extract_event uses all four
pipeline/flow_uploader.pyupload_auto + confidence in the payload
pipeline/event_puller.pydelivery hooked to the collection round

Open Items

Warning

tools/ is not under version control. The project root is not a git repository, so everything described here exists only on disk. This was already flagged when the collector was built; it now holds real logic and a cloud cron would run from it. Versioning it is the first thing to do.

  • Images: 11 of 166. The provider returns a post picture and _best_image exists, but almost nothing arrives. Worth tracing before launch — an event discovery app without images looks broken even when it works.
  • Category is a keyword scan and defaults to musica for everything. It should resolve against the real 34-row taxonomy.
  • The confidence thresholds are a first guess. Watch the first weeks of the review queue and move the line where the data says.

business-plan — the collection engine and its credit budget target-study — why nightlife supply comes from IG and not open data

Graph View