Context
Attendees want notes without taking them. Accessibility needs captions. Organisers want a searchable archive and material for the follow-up email. Speakers want their talk represented accurately. The AI for events feature set in Eventify started with this one, because it creates the content that the support bot and search later depend on.
The challenge
- Venue audio. Room microphones, crosstalk, applause and the occasional fire alarm.
- Names and jargon. Speaker names, sponsor names and product names are exactly the words a general model gets wrong, and exactly the words that matter.
- Latency against accuracy. Live captions must appear within seconds; a published transcript must be right.
- Languages. Multi-track events often mix languages across rooms.
- Consent and retention. Not every speaker agrees to be recorded, and not every organiser wants audio kept.
- Cost. A multi-track event can produce many hours of audio a day.
What we built
- Capture. Audio from the AV desk feed where available, or a phone or laptop in the room as the fallback.
- Live pass. Streaming speech-to-text with a custom vocabulary generated from the agenda: speaker names, companies, session titles, sponsor and exhibitor names. Captions appear in the app a few seconds behind the speaker.
- Post pass. After the session, the full recording goes through a higher-accuracy batch model with the same vocabulary, then speaker-turn detection, punctuation and cleanup.
- Summaries. A language model produces the main points, decisions, and the questions and answers from the Q&A, each labelled as AI-generated and held for organiser review before publication.
- Indexing. Transcript chunks are embedded into the event's vector index, which powers in-app search and the support bot.
- Controls. Per-session enablement, a consent flag on the speaker record, a language per session, and a retention setting per event.
Architecture
Key decisions
1. Two passes instead of one compromise
A single model is either fast enough for captions or accurate enough to publish, rarely both. Running a streaming model live and a batch model afterwards costs a little more per session and removes the compromise entirely.
2. Vocabulary from the agenda
The largest accuracy improvement did not come from a bigger model. It came from feeding the recogniser the names already in the event: speakers, sponsors, exhibitors, session titles and product names. The list is generated automatically from the agenda, so it is never stale.
3. Route by language rather than guessing
Each session declares a language, and the pipeline chooses the model accordingly. Mixed-language sessions are flagged for review rather than silently mangled. The routing layer is the same approach we describe under speech-to-text services.
4. Summaries are labelled and reviewed
Generated summary points carry an AI label and are not published until an organiser approves them. The model is asked to quote rather than paraphrase for anything that looks like a commitment or a figure.
5. Consent lives in the data model
A consent flag on the speaker record decides whether capture runs. Retention is an event setting. Both are enforced in the pipeline, not in a policy document.
6. Pay only for the sessions you switch on
Organisers switch transcription on per session. Audio is processed once, and the expensive summary step runs after the accurate transcript exists, never on the live stream.
7. Evaluate on real recordings
We keep a set of session recordings with hand-corrected transcripts and measure every model or vocabulary change against them before it ships, using the discipline in fine-tuning and evals.
Outcome
Attendees follow sessions with captions and search transcripts afterwards. Organisers publish reviewed summaries in follow-up emails instead of asking speakers for slides. The transcript index became the richest source for the support bot, which can now answer "what did the keynote say about pricing" from the event's own content. Names and sponsors are spelt correctly, which sounds minor until a sponsor reads the transcript.
What this means for you
The same pipeline applies to meetings, webinars, lectures, call recordings and field reports: capture, a fast pass where latency matters, an accurate pass for the record, domain vocabulary from the data you already hold, and reviewed summaries that feed a knowledge base. Our meeting intelligence page and the speech-to-text glossary entry are good next steps.