Context
Two audiences share one support channel. Attendees ask about rooms, parking, Wi-Fi, their own sessions and their ticket. Organisers ask product questions: how to set up a coupon, why an email did not send, how to export check-ins. On event days a small team cannot keep up, and the questions repeat. The bot described here is the AI support and knowledge feature of the Eventify platform, and the pattern is the one we deliver as customer support AI for clients.
The challenge
- Grounding per tenant and per event. An answer about the wrong event is worse than no answer.
- Time-sensitive facts. The agenda changes during the day. A chunk indexed last night is already wrong.
- Hallucination has a cost. A confidently wrong room number sends people to the wrong floor.
- Prompt injection. Attendee bios, exhibitor descriptions and session abstracts are user-generated and all end up in the index.
- Knowing when to stop. Refund requests and complaints need a person.
- Change safely. Models and prompts change; the bot must not regress on event day.
What we built
Ingestion
Event content (agenda, venue information, FAQs, policies, exhibitor and sponsor listings, speaker bios and session transcripts) and the product documentation are chunked, embedded and stored with the tenant and event identifiers on every chunk. Re-indexing is incremental and triggered by content changes in the admin console.
Answering
- Guardrails. The question is classified by intent, screened for injection patterns, and personal data is handled according to the event's settings.
- Retrieval. The vector index is queried with the tenant and event filter applied in the query itself, using hybrid retrieval so exact names and room codes are not lost.
- Live tools. Anything personal or time-sensitive, such as my sessions, my ticket, or the current agenda, is fetched through the same API the apps use, so the attendee's permissions hold.
- Compose. The model drafts an answer that must cite its sources. A confidence gate decides between answering with sources and escalating.
- Handoff. Escalations go to the organiser's support queue with the conversation, the sources that were considered and the attendee context attached.
Organiser console
Organisers see the questions asked, the clusters that went unanswered, and can add an FAQ entry that is indexed immediately. The bot improves the event content as much as it answers from it.
Architecture
Key decisions
1. Filter at the index rather than in the prompt
Telling the model "only answer about event X" is a suggestion. Filtering the index query on tenant and event is a guarantee. The prompt never sees chunks from another event, so it cannot leak them.
2. Structured tools for anything that changes
Agenda times, room allocations and the attendee's own bookings come from live API calls with the attendee's own token. The model is told to prefer a tool result over an indexed chunk when both exist. Stale answers disappeared once this rule was in place.
3. "I don't know" is a feature
A confidence gate compares the answer against its sources. Weak support means a short honest reply and an offer to hand over. Organisers would rather see an escalation than a confident guess.
4. Treat every indexed chunk as untrusted
Exhibitor descriptions and bios can contain instructions aimed at the model. Retrieved text is wrapped as data, instructions inside it are ignored by design, and the output is screened before it is shown. The approach follows our prompt injection defence practice.
5. Citations are mandatory
An answer without a source is not shown. Citations link to the agenda item, FAQ entry or documentation page, so the attendee can verify in one tap and the organiser can fix the source if it is wrong.
6. Evaluate before every change
A golden set of questions and expected sources per event type runs on every prompt or model change. Regressions block the release. The same evaluation loop is described under LLMOps and monitoring.
7. One tool layer, many front ends
The tools the bot calls are the same ones the admin console uses, and the same layer is exposed through the Model Context Protocol so that an organiser can drive the platform from an AI assistant. Building the tool layer once paid off twice; see MCP server development.
Outcome
Repetitive questions are answered immediately with a link to the source, and the support team spends event days on the cases that need a person. Escalations arrive with context, so nobody repeats themselves. Organisers use the unanswered-question clusters to improve their event content before the next day. Model upgrades ship without surprises because the golden set catches regressions first.
What this means for you
Any support or knowledge assistant that serves multiple customers, or multiple departments, needs the same four things: a hard scope filter in retrieval, live tools for facts that change, mandatory citations and a graceful handoff. We build this as support copilots and enterprise RAG systems, and the secure RAG architecture page explains the controls in more depth.