Build vs buy: should you build your own AI chatbot over Slack, Notion and GitHub?
What building an internal RAG chatbot over Slack, Notion and GitHub involves: connectors, permissions, sync, evals and costs, and when building makes sense.
The KnowWhyy team · · 9 min read
Most engineering teams that have tried an LLM API have had the same thought: "We could index Slack, Notion and GitHub ourselves and ask questions over it." A weekend prototype proves it can work. You embed a few channels, put the chunks in a vector store, and the first answers look great.
The hard part is everything a demo skips: keeping data in sync, respecting who can see what, knowing when to say "I don't know", and keeping it running after the engineer who built it moves on. This guide covers what building involves, what it costs at current API prices, and when building beats buying.
What a demo skips
A demo answers questions from a static snapshot, for one person, with no access rules. A tool your whole team relies on needs all of this:
Layer
Demo
Production
Data access
One token, a few channels
OAuth per source, token refresh, rate limits
Freshness
One-off export
Backfill, incremental sync, edits and deletions
Permissions
Everyone sees everything
Per-user filtering that matches each source
Retrieval
Vector search
Hybrid search, reranking, thread and PR context
Answers
Whatever the model says
Ask your first question in minutes.
Create a workspace, connect a tool and choose what gets indexed. The free plan includes GitHub, Slack and Notion.
Citations, abstention, checks for unsupported claims
Quality
"Looks good"
An eval set you run on every change
Operations
Runs on a laptop
Monitoring, retries, on-call, security review
1. Connectors and OAuth
Each source has its own auth model, scopes and limits, and they change.
Slack. A bot token reads any conversation the bot is a member of, given the relevant *:history scope (conversations.history). Rate limits depend on how your app is distributed. Slack's May 2025 change cut conversations.history and conversations.replies to 1 request a minute and 15 messages per request for commercially distributed apps outside the Slack Marketplace. Internal, customer-built apps keep 50+ requests a minute and up to 1,000 messages per request. An internal tool is fine; sharing it with other companies means Marketplace review. Slack also offers a Real-time Search API that queries Slack without storing data on your servers.
GitHub. A GitHub App installation gets 5,000 requests an hour, plus 50 an hour for each repository over 20 and each organization user over 20, up to 12,500 (15,000 on GitHub Enterprise Cloud). Secondary limits cap you at 100 concurrent requests and 900 points a minute for REST endpoints. Backfilling years of pull requests and review comments takes careful pacing.
Notion. The API allows 180 requests a minute per connection on most plans and 600 on Business and Enterprise, with a separate per-workspace limit shared across connections. More important for design: an internal connection only sees pages that someone has explicitly shared with it. Miss a parent page and its whole subtree is invisible, with no error.
2. Sync, edits and deletions
A stale index gives confidently wrong answers. A deletion that never reaches your index is worse: someone removed a message for a reason, and your bot keeps quoting it.
Slack sends a message_deleted event with the deleted_ts of the removed message. You need to listen for edits and deletions, not just new messages.
Notionwebhooks include page.deleted (moved to trash), page.undeleted, page.moved and page.content_updated. Events are signals only, so you fetch the page again to see what changed, and high-frequency events are aggregated.
GitHub sends webhooks for issues, issue comments, pull requests and reviews, repositories and team membership.
Webhooks get missed, so you also need a nightly reconcile job that compares your index against each source and deletes anything no longer returned, plus alerts when sync falls behind or a token is revoked.
3. Permissions for every user
This is where most internal builds go wrong, and it's the one you can't fix later with a better prompt.
Your connectors usually authenticate as a bot that sees more than any single employee. The bot is in the private #exec channel; your intern isn't. So every chunk needs an access list, and every query must filter by the asker's identity before results reach the model. Never put restricted text in the prompt and ask the model not to repeat it.
With Postgres and pgvector, the core query looks like this:
-- Only retrieve chunks the asker may see
SET hnsw.iterative_scan = strict_order; -- keep filtered queries returning enough rows
SELECT c.id, c.source_url, c.text
FROM chunks c
WHERE c.acl && $1::text[] -- asker's user id + group/channel ids
ORDER BY c.embedding <=> $2 -- cosine distance to the question
LIMIT 20;
pgvector's docs explain the SET line: with approximate indexes, filtering happens after the index scan, so a restrictive filter can return too few rows unless you enable iterative scans (available since 0.8.0). Access lists must also stay current: when someone leaves a private channel or GitHub team, the index should reflect it within minutes.
Test it: ask, as a user without access, a question only a private source can answer. If the bot answers, you have a data leak, not a quality bug.
4. Chunking and embeddings
How you split content decides what retrieval can find.
Slack: chunk by thread, not by message. The decision is usually reply 23, and it means nothing without the question that started the thread.
GitHub: keep a pull request's description, review comments and linked issue together, with the merge date and author.
Notion: split by heading, and keep the page title and breadcrumb on every chunk.
Store metadata (source, author, date, URL) on every chunk. You'll need it for citations, filters and recency.
Embedding is the cheap part. As of November 2026, OpenAI lists text-embedding-3-small at $0.02 per million tokens and text-embedding-3-large at $0.13. Voyage AI lists voyage-4-lite at $0.02, voyage-4 at $0.06 and voyage-4-large at $0.12, with the first 200 million tokens free per account. For an illustrative 50 million tokens of history, a full pass costs about $1 with text-embedding-3-small or $6.50 with text-embedding-3-large. The real cost is re-embedding safely whenever you change models or chunking.
5. Retrieval quality
Pure vector search misses exact terms: error codes, ticket keys, service names, people. Most teams end up with:
Hybrid search. pgvector's README suggests combining it with Postgres full-text search, merged with Reciprocal Rank Fusion or a cross-encoder.
Reranking. A reranker reorders the top candidates. Voyage's rerank-3 is $0.05 per million tokens on the same pricing page.
Context expansion. When a reply matches, pull in the whole thread; when a review comment matches, pull in the PR description.
Recency and source weighting. A merged PR from last month usually beats a Slack guess from two years ago.
6. Citations and "I don't know"
An internal bot that guesses is worse than no bot, because people stop checking. Two controls help most.
First, make citations and abstention part of the prompt contract:
You answer questions about our team's work using ONLY the sources below.
Each source has an id like [S3].
Rules:
- Every factual sentence must end with one or more source ids.
- If the sources don't contain the answer, reply exactly:
"I couldn't find this in the indexed sources." Then list the closest sources.
- Never use outside knowledge about our company.
- If sources disagree, say so and cite both, newest first.
Sources:
[S1] <slack thread, #payments, 2026-03-14, url>
[S2] <PR #812 description + review, merged 2026-03-18, url>
...
Question: <question>
Second, check the output in code: reject any cited id that wasn't in the retrieved set, and flag sentences with no citation.
7. Evals
Without an eval set, every prompt or model change is a guess. Start with real questions from your own Slack, each with the source a good answer must cite:
- id: payments-retry-why
question: "Why do we retry failed card charges only once?"
must_cite: ["github.com/acme/api/pull/812"]
asker: alice # has access
- id: exec-offsite-leak
question: "What was decided at the exec offsite?"
expect: abstain
asker: intern # no access to #exec
- id: not-in-sources
question: "What's our parental leave policy?"
expect: abstain # HR docs are not indexed
Track retrieval recall, citation correctness, correct abstention and permission leaks, and run the set on every change.
8. Cost drivers
Model tokens are usually the smallest line. Here are current list prices for a few models you might use, per million tokens:
A worked example, with assumptions you should replace with your own: each question sends 8,000 input tokens (instructions plus retrieved chunks) and gets 500 output tokens back. On Claude Sonnet 5.5 that's $0.016 + $0.005 = about $0.021 a question, or about $105 for 5,000 questions a month. On Claude Haiku 5.5 it's about $0.001 a question. Prompt caching and batch processing reduce this further; Anthropic lists a 50% batch discount.
The costs that actually decide build vs buy are harder to see:
Engineering time for connectors, the permission model, evals and the UI.
Infrastructure: a database with vector search, sync workers, a queue, monitoring.
Maintenance: API changes (Slack's 2025 rate limits; Notion's 2025-09-03 API version, which replaced some database webhook events with data source events), model deprecations, and broken syncs someone has to notice.
Security review: encrypted token storage for every source, and your security team's questions.
A realistic build checklist
## Internal AI chatbot: build checklist
Connectors
- [ ] Slack app (internal), scopes reviewed, private channels opt-in only
- [ ] GitHub App with read-only permissions, rate limit pacing
- [ ] Notion connection, documented list of shared root pages
- [ ] Token storage encrypted at rest, rotation and revocation handled
Sync
- [ ] Resumable backfill per source
- [ ] Handlers for create, edit and delete events
- [ ] Nightly reconcile and alerts on sync lag
Permissions
- [ ] ACL on every chunk, filtered before the model sees results
- [ ] Membership changes propagate within minutes
- [ ] Leak test in the eval set
Retrieval and answers
- [ ] Thread- and PR-aware chunking with metadata
- [ ] Hybrid search and reranking
- [ ] Citations validated in code; abstains when sources are silent
Operations
- [ ] Eval set of real questions, run in CI
- [ ] Named owner and on-call
- [ ] DMs and source code excluded unless explicitly decided otherwise
When to build and when to buy
Build makes sense when...
Buy makes sense when...
Your sources are unusual (internal systems no vendor connects to)
Your sources are mainstream: Slack, GitHub, Notion, Jira
Data can't leave your infrastructure, and self-hosting a vendor or open-source tool isn't allowed
A vendor's security posture passes your review
You have a platform team that will own it for years
The builder would be one engineer, part-time
The chatbot is part of your product, not an internal tool
You want answers next week, not next quarter
There's also a middle path: run an open-source platform such as Onyx, which has an MIT-licensed community edition you can self-host, and own the operations rather than the code. Our Glean alternatives roundup compares that with hosted options, and giving AI coding agents project context covers a related problem.
Where KnowWhyy fits
KnowWhyy is the "buy" option for this exact stack. It connects GitHub, Slack and Notion with read-only scopes, indexes only the repositories, channels and pages you choose, and answers only from sources the asker is allowed to see, with every claim linked to its source. If the sources don't contain the answer, it says so. Slack DMs and source code are never indexed, organizations are isolated with PostgreSQL row-level security, and tokens are encrypted with AES-256-GCM. You ask in the KnowWhyy web app, not in Slack, and GitLab, Jira and Confluence are planned, not available. The free plan covers 2 members; see an AI chatbot for GitHub, Slack and Notion.
The short version
A RAG prototype over Slack, Notion and GitHub takes a weekend; a production tool needs connectors, sync with deletions, per-user permissions, citations, evals and an owner.
Check each API's rules first: Slack limits non-Marketplace commercial apps to 1 history request a minute (internal apps keep 50+), GitHub App installations get 5,000 to 12,500 requests an hour, and Notion connections only see pages shared with them.
Permissions must be enforced at retrieval time, before anything reaches the model, and tested with a leak check.
Embedding and model tokens are cheap at November 2026 prices (from $0.02 per million tokens to embed); engineering and maintenance time are the real cost.
Build when your sources are unusual, your data can't leave your infrastructure, or the chatbot is part of your product. Buy when your stack is mainstream and you want it working this month.