Club Sentiment maintains a known set of team-to-subreddit mappings for clubs with established supporter communities. When a known mapping exists, that subreddit is searched first.

Additional candidate subreddit names are also generated from the team slug, normalized club name, and common naming variations.

The Reddit collector first reads the club subreddit’s recent RSS feed. This provides recent post titles and summaries without requiring a Reddit API key.

Each item is stored with metadata such as source kind, publication time, subreddit name, post id, and whether the post appears to be a match thread.

After fetching the freshest posts from club subreddits, the system prioritizes likely match threads and recent team posts with commentable post ids. It then fetches top comments from a limited number of those posts.

Match threads are allowed a higher comment cap than ordinary posts because they are often the highest-value expressions of real supporter mood immediately before, during, and after matches.

If the direct team subreddit is weak, missing, or uncertain, the system searches Reddit’s subreddit directory for communities that resemble the club’s name and football context.

Candidate subreddits are scored using their display name, title, query-term overlap, and basic community size signals.

If club-specific subreddit evidence is still thin, the system searches broader football communities such as league-level or general-football subreddits for recent posts mentioning the club.

This allows the platform to capture relevant discussion even when a club’s own subreddit is inactive or very small.

Reddit evidence is bucketed into team, discovered, and fallback categories. Within those buckets, evidence is sorted by source type and freshness, then de-duplicated at the text level.

This preserves the highest-value direct supporter evidence while still allowing broader football discussion to supplement sparse clubs.

For selected clubs, Club Sentiment uses club-specific X query overrides that combine exact club names, major hashtags, and official or highly associated handles.

For clubs without an explicit override, the system builds a simpler search around the club name and a normalized hashtag form. Ambiguous clubs receive extra football-context terms to reduce drift.

Before spending search calls on full result collection, the system checks the recent tweet count for the query. If the count falls below a configured threshold, X evidence is skipped for that run.

This avoids wasting usage on clubs or days where X is unlikely to add meaningful signal.

When a topic shows enough recent activity, the system uses X’s recent-search endpoint to fetch the newest relevant posts. For high-volume clubs or major stories, a second page may be requested.

This keeps the collector efficient: it expands when the topic is busy, but remains conservative when the topic is quiet.

X posts are not used in raw API order alone. The system ranks them using a hybrid of freshness and engagement. Extremely fresh posts are favored, while public engagement helps separate meaningful supporter reaction from low-signal chatter.

The output is then de-duplicated so repeated phrasing or near duplicates do not overwhelm the day’s sample.

The X collector inspects usage and rate-limit information and slows down when necessary. This protects system stability and helps preserve the source as a sustainable daily signal rather than a source that works only on some days.

The YouTube collector supports multiple football-specific query templates (for example match reaction, fan reaction, and post-match analysis) around the club’s search name.

In production, query expansion is currently constrained for quota control, so only a limited subset of those templates may run on a given scrape.

Before comments are collected, candidate videos are screened using the video title and description. The system checks whether the club is actually being discussed and whether the surrounding context is recognizably football-related.

Ambiguous club names are held to a stricter standard and must show both club identity and football context.

Once relevant videos are identified, the system collects top level comments from a limited number of the most recent eligible videos. Comments are trimmed to a bounded length and de-duplicated.

YouTube is therefore not being used as a raw video crawler. It is being used as a football fan-reaction sampler.

YouTube search calls are quota-expensive relative to comment fetch calls, so the collector is intentionally selective. Only a subset of clubs are enabled by default for YouTube, with a focus on clubs and situations where YouTube adds the strongest signal.

Club Sentiment uses a recent Google News RSS search and limits attention to the last seven days. For ambiguous clubs, football context is injected into the query itself.

The collector reads both headlines and article summaries, strips markup, removes obvious junk, and stores the resulting snippets as candidate evidence.

News evidence is filtered for football context and common low-value markers such as betting-style language, live-update pages, or clearly non-club topics.

This makes the news layer more useful as a media-tone signal rather than a broad web-search feed.

News snippets are ranked with a strong recency bias. The aim is to measure the club’s current narrative environment, not to mix last week’s framing with stories that are already obsolete.

For selected ambiguous clubs, the system defines strong terms, football-context terms, and excluded terms. A snippet can be retained because it strongly identifies the club, or because it mentions the club alongside football context.

Snippets that match known non-football ambiguity patterns are excluded.

When enabled, a second layer sends batches of snippets to Gemini and asks the model to classify whether each snippet is truly about the football club in question.

This is used selectively, especially for ambiguous clubs, to keep cost under control while improving relevance quality.

The system is intentionally designed not to collapse coverage when filtering becomes too strict. If evidence volume falls below the minimum threshold after full filtering, it retries with rules only. If evidence is still too thin, it falls back to raw source evidence.

This preserves continuity while still preferring cleaner evidence whenever possible.