Course outline
AcademyModule 8 · Source collection and practical workflows
Collect feeds, news, and market data
What you'll learn
- you can explain the three ways live data enters your graph: look up now, follow an entity, and standing feeds
- you can trace how a feed works under the hood: schedule, fetch, dedup, then a live/ note or a numeric observation
- you can use the titlebar live window to choose when minute-scale collection runs, and tell live feeds apart from scheduled (hourly/daily) ones
- you can turn on a source from the catalog or paste your own RSS feed, and follow an entity for news
- you can track a stock or crypto price as a sparkline and watch for a keyword across every feed
Everything you've built so far is yours: notes you wrote, files you imported, a graph the agent reasons over.
But the world outside keeps moving. A competitor cuts prices, a company files an 8-K, a stock gaps down, a story breaks. A knowledge graph that only knows what you typed last week is always a step behind.
Live data closes that gap. Consilience pulls news, filings, research, and prices straight into your workspace, where they become ordinary notes and get extracted into the same graph. This lesson is the last ingredient before the recipes ahead: the three ways that happens, and exactly how a feed works underneath.
- Live feed
- A source Consilience checks on a schedule (news, filings, prices…). New items are recorded into your workspace automatically.
- Live window
- A per-workspace switch for when minute-scale feeds poll. It lives on the titlebar's antenna button — off, 30 min, a few hours, or always. Hourly and daily feeds are unaffected.
- live/ folder
- Where recorded feed items land as markdown notes. Append-only, timestamped, and extracted into your graph like any note you write.
- Follow
- A one-click subscription on an entity's page that spins up a news feed bound to that entity.
- Observation
- A single numeric sample (a price, a value) kept in its own time-series store and drawn as a sparkline. Never a note.
- Watch
- A keyword rule. When any feed records an item matching it, you get a notification.
Three ways to receive external data
Live data comes in three modes, from lightest to always-on. Pick by how much you want to keep watching:
| Mode | What it does | Reach for it when |
|---|---|---|
| Look up now | A one-shot web search on an entity, right now. Nothing scheduled. | you just want the latest on one thing, once |
| Follow an entity | Creates a news feed bound to an entity, so its stories flow in over time. | you care about an entity and want to keep up |
| Standing feed | A source (a publication, a filing stream, a price) checked on a schedule. | you want a whole stream, not just one entity |
The one idea underneath all three
Whatever the mode, the result has the same shape: an item becomes a markdown note in your `live/` folder, timestamped and append-only. Because it's an ordinary note, it gets extracted into your knowledge graph exactly like something you typed. The people, companies, and claims it mentions join your entities and connections. Live data is simply more notes, arriving on their own.
The one exception is pure numbers. A price tick has no prose to extract, so it's kept as an observation in a lightweight time-series store and drawn as a sparkline, never as a note and never in the graph. Prose has one home (live/), numbers have another (observations).
How a feed works under the hood
A single scheduler ticks in the background and asks each feed one question: are you due? If a feed is due, it runs a small, boring pipeline. Understanding it explains everything you'll see in the UI:
What happens on each tick
- Schedule
The scheduler checks each feed against its cadence. Only feeds that are due run; the rest wait. A feed only ticks while its workspace is open, so close the workspace and nothing polls. On top of that, minute-scale (5–30 min) feeds run only while the live window is on; hourly and daily feeds collect on schedule regardless (covered just below).
- Fetch
The connector pulls the source: an RSS/Atom feed, a GDELT news query, an SEC EDGAR filing stream, a price endpoint. Conditional requests mean an unchanged source costs almost nothing.
- Dedup
Every item is checked against what's already been recorded. Seen-before items are dropped here, before any extraction. The check spans the whole workspace, so two overlapping feeds pulling the same story point at the note that already exists rather than writing it twice. That's the volume defense that keeps a busy feed from flooding your graph.
- Record
A genuinely new item is written to your live/ folder as a note (and queued for extraction), or, if it's a number, appended to the observations store. Either way it shows up without you revisiting.
The catalog, and your own feeds
Open Live feeds from the left sidebar (or the command palette, or the macOS Go menu). "Add a source" opens a catalog of about sixty built-in sources across eight categories: news, finance, research, social, and more. Most are keyless and record straight away; a few, like Finnhub for stock prices, ask for a free API key, and the connect dialog links you straight to it.
Not in the catalog? The dashed "Add your own feed" box at the top takes any RSS or Atom URL. Paste it, click Add, and it becomes a feed like any other, with a live/ folder in your file tree holding the notes.
Feeds are per-workspace and opt-in
A feed belongs to the workspace you added it in, and it only polls while that workspace is open. There is no global background crawler running when the app is closed.
Recorded notes are append-only. Turning a feed off (or unfollowing an entity) keeps the notes it already wrote and stops new ones — and keeps the dedup ledger that remembers what it has seen. So turning it back on later resumes from new items instead of re-recording the source's whole backlog.
When to collect: the live window
Live doesn't have to mean a minute-by-minute firehose the whole time the app is open. The antenna button on the right of the titlebar, just left of the Entity table button, is the tap. Click it and choose off · 30 min · 1 hour · 4 hours · until end of today · always on — minute-scale feeds pull new items only for that long.
Live and scheduled collection are different
Minute-scale feeds (5, 15, 30-minute cadences) poll only while the live window is on. Turn it on when you want to watch for a while; turn it off and they go quiet. It defaults to off, so an open app doesn't mean data pours in forever.
Hourly and daily feeds are scheduled collection. They run on their cadence whenever the workspace is open, live window or not — a "skim once each morning" rhythm needs no window at all. The Live / Scheduled chip in the feed list tells you which is which.
When the window is off but minute-scale feeds are waiting, the antenna button shows a count badge (the same idea as the activity-center's unread count), and the feeds tab carries a banner offering to turn it on. It keeps a quiet "off" from being left on by accident.
Follow an entity for news
Open any entity's profile page and find External signals. The Follow news button creates a news feed bound to that entity: under the hood, a query for its name. Stories about it flow into your live/ folder and its graph.
Click again to unfollow, which turns the feed off but keeps the notes. It disables rather than deletes, so following again later resumes instead of re-recording the same stories as duplicates. It's the fastest way to keep one company, person, or product current without hunting for the right source.
Track a price as a sparkline
The same External signals section has Track a price, with a Stock / Crypto toggle. Crypto uses a keyless source, so a symbol like BTC is all it needs. Stock uses Finnhub, so you paste a free API key the first time and only the first time; it's saved in your operating system's keychain and reused on every later ticker.
- Each tracked price draws a sparkline with the latest value and the change since the series began. The first sample shows up immediately; the trend fills in as samples accumulate.
- You choose how often it samples: 5 min, 15 min, 1 hour, or daily (15 minutes by default). The 5- and 15-minute intervals sample while the live window is on; 1 hour and daily are scheduled collection and run regardless. Re-opening the editor pre-selects the interval you already picked, so it never silently resets.
- Prices are observations, remember: a time series in its own store, not notes. They never clutter your graph.
Watch for a keyword across every feed
At the bottom of Live feeds, "Watch for keywords" adds a monitoring rule. Type a term (a product name, a competitor, a phrase), and whenever any feed records a matching item, a notification fires ("Watch: Nvidia") that takes you straight to the note. That's the difference between a firehose of feeds and being told the one thing you actually care about.
Add a feed and follow an entity
- Open Live feeds from the left sidebar. In the dashed "Add your own feed" box, paste an RSS URL you read (a news site's feed, say) and click Add.
- Watch a live/ folder appear in your file tree, with notes arriving inside it. Open one and see the headline, source, and timestamp.
- Now open an entity's profile page and find External signals. Click Follow news.
- Give it a moment, then check the entity's signals list and the live/ folder for its first stories.
Feed notes land in live/ and get extracted into your graph; the followed entity starts collecting stories bound to it, all without you polling anything by hand.
Where to watch it all
The knowledge-graph view has its own External signals panel that gathers recent feed items grouped by source, so you can skim everything that arrived without opening each note.
And when you just want one fresh answer without committing to a feed, use Look up on the web now on an entity's page: a one-shot web search you can clip into your workspace. It sits beside the feeds in External signals.
That's the whole model: a schedule, a fetch, a dedup gate, and then either a note or a number. Notes flow into the same graph you've been building all along. So the next question you ask the agent can reason over what happened this morning, not just what you wrote last month.
Next up: the first end-to-end recipe, where research, reading, and decisions come together.