Real time, events and background work: SignalR, the outbox and hosted services

How a server speaks first (polling, server-sent events, WebSockets and SignalR), how modules hear each other's facts without losing one (in-process events and the transactional outbox), and how work runs by itself - hosted services, distributed leases, caching and signed webhooks - step by step, with pictures.

In short

On the web the browser normally speaks first and the server only answers. Real time turns that round: the server tells open screens that something changed, over WebSockets through SignalR. Inside the server, one module learns what another did through events, and the facts that must never be lost travel through a transactional outbox in PostgreSQL instead of a message broker. Work nobody clicked - schedules, retries, clean-up - runs in hosted services, and a lease makes sure only one server does each job at a time.

After this lesson you can Choose between polling, server-sent events and WebSockets; explain the transactional outbox and at-least-once delivery step by step; write an idempotent handler and a background worker that is safe on several servers; and sign and verify a webhook.

Words you need

How a server speaks first: polling, SSE and WebSockets

HTTP is request and response: the browser asks, the server answers, and that exchange is over. To show a change someone else made, the server needs a way to speak first. There are four classic ways, from the simplest to the most capable.

Why {{brand}} streams AI answers with SSE An answer flows one way and starts with a question. So the question goes out as an ordinary POST, through the same checks as every request, and the answer is read as server-sent events with fetch and a stream reader - the browser's EventSource cannot POST. Because proxies sometimes buffer streams, a robust client keeps a plain request as a fallback.

SignalR: hubs, groups, reconnects and the backplane

SignalR is the ASP.NET Core library built on those transports. It negotiates the best one the browser and the network allow - WebSockets first, then server-sent events, then long polling - and gives both sides one simple model: the browser invokes methods on a hub, and the server sends named messages to one connection, one person, a group or everyone.

A connection lives in the memory of one server. Put two servers behind a load balancer and Aynur may be connected to server A while Rustam saves on server B: B's message to the group never reaches A. A backplane fixes this - every server publishes hub messages to Redis pub/sub, and every server delivers what it hears to its own connections.

A long connection must keep proving who it is A WebSocket can stay open for hours, but a session can be signed out or a person removed in a minute. Teams usually combine three techniques: check access on every join, give the connection a short authentication lifetime so an automatic reconnect proves the session again, and put a version number in the group name so that when access to a record narrows, the old group goes silent.

Signal, then refetch: why the message carries no data

The simplest live message is a signal: this document changed, here is its id, nothing more. The screen then refetches it over normal HTTP. One small idea solves three hard problems at once - permissions, lost messages and ordering. A live board of cards uses exactly this technique.

Pushing the data is still right when a message is small, final and addressed to people the server already chose - for example a chat message going to the members of one conversation. Where a screen shows a filtered view of many records, such as a board, send a signal.

Events: in-process dispatch, and the three ways to deliver

An event says that something already happened. The module that publishes it does not know who listens; each listener registers a handler. Inside one process this needs no infrastructure at all: a dispatcher asks the dependency-injection container for every handler of the event's type and calls them in turn.

The id of the stored event travels with every delivery, so a handler can tell a repeat from a new event. Later sections use it for idempotency.

Never publish a fact before its commit If handlers hear MemberLeft and the save then rolls back, other modules have already revoked access for someone who never left. A fact must be heard only once the change is durable - which is exactly what the outbox below guarantees. The next section compares the ways to deliver such facts; the transactional outbox, one of them, is taken apart step by step afterwards.

The options in the market

What {{brand}} chose, and why

{{brand}} is a modular monolith: many modules, one deployable, one PostgreSQL. A message broker would add a server to run, secure, monitor and back up - for messages that never leave the process. The outbox gives the one thing a broker would add that matters here, durability, with the database that is already there.

The trade-off accepted Every committed fact costs the database one extra row, events can arrive out of order and more than once, and there is no history to replay for a new subscriber. Handlers are therefore written to read what is true now rather than trust the order of facts.

When a broker would be the right call Separate services written in different languages, very high event volumes, consumers that must replay history, or a team that already runs Kafka or Service Bus. Even then, keep the outbox: it becomes the reliable way to publish to the broker.

The transactional outbox, step by step

The outbox solves the dual write. Saving and publishing are two actions, and a crash can land between them. Save first and crash: the change exists but nobody hears about it. Publish first and the save fails: everyone hears about a change that never happened. The outbox turns the two actions into one transaction.

After a handler succeeds, the relay appends its name to handled_by: update outbox set handled_by = handled_by || 'RevokeTokensOnLeave' where id = $1;. A retry skips every handler already listed, so the ones that worked never hear the event twice.

{{brand}} delivers almost every event on the fast path and runs the relay as one worker under a lease. The event row is written in the same transaction as the data it describes, so a fact is never separated from its change.

At-least-once delivery and idempotency

The processed row and the changes are saved together. If the processed row already exists, the unique index rejects the whole save, so the work is not applied twice. IsUniqueViolation is a small helper that checks the database error code (23505 in PostgreSQL).

Order is not guaranteed either A retried event can arrive after a newer one. Handlers that read the current state - who is a member now, which column the card is in now - end in the same place whatever the order.

Background work: hosted services and BackgroundService

In .NET an IHostedService is started and stopped together with the application. BackgroundService is its convenient base class: you write one method, ExecuteAsync - usually a loop - and it receives a stopping token that fires when the server shuts down. {{brand}} runs its background work as hosted services of the same application: schedules, reminders, outgoing email, webhook delivery, clean-up and the outbox relay.

TryRunAsync returns false at once when another server holds the lease, so the loop just waits for the next round. WaitAfter doubles the wait with each failure in a row and stops at a ceiling.

Background work that acts for a person A schedule runs a test as the person who set it up. Run that work as that person - same permissions, same audit trail - and check at run time that they may still act. Never give background work an administrator's power just because nobody is watching.

Distributed leases with PostgreSQL advisory locks

Run two copies of an app and every worker runs twice: two reminder emails, two copies of a scheduled report. A lease lets the copies agree that one of them does each job. PostgreSQL has one ready-made: the advisory lock, a lock on a plain number that means whatever your application says it means.

Connection poolers can break session locks A pooler such as PgBouncer in transaction mode hands your server connection to another client between transactions, so a session-level advisory lock can be held for the wrong client or vanish early. Give lease connections session pooling.

Caching: IMemoryCache, Redis and cache-aside

A cache keeps a copy of something expensive to read or compute, for a short time. The common pattern is cache-aside: look in the cache first; on a miss read the source, store the result with a time to live (TTL) and return it. On a write, save to the database and then delete the cached copy, so the next read fetches the new value.

A cache is never the source of truth If Redis is slow or down, a request becomes slower, never failed - every miss falls through to the database. Design each cached value so that reading the source instead is always correct.

Outgoing webhooks: signing, retries and backoff

A webhook is an HTTP POST your system sends to someone else's address when something happens - a monitor went down, a card was completed. The receiver cannot ask questions first, so the message must prove where it came from, survive the receiver being down, and never let a customer's address be used to reach your own internal network.

The scheme above is the one {{brand}} documents for webhook receivers.

If you are the one receiving webhooks Verify the signature first, answer 2xx quickly, put the real work on your own queue, and store the delivery id so that a repeat does nothing.

Try it yourself: an outbox and a lease in one evening

All you need is Docker and psql. You will see that a rolled-back change leaves no event, watch a duplicate appear after a simulated crash, and remove it with idempotency.

Common mistakes

Interview questions

Related