Files, email and push notifications - storing, sending and fanning out

How files live in object storage behind one interface and travel as streams, how signed links and virus scanning work, how an email travels with SMTP and MIME and reaches the inbox (SPF, DKIM, DMARC, one-click unsubscribe), how Web Push with VAPID and a service worker reaches a phone, and how one event fans out to many people while storing them only as ids.

In short

Three jobs sit at the edge of almost every web product: keeping files, sending email and reaching a phone with a push notification. Each one talks to a system outside the product - a storage service, a mail provider, a browser's push service. {{brand}} puts each behind one small interface and chooses the vendor by configuration, so features never change when the vendor does. On top sits a notification pipeline: one event becomes one row per person, grouped, filtered by their preferences, and delivered to the inbox, a device or an email digest.

After this lesson you can Design file storage that moves from local disk to Azure Blob Storage or S3 without touching a feature, explain SMTP, MIME, SPF, DKIM and DMARC to a colleague, send a Web Push message with VAPID and a service worker from scratch, and design a notification fan-out with grouping, digests and preferences that stores people only as ids.

Words you need

Files: one interface, three places to keep them

A feature that stores a file should not care where the bytes go. {{brand}} gives every module the same tiny contract: save a stream and get a key back, open a key, delete a key. An adapter behind it talks to the real storage. The module keeps only the key, next to its own row with the file name, size and type.

Never use the uploaded name as the key A name such as ../../app.config or invoice.pdf.exe is chosen by the uploader. Store the bytes under a random key with at most a short, safe extension, keep the real name in the database, and give it back only in the Content-Disposition header of a download.

Uploads and downloads as streams

A server that reads a whole upload into memory falls over when a few people upload large files at once. Streaming means the bytes flow through in small buffers: from the network, through the scanner, into storage - and back the same way on download. Memory use stays flat whatever the file size.

How an email travels: SMTP, email APIs and MIME

Your server never delivers an email to someone's inbox itself. It hands the message to a mail provider, which signs it, sends it to the receiver's mail server and deals with refusals. The hand-off is either SMTP, the old text protocol every mail system speaks, or an email API over HTTPS.

How Web Push works, step by step

Three parties take part: your server, the push service run by the browser maker (Google's FCM for Chrome, Mozilla's for Firefox, Apple's for Safari, Microsoft's for Edge on Windows) and the service worker in the person's browser. Your server never connects to the phone. It posts an encrypted message to the push service, which holds it until the device is online.

iPhone and iPad On Apple's phones and tablets a web app can receive pushes only after it is added to the home screen (iOS 16.4 and later). Detect this and explain it, instead of showing a button that cannot work.

From one event to many people: fan-out

When a card is assigned or a check starts failing, several people may need to know. Fan-out is the step that turns one event into one delivery per person, on each channel they want. {{brand}} uses fan-out on write: at the moment of the event, one inbox row is written for each recipient.

The options in the market

What {{brand}} chose, and why

The same product runs as a cloud service, on a single machine and inside a customer's own network. That one fact drives every choice here: each outside system sits behind an interface, a configuration value picks the adapter, and a deployment with a half-set provider refuses to start rather than failing someone's upload an hour later.

The trade-offs accepted Features get only what every backend can do, so no image resizing or lifecycle rules inside the storage. Web Push is best effort: a phone that stays off past the TTL misses the message, which is why the inbox row - not the push - is the record. Fan-out on write costs one row per person per event; grouping keeps that small.

Another choice would be right elsewhere. A social product whose accounts have millions of followers needs fan-out on read for its biggest audiences. A mobile-first product would build native apps. A product that only ever runs on one cloud could use that cloud's storage features directly and skip the interface.

Signed links: let the storage serve the file

Normally the API streams every download itself. When the file sits in an organization's own storage, streaming it through the API would double the traffic and the bill. Instead the API checks access as usual and answers with a signed URL: the storage itself checks the signature and the expiry, then serves the bytes.

A signed link is a bearer credential Whoever holds the link can download the file until it expires - no sign-in needed. Keep lifetimes short, sign only after the permission check, never log the full URL and never put one in an email that will sit in an inbox for weeks.

Trust the bytes, not the label

The browser sends a Content-Type with each upload, but the uploader controls it. A file named cat.png can be an HTML page with a script in it; served as a page from your domain, that script runs as your site. Most file formats begin with fixed magic bytes, so the server can check what a file really is.

Virus scanning with ClamAV

ClamAV is a free, open-source antivirus engine. Its daemon clamd loads the signature database once (the companion tool freshclam keeps it up to date) and listens on TCP port 3310 or a Unix socket. An application streams the file to it with the INSTREAM command and reads a one-line verdict - no file has to be written anywhere first.

Test with EICAR, never with real malware The EICAR test file is a harmless 68-byte text that every antivirus engine reports as infected. Download it from eicar.org to test your upload path; an error reply must always count as no answer, never as clean.

Deliverability: SPF, DKIM and DMARC

Anyone can write any From address into an email. Receivers decide whether to believe it with three DNS records on the sending domain. Without them, confirmation and password-reset emails land in spam or are refused.

One-click unsubscribe, done right

A recurring email - a weekly summary, a digest, tips for the first week - must be easy to stop. The List-Unsubscribe header (RFC 2369) gives mail clients their own Unsubscribe button; List-Unsubscribe-Post (RFC 8058) makes it one click: the client sends a POST to the address and the email stops, with no page and no sign-in.

A GET must never unsubscribe Company mail scanners open every link in incoming email to check it. If opening the link stopped the email, scanners would unsubscribe people who never asked. Only a POST changes anything.

Priority lanes and pacing for outgoing mail

Email providers allow only so many messages per minute and per hour. A good design keeps two lanes: urgent mail such as a password reset goes out at once, and bulk mail such as invitations takes its turn within a share of the quota, so a burst of bulk mail can never delay an urgent message.

The counters must live in a store that every server shares, such as a cache server. Otherwise each server would think it had the whole share to itself. A fixed window (this minute) is simple, but a burst at the edge of two minutes can double up; a sliding window is smoother but costs more to keep.

Inside a push message: encryption and VAPID

A push travels through a company you do not control, so the payload is encrypted end to end for one browser, and the request carries a VAPID signature so the push service knows which server sends it. Both are short standards worth knowing for interviews.

The subscription is untrusted input The endpoint URL comes from the browser. Accept only https addresses of real push services, do not follow redirects, and treat the endpoint and auth secret as a credential: never log them, and delete them on 404, 410 and sign-out.

Taming volume: grouping, priorities and digests

More notifications do not mean better ones. Without grouping and batching, a busy team trains people to block notifications, and a blocked permission is rarely given back. Three techniques keep the volume humane: grouping folds repeats into one row with a count, priorities put what needs action first, and preferences let each person choose per module how much reaches them.

Storing people as ids, not names

A notification sentence such as Aynur assigned you Fix the login redirect is not stored as words. The row keeps a template key, the values that fill it and the ids of the people it names. The sentence is written when it is read, in the reader's language, with names looked up at that moment.

Try it yourself: Web Push, a virus scan, email and signed links

All the exercises use free tools only. Service workers need https or localhost, so serve the page with a local web server on localhost.

Common mistakes

Interview questions

Related