When you're building a wearable app, you quickly learn that the hardest part isn't the UI or the sensor integration. It's everything that happens after the device sends data.
I learned this the hard way.
What's NAND and Why Does It Matter?
NAND is the storage chip on the wearable device. Think of it like the device's hard drive—it stores sensor data locally when there's no Bluetooth connection. When the device connects to the phone, it uploads all that buffered data so the app can sync everything.
The problem: that data isn't immediately usable. It's compressed, encoded, and needs decoding before it becomes actual sensor readings. That decoding is CPU-heavy and happens all the time.
The Problem: Synchronous Decoding
Originally, here's how it worked:
Device connects → App sends NAND file to backend → API immediately starts decoding → Parsing and processing happens → Data written to database → Response sent to client
Sounds straightforward. It wasn't.
When users uploaded a full day's worth of sensor data (which could be 10-50 MB), the API would get hammered. Decoding took minutes. The API process would spike in CPU usage, memory would balloon, and sometimes the entire request would time out. Users got 502/503 errors. Engineers got paged at midnight.
The root cause was tight coupling: the API was responsible for everything in the request lifecycle. If decoding failed, the user request failed. If processing was slow, the user waited. If the system was under load, everyone suffered.
Why This Actually Breaks Everything
Performance: Decoding is heavy. A single large upload could block the API for 2-3 minutes. Other requests had to wait or get rejected.
Reliability: One failed decoding job meant a failed API response. No retry logic, no fallback. The user had to re-upload and try again.
Observability: When something went wrong, I couldn't tell if it was the upload, the decoding, the parsing, or the database write. Everything was one big black box.
Scaling: Throwing more API instances at the problem didn't help. Every instance had the same issue. I needed to process data faster, not spawn more servers.
User experience: Users would see "uploading..." for minutes, then get an error. Then they'd have to troubleshoot why their data didn't sync.
The Solution: Async Workers
I decoupled the API from the heavy lifting.
New flow:
Device connects → API receives file and creates metadata → Job queued immediately → API responds instantly to user → Background worker picks up job → Heavy decoding happens independently → Data processed and stored → User sees results when ready
Key changes:
1. Loose coupling The API now just accepts data and queues a job. It doesn't decode anything. The backend worker handles all the heavy lifting independently. If the worker fails, the API doesn't care. The job retries.
2. Fault isolation If a decode job fails, it doesn't impact the API. I can log it, retry it, and the user can still use the app. The error is isolated to that one job.
3. Independent scaling API instances handle requests. Worker instances handle decoding. If I'm getting tons of uploads, I spin up more workers. If I'm getting tons of regular API requests, I scale the API separately. They don't compete.
4. Better observability Each step of the pipeline is separate and trackable. I can see exactly where a job is: queued, processing, completed, or failed. I can monitor worker health independently from API health.
5. Performance API response time dropped from 2-3 minutes to under 100ms. Users get instant feedback. Processing happens in the background without blocking them.
What I Actually Built
The architecture looks like this:
- API layer handles authentication, validation, and file metadata. Creates a job record and queues it.
- Message queue holds all pending decoding jobs with retry logic built in.
- Background workers subscribe to the queue, pick up jobs, and process them independently. If a worker crashes, another picks up the job.
- Database stores final results. Workers write directly once processing is done.
- Monitoring tracks queue depth, worker health, job status, and processing time for each job.
The Lessons
1. Synchronous operations don't scale If your API is doing heavy processing, you're going to hit a wall. Users will wait. Errors will compound. Decouple early.
2. Tight coupling creates cascade failures When the API owns the entire lifecycle, one failure takes everything down. Separating concerns means failures are contained.
3. Observability matters With the old system, I had no idea what was happening inside a request. With async jobs, I can monitor each step independently and pinpoint exactly where things break.
4. Better UX comes from decoupling Users don't need a 2-minute API response. They need instant feedback ("Your data is syncing") and results when they're ready. Async lets you deliver that.
5. Scaling is about architecture, not just hardware Throwing more servers at the problem only works if your architecture allows it. Workers let you scale horizontally. The API can handle more requests. Workers can handle more jobs.
Building for Real-World Constraints
This pattern matters especially for data-heavy applications: health wearables, IoT, financial systems, anything where you're processing tons of data in the background.
The principles are simple:
- Don't block users on heavy processing
- Decouple components so failures don't cascade
- Make monitoring and debugging straightforward
- Build for independent scaling
It's not groundbreaking architecture, but it's the difference between an app that works and one that breaks under real usage.
That's the story of how I went from API timeouts and 502 errors to a system that handles thousands of sync operations reliably.