CozySell

Why it was hard

When a person orders from the supplier directly, no auto-purchase run record is left, so duplicate checks that only look at the DB couldn't catch it. There was a guard that looks up the supplier's order history, but I had set it to turn on only for retries. I had assumed first attempts were safe. On top of that, when the status changed on the product management tool's side, the sync didn't re-read that row, so an order that had already been handled stayed a purchase target.

How I traced it

1. 1. Were the matching criteria wrong?

   The start of the guard showed that on a first attempt, it never looked up the supplier's order history at all and just returned 'no duplicate'.

2. 2. Why was an already-handled order still a candidate?

   When the status changed in the product management tool, the sync didn't re-read that row, so the local row was stuck in its old state.

3. 3. Could I just delete the condition?

   No. On a first attempt, the lookup started from the run record's creation time, so an order a person had placed directly before that fell outside the range.

4. 4. Which account should the lookup use?

   A first attempt may have no account-selection record, so I made it use the supplier account set in the order routing instead.

Root cause

The duplicate guard (evaluate_run_for_duplicate) skipped the supplier lookup and returned 'no duplicate' right away whenever the run wasn't a retry.

Fix

It now looks up the supplier's order history on first attempts too. The lookup starts from the time the customer placed the order, and if there's no account-selection record, it uses the account set in the routing. It blocks payment and removes the candidate only when product number, option, quantity and amount clearly match. I kept one exception: if the supplier lookup itself fails, it lets the order through so normal orders don't all halt. That is a separate decision from blocking when a payment result is uncertain.

if not _is_redispatch(run):
    return DuplicateMatch("none", reason="first_attempt")
since = _first_attempt_at(run)

is_redispatch = _is_redispatch(run)
account_ids = _supplier_account_ids_for_order(run)
if not account_ids:
    fb = _dispatch_supplier_account_id(run)  # account set in the routing
    if fb:
        account_ids = [fb]
since = _first_attempt_at(run) if is_redispatch else _candidate_order_datetime(run)

Why it was hard

The bug had 2 layers. The backend container had no docker tools or socket. Even with those added, the update recreated the backend too, and the process running the update died with it. The updater was replacing the very container it was running in. And simply recreating a worker that held a payment session would also cut off the payment in progress.

How I traced it

1. 1. Where does the 502 come from?

   The update API ran the update synchronously inside the request, and it failed right at the first image pull.

2. 2. Would attaching the docker socket be enough?

   No. The moment the container was recreated, the updater died with it and never reached the health check or rollback. I concluded the update logic was fine and only the place it ran was wrong.

3. 3. Then where should it run?

   I had a systemd watcher on the host see a request file and start a one-off container outside the compose project. I dropped the idea of rewriting it in bash because it would duplicate the verification and rollback logic.

4. 4. Would it work on a real server?

   A smoke run that changed nothing turned up 2 problems. The systemd environment had no $HOME, so the script exited right away, and the working-directory option overrode the image's settings.

5. 5. What about the first real update?

   The health check request got a 400 for a host name that wasn't allowed, and file permissions kept it from reading or writing settings. Each time, the automatic rollback protected the server. I fixed them one by one, and the update from 1.0.9 to 1.0.10 succeeded.

Root cause

The update API (FleetApplyUpdateView) ran the update synchronously while handling the request, inside the backend container it was about to replace.

Fix

The API no longer runs the update itself. It only leaves a request file and returns 202. If a request already exists, it returns 409. I set up a systemd watcher on the host that watches for a request file and runs a script. The script takes a lock, pauses auto-purchasing, waits until in-progress payments reach 0 and then starts a one-off update container from the current version's image. This container sits outside the compose project, so it survives when the other containers are recreated.

try:
    result = run_fleet_update(str(offer["target_version"]), ...)
except FleetUpdateError:
    return Response({"detail": "...restored to the previous version"}, status=502)

# API: leave only a request and respond right away
if self_update_request_pending():
    return Response({...}, status=409)
if not request_self_update(offer):   # .fleet-control/request.json
    return Response({...}, status=501)
return Response({"queued": True}, status=202)

# host script (run by a systemd path watcher, shortened)
flock -n 9 || exit 0; trap cleanup EXIT INT TERM
pause; drain until inflight == 0        # abort the update after 15 minutes
docker run --rm -v /var/run/docker.sock:... <current version image> \
  python manage.py fleet_update --request request.json

Why it was hard

Chromium's profile lock has the form 'hostname-PID', and inside a container the hostname is the container ID. When the container is recreated, the leftover lock looks like 'a profile in use by another computer'. The app's cleanup logic was set not to touch locks with a different hostname, and since it was a different PID namespace, there was no way to check whether that process was alive. It was also intermittent: it only happened if a browser was open at the moment of the forced kill.

How I traced it

1. 1. Is the lock owner still alive?

   The hostname the lock pointed to didn't belong to any container, and there were 0 Chromium processes using that profile.

2. 2. Did the cleanup code not get called?

   It was called normally. By design, it didn't reclaim locks from another host.

3. 3. Is an offline recovery tool enough?

   The first fix required stopping the 5 workers that share the same volume, so it couldn't run in production.

4. 4. When was the lock created?

   Lining up the lock file's timestamp with that day's deploy log, there was a record 14 minutes later of a forced kill for 'not stopping within 2 minutes'.

5. 5. Do all the failures have the same cause?

   Failures that ended at exactly 30 seconds had a different cause, a page-load wait timeout. I classified them separately.

Root cause

During a deploy, the container went past its 2-minute stop grace period and was force-killed, and Chromium couldn't release the lock. A lock pointing to a container that no longer existed was left behind, with no path to reclaim it. The same server saw 40 forced kills during deploys in 60 days.

Fix

If the lock's hostname looks like a container ID, it checks whether any container with that ID still exists, in any state. Then it scans every process on the host to see whether any Chromium process is using a profile directory with the same device and inode. If any check is incomplete, it doesn't delete. I also added an online recovery mode that doesn't stop the workers, and a step that reclaims locks automatically after every deploy.

host, pid_text = os.readlink(lock_path).rsplit("-", 1)
if host != socket.gethostname() or not pid_text.isdigit():
    return False   # once the container changes, always False

m = re.fullmatch(r"([a-f0-9]{12}|[a-f0-9]{64})-([1-9][0-9]*)", target)
for c in containers:                     # containers in every state
    if c["Id"].startswith(m[1]) or c["Config"].get("Hostname") == m[1]:
        raise ProfileInUseError("former_owner_container_present")
ident = (info.st_dev, info.st_ino)
for ref in process_refs:                 # --user-data-dir resolved against /proc/<pid>/root
    if ref.error:
        raise ProfileInUseError("host_process_scan_incomplete")
    if ref.identity == ident:
        raise ProfileInUseError("host_profile_process_still_active")