Data and search

02 What I built

• 01

  Items pipeline

  Takes in data feeds, cleans them, applies each customer's rules, and puts each customer's items into a search index.

• 02

  Search engine as main store

  Moved the main data store to the search engine safely, step by step: a shadow copy, writes to both, copied the old data, compared the two, then switched.

• 03

  Safe index rebuilds

  A new search index is built next to the old one and swapped in one step. A lock that a late worker cannot break stops two rebuilds from fighting, and a loop finishes any rebuild that stopped halfway.

• 04

  Feed change check

  Compares each new feed with what is stored, using hashes so only real changes are written, and marks missing items as gone.

• 05

  Stable search paging

  Results are always sorted the same way and paged by position, not by page number, so going deep into search results never skips or repeats an item.

• 06

  Rare features first

  Ranks item features by how rare they are, the way search engines rank words, so the most special ones show first.

• 07

  Search and database clusters

  Created the container images for the company's own self-hosted OpenSearch cluster and its dashboards. I also created the database connection layer for the whole platform. I am the only developer of both.

03 How it works

1. 01Data feeds come in, and only real changes are saved.
2. 02Items are cleaned, and each customer's rules are applied.
3. 03A new search index is built next to the old one.
4. 04It is swapped in one step, so shoppers never see half-built results.
5. 05Items missing from a feed are marked as gone.