Back to AI projects
Self-developed Agentic AI App · Dec 2025

Spending

An Agentic AI Pipeline for Family Spending — From Receipts to Insights

Stack Python · Flask · Postgres · Gemini Flash
Hosting Google Cloud Run · Cloud SQL · Firebase Auth
Time saved 3 days → under 2 hours
The story

Most spending apps tell you "You spent $342 on groceries last month." That's it. They don't keep the milk-by-milk, store-by-store detail that you actually need six months later, when the real question is "which store has the better deal on this brand?" or "has travel spending crept up since last year?"

I built one that does. Snap a receipt, paste an online order confirmation, or just type "Lunch yesterday $12," and every line item gets pulled apart and saved with date, store, product, price, and category. The AI handles the extraction. Postgres stores it cleanly. SQL turns it into answers.

Real-world impact
3 days under 2 hours
Manual processing time, before and after the agent went live
Questions the data can answer
Which store is actually cheaper for the brand of olive oil I buy every month?
→ See the price gap, every visit
Is this month's travel spending tracking under last year's, or did I drift?
→ Compare month against month, year against year
Has the same item gotten more expensive at the same store over the past year?
→ Watch a single item's price climb (or fall)
How has our family's category mix shifted between this quarter and last?
→ See where the money quietly moved
Three designs that paid off
📥
Design 01
Send it anything
InputPhoto · Text · PDF
ResultSame clean rows
🧬
Design 02
Tune without code
RulesIn the database
UpdateIn seconds
🧠
Design 03
Learns your habits
Looks atPast purchases
KeepsLabels consistent
How it works

The Receipt-to-Database Pipeline

Four stages · one round-trip · under 5 seconds

Stage 1
Capture
Photo, PDF, online order confirmation, or natural-language text
Mobile
Stage 2
Parse
Gemini Flash with a system prompt assembled from database rules + 300-row history
Gemini
Stage 3
Validate
Duplicates checked by AI flag and Python date+amount match. Math reconciliation forces sum=total
Python
Stage 4
Persist
Item-level rows written to a shared, queryable database, ready for SQL analysis
Postgres

All decision-making is AI-driven. Persistence and a duplicate safety net are deterministic code.

The receipt is just the entry point. The real value is being able to ask the data anything, six months later, in plain SQL.

What makes it interesting
01

Every line, not just the total

Most apps log "$24 at Whole Foods" and move on. Mine records every line on the receipt ($4 milk, $3 bread, $5 eggs), each as its own row, with the date, store, category, and subcategory attached. That's why I can ask, six months later, "has milk gotten more expensive at Whole Foods?" or "where do I get the best price on coffee?" The detail is the whole point.

02

The whole rulebook lives in the database, not the code

This is what makes it agentic. Traditional AI apps bake their prompts into the source code, so changing the AI's behavior means rewriting and redeploying. Here, both the standard rules and the time-limited overrides (like "for Jan 7–9, everything counts as Travel") live as rows in the database. The agent re-reads its rules on every call, so behavior changes the moment I edit a row.

03

A RAG pattern over my family's own purchase history

This is a textbook Retrieval-Augmented Generation (RAG) setup, but the "knowledge base" isn't Wikipedia or some scraped corpus. It's my family's own past spending. Before the AI sees a new receipt, the backend retrieves the last 300 transactions from Postgres, formats them as a compact log, and injects them into the system prompt as context. So when "Starbucks" shows up again, the AI sees that previous Starbucks entries were filed under Dining / Coffee Shop, and follows the pattern. Categories stay consistent across thousands of rows, which is exactly what makes period-over-period comparisons trustworthy.

04

Two-layer duplicate detection

Submit the same receipt twice and the system catches it. Twice. First, the AI looks at your last 300 transactions and flags anything that looks like a repeat. Then Python double-checks by date and amount over the last 30 days. If either layer spots a likely duplicate, you get a "did you mean to record this again?" prompt before it saves. Clean data going in is what makes the answers trustworthy coming out.

05

One household, many phones, one database

My partner records dinner from his phone. I scan a grocery receipt from mine. Both entries land in the same shared database, tagged with our family ID. Joining the household takes a 6-character code. One tap.

Architecture
Backend
Python · Flask · SQLAlchemy
AI
Google Gemini Flash (multimodal)
Database
PostgreSQL on Cloud SQL
Auth
Firebase Auth + Firestore allow-list
Hosting
Google Cloud Run
API surface
REST · JSON · CORS-enabled
Agentic AI RAG Prompt Engineering Python