Hithesh Shetty.

The problem

My first version had a bad habit. I'd ask it to send a WhatsApp message, it would reply "Sent.", and the message would still be sitting in the search box. So I changed the rules: it's only allowed to say something worked after it has looked at the screen and confirmed it.

How it works

1. Perception. It reads the screen through Windows UI Automation (the accessibility tree) first, OCR second, and screenshots only when text can't see the target. That is cheaper and more predictable than working from pixels alone.
2. About 48 tools. Click by label, type, drag, manage windows, set volume, find and edit files, search code, read .docx, run git, drive its own Chromium over CDP, and draft Gmail emails.
3. Policy before action. Every call passes a policy engine: a hard deny list for destructive shell commands, a confirm list, and a terminal that is off by default and runs inside Windows Sandbox.
4. Receipts after action. Every result carries a typed receipt: what was observed, who acted, who checked, and a verdict. "Sent." can only be rendered from a CONFIRMED receipt, and a receipt is rejected if the checker is the same component as the actor. UNCONFIRMED exists for "nobody could look", which is different from "it failed".
5. Skills. Once a task has been done and verified it can be saved and replayed later with zero model calls.

Things I learned

• It drafts emails but never presses Send. The Send button is exactly where a prompt injection wants to end up. Text read from a page or email can't choose where things go, and that rule is enforced at the tool boundary rather than by trying to spot suspicious English.
• I deleted an entire staged planning pipeline (and the eight packages only it used) after instrumentation showed it ran 0 times across 335 eval runs and 143 real sessions.
• A unit test fails the build if anyone adds a tool without a receipt.

Results

The eval checks the machine itself (the file on disk, the volume level, the message in the chat), never the agent's own answer. Plus about 1,500 unit and integration tests.

Stack

• Node 22
• Electron
• PowerShell host
• UI Automation
• OCR
• Chrome DevTools Protocol
• Windows Sandbox
• DPAPI
• Any LLM: OpenAI-compatible, Anthropic, Gemini, Mistral, DeepSeek

The problem

AI tools are happy to add citations, but click the link and it often says something else entirely. I wanted one where you can trust the sources, so the model never writes the citations itself. The code does, and only from quotes it has actually checked.

How it works

1. Plan, then pause. The planner splits the question into 3 to 6 subquestions with search queries. The graph then stops at a LangGraph interrupt() so you can approve, edit, replan or cancel.
2. Research in parallel. One worker subgraph per subquestion is fanned out with Send(). Each worker is a small tool-using agent that picks between web search (Tavily) and academic search (arXiv, Semantic Scholar), then fetches, parses PDF and HTML, chunks, indexes into pgvector and retrieves passages.
3. Evidence or nothing. Claims are extracted with exact quotes, and any quote that isn't found word for word in the source is thrown away. Sources are scored on authority, recency, relevance, primary vs. secondary, methodology and citations. Search rank never counts as credibility.
4. Gates that loop. An evidence gate sends thin subquestions back for more research. An independent critic (rules plus an LLM review) looks for overclaiming and single-source conclusions, and a deterministic quality gate decides to publish, research more, or publish with stated limitations.
5. Citations built in code. The writer only tags sentences with evidence IDs. Numbering, URLs and the bibliography are generated from stored provenance. Every sentence must cite something, the evidence must cover its key terms, and every number in it must appear in that evidence. Sentences that fail are re-attributed or removed, and removals are listed in the report.

Things I learned

• Only 8 steps use an LLM. Source scoring, the gates, citation numbering and verification are plain, unit-tested Python.
• Runs are durable. State is checkpointed after every step, so a plan approval still works after a server restart. Progress streams to the UI over Server-Sent Events with Last-Event-ID replay.
• Parallel workers merge through commutative, idempotent reducers with content-addressed IDs, so two workers that find the same source converge on one entry.
• Built-in fault injection: failed source, LLM timeout, invalid model output, thin evidence and critic rejection, each recovered from end to end.

Numbers

The hosted demo runs offline with a mock model and a synthetic corpus, so it's free to try. It sleeps on Render's free tier, so the first request can take up to a minute.

Stack

• Python 3.12
• LangGraph 1.2
• FastAPI
• Server-Sent Events
• Pydantic v2
• PostgreSQL + pgvector
• SQLAlchemy async
• React 19
• TypeScript
• Docker
• GitHub Actions

Type in either editor. Hit Run, or press Ctrl+Enter.

The problem

Most shared editors I tried only synced the text. I wanted to be able to actually run the code together too, and see each other's cursors while typing on the same line without messing each other up.

How it works

1. Shared document. Monaco (the editor inside VS Code) is bound to a Yjs document, and every client keeps a full copy.
2. CRDT merging. Yjs guarantees that clients who have seen the same edits end up with identical text in any arrival order, so two people typing on the same line both keep their changes.
3. Thin relay. The Node server is a y-websocket relay with one document per room. The room ID in the URL is the document name, and that's all the isolation rooms need.
4. Presence. Yjs awareness gives everyone a color, a name, and a cursor with a name tag.
5. Execution. Run posts to /api/execute. An Express proxy adds JDoodle credentials, rate-limits per IP and rejects huge payloads. Python, JavaScript, C++ and Java are supported with stdin, and output is classified as normal, error or timeout by matching tracebacks, error: lines, segfaults and timeout messages.
6. Shared language. Switch the room to Java and everyone switches, so nobody runs your code with the wrong compiler.

Things I learned

• CRDT over operational transform. With OT the server has to own the document and transform every operation against every concurrent one. With Yjs the merging happens on clients through a well-tested library, and the server stays dumb. The cost is a little memory per character, which doesn't matter for a code file.
• Split deploy: the client on Vercel and the server on Render, because WebSockets need a long-lived process. vercel.json forwards /api, so the browser only ever talks to its own origin.
• Deliberately one file per room, with no accounts and no persistence, to keep the focus on sync and execution.

Numbers

The server sleeps on Render's free tier, so the first sync after a quiet spell can take up to a minute.

Stack

• React
• Vite
• Monaco
• Yjs
• y-websocket
• Node
• Express
• JDoodle API
• Vitest
• fast-check

The problem

I kept wanting to know what the engine thought of a position, and setting up the board again by hand got old fast. So I trained a model to read the board straight off my screen.

How it works

1. Capture. mss grabs the screen region where the board lives.
2. Find the board. Adaptive thresholding, then a contour search for the largest roughly square shape with four corners. A four-point perspective transform turns it into a clean top-down board even if the capture is skewed.
3. Classify 64 squares in one batch. EfficientNetB0, pretrained on ImageNet, with a custom head (global average pooling, then 256 and 128 unit dense layers with batch norm, dropout and L2, then a 13-way softmax) at 128×128.
4. Build the position. Square indices map to a1–h8, flipped by which side you play, and go to the web app as {"e4": "wP", "g8": "bN", …}.
5. Ask the engine. A Node API talks UCI to Stockfish at depth 12, or 18 with deeper search. FENs are validated before they reach the engine (400 on bad input), and slow searches return 504.
6. Play on. A React app built on chess.js handles castling, en passant, promotion and repetition, with legal-move highlights, custom position setup, and free or strict turn modes. There's also a small Tkinter tool for a quick best move.

Training notes

• About 67,000 labeled square images (4,000 to 11,500 per class), trained in two phases: head only on a frozen backbone at lr 1e-3, then fine-tuning from block 4b up at 1e-4. It uses class weights for the imbalance, early stopping, LR reduction on plateau, and tracks top-2 accuracy.
• Augmentation covers small shifts, zoom, brightness and contrast, but no flips or rotations. Pieces on a screen are always upright, so the model should never learn otherwise.
• The bug that bit me: training and inference have to feed images exactly the same way (OpenCV BGR order, scaled to 0–1). When they didn't, accuracy dropped quietly.
• An earlier from-scratch CNN (depthwise-separable convolutions with squeeze-and-excitation) trained faster but lost on accuracy, so EfficientNet shipped.

Numbers

Stack

• Python
• TensorFlow 2.19
• Keras 3.9
• OpenCV
• mss
• React
• chess.js
• Node
• Express
• Stockfish (UCI)
• Tkinter