# Unify position math + real UTF-16

**Effort:** M · **Impact:** L · **Immediate roc UX:** yes

## Repro (broken today)

1. Open a `.roc` file whose first line (or the line under the caret) contains multi-byte UTF-8 **before** the identifier you care about, e.g.:

   ```roc
   app [main!] { pf: platform "../basic-cli/platform.roc" }

   main! = |_| {
       # café is 4 UTF-8 bytes for "é" but 1 UTF-16 code unit in many editors
       café_value = 1
       café_value  # hover / go-to / completion here
   }
   ```

   Or simpler one-liner style the editor will still send UTF-16 columns for:

   ```roc
   x = "😀"  # or any non-ASCII in a string / name before the target token on the same line
   ```

2. Place the caret on a binding / call **after** non-ASCII on that line.
3. Trigger hover, go-to-definition, completion, document highlight, or selection range.
4. Observe: wrong symbol, empty result, or highlight/selection on the wrong span — while **incremental edits** (`didChange` range replace) still land correctly.

**Why:** Capabilities advertise `positionEncoding: "utf-16"`. `DocumentStore` converts UTF-16 ↔ bytes correctly for edits. Query paths treat `character` as a **byte** offset:

```zig
// src/lsp/position.zig — positionToOffset
// For simplicity, treat character as byte offset within line
// (proper UTF-16 handling would require more work)
return line_start + character;
```

Handler-local copies (`document_highlight.zig`, `selection_range.zig`, …) repeat byte math and add a **4096-line stack cap** in places — large files silently truncate.

## Evidence

- Cap: `src/lsp/capabilities.zig` → `positionEncoding = "utf-16"`
- Correct converter: `src/lsp/document_store.zig` → private `positionToOffset` (UTF-16 aware)
- Wrong shared helper: `src/lsp/position.zig` → `positionToOffset` / `offsetToPosition` (byte `character`)
- Also: `src/lsp/line_info.zig`; local `LineOffsets` / `positionToOffset` in highlight & selection_range handlers

## Goal

One module owns **bidirectional** LSP Position ↔ byte offset under the negotiated encoding (utf-16 today). Every feature uses it. No 4096 hard caps. Edits and queries agree.

## Fix shape

1. Lift / share the UTF-16 logic from `document_store.zig` into `position.zig` (or a dedicated `lsp_encoding.zig`) with:
   - `positionToByteOffset(text, line, character_utf16)`
   - `byteOffsetToPosition(text, offset)` (or line-table accelerated variants)
2. Make `ModuleEnv`-based helpers call the same codec (line starts + UTF-16 within line).
3. Delete handler-local line tables / 4096 caps; allocate line starts like `position.buildLineOffsets` already does.
4. Optionally: if client offers `utf-8` in `initialize`, negotiate it and skip UTF-16 scans — still one API.
5. Tests:
   - Non-ASCII before caret → hover/definition hit the intended binding.
   - Round-trip: UTF-16 position → byte → UTF-16 position.
   - File with >4096 lines: selection_range / highlight still works.

## Acceptance

- [ ] ASCII-only behavior unchanged.
- [ ] Non-ASCII-before-caret repro fixed for at least hover + definition + completion.
- [ ] No remaining “treat character as byte offset” comments.
- [ ] No stack-capped line tables in handlers.

## Out of scope

- Full semantic-token delta encoding audits beyond position fields (do if you touch those ranges in the same PR).
- Async message loop.