feat(android): generalize generated media blocks to video/audio/file (#2162)

* feat(android): render chat markdown math with offline KaTeX

Port the web math pipeline to Android chat markdown. A new
MarkdownMath pre-pass in :core:protocol extracts math with web
semantics (remark-math singleDollarTextMath:false + remark-latex-
bracket-math): $$ display flows, $$ text math only when the closer
run equals the opener and stays inside the paragraph, single $
never triggers math (currency stays prose), \(..\)/\[..\] bracket
pairs, backslash-escaped delimiters stay literal, and code spans,
fences, links and link reference definitions are fully protected.
Matches are lifted behind private-use placeholder tokens and
re-inserted as MathInlineNode/MathBlockNode custom nodes, hoisting
display math out of paragraphs while blockquote/list containers
survive the reparse. 37 JVM tests mirror the web suite's cases.

Rendering is a shared offline WebView (ui/markdown/MathWebView.kt)
fed a self-built HTML document (MathDocument.kt) against bundled
assets/katex/ (KaTeX 0.16.45, same version as web). The view is
network-blocked (blockNetworkLoads + shouldInterceptRequest
refuses http/https/ws/wss) with file/content access disabled;
asset URLs still resolve through the file:///android_asset/katex/
base URL. Theme color and font size are injected per render and
the document reports its height through a JS bridge
(ResizeObserver + onPageFinished re-measure). KaTeX failures
follow the web's three-tier semantics, degrading to monospace
raw TeX.

Inline \(..\) and single-paragraph $$ math degrades to monospace
TeX for v1: AnnotatedString cannot host a WebView and one WebView
per fragment is too costly in chat lists. The parser still emits
MathInlineNode so a future renderer can upgrade inline rendering
without touching the extraction layer.

Co-Authored-By: Claude Code <noreply@anthropic.com>
via [HAPI](https://hapi.run)
Co-Authored-By: HAPI <noreply@hapi.run>

* fix(android): convert CSS pixels correctly at the math WebView boundary

The JS bridge reports scrollHeight in CSS pixels, but KaTeXMathBlock
treated the value as Android physical pixels and divided it by the
screen density when sizing the modifier: on a density-3 device a
reported 90 CSS px block collapsed to 30dp instead of ~90dp. The font
conversion had the opposite mismatch — 16sp chat text went through
TextUnit.toPx() (physical pixels) and was injected as `font-size: 48px`
CSS, making formulas render ~3x too large on the same screen.

Define the two boundary conversions explicitly as pure functions next
to the document builder and unit-test them:

- spFontToCssPx: CSS px and sp are both density-independent, so only
  the user's font-scale preference crosses the boundary; density never
  enters (16sp stays 16 CSS px at any density, 21 at fontScale 1.3).
- cssPxToDp: reported CSS px map 1:1 to dp at the default WebView
  scale, multiplied by WebView.scale when the view is zoomed.

The document now also pins the viewport with
`width=device-width, initial-scale=1`, making the 1 CSS px = 1 dp
default scale an explicit property of the document rather than a
WebView default, with a regression test asserting the meta tag.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(android): refresh the math WebView height callback across spec changes

KaTeXMathBlock keyed its height state with remember(spec) but assigned
MathWebView.onHeightChanged inside AndroidView.factory, which runs only
when the Android view is created. The callback therefore captured the
initial spec's state object: when LaTeX, theme color or font size
changed, remember(spec) minted a new state while AndroidView retained
the existing view and its stale callback, so every measurement from the
reloaded document updated the orphaned state and the current block
stayed at its 22dp minimum.

Keep one stable holder (MathViewHeightState) remembered unkeyed for the
mounted view, reset it explicitly in onRender when the spec changes, and
re-point the view's callback plus reset the height inside
AndroidView.update before rendering — the creation-time callback then
always feeds live state. JVM tests drive the holder through spec
changes on one mounted view (the pre-fix wiring cannot compile against
this shape: the factory-captured remember(spec) state is exactly the
orphan the holder removes).

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(protocol): bound dollar flow math closing to the enclosing container

findFlowClose kept scanning after the opening line left its container
and accepted an unprefixed closing run, so `> $$\n> x\n\nOrdinary
prose\n\n$$\ny\n$$` paired the quoted opener with the later top-level
`$$`: the prose was swallowed into the formula and the final `$$` lost
its own opener. remark-math (verified against micromark-extension-math,
whose math flow is `concrete` and dies with its container) renders the
quote's math, the prose and a separate top-level equation instead.

Bound matching to the opening container, distinguishing container
shapes: a closer is only accepted from a line that still carries the
container prefix; a blank line ends a blockquote (a following `> $$`
opens a NEW quote, while a `>`-only line is an in-quote blank the
container survives); a list item may resume after blank lines (loose
lists), so blanks are skipped and the next content line decides; an
unrecognized prefix is plain <= 3 space indentation, not a container,
and keeps the unbounded scan (blank lines and fences stay math content).

Regression tests: the quoted example (prose stays prose, the later
equation keeps its own opener), a list-container variant, a
blank-line/new-quote guard, plus two in-container close locks — the
first three fail on the pre-fix scan.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* feat(android): render mermaid fences as offline diagrams

Port the web MermaidDiagram component to Android chat markdown. A fence
whose language is exactly `mermaid` (case-sensitive, like the web's
MARKDOWN_COMPONENTS_BY_LANGUAGE key) now dispatches to MermaidDiagramView
instead of the plain code block; every other fence keeps the CodeBlock
path.

Rendering runs in a hardened offline WebView fed a self-built HTML
document against bundled assets/mermaid/mermaid.min.js (Mermaid 11.14.0,
curl-fetched, sha256-identical to the web app's locked dependency). That
dist file is a self-contained classic script assigning globalThis
mermaid - no imports, no runtime fetches - so no ES-module/CORS
workaround is needed. The view shares the KaTeX skeleton: network stack
blocked, file/content access off, asset URLs through the
file:///android_asset/mermaid/ base URL (OfflineMathWebViewClient
generalized to OfflineMarkdownWebViewClient, no behavior change).

Behavior mirrors the web component line by line: initialize with
startOnLoad:false, securityLevel:'strict', suppressErrorRendering:true
and the web's dark/light themeVariables palettes verbatim (theme flips
re-render); the validate-then-render gate parses with suppressErrors
first and never calls render() after a failed validation, re-parsing
once without suppression only to surface a human-readable reason; svg
element ids are per-instance (`mermaid-<n>`, web useId) so the scoped
styles mermaid emits cannot collide across diagrams in one message;
wide diagrams scroll horizontally inside the document while the svg is
centered at max-width:100% and height:auto.

The code header (language chip + copy) stays above the body like the
web's retained DefaultCodeBlock header; until the diagram renders - and
on failure - the raw source is shown below it (the block's source exit),
with the failure notice appearing only for actual errors so the loading
window never flashes a render-error line. mermaidFallbackUi maps state
to those surfaces as a pure function.

15 JVM tests cover dispatch, offline-ness, palettes, security flags,
the validation gate structure, id uniqueness, generation filtering,
script-breakout safety and the fallback mapping; the driver's runtime
semantics were additionally validated against the real bundle in jsdom
(flowchart/sequence render, invalid source surfaces the parse reason,
hostile labels are rejected/sanitized under strict).

Co-Authored-By: Claude Code <noreply@anthropic.com>
via [HAPI](https://hapi.run)
Co-Authored-By: HAPI <noreply@hapi.run>

* fix(android): convert CSS pixel heights correctly in the mermaid WebView

Both height reporters feeding MermaidDiagramView — the document bridge's
report() (documentElement/body scrollHeight) and
OfflineMarkdownWebViewClient's onPageFinished re-measure — return CSS
pixels, but the rendered branch ran the value through
with(density) { heightPx.toDp() }, which treats it as Android physical
pixels and divides by display density: on a 3x display a diagram
measuring 180 CSS px received a 60dp box and was clipped into an
unnecessarily scrolling WebView.

Fix it exactly like the KaTeX base now does: the document pins
`width=device-width, initial-scale=1` (making 1 CSS px = 1 dp at the
default WebView scale an explicit property of the document rather than a
WebView default), and the reported value crosses the boundary through
the shared density-independent cssPxToDp helper — extracted as the pure
mermaidRenderedHeightDp so the composable no longer holds any density
conversion at all, keeping both markdown renderers on one boundary
contract. (The KaTeXMathBlock repeat of the same bug is fixed by the
rebased base; MathWebView.kt arrives converted already.)

JVM tests: the viewport pin is asserted against the built document
(demonstrated failing on the pre-fix template) and the conversion math
is driven through the pure helper — 180 CSS px stays 180dp because the
boundary carries no density parameter, mirroring the base's
MathDocumentTest contract.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* feat(android): generalize generated media blocks to video/audio/file

The hub's /generated-images/:id endpoint already serves any mime type
(video, audio, arbitrary files arrive as the same generated-image wire
block with a mimeType), but the Android chat rendered every block as an
image. This ports the web GeneratedImageCard generalization so the card
routes by display intent, exactly like web generatedInlineMedia.ts:

- mimeType routing is the web's literal prefix check: video/… gets a
  player, audio/… an audio player, image/… or null keep the existing
  image treatment (unchanged AsyncImage thumbnail + tap-to-zoom dialog),
  everything else is a file card (no separate "unsupported" class —
  unknown mimes ARE the file UI, matching web G3/G8).
- non-image media never downloads on its own: "Load video" / "Load
  audio" / "Prepare download" gate buttons (web's explicit-intent rule
  G4; English labels verbatim from the web card), pulsing skeleton
  while fetching (G7), and the shared "{label} is unavailable. {error}"
  error line (G8). The wire carries no tiny/poster fields, so video
  renders bare — nothing to special-case.
- every kind now sits in the web's G14 card shell with the localized
  "{label}: {fileName}" hint header; images additionally surface the
  error line on Coil failure instead of a silent empty hole.
- downloads go through the same authed OkHttp client Coil uses (JWT +
  silent 401 re-auth + ETag disk cache), staged into a temp file and
  atomically renamed under cache/generated-media/; a completed file
  short-circuits later fetches (immutable, ETag-stable content) and
  non-2xx maps to "HTTP <code>" exactly like the web's ApiError.
- video plays in-app via framework VideoView + MediaController (no new
  media dependencies: Media3 is not on the classpath and pulling it in
  for this card is not worth the MB); audio is a compact
  play/pause + scrub + m:ss row over MediaPlayer (web's native <audio>
  chrome degraded: volume/speed omitted, no waveform — web has none);
  files get Open (ACTION_VIEW over the existing FileProvider, new
  generated-media cache path) and Save (ACTION_CREATE_DOCUMENT picker)
  outlets in place of the browser download link.
- ChatMedia gains a nullable fetchGeneratedMedia field (default null:
  previews/tests degrade to the unavailable line, so stack-only call
  sites merge unchanged).

17 new strings added to en/ru/zh-CN in one commit, mirroring en key
order (KEYSYNC clean; zh count equal — its pre-existing regional order
drift is untouched). 10 JVM tests cover routing (including case
sensitivity and null-mime), mount-gating, label/action mapping, cache
file name sanitizing (path traversal), the MockWebServer round-trip
with status mapping + cached-file short-circuit, and the time
formatter. Gate: :app:testDebugUnitTest :core:data:test
:core:protocol:test :app:lintDebug :app:compileDebugAndroidTestKotlin
BUILD SUCCESSFUL.

Co-Authored-By: Claude Code <noreply@anthropic.com>
via [HAPI](https://hapi.run)
Co-Authored-By: HAPI <noreply@hapi.run>

* fix(android): refine generic file MIME types from the filename

The file card passed the wire mimeType straight through to ACTION_VIEW,
substituting application/octet-stream only when the wire type was null
(r4370795783). Passing octet-stream explicitly defeats Android's viewer
matching: a PDF viewer registered for application/pdf is not matched by
an intent whose type says application/octet-stream, so PDFs (and every
other ordinary file whose hub type was octet-stream or missing) opened
with no eligible activity.

resolveFileMimeType now passes a specific wire type through untouched,
but when the wire type is missing, empty, or octet-stream (compared
case-insensitively) it re-derives the type from the original filename's
extension through an injected resolver — android.webkit.MimeTypeMap at
the call site, following the established AttachmentPreparer /
AndroidShareIntake pattern (lowercased extension, slash/wildcard sanity
check on the resolved value). Unknown extensions keep the generic
octet-stream fallback. The resolved type feeds both outlets: the Open
ACTION_VIEW intent and the Save ACTION_CREATE_DOCUMENT contract.

The resolver is a required parameter rather than a default, which keeps
GeneratedMedia.kt free of android.* imports and forces the seam: 3 JVM
tests drive it with a fake table (pdf->application/pdf via extension for
null/octet-stream/empty wire types; unknown extension and no extension
-> octet-stream fallback; specific wire types pass through untouched
even when the extension would map elsewhere). All three fail on the
pre-fix code by construction — resolveFileMimeType did not exist and the
null-wire PDF case asserted the old octet-stream handoff.

Gate: :app:testDebugUnitTest --tests GeneratedMediaTest BUILD SUCCESSFUL
(13 tests, 0 failures).

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(android): bound generated media cache names by UTF-8 bytes

sanitizedGeneratedMediaFileName capped the stem at 96 UTF-16 chars and
relied on that alone (r4370795787). Android filename limits are byte
limits (NAME_MAX 255): 80 hanzi + ".pdf" is a 122-char / 281-byte
component once the UUID imageId prefix is attached (verified against the
pre-fix logic: 36-byte id + 1 + 240 + 4 = 281 bytes), so long Chinese
names failed the atomic cache rename outright.

The complete component — id prefix + stem + extension — is now assembled
under a 255-byte UTF-8 budget, with pathological ids and extensions
capped at 64/32 bytes so the stem always keeps room. The stem keeps its
historical 96-char cap (ASCII names keep their exact prior shape), bytes
beyond the budget are cut from the stem only, on code-point boundaries,
and the extension always survives intact. The char-level whitelist
already drops surrogate halves (a lone half is never a letter), so every
surviving char is one complete BMP code point; the truncation helpers
are additionally pair-safe for any future code-point-aware whitelist.

JVM tests: the reviewer's UUID + 80-hanzi ".pdf" case lands at <= 255
bytes with the extension and id prefix preserved (red on the pre-fix
logic, which produced exactly 281); ".tar.gz"-style extensions survive
stem truncation; a battery of multibyte names re-decode byte-identically
under the cap; ASCII names keep their historical shapes ("i-" + 96 a's,
"img-1-media", "id-x.bin", "i-截图01.png").

Gate: :app:testDebugUnitTest --tests GeneratedMediaTest BUILD SUCCESSFUL
(16 tests, 0 failures).

Co-Authored-By: Claude Code <noreply@anthropic.com>

* test(android): cover audio playback wiring and the file outlets' types

The play/pause/seek decisions and the MediaPlayer lifecycle lived inline
in the audio composable, which made them unreachable from JVM tests (no
emulator here; androidTest only compile-checks). Extract them into
GeneratedAudioSession over a GeneratedAudioPlayer seam — the app side
adapts android.media.MediaPlayer, tests pass a scriptable fake — and the
card now collects the session's flows, with DisposableEffect.onDispose
forwarding to session.release().

JVM coverage through the fake:
 - prepare wires the data source + prepareAsync and reports duration on
   the prepared callback; pre-preparation toggles/seeks stay inert;
 - the toggle maps to start/pause (turbine on the playing flow) and
   natural completion parks the row;
 - seekToFraction lands fraction*duration and clamps to [0, duration],
   reading the position back afterwards;
 - pollPosition only reads while playing (the 250 ms LaunchedEffect);
 - release — the dispose hook — releases the framework player exactly
   once across double-dispose, stops playback, and every later control
   call is inert (a released MediaPlayer throws);
 - player errors and a failing data source both map to the failed line
   and the row goes dead.

The file card's Open/Save construction moves behind generatedFileOutlets
so a JVM test pins that ACTION_VIEW and ACTION_CREATE_DOCUMENT both
carry the *resolved* MIME type (inferred pdf, generic fallback, specific
wire passthrough) plus the suggested save name — android.content.Intent
is not on the unit-test classpath. A MockWebServer fetch with the
reviewer's 80-hanzi + UUID name proves the byte-bounded component really
round-trips the local filesystem: under 255 bytes, extension kept, no
staging litter beside the final file.

These tests fail on the pre-change code by construction: the session,
seam, and outlets did not exist and the logic was not reachable from
the JVM. Explicit loading (loadsOnMount gate), authenticated caching
(hubGraph.session.imageClient: JWT + silent 401 re-auth + ETag cache),
and download cleanup (staged temp deleted on failure, atomic rename on
success) are untouched — only their test surface grew.

Gate: :app:testDebugUnitTest --tests GeneratedMediaTest BUILD SUCCESSFUL
(24 tests, 0 failures).

Co-Authored-By: Claude Code <noreply@anthropic.com>

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: HAPI <noreply@hapi.run>
This commit is contained in:
admin
2026-11-02 18:09:49 +08:00
committed by GitHub
co-authored by Claude Code HAPI
parent 3208199f99
commit be5f37b9bf
89 changed files with 7815 additions and 51 deletions
@@ -0,0 +1,635 @@
package app.hapi.protocol.markdown
import org.commonmark.node.Code
import org.commonmark.node.Document
import org.commonmark.node.FencedCodeBlock
import org.commonmark.node.HardLineBreak
import org.commonmark.node.HtmlBlock
import org.commonmark.node.HtmlInline
import org.commonmark.node.Image
import org.commonmark.node.IndentedCodeBlock
import org.commonmark.node.Link
import org.commonmark.node.LinkReferenceDefinition
import org.commonmark.node.Node
import org.commonmark.node.Paragraph
import org.commonmark.node.SoftLineBreak
import org.commonmark.node.Text
/**
* Math extraction, port of the web pipeline so both renderers agree on chat
* markdown (`web/src/lib/remark-latex-bracket-math.ts` + `remark-math`
* `singleDollarTextMath: false` semantics):
*
* - `$$...$$` on its own lines -> [MathKind.DISPLAY] (micromark math flow);
* mid-paragraph `$$...$$` -> [MathKind.INLINE] (math text), closing run must
* be the same size as the opening run.
* - Single `$...$` NEVER opens math (currency protection, GitHub parity).
* - `\[...\]` -> [MathKind.DISPLAY], `\( ... \)` -> [MathKind.INLINE]; the
* opener `\(`/`\[` may be escaped by an odd run of backslashes.
* - Code spans, fenced/indented code, html and link/image/definition
* destinations never trigger math, and an unclosed opener is left as text.
*
* Mechanism (same as the web plugin): parse the repaired source once to learn
* which source ranges are protected, replace every formula with private-use
* placeholder tokens that keep the line shape (so the re-parse does not split
* the paragraph), re-parse the safe source, then restore the tokens as
* [MathBlockNode]/[MathInlineNode] and hoist display math out of paragraphs.
*
* Pure JVM on purpose: unit-tested in `:core:protocol` (MarkdownMathTest
* mirrors the web remark-latex-bracket-math/markdown-text test names).
*/
internal object MarkdownMath {
/** Cheap pre-check: sources without any of these substrings skip the whole two-parse dance. */
val MATH_HINT: Regex = Regex("""\$\$|\\\[|\\\(""")
private const val PLACEHOLDER_BASE = 0xE000
private const val PLACEHOLDER_END = 0xF8FF
private val LIST_MARKER: Regex = Regex("""^(?:[-+*]|\d{1,9}[.)])[ \t]+""")
// ── Result types ─────────────────────────────────────────────────────────
/** One extracted formula plus its source span and its container prefix. */
internal class MathMatch(
val kind: MathKind,
val value: String,
val start: Int,
val end: Int,
val prefix: String,
) {
internal var token: Char = ' '
}
/** The token-replaced source plus every extracted formula. */
internal class PreparedMath(val safeSource: String, val matches: List<MathMatch>)
/**
* Full pipeline for sources that pass [MATH_HINT]: mask-protected parse,
* extraction, placeholder re-parse and restore. Returns the final document.
*/
fun parseWithMath(
source: String,
spannedParse: (String) -> Node,
plainParse: (String) -> Node,
): Document {
val maskTree = spannedParse(source)
val prepared = extract(source, maskTree as Document) ?: return maskTree as Document
val document = plainParse(prepared.safeSource) as Document
restore(document, prepared)
return document
}
// ── Extraction ───────────────────────────────────────────────────────────
/**
* Find every formula in [source], using [tree] (parsed with source spans
* enabled) to learn protected ranges. Returns null when there is no math,
* in which case the caller keeps the original tree as-is.
*/
fun extract(source: String, tree: Document): PreparedMath? {
val mask = BooleanArray(source.length)
markProtectedRanges(source, tree, mask)
val matches = mutableListOf<MathMatch>()
matches += findDollarMath(source, mask)
// Existing $$ math is opaque to the bracket scan (web SOURCE_OPAQUE
// contains math/inlineMath: `$$\verb|\(x\)|$$` keeps its inner `\(`).
for (match in matches) markRange(mask, match.start, match.end)
matches += findBracketMathMatches(source, mask)
if (matches.isEmpty()) return null
matches.sortBy { it.start }
val out = StringBuilder(source.length)
var cursor = 0
var nextToken = PLACEHOLDER_BASE
for (match in matches) {
val token = findUnusedPlaceholderToken(source, nextToken)
nextToken = token.code + 1
match.token = token
out.append(source, cursor, match.start)
out.append(makeSourcePlaceholder(source.substring(match.start, match.end), match.prefix, token))
cursor = match.end
}
out.append(source, cursor, source.length)
return PreparedMath(out.toString(), matches.toList())
}
// ── Dollar math (remark-math `singleDollarTextMath: false` semantics) ────
/**
* `$$` runs: flow math when the run opens a line (after a container
* prefix or <= 3 spaces) with only whitespace after it, closing on a later
* line inside the same container that is just a `$$` run of >= the opening
* size; otherwise text math inside a paragraph, closing on an equal-sized
* run with no blank line or protected range in between. Single `$` never
* opens math.
*/
private fun findDollarMath(source: String, mask: BooleanArray): List<MathMatch> {
val matches = mutableListOf<MathMatch>()
var cursor = 0
while (cursor < source.length) {
val p = source.indexOf('$', cursor)
if (p < 0) break
var size = 0
while (p + size < source.length && source[p + size] == '$') size++
if (size < 2 || isProtected(mask, p, size) || !isUnescapedDelimiter(source, p)) {
cursor = p + 1
continue
}
val lineStart = source.lastIndexOf('\n', p - 1) + 1
val lineEnd = source.indexOf('\n', p).let { if (it < 0) source.length else it }
val container = getMarkdownContainerPrefix(source, p)
val lineIndent = source.substring(lineStart, p).takeWhile { it == ' ' || it == '\t' }.length
val restOfLineBlank = source.substring(p + size, lineEnd).isBlank()
val atFlowPosition = p - lineStart == container.prefix.length &&
(container.recognized || lineIndent <= 3)
if (atFlowPosition && restOfLineBlank) {
val close = findFlowClose(source, mask, size, container.prefix, container.recognized, lineEnd)
if (close != null) {
val value = stripLinePrefix(source.substring(lineEnd + 1, close.start), container.prefix).trim()
if (value.isNotEmpty()) {
matches.add(MathMatch(MathKind.DISPLAY, value, p, close.end, container.prefix))
cursor = close.end
continue
}
}
// Unclosed flow math falls through to the paragraph-text rules,
// matching micromark: the opener line stays a paragraph.
}
val textClose = findTextClose(source, mask, p, size)
if (textClose != null) {
val raw = source.substring(p + size, textClose)
val value = trimMirroredWhitespace(raw)
if (value.isNotEmpty()) {
matches.add(MathMatch(MathKind.INLINE, value, p, textClose + size, ""))
cursor = textClose + size
continue
}
}
cursor = p + size
}
return matches
}
private class CloseSpan(val start: Int, val end: Int)
/**
* Closing `$$` line for flow math: container prefix / <= 3 spaces, then a `$` run >= [size].
*
* Bounded to the opening container (micromark math flow is `concrete` and
* ends with its container): when [containerRecognized] the closer must
* come from a line that still carries [prefix] — a quoted opener never
* pairs with a later top-level or sibling `$$`. A blank line closes a
* blockquote (a following `> $$` opens a NEW quote, not a closer), while a
* list item may resume after blank lines (loose lists), so blanks are
* skipped and the next content line decides. An unrecognized prefix is
* plain <= 3 space indentation, not a container: no bounding, blank lines
* are math content (web parity with `$$\nx\n\ny\n$$`).
*/
private fun findFlowClose(
source: String,
mask: BooleanArray,
size: Int,
prefix: String,
containerRecognized: Boolean,
openLineEnd: Int,
): CloseSpan? {
val marker = prefix.trimEnd()
val quoted = marker.isNotEmpty() && marker.last() == '>'
var lineStart = openLineEnd + 1
while (lineStart < source.length) {
val lineEnd = source.indexOf('\n', lineStart).let { if (it < 0) source.length else it }
val line = source.substring(lineStart, lineEnd)
if (containerRecognized && !line.startsWith(prefix)) {
when {
// A `>`-only line is a blank line inside the quote: the
// container survives it ("> $$\n> x\n>\n> $$" closes).
quoted && line.trimEnd(' ', '\t') == marker -> Unit
// Blank line: the blockquote dies here, a list item may
// resume after it — keep scanning for the list case only.
line.isBlank() && !quoted -> Unit
else -> return null
}
lineStart = lineEnd + 1
continue
}
var contentStart = lineStart
var content = line
if (prefix.isNotEmpty() && content.startsWith(prefix)) {
contentStart += prefix.length
content = content.substring(prefix.length)
}
var indent = 0
while (indent < content.length && indent < 3 && (content[indent] == ' ' || content[indent] == '\t')) indent++
val rest = content.substring(indent)
var run = 0
while (run < rest.length && rest[run] == '$') run++
if (run >= size && rest.substring(run).isBlank() && !isProtected(mask, contentStart + indent, run)) {
val runStart = contentStart + indent
return CloseSpan(runStart, runStart + run)
}
lineStart = lineEnd + 1
}
return null
}
/**
* Lines that would interrupt the opening paragraph (heading, blockquote,
* fence, thematic break): micromark math-text cannot cross them, so the
* search fails there. A blank line is handled inline below.
*/
private val PARAGRAPH_INTERRUPT: Regex = Regex("""^[ \t]{0,3}(?:#|>|```|~~~|[-*_][ \t]*[-*_][ \t]*[-*_])""")
/** Closing equal-sized `$$` run for text math: same paragraph, unescaped, nothing protected in between. */
private fun findTextClose(source: String, mask: BooleanArray, openStart: Int, size: Int): Int? {
var i = openStart + size
while (i < source.length) {
val c = source[i]
if (c == '\n') {
// A blank line ends the paragraph; text math cannot cross it.
var j = i + 1
while (j < source.length && (source[j] == ' ' || source[j] == '\t')) j++
if (j >= source.length || source[j] == '\n') return null
// Nor can another block construct that interrupts paragraphs.
if (PARAGRAPH_INTERRUPT.containsMatchIn(source.substring(j).lineSequence().first())) return null
i++
} else if (c == '$') {
var run = 0
while (i + run < source.length && source[i + run] == '$') run++
if (run == size && !isProtected(mask, i, run) && isUnescapedDelimiter(source, i)) {
return if (containsProtectedRange(mask, openStart, i)) null else i
}
i += run
} else {
i++
}
}
return null
}
/** micromark math-text resolve: trimmed only when BOTH edges are whitespace. */
private fun trimMirroredWhitespace(value: String): String {
if (value.isEmpty()) return value
val firstWs = value.first().isWhitespace()
val lastWs = value.last().isWhitespace()
return if (firstWs && lastWs) value.trim() else value
}
// ── Bracket math (remark-latex-bracket-math port) ────────────────────────
private fun findBracketMathMatches(source: String, mask: BooleanArray): List<MathMatch> {
val matches = mutableListOf<MathMatch>()
var cursor = 0
while (true) {
val displayStart = findNextDelimiter(source, "\\[", cursor, mask)
val inlineStart = findNextDelimiter(source, "\\(", cursor, mask)
val start: Int
val kind: MathKind
if (displayStart >= 0 && (inlineStart < 0 || displayStart < inlineStart)) {
start = displayStart
kind = MathKind.DISPLAY
} else if (inlineStart >= 0) {
start = inlineStart
kind = MathKind.INLINE
} else {
break
}
val closingDelimiter = if (kind == MathKind.DISPLAY) "\\]" else "\\)"
val closingStart = findClosingDelimiter(source, closingDelimiter, start + 2, mask)
if (closingStart < 0) {
cursor = start + 2
continue
}
val prefix = getMarkdownContainerPrefix(source, start).prefix
val value = stripLinePrefix(source.substring(start + 2, closingStart), prefix).trim()
val end = closingStart + 2
if (value.isNotEmpty()) matches.add(MathMatch(kind, value, start, end, prefix))
cursor = end
}
return matches
}
/** An odd run of backslashes before the delimiter escapes the final backslash. */
private fun isUnescapedDelimiter(source: String, offset: Int): Boolean {
var precedingBackslashes = 0
var index = offset - 1
while (index >= 0 && source[index] == '\\') {
precedingBackslashes++
index--
}
return precedingBackslashes % 2 == 0
}
private fun findNextDelimiter(source: String, delimiter: String, from: Int, mask: BooleanArray): Int {
var offset = source.indexOf(delimiter, from)
while (offset >= 0 && (isProtected(mask, offset, delimiter.length) || !isUnescapedDelimiter(source, offset))) {
offset = source.indexOf(delimiter, offset + 1)
}
return offset
}
private fun findClosingDelimiter(source: String, delimiter: String, from: Int, mask: BooleanArray): Int {
val offset = findNextDelimiter(source, delimiter, from, mask)
return if (offset >= 0 && !containsProtectedRange(mask, from, offset)) offset else -1
}
// ── Container prefixes ───────────────────────────────────────────────────
internal class ContainerPrefix(val prefix: String, val recognized: Boolean)
/**
* Container prefix (blockquote markers, list markers, <= 3 spaces) that
* precedes [offset] on its line; list markers normalize to spaces
* (continuation indent). Returns the raw leading text when no container is
* recognized and the delimiter is preceded by non-blank prose (web parity).
*/
private fun getMarkdownContainerPrefix(source: String, offset: Int): ContainerPrefix {
val lineStart = source.lastIndexOf('\n', offset - 1) + 1
val beforeDelimiter = source.substring(lineStart, offset)
var cursor = 0
val continuation = StringBuilder()
var recognizedContainer = false
while (cursor < beforeDelimiter.length) {
val containerStart = cursor
while (cursor < beforeDelimiter.length &&
(beforeDelimiter[cursor] == ' ' || beforeDelimiter[cursor] == '\t')
) cursor++
if (cursor - containerStart > 3) {
cursor = containerStart
break
}
if (cursor < beforeDelimiter.length && beforeDelimiter[cursor] == '>') {
cursor++
if (cursor < beforeDelimiter.length &&
(beforeDelimiter[cursor] == ' ' || beforeDelimiter[cursor] == '\t')
) cursor++
continuation.append(beforeDelimiter, containerStart, cursor)
recognizedContainer = true
continue
}
val listMarker = LIST_MARKER.find(beforeDelimiter.substring(cursor))
if (listMarker != null) {
repeat(cursor - containerStart + listMarker.value.length) { continuation.append(' ') }
cursor += listMarker.value.length
recognizedContainer = true
continue
}
cursor = containerStart
break
}
if (!recognizedContainer && beforeDelimiter.isNotBlank()) return ContainerPrefix("", false)
val prefix = continuation.toString().ifEmpty { beforeDelimiter }
return ContainerPrefix(prefix, recognizedContainer)
}
/** Remove [prefix] from the start of every line in [value], including the first. */
private fun stripLinePrefix(value: String, prefix: String): String {
if (prefix.isEmpty()) return value
return value.replace(Regex("(^|\\n)" + Regex.escape(prefix)), "$1")
}
// ── Placeholder construction ─────────────────────────────────────────────
private fun findUnusedPlaceholderToken(source: String, start: Int): Char {
for (code in start..PLACEHOLDER_END) {
if (!source.contains(code.toChar())) return code.toChar()
}
throw IllegalStateException("Unable to allocate a LaTeX placeholder token")
}
/**
* Replace every line of the raw span with the placeholder token, keeping
* the line shape (newlines, container prefix) so the re-parse keeps the
* formula inside its original block; an empty line takes one token so the
* paragraph is not split (web `makeSourcePlaceholder`).
*/
private fun makeSourcePlaceholder(raw: String, prefix: String, token: Char): String {
val out = StringBuilder(raw.length)
val lines = raw.split('\n')
for (index in lines.indices) {
if (index > 0) out.append('\n')
val line = lines[index]
if (index == 0) {
repeat(maxOf(1, line.length)) { out.append(token) }
} else {
val kept = if (prefix.isNotEmpty() && line.startsWith(prefix)) prefix else ""
out.append(kept)
repeat(maxOf(1, line.length - kept.length)) { out.append(token) }
}
}
return out.toString()
}
// ── Protected ranges (mask) ──────────────────────────────────────────────
/**
* Ranges the math scan must never touch, learned from a parse with
* `IncludeSourceSpans.BLOCKS_AND_INLINES`: code blocks/html blocks/
* definitions consume whole lines; inline code, inline html and links
* (incl. images and autolinks, which the autolink extension emits as
* [Link]s) protect their exact span. Web parity note: the web plugin only
* protects link DESTINATIONS (math inside `[label]` renders); here the
* whole link is protected, which cannot corrupt anything and at most
* leaves label math as literal text.
*/
private fun markProtectedRanges(source: String, tree: Document, mask: BooleanArray) {
val lineStarts = lineStartOffsets(source)
val stack = ArrayDeque<Node>()
stack.addLast(tree)
while (stack.isNotEmpty()) {
val node = stack.removeLast()
when (node) {
is FencedCodeBlock, is IndentedCodeBlock, is HtmlBlock, is LinkReferenceDefinition -> {
for (span in node.sourceSpans) protectFullLine(mask, lineStarts, source.length, span.lineIndex)
}
is Code, is HtmlInline, is Link, is Image -> {
for (span in node.sourceSpans) {
if (span.lineIndex < lineStarts.size) {
val start = lineStarts[span.lineIndex] + span.columnIndex
markRange(mask, start, start + span.length)
}
}
}
}
var child = node.firstChild
while (child != null) {
stack.addLast(child)
child = child.next
}
}
}
private fun lineStartOffsets(source: String): IntArray {
val starts = ArrayList<Int>()
starts.add(0)
for (index in source.indices) {
if (source[index] == '\n') starts.add(index + 1)
}
return starts.toIntArray()
}
private fun protectFullLine(mask: BooleanArray, lineStarts: IntArray, sourceLength: Int, lineIndex: Int) {
if (lineIndex >= lineStarts.size) return
val start = lineStarts[lineIndex]
val end = if (lineIndex + 1 < lineStarts.size) lineStarts[lineIndex + 1] - 1 else sourceLength
markRange(mask, start, end)
}
private fun markRange(mask: BooleanArray, start: Int, end: Int) {
for (index in maxOf(0, start) until minOf(mask.size, end)) mask[index] = true
}
private fun isProtected(mask: BooleanArray, start: Int, length: Int): Boolean {
for (index in start until minOf(mask.size, start + length)) {
if (mask[index]) return true
}
return false
}
private fun containsProtectedRange(mask: BooleanArray, start: Int, end: Int): Boolean {
for (index in maxOf(0, start) until minOf(mask.size, end)) {
if (mask[index]) return true
}
return false
}
// ── Restore ──────────────────────────────────────────────────────────────
/**
* Turn placeholder tokens in the re-parsed [document] into math nodes:
* split text node runs into [Text]/[MathInlineNode]/[MathBlockNode]
* (fragments of one formula across soft line breaks collapse to a single
* node), then hoist display math out of paragraphs (web
* `splitParagraphAroundDisplayMath`), dropping whitespace-only leftovers.
*/
fun restore(document: Document, prepared: PreparedMath) {
val byToken = HashMap<Char, MathMatch>()
for (match in prepared.matches) byToken[match.token] = match
val emitted = HashSet<Char>()
// Pass 1: split every text node that contains a placeholder run.
val texts = ArrayList<Text>()
val stack = ArrayDeque<Node>()
stack.addLast(document)
while (stack.isNotEmpty()) {
val node = stack.removeLast()
if (node is Text) texts.add(node)
var child = node.firstChild
while (child != null) {
stack.addLast(child)
child = child.next
}
}
for (text in texts) {
val literal = text.literal.orEmpty()
var hasToken = false
for (c in literal) {
if (byToken.containsKey(c)) {
hasToken = true
break
}
}
if (!hasToken) continue
var cursor = 0
var index = 0
while (index < literal.length) {
val match = byToken[literal[index]]
if (match == null) {
index++
continue
}
var runEnd = index
while (runEnd < literal.length && literal[runEnd] == literal[index]) runEnd++
if (index > cursor) text.insertBefore(Text(literal.substring(cursor, index)))
if (emitted.add(match.token)) {
text.insertBefore(
if (match.kind == MathKind.DISPLAY) MathBlockNode(match.value) else MathInlineNode(match.value),
)
}
cursor = runEnd
index = runEnd
}
if (cursor < literal.length) text.insertBefore(Text(literal.substring(cursor)))
text.unlink()
}
// Pass 2: hoist display math out of paragraphs.
val paragraphs = ArrayList<Paragraph>()
stack.addLast(document)
while (stack.isNotEmpty()) {
val node = stack.removeLast()
if (node is Paragraph && node.firstChild?.next == null) {
// Fast path: single-child paragraphs either hold one math node
// (hoist) or nothing to do; still collected for uniform handling.
}
if (node is Paragraph) paragraphs.add(node)
var child = node.firstChild
while (child != null) {
stack.addLast(child)
child = child.next
}
}
for (paragraph in paragraphs) {
var hasDisplay = false
var child = paragraph.firstChild
while (child != null) {
if (child is MathBlockNode) {
hasDisplay = true
break
}
child = child.next
}
if (!hasDisplay) continue
var group = ArrayList<Node>()
val flush = {
val hasContent = group.any { !isBlankInline(it) }
if (hasContent) {
val rest = Paragraph()
for (inline in group) {
inline.unlink()
rest.appendChild(inline)
}
paragraph.insertBefore(rest)
} else {
for (inline in group) inline.unlink()
}
group = ArrayList()
}
var node = paragraph.firstChild
while (node != null) {
val next = node.next
if (node is MathBlockNode) {
flush()
node.unlink()
paragraph.insertBefore(node)
} else {
group.add(node)
}
node = next
}
flush()
paragraph.unlink()
}
}
/** Whitespace-only text, soft and hard line breaks carry no paragraph content. */
private fun isBlankInline(node: Node): Boolean = when (node) {
is Text -> node.literal.orEmpty().isBlank()
is SoftLineBreak, is HardLineBreak -> true
else -> false
}
}
@@ -11,6 +11,7 @@ import org.commonmark.node.HtmlBlock
import org.commonmark.node.ListBlock
import org.commonmark.node.Node
import org.commonmark.node.ThematicBreak
import org.commonmark.parser.IncludeSourceSpans
import org.commonmark.parser.Parser
/**
@@ -24,6 +25,9 @@ import org.commonmark.parser.Parser
* - [stripCjkAutolinkArtifacts] <- `web/src/lib/remark-strip-cjk-autolink.ts`
* - [detectFilePathLinks] / [matchWholeFilePath] / [rewriteExplicitLinkTarget]
* <- `web/src/lib/remark-file-path-links.ts`
* - [MarkdownMath] <- `web/src/lib/remark-latex-bracket-math.ts` +
* `remark-math {singleDollarTextMath: false}`
* (`parse` extracts `$$`/`\[...\]`/`\(...\)` math nodes)
*
* Pure JVM on purpose: everything here is unit-tested in `:core:protocol`.
*/
@@ -182,12 +186,12 @@ object MarkdownTransforms {
builder.enabledBlockTypes(BLOCK_TYPES_WITHOUT_INDENTED_CODE)
/**
* The canonical HAPI markdown parser: GFM tables + strikethrough (double
* tilde only, matching web `remark-gfm {singleTilde: false}` -- a single
* tilde is common in shell prompts) + bare-URL autolink, with indented code
* blocks disabled. Thread-safe and reusable.
* The canonical HAPI markdown parser builder: GFM tables + strikethrough
* (double tilde only, matching web `remark-gfm {singleTilde: false}` -- a
* single tilde is common in shell prompts) + bare-URL autolink, with
* indented code blocks disabled. Thread-safe and reusable once built.
*/
fun newParser(): Parser = disableIndentedCode(
fun newParserBuilder(): Parser.Builder = disableIndentedCode(
Parser.builder().extensions(
listOf(
TablesExtension.create(),
@@ -195,12 +199,38 @@ object MarkdownTransforms {
AutolinkExtension.create(),
)
)
).build()
)
/** [newParserBuilder] built: the canonical parser. */
fun newParser(): Parser = newParserBuilder().build()
private val sharedParser: Parser by lazy { newParser() }
/** Full source pipeline: [repairTables] then parse with [newParser] config. */
fun parse(source: String): Node = sharedParser.parse(repairTables(source))
/**
* Same configuration plus source spans on blocks and inlines; used only by
* [MarkdownMath] to learn which source ranges are protected (code, links,
* definitions) before scanning for math delimiters.
*/
private val spannedParser: Parser by lazy {
newParserBuilder().includeSourceSpans(IncludeSourceSpans.BLOCKS_AND_INLINES).build()
}
/**
* Full source pipeline: [repairTables] then parse with [newParser] config.
* Sources that look like they may contain math (`$$`, `\[`, `\(`) go
* through [MarkdownMath.parseWithMath], which extracts
* [MathBlockNode]/[MathInlineNode] nodes; everything else takes the plain
* single-parse path.
*/
fun parse(source: String): Node {
val repaired = repairTables(source)
if (!MarkdownMath.MATH_HINT.containsMatchIn(repaired)) return sharedParser.parse(repaired)
return MarkdownMath.parseWithMath(
repaired,
spannedParse = spannedParser::parse,
plainParse = sharedParser::parse,
)
}
// ── CJK autolink artifact strip (remark-strip-cjk-autolink port) ─────────
@@ -0,0 +1,42 @@
package app.hapi.protocol.markdown
import org.commonmark.node.CustomBlock
import org.commonmark.node.CustomNode
/** Whether a formula came from block-level or inline-level delimiters. */
enum class MathKind {
/** `$$...$$` on its own lines or `\[...\]` — web `math` node, `math-display`. */
DISPLAY,
/** `\( ... \)` or `$$...$$` mid-paragraph — web `inlineMath` node, `math-inline`. */
INLINE,
}
/**
* Block-level display math extracted from the source by [MarkdownMath].
*
* Mirrors the web `math` mdast node produced by `remark-latex-bracket-math`
* (`\[...\]`) and `remark-math` flow math (`$$...$$` on their own lines): both
* end up as `pre > code.language-math.math-display`, rendered by rehype-katex
* with `displayMode: true`.
*
* Extends [CustomBlock] so it can be hoisted to a top-level block sibling
* (web `splitParagraphAroundDisplayMath` parity); before hoisting it may
* transiently sit in inline position, where the Compose walker degrades it to
* the monospace inline fallback instead.
*/
class MathBlockNode(val latex: String) : CustomBlock() {
override fun toString(): String = "MathBlockNode(${latex.replace("\n", "\\n")})"
}
/**
* Inline math extracted from `\( ... \)` or mid-paragraph `$$...$$`.
*
* Mirrors the web `inlineMath` mdast node (rendered by rehype-katex without
* display mode). Produced by splitting [org.commonmark.node.Text] nodes that
* contain a placeholder token, so it appears at any inline nesting depth
* (emphasis, links labels excluded — those are protected at extraction time).
*/
class MathInlineNode(val latex: String) : CustomNode() {
override fun toString(): String = "MathInlineNode(${latex.replace("\n", "\\n")})"
}
@@ -0,0 +1,798 @@
package app.hapi.protocol.markdown
import org.commonmark.ext.gfm.tables.TableBlock
import org.commonmark.node.BlockQuote
import org.commonmark.node.Code
import org.commonmark.node.Document
import org.commonmark.node.FencedCodeBlock
import org.commonmark.node.Image
import org.commonmark.node.Link
import org.commonmark.node.ListItem
import org.commonmark.node.Node
import org.commonmark.node.Paragraph
import org.commonmark.node.StrongEmphasis
import org.commonmark.node.Text
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
/**
* Math extraction semantics, mirroring the web test names in
* `web/src/lib/remark-latex-bracket-math.test.ts` and the currency suite of
* `web/src/components/assistant-ui/markdown-text.test.ts` (remark-math with
* `singleDollarTextMath: false`). Assertions are AST-level: which
* [MathBlockNode]/[MathInlineNode] nodes exist, their latex values and block
* placement, and that no private-use placeholder token ever leaks into the
* final tree.
*/
class MarkdownMathTest {
private fun parse(markdown: String): Document = MarkdownTransforms.parse(markdown) as Document
private fun <T : Node> Node.collectOfType(type: Class<T>): List<T> {
val out = ArrayList<T>()
fun walk(node: Node) {
var child = node.firstChild
while (child != null) {
if (type.isInstance(child)) out.add(type.cast(child))
walk(child)
child = child.next
}
}
walk(this)
return out
}
private inline fun <reified T : Node> Node.collect(): List<T> = collectOfType(T::class.java)
private fun Node.allText(): String {
val out = StringBuilder()
fun walk(node: Node) {
var child = node.firstChild
while (child != null) {
if (child is Text) out.append(child.literal)
else walk(child)
child = child.next
}
}
walk(this)
return out.toString()
}
private fun Document.mathBlocks(): List<MathBlockNode> = collectOfType(MathBlockNode::class.java)
private fun Document.mathInlines(): List<MathInlineNode> = collectOfType(MathInlineNode::class.java)
private fun Document.topBlocks(): List<Node> {
val blocks = ArrayList<Node>()
var child = firstChild
while (child != null) {
blocks.add(child)
child = child.next
}
return blocks
}
/** Web parity: `expect(html).not.toContain('\uE000')`. */
private fun assertNoPlaceholderTokens(document: Document) {
for (text in document.collect<Text>()) {
for (ch in text.literal.orEmpty()) {
assertTrue(
ch.code !in 0xE000..0xF8FF,
"placeholder token leaked into text: \"${text.literal}\"",
)
}
}
for (code in document.collect<Code>()) {
for (ch in code.literal.orEmpty()) {
assertTrue(ch.code !in 0xE000..0xF8FF, "placeholder token leaked into code span")
}
}
for (fence in document.collect<FencedCodeBlock>()) {
for (ch in fence.literal.orEmpty()) {
assertTrue(ch.code !in 0xE000..0xF8FF, "placeholder token leaked into fenced code")
}
}
}
// ── Bracket math ─────────────────────────────────────────────────────────
@Test
fun `renders display bracket math`() {
val document = parse("""\[E = mc^2\]""")
val math = document.collect<MathBlockNode>()
assertEquals(listOf("E = mc^2"), math.map { it.latex })
assertEquals(0, document.collect<MathInlineNode>().size)
assertEquals(0, document.collect<Paragraph>().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `renders parenthesized inline math`() {
val document = parse("""Result: \(x^2 + y^2\).""")
val math = document.collect<MathInlineNode>()
assertEquals(listOf("x^2 + y^2"), math.map { it.latex })
assertEquals(0, document.collect<MathBlockNode>().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `renders multiline display math before optional hard-break processing`() {
val document = parse(
"""
Before
\[
E = mc^2
\]
After
""".trimIndent(),
)
val math = document.collect<MathBlockNode>()
assertEquals(listOf("E = mc^2"), math.map { it.latex })
val paragraphs = document.collect<Paragraph>().map { it.allText() }
assertEquals(listOf("Before", "After"), paragraphs)
assertNoPlaceholderTokens(document)
}
@Test
fun `renders multiple inline formulas in one paragraph`() {
val document = parse("""Compare \(a^2\) with \(b^2\).""")
assertEquals(listOf("a^2", "b^2"), document.collect<MathInlineNode>().map { it.latex })
assertEquals(1, document.collect<Paragraph>().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `renders aligned display math with TeX line breaks and indentation`() {
val document = parse(
"""
\[
\begin{aligned}
f(x) &= x^2+2x+1 \\
&= (x+1)^2
\end{aligned}
\]
""".trimIndent(),
)
val math = document.collect<MathBlockNode>()
assertEquals(1, math.size)
assertEquals(
"\\begin{aligned}\nf(x) &= x^2+2x+1 \\\\\n &= (x+1)^2\n\\end{aligned}",
math.single().latex,
)
assertNoPlaceholderTokens(document)
}
@Test
fun `hoists display math out of surrounding prose`() {
val document = parse("""Before \[x^2\] after""")
val math = document.collect<MathBlockNode>()
assertEquals(listOf("x^2"), math.map { it.latex })
val paragraphs = document.collect<Paragraph>().map { it.allText() }
assertEquals(listOf("Before ", " after"), paragraphs)
// The hoisted math must sit between the two prose paragraphs.
val blocks = document.topBlocks()
assertEquals(3, blocks.size)
assertTrue(
blocks[1] is MathBlockNode,
"expected math between the paragraphs, got ${blocks.map { it.javaClass.simpleName }}",
)
assertNoPlaceholderTokens(document)
}
@Test
fun `preserves common LaTeX commands inside display math`() {
val document = parse(
"""
\[
\lim_{x\to 0}\frac{\sin x}{x}=1
\]
""".trimIndent(),
)
assertEquals(
listOf("\\lim_{x\\to 0}\\frac{\\sin x}{x}=1"),
document.collect<MathBlockNode>().map { it.latex },
)
}
@Test
fun `renders the multiline transfer-function formulas from the real session`() {
val document = parse(
"""
\[
\Phi(s)=
\frac{\dfrac{K^*(s+4)}{s(s+2)(s+3)}}
{1+\dfrac{K^*}{s(s+3)}}
\]
\[
\Phi(s)
=
\frac{K^*(s+4)}
{(s+2)\left[s(s+3)+K^*\right]}
\]
\[
\Phi(s)=
\frac{9(s+4)}
{(s+2)(s^2+3s+9)}
\]
\[
\Phi(s)
=
\frac{9}{s^2+3s+9}
\frac{s+4}{s+2}
\]
""".trimIndent(),
)
val math = document.collect<MathBlockNode>()
assertEquals(4, math.size)
assertTrue(math[0].latex.contains("\\dfrac"), "first formula lost its nested fraction")
assertTrue(math[1].latex.contains("\\left[s(s+3)+K^*\\right]"))
assertNoPlaceholderTokens(document)
}
@Test
fun `renders percent and text commands from the real session`() {
val document = parse(
"""
标准二阶系统超调量:
\[
\sigma\% =
e^{-\frac{\pi\zeta}{\sqrt{1-\zeta^2}}}\times100\%
\]
\[
t_s\approx\frac{4}{0.5\times3}
\approx2.67\text{ s}
\]
""".trimIndent(),
)
val math = document.collect<MathBlockNode>()
assertEquals(2, math.size)
assertTrue(math[1].latex.contains("\\text{ s}"))
assertNoPlaceholderTokens(document)
}
@Test
fun `recurses through inline formatting while keeping the display node block-level`() {
val document = parse("""**Before \(x^2\) after**""")
val emphasis = document.collect<StrongEmphasis>()
assertEquals(1, emphasis.size)
assertEquals(listOf("x^2"), emphasis.single().collect<MathInlineNode>().map { it.latex })
assertEquals(0, document.collect<MathBlockNode>().size)
assertNoPlaceholderTokens(document)
}
// ── Literal passthrough ──────────────────────────────────────────────────
@Test
fun `leaves inline code as text`() {
val document = parse("""Use `\(x^2\)` here.""")
assertEquals(0, document.collect<MathInlineNode>().size + document.collect<MathBlockNode>().size)
assertEquals(listOf("\\(x^2\\)"), document.collect<Code>().map { it.literal })
}
@Test
fun `leaves fenced code as text`() {
val document = parse("```latex\n\\[x^2\\]\n```")
assertEquals(0, document.collect<MathInlineNode>().size + document.collect<MathBlockNode>().size)
assertEquals(listOf("\\[x^2\\]\n"), document.collect<FencedCodeBlock>().map { it.literal })
}
@Test
fun `leaves escaped delimiters as text`() {
val document = parse("""Literal \\(x^2\\) and \\[y^2\\]""")
assertEquals(0, document.collect<MathInlineNode>().size + document.collect<MathBlockNode>().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `leaves an unclosed delimiter as text`() {
val document = parse("""Literal \(x^2""")
assertEquals(0, document.collect<MathInlineNode>().size + document.collect<MathBlockNode>().size)
}
@Test
fun `does not let an unmatched opener consume a protected code block`() {
val document = parse(
listOf(
"""Literal \(""",
"```latex",
"""\[x^2\]""",
"```",
"",
"""\(y^2\)""",
).joinToString("\n"),
)
assertEquals(listOf("\\[x^2\\]\n"), document.collect<FencedCodeBlock>().map { it.literal })
assertEquals(listOf("y^2"), document.collect<MathInlineNode>().map { it.latex })
assertTrue(document.allText().contains("Literal ("), "the lone opener must stay literal text")
assertNoPlaceholderTokens(document)
}
@Test
fun `preserves quoted fences and multiline code spans as literal Markdown`() {
val quotedFence = parse(
listOf(
"> ```latex",
"> \\[x^2\\]",
"> ````",
).joinToString("\n"),
)
assertEquals(0, quotedFence.collect<MathInlineNode>().size + quotedFence.collect<MathBlockNode>().size)
assertEquals(listOf("\\[x^2\\]\n"), quotedFence.collect<FencedCodeBlock>().map { it.literal })
val multilineCodeSpan = parse(
listOf(
"Use `before",
"""\[x^2\]""",
"after` here.",
).joinToString("\n"),
)
assertEquals(0, multilineCodeSpan.collect<MathInlineNode>().size + multilineCodeSpan.collect<MathBlockNode>().size)
// CommonMark converts newlines inside a code span to spaces; the
// delimiters just have to stay literal.
assertEquals(listOf("before \\[x^2\\] after"), multilineCodeSpan.collect<Code>().map { it.literal })
}
// ── Container prefixes ───────────────────────────────────────────────────
@Test
fun `removes blockquote prefixes from multiline bracket math`() {
val document = parse(
listOf(
"> \\[",
"> x = 1",
"> \\]",
).joinToString("\n"),
)
val math = document.collect<MathBlockNode>()
assertEquals(listOf("x = 1"), math.map { it.latex })
assertEquals(1, document.collect<BlockQuote>().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `removes blockquote prefixes from dollar flow math`() {
val document = parse(
listOf(
"> $$",
"> E = mc^2",
"> F = ma",
"> $$",
).joinToString("\n"),
)
val math = document.collect<MathBlockNode>()
assertEquals(listOf("E = mc^2\nF = ma"), math.map { it.latex })
assertEquals(1, document.collect<BlockQuote>().size)
assertNoPlaceholderTokens(document)
val singleLine = parse(
listOf(
"> $$",
"> E",
"> $$",
).joinToString("\n"),
)
assertEquals(listOf("E"), singleLine.collect<MathBlockNode>().map { it.latex })
assertNoPlaceholderTokens(singleLine)
}
@Test
fun `preserves blockquote prefixes when prose precedes multiline bracket math`() {
val document = parse(
listOf(
"> Result: \\[",
"> x = 1",
"> \\]",
).joinToString("\n"),
)
val math = document.collect<MathBlockNode>()
assertEquals(listOf("x = 1"), math.map { it.latex })
val quotes = document.collect<BlockQuote>()
assertEquals(1, quotes.size)
assertEquals(listOf("Result: "), quotes.single().collect<Paragraph>().map { it.allText() })
assertNoPlaceholderTokens(document)
}
@Test
fun `keeps mathematical greater-than signs in unquoted multiline bracket math`() {
val document = parse(
listOf(
"\\[",
"x",
"> 0",
"\\]",
).joinToString("\n"),
)
assertEquals(listOf("x\n> 0"), document.collect<MathBlockNode>().map { it.latex })
assertNoPlaceholderTokens(document)
}
// ── Lists and fences ─────────────────────────────────────────────────────
@Test
fun `preserves list-contained fences and renders following bracket math`() {
val document = parse(
listOf(
"- ```latex",
" \\[x^2\\]",
" ````",
"",
"""\(y^2\)""",
).joinToString("\n"),
)
assertEquals(listOf("\\[x^2\\]\n"), document.collect<FencedCodeBlock>().map { it.literal })
assertEquals(listOf("y^2"), document.collect<MathInlineNode>().map { it.latex })
assertNoPlaceholderTokens(document)
}
@Test
fun `recognizes ordered-list fence indentation before following bracket math`() {
val document = parse(
listOf(
"10. ```latex",
" \\[x^2\\]",
" ````",
"",
"""\(y^2\)""",
).joinToString("\n"),
)
assertEquals(listOf("\\[x^2\\]\n"), document.collect<FencedCodeBlock>().map { it.literal })
assertEquals(listOf("y^2"), document.collect<MathInlineNode>().map { it.latex })
}
@Test
fun `allows a list-relative closing fence indentation`() {
val document = parse(
listOf(
"- ```latex",
" \\[x^2\\]",
" ````",
"",
"""\(y^2\)""",
).joinToString("\n"),
)
assertEquals(listOf("\\[x^2\\]\n"), document.collect<FencedCodeBlock>().map { it.literal })
assertEquals(listOf("y^2"), document.collect<MathInlineNode>().map { it.latex })
}
@Test
fun `keeps multiline bracket math inside a list item`() {
val document = parse(
listOf(
"- \\[",
" x = 1",
" \\]",
).joinToString("\n"),
)
val math = document.collect<MathBlockNode>()
assertEquals(listOf("x = 1"), math.map { it.latex })
val items = document.collect<ListItem>()
assertEquals(1, items.size)
assertTrue(items.single().collect<MathBlockNode>().isNotEmpty(), "math must stay inside the list item")
assertNoPlaceholderTokens(document)
}
@Test
fun `normalizes nested list and blockquote prefixes in multiline bracket math`() {
val document = parse(
listOf(
"- > \\[",
" > x = 1",
" > \\]",
).joinToString("\n"),
)
val math = document.collect<MathBlockNode>()
assertEquals(listOf("x = 1"), math.map { it.latex })
assertEquals(1, document.collect<BlockQuote>().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `renders bracket math across an empty TeX line`() {
val document = parse(
listOf(
"\\[",
"a",
"",
"+ b",
"\\]",
).joinToString("\n"),
)
assertEquals(listOf("a\n\n+ b"), document.collect<MathBlockNode>().map { it.latex })
assertNoPlaceholderTokens(document)
}
// ── Link / image / definition metadata ───────────────────────────────────
@Test
fun `keeps escaped parentheses in link, image, and definition destinations`() {
val link = parse("""[source](https://example.com/foo\(bar\).ts)""")
assertEquals(listOf("https://example.com/foo(bar).ts"), link.collect<Link>().map { it.destination })
assertEquals(0, link.collect<MathInlineNode>().size + link.collect<MathBlockNode>().size)
val image = parse("""![source](https://example.com/foo\(bar\).png)""")
assertEquals(listOf("https://example.com/foo(bar).png"), image.collect<Image>().map { it.destination })
assertEquals(0, image.collect<MathInlineNode>().size + image.collect<MathBlockNode>().size)
val definition = parse("[source][id]\n\n[id]: https://example.com/foo\\(bar\\).ts")
val resolved = definition.collect<Link>().map { it.destination }
assertEquals(listOf("https://example.com/foo(bar).ts"), resolved)
assertEquals(0, definition.collect<MathInlineNode>().size + definition.collect<MathBlockNode>().size)
assertNoPlaceholderTokens(definition)
}
@Test
fun `preserves image alt text and reference identifiers containing delimiters`() {
val image = parse("""![\(x\)](https://example.com/image.png)""")
assertEquals(1, image.collect<Image>().size)
assertEquals(0, image.collect<MathInlineNode>().size + image.collect<MathBlockNode>().size)
assertNoPlaceholderTokens(image)
val reference = parse("[source][\\(id\\)]\n\n[\\(id\\)]: https://example.com/foo")
assertEquals(1, reference.collect<Link>().size)
assertEquals(0, reference.collect<MathInlineNode>().size + reference.collect<MathBlockNode>().size)
assertNoPlaceholderTokens(reference)
}
@Test
fun `preserves reference-image alt text and implicit link identifiers`() {
val imageReference = parse("![\\(x\\)][id]\n\n[id]: https://example.com/image.png")
assertEquals(1, imageReference.collect<Image>().size)
assertEquals(0, imageReference.collect<MathInlineNode>().size + imageReference.collect<MathBlockNode>().size)
assertNoPlaceholderTokens(imageReference)
val collapsed = parse("[\\(x\\)][]\n\n[\\(x\\)]: https://example.com")
assertEquals(1, collapsed.collect<Link>().size)
assertNoPlaceholderTokens(collapsed)
val shortcut = parse("[\\(x\\)]\n\n[\\(x\\)]: https://example.com")
assertEquals(1, shortcut.collect<Link>().size)
assertNoPlaceholderTokens(shortcut)
}
// ── Dollar math (remark-math `singleDollarTextMath: false`) ──────────────
@Test
fun `does not render single-dollar currency amounts as math`() {
val document = parse("The plan is \$200/mo and the bill is \$80 — total \$400 saved.")
assertEquals(0, document.collect<MathInlineNode>().size + document.collect<MathBlockNode>().size)
assertTrue(document.allText().contains("\$200"))
assertTrue(document.allText().contains("\$80"))
assertTrue(document.allText().contains("\$400"))
}
@Test
fun `does not render the reported real-world prose as math`() {
val document = parse("Cursor's UI quotes the ratio: at least \$400 of API usage on a \$200 plan. That's 2:1.")
assertEquals(0, document.collect<MathInlineNode>().size + document.collect<MathBlockNode>().size)
assertTrue(document.allText().contains("\$400"))
assertTrue(document.allText().contains("\$200"))
}
@Test
fun `still renders block math with dollar-dollar on its own lines`() {
val document = parse("Before\n\n\$\$\nE = mc^2\n\$\$\n\nAfter")
assertEquals(listOf("E = mc^2"), document.collect<MathBlockNode>().map { it.latex })
assertEquals(listOf("Before", "After"), document.collect<Paragraph>().map { it.allText() })
assertNoPlaceholderTokens(document)
}
@Test
fun `renders mid-paragraph dollar-dollar as inline math`() {
val document = parse("Euler says \$\$e^{i\\pi} + 1 = 0\$\$ exactly.")
assertEquals(listOf("e^{i\\pi} + 1 = 0"), document.collect<MathInlineNode>().map { it.latex })
assertEquals(0, document.collect<MathBlockNode>().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `requires a matching dollar run size for inline math`() {
// A `$$$` opener needs an equal three-dollar closer; without one the
// whole run stays literal (the inner `$$` is part of the opener).
val document = parse("a \$\$\$x\$\$ b")
assertEquals(0, document.collect<MathInlineNode>().size + document.collect<MathBlockNode>().size)
}
@Test
fun `treats backslash-escaped dollars as literal`() {
val document = parse("He said " + "\\$" + "\\$" + "x" + "\\$" + "\\$" + " loudly.")
assertEquals(0, document.collect<MathInlineNode>().size + document.collect<MathBlockNode>().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `does not close flow math through a fenced code block`() {
val document = parse(
listOf(
"\$\$",
"```",
"not a closer \$\$",
"```",
"\$\$",
).joinToString("\n"),
)
// The `$$` inside the fence is protected, so the real closer wins and
// the fence lines become math content (math flow, like code fences,
// is not interrupted by ```).
assertEquals(
listOf("```\nnot a closer \$\$\n```"),
document.collect<MathBlockNode>().map { it.latex },
)
assertEquals(0, document.collect<FencedCodeBlock>().size)
}
@Test
fun `does not close quoted flow math from a later top-level opener`() {
// Web parity (remark-math): math flow is concrete and ends with its
// container, so the quoted `> $$` opener must not pair with the
// top-level `$$` after the blockquote ended — ordinary prose stays
// prose and the later equation keeps its own opener.
val document = parse(
listOf(
"> \$\$",
"> x",
"",
"Ordinary prose",
"",
"\$\$",
"y",
"\$\$",
).joinToString("\n"),
)
assertEquals(listOf("y"), document.mathBlocks().map { it.latex })
val paragraphs = document.collect<Paragraph>().map { it.allText() }
assertTrue("Ordinary prose" in paragraphs, "prose after the quote stays a prose paragraph")
assertTrue(
paragraphs.any { it.startsWith("\$\$") && it.endsWith("x") },
"the unclosed quoted opener stays literal text (got: $paragraphs)",
)
assertEquals(1, document.collect<BlockQuote>().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `does not close list-contained flow math from a later top-level opener`() {
val document = parse(
listOf(
"- \$\$",
" x",
"",
"Ordinary prose",
"",
"\$\$",
"y",
"\$\$",
).joinToString("\n"),
)
assertEquals(listOf("y"), document.mathBlocks().map { it.latex })
val paragraphs = document.collect<Paragraph>().map { it.allText() }
assertTrue("Ordinary prose" in paragraphs, "prose after the list stays a prose paragraph")
assertTrue(
paragraphs.any { it.startsWith("\$\$") && it.endsWith("x") },
"the unclosed list opener stays literal text (got: $paragraphs)",
)
assertNoPlaceholderTokens(document)
}
@Test
fun `does not close quoted flow math across a blank line into a new quote`() {
// A blank line closes the blockquote; the later `> $$` re-opens a NEW
// quote and is not a closer for the first one (remark-math renders
// two separate blocks here).
val document = parse(
listOf(
"> \$\$",
"> x",
"",
"> \$\$",
).joinToString("\n"),
)
assertEquals(0, document.mathBlocks().size)
assertEquals(0, document.mathInlines().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `closes list-contained flow math on a prefixed closer line`() {
val document = parse(
listOf(
"- \$\$",
" E = mc^2",
" \$\$",
).joinToString("\n"),
)
// Bounded matching keeps in-container closers working.
assertEquals(listOf("E = mc^2"), document.mathBlocks().map { it.latex })
val items = document.collect<ListItem>()
assertEquals(1, items.size)
assertEquals(1, items.single().collect<MathBlockNode>().size)
assertNoPlaceholderTokens(document)
}
@Test
fun `closes list-contained flow math across a blank line in a loose item`() {
val document = parse(
listOf(
"- \$\$",
" x",
"",
" \$\$",
).joinToString("\n"),
)
// A list item may resume after a blank line (loose list), so the
// indented closer still pairs — only blockquotes die at blank lines.
assertEquals(listOf("x"), document.mathBlocks().map { it.latex })
assertNoPlaceholderTokens(document)
}
@Test
fun `preserves bracket-like TeX commands inside existing dollar math`() {
val document = parse("\$\$\\verb|\\(x\\)|\$\$")
assertEquals(listOf("""\verb|\(x\)|"""), document.collect<MathInlineNode>().map { it.latex })
assertNoPlaceholderTokens(document)
}
// ── Interaction with table repair ────────────────────────────────────────
@Test
fun `keeps bracket math after the existing table source repair`() {
val document = parse(
listOf(
"| A | B | C |",
"|---|---|",
"| x | y | z |",
"",
"""\[E = mc^2\]""",
).joinToString("\n"),
)
assertEquals(1, document.collect<TableBlock>().size)
assertEquals(listOf("E = mc^2"), document.collect<MathBlockNode>().map { it.latex })
assertNoPlaceholderTokens(document)
}
}