Currently when an inintial index takes a long time, navigation will
basically be blocked until it's finished which isn't great. This
special-cases that scenario and doesn't block (at the cost of possibly
not showing up to date widgets on the target page).
No longer cache widget HTML content just widget heights, either way the
widgets will briefly flash as the widget is recalculated and this saves
storage space and reduces issues in Webkit
* [LIQ] Fix: Coerce values to plain numbers in numeric aggregate functions
All numeric aggregate iterate functions (`sum`, `product`, `avg`,
`stddev`, variance, covariance, correlation, bitwise, quantile, ...)
used TypeScript as number type assertions which have no effect at
runtime. When a value was a string (e.g., YAML `rs: '3000.0'`) or
a `LuaTaggedFloat` object JS arithmetic operators like `+=` would
silently perform string concatenation or produce `[object Object]`
nonsense instead of numeric addition.
The fix introduces a small helper wrapping the `coerceToNumber` that
correctly unwraps plain numbers, `LuaTaggedFloat` objects and numeric
strings to plain JS numbers, and applies it at every numeric aggregate
intake point.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* [LIQ] Fix: Unwrap `LuaTaggedFloat` before passing values iterate function
`executeAggregate` now unwraps `LuaTaggedFloat` boxing before passing
values to any iterate function ensuring user-defined aggregates also
receive plain JS numbers instead of opaque internal objects.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Remove forgotten trivial wrapper `aggNum`
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Now produce "markdown" (or <table> tags for tables) instead of pure
HTML. Hopefully doesn't introduce regressions. Added a test suite for
future iterations.
* Fine tunes a few behaviors (like not adding extra newlines, ever
(should fix#1629))
* When a list item ends with :, it automaticaly indents the next item
* Adds an extensive test suite with various cases
This needs some more testing, but should be a good starting point for
adding more tweaks later.
* tag.define now supports a `tagPage` attribute, allowing what page to
send the user to when clicking on a hash tag in the editor
* index.contentPages() and index.metaPages() are convenience wrappers
around the index.tag("page") collection that filter on content and
meta pages respectively.
Even though the service worker will proactively cache them in the
browser, the browser will not have to load them, parse them, interpret
them unless these languages are actually used. Less JS to load, less
waiting.
* Website: Make manual build ref configurable
* [LIQ] Unify `from`, `select`, and `group by` clauses to use field lists
Query clauses `from`, `select`, and `group by` now accept the same field
syntax as Lua table constructors (`name = expr`, bare expr, or `[expr]
= expr`), giving them a consistent grammar and enabling named bindings
everywhere. #feature
What changed:
* New `FieldList` rule in `lua.grammar`. All three clauses now parse
their arguments through it instead of using expression lists or the
special `Name "=" exp` pattern.
* AST types `LuaFromClause`, `LuaSelectClause`, and `LuaGroupByClause`
now carry fields: `LuaTableField[]` instead of a single expression or
an expression array. The parser, static analysis helpers and the
evaluator are all updated to work with this new shape.
Named bindings in `select`
--------------------------
`select name, age` now derives column names from bare variable or
property access expressions so the result table has string keys (`name`,
`age`) rather than integer indices. Explicit naming still works.
Named bindings in `group by`
----------------------------
`group b`y fields can carry explicit aliases that propagate into the
post-grouping scope. For example:
```sql
group by
n = name
select {
label = n
}
```
The alias `n` is now visible in `select`, `having`, and `order by`.
Previously only the raw expression form (`name`) was bound and an
explicit alias was silently lost. #bugfix
A new `LuaGroupByEntry` type pairs each `group by` expression with its
optional alias. The environment builder binds every
declared alias to the corresponding key value, regardless of whether the
group key is a scalar or a multi-key table.
Named bindings in `from`
------------------------
`from p = page` continues to work as before. The clause now also accepts
the full field-list syntax which lays the groundwork for multi-source
from (cross join) in a future change. #todo #feature
Complete LIQ syntax now
-----------------------
Complete LIQ syntax (in Postgres documentation style) could be now
roughly expressed as follows: #todo #documentation
```text
query[[ clause [...] ]]
where clause is one of:
FROM field [, ...]
WHERE condition
GROUP BY field [, ...]
HAVING condition
SELECT field [, ...]
ORDER BY sort_key [, ...]
LIMIT count [, offset]
OFFSET start
where field is one of:
expression
name = expression
[expression] = expression
where sort_key is:
expression [ ASC | DESC ]
[ NULLS { FIRST | LAST } ]
[ USING name ]
[ USING FUNCTION function_body ]
where expression is any valid Lua expression, including:
aggregate_call
aggregate_call FILTER ( WHERE condition )
```
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix type error caused by earlier merge conflict resolution
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* [LIQ] Cross-join support for multi-source `from`
Examples of what works now:
```
${query [[
from
x = { 1, 2 },
y = { 10, 20 },
z = { 100, 200 }
select {
sum = x + y + z
}
]]}
```
Or (more complicated three-way cross-join) to illustrate:
```
${query [[
from
s = {
{ id = 1, name = 'Eva', },
{ id = 2, name = 'Adam', },
{ id = 3, name = 'John', },
{ id = 4, name = 'Zef', },
},
c = {
{ id = 101, title = 'Mathematics', },
{ id = 102, title = 'Arts', },
{ id = 103, title = 'Physics', },
{ id = 104, title = 'Computer Science', },
{ id = 105, title = 'Literature', },
},
e = {
{ sid = 1, cid = 101, },
{ sid = 1, cid = 103, },
{ sid = 2, cid = 101, },
{ sid = 2, cid = 102, },
{ sid = 2, cid = 103, },
{ sid = 3, cid = 102, },
{ sid = 3, cid = 105, },
{ sid = 4, cid = 101, },
{ sid = 4, cid = 103, },
{ sid = 4, cid = 104, },
}
where
s.id == e.sid and
e.cid == c.id
order by
s.name
group by
s.name
having
s.name:match('^A') or
s.name:match('f$')
select
student = s.name:upper(),
courses = string_agg(c.title, ', '
order by c.title desc
)
]]}
```
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Co-authored-by: Zef Hemel <zef@zef.me>
* Allow aggregate functions in `select`/`having` without an explicit
`group by` clause. The entire result set is treated as one group.
Note: Uses a `Symbol` as the map key for the implicit single group to
avoid any collision with string keys from `generateKey`.
* Extend documentation.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix (tasks): Toggle dropdown on re-click and narrow mark decoration range
- Clicking the dropdown icon now dismisses the completion menu if it
is already open (`closeCompletion`).
- The `data-task-state` mark decoration now covers only the inner state
text (excluding brackets). This prevents CodeMirror cursor/editing
issues when the caret is inside the brackets and allows bracket-aware
CSS styling. Example:
```space-style
[data-task-state="DONE"] { color: green; }
[data-task-state="PLANNED"] { color: orange; }
[data-task-state] { font-weight: bold; }
```
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Refactor (tasks): Unify task state data attribute to `data-task-state`
Rename `data-state` to `data-task-state` on extended task state spans in
Markdown renderer and update the corresponding places.
Previously the editor (CodeMirror mark decoration) used
`data-task-state` while the widget HTML renderer used `data-state`,
requiring duplicate CSS selectors. Now a single attribute selector works
everywhere (for example tasks rendered using `template.taskItem` also
follow the styling now):
```css
[data-task-state="PLANNED"] { /* cool CSS here */; }
```
Note: Checkbox inputs (`[ ]`/`[x]`/`[X]`) still use `data-state` since
they follow a different code path.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Add a dropdown picker for custom task states
Add a dropdown picker for custom task states that appears as a small
triangle next to the `[state]` allowing to quickly switch between
defined states with autocomplete.
States marked as done receive strikethrough styling on the task text.
Each state can be styled individually using the `data-task-state`
attribute, for example:
```space-style
[data-task-state=PLANNED] .sb-task-state {
color: black;
background-color: orange;
}
```
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Keep the current task state visible while the dropdown picker is open
Keep the current task state visible while the dropdown picker is open by
disabling built-in filtering and handling it manually — when the state
text exactly matches a known state all options are shown, otherwise
results are filtered by what the user has typed so far.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Added on-hover completion styling
* Fix: Show all defined task states in the dropdown when clicking the picker on an unrecognized state instead of showing an empty menu
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Co-authored-by: Zef Hemel <zef@zef.me>
* The interpreter wasted cycles on unnecessary async/Promise wrapping
and redundant allocations even for synchronous operations.
* Now: more "sync-first" evaluation — try the fast synchronous path
first, only fall back to Promises when actually needed.
* In addition: fast paths for table access, variable lookups, string
concat, arithmetic, and function calls; loops can re-enter sync mode
after an async iteration resolves.
* Added benchmarks
Results: 1.3x–1.7x speedup across typical workloads (API calls, table
traversal, string ops), up to 1.92x in some cases.
* [LIQ] Add more tests for builtin aggregates
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* [LIQ] Add `first`, `last`, `mode` and `median` aggregates and `group_concat` alias
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry
* Extend with 13 new built-in aggregates: `product`, `string_agg`,
`yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`,
`bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`.
* Introduce `aggregate.alias` API allowing users to define custom
aliases for any aggregate. Standard aliases (`every`, `std`, `stddev`
and `variance`) are now defined via this API rather than hardcoded.
* Add `index.aggregates` queryable collection so users can discover
all available aggregates directly from LIQ queries.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix config pass through query path so custom aggregates work
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage
`config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on
all arguments. This converted the aggregate `LuaTable` to a plain JS
object and wrapped `LuaFunction` callbacks in JS functions that also
converted their returned values via `luaValueToJS`. The result was that
state returned by initialize (a `LuaTable`) got converted to a plain JS
object before being passed to `iterate`. Therefor Lua operations like
`table.insert` on that were failing because they expected a `LuaTable`
and not a plain JS array.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix formatting
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Improve aggregate functions descriptions, fix `sum` divergence
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Align `product` with `sum`
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Rename `alias` in `aggregates()` to `target` for clarity
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add a null guard at the top of `jsToLuaValue`
This preserves `null`/`undefined` as-is (both map to Lua nil) and
prevents them from falling through to the `typeof` "object" branch.
For this PR it means that null `target` in our `aggregates` entries will
correctly show as empty/`nil` in query results rather than `{}`.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Documentation reflects recent changes
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks
* `sum(`) and` product(`) now return null when no rows match (matching
Postgres semantics) instead of returning 0 and 1 respectively.
* Query result columns that hold null are internally preserved using
a `LIQ_NULL` sentinel so that column keys survive in `LuaTable`
storage. This sentinel was leaking into Lua code as "userdata"
through three read paths:
* `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when
accessing table fields,
* `rawget` (stdlib): the builtin `rawget` function exposed the
sentinel without converting it back to `nil`,
* `createAugmentedEnv`: string interpolation unpacked table values via
`rawGet` into local variables, making the sentinel visible in
template expressions like `${var}`.
All three now convert `LIQ_NULL` to `nil` at the read boundary,
keeping the sentinel internal to table storage where it belongs.
* Update affected test expectations accordingly.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: `array_agg` preserves NULL positions
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add symbol guard to `json_agg`
`JSON.stringify(Symbol(...))` in an array produces null by accident.
That is a JS implementation detail we **MUST NOT** rely on. Explicit
null push makes intent clear and avoids surprises if the `Symbol`
representation ever changes.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add symbol guard to `yaml_agg` (ditto)
`js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes
it emit YAML null (or `~`), which is the correct YAML representation of
a missing value and matches standard `json_agg`/`yaml_agg`
NULL-inclusion semantics.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add intra-aggregate ordering null guards
Without this, `LIQ_NULL` sort keys would fall through to `valA < valB`
which is always false for `Symbol`s which is breaking the `nulls
first`/`nulls last` contract...
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Ditto, but for `order by` null comparisons
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates
Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated
against the outer query environment where the object variable is not
bound. This caused multi-argument aggregates like `covar_samp(data.y,
data.x)` to fail with nil reference errors. This commit addresses this
by evaluating extra args per-item inside the iterate loop using the item
environment so all arguments resolve correctly.
We also add few common aggregates:
- `covar_pop`, `covar_samp`, `corr`: population/sample covariance and
correlation coefficient using online co-moment algorithm.
- `quantile(value, q, method)`: general quantile with interpolation
methods: lower, higher, nearest, midpoint and default linear.
- `percentile_cont(value, q)`: continuous percentile (linear)
- `percentile_disc(value, q)`: discrete percentile (lower)
Note: `percentile_cont` and `percentile_disc` share the `quantile`
implementation through `ctx.name` at initialize time.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Update docs
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Make the ordering for quantile aggregates explicit
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Update docs
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Improve docs
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Zef Hemel