* [LIQ] Fix: Coerce values to plain numbers in numeric aggregate functions
All numeric aggregate iterate functions (`sum`, `product`, `avg`,
`stddev`, variance, covariance, correlation, bitwise, quantile, ...)
used TypeScript as number type assertions which have no effect at
runtime. When a value was a string (e.g., YAML `rs: '3000.0'`) or
a `LuaTaggedFloat` object JS arithmetic operators like `+=` would
silently perform string concatenation or produce `[object Object]`
nonsense instead of numeric addition.
The fix introduces a small helper wrapping the `coerceToNumber` that
correctly unwraps plain numbers, `LuaTaggedFloat` objects and numeric
strings to plain JS numbers, and applies it at every numeric aggregate
intake point.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* [LIQ] Fix: Unwrap `LuaTaggedFloat` before passing values iterate function
`executeAggregate` now unwraps `LuaTaggedFloat` boxing before passing
values to any iterate function ensuring user-defined aggregates also
receive plain JS numbers instead of opaque internal objects.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Remove forgotten trivial wrapper `aggNum`
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Now produce "markdown" (or <table> tags for tables) instead of pure
HTML. Hopefully doesn't introduce regressions. Added a test suite for
future iterations.
* Website: Make manual build ref configurable
* [LIQ] Unify `from`, `select`, and `group by` clauses to use field lists
Query clauses `from`, `select`, and `group by` now accept the same field
syntax as Lua table constructors (`name = expr`, bare expr, or `[expr]
= expr`), giving them a consistent grammar and enabling named bindings
everywhere. #feature
What changed:
* New `FieldList` rule in `lua.grammar`. All three clauses now parse
their arguments through it instead of using expression lists or the
special `Name "=" exp` pattern.
* AST types `LuaFromClause`, `LuaSelectClause`, and `LuaGroupByClause`
now carry fields: `LuaTableField[]` instead of a single expression or
an expression array. The parser, static analysis helpers and the
evaluator are all updated to work with this new shape.
Named bindings in `select`
--------------------------
`select name, age` now derives column names from bare variable or
property access expressions so the result table has string keys (`name`,
`age`) rather than integer indices. Explicit naming still works.
Named bindings in `group by`
----------------------------
`group b`y fields can carry explicit aliases that propagate into the
post-grouping scope. For example:
```sql
group by
n = name
select {
label = n
}
```
The alias `n` is now visible in `select`, `having`, and `order by`.
Previously only the raw expression form (`name`) was bound and an
explicit alias was silently lost. #bugfix
A new `LuaGroupByEntry` type pairs each `group by` expression with its
optional alias. The environment builder binds every
declared alias to the corresponding key value, regardless of whether the
group key is a scalar or a multi-key table.
Named bindings in `from`
------------------------
`from p = page` continues to work as before. The clause now also accepts
the full field-list syntax which lays the groundwork for multi-source
from (cross join) in a future change. #todo #feature
Complete LIQ syntax now
-----------------------
Complete LIQ syntax (in Postgres documentation style) could be now
roughly expressed as follows: #todo #documentation
```text
query[[ clause [...] ]]
where clause is one of:
FROM field [, ...]
WHERE condition
GROUP BY field [, ...]
HAVING condition
SELECT field [, ...]
ORDER BY sort_key [, ...]
LIMIT count [, offset]
OFFSET start
where field is one of:
expression
name = expression
[expression] = expression
where sort_key is:
expression [ ASC | DESC ]
[ NULLS { FIRST | LAST } ]
[ USING name ]
[ USING FUNCTION function_body ]
where expression is any valid Lua expression, including:
aggregate_call
aggregate_call FILTER ( WHERE condition )
```
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix type error caused by earlier merge conflict resolution
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* [LIQ] Cross-join support for multi-source `from`
Examples of what works now:
```
${query [[
from
x = { 1, 2 },
y = { 10, 20 },
z = { 100, 200 }
select {
sum = x + y + z
}
]]}
```
Or (more complicated three-way cross-join) to illustrate:
```
${query [[
from
s = {
{ id = 1, name = 'Eva', },
{ id = 2, name = 'Adam', },
{ id = 3, name = 'John', },
{ id = 4, name = 'Zef', },
},
c = {
{ id = 101, title = 'Mathematics', },
{ id = 102, title = 'Arts', },
{ id = 103, title = 'Physics', },
{ id = 104, title = 'Computer Science', },
{ id = 105, title = 'Literature', },
},
e = {
{ sid = 1, cid = 101, },
{ sid = 1, cid = 103, },
{ sid = 2, cid = 101, },
{ sid = 2, cid = 102, },
{ sid = 2, cid = 103, },
{ sid = 3, cid = 102, },
{ sid = 3, cid = 105, },
{ sid = 4, cid = 101, },
{ sid = 4, cid = 103, },
{ sid = 4, cid = 104, },
}
where
s.id == e.sid and
e.cid == c.id
order by
s.name
group by
s.name
having
s.name:match('^A') or
s.name:match('f$')
select
student = s.name:upper(),
courses = string_agg(c.title, ', '
order by c.title desc
)
]]}
```
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Co-authored-by: Zef Hemel <zef@zef.me>
* Allow aggregate functions in `select`/`having` without an explicit
`group by` clause. The entire result set is treated as one group.
Note: Uses a `Symbol` as the map key for the implicit single group to
avoid any collision with string keys from `generateKey`.
* Extend documentation.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* The interpreter wasted cycles on unnecessary async/Promise wrapping
and redundant allocations even for synchronous operations.
* Now: more "sync-first" evaluation — try the fast synchronous path
first, only fall back to Promises when actually needed.
* In addition: fast paths for table access, variable lookups, string
concat, arithmetic, and function calls; loops can re-enter sync mode
after an async iteration resolves.
* Added benchmarks
Results: 1.3x–1.7x speedup across typical workloads (API calls, table
traversal, string ops), up to 1.92x in some cases.
* [LIQ] Add more tests for builtin aggregates
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* [LIQ] Add `first`, `last`, `mode` and `median` aggregates and `group_concat` alias
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry
* Extend with 13 new built-in aggregates: `product`, `string_agg`,
`yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`,
`bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`.
* Introduce `aggregate.alias` API allowing users to define custom
aliases for any aggregate. Standard aliases (`every`, `std`, `stddev`
and `variance`) are now defined via this API rather than hardcoded.
* Add `index.aggregates` queryable collection so users can discover
all available aggregates directly from LIQ queries.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix config pass through query path so custom aggregates work
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage
`config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on
all arguments. This converted the aggregate `LuaTable` to a plain JS
object and wrapped `LuaFunction` callbacks in JS functions that also
converted their returned values via `luaValueToJS`. The result was that
state returned by initialize (a `LuaTable`) got converted to a plain JS
object before being passed to `iterate`. Therefor Lua operations like
`table.insert` on that were failing because they expected a `LuaTable`
and not a plain JS array.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix formatting
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Improve aggregate functions descriptions, fix `sum` divergence
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Align `product` with `sum`
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Rename `alias` in `aggregates()` to `target` for clarity
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add a null guard at the top of `jsToLuaValue`
This preserves `null`/`undefined` as-is (both map to Lua nil) and
prevents them from falling through to the `typeof` "object" branch.
For this PR it means that null `target` in our `aggregates` entries will
correctly show as empty/`nil` in query results rather than `{}`.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Documentation reflects recent changes
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks
* `sum(`) and` product(`) now return null when no rows match (matching
Postgres semantics) instead of returning 0 and 1 respectively.
* Query result columns that hold null are internally preserved using
a `LIQ_NULL` sentinel so that column keys survive in `LuaTable`
storage. This sentinel was leaking into Lua code as "userdata"
through three read paths:
* `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when
accessing table fields,
* `rawget` (stdlib): the builtin `rawget` function exposed the
sentinel without converting it back to `nil`,
* `createAugmentedEnv`: string interpolation unpacked table values via
`rawGet` into local variables, making the sentinel visible in
template expressions like `${var}`.
All three now convert `LIQ_NULL` to `nil` at the read boundary,
keeping the sentinel internal to table storage where it belongs.
* Update affected test expectations accordingly.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: `array_agg` preserves NULL positions
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add symbol guard to `json_agg`
`JSON.stringify(Symbol(...))` in an array produces null by accident.
That is a JS implementation detail we **MUST NOT** rely on. Explicit
null push makes intent clear and avoids surprises if the `Symbol`
representation ever changes.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add symbol guard to `yaml_agg` (ditto)
`js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes
it emit YAML null (or `~`), which is the correct YAML representation of
a missing value and matches standard `json_agg`/`yaml_agg`
NULL-inclusion semantics.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add intra-aggregate ordering null guards
Without this, `LIQ_NULL` sort keys would fall through to `valA < valB`
which is always false for `Symbol`s which is breaking the `nulls
first`/`nulls last` contract...
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Ditto, but for `order by` null comparisons
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates
Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated
against the outer query environment where the object variable is not
bound. This caused multi-argument aggregates like `covar_samp(data.y,
data.x)` to fail with nil reference errors. This commit addresses this
by evaluating extra args per-item inside the iterate loop using the item
environment so all arguments resolve correctly.
We also add few common aggregates:
- `covar_pop`, `covar_samp`, `corr`: population/sample covariance and
correlation coefficient using online co-moment algorithm.
- `quantile(value, q, method)`: general quantile with interpolation
methods: lower, higher, nearest, midpoint and default linear.
- `percentile_cont(value, q)`: continuous percentile (linear)
- `percentile_disc(value, q)`: discrete percentile (lower)
Note: `percentile_cont` and `percentile_disc` share the `quantile`
implementation through `ctx.name` at initialize time.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Update docs
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Make the ordering for quantile aggregates explicit
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Update docs
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Improve docs
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Add support for `order by ...` inside aggregate function calls, e.g.,
`array_agg(p.name order by p.name asc)` matching Postgres
per-aggregate ordering semantics.
* Add support for extra arguments for aggregates.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Tagged floats (`{value: N, isFloat: true}`) were matched by
`isPlainObject` and rendered as a two-column table with "value" and
"isFloat" headers instead of as a formatted number. This is now fixed.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Lua `${...}` expression results are rendered using two parallel paths:
- **HTML path** (display path) for visual display in the editor and
- **Markdown path** (copy path) for the *copy button* output.
Every Lua value is classified as one of:
- **Scalar:** number, string, boolean, or `nil`
- **Empty table:** `{}`
- **Array:** a `LuaTable` with only sequential integer keys (no string keys)
- **Array of tables:** an array where every element is a `LuaTable`
- **Record:** a `LuaTable` with any string key (may also have integer keys)
Scalars are rendered as inline Markdown - wiki links, tags, bold, and
other inline syntax are resolved, `nil` renders as `<span data-empty>`.
Empty tables render as `<table data-table-empty>`.
An array of tables renders as a multi-row `<table>`. Column headers
are the union of all element keys. Each cell value recurses through
the same classification.
An array of scalars (or a mix of scalars and tables) renders as `<ul>`
with one `<li>` per element. Each item value recurses.
A record renders as a single-row `<table>` with one column per key.
Cell values recurse.
Every `<td>` carries `data-table-cell-type` and optionally
`data-table-cell-empty`. Every `<li>` carries `data-list-item-type` and
optionally `data-list-item-empty`. The outermost element carries
`data-type`. These attributes allow CSS styling of rendered widgets.
Nesting is **unlimited**. Tables inside cells, lists inside cells,
tables inside list items all render correctly in HTML.
Scalars render as plain text. Lua `nil` renders as `nil`. Empty tables
render as `*(empty table)*`.
An array of tables renders as a pipe-delimited Markdown table. Nested
values in cells are rendered as Lua literals via `toStringAsync()`
(e.g. `{1, 2, 3}` or `{a = 1}`), since Markdown table cells cannot
contain block-level content.
An array of scalars renders as a Markdown list (`- item`). When a list
item is itself multi-line (a sublist) it uses an empty bullet with
two-space indented content on the next line:
```
- 1
- 2
-
- nested 1
- nested 2
```
Nested tables or arrays **beyond the first list nesting** level are
rendered as Lua literals.
A record renders as a single-row Markdown table.
Column headers follow the order returned by `LuaTable.keys`: string keys
in their insertion order, followed by integer keys in ascending order.
For example, `{a = 1, b = 2, 3, d = 4}` produces columns `a
| b | d | 1`. This is consequence of `LuaTable` storing string keys
and the array part separately (for performance).
Arrays mixing plain scalars and record-like tables (e.g. `{1, 2, 3,
{a=3}}`) were previously rendered as a single flat table due to an
internal JS conversion (`toJS`). They now render as a list, with record
elements shown as nested tables - which more accurately reflects their
Lua structure. This is the only divergence from the original
presentation.
To restore the old tabular layout, wrap all elements as records with
a consistent key:
```lua
-- Before (was a flat table, now a list with nested table):
${{1, 2, 3, {a=3}}}
-- After (explicit records, renders as a table):
${{{value=1}, {value=2}, {value=3}, {a=3}}}
```
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Items from `index.tag` are stored as plain JS objects when no metatable
is defined. The `group by` clause wraps rows in `LuaTable` (`key`,
`group`) but leaves the individual `group` items as plain JS objects.
This causes `t.keys is not a function` errors when passing `row.group`
to APIs like `template.each` that expect `LuaTable` instances.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Evaluate all `order by` expressions once per item in a decorate pass,
then compare cached keys during sort. Reduces Lua eval calls from
*O(kn log n)* to *O(kn)* where *k* is the number of sort keys.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
- Add iterative async stable merge sort (*O(n log n)*).
- Replace Quicksort with the new merge sort (kept Quicksort for tables).
- Update documentation (reorganize for logical flow, add SWO section
with invalid comparator example that really throws).
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* support for custom comparators in LIQ `order by` clauses via the
`using` keyword.
* `using` is mutually exclusive with `asc`/`desc` and accepts either
a *named function* or an *inline anonymous function*. The comparator
receives two values and returns true if the first should come before the
second.
* `nulls first`/ `nulls last` works independently of `using`.
* `using` is now a reserved keyword (`kw`; this cannot be avoided but
the chance to collide with user scripts is negligable),
* documentation updated + polished a little
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Implements `filter(where <cond>)` for aggregate functions in LIQ
queries which allows per-row conditional filtering within grouped
aggregates. Also extends test suite and documentation.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Extend the `order by` clause with optional `nulls first` and `nulls
last` modifiers following standard SQL semantics. Each sort key can
independently specify where `nil` values should appear in the result. By
default, ascending sorts place nulls last and descending sorts place
nulls first. An explicit nulls first or nulls last overrides this
default. The `asc` keyword is also now accepted as an explicit
alternative to the default ascending direction. Documentation and tests
were updated accordingly.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Check for already-queryable objects (e.g. `DataStoreQueryCollection`)
before `LuaTable` extraction or `luaValueToJS` conversion, preventing
them from being destroyed by the collection normalization pipeline.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
- Fix `group by` key building: populate array indices from
`keyParts` instead of only from `keyRecord`, which was empty for
some expressions (e.g. `tags[1]`).
- Fix collection handling: treat record-like `LuaTables` (no array part)
as items instead of extracting zero items.
- Update test suite.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix `LUA_SQL_NULL` leaking into `order by` comparisons
Note: SQL `NULLS LAST` is now default behaviour!
* Fix `objectVariable` unset for empty groups
* Fix non-grouped `having` used wrong environment builder
Note: Queries similar to the following one now works correctly:
```sql
${query[[
from
tags.page
having
name == 'index'
]]}
```
* Fix `generateKey` did not handle `LUA_SQL_NULL`
* Refactor/deduplicate duplicated `order by` comparator
* Add `isSqlNull` helper
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
1. Fix `order by` inability to access `select` output aliases and
aggregate functions after `group by`.
2. Introduce `LUA_SQL_NULL` sentinel to preserve column ordering when
`select` expressions evaluate to `nil`.
Note: In Lua, setting a table key to `nil` deletes it so rows with
`nil` values lose those keys and produced inconsistent insertion
order across result tables.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Zef Hemel