6cbb61c7db568cdb6f8e40ba589c821c14b6feff
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matouš Jan FialkaandGitHub
|
6cbb61c7db |
[LIQ] Add new aggregate functions, aliases, and queryable aggregate registry (#1891)
* [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry * Extend with 13 new built-in aggregates: `product`, `string_agg`, `yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`, `bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`. * Introduce `aggregate.alias` API allowing users to define custom aliases for any aggregate. Standard aliases (`every`, `std`, `stddev` and `variance`) are now defined via this API rather than hardcoded. * Add `index.aggregates` queryable collection so users can discover all available aggregates directly from LIQ queries. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix config pass through query path so custom aggregates work Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage `config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on all arguments. This converted the aggregate `LuaTable` to a plain JS object and wrapped `LuaFunction` callbacks in JS functions that also converted their returned values via `luaValueToJS`. The result was that state returned by initialize (a `LuaTable`) got converted to a plain JS object before being passed to `iterate`. Therefor Lua operations like `table.insert` on that were failing because they expected a `LuaTable` and not a plain JS array. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix formatting Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Improve aggregate functions descriptions, fix `sum` divergence Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Align `product` with `sum` Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Rename `alias` in `aggregates()` to `target` for clarity Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add a null guard at the top of `jsToLuaValue` This preserves `null`/`undefined` as-is (both map to Lua nil) and prevents them from falling through to the `typeof` "object" branch. For this PR it means that null `target` in our `aggregates` entries will correctly show as empty/`nil` in query results rather than `{}`. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Documentation reflects recent changes Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks * `sum(`) and` product(`) now return null when no rows match (matching Postgres semantics) instead of returning 0 and 1 respectively. * Query result columns that hold null are internally preserved using a `LIQ_NULL` sentinel so that column keys survive in `LuaTable` storage. This sentinel was leaking into Lua code as "userdata" through three read paths: * `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when accessing table fields, * `rawget` (stdlib): the builtin `rawget` function exposed the sentinel without converting it back to `nil`, * `createAugmentedEnv`: string interpolation unpacked table values via `rawGet` into local variables, making the sentinel visible in template expressions like `${var}`. All three now convert `LIQ_NULL` to `nil` at the read boundary, keeping the sentinel internal to table storage where it belongs. * Update affected test expectations accordingly. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: `array_agg` preserves NULL positions Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add symbol guard to `json_agg` `JSON.stringify(Symbol(...))` in an array produces null by accident. That is a JS implementation detail we **MUST NOT** rely on. Explicit null push makes intent clear and avoids surprises if the `Symbol` representation ever changes. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add symbol guard to `yaml_agg` (ditto) `js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes it emit YAML null (or `~`), which is the correct YAML representation of a missing value and matches standard `json_agg`/`yaml_agg` NULL-inclusion semantics. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add intra-aggregate ordering null guards Without this, `LIQ_NULL` sort keys would fall through to `valA < valB` which is always false for `Symbol`s which is breaking the `nulls first`/`nulls last` contract... Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Ditto, but for `order by` null comparisons Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated against the outer query environment where the object variable is not bound. This caused multi-argument aggregates like `covar_samp(data.y, data.x)` to fail with nil reference errors. This commit addresses this by evaluating extra args per-item inside the iterate loop using the item environment so all arguments resolve correctly. We also add few common aggregates: - `covar_pop`, `covar_samp`, `corr`: population/sample covariance and correlation coefficient using online co-moment algorithm. - `quantile(value, q, method)`: general quantile with interpolation methods: lower, higher, nearest, midpoint and default linear. - `percentile_cont(value, q)`: continuous percentile (linear) - `percentile_disc(value, q)`: discrete percentile (lower) Note: `percentile_cont` and `percentile_disc` share the `quantile` implementation through `ctx.name` at initialize time. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Update docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Make the ordering for quantile aggregates explicit Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Update docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Improve docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> --------- Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> |
||
|
Zef Hemel
|
799e27a9b4 | Enable biome based formatter and reformat | ||
|
Zef Hemel
|
cce5b669f6 | Merge branch 'main' into to-node | ||
|
Matouš Jan Fialka
|
9c9ca047f0 |
Fix: Handle tagged floats before plain object check in renderCellContent
Tagged floats (`{value: N, isFloat: true}`) were matched by
`isPlainObject` and rendered as a two-column table with "value" and
"isFloat" headers instead of as a formatted number. This is now fixed.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
|
||
|
Matouš Jan Fialka
|
2aaa797bda |
Improve Lua Widget Rendering
Lua `${...}` expression results are rendered using two parallel paths:
- **HTML path** (display path) for visual display in the editor and
- **Markdown path** (copy path) for the *copy button* output.
Every Lua value is classified as one of:
- **Scalar:** number, string, boolean, or `nil`
- **Empty table:** `{}`
- **Array:** a `LuaTable` with only sequential integer keys (no string keys)
- **Array of tables:** an array where every element is a `LuaTable`
- **Record:** a `LuaTable` with any string key (may also have integer keys)
Scalars are rendered as inline Markdown - wiki links, tags, bold, and
other inline syntax are resolved, `nil` renders as `<span data-empty>`.
Empty tables render as `<table data-table-empty>`.
An array of tables renders as a multi-row `<table>`. Column headers
are the union of all element keys. Each cell value recurses through
the same classification.
An array of scalars (or a mix of scalars and tables) renders as `<ul>`
with one `<li>` per element. Each item value recurses.
A record renders as a single-row `<table>` with one column per key.
Cell values recurse.
Every `<td>` carries `data-table-cell-type` and optionally
`data-table-cell-empty`. Every `<li>` carries `data-list-item-type` and
optionally `data-list-item-empty`. The outermost element carries
`data-type`. These attributes allow CSS styling of rendered widgets.
Nesting is **unlimited**. Tables inside cells, lists inside cells,
tables inside list items all render correctly in HTML.
Scalars render as plain text. Lua `nil` renders as `nil`. Empty tables
render as `*(empty table)*`.
An array of tables renders as a pipe-delimited Markdown table. Nested
values in cells are rendered as Lua literals via `toStringAsync()`
(e.g. `{1, 2, 3}` or `{a = 1}`), since Markdown table cells cannot
contain block-level content.
An array of scalars renders as a Markdown list (`- item`). When a list
item is itself multi-line (a sublist) it uses an empty bullet with
two-space indented content on the next line:
```
- 1
- 2
-
- nested 1
- nested 2
```
Nested tables or arrays **beyond the first list nesting** level are
rendered as Lua literals.
A record renders as a single-row Markdown table.
Column headers follow the order returned by `LuaTable.keys`: string keys
in their insertion order, followed by integer keys in ascending order.
For example, `{a = 1, b = 2, 3, d = 4}` produces columns `a
| b | d | 1`. This is consequence of `LuaTable` storing string keys
and the array part separately (for performance).
Arrays mixing plain scalars and record-like tables (e.g. `{1, 2, 3,
{a=3}}`) were previously rendered as a single flat table due to an
internal JS conversion (`toJS`). They now render as a list, with record
elements shown as nested tables - which more accurately reflects their
Lua structure. This is the only divergence from the original
presentation.
To restore the old tabular layout, wrap all elements as records with
a consistent key:
```lua
-- Before (was a flat table, now a list with nested table):
${{1, 2, 3, {a=3}}}
-- After (explicit records, renders as a table):
${{{value=1}, {value=2}, {value=3}, {a=3}}}
```
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
|