* Allow aggregate functions in `select`/`having` without an explicit
`group by` clause. The entire result set is treated as one group.
Note: Uses a `Symbol` as the map key for the implicit single group to
avoid any collision with string keys from `generateKey`.
* Extend documentation.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry
* Extend with 13 new built-in aggregates: `product`, `string_agg`,
`yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`,
`bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`.
* Introduce `aggregate.alias` API allowing users to define custom
aliases for any aggregate. Standard aliases (`every`, `std`, `stddev`
and `variance`) are now defined via this API rather than hardcoded.
* Add `index.aggregates` queryable collection so users can discover
all available aggregates directly from LIQ queries.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix config pass through query path so custom aggregates work
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage
`config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on
all arguments. This converted the aggregate `LuaTable` to a plain JS
object and wrapped `LuaFunction` callbacks in JS functions that also
converted their returned values via `luaValueToJS`. The result was that
state returned by initialize (a `LuaTable`) got converted to a plain JS
object before being passed to `iterate`. Therefor Lua operations like
`table.insert` on that were failing because they expected a `LuaTable`
and not a plain JS array.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix formatting
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Improve aggregate functions descriptions, fix `sum` divergence
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Align `product` with `sum`
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Rename `alias` in `aggregates()` to `target` for clarity
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add a null guard at the top of `jsToLuaValue`
This preserves `null`/`undefined` as-is (both map to Lua nil) and
prevents them from falling through to the `typeof` "object" branch.
For this PR it means that null `target` in our `aggregates` entries will
correctly show as empty/`nil` in query results rather than `{}`.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Documentation reflects recent changes
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks
* `sum(`) and` product(`) now return null when no rows match (matching
Postgres semantics) instead of returning 0 and 1 respectively.
* Query result columns that hold null are internally preserved using
a `LIQ_NULL` sentinel so that column keys survive in `LuaTable`
storage. This sentinel was leaking into Lua code as "userdata"
through three read paths:
* `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when
accessing table fields,
* `rawget` (stdlib): the builtin `rawget` function exposed the
sentinel without converting it back to `nil`,
* `createAugmentedEnv`: string interpolation unpacked table values via
`rawGet` into local variables, making the sentinel visible in
template expressions like `${var}`.
All three now convert `LIQ_NULL` to `nil` at the read boundary,
keeping the sentinel internal to table storage where it belongs.
* Update affected test expectations accordingly.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: `array_agg` preserves NULL positions
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add symbol guard to `json_agg`
`JSON.stringify(Symbol(...))` in an array produces null by accident.
That is a JS implementation detail we **MUST NOT** rely on. Explicit
null push makes intent clear and avoids surprises if the `Symbol`
representation ever changes.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add symbol guard to `yaml_agg` (ditto)
`js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes
it emit YAML null (or `~`), which is the correct YAML representation of
a missing value and matches standard `json_agg`/`yaml_agg`
NULL-inclusion semantics.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Add intra-aggregate ordering null guards
Without this, `LIQ_NULL` sort keys would fall through to `valA < valB`
which is always false for `Symbol`s which is breaking the `nulls
first`/`nulls last` contract...
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Ditto, but for `order by` null comparisons
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates
Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated
against the outer query environment where the object variable is not
bound. This caused multi-argument aggregates like `covar_samp(data.y,
data.x)` to fail with nil reference errors. This commit addresses this
by evaluating extra args per-item inside the iterate loop using the item
environment so all arguments resolve correctly.
We also add few common aggregates:
- `covar_pop`, `covar_samp`, `corr`: population/sample covariance and
correlation coefficient using online co-moment algorithm.
- `quantile(value, q, method)`: general quantile with interpolation
methods: lower, higher, nearest, midpoint and default linear.
- `percentile_cont(value, q)`: continuous percentile (linear)
- `percentile_disc(value, q)`: discrete percentile (lower)
Note: `percentile_cont` and `percentile_disc` share the `quantile`
implementation through `ctx.name` at initialize time.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Update docs
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Make the ordering for quantile aggregates explicit
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Update docs
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Improve docs
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Implements `filter(where <cond>)` for aggregate functions in LIQ
queries which allows per-row conditional filtering within grouped
aggregates. Also extends test suite and documentation.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* [LIQ] Extend with `group by` and `having` support
Also update documentation with extensive examples and add some parser
tests.
Note: `make generate` JS files included.
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Merge `liq-add-support-for-aggregators` branch
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fix link, remove `_` example
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Wrap example LIQ queries inside `query [[ .. ]]` and replace `sql` code block type to `lua`
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Example self-consistency fix
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
* Fixed some queries
* Add experimental tags to LIQ aggregation/grouping
---------
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Co-authored-by: Zef Hemel <zef@zef.me>
Zef Hemel