Files
plainleaf/client/space_lua/stdlib/space_lua.ts
T
Matouš Jan FialkaandGitHub 6cbb61c7db [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry (#1891)
* [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry

* Extend with 13 new built-in aggregates: `product`, `string_agg`,
  `yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`,
  `bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`.

* Introduce `aggregate.alias` API allowing users to define custom
  aliases for any aggregate. Standard aliases (`every`, `std`, `stddev`
  and `variance`) are now defined via this API rather than hardcoded.

* Add `index.aggregates` queryable collection so users can discover
  all available aggregates directly from LIQ queries.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix config pass through query path so custom aggregates work

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage

`config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on
all arguments. This converted the aggregate `LuaTable` to a plain JS
object and wrapped `LuaFunction` callbacks in JS functions that also
converted their returned values via `luaValueToJS`. The result was that
state returned by initialize (a `LuaTable`) got converted to a plain JS
object before being passed to `iterate`. Therefor Lua operations like
`table.insert` on that were failing because they expected a `LuaTable`
and not a plain JS array.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix formatting

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Improve aggregate functions descriptions, fix `sum` divergence

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Align `product` with `sum`

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Rename `alias` in `aggregates()` to `target` for clarity

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add a null guard at the top of `jsToLuaValue`

This preserves `null`/`undefined` as-is (both map to Lua nil) and
prevents them from falling through to the `typeof` "object" branch.

For this PR it means that null `target` in our `aggregates` entries will
correctly show as empty/`nil` in query results rather than `{}`.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Documentation reflects recent changes

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks

* `sum(`) and` product(`) now return null when no rows match (matching
  Postgres semantics) instead of returning 0 and 1 respectively.

* Query result columns that hold null are internally preserved using
  a `LIQ_NULL` sentinel so that column keys survive in `LuaTable`
  storage.  This sentinel was leaking into Lua code as "userdata"
  through three read paths:

  * `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when
    accessing table fields,

  * `rawget` (stdlib): the builtin `rawget` function exposed the
    sentinel without converting it back to `nil`,

  * `createAugmentedEnv`: string interpolation unpacked table values via
    `rawGet` into local variables, making the sentinel visible in
    template expressions like `${var}`.

  All three now convert `LIQ_NULL` to `nil` at the read boundary,
  keeping the sentinel internal to table storage where it belongs.

* Update affected test expectations accordingly.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: `array_agg` preserves NULL positions

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add symbol guard to `json_agg`

`JSON.stringify(Symbol(...))` in an array produces null by accident.
That is a JS implementation detail we **MUST NOT** rely on. Explicit
null push makes intent clear and avoids surprises if the `Symbol`
representation ever changes.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add symbol guard to `yaml_agg` (ditto)

`js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes
it emit YAML null (or `~`), which is the correct YAML representation of
a missing value and matches standard `json_agg`/`yaml_agg`
NULL-inclusion semantics.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add intra-aggregate ordering null guards

Without this, `LIQ_NULL` sort keys would fall through to `valA < valB`
which is always false for `Symbol`s which is breaking the `nulls
first`/`nulls last` contract...

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Ditto, but for `order by` null comparisons

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates

Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated
against the outer query environment where the object variable is not
bound. This caused multi-argument aggregates like `covar_samp(data.y,
data.x)` to fail with nil reference errors. This commit addresses this
by evaluating extra args per-item inside the iterate loop using the item
environment so all arguments resolve correctly.

We also add few common aggregates:

- `covar_pop`, `covar_samp`, `corr`: population/sample covariance and
  correlation coefficient using online co-moment algorithm.

- `quantile(value, q, method)`: general quantile with interpolation
  methods: lower, higher, nearest, midpoint and default linear.

- `percentile_cont(value, q)`: continuous percentile (linear)

- `percentile_disc(value, q)`: discrete percentile (lower)

Note: `percentile_cont` and `percentile_disc` share the `quantile`
implementation through `ctx.name` at initialize time.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Update docs

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Make the ordering for quantile aggregates explicit

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Update docs

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Improve docs

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

---------

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
2026-03-19 09:37:09 +01:00

144 lines
4.2 KiB
TypeScript

import { parseExpressionString } from "../parse.ts";
import type { LuaExpression } from "../ast.ts";
import { evalExpression } from "../eval.ts";
import {
LuaBuiltinFunction,
LuaEnv,
LuaRuntimeError,
type LuaStackFrame,
LuaTable,
luaToString,
luaValueToJS,
singleResult,
} from "../runtime.ts";
import { isSqlNull } from "../liq_null.ts";
/**
* These are Space Lua specific functions that are available to all scripts, but are not part of the standard Lua language.
*/
/**
* Helper function to create an augmented environment
*/
function createAugmentedEnv(
sf: LuaStackFrame,
envAugmentation?: LuaTable,
): LuaEnv {
const globalEnv = sf.threadLocal.get("_GLOBAL");
if (!globalEnv) {
throw new Error("_GLOBAL not defined");
}
const env = new LuaEnv(globalEnv);
if (envAugmentation) {
env.setLocal("_", envAugmentation);
for (const key of envAugmentation.keys()) {
const v = envAugmentation.rawGet(key);
env.setLocal(key, isSqlNull(v) ? null : v);
}
}
return env;
}
/**
* Interpolates a string with lua expressions and returns the result.
*
* @param sf - The current space_lua state.
* @param template - The template string to interpolate.
* @param envAugmentation - An optional environment to augment the global environment with.
* @returns The interpolated string.
*/
export async function interpolateLuaString(
sf: LuaStackFrame,
template: string,
envAugmentation?: LuaTable,
): Promise<string> {
let result = "";
let currentIndex = 0;
while (true) {
const startIndex = template.indexOf("${", currentIndex);
if (startIndex === -1) {
result += template.slice(currentIndex);
break;
}
result += template.slice(currentIndex, startIndex);
// Find matching closing brace by counting nesting
let nestLevel = 1;
let endIndex = startIndex + 2;
while (nestLevel > 0 && endIndex < template.length) {
if (template[endIndex] === "{") {
nestLevel++;
} else if (template[endIndex] === "}") {
nestLevel--;
}
if (nestLevel > 0) {
endIndex++;
}
}
if (nestLevel > 0) {
throw new LuaRuntimeError("Unclosed interpolation expression", sf);
}
const expr = template.slice(startIndex + 2, endIndex);
try {
const parsedExpr = parseExpressionString(expr);
const env = createAugmentedEnv(sf, envAugmentation);
// Do `luaToString` before `luaValueToJS` to preserve tagged float
// formatting.
const luaResult = singleResult(await evalExpression(parsedExpr, env, sf));
result += await luaToString(luaResult);
} catch (e: any) {
throw new LuaRuntimeError(`Error evaluating "${expr}": ${e.message}`, sf);
}
currentIndex = endIndex + 1;
}
return result;
}
export const spaceluaApi = new LuaTable({
/**
* Parses a lua expression and returns the parsed expression.
*
* @param sf - The current space_lua state.
* @param luaExpression - The lua expression to parse.
* @returns The parsed expression.
*/
parseExpression: new LuaBuiltinFunction((_sf, luaExpression: string) => {
return parseExpressionString(luaExpression);
}),
/**
* Evaluates a parsed lua expression and returns the result.
*
* @param sf - The current space_lua state.
* @param parsedExpr - The parsed lua expression to evaluate.
* @param envAugmentation - An optional environment to augment the global environment with.
* @returns The result of the evaluated expression.
*/
evalExpression: new LuaBuiltinFunction(
async (sf, parsedExpr: LuaExpression, envAugmentation?: LuaTable) => {
const env = createAugmentedEnv(sf, envAugmentation);
return luaValueToJS(await evalExpression(parsedExpr, env, sf), sf);
},
),
/**
* Interpolates a string with lua expressions and returns the result.
*/
interpolate: new LuaBuiltinFunction(
(sf, template: string, envAugmentation?: LuaTable) => {
return interpolateLuaString(sf, template, envAugmentation);
},
),
/**
* Returns your SilverBullet instance's base URL
*/
baseUrl: new LuaBuiltinFunction(() => {
//NOTE: Removing trailing slash to stay compatible with original code: `location.protocol + "//" + location.host;`
return document.baseURI.replace(/\/*$/, "");
}),
});