Files
plainleaf/client/space_lua/stdlib.ts
T
Matouš Jan FialkaandGitHub 6cbb61c7db [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry (#1891)
* [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry

* Extend with 13 new built-in aggregates: `product`, `string_agg`,
  `yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`,
  `bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`.

* Introduce `aggregate.alias` API allowing users to define custom
  aliases for any aggregate. Standard aliases (`every`, `std`, `stddev`
  and `variance`) are now defined via this API rather than hardcoded.

* Add `index.aggregates` queryable collection so users can discover
  all available aggregates directly from LIQ queries.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix config pass through query path so custom aggregates work

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage

`config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on
all arguments. This converted the aggregate `LuaTable` to a plain JS
object and wrapped `LuaFunction` callbacks in JS functions that also
converted their returned values via `luaValueToJS`. The result was that
state returned by initialize (a `LuaTable`) got converted to a plain JS
object before being passed to `iterate`. Therefor Lua operations like
`table.insert` on that were failing because they expected a `LuaTable`
and not a plain JS array.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix formatting

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Improve aggregate functions descriptions, fix `sum` divergence

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Align `product` with `sum`

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Rename `alias` in `aggregates()` to `target` for clarity

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add a null guard at the top of `jsToLuaValue`

This preserves `null`/`undefined` as-is (both map to Lua nil) and
prevents them from falling through to the `typeof` "object" branch.

For this PR it means that null `target` in our `aggregates` entries will
correctly show as empty/`nil` in query results rather than `{}`.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Documentation reflects recent changes

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks

* `sum(`) and` product(`) now return null when no rows match (matching
  Postgres semantics) instead of returning 0 and 1 respectively.

* Query result columns that hold null are internally preserved using
  a `LIQ_NULL` sentinel so that column keys survive in `LuaTable`
  storage.  This sentinel was leaking into Lua code as "userdata"
  through three read paths:

  * `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when
    accessing table fields,

  * `rawget` (stdlib): the builtin `rawget` function exposed the
    sentinel without converting it back to `nil`,

  * `createAugmentedEnv`: string interpolation unpacked table values via
    `rawGet` into local variables, making the sentinel visible in
    template expressions like `${var}`.

  All three now convert `LIQ_NULL` to `nil` at the read boundary,
  keeping the sentinel internal to table storage where it belongs.

* Update affected test expectations accordingly.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: `array_agg` preserves NULL positions

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add symbol guard to `json_agg`

`JSON.stringify(Symbol(...))` in an array produces null by accident.
That is a JS implementation detail we **MUST NOT** rely on. Explicit
null push makes intent clear and avoids surprises if the `Symbol`
representation ever changes.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add symbol guard to `yaml_agg` (ditto)

`js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes
it emit YAML null (or `~`), which is the correct YAML representation of
a missing value and matches standard `json_agg`/`yaml_agg`
NULL-inclusion semantics.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add intra-aggregate ordering null guards

Without this, `LIQ_NULL` sort keys would fall through to `valA < valB`
which is always false for `Symbol`s which is breaking the `nulls
first`/`nulls last` contract...

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Ditto, but for `order by` null comparisons

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates

Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated
against the outer query environment where the object variable is not
bound. This caused multi-argument aggregates like `covar_samp(data.y,
data.x)` to fail with nil reference errors. This commit addresses this
by evaluating extra args per-item inside the iterate loop using the item
environment so all arguments resolve correctly.

We also add few common aggregates:

- `covar_pop`, `covar_samp`, `corr`: population/sample covariance and
  correlation coefficient using online co-moment algorithm.

- `quantile(value, q, method)`: general quantile with interpolation
  methods: lower, higher, nearest, midpoint and default linear.

- `percentile_cont(value, q)`: continuous percentile (linear)

- `percentile_disc(value, q)`: discrete percentile (lower)

Note: `percentile_cont` and `percentile_disc` share the `quantile`
implementation through `ctx.name` at initialize time.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Update docs

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Make the ordering for quantile aggregates explicit

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Update docs

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Improve docs

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

---------

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
2026-03-19 09:37:09 +01:00

504 lines
15 KiB
TypeScript

import {
getMetatable,
type ILuaFunction,
isILuaFunction,
isLuaTable,
LuaBuiltinFunction,
luaCall,
luaCloseFromMark,
luaEnsureCloseStack,
LuaEnv,
luaGet,
luaKeys,
luaLen,
LuaMultiRes,
LuaRuntimeError,
type LuaStackFrame,
type LuaTable,
luaToString,
luaTypeOf,
type LuaValue,
singleResult,
} from "./runtime.ts";
import { stringApi } from "./stdlib/string.ts";
import { tableApi } from "./stdlib/table.ts";
import { osApi } from "./stdlib/os.ts";
import { jsApi } from "./stdlib/js.ts";
import { spaceluaApi } from "./stdlib/space_lua.ts";
import { mathApi } from "./stdlib/math.ts";
import { parse } from "./parse.ts";
import { evalStatement } from "./eval.ts";
import { encodingApi } from "./stdlib/encoding.ts";
import { luaToNumberDetailed } from "./tonumber.ts";
import { luaLoad } from "./stdlib/load.ts";
import { cryptoApi } from "./stdlib/crypto.ts";
import { netApi } from "./stdlib/net.ts";
import { isTaggedFloat, makeLuaFloat } from "./numeric.ts";
import { isPromise } from "./rp.ts";
import { isSqlNull } from "./liq_null.ts";
const printFunction = new LuaBuiltinFunction(async (_sf, ...args) => {
console.log("[Lua]", ...(await Promise.all(args.map((v) => luaToString(v)))));
});
const assertFunction = new LuaBuiltinFunction(
async (sf, value: any, message?: string) => {
if (!(await value)) {
throw new LuaRuntimeError(`Assertion failed: ${message}`, sf);
}
},
);
const ipairsFunction = new LuaBuiltinFunction((sf, t: LuaTable | any[]) => {
let i = 0;
return async () => {
i = i + 1;
const v = await luaGet(t, i, sf.astCtx ?? null, sf);
if (v === null || v === undefined) {
return;
}
return new LuaMultiRes([i, v]);
};
});
const pairsFunction = new LuaBuiltinFunction(
(sf, t: LuaTable | any[] | Record<string, any>) => {
// Respect `__pairs` metamethod for Lua tables
if (isLuaTable(t)) {
const mt = (t as any).metatable as LuaTable | null | undefined;
if (mt) {
const mm = mt.get("__pairs", sf);
if (mm && (typeof mm === "function" || isILuaFunction(mm))) {
// __pairs must return (iter, state, control, closing)
return luaCall(mm, [t], sf.astCtx ?? {}, sf);
}
}
}
let keys: (string | number)[];
if (Array.isArray(t)) {
keys = Array.from({ length: t.length }, (_, i) => i + 1); // For arrays, generate 1-based indices
} else if (isLuaTable(t) || t instanceof LuaEnv) {
keys = t.keys();
} else {
// For plain JavaScript objects case, note: this will also include keys from the prototype
keys = [];
for (const key in t) {
keys.push(key);
}
}
let i = 0;
const iter = async () => {
if (i >= keys.length) {
return;
}
const key = keys[i];
i++;
const value = await luaGet(t, key, sf.astCtx ?? null, sf);
return new LuaMultiRes([key, value]);
};
// Must return (iter, state, control) for generic for
return new LuaMultiRes([iter, t, null]);
},
);
export const eachFunction = new LuaBuiltinFunction(
(sf, ar: LuaTable | any[]) => {
let i = 1;
const length = (ar as any).length;
return async () => {
if (i > length) {
return;
}
const result = await luaGet(ar, i, sf.astCtx ?? null, sf);
i++;
return result;
};
},
);
const typeFunction = new LuaBuiltinFunction(
(_sf, value: LuaValue): string | Promise<string> => {
return luaTypeOf(value);
},
);
// tostring() checks `__tostring` metamethod first (with live SF), then
// falls back to the default `luaToString` representation.
const tostringFunction = new LuaBuiltinFunction(
(sf, value: any): string | Promise<string> => {
const mt = getMetatable(value, sf);
if (mt) {
const mm = mt.rawGet("__tostring");
if (mm !== undefined && mm !== null) {
const ctx = sf.astCtx ?? {};
const r = luaCall(mm, [value], ctx as any, sf);
const unwrap = (v: any): string => {
const s = singleResult(v);
if (typeof s !== "string") {
throw new LuaRuntimeError("'__tostring' must return a string", sf);
}
return s;
};
if (isPromise(r)) {
return (r as Promise<any>).then(unwrap);
}
return unwrap(r);
}
}
return luaToString(value);
},
);
const tonumberFunction = new LuaBuiltinFunction(
(sf, value: LuaValue, base?: number) => {
if (base !== undefined) {
if (!(typeof base === "number" && base >= 2 && base <= 36)) {
throw new LuaRuntimeError(
"bad argument #2 to 'tonumber' (base out of range)",
sf,
);
}
}
if (typeof value === "number") {
return value;
}
if (isTaggedFloat(value)) {
return value;
}
if (typeof value !== "string") {
return null;
}
const result = luaToNumberDetailed(value, base);
if (result === null) {
return null;
}
if (result.numericType === "float") {
return makeLuaFloat(result.value);
}
return result.value;
},
);
const errorFunction = new LuaBuiltinFunction((sf, message: string) => {
throw new LuaRuntimeError(message, sf);
});
async function pcallBoundary(
sf: LuaStackFrame,
fn: ILuaFunction,
args: LuaValue[],
): Promise<{ ok: true; values: LuaValue[] } | { ok: false; message: string }> {
const closeStack = luaEnsureCloseStack(sf);
const mark = closeStack.length;
const errMsgOf = (e: any): string =>
e instanceof LuaRuntimeError ? e.message : (e?.message ?? String(e));
try {
const r = await luaCall(fn, args, sf.astCtx!, sf);
await luaCloseFromMark(sf, mark, null);
const values = r instanceof LuaMultiRes ? r.flatten().values : [r];
return { ok: true, values };
} catch (e: any) {
const msg = errMsgOf(e);
try {
await luaCloseFromMark(sf, mark, msg);
return { ok: false, message: msg };
} catch (closeErr: any) {
return { ok: false, message: errMsgOf(closeErr) };
}
}
}
const pcallFunction = new LuaBuiltinFunction(
async (sf, fn: ILuaFunction, ...args) => {
// To-be-closed variables must be closed when unwinding to the
// protected call boundary. Space Lua uses a per-thread close
// stack, so we snapshot its length and close anything pushed
// after that.
//
// The protected call boundary must be established *before*
// evaluating the function and its arguments. Otherwise, any
// `<close>` locals created while evaluating `pcall`'s arguments
// will be wrongly treated as "inside" the protected call, and
// `pcall` may end up closing them (or affecting close ordering).
//
// `threadState` is read-only on the stack frame; do not reassign!
const res = await pcallBoundary(sf, fn, args);
if (res.ok) {
return new LuaMultiRes([true, ...res.values]);
}
return new LuaMultiRes([false, res.message]);
},
);
const xpcallFunction = new LuaBuiltinFunction(
async (sf, fn: ILuaFunction, errorHandler: ILuaFunction, ...args) => {
// Same semantic as `pcall` (see comments there)
const res = await pcallBoundary(sf, fn, args);
if (res.ok) {
return new LuaMultiRes([true, ...res.values]);
}
const hr = await luaCall(errorHandler, [res.message], sf.astCtx!, sf);
const outVals = hr instanceof LuaMultiRes ? hr.flatten().values : [hr];
return new LuaMultiRes([false, ...outVals]);
},
);
const setmetatableFunction = new LuaBuiltinFunction(
(sf, table: LuaTable, metatable: LuaTable) => {
if (!metatable) {
throw new LuaRuntimeError("metatable cannot be set to nil", sf);
}
table.metatable = metatable;
return table;
},
);
const rawlenFunction = new LuaBuiltinFunction((_sf, value: LuaValue) => {
return luaLen(value, _sf, true);
});
const rawsetFunction = new LuaBuiltinFunction(
(_sf, table: LuaTable, key: LuaValue, value: LuaValue) => {
return (table as any).rawSet(key, value);
},
);
const rawgetFunction = new LuaBuiltinFunction(
(_sf, table: any, key: LuaValue) => {
const isArray = Array.isArray(table);
const isPlainObj =
typeof table === "object" &&
table !== null &&
(table as any).constructor === Object;
if (!isLuaTable(table) && !isArray && !isPlainObj) {
let typeName = "userdata";
if (table === null || table === undefined) {
typeName = "nil";
} else if (typeof table === "boolean") {
typeName = "boolean";
} else if (typeof table === "number" || isTaggedFloat(table)) {
typeName = "number";
} else if (typeof table === "string") {
typeName = "string";
} else if (
typeof table === "function" ||
(typeof table === "object" &&
table !== null &&
typeof (table as any).call === "function")
) {
typeName = "function";
}
throw new LuaRuntimeError(
`bad argument #1 to 'rawget' (table expected, got ${typeName})`,
_sf,
);
}
if (isLuaTable(table)) {
const v = table.rawGet(key);
return v === undefined || isSqlNull(v) ? null : v;
}
const k = isTaggedFloat(key) ? key.value : key;
if (isArray) {
if (typeof k === "number") {
const v = (table as any[])[k - 1];
return v === undefined ? null : v;
}
const v = (table as Record<string, any>)[k];
return v === undefined ? null : v;
}
const v = (table as Record<string | number, any>)[k as any];
return v === undefined ? null : v;
},
);
const rawequalFunction = new LuaBuiltinFunction((_sf, a: any, b: any) => {
const av = isTaggedFloat(a) ? a.value : a;
const bv = isTaggedFloat(b) ? b.value : b;
return av === bv;
});
const getmetatableFunction = new LuaBuiltinFunction((_sf, table: LuaTable) => {
return (table as any).metatable;
});
const dofileFunction = new LuaBuiltinFunction(async (sf, filename: string) => {
const global = sf.threadLocal.get("_GLOBAL") as LuaEnv;
const file = (await luaCall(
(global.get("space") as any).get("readFile"),
[filename],
sf.astCtx!,
sf,
)) as Uint8Array;
const code = new TextDecoder().decode(file);
try {
const parsedExpr = parse(code);
const env = new LuaEnv(global);
await evalStatement(parsedExpr, env, sf.withCtx(parsedExpr.ctx));
} catch (e: any) {
throw new LuaRuntimeError(
`Error evaluating "${filename}": ${e.message}`,
sf,
);
}
});
/**
* From the Lua docs:
*
* If index is a number, returns all arguments after argument number
* index; a negative number indexes from the end (-1 is the last
* argument). Otherwise, index must be the string "#", and select
* returns the total number of extra arguments it received.
*/
const selectFunction = new LuaBuiltinFunction(
(_sf, index: number | "#", ...args: LuaValue[]) => {
if (index === "#") {
return args.length;
}
if (typeof index === "number") {
if (index >= 0) {
return new LuaMultiRes(args.slice(index - 1));
}
return new LuaMultiRes(args.slice(args.length + index));
}
},
);
/**
* From the Lua docs:
*
* Allows a program to traverse all fields of a table. Its first
* argument is a table and its second argument is an index in this
* table. A call to next returns the next index of the table and its
* associated value. When called with nil as its second argument, next
* returns an initial index and its associated value. When called with
* the last index, or with nil in an empty table, next returns nil. If
* the second argument is absent, then it is interpreted as nil. In
* particular, you can use next(t) to check whether a table is empty.
*
* The order in which the indices are enumerated is not specified, even
* for numeric indices. (To traverse a table in numerical order, use
* a numerical for.)
*
* You should not assign any value to a non-existent field in a table
* during its traversal. You may however modify existing fields. In
* particular, you may set existing fields to nil.
*/
const nextFunction = new LuaBuiltinFunction(
(sf, table: LuaTable | Record<string, any>, index: number | null = null) => {
if (!table) {
// When nil value
return null;
}
const keys = luaKeys(table);
// Empty table -> null return value
if (keys.length === 0) {
return null;
}
if (index === null) {
// Return the first key, value
const key = keys[0];
return new LuaMultiRes([key, luaGet(table, key, sf.astCtx ?? null, sf)]);
}
// Find index in the key list
const idx = keys.indexOf(index);
if (idx === -1) {
// Not found
throw new LuaRuntimeError("invalid key to 'next': key not found", sf);
}
const key = keys[idx + 1];
if (key === undefined) {
// When called with the last key, should return nil
return null;
}
return new LuaMultiRes([key, luaGet(table, key, sf.astCtx ?? null, sf)]);
},
);
// Non-standard, but useful
const someFunction = new LuaBuiltinFunction(async (_sf, value: any) => {
switch (await luaTypeOf(value)) {
case "number":
if (!Number.isFinite(value)) return null;
break;
case "string":
if (value.trim() === "") return null;
break;
case "table":
if (luaKeys(value).length === 0) return null;
}
return value;
});
const loadFunction = new LuaBuiltinFunction((sf, s) => luaLoad(s, sf));
export function luaBuildStandardEnv() {
const env = new LuaEnv();
// _G global
env.set("_G", env);
// Lua version string - for now it signals Lua 5.4 compatibility with
// selective 5.5 features; kept non-standard so callers can distinguish
// Space Lua from a plain Lua runtime.
env.set("_VERSION", "Lua 5.4+");
// Top-level builtins
env.set("print", printFunction);
env.set("assert", assertFunction);
env.set("type", typeFunction);
env.set("tostring", tostringFunction);
env.set("tonumber", tonumberFunction);
env.set("select", selectFunction);
env.set("next", nextFunction);
// Iterators
env.set("pairs", pairsFunction);
env.set("ipairs", ipairsFunction);
// meta table stuff
env.set("setmetatable", setmetatableFunction);
env.set("getmetatable", getmetatableFunction);
env.set("rawlen", rawlenFunction);
env.set("rawset", rawsetFunction);
env.set("rawget", rawgetFunction);
env.set("rawequal", rawequalFunction);
env.set("dofile", dofileFunction);
// Error handling
env.set("error", errorFunction);
env.set("pcall", pcallFunction);
env.set("xpcall", xpcallFunction);
// Evaluation
env.set("load", loadFunction);
// APIs
env.set("string", stringApi);
env.set("table", tableApi);
env.set("os", osApi);
env.set("js", jsApi);
env.set("math", mathApi);
// Non-standard
env.set("each", eachFunction);
env.set("spacelua", spaceluaApi);
env.set("encoding", encodingApi);
env.set("crypto", cryptoApi);
env.set("net", netApi);
env.set("some", someFunction);
return env;
}