From 6cbb61c7db568cdb6f8e40ba589c821c14b6feff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Matou=C5=A1=20Jan=20Fialka?= Date: Thu, 19 Mar 2026 09:37:09 +0100 Subject: [PATCH] [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry (#1891) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry * Extend with 13 new built-in aggregates: `product`, `string_agg`, `yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`, `bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`. * Introduce `aggregate.alias` API allowing users to define custom aliases for any aggregate. Standard aliases (`every`, `std`, `stddev` and `variance`) are now defined via this API rather than hardcoded. * Add `index.aggregates` queryable collection so users can discover all available aggregates directly from LIQ queries. Signed-off-by: Matouš Jan Fialka * Fix config pass through query path so custom aggregates work Signed-off-by: Matouš Jan Fialka * Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage `config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on all arguments. This converted the aggregate `LuaTable` to a plain JS object and wrapped `LuaFunction` callbacks in JS functions that also converted their returned values via `luaValueToJS`. The result was that state returned by initialize (a `LuaTable`) got converted to a plain JS object before being passed to `iterate`. Therefor Lua operations like `table.insert` on that were failing because they expected a `LuaTable` and not a plain JS array. Signed-off-by: Matouš Jan Fialka * Fix formatting Signed-off-by: Matouš Jan Fialka * Improve aggregate functions descriptions, fix `sum` divergence Signed-off-by: Matouš Jan Fialka * Align `product` with `sum` Signed-off-by: Matouš Jan Fialka * Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry Signed-off-by: Matouš Jan Fialka * Rename `alias` in `aggregates()` to `target` for clarity Signed-off-by: Matouš Jan Fialka * Fix: Add a null guard at the top of `jsToLuaValue` This preserves `null`/`undefined` as-is (both map to Lua nil) and prevents them from falling through to the `typeof` "object" branch. For this PR it means that null `target` in our `aggregates` entries will correctly show as empty/`nil` in query results rather than `{}`. Signed-off-by: Matouš Jan Fialka * Fix: Documentation reflects recent changes Signed-off-by: Matouš Jan Fialka * Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks * `sum(`) and` product(`) now return null when no rows match (matching Postgres semantics) instead of returning 0 and 1 respectively. * Query result columns that hold null are internally preserved using a `LIQ_NULL` sentinel so that column keys survive in `LuaTable` storage. This sentinel was leaking into Lua code as "userdata" through three read paths: * `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when accessing table fields, * `rawget` (stdlib): the builtin `rawget` function exposed the sentinel without converting it back to `nil`, * `createAugmentedEnv`: string interpolation unpacked table values via `rawGet` into local variables, making the sentinel visible in template expressions like `${var}`. All three now convert `LIQ_NULL` to `nil` at the read boundary, keeping the sentinel internal to table storage where it belongs. * Update affected test expectations accordingly. Signed-off-by: Matouš Jan Fialka * Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s Signed-off-by: Matouš Jan Fialka * Fix: `array_agg` preserves NULL positions Signed-off-by: Matouš Jan Fialka * Fix: Add symbol guard to `json_agg` `JSON.stringify(Symbol(...))` in an array produces null by accident. That is a JS implementation detail we **MUST NOT** rely on. Explicit null push makes intent clear and avoids surprises if the `Symbol` representation ever changes. Signed-off-by: Matouš Jan Fialka * Fix: Add symbol guard to `yaml_agg` (ditto) `js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes it emit YAML null (or `~`), which is the correct YAML representation of a missing value and matches standard `json_agg`/`yaml_agg` NULL-inclusion semantics. Signed-off-by: Matouš Jan Fialka * Fix: Add intra-aggregate ordering null guards Without this, `LIQ_NULL` sort keys would fall through to `valA < valB` which is always false for `Symbol`s which is breaking the `nulls first`/`nulls last` contract... Signed-off-by: Matouš Jan Fialka * Fix: Ditto, but for `order by` null comparisons Signed-off-by: Matouš Jan Fialka * Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel Signed-off-by: Matouš Jan Fialka * Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text Signed-off-by: Matouš Jan Fialka * Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated against the outer query environment where the object variable is not bound. This caused multi-argument aggregates like `covar_samp(data.y, data.x)` to fail with nil reference errors. This commit addresses this by evaluating extra args per-item inside the iterate loop using the item environment so all arguments resolve correctly. We also add few common aggregates: - `covar_pop`, `covar_samp`, `corr`: population/sample covariance and correlation coefficient using online co-moment algorithm. - `quantile(value, q, method)`: general quantile with interpolation methods: lower, higher, nearest, midpoint and default linear. - `percentile_cont(value, q)`: continuous percentile (linear) - `percentile_disc(value, q)`: discrete percentile (lower) Note: `percentile_cont` and `percentile_disc` share the `quantile` implementation through `ctx.name` at initialize time. Signed-off-by: Matouš Jan Fialka * Update docs Signed-off-by: Matouš Jan Fialka * Make the ordering for quantile aggregates explicit Signed-off-by: Matouš Jan Fialka * Update docs Signed-off-by: Matouš Jan Fialka * Improve docs Signed-off-by: Matouš Jan Fialka --------- Signed-off-by: Matouš Jan Fialka --- client/data/datastore.ts | 4 +- client/data/object_index.ts | 90 ++- client/markdown_renderer/result_render.ts | 10 +- client/plugos/syscalls/index.ts | 3 + client/space_lua/aggregates.ts | 542 ++++++++++++++++-- client/space_lua/liq_null.ts | 6 + client/space_lua/query_collection.ts | 57 +- client/space_lua/query_test.lua | 8 +- client/space_lua/render_lua_html.ts | 2 +- client/space_lua/runtime.ts | 11 +- client/space_lua/stdlib.ts | 3 +- client/space_lua/stdlib/space_lua.ts | 4 +- libraries/Library/Std/APIs/Aggregate.md | 117 +++- .../Lua Integrated Query/Aggregating.md | 28 +- .../Lua Integrated Query/Grouping.md | 2 +- 15 files changed, 782 insertions(+), 105 deletions(-) create mode 100644 client/space_lua/liq_null.ts diff --git a/client/data/datastore.ts b/client/data/datastore.ts index be41cfe2..61a4faf8 100644 --- a/client/data/datastore.ts +++ b/client/data/datastore.ts @@ -4,6 +4,7 @@ import { } from "../space_lua/query_collection.ts"; import { LuaEnv, LuaStackFrame } from "../space_lua/runtime.ts"; import type { KvPrimitives, KvQueryOptions } from "./kv_primitives.ts"; +import type { Config } from "../config.ts"; import type { KV, KvKey } from "../../plug-api/types/datastore.ts"; @@ -76,7 +77,8 @@ export class DataStore { env: LuaEnv = new LuaEnv(), sf: LuaStackFrame = LuaStackFrame.lostFrame, enricher?: (key: KvKey, item: any) => any, + config?: Config, ): Promise { - return queryLua(this.kv, prefix, query, env, sf, enricher); + return queryLua(this.kv, prefix, query, env, sf, enricher, config); } } diff --git a/client/data/object_index.ts b/client/data/object_index.ts index ab829168..ecd55318 100644 --- a/client/data/object_index.ts +++ b/client/data/object_index.ts @@ -1,14 +1,15 @@ import type { ObjectValue } from "@silverbulletmd/silverbullet/type/index"; import type { Config } from "../config.ts"; -import type { - LuaCollectionQuery, - LuaQueryCollection, +import { + ArrayQueryCollection, + type LuaCollectionQuery, + type LuaQueryCollection, } from "../space_lua/query_collection.ts"; import { jsToLuaValue, LuaEnv, LuaStackFrame, - type LuaTable, + LuaTable, } from "../space_lua/runtime.ts"; import type { DataStore } from "./datastore.ts"; import type { KV, KvKey } from "@silverbulletmd/silverbullet/type/datastore"; @@ -16,6 +17,10 @@ import type { EventHook } from "../plugos/hooks/event.ts"; import type { DataStoreMQ } from "./mq.datastore.ts"; import type { Space } from "../space.ts"; import { validateObject } from "../plugos/syscalls/jsonschema.ts"; +import { + getAggregateSpec, + getBuiltinAggregateEntries, +} from "../space_lua/aggregates.ts"; const indexKey = "idx"; const pageKey = "ridx"; @@ -105,6 +110,7 @@ export class ObjectIndex { query: LuaCollectionQuery, env: LuaEnv, sf: LuaStackFrame, + config?: Config, ): Promise => { return this.ds.luaQuery( ["idx", tagName], @@ -126,11 +132,87 @@ export class ObjectIndex { value.metatable = mt; return value; }, + config, ); }, }; } + /** + * Returns a queryable collection of all aggregate functions: + * + * - builtin, + * - user-defined, and + * - aliases. + * + * Every row has all columns: `builtin`, `name`, `description`, + * `initialize`, `iterate`, `finish` and `target`. + */ + aggregates(): LuaQueryCollection { + const entries: Record[] = []; + + // Builtins are always listed (even if overridden) + for (const entry of getBuiltinAggregateEntries()) { + entries.push({ + builtin: true, + name: entry.name, + description: entry.description, + initialize: true, + iterate: true, + finish: entry.hasFinish, + target: null, + }); + } + + // Config entries (user-defined overrides and aliases) + const userAggs: Record = this.config.get("aggregates", {}); + for (const [key, spec] of Object.entries(userAggs)) { + const aliasTarget = spec instanceof LuaTable + ? spec.rawGet("alias") + : spec?.alias ?? null; + if (typeof aliasTarget === "string") { + const resolved = getAggregateSpec(aliasTarget, this.config); + entries.push({ + builtin: false, + name: key, + description: spec instanceof LuaTable + ? spec.rawGet("description") ?? resolved?.description ?? "" + : spec?.description ?? resolved?.description ?? "", + initialize: resolved ? !!resolved.initialize : false, + iterate: resolved ? !!resolved.iterate : false, + finish: resolved ? !!resolved.finish : false, + target: aliasTarget, + }); + } else { + let hasInit = false; + let hasIter = false; + let hasFin = false; + let desc = ""; + if (spec instanceof LuaTable) { + hasInit = !!spec.rawGet("initialize"); + hasIter = !!spec.rawGet("iterate"); + hasFin = !!spec.rawGet("finish"); + desc = spec.rawGet("description") ?? ""; + } else if (spec) { + hasInit = !!spec.initialize; + hasIter = !!spec.iterate; + hasFin = !!spec.finish; + desc = spec.description ?? ""; + } + entries.push({ + builtin: false, + name: key, + description: desc, + initialize: hasInit, + iterate: hasIter, + finish: hasFin, + target: null, + }); + } + } + return new ArrayQueryCollection(entries); + } + getObjectByRef(page: string, tag: string, ref: string) { return this.ds.get([indexKey, tag, this.cleanKey(ref, page), page]); } diff --git a/client/markdown_renderer/result_render.ts b/client/markdown_renderer/result_render.ts index 2176fdf5..2881d93a 100644 --- a/client/markdown_renderer/result_render.ts +++ b/client/markdown_renderer/result_render.ts @@ -4,10 +4,10 @@ import { luaToString, } from "../space_lua/runtime.ts"; import { isTaggedFloat } from "../space_lua/numeric.ts"; -import { isSqlNull } from "../space_lua/query_collection.ts"; +import { isSqlNull } from "../space_lua/liq_null.ts"; export function defaultTransformer(v: any, _k: string): Promise { - if (v === undefined || isSqlNull(v)) { + if (v === undefined || v === null || isSqlNull(v)) { return Promise.resolve(""); } if (typeof v === "string") { @@ -98,7 +98,7 @@ export function renderExpressionResult( result: any, cellTransformer: (v: any, k: string) => Promise = refCellTransformer, ): Promise { - if (result === undefined || result === null) { + if (result === undefined || result === null || isSqlNull(result)) { return Promise.resolve("nil"); } // LuaTable: render natively without `.toJS` @@ -217,7 +217,9 @@ function renderItemToMarkdown( cellTransformer: (v: any, k: string) => Promise, nested: boolean, ): Promise { - if (item === undefined || item === null) return Promise.resolve(""); + if (item === undefined || item === null || isSqlNull(item)) { + return Promise.resolve(""); + } if (item instanceof LuaTable) { if (item.empty()) return Promise.resolve("*(empty table)*"); if (nested) { diff --git a/client/plugos/syscalls/index.ts b/client/plugos/syscalls/index.ts index 8ad7cf6c..4b7a66b0 100644 --- a/client/plugos/syscalls/index.ts +++ b/client/plugos/syscalls/index.ts @@ -20,6 +20,9 @@ export function indexSyscalls( "index.tag": (_ctx, tagName: string): LuaQueryCollection => { return objectIndex.tag(tagName); }, + "index.aggregates": (_ctx): LuaQueryCollection => { + return objectIndex.aggregates(); + }, "index.ensureFullIndex": (_ctx) => { return objectIndex.ensureFullIndex(client.space); }, diff --git a/client/space_lua/aggregates.ts b/client/space_lua/aggregates.ts index 2fbcf9ad..4e3fd5a7 100644 --- a/client/space_lua/aggregates.ts +++ b/client/space_lua/aggregates.ts @@ -1,10 +1,6 @@ /** * Aggregate function definitions and execution for LIQ. * - * Built-in aggregates (sum, count, min, max, avg, array_agg) are - * implemented in TypeScript for speed. Users can override any builtin - * via `aggregate.define` or `aggregate.update`. - * * Builtins implement ILuaFunction via plain objects rather than * LuaBuiltinFunction instances. This avoids ES module TDZ issues: * `class` exports are not available during circular module init, @@ -17,12 +13,15 @@ import { type LuaEnv, LuaTable, luaTruthy, + luaValueToJS, type LuaValue, } from "./runtime.ts"; +import { isSqlNull } from "./liq_null.ts"; import type { LuaExpression } from "./ast.ts"; import { buildItemEnv } from "./query_env.ts"; import { asyncMergeSort } from "./util.ts"; import type { Config } from "../config.ts"; +import YAML from "js-yaml"; export interface AggregateSpec { name: string; @@ -47,52 +46,207 @@ function aggFn( }; } +// Welford's online algorithm (for variance and standard deviation) +interface WelfordState { + n: number; + mean: number; + m2: number; +} + +function welfordInit(): WelfordState { + return { n: 0, mean: 0, m2: 0 }; +} + +function welfordIterate(state: WelfordState, value: any): WelfordState { + if (value === null || value === undefined || isSqlNull(value)) return state; + const x = value as number; + state.n += 1; + const delta = x - state.mean; + state.mean += delta / state.n; + const delta2 = x - state.mean; + state.m2 += delta * delta2; + return state; +} + +interface CovarState extends WelfordState { + meanY: number; + m2y: number; + c: number; // co-moment +} + +function covarInit(): CovarState { + return { n: 0, mean: 0, m2: 0, meanY: 0, m2y: 0, c: 0 }; +} + +function covarIterate(state: CovarState, x: any, y: any): CovarState { + if ( + x === null || + x === undefined || + isSqlNull(x) || + y === null || + y === undefined || + isSqlNull(y) + ) + return state; + state.n += 1; + const dx = (x as number) - state.mean; + state.mean += dx / state.n; + const dy = (y as number) - state.meanY; + state.meanY += dy / state.n; + const dx2 = (x as number) - state.mean; + const dy2 = (y as number) - state.meanY; + state.c += dx * dy2; + state.m2 += dx * dx2; + state.m2y += dy * dy2; + return state; +} + +// Quantile interpolation methods +type QuantileMethod = + | "linear" // percentile_cont + | "lower" // percentile_disc + | "higher" + | "nearest" + | "midpoint"; + +interface QuantileState { + values: number[]; + q: number; + method: QuantileMethod; +} + +// Default method based on aggregate invocation name +const quantileNameDefaults: Record = { + percentile_cont: "linear", + percentile_disc: "lower", +}; + +function quantileFinish(state: QuantileState): number | null { + const { values, q, method } = state; + if (values.length === 0) return null; + const n = values.length; + if (n === 1) return values[0]; + const idx = q * (n - 1); + const lo = Math.floor(idx); + const hi = Math.ceil(idx); + switch (method) { + case "lower": + return values[lo]; + case "higher": + return values[hi]; + case "nearest": + return idx - lo <= 0.5 ? values[lo] : values[hi]; + case "midpoint": + return (values[lo] + values[hi]) / 2; + case "linear": { + if (lo === hi) return values[lo]; + const frac = idx - lo; + return values[lo] + frac * (values[hi] - values[lo]); + } + default: + throw new Error(`quantile: unsupported interpolation method '${method}'`); + } +} + +// Shared spec — branching on `ctx.name` for the default method +function makeQuantileSpec(name: string, description: string): AggregateSpec { + return { + name, + description, + initialize: aggFn((_sf, ctx: any, q: any, method: any) => { + const qVal = q ?? 0.5; + if (typeof qVal !== "number" || qVal < 0 || qVal > 1) { + throw new Error(`${name}: quantile must be between 0 and 1`); + } + const ctxName = ctx instanceof LuaTable ? ctx.rawGet("name") : name; + const m = (method ?? + quantileNameDefaults[ctxName] ?? + "linear") as QuantileMethod; + return { values: [] as number[], q: qVal, method: m } as QuantileState; + }), + iterate: aggFn((_sf, state: any, value: any) => { + if (value === null || value === undefined || isSqlNull(value)) + return state; + state.values.push(value as number); + return state; + }), + finish: aggFn((_sf, state: any) => quantileFinish(state as QuantileState)), + }; +} + // Built-in aggregate specs const builtinAggregates: Record = { - sum: { - name: "sum", - description: "Sum of numeric values", - initialize: aggFn((_sf) => 0), - iterate: aggFn((_sf, state: any, value: any) => { - if (value === null || value === undefined) return state; - return (state as number) + (value as number); - }), - }, + // General purpose count: { name: "count", - description: "Count of values; count() with no argument counts all rows", + description: + "Non-null row count for arguments; total row count without argument", initialize: aggFn((_sf) => 0), iterate: aggFn((_sf, state: any, value: any) => { - if (value === null || value === undefined) return state; + if (value === null || value === undefined || isSqlNull(value)) + return state; return (state as number) + 1; }), }, + sum: { + name: "sum", + description: "Arithmetic sum of all non-null input values", + initialize: aggFn((_sf) => ({ result: 0, hasValue: false })), + iterate: aggFn((_sf, state: any, value: any) => { + if (value === null || value === undefined || isSqlNull(value)) + return state; + state.result += value as number; + state.hasValue = true; + return state; + }), + finish: aggFn((_sf, state: any) => { + return state.hasValue ? state.result : null; + }), + }, + product: { + name: "product", + description: "Product of all non-null input values", + initialize: aggFn((_sf) => ({ result: 1, hasValue: false })), + iterate: aggFn((_sf, state: any, value: any) => { + if (value === null || value === undefined || isSqlNull(value)) + return state; + state.result *= value as number; + state.hasValue = true; + return state; + }), + finish: aggFn((_sf, state: any) => { + return state.hasValue ? state.result : null; + }), + }, min: { name: "min", - description: "Minimum value", + description: "Minimum value among non-null inputs", initialize: aggFn((_sf) => null), iterate: aggFn((_sf, state: any, value: any) => { - if (value === null || value === undefined) return state; + if (value === null || value === undefined || isSqlNull(value)) + return state; if (state === null || value < state) return value; return state; }), }, max: { name: "max", - description: "Maximum value", + description: "Maximum value among non-null inputs", initialize: aggFn((_sf) => null), iterate: aggFn((_sf, state: any, value: any) => { - if (value === null || value === undefined) return state; + if (value === null || value === undefined || isSqlNull(value)) + return state; if (state === null || value > state) return value; return state; }), }, avg: { name: "avg", - description: "Average of numeric values", + description: "Arithmetic mean of all non-null input values", initialize: aggFn((_sf) => ({ sum: 0, count: 0 })), iterate: aggFn((_sf, state: any, value: any) => { - if (value === null || value === undefined) return state; + if (value === null || value === undefined || isSqlNull(value)) + return state; state.sum += value as number; state.count += 1; return state; @@ -102,18 +256,249 @@ const builtinAggregates: Record = { return state.sum / state.count; }), }, + // Collection and format array_agg: { name: "array_agg", - description: "Collect values into an array", + description: "Input values concatenated into an array", initialize: aggFn((_sf) => new LuaTable()), iterate: aggFn((_sf, state: any, value: any) => { (state as LuaTable).rawSetArrayIndex( (state as LuaTable).length + 1, - value, + isSqlNull(value) ? null : value, ); return state; }), }, + string_agg: { + name: "string_agg", + description: + "Concatenated non-null values; argument: delimiter (default: ',')", + initialize: aggFn((_sf, _ctx: any, sep: any) => { + return { sep: sep ?? ",", parts: [] as string[] }; + }), + iterate: aggFn((_sf, state: any, value: any) => { + if (value === null || value === undefined || isSqlNull(value)) + return state; + state.parts.push(String(value)); + return state; + }), + finish: aggFn((_sf, state: any) => { + return state.parts.join(state.sep); + }), + }, + yaml_agg: { + name: "yaml_agg", + description: "Input values aggregated into a YAML string", + initialize: aggFn((_sf) => [] as any[]), + iterate: aggFn((sf, state: any, value: any) => { + if (isSqlNull(value)) { + state.push(null); + } else if (value instanceof LuaTable) { + state.push(luaValueToJS(value, sf)); + } else { + state.push(value); + } + return state; + }), + finish: aggFn((_sf, state: any) => { + return YAML.dump(state, { quotingType: '"', noCompatMode: true }); + }), + }, + json_agg: { + name: "json_agg", + description: "Input values aggregated into a JSON string", + initialize: aggFn((_sf) => [] as any[]), + iterate: aggFn((sf, state: any, value: any) => { + if (isSqlNull(value)) { + state.push(null); + } else if (value instanceof LuaTable) { + state.push(luaValueToJS(value, sf)); + } else { + state.push(value); + } + return state; + }), + finish: aggFn((_sf, state: any) => { + return JSON.stringify(state); + }), + }, + // Bitwise and boolean + bit_and: { + name: "bit_and", + description: "Bitwise AND of all non-null input values", + initialize: aggFn((_sf) => ({ result: ~0, hasValue: false })), + iterate: aggFn((_sf, state: any, value: any) => { + if (value === null || value === undefined || isSqlNull(value)) + return state; + state.result &= value as number; + state.hasValue = true; + return state; + }), + finish: aggFn((_sf, state: any) => { + return state.hasValue ? state.result : null; + }), + }, + bit_or: { + name: "bit_or", + description: "Bitwise OR of all non-null input values", + initialize: aggFn((_sf) => ({ result: 0, hasValue: false })), + iterate: aggFn((_sf, state: any, value: any) => { + if (value === null || value === undefined || isSqlNull(value)) + return state; + state.result |= value as number; + state.hasValue = true; + return state; + }), + finish: aggFn((_sf, state: any) => { + return state.hasValue ? state.result : null; + }), + }, + bit_xor: { + name: "bit_xor", + description: "Bitwise exclusive OR of all non-null input values", + initialize: aggFn((_sf) => ({ result: 0, hasValue: false })), + iterate: aggFn((_sf, state: any, value: any) => { + if (value === null || value === undefined || isSqlNull(value)) + return state; + state.result ^= value as number; + state.hasValue = true; + return state; + }), + finish: aggFn((_sf, state: any) => { + return state.hasValue ? state.result : null; + }), + }, + bool_and: { + name: "bool_and", + description: "True if all non-null inputs are true, otherwise false", + initialize: aggFn((_sf) => ({ result: true, hasValue: false })), + iterate: aggFn((_sf, state: any, value: any) => { + if (value === null || value === undefined || isSqlNull(value)) + return state; + state.result = state.result && !!value; + state.hasValue = true; + return state; + }), + finish: aggFn((_sf, state: any) => { + return state.hasValue ? state.result : null; + }), + }, + bool_or: { + name: "bool_or", + description: "True if at least one non-null input is true, otherwise false", + initialize: aggFn((_sf) => ({ result: false, hasValue: false })), + iterate: aggFn((_sf, state: any, value: any) => { + if (value === null || value === undefined || isSqlNull(value)) + return state; + state.result = state.result || !!value; + state.hasValue = true; + return state; + }), + finish: aggFn((_sf, state: any) => { + return state.hasValue ? state.result : null; + }), + }, + // Statistical + stddev_pop: { + name: "stddev_pop", + description: "Population standard deviation of non-null inputs", + initialize: aggFn((_sf) => welfordInit()), + iterate: aggFn((_sf, state: any, value: any) => + welfordIterate(state, value), + ), + finish: aggFn((_sf, state: any) => { + if (state.n === 0) return null; + return Math.sqrt(state.m2 / state.n); + }), + }, + stddev_samp: { + name: "stddev_samp", + description: "Sample standard deviation of non-null inputs", + initialize: aggFn((_sf) => welfordInit()), + iterate: aggFn((_sf, state: any, value: any) => + welfordIterate(state, value), + ), + finish: aggFn((_sf, state: any) => { + if (state.n < 2) return null; + return Math.sqrt(state.m2 / (state.n - 1)); + }), + }, + var_pop: { + name: "var_pop", + description: + "Population variance (square of population standard deviation)", + initialize: aggFn((_sf) => welfordInit()), + iterate: aggFn((_sf, state: any, value: any) => + welfordIterate(state, value), + ), + finish: aggFn((_sf, state: any) => { + if (state.n === 0) return null; + return state.m2 / state.n; + }), + }, + var_samp: { + name: "var_samp", + description: "Sample variance (square of sample standard deviation)", + initialize: aggFn((_sf) => welfordInit()), + iterate: aggFn((_sf, state: any, value: any) => + welfordIterate(state, value), + ), + finish: aggFn((_sf, state: any) => { + if (state.n < 2) return null; + return state.m2 / (state.n - 1); + }), + }, + covar_pop: { + name: "covar_pop", + description: "Population covariance of non-null input pairs", + initialize: aggFn((_sf) => covarInit()), + iterate: aggFn((_sf, state: any, y: any, _ctx: any, x: any) => + covarIterate(state, x, y), + ), + finish: aggFn((_sf, state: any) => { + if (state.n === 0) return null; + return state.c / state.n; + }), + }, + covar_samp: { + name: "covar_samp", + description: "Sample covariance of non-null input pairs", + initialize: aggFn((_sf) => covarInit()), + iterate: aggFn((_sf, state: any, y: any, _ctx: any, x: any) => + covarIterate(state, x, y), + ), + finish: aggFn((_sf, state: any) => { + if (state.n < 2) return null; + return state.c / (state.n - 1); + }), + }, + corr: { + name: "corr", + description: "Correlation coefficient of non-null input pairs", + initialize: aggFn((_sf) => covarInit()), + iterate: aggFn((_sf, state: any, y: any, _ctx: any, x: any) => + covarIterate(state, x, y), + ), + finish: aggFn((_sf, state: any) => { + if (state.n < 2) return null; + const denom = Math.sqrt(state.m2 * state.m2y); + if (denom === 0) return null; + return state.c / denom; + }), + }, + // Quantile and percentile + quantile: makeQuantileSpec( + "quantile", + "Quantile of ordered set of non-null inputs; arguments: value, quantile (0-1), interpolation ('lower', 'higher', 'nearest', 'midpoint' and default: 'linear')", + ), + percentile_cont: makeQuantileSpec( + "percentile_cont", + "Continuous percentile (linear interpolation) on ordered set of non-null inputs; arguments: value, fraction (0-1)", + ), + percentile_disc: makeQuantileSpec( + "percentile_disc", + "Discrete percentile (nearest lower value) on ordered set of non-null inputs; arguments: value, fraction (0-1)", + ), }; const noCtx = {}; @@ -125,33 +510,67 @@ function buildAggCtx(name: string, config: Config): LuaTable { return ctx; } +/** + * Resolve name through config following alias chains (cycles detected) + */ export function getAggregateSpec( name: string, config?: Config, ): AggregateSpec | null { - if (config) { - const spec: any = config.get(`aggregates.${name}`, null); - if (spec) { - let candidate: AggregateSpec | null = null; - if (spec instanceof LuaTable) { - const init = spec.rawGet("initialize"); - const iter = spec.rawGet("iterate"); - if (init && iter) { - candidate = { - name: spec.rawGet("name") ?? name, - description: spec.rawGet("description"), - initialize: init, - iterate: iter, - finish: spec.rawGet("finish"), - }; - } - } else if (spec.initialize && spec.iterate) { - candidate = spec as AggregateSpec; - } - if (candidate) return candidate; + const visited = new Set(); + let current = name; + + while (config) { + if (visited.has(current)) return null; // cycle + visited.add(current); + + const spec: any = config.get(`aggregates.${current}`, null); + if (!spec) break; + + // Check for alias redirect + const alias = spec instanceof LuaTable ? spec.rawGet("alias") : spec.alias; + if (typeof alias === "string") { + current = alias; + continue; } + + // Full definition in config + let candidate: AggregateSpec | null = null; + if (spec instanceof LuaTable) { + const init = spec.rawGet("initialize"); + const iter = spec.rawGet("iterate"); + if (init && iter) { + candidate = { + name: spec.rawGet("name") ?? current, + description: spec.rawGet("description"), + initialize: init, + iterate: iter, + finish: spec.rawGet("finish"), + }; + } + } else if (spec.initialize && spec.iterate) { + candidate = spec as AggregateSpec; + } + if (candidate) return candidate; + break; } - return builtinAggregates[name] ?? null; + + return builtinAggregates[current] ?? null; +} + +/** + * Returns info about all built-in aggregates + */ +export function getBuiltinAggregateEntries(): { + name: string; + description: string; + hasFinish: boolean; +}[] { + return Object.values(builtinAggregates).map((spec) => ({ + name: spec.name, + description: spec.description ?? "", + hasFinish: !!spec.finish, + })); } /** @@ -176,16 +595,25 @@ export async function executeAggregate( ): Promise { const ctx = buildAggCtx(spec.name, config); - // Evaluate extra args once (before the loop) + // Evaluate extra args using the first item's env so that references + // to the object variable (e.g. `data.x`) resolve correctly. + // These are used for initialize and finish; iterate re-evaluates per-item. const extraArgs: LuaValue[] = []; - for (const argExpr of extraArgExprs) { - extraArgs.push(await evalExprFn(argExpr, env, sf)); + if (extraArgExprs.length > 0) { + const firstItem = items.length > 0 ? items.rawGet(1) : undefined; + const firstEnv = + firstItem !== undefined + ? buildItemEnv(objectVariable, firstItem, env, sf) + : env; + for (const argExpr of extraArgExprs) { + extraArgs.push(await evalExprFn(argExpr, firstEnv, sf)); + } } // Initialize let state = await luaCall(spec.initialize, [ctx, ...extraArgs], noCtx, sf); - // Iterate + // Collect filtered items const filteredItems: LuaValue[] = []; const len = items.length; for (let i = 1; i <= len; i++) { @@ -202,7 +630,10 @@ export async function executeAggregate( filteredItems.push(item); } - // Intra-aggregate ordering + // Intra-aggregate ordering: sorts items before iteration. + // This is required for ordered-set aggregates (quantile, percentile_cont, + // percentile_disc) which expect values in a specific order. The user + // must provide `order by` for these aggregates to produce correct results. if (orderBy && orderBy.length > 0) { await asyncMergeSort(filteredItems, async (a: any, b: any) => { for (const ob of orderBy) { @@ -210,8 +641,8 @@ export async function executeAggregate( const envB = buildItemEnv(objectVariable, b, env, sf); const valA = await evalExprFn(ob.expression, envA, sf); const valB = await evalExprFn(ob.expression, envB, sf); - const aNull = valA === null || valA === undefined; - const bNull = valB === null || valB === undefined; + const aNull = valA === null || valA === undefined || isSqlNull(valA); + const bNull = valB === null || valB === undefined || isSqlNull(valB); if (aNull && bNull) continue; if (aNull) return ob.nulls === "first" ? -1 : 1; if (bNull) return ob.nulls === "first" ? 1 : -1; @@ -228,16 +659,21 @@ export async function executeAggregate( // Iterate for (const item of filteredItems) { + const itemEnv = buildItemEnv(objectVariable, item, env, sf); let value: LuaValue; if (valueExpr === null) { value = item; } else { - const itemEnv = buildItemEnv(objectVariable, item, env, sf); value = await evalExprFn(valueExpr, itemEnv, sf); } + // Evaluate extra args per-item so they can reference item fields + const iterExtraArgs: LuaValue[] = []; + for (const argExpr of extraArgExprs) { + iterExtraArgs.push(await evalExprFn(argExpr, itemEnv, sf)); + } state = await luaCall( spec.iterate, - [state, value, ctx, ...extraArgs], + [state, value, ctx, ...iterExtraArgs], noCtx, sf, ); diff --git a/client/space_lua/liq_null.ts b/client/space_lua/liq_null.ts new file mode 100644 index 00000000..a66a793b --- /dev/null +++ b/client/space_lua/liq_null.ts @@ -0,0 +1,6 @@ +// Sentinel value representing SQL NULL in query results. +export const LIQ_NULL = Symbol.for("silverbullet.sqlNull"); + +export function isSqlNull(v: any): boolean { + return v === LIQ_NULL; +} diff --git a/client/space_lua/query_collection.ts b/client/space_lua/query_collection.ts index 757ae129..96baef34 100644 --- a/client/space_lua/query_collection.ts +++ b/client/space_lua/query_collection.ts @@ -26,6 +26,7 @@ import { type LuaValue, singleResult, } from "./runtime.ts"; +import { isSqlNull, LIQ_NULL } from "./liq_null.ts"; import { evalExpression, luaOp } from "./eval.ts"; import { asyncMergeSort } from "./util.ts"; import type { DataStore } from "../data/datastore.ts"; @@ -38,13 +39,6 @@ import type { KvKey } from "../../plug-api/types/datastore.ts"; import { executeAggregate, getAggregateSpec } from "./aggregates.ts"; import { Config } from "../config.ts"; -// Sentinel value representing SQL NULL in query results. -export const LIQ_NULL = Symbol.for("silverbullet.sqlNull"); - -export function isSqlNull(v: any): boolean { - return v === LIQ_NULL; -} - // Build environment for post-`group by` clauses. Injects `key` and `group` // as top-level variables. Unpacks first group item fields and group-by key // fields as locals so that bare field access works after grouping. @@ -203,56 +197,71 @@ export function toCollection(obj: any): LuaQueryCollection { return new ArrayQueryCollection([obj]); } -function containsAggregate(expr: LuaExpression): boolean { +function containsAggregate(expr: LuaExpression, config?: Config): boolean { switch (expr.type) { case "FilteredCall": { const fc = (expr as LuaFilteredCallExpression).call; - if (fc.prefix.type === "Variable" && getAggregateSpec(fc.prefix.name)) { + if ( + fc.prefix.type === "Variable" && + getAggregateSpec(fc.prefix.name, config) + ) { return true; } return ( - containsAggregate(fc) || - containsAggregate((expr as LuaFilteredCallExpression).filter) + containsAggregate(fc, config) || + containsAggregate((expr as LuaFilteredCallExpression).filter, config) ); } case "AggregateCall": { const ac = expr as LuaAggregateCallExpression; const fc = ac.call; - if (fc.prefix.type === "Variable" && getAggregateSpec(fc.prefix.name)) { + if ( + fc.prefix.type === "Variable" && + getAggregateSpec(fc.prefix.name, config) + ) { return true; } - return containsAggregate(fc); + return containsAggregate(fc, config); } case "FunctionCall": { const fc = expr as LuaFunctionCallExpression; - if (fc.prefix.type === "Variable" && getAggregateSpec(fc.prefix.name)) { + if ( + fc.prefix.type === "Variable" && + getAggregateSpec(fc.prefix.name, config) + ) { return true; } - return fc.args.some(containsAggregate); + return fc.args.some((a) => containsAggregate(a, config)); } case "Binary": { const bin = expr as LuaBinaryExpression; - return containsAggregate(bin.left) || containsAggregate(bin.right); + return ( + containsAggregate(bin.left, config) || + containsAggregate(bin.right, config) + ); } case "Unary": { const un = expr as LuaUnaryExpression; - return containsAggregate(un.argument); + return containsAggregate(un.argument, config); } case "Parenthesized": { const p = expr as LuaParenthesizedExpression; - return containsAggregate(p.expression); + return containsAggregate(p.expression, config); } case "TableConstructor": return expr.fields.some((f) => { switch (f.type) { case "PropField": - return containsAggregate((f as LuaPropField).value); + return containsAggregate((f as LuaPropField).value, config); case "DynamicField": { const df = f as LuaDynamicField; - return containsAggregate(df.key) || containsAggregate(df.value); + return ( + containsAggregate(df.key, config) || + containsAggregate(df.value, config) + ); } case "ExpressionField": - return containsAggregate((f as LuaExpressionField).value); + return containsAggregate((f as LuaExpressionField).value, config); default: return false; } @@ -284,7 +293,7 @@ export async function evalExpressionWithAggregates( outerEnv: LuaEnv, config: Config, ): Promise { - if (!containsAggregate(expr)) { + if (!containsAggregate(expr, config)) { return evalExpression(expr, env, sf); } const recurse = (e: LuaExpression) => @@ -592,8 +601,8 @@ async function sortKeyCompare( const bVal = bKeys[idx]; // Handle nulls positioning - const aIsNull = aVal === null || aVal === undefined; - const bIsNull = bVal === null || bVal === undefined; + const aIsNull = aVal === null || aVal === undefined || isSqlNull(aVal); + const bIsNull = bVal === null || bVal === undefined || isSqlNull(bVal); if (aIsNull || bIsNull) { if (aIsNull && bIsNull) continue; // Default: nulls last for asc, nulls first for desc diff --git a/client/space_lua/query_test.lua b/client/space_lua/query_test.lua index 4b208647..eaa893f4 100644 --- a/client/space_lua/query_test.lua +++ b/client/space_lua/query_test.lua @@ -1787,9 +1787,9 @@ do assertEquals(row.total_size, 20) assertEquals(row.big_size, 15) elseif row.tag == "random" then - -- Fran(1) total=1, big: 0 (none pass) + -- Fran(1) total=1, big: nil (none pass) assertEquals(row.total_size, 1) - assertEquals(row.big_size, 0) + assertEquals(row.big_size, nil) end end end @@ -1906,7 +1906,7 @@ do } ]] assertEquals(r[1].n, 0) - assertEquals(r[1].s, 0) + assertEquals(r[1].s, nil) end -- 53. Multiple filters in one select @@ -3105,4 +3105,4 @@ do ]] assertEquals(#r, 3) assertEquals(r[1].name, "Carol") -end \ No newline at end of file +end diff --git a/client/space_lua/render_lua_html.ts b/client/space_lua/render_lua_html.ts index 82087905..773e7cdf 100644 --- a/client/space_lua/render_lua_html.ts +++ b/client/space_lua/render_lua_html.ts @@ -1,6 +1,6 @@ import { luaFormatNumber, LuaTable } from "../space_lua/runtime.ts"; import { isTaggedFloat } from "../space_lua/numeric.ts"; -import { isSqlNull } from "../space_lua/query_collection.ts"; +import { isSqlNull } from "../space_lua/liq_null.ts"; function escapeHtml(s: string): string { return s diff --git a/client/space_lua/runtime.ts b/client/space_lua/runtime.ts index 4cc5abf3..343cd321 100644 --- a/client/space_lua/runtime.ts +++ b/client/space_lua/runtime.ts @@ -3,6 +3,7 @@ import { evalStatement } from "./eval.ts"; import { asyncQuickSort } from "./util.ts"; import { isPromise, rpAll } from "./rp.ts"; import { isNegativeZero, isTaggedFloat } from "./numeric.ts"; +import { isSqlNull } from "./liq_null.ts"; import { luaFormat } from "./stdlib/format.ts"; export type LuaType = @@ -79,7 +80,7 @@ function isLuaNumber(v: any): boolean { } export function luaTypeName(val: any): LuaType { - if (val === null || val === undefined) { + if (val === null || val === undefined || isSqlNull(val)) { return "nil"; } @@ -1153,6 +1154,7 @@ export function luaIndexValue( if (t instanceof LuaTable) { const raw = t.rawGet(key); if (raw !== undefined) { + if (isSqlNull(raw)) return null; return raw; } // If no metatable, raw miss => nil @@ -1391,7 +1393,7 @@ export function luaKeys(val: any): any[] { } export function luaTypeOf(val: any): LuaType | Promise { - if (val === null || val === undefined) { + if (val === null || val === undefined || isSqlNull(val)) { return "nil"; } if (isPromise(val)) { @@ -1503,7 +1505,7 @@ export function luaToString( value: any, visited: Set = new Set(), ): string | Promise { - if (value === null || value === undefined) { + if (value === null || value === undefined || isSqlNull(value)) { return "nil"; } if (isPromise(value)) { @@ -1635,6 +1637,9 @@ export function getMetatable( } export function jsToLuaValue(value: any): any { + if (value === null || value === undefined) { + return value; + } if (isPromise(value)) { return (value as Promise).then(jsToLuaValue); } diff --git a/client/space_lua/stdlib.ts b/client/space_lua/stdlib.ts index d85f0e36..d182c2f9 100644 --- a/client/space_lua/stdlib.ts +++ b/client/space_lua/stdlib.ts @@ -35,6 +35,7 @@ import { cryptoApi } from "./stdlib/crypto.ts"; import { netApi } from "./stdlib/net.ts"; import { isTaggedFloat, makeLuaFloat } from "./numeric.ts"; import { isPromise } from "./rp.ts"; +import { isSqlNull } from "./liq_null.ts"; const printFunction = new LuaBuiltinFunction(async (_sf, ...args) => { console.log("[Lua]", ...(await Promise.all(args.map((v) => luaToString(v))))); @@ -310,7 +311,7 @@ const rawgetFunction = new LuaBuiltinFunction( if (isLuaTable(table)) { const v = table.rawGet(key); - return v === undefined ? null : v; + return v === undefined || isSqlNull(v) ? null : v; } const k = isTaggedFloat(key) ? key.value : key; diff --git a/client/space_lua/stdlib/space_lua.ts b/client/space_lua/stdlib/space_lua.ts index 63853cc4..678952ea 100644 --- a/client/space_lua/stdlib/space_lua.ts +++ b/client/space_lua/stdlib/space_lua.ts @@ -11,6 +11,7 @@ import { luaValueToJS, singleResult, } from "../runtime.ts"; +import { isSqlNull } from "../liq_null.ts"; /** * These are Space Lua specific functions that are available to all scripts, but are not part of the standard Lua language. @@ -31,7 +32,8 @@ function createAugmentedEnv( if (envAugmentation) { env.setLocal("_", envAugmentation); for (const key of envAugmentation.keys()) { - env.setLocal(key, envAugmentation.rawGet(key)); + const v = envAugmentation.rawGet(key); + env.setLocal(key, isSqlNull(v) ? null : v); } } return env; diff --git a/libraries/Library/Std/APIs/Aggregate.md b/libraries/Library/Std/APIs/Aggregate.md index 08c45fa6..7d057b3a 100644 --- a/libraries/Library/Std/APIs/Aggregate.md +++ b/libraries/Library/Std/APIs/Aggregate.md @@ -5,7 +5,27 @@ tags: meta/api APIs to define and override aggregate functions used in LIQ `select` and `having` clauses after `group by`. -Built-in aggregates: `count`, `sum`, `min`, `max`, `avg` and `array_agg`. +All aggregates skip null/nil values by convention. Empty groups return null (except `count` which returns 0 and `string_agg` which returns an empty string). + +# Querying available aggregates + +All aggregates (built-in, user-defined, and aliases) are queryable via `index.aggregates()`: + +```lua +-- List all +${query[[from index.aggregates()]]} + +-- Only builtins +${query[[from index.aggregates() where builtin]]} + +-- Only aliases +${query[[from index.aggregates() where target]]} + +-- Only user-defined (non-builtin, non-alias) +${query[[from index.aggregates() where not builtin and not target]]} +``` + +Each row includes the following columns: `builtin`, `name`, `description`, `initialize`, `iterate`, `finish`, and `target`. The `initialize`, `iterate`, and `finish` columns are represented by boolean values. # API @@ -30,15 +50,24 @@ Aggregate functions can accept additional arguments beyond the first value expre * `iterate(state, value, ctx, ...extraArgs)` — receives extra args after the context table * `finish(state, ctx, ...extraArgs)` — receives extra args after the context table -This allows parameterized aggregates, for example a separator argument for string concatenation. +This allows parameterized aggregates, for example a separator argument for string concatenation or boundary arguments for clamped sums. ## aggregate.update(spec) Updates an existing aggregate definition. Same keys as `aggregate.define`. Only the provided keys are overwritten. +## aggregate.alias(name, target, description?) + +Creates an alias so that `name` resolves to `target` at query time. The target may be a builtin, a user-defined aggregate, or another alias (chains are followed with cycle detection). + +```lua +aggregate.alias("total", "sum") +aggregate.alias("stdev", "stddev_pop", "My stddev alias") +``` + # Examples -## Define a custom aggregate +## Define a custom aggregate with one extra argument Define a custom aggregate `concat` that concatenates strings with a configurable separator (defaulting to `", "`): @@ -50,7 +79,7 @@ aggregate.define { return { sep = sep or ', ', parts = {} } end, - iterate = function(state, value, ctx, sep) + iterate = function(state, value) if value ~= nil then state.parts[#state.parts + 1] = tostring(value) end @@ -77,6 +106,55 @@ query [[ ]] ``` +## Define a custom aggregate with two extra arguments + +Define a custom aggregate `clamp_sum` that sums non-null inputs and clamps the result to a `[min, max]` range: + +```lua +aggregate.define { + name = 'clamp_sum', + description = 'Sum of non-null inputs clamped to [min, max]', + + initialize = function(ctx, lo, hi) + return { total = 0, lo = lo or -math.huge, hi = hi or math.huge } + end, + + iterate = function(state, value) + if value ~= nil then + state.total = state.total + value + end + return state + end, + + finish = function(state) + if state.total < state.lo then return state.lo end + if state.total > state.hi then return state.hi end + return state.total + end, +} +``` + +Usage in a query: + +```lua +query [[ + from + d = { + { dept = "eng", hours = 12 }, + { dept = "eng", hours = 35 }, + { dept = "sales", hours = 8 }, + { dept = "sales", hours = 6 }, + } + group by d.dept + select { + dept = d.dept, + total = clamp_sum(d.hours, 0, 40), + } +]] +``` + +Here `eng` sums to 47 but is clamped to `40`, while `sales` sums to `14` which is within range. + ## Update an existing aggregate ```lua @@ -90,6 +168,13 @@ aggregate.update { } ``` +## Create an alias + +```lua +aggregate.alias("total", "sum") +aggregate.alias("stdev", "stddev_pop", "Shorthand for population stddev") +``` + # Implementation ```space-lua @@ -121,7 +206,7 @@ function aggregate.define(spec) error('aggregate.define: ' .. validationResult) end - config.set({'aggregates', spec.name}, spec) + config.setLuaValue({'aggregates', spec.name}, spec) end function aggregate.update(spec) @@ -145,6 +230,26 @@ function aggregate.update(spec) .. spec.name .. ' has no iterate after merge') end - config.set({'aggregates', spec.name}, existing) + config.setLuaValue({'aggregates', spec.name}, existing) end + +function aggregate.alias(name, target, description) + if not name or not target then + error('aggregate.alias: both name and target are required') + end + if name == target then + error('aggregate.alias: name and target must differ') + end + local entry = { alias = target } + if description then + entry.description = description + end + config.setLuaValue({'aggregates', name}, entry) +end + +-- Standard aliases +aggregate.alias('every', 'bool_and') +aggregate.alias('std', 'stddev_pop') +aggregate.alias('stddev', 'stddev_pop') +aggregate.alias('variance', 'var_pop') ``` diff --git a/website/Space Lua/Lua Integrated Query/Aggregating.md b/website/Space Lua/Lua Integrated Query/Aggregating.md index 2f79584d..2d5da784 100644 --- a/website/Space Lua/Lua Integrated Query/Aggregating.md +++ b/website/Space Lua/Lua Integrated Query/Aggregating.md @@ -14,6 +14,28 @@ Field names used in `group by` are exposed as locals in `having`, `select`, and > **note** Note > The `having` clause acts only on grouped output. For filtering individual items, use `where` prior to grouping. +# Available aggregates + +All registered aggregate functions — built-in, user-defined, and aliases — can be listed via `index.aggregates()`: + +${query[[ + select + { + Name = '`' .. name .. '`', + Description = description, + Kind = + (builtin and 'builtin' or 'custom') .. + (target and ' alias for ' .. '`' .. target .. '`' or ''), + } + from + index.aggregates() + order by + builtin desc, + name +]]} + +See [[Library/Std/APIs/Aggregate|Aggregate API]] for how to define custom aggregates and aliases. + # Examples All example queries operate on `tags.page`, but will work with any query collection. As always, to see the underlying query, hover over the result table and click the _Edit_ button to see the underlying query. @@ -101,13 +123,15 @@ ${query [[ tag ]]} -The filter clause works with all aggregate functions: `count`, `sum`, `min`, `max`, `avg`, `array_agg`, and custom aggregates. When no rows match the filter condition, aggregates return their identity value: `0` for `count` and `sum`, `nil` for `min`, `max`, and `avg`, and an empty table `{}` for `array_agg`. +The filter clause works with all aggregate functions: `count`, `sum`, `min`, `max`, `avg`, `array_agg`, and custom aggregates. When no rows match the filter condition, aggregates return their empty-group value: `0` for `count`, `nil` for `sum`, `min`, `max`, and `avg`, and an empty table `{}` for `array_agg`. ## Intra-aggregate `order by` Aggregate functions can include an `order by` clause **inside** the function call to control the order in which values are processed. For commutative aggregates like `sum`, `count`, `min`, `max`, and `avg`, the intra-aggregate `order by` has no effect on the result because the value is the same regardless of iteration order. It is only meaningful for order-dependent aggregates like `array_agg`. +Ordered-set aggregates such as `quantile`, `percentile_cont`, and `percentile_disc` require an intra-aggregate `order by` clause to produce correct results, as they depend on the iteration order of input values. Without `order by`, results are undefined. + ### Basic example Collect page names sorted alphabetically within each group: @@ -185,4 +209,4 @@ Custom aggregator functions may be defined by the user using [[Library/Std/APIs/ # See also * [[Space Lua/Lua Integrated Query/Grouping]] — grouping queries without aggregation -* [[Space Lua/Lua Integrated Query]] — full LIQ language reference and listing available aggregators +* [[Space Lua/Lua Integrated Query]] — full LIQ language reference and listing available aggregates diff --git a/website/Space Lua/Lua Integrated Query/Grouping.md b/website/Space Lua/Lua Integrated Query/Grouping.md index 6f238729..f88e580c 100644 --- a/website/Space Lua/Lua Integrated Query/Grouping.md +++ b/website/Space Lua/Lua Integrated Query/Grouping.md @@ -10,7 +10,7 @@ After `group by`, each result row has two fields: The field names used in `group by` are also available as bare variables in `having`, `select`, and `order by`. Use `#group` to count items per group. > **note** Note -> `having` can only reference group key fields, `key`, `group`, and aggregates like `#group`. To filter individual rows, use `where`. +> `having` can only reference group key fields, `key`, `group`, aggregate expressions like `#group`, and aggregate functions like `count()`. To filter individual rows, use `where`. # Examples