* [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry * Extend with 13 new built-in aggregates: `product`, `string_agg`, `yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`, `bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`. * Introduce `aggregate.alias` API allowing users to define custom aliases for any aggregate. Standard aliases (`every`, `std`, `stddev` and `variance`) are now defined via this API rather than hardcoded. * Add `index.aggregates` queryable collection so users can discover all available aggregates directly from LIQ queries. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix config pass through query path so custom aggregates work Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage `config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on all arguments. This converted the aggregate `LuaTable` to a plain JS object and wrapped `LuaFunction` callbacks in JS functions that also converted their returned values via `luaValueToJS`. The result was that state returned by initialize (a `LuaTable`) got converted to a plain JS object before being passed to `iterate`. Therefor Lua operations like `table.insert` on that were failing because they expected a `LuaTable` and not a plain JS array. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix formatting Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Improve aggregate functions descriptions, fix `sum` divergence Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Align `product` with `sum` Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Rename `alias` in `aggregates()` to `target` for clarity Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add a null guard at the top of `jsToLuaValue` This preserves `null`/`undefined` as-is (both map to Lua nil) and prevents them from falling through to the `typeof` "object" branch. For this PR it means that null `target` in our `aggregates` entries will correctly show as empty/`nil` in query results rather than `{}`. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Documentation reflects recent changes Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks * `sum(`) and` product(`) now return null when no rows match (matching Postgres semantics) instead of returning 0 and 1 respectively. * Query result columns that hold null are internally preserved using a `LIQ_NULL` sentinel so that column keys survive in `LuaTable` storage. This sentinel was leaking into Lua code as "userdata" through three read paths: * `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when accessing table fields, * `rawget` (stdlib): the builtin `rawget` function exposed the sentinel without converting it back to `nil`, * `createAugmentedEnv`: string interpolation unpacked table values via `rawGet` into local variables, making the sentinel visible in template expressions like `${var}`. All three now convert `LIQ_NULL` to `nil` at the read boundary, keeping the sentinel internal to table storage where it belongs. * Update affected test expectations accordingly. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: `array_agg` preserves NULL positions Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add symbol guard to `json_agg` `JSON.stringify(Symbol(...))` in an array produces null by accident. That is a JS implementation detail we **MUST NOT** rely on. Explicit null push makes intent clear and avoids surprises if the `Symbol` representation ever changes. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add symbol guard to `yaml_agg` (ditto) `js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes it emit YAML null (or `~`), which is the correct YAML representation of a missing value and matches standard `json_agg`/`yaml_agg` NULL-inclusion semantics. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add intra-aggregate ordering null guards Without this, `LIQ_NULL` sort keys would fall through to `valA < valB` which is always false for `Symbol`s which is breaking the `nulls first`/`nulls last` contract... Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Ditto, but for `order by` null comparisons Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated against the outer query environment where the object variable is not bound. This caused multi-argument aggregates like `covar_samp(data.y, data.x)` to fail with nil reference errors. This commit addresses this by evaluating extra args per-item inside the iterate loop using the item environment so all arguments resolve correctly. We also add few common aggregates: - `covar_pop`, `covar_samp`, `corr`: population/sample covariance and correlation coefficient using online co-moment algorithm. - `quantile(value, q, method)`: general quantile with interpolation methods: lower, higher, nearest, midpoint and default linear. - `percentile_cont(value, q)`: continuous percentile (linear) - `percentile_disc(value, q)`: discrete percentile (lower) Note: `percentile_cont` and `percentile_disc` share the `quantile` implementation through `ctx.name` at initialize time. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Update docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Make the ordering for quantile aggregates explicit Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Update docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Improve docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> --------- Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
340 lines
9.2 KiB
TypeScript
340 lines
9.2 KiB
TypeScript
import { luaFormatNumber, LuaTable } from "../space_lua/runtime.ts";
|
|
import { isTaggedFloat } from "../space_lua/numeric.ts";
|
|
import { isSqlNull } from "../space_lua/liq_null.ts";
|
|
|
|
function escapeHtml(s: string): string {
|
|
return s
|
|
.replace(/&/g, "&")
|
|
.replace(/</g, "<")
|
|
.replace(/>/g, ">")
|
|
.replace(/"/g, """);
|
|
}
|
|
|
|
// Render inline Markdown to HTML (caller provides the implementation)
|
|
export type InlineRenderer = (text: string) => string;
|
|
|
|
function defaultInlineRenderer(text: string): string {
|
|
return escapeHtml(text);
|
|
}
|
|
|
|
function luaTypeName(
|
|
v: any,
|
|
): "number" | "string" | "boolean" | "table" | "array" | undefined {
|
|
if (v === undefined || v === null || isSqlNull(v)) return undefined;
|
|
if (typeof v === "number" || isTaggedFloat(v)) return "number";
|
|
if (typeof v === "string") return "string";
|
|
if (typeof v === "boolean") return "boolean";
|
|
if (v instanceof LuaTable) {
|
|
return hasStringKeys(v) ? "table" : "array";
|
|
}
|
|
if (Array.isArray(v)) return "array";
|
|
if (isPlainObject(v)) return "table";
|
|
return "string";
|
|
}
|
|
|
|
function isEmpty(v: any): boolean {
|
|
return v === undefined || v === null || isSqlNull(v);
|
|
}
|
|
|
|
function isPlainObject(v: any): v is Record<string, any> {
|
|
return (
|
|
typeof v === "object" &&
|
|
v !== null &&
|
|
!Array.isArray(v) &&
|
|
v.constructor === Object
|
|
);
|
|
}
|
|
|
|
function hasStringKeys(tbl: LuaTable): boolean {
|
|
for (const k of tbl.keys()) {
|
|
if (typeof k === "string") return true;
|
|
}
|
|
return false;
|
|
}
|
|
|
|
function formatScalar(v: any): string {
|
|
if (isEmpty(v)) return "";
|
|
if (isTaggedFloat(v)) return luaFormatNumber(v.value, "float");
|
|
if (typeof v === "number") return luaFormatNumber(v);
|
|
if (typeof v === "boolean") return v ? "true" : "false";
|
|
return `${v}`;
|
|
}
|
|
|
|
/**
|
|
* Render any Lua/JS value to an HTML string with data attributes.
|
|
* Also sets data-type and data-empty on the outermost element when
|
|
* used for widget display.
|
|
*/
|
|
export async function renderResultToHtml(
|
|
result: any,
|
|
renderInline: InlineRenderer = defaultInlineRenderer,
|
|
): Promise<{
|
|
html: string;
|
|
dataType: string;
|
|
}> {
|
|
if (isEmpty(result)) {
|
|
return {
|
|
html: `<span data-empty></span>`,
|
|
dataType: "nil",
|
|
};
|
|
}
|
|
if (typeof result === "string") {
|
|
return {
|
|
html: renderInline(result),
|
|
dataType: "string",
|
|
};
|
|
}
|
|
if (isTaggedFloat(result)) {
|
|
return {
|
|
html: escapeHtml(luaFormatNumber(result.value, "float")),
|
|
dataType: "number",
|
|
};
|
|
}
|
|
if (typeof result === "number") {
|
|
return {
|
|
html: escapeHtml(luaFormatNumber(result)),
|
|
dataType: "number",
|
|
};
|
|
}
|
|
if (typeof result === "boolean") {
|
|
return {
|
|
html: result ? "true" : "false",
|
|
dataType: "boolean",
|
|
};
|
|
}
|
|
if (result instanceof LuaTable) {
|
|
if (result.empty()) {
|
|
return {
|
|
html: `<table data-table-empty></table>`,
|
|
dataType: "table",
|
|
};
|
|
}
|
|
const html = await renderLuaTableToHtml(result, renderInline);
|
|
const type = hasStringKeys(result) ? "table" : "list";
|
|
return { html, dataType: type };
|
|
}
|
|
if (Array.isArray(result)) {
|
|
if (result.length === 0) {
|
|
return {
|
|
html: `<table data-table-empty></table>`,
|
|
dataType: "table",
|
|
};
|
|
}
|
|
if (result.every(isPlainObject)) {
|
|
return {
|
|
html: await renderJsObjectArrayToHtml(result, renderInline),
|
|
dataType: "table",
|
|
};
|
|
}
|
|
return {
|
|
html: await renderJsArrayToHtml(result, renderInline),
|
|
dataType: "list",
|
|
};
|
|
}
|
|
if (isPlainObject(result)) {
|
|
if (Object.keys(result).length === 0) {
|
|
return {
|
|
html: `<table data-table-empty></table>`,
|
|
dataType: "table",
|
|
};
|
|
}
|
|
return {
|
|
html: await renderJsObjectArrayToHtml([result], renderInline),
|
|
dataType: "table",
|
|
};
|
|
}
|
|
return {
|
|
html: renderInline(`${result}`),
|
|
dataType: "string",
|
|
};
|
|
}
|
|
|
|
async function renderLuaTableToHtml(
|
|
tbl: LuaTable,
|
|
renderInline: InlineRenderer,
|
|
): Promise<string> {
|
|
const keys = tbl.keys();
|
|
if (keys.length === 0) return `<table data-table-empty></table>`;
|
|
|
|
const arrayLen = tbl.length;
|
|
const stringKeys: string[] = [];
|
|
for (const k of keys) {
|
|
if (typeof k === "string") stringKeys.push(k);
|
|
}
|
|
|
|
const hasArrayPart = arrayLen > 0;
|
|
const hasStrKeys = stringKeys.length > 0;
|
|
|
|
// Pure array (render as `<ul>` or multi-row `<table>`)
|
|
if (hasArrayPart && !hasStrKeys) {
|
|
const elements: any[] = [];
|
|
for (let i = 1; i <= arrayLen; i++) elements.push(tbl.rawGet(i));
|
|
// If all elements are `LuaTables` then render as multi-row table
|
|
if (elements.every((el) => el instanceof LuaTable)) {
|
|
return renderLuaTableArrayToHtml(elements as LuaTable[], renderInline);
|
|
}
|
|
return renderArrayToHtml(elements, renderInline);
|
|
}
|
|
|
|
// Has string keys (record or mixed) - use `keys` order directly
|
|
const lines: string[] = [];
|
|
lines.push("<table>");
|
|
lines.push("<thead><tr>");
|
|
for (const k of keys) lines.push(`<th>${escapeHtml(String(k))}</th>`);
|
|
lines.push("</tr></thead>");
|
|
lines.push("<tbody><tr>");
|
|
for (const k of keys) {
|
|
const v = tbl.rawGet(k);
|
|
lines.push(await renderTdHtml(v, renderInline));
|
|
}
|
|
lines.push("</tr></tbody>");
|
|
lines.push("</table>");
|
|
return lines.join("");
|
|
}
|
|
|
|
// Array of record-like LuaTables to multi-row `<table>`
|
|
async function renderLuaTableArrayToHtml(
|
|
tables: LuaTable[],
|
|
renderInline: InlineRenderer,
|
|
): Promise<string> {
|
|
const headerSet = new Set<string>();
|
|
for (const tbl of tables) {
|
|
for (const k of tbl.keys()) headerSet.add(String(k));
|
|
}
|
|
if (headerSet.size === 0) return `<table data-table-empty></table>`;
|
|
const headers = [...headerSet];
|
|
|
|
const lines: string[] = [];
|
|
lines.push("<table>");
|
|
lines.push("<thead><tr>");
|
|
for (const h of headers) lines.push(`<th>${escapeHtml(h)}</th>`);
|
|
lines.push("</tr></thead>");
|
|
lines.push("<tbody>");
|
|
for (const tbl of tables) {
|
|
lines.push("<tr>");
|
|
for (const h of headers) {
|
|
const key = /^\d+$/.test(h) ? Number(h) : h;
|
|
const v = tbl.rawGet(key);
|
|
lines.push(await renderTdHtml(v, renderInline));
|
|
}
|
|
lines.push("</tr>");
|
|
}
|
|
lines.push("</tbody>");
|
|
lines.push("</table>");
|
|
return lines.join("");
|
|
}
|
|
|
|
async function renderJsObjectArrayToHtml(
|
|
jsonArray: Record<string, any>[],
|
|
renderInline: InlineRenderer,
|
|
): Promise<string> {
|
|
const headerSet = new Set<string>();
|
|
for (const entry of jsonArray) {
|
|
for (const k of Object.keys(entry)) headerSet.add(k);
|
|
}
|
|
if (headerSet.size === 0) return `<table data-table-empty></table>`;
|
|
const headers = [...headerSet];
|
|
|
|
const lines: string[] = [];
|
|
lines.push("<table>");
|
|
lines.push("<thead><tr>");
|
|
for (const h of headers) lines.push(`<th>${escapeHtml(h)}</th>`);
|
|
lines.push("</tr></thead>");
|
|
lines.push("<tbody>");
|
|
for (const row of jsonArray) {
|
|
lines.push("<tr>");
|
|
for (const h of headers) {
|
|
lines.push(await renderTdHtml(row[h], renderInline));
|
|
}
|
|
lines.push("</tr>");
|
|
}
|
|
lines.push("</tbody>");
|
|
lines.push("</table>");
|
|
return lines.join("");
|
|
}
|
|
|
|
async function renderArrayToHtml(
|
|
items: any[],
|
|
renderInline: InlineRenderer,
|
|
): Promise<string> {
|
|
const lines: string[] = [];
|
|
lines.push("<ul>");
|
|
for (const item of items) {
|
|
lines.push(await renderLiHtml(item, renderInline));
|
|
}
|
|
lines.push("</ul>");
|
|
return lines.join("");
|
|
}
|
|
|
|
function renderJsArrayToHtml(
|
|
items: any[],
|
|
renderInline: InlineRenderer,
|
|
): Promise<string> {
|
|
return renderArrayToHtml(items, renderInline);
|
|
}
|
|
|
|
async function renderTdHtml(
|
|
v: any,
|
|
renderInline: InlineRenderer,
|
|
): Promise<string> {
|
|
const type = luaTypeName(v);
|
|
const attrs: string[] = [];
|
|
if (type) {
|
|
attrs.push(`data-table-cell-type="${type}"`);
|
|
}
|
|
if (isEmpty(v)) {
|
|
attrs.push("data-table-cell-empty");
|
|
}
|
|
const attrStr = attrs.length > 0 ? ` ${attrs.join(" ")}` : "";
|
|
const content = await renderCellContent(v, renderInline);
|
|
return `<td${attrStr}>${content}</td>`;
|
|
}
|
|
|
|
async function renderLiHtml(
|
|
v: any,
|
|
renderInline: InlineRenderer,
|
|
): Promise<string> {
|
|
const type = luaTypeName(v);
|
|
const attrs: string[] = [];
|
|
if (type) {
|
|
attrs.push(`data-list-item-type="${type}"`);
|
|
}
|
|
if (isEmpty(v)) {
|
|
attrs.push("data-list-item-empty");
|
|
}
|
|
const attrStr = attrs.length > 0 ? ` ${attrs.join(" ")}` : "";
|
|
const content = await renderCellContent(v, renderInline);
|
|
return `<li${attrStr}>${content}</li>`;
|
|
}
|
|
|
|
// Render a value content (recurse for nested tables/arrays)
|
|
function renderCellContent(
|
|
v: any,
|
|
renderInline: InlineRenderer,
|
|
): Promise<string> {
|
|
if (isEmpty(v)) return Promise.resolve("");
|
|
if (isTaggedFloat(v)) {
|
|
return Promise.resolve(renderInline(luaFormatNumber(v.value, "float")));
|
|
}
|
|
if (v instanceof LuaTable) {
|
|
if (v.empty()) return Promise.resolve(`<table data-table-empty></table>`);
|
|
return renderLuaTableToHtml(v, renderInline);
|
|
}
|
|
if (Array.isArray(v)) {
|
|
if (v.length === 0) {
|
|
return Promise.resolve(`<table data-table-empty></table>`);
|
|
}
|
|
if (v.every(isPlainObject)) {
|
|
return renderJsObjectArrayToHtml(v, renderInline);
|
|
}
|
|
return renderArrayToHtml(v, renderInline);
|
|
}
|
|
if (isPlainObject(v)) {
|
|
if (Object.keys(v).length === 0) {
|
|
return Promise.resolve(`<table data-table-empty></table>`);
|
|
}
|
|
return renderJsObjectArrayToHtml([v], renderInline);
|
|
}
|
|
return Promise.resolve(renderInline(formatScalar(v)));
|
|
}
|