02ee8c79421bfbceb2bda138841e6a489d5281c0
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zef Hemel
|
2020fbb16d | refactor: flatten documented callback metadata | ||
|
Zef Hemel
|
37d6c59555 | Docs in code: unify documented function definitions | ||
|
Zef Hemel
|
23f338dc14 | Runtime-expose API documentation (base) | ||
|
Zef Hemel
|
a4f738815f | Lua: add AST pretty-printer and align parse/print naming | ||
|
Zef Hemel
|
8b241f9448 | Rebranding LIQ to SLIQ! | ||
|
Zef Hemel
|
75d992c10e | Support JS objects for spacelua.interpolate | ||
|
Matouš Jan FialkaandGitHub
|
6cbb61c7db |
[LIQ] Add new aggregate functions, aliases, and queryable aggregate registry (#1891)
* [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry * Extend with 13 new built-in aggregates: `product`, `string_agg`, `yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`, `bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`. * Introduce `aggregate.alias` API allowing users to define custom aliases for any aggregate. Standard aliases (`every`, `std`, `stddev` and `variance`) are now defined via this API rather than hardcoded. * Add `index.aggregates` queryable collection so users can discover all available aggregates directly from LIQ queries. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix config pass through query path so custom aggregates work Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage `config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on all arguments. This converted the aggregate `LuaTable` to a plain JS object and wrapped `LuaFunction` callbacks in JS functions that also converted their returned values via `luaValueToJS`. The result was that state returned by initialize (a `LuaTable`) got converted to a plain JS object before being passed to `iterate`. Therefor Lua operations like `table.insert` on that were failing because they expected a `LuaTable` and not a plain JS array. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix formatting Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Improve aggregate functions descriptions, fix `sum` divergence Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Align `product` with `sum` Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Rename `alias` in `aggregates()` to `target` for clarity Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add a null guard at the top of `jsToLuaValue` This preserves `null`/`undefined` as-is (both map to Lua nil) and prevents them from falling through to the `typeof` "object" branch. For this PR it means that null `target` in our `aggregates` entries will correctly show as empty/`nil` in query results rather than `{}`. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Documentation reflects recent changes Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks * `sum(`) and` product(`) now return null when no rows match (matching Postgres semantics) instead of returning 0 and 1 respectively. * Query result columns that hold null are internally preserved using a `LIQ_NULL` sentinel so that column keys survive in `LuaTable` storage. This sentinel was leaking into Lua code as "userdata" through three read paths: * `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when accessing table fields, * `rawget` (stdlib): the builtin `rawget` function exposed the sentinel without converting it back to `nil`, * `createAugmentedEnv`: string interpolation unpacked table values via `rawGet` into local variables, making the sentinel visible in template expressions like `${var}`. All three now convert `LIQ_NULL` to `nil` at the read boundary, keeping the sentinel internal to table storage where it belongs. * Update affected test expectations accordingly. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: `array_agg` preserves NULL positions Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add symbol guard to `json_agg` `JSON.stringify(Symbol(...))` in an array produces null by accident. That is a JS implementation detail we **MUST NOT** rely on. Explicit null push makes intent clear and avoids surprises if the `Symbol` representation ever changes. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add symbol guard to `yaml_agg` (ditto) `js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes it emit YAML null (or `~`), which is the correct YAML representation of a missing value and matches standard `json_agg`/`yaml_agg` NULL-inclusion semantics. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add intra-aggregate ordering null guards Without this, `LIQ_NULL` sort keys would fall through to `valA < valB` which is always false for `Symbol`s which is breaking the `nulls first`/`nulls last` contract... Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Ditto, but for `order by` null comparisons Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated against the outer query environment where the object variable is not bound. This caused multi-argument aggregates like `covar_samp(data.y, data.x)` to fail with nil reference errors. This commit addresses this by evaluating extra args per-item inside the iterate loop using the item environment so all arguments resolve correctly. We also add few common aggregates: - `covar_pop`, `covar_samp`, `corr`: population/sample covariance and correlation coefficient using online co-moment algorithm. - `quantile(value, q, method)`: general quantile with interpolation methods: lower, higher, nearest, midpoint and default linear. - `percentile_cont(value, q)`: continuous percentile (linear) - `percentile_disc(value, q)`: discrete percentile (lower) Note: `percentile_cont` and `percentile_disc` share the `quantile` implementation through `ctx.name` at initialize time. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Update docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Make the ordering for quantile aggregates explicit Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Update docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Improve docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> --------- Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> |
||
|
Zef Hemel
|
799e27a9b4 | Enable biome based formatter and reformat | ||
|
Zef Hemel
|
380b5d90ca | More Deno cleanup | ||
|
|
75cc79800f |
Space Lua: Align numeric/table semantics with Lua, align number formating, optimize loops allocations (#1823)
* Don't know what's going on with the Deno docker image, disabling the deno.lock file for now * Another try * Space Lua: Align numeric and table semantics with Lua This change improves Space Lua compatibility with standard Lua 5.4, focusing on numeric subtypes and table behavior. The test suite is extended to lock in the expected semantics and should pass under both Space Lua and a Lua interpreter. SUMMARY OF CHANGES ------------------ Tighten Lua compatibility across evaluator and runtime: - correct metamethod dispatch (`__index`, `__newindex`, `__call`, comparison metamethods), - loop limits, - raw metamethod lookups. Rework numeric semantics to preserve Lua **integer** vs **float** behavior: - $0$ vs $0.0$ and $-0.0$, - explicit zero kind representation, and - updates arithmetic/bitwise coercions accordingly. Improve parser correctness by rejecting **unary plus** with aligned Lua errors and better parsing errors reporting. Fix `stdlib` behavior to match that of Lua: - `table` function `concat`, `insert`, `remove`, `sort` and `unpack` gain metamethod awareness and enforce Lua errors, - `ipairs` iteration updated to stop on first nil and honor `__index`, - `tonumber` updated to Luae conversion using `luaToNumberDetailed`, - `math.modf` return corrected, - `math.type` accuracy improvements for float/integer and $-0.0$, - `math.pi` added. Fix numeric subtypes for `/` and `^` operators so `math.type` matches Lua results using tagging as well as unary `-`. Expand test coverage: - new `metamethods_test.lua` for Lua metamethod/operator semantics, - extend arithmetic and length tests for zero-kind propagation and `rawlen` vs `__len` metamethod, - Extend `math` test suite to test proper Lua alignment (`math.type` and more), and - update context error expectations for Lua error messages. RATIONALE --------- Lua differs from JavaScript by having two numeric subtypes: **integer** and **float**. Operators depend on the subtype: `+`, `-`, `*`, `//` and `%` use integer mode when both operands are integers and float mode otherwise. Bitwise operators require integers and `math.type(x)` reports "integer" or "float". JavaScript has one numeric primitive type (`number`) so a plain number value cannot record whether Lua considers a value to be a float when the value has no fractional part (for example $2.0$). Lua also differs from "everything is IEEE 754 double" because the rules are defined in terms of integer and float subtypes. Float operations preserve IEEE 754 behavior including `NaN`, infinities and signed zero ($-0.0$) which affects results like $1/0.0$ versus $1/-0.0$. Integer arithmetic does not preserve $-0$ and collapses it to $0$. Lua integer arithmetic is exact within its integer range while JavaScript `number` cannot exactly represent all integers in that range. Lua numbers are integers or floats. Numeric strings coerce to integer or float based on _lexical_ form. Each arithmetic operator selects the result subtype from the operator rules and operand subtypes. Integer only operators (bitwise and `//` as integer division) require integer representability. Mixed arithmetic promotes to float as needed. Two operators are **always float** typed: division (`/`) and exponentiation (`^`) produce floats even if both operands are integers and even if the numeric value has no fractional part. `math.type` reports that internal subtype. Tables are associative arrays and assigning `nil` removes a key. The length operator `#` uses `__len` metamethod if present otherwise it uses the raw length rule. `rawlen(table)` ignores `__len`. Without `__len` Lua defines `#` as some boundary `N` such that `table[N]` is not `nil` and `table[N+1]` is `nil`. If the table has holes (missing or `nil` entries in the positive integer key sequence) the boundary may be non unique so `#` is stable only for proper sequences without holes. PERFORMANCE NOTES ----------------- Numeric changes add small checks to preserve Lua integer and float subtype semantics and avoid allocations except when the subtype would otherwise be lost. Some table operations may be slower due to stricter Lua 5.4 behavior especially around length and sequence boundary handling which currently requires extra metadata tracking and scans and cannot be avoided without a completely different internal table representation. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Another try * Replace `LuaFloatTag` plain objects with boxed Number for float tagging * Restore pre-merge eval/numeric architecture and fix regressions This commit restores the original branch architecture. On top of the restored foundation, float-typed integer results (e.g. `1.0 + 1.0` = `2.0`, `0.0 // 1.0` = `0.0`) are now correctly tagged via `makeLuaFloat` so that `tostring` and `math.type` report them as floats. The *unary minus* fast path for float literals and the `tonumber` function also preserve float tagging. Performance regressions from the merge are addressed by avoiding `Number` boxing for non-integer floats (`3.14` needs no tag — it is unambiguously float), adding *string key* fast paths in `LuaTable` `has`/`rawGet`/`rawSet` to skip numeric normalization for the dominant case, and inlining a `typeof` check in math standard library functions to avoid function call overhead on plain numbers. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Replace boxed `Number` float tagging with plain tagged objects * Replace `new Number()` boxing with plain tagged float objects for Lua float type tracking. * Integer-valued floats that need type disambiguation are now represented as `{ value: number, isFloat: true }` instead of boxed `Number` objects with a symbol property. * Pre-allocated singletons are used for positive and negative float zeros to avoid allocation entirely in common cases. * Updated all detection, unwrapping, and coercion paths across `numeric.ts`, `runtime.ts`, `eval.ts`, `stdlib.ts`, and `stdlib/` modules to use the new `isTaggedFloat` type guard. * Removed all `instanceof Number` checks. * Deleted the `FloatKind` symbol and eliminated redundant helpers `isLuaFloat`, `isFloatTag`, `getZeroBoxKind` and `toPlainNumber` that became dead code. * Simplified `math.type`, `luaToString`, `luaEquals`, `luaTypeName` and various other key normalization paths. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Add fast paths in `coerceNumericPair` for tagged float operands Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix copy/paste typo Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Avoid extra `LuaEnv` allocations in "For" and "ForIn" loops Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * [Space Lua] Reuse loop variable environment in `for` and `for-in` loops Numeric `for` and generic `for-in` loops allocated a fresh `LuaEnv` on every iteration to hold loop variables. But this is only necessary when a closure inside the loop body captures the loop variable. The optimization uses a two-level check computed at parse time. If no function definition exists in the loop's subtree, environment reuse is safe. When a function definition is present a deeper analysis walks the block to determine whether any function body references the loop variable names without them being shadowed by its own parameters. When a closure captures a loop variable the loop fall back to per-iteration allocation. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * [Space Lua] Format numbers using standard Lua rules Standard Lua formats floats via C `sprintf("%.14g")` (14 significant digits, scientific notation when shorter, exponent padded to 2 digits, and a guaranteed `.0` suffix for integer-valued floats). * Replace the old `luaFormatNumber` with JS `toPrecision(14)`-based implementation that reproduces this behavior. * Integrates it so that `${}` expressions in the UI also display numbers correctly. * Fixes tagged floats (`{ value, isFloat }`) were being stripped by `luaValueToJS` or matched as plain objects before reaching the number formatter. This caused `${}` expressions to render raw JS numbers. Examples: ``` - ${tostring(2^63)} - ${2^63} - ${(2^63)} ``` All of the the above examples show correct `9.2233720368548e+18` now. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * [Space Lua] Fix `string.format` for floats and tagged numbers Unwrap tagged floats before `printf`, handle `inf`/`-inf`/`-nan` in `formatDouble`, and fix `%g` producing `0e+00` for zero. Hopefuly it's enough to gain Lua formatting. Tests: ``` - ${string.format("%.14g", 0.0)} - `0` - ${string.format("%.14g", 1.0)} - `1` - ${string.format("%.14g", 1/3)} - `0.33333333333333` - ${string.format("%.14g", math.pi)} - `3.1415926535898` - ${string.format("%.14g", 1e-10)} - `1e-10` - ${string.format("%.14g", 1e18)} - `1e+18` - ${string.format("%.14g", 2^63)} - `9.2233720368548e+18` - ${string.format("%.14g", 2^53)} - `9.007199254741e+15` - ${string.format("%.14g", 1.7976931348623e+308)} - `1.7976931348623e+308` - ${string.format("%.14g", 5e-324)} - `4.9406564584125e-324` - ${string.format("%.14g", 0/0)} - `-nan` - ${string.format("%.14g", 1/0)} - `inf` - ${string.format("%.14g", -1/0)} - `-inf` ``` Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * [Space Lua] Add and integrate new `luaFormat` utility and test suite * Add `luaFormat` string formatting function compatible with Lua, and integrate it across the codebase as a replacement for prior formatting approaches. * Add extensive test suite. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix check Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * [Space Lua] Add `%a`/`%A` and `%q` format specifiers to `string.format` Implement hexadecimal floating-point (`%a`/`%A`) and quoted literal (`%q`) specifiers. * Add `%a`/`%A` as IEEE 754 double decomposition with full flag, width and precision support. * Add `%q` as producind valid Lua literals for strings, numbers, booleans and nil. * Use `Math.PI` for `math.pi` to preserve full double precision. * Remove Deno based test suite and replace it with native Lua test suite. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * [Space Lua] Add `%p` format specifier to `string.format` In standard Lua, `%p` formats the internal C heap address of a value, producing output like `0x55a3bc4e2f10`. It works on tables, functions, threads, strings, and userdata (GC-ed objects). For `nil`, booleans, and numbers it returns `(null)`. In Space Lua, there are no *raw memory addresses* since the runtime is JavaScript. Instead, `%p` assigns a *stable sequential integer* to each object via a `WeakMap`, formatted as a 14-digit zero-padded hex value. The key difference is that identifiers are deterministic and sequential rather than random-looking heap addresses: ```lua local t = {} print(string.format("identifier: %p", t)) -- 0x00000000000001 print(string.format("the same: %p", t)) -- 0x00000000000001 print(string.format("another: %p", {}) -- 0x00000000000002 ``` For strings, a regular `Map` is used so identical string content always produces the same identifier. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Revert merge changes to the deno.json Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Replace `interface` with `type` Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Remove `has_math()` relict function test Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Refactor loop to map Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Remove `Deno.remove("deno.lock")` weirdness Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Refactor: Early return undefined in `astNumberKind` instead of assigning Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> --------- Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> Co-authored-by: Zef Hemel <zef@zef.me> |
||
|
Zef Hemel
|
f4336bf09f | Code restructuring: eliminated the top-level lib/ directory |