Matouš Jan FialkaandGitHub 6cbb61c7db [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry (#1891)
* [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry

* Extend with 13 new built-in aggregates: `product`, `string_agg`,
  `yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`,
  `bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`.

* Introduce `aggregate.alias` API allowing users to define custom
  aliases for any aggregate. Standard aliases (`every`, `std`, `stddev`
  and `variance`) are now defined via this API rather than hardcoded.

* Add `index.aggregates` queryable collection so users can discover
  all available aggregates directly from LIQ queries.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix config pass through query path so custom aggregates work

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage

`config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on
all arguments. This converted the aggregate `LuaTable` to a plain JS
object and wrapped `LuaFunction` callbacks in JS functions that also
converted their returned values via `luaValueToJS`. The result was that
state returned by initialize (a `LuaTable`) got converted to a plain JS
object before being passed to `iterate`. Therefor Lua operations like
`table.insert` on that were failing because they expected a `LuaTable`
and not a plain JS array.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix formatting

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Improve aggregate functions descriptions, fix `sum` divergence

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Align `product` with `sum`

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Rename `alias` in `aggregates()` to `target` for clarity

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add a null guard at the top of `jsToLuaValue`

This preserves `null`/`undefined` as-is (both map to Lua nil) and
prevents them from falling through to the `typeof` "object" branch.

For this PR it means that null `target` in our `aggregates` entries will
correctly show as empty/`nil` in query results rather than `{}`.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Documentation reflects recent changes

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks

* `sum(`) and` product(`) now return null when no rows match (matching
  Postgres semantics) instead of returning 0 and 1 respectively.

* Query result columns that hold null are internally preserved using
  a `LIQ_NULL` sentinel so that column keys survive in `LuaTable`
  storage.  This sentinel was leaking into Lua code as "userdata"
  through three read paths:

  * `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when
    accessing table fields,

  * `rawget` (stdlib): the builtin `rawget` function exposed the
    sentinel without converting it back to `nil`,

  * `createAugmentedEnv`: string interpolation unpacked table values via
    `rawGet` into local variables, making the sentinel visible in
    template expressions like `${var}`.

  All three now convert `LIQ_NULL` to `nil` at the read boundary,
  keeping the sentinel internal to table storage where it belongs.

* Update affected test expectations accordingly.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: `array_agg` preserves NULL positions

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add symbol guard to `json_agg`

`JSON.stringify(Symbol(...))` in an array produces null by accident.
That is a JS implementation detail we **MUST NOT** rely on. Explicit
null push makes intent clear and avoids surprises if the `Symbol`
representation ever changes.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add symbol guard to `yaml_agg` (ditto)

`js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes
it emit YAML null (or `~`), which is the correct YAML representation of
a missing value and matches standard `json_agg`/`yaml_agg`
NULL-inclusion semantics.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Add intra-aggregate ordering null guards

Without this, `LIQ_NULL` sort keys would fall through to `valA < valB`
which is always false for `Symbol`s which is breaking the `nulls
first`/`nulls last` contract...

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Ditto, but for `order by` null comparisons

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates

Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated
against the outer query environment where the object variable is not
bound. This caused multi-argument aggregates like `covar_samp(data.y,
data.x)` to fail with nil reference errors. This commit addresses this
by evaluating extra args per-item inside the iterate loop using the item
environment so all arguments resolve correctly.

We also add few common aggregates:

- `covar_pop`, `covar_samp`, `corr`: population/sample covariance and
  correlation coefficient using online co-moment algorithm.

- `quantile(value, q, method)`: general quantile with interpolation
  methods: lower, higher, nearest, midpoint and default linear.

- `percentile_cont(value, q)`: continuous percentile (linear)

- `percentile_disc(value, q)`: discrete percentile (lower)

Note: `percentile_cont` and `percentile_disc` share the `quantile`
implementation through `ctx.name` at initialize time.

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Update docs

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Make the ordering for quantile aggregates explicit

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Update docs

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

* Improve docs

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>

---------

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
2026-03-19 09:37:09 +01:00
2026-02-27 16:25:34 +01:00
2026-02-27 15:35:34 +01:00
2026-03-17 09:57:08 +01:00
2026-02-27 15:35:34 +01:00
2026-01-21 09:22:13 +01:00
2026-03-17 10:22:16 +01:00
2026-02-14 22:17:37 +01:00
2026-02-14 22:17:37 +01:00
2026-03-10 09:42:48 +01:00
2022-08-02 14:40:04 +02:00
2025-09-28 17:24:25 +02:00
2026-02-14 22:17:37 +01:00
2026-02-06 14:49:20 +01:00
2026-03-17 09:57:08 +01:00
2026-03-17 09:57:08 +01:00
2022-10-12 11:47:13 +02:00
2026-03-17 09:57:08 +01:00
2026-02-27 15:35:34 +01:00
2026-03-06 14:36:23 +01:00
2026-02-14 22:08:35 +01:00

GitHub Repo stars Docker Pulls GitHub Downloads (all assets, all releases) GitHub contributors

SilverBullet

SilverBullet is a Programmable, Private, Browser-based, Open Source, Self Hosted, Personal Knowledge Management Platform.

Yowza! That surely is a lot of adjectives to describe a browser-based Markdown editor programmable with Lua.

Let’s get more specific.

In SilverBullet you keep your content as a collection of Markdown Pages (called a Space). You navigate your space using the Page Picker like a traditional notes app, or through Links like a wiki (except they are bi-directional).

If you are the writer type, you’ll appreciate SilverBullet as a clean Markdown editor with Live Preview. If you have more of an outliner personality, SilverBullet has Outlining tools for you. Productivity freak? Have a look at Tasks. More of a database person? You will appreciate Objects and Queries.

And if you are comfortable programming a little bit — now we’re really talking. You will love dynamically generating content with Space Lua (SilverBullet’s Lua dialect), or to use it to create custom Commands, Page Templates or Widgets.

Much more detail can be found on silverbullet.md

Installing SilverBullet

Check out the instructions.

Developing SilverBullet

SilverBullet's frontend is written in TypeScript and built on top of the excellent CodeMirror 6 editor component. Additional UI is built using Preact. ESBuild is used to build the frontend.

The server backend is written in Go.

If you're considering contributing changes, be aware of the LLM use policy.

Code structure

  • client/: The SilverBullet client, implemented with TypeScript
  • server/: The SilverBullet server, written in Go
  • plugs: Set of built-in plugs that are distributed with SilverBullet
  • libraries: A set of libraries (space scripts, page templates, slash templates) distributed with SilverBullet
  • plug-api/: Useful APIs for use in plugs
    • lib/: Useful libraries to be used in plugs
    • syscalls/: TypeScript wrappers around syscalls
    • types/: Various (client) types that can be references from plugs
  • bin
    • plug_compile.ts the plug compiler
  • scripts/: Useful scripts
  • website/: silverbullet.md website content

Requirements

  • Node.js 24+ and npm 10+: Used to build the frontend and plugs
  • Go: Used to build the backend

The project includes .nvmrc and .node-version files. If you use nvm or another Node version manager, it will automatically use the correct Node.js version:

nvm use  # If using nvm

It's convenient to also install air for development, this will automatically rebuild both the frontend and backend when changes are made:

go install github.com/air-verse/air@latest

Make sure your $GOPATH/bin is in your $PATH.

First, install dependencies:

npm install

To build everything and run the server:

air <PATH-TO-YOUR-SPACE>

Alternatively, to build:

make build
# or
npm run build

To run the resulting server:

./silverbullet <PATH-TO-YOUR-SPACE>

Useful development tasks

# Clean all generated files
make clean
# Typecheck and lint all code
make check
# Format all code
make fmt
# Run all tests
make test
# Run benchmarks
make bench

Build a docker container

Note, you do not need Node.js nor Go locally installed for this to work:

docker build -t silverbullet .

To run:

docker run -p 3000:3000 -v <PATH-TO-YOUR-SPACE>:/space silverbullet
S
Description
No description provided
Readme
51 MiB
Languages
TypeScript 69.8%
Rust 20.2%
Lua 5.2%
SCSS 4.1%
Shell 0.2%
Other 0.3%