* [LIQ] Add new aggregate functions, aliases, and queryable aggregate registry * Extend with 13 new built-in aggregates: `product`, `string_agg`, `yaml_agg`, `json_agg`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`, `bool_or`, `stddev_pop`, `stddev_samp`, `var_pop` and `var_samp`. * Introduce `aggregate.alias` API allowing users to define custom aliases for any aggregate. Standard aliases (`every`, `std`, `stddev` and `variance`) are now defined via this API rather than hardcoded. * Add `index.aggregates` queryable collection so users can discover all available aggregates directly from LIQ queries. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix config pass through query path so custom aggregates work Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Preserve `LuaTable`/`LuaFunction` values in aggregate config storage `config.set` uses `LuaNativeJSFunction` which calls `luaValueToJS` on all arguments. This converted the aggregate `LuaTable` to a plain JS object and wrapped `LuaFunction` callbacks in JS functions that also converted their returned values via `luaValueToJS`. The result was that state returned by initialize (a `LuaTable`) got converted to a plain JS object before being passed to `iterate`. Therefor Lua operations like `table.insert` on that were failing because they expected a `LuaTable` and not a plain JS array. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix formatting Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Improve aggregate functions descriptions, fix `sum` divergence Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Align `product` with `sum` Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: extract `alias` from `LuaTable` via `rawGet` in `aggregates()` registry Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Rename `alias` in `aggregates()` to `target` for clarity Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add a null guard at the top of `jsToLuaValue` This preserves `null`/`undefined` as-is (both map to Lua nil) and prevents them from falling through to the `typeof` "object" branch. For this PR it means that null `target` in our `aggregates` entries will correctly show as empty/`nil` in query results rather than `{}`. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Documentation reflects recent changes Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: make `sum`/`product` return null on empty input; stop `LIQ_NULL` leaks * `sum(`) and` product(`) now return null when no rows match (matching Postgres semantics) instead of returning 0 and 1 respectively. * Query result columns that hold null are internally preserved using a `LIQ_NULL` sentinel so that column keys survive in `LuaTable` storage. This sentinel was leaking into Lua code as "userdata" through three read paths: * `luaIndexValue`: `rawGet` returned the sentinel directly to Lua when accessing table fields, * `rawget` (stdlib): the builtin `rawget` function exposed the sentinel without converting it back to `nil`, * `createAugmentedEnv`: string interpolation unpacked table values via `rawGet` into local variables, making the sentinel visible in template expressions like `${var}`. All three now convert `LIQ_NULL` to `nil` at the read boundary, keeping the sentinel internal to table storage where it belongs. * Update affected test expectations accordingly. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Remove duplicated LIQ_NULL hazard, add guard for all builtin aggregate `iterate`s Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: `array_agg` preserves NULL positions Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add symbol guard to `json_agg` `JSON.stringify(Symbol(...))` in an array produces null by accident. That is a JS implementation detail we **MUST NOT** rely on. Explicit null push makes intent clear and avoids surprises if the `Symbol` representation ever changes. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add symbol guard to `yaml_agg` (ditto) `js-yaml` has no knowledge of the `LIQ_NULL` symbol. Passing null makes it emit YAML null (or `~`), which is the correct YAML representation of a missing value and matches standard `json_agg`/`yaml_agg` NULL-inclusion semantics. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Add intra-aggregate ordering null guards Without this, `LIQ_NULL` sort keys would fall through to `valA < valB` which is always false for `Symbol`s which is breaking the `nulls first`/`nulls last` contract... Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Ditto, but for `order by` null comparisons Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Guard `luaTypeName`, `luaTypeOf` and `luaToString` against `LIQ_NULL` sentinel Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Guard presentation layer against `LIQ_NULL` sentinel leaking as visible text Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Fix: Evaluate extra args per-item in `executeAggregate`; add new aggregates Extra arguments (2nd, 3rd, etc.) to aggregate functions were evaluated against the outer query environment where the object variable is not bound. This caused multi-argument aggregates like `covar_samp(data.y, data.x)` to fail with nil reference errors. This commit addresses this by evaluating extra args per-item inside the iterate loop using the item environment so all arguments resolve correctly. We also add few common aggregates: - `covar_pop`, `covar_samp`, `corr`: population/sample covariance and correlation coefficient using online co-moment algorithm. - `quantile(value, q, method)`: general quantile with interpolation methods: lower, higher, nearest, midpoint and default linear. - `percentile_cont(value, q)`: continuous percentile (linear) - `percentile_disc(value, q)`: discrete percentile (lower) Note: `percentile_cont` and `percentile_disc` share the `quantile` implementation through `ctx.name` at initialize time. Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Update docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Make the ordering for quantile aggregates explicit Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Update docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> * Improve docs Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz> --------- Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
SilverBullet
SilverBullet is a Programmable, Private, Browser-based, Open Source, Self Hosted, Personal Knowledge Management Platform.
Yowza! That surely is a lot of adjectives to describe a browser-based Markdown editor programmable with Lua.
Let’s get more specific.
In SilverBullet you keep your content as a collection of Markdown Pages (called a Space). You navigate your space using the Page Picker like a traditional notes app, or through Links like a wiki (except they are bi-directional).
If you are the writer type, you’ll appreciate SilverBullet as a clean Markdown editor with Live Preview. If you have more of an outliner personality, SilverBullet has Outlining tools for you. Productivity freak? Have a look at Tasks. More of a database person? You will appreciate Objects and Queries.
And if you are comfortable programming a little bit — now we’re really talking. You will love dynamically generating content with Space Lua (SilverBullet’s Lua dialect), or to use it to create custom Commands, Page Templates or Widgets.
Much more detail can be found on silverbullet.md
Installing SilverBullet
Check out the instructions.
Developing SilverBullet
SilverBullet's frontend is written in TypeScript and built on top of the excellent CodeMirror 6 editor component. Additional UI is built using Preact. ESBuild is used to build the frontend.
The server backend is written in Go.
If you're considering contributing changes, be aware of the LLM use policy.
Code structure
client/: The SilverBullet client, implemented with TypeScriptserver/: The SilverBullet server, written in Goplugs: Set of built-in plugs that are distributed with SilverBulletlibraries: A set of libraries (space scripts, page templates, slash templates) distributed with SilverBulletplug-api/: Useful APIs for use in plugslib/: Useful libraries to be used in plugssyscalls/: TypeScript wrappers around syscallstypes/: Various (client) types that can be references from plugs
binplug_compile.tsthe plug compiler
scripts/: Useful scriptswebsite/: silverbullet.md website content
Requirements
The project includes .nvmrc and .node-version files. If you use nvm or another Node version manager, it will automatically use the correct Node.js version:
nvm use # If using nvm
It's convenient to also install air for development, this will automatically rebuild both the frontend and backend when changes are made:
go install github.com/air-verse/air@latest
Make sure your $GOPATH/bin is in your $PATH.
First, install dependencies:
npm install
To build everything and run the server:
air <PATH-TO-YOUR-SPACE>
Alternatively, to build:
make build
# or
npm run build
To run the resulting server:
./silverbullet <PATH-TO-YOUR-SPACE>
Useful development tasks
# Clean all generated files
make clean
# Typecheck and lint all code
make check
# Format all code
make fmt
# Run all tests
make test
# Run benchmarks
make bench
Build a docker container
Note, you do not need Node.js nor Go locally installed for this to work:
docker build -t silverbullet .
To run:
docker run -p 3000:3000 -v <PATH-TO-YOUR-SPACE>:/space silverbullet