# EdgeDB v4

This release cycle is much shorter than the previous ones. It reflects our new approach at EdgeDB where the goal is to provide improvements at a steady regular pace rather than in big, but infrequent batches. Going forward we expect to maintain this shorter release cadence focusing on a few features at a time.

To play with the new features, install the CLI using [our installation guide](https://www.edgedb.com/install) and initialize a new project.

```bash
$ edgedb project init
```

## Upgrading

**Local and Cloud instances**

To upgrade a local project, first ensure that your CLI is up to date with `edgedb cli upgrade`. Then run the following command inside the project directory.

```bash
$ edgedb project upgrade
```

Alternatively, specify an instance name if you aren't using a project.

```bash
$ edgedb instance upgrade -I my_instance
```

The CLI will first check to see if your schema will migrate cleanly to EdgeDB 4.0. If the upgrade check finds any problems, it will report them back to you.

**Hosted instances**

To upgrade a remote (hosted) instance, we recommend the following dump-and-restore process.

1. EdgeDB v4.0 only supports PostgreSQL 14 (or above). So check the version of PostgreSQL you are using before upgrading EdgeDB. If you're using Postgres 13 or below, you should upgrade Postgres first.

2. Spin up an empty 4.0 instance. You can use one of our [deployment guides](https://docs.geldata.com/reference/running/deployment.md#ref-guide-deployment).
   
   Under Debian/Ubuntu, when adding the EdgeDB package repository, use this command instead:
   
   ```bash
   $ echo deb [signed-by=/usr/local/share/keyrings/edgedb-keyring.gpg] \
       https://packages.edgedb.com/apt \
       $(grep "VERSION_CODENAME=" /etc/os-release | cut -d= -f2) main \
       | sudo tee /etc/apt/sources.list.d/edgedb.list
   ```
   
   Use this command for installation under Debian/Ubuntu:
   
   ```bash
   $ sudo apt-get update && sudo apt-get install edgedb-4
   ```
   
   Under CentOS/RHEL, use this installation command:
   
   ```bash
   $ sudo yum install edgedb-4
   ```
   
   In any required `systemctl` commands, replace `edgedb-server-3` with `edgedb-server-4`.
   
   Under any Docker setups, supply the `4.0` tag.

3. Take your application offline, then dump your v3.x database with the CLI
   
   ```bash
   $ gel dump --dsn <old dsn> --all --format dir my_database.dump/
   ```
   
   This will dump the schema and contents of your current database to a directory on your local disk called `my_database.dump`. The directory name isn't important.

4. Restore the empty v4.x instance from the dump
   
   ```bash
   $ gel restore --all my_database.dump/ --dsn <new dsn>
   ```
   
   Once the restore is complete, update your application to connect to the new instance.
   
   This process will involve some downtime, specifically during steps 2 and 3.


> Note: If your Postgres cluster is also backing other versions of EdgeDB, make sure you start your new instance with the `--ignore-other-tenants` option when bootstrapping your new instance.

## New features

### Full-text Search

EdgeDB 4.0 adds [full-text search](https://docs.geldata.com/reference/stdlib/fts.md#ref-std-fts) functionality packaged in the `fts` module. By adding an `fts::index` to an object type you can transform any object into a searchable document:

```sdl
type Item {
  required available: bool {
    default := false;
  };
  required name: str;
  required description: str;

  index fts::index on (
    fts::with_options(
      .name,
      language := fts::Language.eng
    )
  );
}
```

The `fts::index` indicates to EdgeDB that this object type is a valid target for full-text search. The property that will be searched as well as the language is provided in the index.

The [`fts::search()`](https://docs.geldata.com/reference/stdlib/fts.md#function::fts::search) function allows searching objects for a particular phrase:

```edgeql-repl
db> select fts::search(Item, 'candy corn', language := 'eng');
{
  (
    object := default::Item {id: 9da06b18-69b2-11ee-96b9-1bedbe75ad4f},
    score := 0.30396354,
  ),
  (
    object := default::Item {id: 92375624-69b2-11ee-96b9-675b9b87ac70},
    score := 0.6079271,
  ),
}
```

The search results are provided as a tuple containing the matching document object and a score. Higher score indicates a better match. So we can use these values to order the results:

```edgeql-repl
db> with res := (
...   select fts::search(Item, 'candy corn', language := 'eng')
... )
... select res.object {name, score := res.score}
... order by res.score desc;
{
  default::Item {name: 'Candy corn', score: 0.6079271},
  default::Item {name: 'Canned corn', score: 0.30396354},
}
```

You can only have at most one `fts::index` defined for any particular type. So if there are multiple properties that should be searchable, they can all be specified in that one index:

```sdl
type Item {
  required available: bool {
    default := false;
  };
  required name: str;
  required description: str;

  index fts::index on ((
    fts::with_options(
      .name,
      language := fts::Language.eng
    ),
    fts::with_options(
      .description,
      language := fts::Language.eng
    )
  ));
}
```

The above schema declares both `name` and `description` as searchable fields:

```edgeql-repl
db> with res := (
...   select fts::search(Item, 'trick or treat', language := 'eng')
... )
... select res.object {name, description, score := res.score}
... order by res.score desc;
{
  default::Item {
    name: 'Candy corn',
    description: 'A great Halloween treat',
    score: 0.30396354,
  },
}
```

### Multiranges

We've made it easier to work with ranges by adding a [multirange](https://docs.geldata.com/reference/stdlib/range.md#ref-std-range) datatype. Multiranges consist of one or more ranges and allow expressing intervals that are not contiguous. Multiranges are automatically normalized to contain non-overlapping ranges that are ordered according to their boundaries. All the usual range operators and functions like `overlaps` or `contains` work with any combination of ranges and multiranges, providing more flexibility in expressions.

```edgeql-repl
db> select multirange([range(8, 10)]) + range(1, 5) - range(3, 4);
{[range(1, 3), range(4, 5), range(8, 10)]}
```

## GraphQL and HTTP authentication

Starting in rc1, the [EdgeQL over HTTP](https://docs.geldata.com/reference/using/http.md#ref-edgeql-http) and [GraphQL](https://docs.geldata.com/reference/using/graphql.md#ref-graphql-index) endpoints support (and by default require) authentication.

By default, [HTTP Basic Authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme) is used.

Full details are available in the [EdgeQL over HTTP documentation](https://docs.geldata.com/reference/using/http.md#ref-http-auth).

This is a backwards-incompatible change. It is possible to opt-in to the old behavior, but not recommended.

## Extensions

### auth

The new `auth` extension adds a full authentication service that runs alongside your database instance, saving you the hassle of having to learn and implement the intricacies of OAuth or secure password storage.

- OAuth Integration: Seamlessly authenticate with GitHub, Google, Apple, and Azure/Microsoft.
- Email & Password Support: Includes robust email+password authentication with reset password functionality.
- Easy Configuration: Set up via our configuration system.
- Hosted UI: Use our hosted authentication UI to quickly add authentication to your app.

When a user signs up, we create a new object of type `ext::auth::Identity`, which you can link to in your own schema. We then provide you with a token that can be set as the global `ext::auth::client_token` which will automatically populate another computed global called `ext::auth::ClientTokenIdentity` which you can use directly in your access policies, or in your own globals.

```sdl
using extension auth;

module default {
    global current_customer := (
        assert_single((
            select Customer
            filter .identity = global ext::auth::ClientTokenIdentity
        ))
    );

    type Customer {
        required text: str;
        required identity: ext::auth::Identity;
    }

    type Item {
        required sku: str;
        required description: str;
    }

    type Cart {
        required customer: Customer;
        multi items: Item {
            quantity: int32;
        };

        access policy customer_has_full_access
            allow all
            using (global current_customer ?= .customer);
    }
}
```

Here's an example query using the TypeScript client:

```typescript
import { createClient } from "edgedb";

declare const tokenFromAuthServer: string;
const client = createClient()
  .withGlobals({
    "ext::auth::client_token": tokenFromAuthServer
  });

const carts = await client.query(`select Cart { * };`);
```

### pgcrypto

We've added [pgcrypto](https://docs.geldata.com/reference/stdlib/pgcrypto.md#ref-ext-pgcrypto) to our extensions. This exposes `digest`, `hmac`, `gen_salt` and `crypt` functions for your hashing, encrypting and salting needs.

```edgeql-repl
db> select ext::pgcrypto::digest('encrypt this', 'sha1');
{b'\x05\x82\xd8YLF\xe7\xd4\x12\x91\n\xdb$\xf1!v\xf9\xd4\x89\xc4'}
db> select ext::pgcrypto::gen_salt('md5');
{'$1$FjNlXgX7'}
```

Standard algorithms are "md5", "sha1", "sha224", "sha256", "sha384" and "sha512". Moreover, any digest algorithm OpenSSL supports is automatically picked up.

### pg_trgm

The [pg_trgm](https://docs.geldata.com/reference/stdlib/pg_trgm.md#ref-ext-pgtrgm) extension provides functionality used to determine string similarity, which makes it a good text search alternative for some use cases:

```edgeql-repl
db> with x := {'hello world', 'word hero', 'help the world'}
... select res := (x, ext::pg_trgm::word_similarity(x, 'hello world'))
... order by res.1 desc;
{('hello world', 1), ('help the world', 0.5), ('word hero', 0.35714287)}
```

## Additional changes

### Performance

We've made a few internal changes affecting performance, the biggest of which was rewriting EdgeQL parser in Rust. Overall we've manged to reduce the baseline server memory consumption by 40%.

### EdgeQL

- Add new style of `if`/`then`/`else` syntax. ([#6074](https://github.com/edgedb/edgedb/issues/6074))
  
  Many people find it more natural to write "if … then .. else …" for conditional expressions because it mirrors the conditional statement from other familiar programming languages.
  
  ```edgeql-repl
  db> select if count(Object) > 0 then 'got data' else 'no data';
  {'got data'}
  ```

- Support conditional DML. ([#6181](https://github.com/edgedb/edgedb/issues/6181))
  
  It can be useful to be able to create, update or delete different objects based on some condition:
  
  ```edgeql
  with
    name := <str>$0,
    admin := <bool>$1
  select if admin then (
      insert AdminUser { name := name }
  ) else (
      insert User { name := name }
  )
  ```
  
  A different use-case of conditional DML is using a [`coalesce`](https://docs.geldata.com/reference/stdlib/set.md#operator::coalesce) operator to express things like "select or insert if missing":
  
  ```edgeql
  select (select User filter .name = 'Alice') ??
         (insert User { name := 'Alice' });
  ```

- Add `contains` for JSON so that it can be used with `pg::gin` index. ([#5910](https://github.com/edgedb/edgedb/issues/5910))
- Add [`to_bytes()`](https://docs.geldata.com/reference/stdlib/bytes.md#function::std::to_bytes) to convert [`str`](https://docs.geldata.com/reference/stdlib/string.md#type::std::str) into [`bytes`](https://docs.geldata.com/reference/stdlib/bytes.md#type::std::bytes) using UTF-8 encoding. ([#5960](https://github.com/edgedb/edgedb/issues/5960))

- Add [`to_str()`](https://docs.geldata.com/reference/stdlib/string.md#function::std::to_str) to convert [`bytes`](https://docs.geldata.com/reference/stdlib/bytes.md#type::std::bytes) into [`str`](https://docs.geldata.com/reference/stdlib/string.md#type::std::str) using UTF-8 encoding. ([#5960](https://github.com/edgedb/edgedb/issues/5960))

- Add `enc::base64_encode` and `enc::base64_decode` functions. ([#5963](https://github.com/edgedb/edgedb/issues/5963))
  
  ```edgeql-repl
  db> select enc::base64_encode(b'hello');
  {'aGVsbG8='}
  db> select enc::base64_decode('aGVsbG8=');
  {b'hello'}
  ```

- Add `when` clause to triggers to enable them to be conditional. ([#6184](https://github.com/edgedb/edgedb/issues/6184))
- Allow empty arrays without cast in `insert`. ([#6218](https://github.com/edgedb/edgedb/issues/6218))

### GraphQL

- Change how globals are passed in GraphQL queries. ([#5864](https://github.com/edgedb/edgedb/issues/5864))
  
  Instead of using a separate `globals` field (which is non-standard), use `variables` to add a `__globals__` object to pass the global variables.
  
  In order to ensure backwards compatibility, the old way of passing globals is still valid. In case both the new and the old methods are used the globals being passed in them must match or else the query will be rejected.

- Fix GraphQL bug with objects without editable fields. ([#6056](https://github.com/edgedb/edgedb/issues/6056))
- Fix GraphQL issues with deeply nested modules. ([#6056](https://github.com/edgedb/edgedb/issues/6056))
- Fix GraphQL `__typename` for non-default modules and mutations. ([#6035](https://github.com/edgedb/edgedb/issues/6035))
- Fix GraphQL fragments on types from non-default module. ([#6035](https://github.com/edgedb/edgedb/issues/6035))

### Bug fixes

- Fix a casting bug for some aliased expressions. ([#5788](https://github.com/edgedb/edgedb/issues/5788))
- Fix cardinality inference of calls to functions with `optional` args. ([#5867](https://github.com/edgedb/edgedb/issues/5867))
- Fix the undefined order of columns in SQL `COPY`. ([#6036](https://github.com/edgedb/edgedb/issues/6036))
- Fix drop of union links when source has a subtype. ([#6044](https://github.com/edgedb/edgedb/issues/6044))
- Fix link deletion policies on links to union types. ([#6033](https://github.com/edgedb/edgedb/issues/6033))
- Fix deletion issues of aliases that use `with` ([#6052](https://github.com/edgedb/edgedb/issues/6052))
- Make `id` of schema objects stable. ([#6058](https://github.com/edgedb/edgedb/issues/6058))
- Allow computed pointers on types to omit link/property kind specification. ([#6073](https://github.com/edgedb/edgedb/issues/6073))
- Support `listen_ports` greater than 32767. ([#6194](https://github.com/edgedb/edgedb/issues/6194))
- Fix migration issues with some overloaded indexes/constraints in SDL. ([#6172](https://github.com/edgedb/edgedb/issues/6172))
- Support DML on right hand side of coalesce expressions. ([#6202](https://github.com/edgedb/edgedb/issues/6202))
- Fix cardinality inference of polymorphic shape elements. ([#6255](https://github.com/edgedb/edgedb/issues/6255))
- Fix migration issue involving property defaults. ([#6265](https://github.com/edgedb/edgedb/issues/6265))
- Fix bugs in `set ... using` statements with `assert_exists` and similar. ([#6267](https://github.com/edgedb/edgedb/issues/6267))
- Fix cardinality bug when a property appears in multiple splats. ([#6255](https://github.com/edgedb/edgedb/issues/6255))
- Make comparison operators non-associative ([#6327](https://github.com/edgedb/edgedb/issues/6327))
- Fix an obscure parser bug caused by constant extraction ([#6328](https://github.com/edgedb/edgedb/issues/6328))
- Cap the size of sets in `multi` configuration values to `128` ([#6402](https://github.com/edgedb/edgedb/issues/6402))

## 4.1

- Fix dump and restore of auth config ([#6414](https://github.com/edgedb/edgedb/issues/6414))
- Clear `_config_cache` in `sys_pgcon` before reloading reported config ([#6427](https://github.com/edgedb/edgedb/issues/6427))
- Fix access policy type rewrites of `std::Object` ([#6420](https://github.com/edgedb/edgedb/issues/6420))
- Fix an error message of affected refs ([#6425](https://github.com/edgedb/edgedb/issues/6425))
- Fix DML with constraints on abstract types ([#6421](https://github.com/edgedb/edgedb/issues/6421))

## 4.2

- Fix schema delta for RESET EXPRESSION ([#6463](https://github.com/edgedb/edgedb/issues/6463))
- Fix plain references to __old__ in rewrites ([#6470](https://github.com/edgedb/edgedb/issues/6470))
- Fix std::range in singleton mode ([#6475](https://github.com/edgedb/edgedb/issues/6475))
- Treat password reset as a verification event ([#6504](https://github.com/edgedb/edgedb/issues/6504))
- Fixes to auth redirect urls ([#6469](https://github.com/edgedb/edgedb/issues/6469))
- Fix SQL introspection of __fts_document__ ([#6507](https://github.com/edgedb/edgedb/issues/6507))
- Type check mutation rewrites at migration time ([#6466](https://github.com/edgedb/edgedb/issues/6466))
  
  Mutation rewrite's `using` expression are now required to be of correct type when the rewrite is created. Up until now, it was possible to migrate to a schema that contained a rewrite rule that would always throw a type error when an object was being inserted or updated.
  
  This might be considered a breaking change, but as it is clearly a bug in user schema and as it could also be considered a bug in the compiler, we are fixing it in a minor version.


## 4.3

- Fix non-rewritten tuple literals ([#6521](https://github.com/edgedb/edgedb/issues/6521))
- Support `connect_timeout` in backend DSN ([#6531](https://github.com/edgedb/edgedb/issues/6531))
- Fix coalesced DML in FOR loops over objects ([#6526](https://github.com/edgedb/edgedb/issues/6526))
- Fix inserts silently failing when a `json->array` handles 'null' ([#6544](https://github.com/edgedb/edgedb/issues/6544))
- Fix tracing of enums in type expressions ([#6548](https://github.com/edgedb/edgedb/issues/6548))
- Fix dumps with FTS indexes ([#6560](https://github.com/edgedb/edgedb/issues/6560))
- Allow indexes that use user-defined functions to actually get hit ([#6551](https://github.com/edgedb/edgedb/issues/6551))
- Fix reloading readiness state and JWT `*_list` files in multi-tenant mode ([#6562](https://github.com/edgedb/edgedb/issues/6562))
- Handle OPTIONS in the extension path which is needed for CORS preflight ([#6577](https://github.com/edgedb/edgedb/issues/6577))
- Optimize the compiler and reduce time of an update test by ~52% ([#6579](https://github.com/edgedb/edgedb/issues/6579))
- Always retry system connection on any BackendError ([#6588](https://github.com/edgedb/edgedb/issues/6588))
- Properly support @source/@target on multi-link constraints ([#6585](https://github.com/edgedb/edgedb/issues/6585))
- Fix constant extraction's interaction with the new if-then-else ([#6591](https://github.com/edgedb/edgedb/issues/6591))
- Fix migrating between aliased and non-aliased computeds ([#6566](https://github.com/edgedb/edgedb/issues/6566))
- Improve error message for illegal casts and parameters. ([#6511](https://github.com/edgedb/edgedb/issues/6511))
- Don't eval ext::auth::ClientTokenIdentity on every row in a filter ([#6607](https://github.com/edgedb/edgedb/issues/6607))
- Expose __type__ via SQL adapter ([#6519](https://github.com/edgedb/edgedb/issues/6519))
- Fix some bugs involving union and coalescing of optional values ([#6590](https://github.com/edgedb/edgedb/issues/6590))
- Avoid doing a join when injecting type ids ([#6601](https://github.com/edgedb/edgedb/issues/6601))
- Generate better code for ?? with multi RHS or object type ([#6532](https://github.com/edgedb/edgedb/issues/6532))
- Fix pgast nullability inference for subqueries and coalesce ([#6529](https://github.com/edgedb/edgedb/issues/6529))
- Fix changing a pointer to be computed when there is a subtype ([#6565](https://github.com/edgedb/edgedb/issues/6565))

## 4.4

- Fix DML-containing FOR when the iterator is an object set with duplicates ([#6609](https://github.com/edgedb/edgedb/issues/6609))
- Fix very slow global introspection query when there are lots of databases ([#6633](https://github.com/edgedb/edgedb/issues/6633))
- Use correct signature for in-memory cache method ([#6643](https://github.com/edgedb/edgedb/issues/6643))

## 4.5

- Fix spurious delete/create of constraints with arguments in migrations ([#6712](https://github.com/edgedb/edgedb/issues/6712))
- Speed up introspection of system objects like Database and Role ([#6665](https://github.com/edgedb/edgedb/issues/6665))
- Fix some alter+rename combos on pointers in POPULATE MIGRATION ([#6666](https://github.com/edgedb/edgedb/issues/6666))
- Fix issue with schema changes when trigger depends on a rewrite ([#6706](https://github.com/edgedb/edgedb/issues/6706))
- Get verification data from request body ([#6723](https://github.com/edgedb/edgedb/issues/6723))
- Make overloading a link when omitting `link` keyword work ([#6718](https://github.com/edgedb/edgedb/issues/6718))

## 4.6

- Fix issues with empty sets leaking out of optional scopes ([#6747](https://github.com/edgedb/edgedb/issues/6747))
- Fix empty array literals in SQL adapter ([#6806](https://github.com/edgedb/edgedb/issues/6806))
- Fix duration/memory config in config objects ([#6827](https://github.com/edgedb/edgedb/issues/6827))
- Fix hanging backend pool with fast connect ([#6813](https://github.com/edgedb/edgedb/issues/6813))
- Fix changing index expressions in migrations ([#6843](https://github.com/edgedb/edgedb/issues/6843))
- Properly report errors involving newly created types ([#6852](https://github.com/edgedb/edgedb/issues/6852))
- Make constraint error details contain useful information for developers ([#6796](https://github.com/edgedb/edgedb/issues/6796))
- Fix DML coalesce inside of IF/ELSE ([#6917](https://github.com/edgedb/edgedb/issues/6917))
- Allow trailing comma for GROUP's BY clause ([#7002](https://github.com/edgedb/edgedb/issues/7002))
- Fix computed single scalar globals ([#6999](https://github.com/edgedb/edgedb/issues/6999))
- Fix query cache dbver issue with concurrent DDL ([#6819](https://github.com/edgedb/edgedb/issues/6819))
- Several UI fixes
  
  - Update behaviour of datatable editor to save input by default on closing ([edgedb/edgedb-ui/#325](https://github.com/edgedb/edgedb-ui/issues/325))
  
  - Store instance version with schema data, so schema is refreshed when instance is upgraded ([edgedb/edgedb-ui/#333](https://github.com/edgedb/edgedb-ui/issues/333))
  
  - De-duplicate queries when navigating repl history with ctrl+up/down ([edgedb/edgedb-ui/7916ee70](https://github.com/edgedb/edgedb-ui/commit/7916ee70))
  
  - Add max height to inline editor in dataview ([edgedb/edgedb-ui/b2fedb72](https://github.com/edgedb/edgedb-ui/commit/b2fedb72))
  - Always order `id` column first in dataview ([edgedb/edgedb-ui/9a7c352e](https://github.com/edgedb/edgedb-ui/commit/9a7c352e))
  - Improve rendering of long/multiline strings in dataview changes preview ([edgedb/edgedb-ui/13511ebd](https://github.com/edgedb/edgedb-ui/commit/13511ebd))
  
  - Fix link props in visual query builder ([edgedb/edgedb-ui/42492465](https://github.com/edgedb/edgedb-ui/commit/42492465))
  - And [many more UI fixes](https://github.com/edgedb/edgedb-ui/commits/4.x/?since=2024-02-25&until=2024-03-15)


## 4.7

- UI: revert a few changes that were meant for 5.x only.
- Revert "Compile single globals into materialized CTEs", since it introduced multiple bugs. These bugs were already fixed in 5.x. ([#6613](https://github.com/edgedb/edgedb/issues/6613))

- Add optional PKCE challenge in email verification ([#7037](https://github.com/edgedb/edgedb/issues/7037))
- Fix very slow parsing performance of large files with unrecoverable errors ([#7046](https://github.com/edgedb/edgedb/issues/7046))
- Fix update rewrites on types that are children of updated type ([#7073](https://github.com/edgedb/edgedb/issues/7073))

## 4.8

- Use SQL's `ON CONFLICT` to implement `UNLESS CONFLICT` more often ([#7472](https://github.com/edgedb/edgedb/issues/7472))
- Prevent dump hangups from leaving stray Postgres queries. ([#7262](https://github.com/edgedb/edgedb/issues/7262))