1.6. 3.0.x Branch — Apache CouchDB® 3.5 Documentation

# 1.6. 3.0.x Branch

## 1.6.1. Upgrade Notes

- #2228: The default maximum document size has been reduced to 8MB. This means that databases with larger documents will not be able to replicate into CouchDB 3.0 correctly without modification. This change has been made in preparation for anticipated hard upper limits on document size imposed by CouchDB 4.0. For 3.x, the max document size setting can be relaxed via the `[couchdb] max_document_size` config setting.

- #2228: The default database sharding factor `q` has been reduced to 2 by default. This, combined with automated database resharding (see below), is a better starting place for new CouchDB databases. As in CouchDB 2.x, specify `?q=#` to change the value upon database creation if desired. The default can be changed via the config `[cluster] q` setting.

- #1523, #2092, #2336, #2475: The “node-local” HTTP interface, by default exposed on port 5986, has been removed. All functionality previously available at that port is now available on the main, clustered interface (by default, port 5984). Examples:

      GET /_node/{nodename}/_stats
      GET /_node/{nodename}/_system
      GET /_node/{nodename}/_all_dbs
      GET /_node/{nodename}/_uuids
      GET /_node/{nodename}/_config
      GET /_node/{nodename}/_config/couchdb/uuid
      POST /_node/{nodename}_config/_reload
      GET /_node/{nodename}/_nodes/_changes?include_docs=true
      PUT /_node/{nodename}/_dbs/{dbname}
      POST /_node/{nodename}/_restart
      GET /_node/{nodename}/{db-shard}
      GET /_node/{nodename}/{db-shard}/{doc}
      GET /_node/{nodename}/{db-shard}/{ddoc}/_info

  …and so on. Documentation has been updated to reflect this change.

  Warning

  The _node endpoint is for adminstrative purposes it is NOT intended as an alternative to the regular endpoints (“GET /dbname”, “PUT /dbname/docid” and so on)

- #2389: CouchDB 3.0 now requires a server admin user to be defined at startup, or will print an error message and exit. If you do not have one, be sure to create an admin user. (The Admin Party is now over.)

CC-BY-NC 2.0: hehaden @ Flickr

- #2576: CouchDB 3.0 now requires admin-level access for the `/_all_dbs` endpoint.

- #2339: All databases are now created by default as admin-only. That is, the default new database `_security` object is now:

      {
        "members" : { "roles" : [ "_admin" ] },
         "admins" : { "roles" : [ "_admin" ] }
      }

  This can be changed after database creation.

- Due to code changes in #2324, it is not possible to upgrade transparently from CouchDB 1.x to 3.x. In addition, the `couchup` utility has been removed from CouchDB 3.0 by #2399. If you are upgrading from CouchDB 1.x, you must first upgrade to CouchDB 2.3.1 to convert your database and indexes, using `couchup` if desired. You can then upgrade to CouchDB 3.0. Or, you can start a new CouchDB 3.0 installation and replicate directly from 1.x to 3.0.

- #1833, #2358, #1871, #1857: CouchDB 3.0 supports running only under the following Erlang/OTP versions:

  - 19.x - “soft” support only. No longer tested, but should work.

  - 20.x - must be newer than 20.3.8.11 (20.0, 20.1, 20.2 versions all invalid)

  - 21.x - for 21.2, must be newer than 21.2.3

  - 22.x - for 22.0, must be newer than 22.0.5

- #1804: By default, views are limited to return a maximum of 2**28 (268435456) results. This limit can be configured separately for views and partitioned views via the `query_limit` and `partition_query_limit` values in the ini file `[query_server_config]` section.

- After upgrading all nodes in a cluster to 3.0, add `[rexi] use_kill_all = true` to `local.ini` to save some intra-cluster network bandwidth.

### 1.6.1.1. Deprecated feature removal

The following features, deprecated in CouchDB 2.x, have been removed or replaced in CouchDB 3.0:

- #2089, #2128, #2251: Local endpoints for replication targets, which never functioned as expected in CouchDB 2.x, have been completely removed. When replicating databases, always specify a full URL for the source and target. In addition, the node local `_replicator` database is no longer automatically created.

- #2163: The `disk_size` and `data_size` fields have been retired from the database info object returned by `GET /{db}/`. These were deprecated in CouchDB 2.x and replaced by the `sizes` object, which contains the improved `file`, `active` and `external` size metrics. Fauxton has been updated to match.

- #2173: The ability to submit multiple queries against a view using the POST to `/{db}/_design/{ddoc}/_view/{view}` with the `?queries=` option has been replaced by the new queries endpoint. The same is true of the _all_docs, _design_docs, and _local_docs endpoints. Specify a `keys` object when POST-ing to these endpoints.

- #2248: CouchDB externals (`_external/`) have been removed entirely.

- #2208: CouchDB no longer supports the `delayed_commits` option in the configuration file. All writes are now full commits. The `/_ensure_full_commit` API endpoint has been retained (as a no-op) for backwards compatibility with old CouchDB replicators.

- #2395: The security object in the `_users` database cannot be edited by default. A setting exists in the configuration file to revert this behaviour. The ability to override the disable setting is expected to be removed in CouchDB 4.0.

### 1.6.1.2. Deprecated feature warnings

The following features are deprecated in CouchDB 3.0 and will be removed in CouchDB 4.0:

- Show functions (`/{db}/{ddoc}/_show`)

- List functions (`/{db}/{ddoc}/_list`)

- Update functions (`/{db}/{ddoc}/_update`)

- Virtual hosts and ini-file rewrites

- Rewrite functions (`/{db}/{ddoc}/_rewrite`)

## 1.6.2. Version 3.0.1

### 1.6.2.1. Features and Enhancements

- Fauxton was updated to version v1.2.3.

### 1.6.2.2. Bugfixes

- #2441: A memory leak when encoding large binary content was patched. This should resolve a long-standing gradual memory increase bug in CouchDB.

- #2613: Simultaneous attempts to create the same new database should no longer result in a 500 Internal Server Error error.

- #2678: Defaults for the `smoosh` compaction daemon are now consistent with the shipped `default.ini` file.

- #2680: The Windows CouchDB startup batch file will no longer fail to start CouchDB if incompatible versions of OpenSSL are on the `PATH`.

- #2741: A small performance improvement in the `couch_server` process was made.

- #2745: The `require_valid_user` exception logic was corrected.

- #2643: The `users_db_security_editable` setting is now in the correct section of the `default.ini` file.

- #2654: Filtered changes feeds that need to rewind partially should no longer rewind all the way to the beginning of the feed.

- #2655: When deleting a session cookie, CouchDB should now respect the operator-specified cookie domain, if set.

- #2690: Nodes that re-enter a cluster after a database was created (while the node was offline or in maintenance mode) should more correctly handle creating local replicas of that database.

- #2805: Mango operators more correctly handle being passed empty arrays.

- #2716, #2738: The `remsh` utility will now try and guess the node name and Erlang cookie of the local installation. It will also respect the `COUCHDB_ARGS_FILE` environment variable.

- #2797: The cluster setup workflow now uses the correct logging module.

- #2818: Mango now uses a safer method of bookmark creation that prevents unexpectedly creating new Erlang atoms.

- #2756: SpiderMonkey 60+ will no longer corrupt UTF-8 strings when various JS functions are applied to them.

- Multiple test case improvements, including more ports of JS tests to Elixir.

## 1.6.3. Version 3.0.0

### 1.6.3.1. Features and Enhancements

- #1789: User-defined partitioned databases.

  These special databases support user-driven placement of documents into the same shard range. JavaScript views and Mango indexes have specific optimizations for partitioned databases as well.

  Two tweakable configuration parameters exist:

  - #1842: Partition size limits. By default, each partition is limited to 10 GiB.

  - #1684: Partitioned database support can be disabled via feature flag in `default.ini`.

- #1972, #2012: Automated shard splitting. Databases can now be re-sharded while online to increase the `q` factor to a larger number. This can be configured to require specific node and range parameters upon execution.

- #1910: Automatic background indexing, internally known as `ken`. This subsystem ensures secondary indexes (such as JavaScript, Mango, and text search) are kept up to date, without requiring an external query to trigger building them. Many configuration parameters are available.

- #1904: Completely rewritten automatic compaction daemon, internally known as `smoosh`. This subsystem automatically triggers background compaction jobs for both databases and views, based on configurable thresholds.

- #1889, #2408: New IO Queue subsystem implementation. This is highly configurable and well-documented.

- #2436, #2455: CouchDB now regression tests against, and officially supports, running on the `arm64v8` (`aarch64`) and `ppc64le` (`ppc64el`) machine architectures. Convenience binaries are generated on these architectures for Debian 10.x (“buster”) packages, and for the Docker containers.

- #1875, #2437, #2423: CouchDB now supports linking against SpiderMonkey 60 or SpiderMonkey 1.8.5. SpiderMonkey 60 provides enhanced support for ES5, ES6, and ES2016+. Full compatibility information is available at the ECMAScript compatibility table. Click on “Show obsolete platforms”, then look for “FF 60 ESR” in the list of engine types.

  However, it was discovered that on some ARM 64-bit distributions, SM 60 segfaults frequently, including the SM 60 packages on CentOS 8 and Debian 10.

  As a result, CouchDB’s convenience binaries only link against SM 60 on the ``x86_64`` and ``ppc64le`` architectures. This includes the Docker image for these architectures.

  At present, CouchDB ships with SM 60 linked in on the following binary distributions:

  - Debian buster (10.x)

  - CentOS / RedHat 8.x

  - macOS (10.10+)

  - Windows (7+)

  - Docker (3.0.0)

  - FreeBSD (CURRENT)

  We expect to add SM 60 support to Ubuntu with Focal Fossa (20.04 LTS) when it ships in April 2020.

  It is unlikely we will backport SM 60 packages to older versions of Debian, CentOS, RedHat, or Ubuntu.

- The Windows installer has many improvements, including:

  - Prompts for an admin user/password as CouchDB 3.0 requires * Will not overwrite existing credentials if in place

  - No longer remove user-modified config files, closing #1989 * Also will not overwrite them on install.

  - Checkbox to disable installation of the Windows service

  - Silent install support.

  - Friendly link to these online release notes in the exit dialog

  - Higher resolution icon for HiDPI (500x500)

Warning

Windows 8, 8.1, and 10 require the .NET Framework v3.5 to be installed.

- #2037: Dreyfus, the CouchDB side of the Lucene-powered search solution, is now shipped with CouchDB. When one or more Clouseau Java nodes are joined to the cluster, text-based indexes can be enabled in CouchDB. It is recommended to have as many Clouseau nodes as you have CouchDB nodes. Search is advertised in the feature list present at `GET /` if configured correctly (#2206). Configuration and installation documentation is available.

- #2411: The `/_up` endpoint no longer requires authentication, even when `require_valid_user` is `true`.

- #2392: A new `_metrics` role can be given to a user. This allows that user access only to the `/_node/{node}/_stats` and `/_node/{node}/_system` endpoints.

- #1912: A new alternative `systemd-journald logging` backend has been added, and can be enabled through the ini file. The new backend does not include CouchDB’s microsecond-accurate timestamps, and uses the `sd-daemon(3)` logging levels.

- #2296, #1977: If the configuration file setting `[couchdb]` `single_node` is set to `true`, CouchDB will automatically create the system databases on startup if they are not present.

- #2338, #2343: POST request to CouchDB views and the `/{db}/_all_docs`, `/{db}/_local_docs` and `/{db}/_design_docs` endpoints now support the same functionality as GET. Parameters are passed in the body as a JSON object, rather than in the URL when using POST.

- #2292: The `_scheduler/docs` and `_scheduler/info` endpoints now return detailed replication stats for running and pending jobs.

- #2282, #2272, #2290: CouchDB now supports specifying separate proxies for both the `source` and `target` in a replication via `source_proxy` and `target_proxy` keys. The API documentation has been updated.

- #2240: Headers are now returned from the `/{db}/_changes` feed immediately, even when there are no changes available. This avoids client blocking.

- #2005, #2006: The name of any node can now be retrieved through the new API endpoint `GET /_node/{node-name}`.

- #1766: Timeouts for requests, `all_docs`, attachments, views, and partitioned view requests can all be specified separately in the ini file under the `[fabric]` section. See `default.ini` for more detail.

- #1963: Metrics are now kept on the number of partition and global view queries, along with the number of timeouts that occur.

- #2452, #2221: A new configuration field `[couch_httpd_auth] same_site` has been added to set the value of the CouchDB auth cookie’s `SameSite` attribute. It may be necessary to set this to `strict` for compatibility with future versions of Google Chrome. If CouchDB CORS support is enabled, set this to `None`.

### 1.6.3.2. Performance

- #2277: The `couch_server` process has been highly optimized, supporting significantly more load than before.

- #2360: It is now possible to make the rexi interface’s unacked message limit configurable. A new, more optimized default (5, lowered from 10) has been set. This results in a ~50% improvement on view queries on large clusters with `q ≥ 8`.

- #2280: Connection sharing for replication now functions correctly when replicating through a forward proxy. Closes #2271.

- #2195, #2207: Metrics aggregation now supports CouchDB systems that sleep or hibernate, ensuring that on wakeup does not trigger thousands of unnecessary function calls.

- #1795: Avoid calling `fabric:update_docs` with empty doc lists.

- #2497: The setup wizard no longer automatically creates the `_global_changes` database, as the majority of users do not need this functionality. This reduces overall CouchDB load.

### 1.6.3.3. Bugfixes

- #1752, #2398, #1803: The cluster setup wizard now ensures a consistent UUID and http secret across all nodes in a cluster. CouchDB admin passwords are also synced when the cluster setup wizard is used. This prevents being logged out when using Fauxton as a server admin user through a load balancer.

- #2388: A compatibility change has been made to support replication with future databases containing per-document access control fields.

- #2379: Any replicator error messages will provide an object in the response, or null, but never a string.

- #2244, #2310: CouchDB will no longer send more data than is requested when retrieving partial attachment data blocks.

- #2138: Manual operator updates to a database’s shard map will not corrupt additional database properties, such as partitioning values.

- #1877: The `_purge` and `_purged_infos_limit` endpoints are now correctly restricted to server admin only.

- #1794: The minimum purge sequence value for a database is now gathered without a clustered `_all_docs` lookup.

- #2351: A timeout case clause in `fabric_db_info` has been normalised to match other case clauses.

- #1897: The `/{db}/_bulk_docs` endpoint now correctly catches invalid (i.e., non-hexadecimal) `_rev_` values and responds with a 400 Bad Request error.

- #2321: CouchDB no longer requires Basic auth credentials to reach the `/_session` endpoint for login, even when `require_valid_user` is enabled.

- #2295: CouchDB no longer marks a job as failed permanently if the internal doc processor crashes.

- #2178: View compaction files are now removed on view cleanup.

- #2179: The error message logged when CouchDB does not have a `_users` database is now less scary.

- #2153: CouchDB no longer may return a `badmatch` error when querying `all_docs` with a passed `keys` array.

- #2137: If search is not available, return a 400 Bad Request instead of a 500 Internal Server Error status code.

- #2077: Any failed `fsync(2)` calls are now correctly raised to avoid data corruption arising from retry attempts.

- #2027: Handle epoch mismatch when duplicate UUIDs are created through invalid operator intervention.

- #2019: If a database is deleted and re-created while internal cluster replication is still active, CouchDB will no longer retry to delete it continuously.

- #2003, #2438: CouchDB will no longer automatically reset an index file if any attempt to read its header fails (such as when the couch_file process terminates unexpectedly). CouchDB now also handles the case when a view file lacks a proper header.

- #1983: Improve database “external” size calcuation to be more precise.

- #1971: Correctly compare ETags using weak comparison methods to support W/ prefix added by some load balancer configurations.

- #1901: Invalid revision specified for a document update will no longer result in a `badarg` crash.

- #1845: The `end_time` field in `/_replicate` now correctly converts time to UTC.

- #1824: `rexi` stream workers are now cleaned up when the coordinator process is killed, such as when the ddoc cache is refreshed.

- #1770: Invalid database `_security` objects no longer return a `function_clause` error and stack trace.

- #2412: Mango execution stats now correctly count documents read which weren’t followed by a match within a given shard.

- #2393, #2143: It is now possible to override the query server environment variables `COUCHDB_QUERY_SERVER_JAVASCRIPT` and `COUCHDB_QUERY_SERVER_COFFEESCRIPT` without overwriting the `couchdb`/`couchdb.cmd` startup scripts.

- #2426, #2415: The replicator now better handles the situation where design document writes to the target fail when replicating with non-admin credentials.

- #2444, #2413: Replicator error messages are now significantly improved, reducing `function_clause` responses.

- #2454: The replication auth session plugin now ignores other cookies it may receive without logging an error.

- #2458: Partitioned queries and dreyfus search functions no longer fail if there is a single failed node or rexi worker error.

- #1783: Mango text indexes no longer error when given an empty selector or operators with empty arrays.

- #2466: Mango text indexes no longer error if the indexed document revision no longer exists in the primary index.

- #2486: The `$lt`, `$lte`, `$gt`, and `$gte` Mango operators are correctly quoted internally when used in conjunction with a text index search.

- #2493: The `couch_auth_cache` no longer has a runaway condition in which it creates millions of monitors on the `_users` database.

### 1.6.3.4. Other

The 3.0.0 release also includes the following minor improvements:

- #2472: CouchDB now logs the correct, clustered URI at startup (by default: port `5984`.)

- #2034, #2416: The path to the Fauxton installation can now be specified via the `COUCHDB_FAUXTON_DOCROOT` environment variable.

- #2447: Replication stats are both persisted when jobs are re-created, as well as properly handled when bulk document batches are split.

- #2410, #2390, #1913: Many metrics were added for Mango use, including counts of unindexed queries, invalid index queries, docs examined that do and don’t meet cluster quorum, query time, etc.

- #2152, #2504: CouchDB can now be started via a symlink to the binary on UNIX-based platforms.

- #1844: A new internal API has been added to write custom Erlang request-level metrics reporting plugins.

- #2293, #1095: The `-args_file`, `-config` and `-couch_ini` parameters may now be overridden via the `COUCHDB_INI_FILES` environment variable on UNIX-based systems.

- #2352: The `remsh` utility now searches for the Erlang cookie in `ERL_FLAGS` as well as `vm.args`.

- #2324: All traces of the (never fully functional) view-based `_changes` feed have been expunged from the code base.

- #2337: The md5 shim (introduced to support FIPS-compliance) is now used consistently throughout the code base.

- #2270: Negative and non-integer `heartbeat` values now return 400 Bad Request.

- #2268: When rescheduling jobs, CouchDB now stops sufficient running jobs to make room for the pending jobs.

- #2186: CouchDB plugin writers have a new field in which endpoint credentials may be stashed for later use.

- #2183: `dev/run` now supports an `--extra-args` flag to modify the Erlang runtime environment during development.

- #2105: `dev/run` no longer fails on unexpected remote end connection close during cluster setup.

- #2118: Improve `couch_epi` process replacement mechanism using map childspecs functionality in modern Erlang.

- #2111: When more than `MaxJobs` replication jobs are defined, CouchDB now correctly handles job rotation when some jobs crash.

- #2020: Fix full ring assertion in fabric stream shard replacements

- #1925: Support list for docid when using `couch_db:purge_docs/3`.

- #1642: `io_priority` is now set properly on view update and compaction processes.

- #1865: Purge now supports >100 document IDs in a single request.

- #1861: The `vm.args` file has improved commentary.

- #1808: Pass document update type for additional checks in `before_doc_update`.

- #1835: Module lists are no longer hardcoded in `.app` files.

- #1798, #1933: Multiple compilation warnings were eliminated.

- #1826: The `couch_replicator_manager` shim has been fully removed.

- #1820: After restarting CouchDB, JS and Elixir tests now wait up to 30s for it to be ready before timing out.

- #1800: `make elixir` supports specifying individual tests to run with `tests=`.

- #1805: `dev/run` supports `--with-haproxy` again.

- #1774: `dev/run` now supports more than 3 nodes.

- #1779: Refactor Elixir test suite initialization.

- #1769: The Elixir test suite uses Credo for static analysis.

- #1776: All Python code is now formatted using Python black.

- #1786: `dev/run`: do not create needless `dev/data/` directory.

- #2482: A redundant `get_ring_opts` call has been removed from `dreyfus_fabric_search`.

- #2506: CouchDB’s release candidates no longer propagate the RC tags into each Erlang application’s version string.

- #2511: recon, the Erlang diagnostic toolkit, has been added to CouchDB’s build process and ships in the release + convenience binaries.

- Fauxton updated to v1.2.3, which includes:

  - Support multiple server-generated warnings when running queries

  - Partitioned database support

  - Search index support

  - Remove references to deprecated dbinfo fields

  - Improve accessibility for screen readers

  - Numerous CSS fixes

- Improved test cases:

  - Many, many test race conditions and bugs have been removed (PR list too long to include here!)

  - More test cases were ported to Elixir, including:

    - Cluster with and without quorum tests (#1812)

    - `delayed_commits` (#1796)

    - `multiple_rows` (#1958)

    - `invalid_docids` (#1968)

    - `replication` (#2090)

    - All `attachment_*` tests (#1999)

    - `copy_doc` (#2000)

    - `attachments` (#1953)

    - `erlang_views` (#2237)

    - `auth_cache`, `cookie_auth`, `lorem*`, `multiple_rows`, `users_db`, `utf8` (#2394)

    - `etags_head` (#2464, #2469)

  - #2431: `chttpd_purge_tests` have been improved in light of CI failures.

  - #2432: Address flaky test failure on `t_invalid_view/1`.

  - #2363: Elixir tests now run against a single node cluster, in line with the original design of the JavaScript test suite. This is a permanent change.

  - #1893: Add “w:3” for lots of doc tests.

  - #1939, #1931: Multiple fixes to improve support in constrained CI environments.

  - #2346: Big-endian support for the `couch_compress` tests.

  - #2314: Do not auto-index when testing `update=false` in Mango.

  - #2141: Fix `couch_views` encoding test.

  - #2123: Timeout added for `fold_docs-with_different_keys` test.

  - #2114: EUnit tests now correctly inherit necessary environment variables.

  - #2122: :meck.unload() is now called automatically after every test.

  - #2098: Fix `cpse_test_purge_replication` eunit test.

  - #2085, #2086: Fix a flaky `mem3_sync_event_listener` test.

  - #2084: Increase timeouts on two slow btree tests.

  - #1960, #1961: Fix for `chttpd_socket_buffer_size_test`.

  - #1922: Tests added for shard splitting functionality.

  - #1869: New test added for doc reads with etag `If-None-Match` header.

  - #1831: Re-introduced cpse_test_purge_seqs test.

  - #1790: Reorganise `couch_flag_config_tests` into a proper suite.

  - #1785: Use `devclean` on elixir target for consistency of Makefile.

  - #2476: For testing, `Triq` has been replaced with `PropEr` as an optional dependency.

- External dependency updates:

  - #1870: Mochiweb has been updated to 2.19.0.

  - #1938: Folsom has been updated to 0.8.3.

  - #2001: ibrowse has been updated to 4.0.1-1.

  - #2400: jiffy has been updated to 1.0.1.

- A llama! OK, no, not really. If you got this far…thank you for reading.