single_node=true setting in local.ini does not auto-create system databases.

[willemodendaal]

Description

I run a CouchDb server in a local container (for local development only). I expect the server to automatically create the system databases on startup (eg. "_users") , because I have the [couchdb] single_node=true setting in local.ini

According to the documentation, this should work. But when my server starts up, there are no system databases (even after waiting several minutes).

I don't want to initialize the system dbs using the web interface, because I need to automate creation of the server.

Steps to Reproduce

Use the following dockerfile:

FROM couchdb:3.1
WORKDIR /src
COPY ./mycustom.ini .
COPY ./local.ini .

RUN cp mycustom.ini /opt/couchdb/etc/local.d/
RUN cp local.ini /opt/couchdb/etc/

And use the following local.ini file (on local machine, copied into the container). A duplicate of the default couchdb ini file, with added single_node line:

[couchdb]
single_node=true

[couch_peruser]

[chttpd]

[httpd]

[couch_httpd_auth]

[ssl]

[vhosts]

[admins]

And use the following mycustom.ini file to use different default port. (I attempted to set single_node=true here as well, but no luck)

[couchdb] 
single_node=true 

[httpd]
enable_cors = true

[chttpd]
port = 31986

Then build and run the container.

Then open the couchDb Fauxton at http://localhost:31986/_utils/ and view list of databases.

Expected Behaviour

The _users database should exist. But no databases exist:

Your Environment

• CouchDB version used:

  • {"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"e6520921195ec9c7db9e6c0baecebc1b","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

• Browser name and version:

  • Chrome v86.0.4240.183 (Official Build) (64-bit)

• Operating system and version:

  • Microsoft Windows 10 Pro, 10.0.19041 N/A Build 19041
  • Docker version 19.03.13, build 4484c46d9d
  • Kubernetes version: Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"windows/amd64"}
    Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Additional Context

The couchDb container logs:

[info] 2020-11-10T13:05:33.247999Z nonode@nohost <0.11.0> -------- Application couch_log started on node nonode@nohost
[info] 2020-11-10T13:05:33.259311Z nonode@nohost <0.11.0> -------- Application folsom started on node nonode@nohost
[info] 2020-11-10T13:05:33.311392Z nonode@nohost <0.11.0> -------- Application couch_stats started on node nonode@nohost
[info] 2020-11-10T13:05:33.311526Z nonode@nohost <0.11.0> -------- Application khash started on node nonode@nohost
[info] 2020-11-10T13:05:33.322683Z nonode@nohost <0.11.0> -------- Application couch_event started on node nonode@nohost
[info] 2020-11-10T13:05:33.322819Z nonode@nohost <0.11.0> -------- Application hyper started on node nonode@nohost
[info] 2020-11-10T13:05:33.331710Z nonode@nohost <0.11.0> -------- Application ibrowse started on node nonode@nohost
[info] 2020-11-10T13:05:33.340585Z nonode@nohost <0.11.0> -------- Application ioq started on node nonode@nohost
[info] 2020-11-10T13:05:33.340716Z nonode@nohost <0.11.0> -------- Application mochiweb started on node nonode@nohost
[info] 2020-11-10T13:05:33.346874Z nonode@nohost <0.216.0> -------- Preflight check: Asserting Admin Account

[info] 2020-11-10T13:05:33.350830Z nonode@nohost <0.216.0> -------- Apache CouchDB 3.1.1 is starting.

[info] 2020-11-10T13:05:33.350889Z nonode@nohost <0.217.0> -------- Starting couch_sup
[notice] 2020-11-10T13:05:33.386403Z nonode@nohost <0.82.0> -------- config: [admins] admin set to redacted for reason nil
[info] 2020-11-10T13:05:33.435463Z nonode@nohost <0.216.0> -------- Apache CouchDB has started. Time to relax.

[info] 2020-11-10T13:05:33.438418Z nonode@nohost <0.11.0> -------- Application couch started on node nonode@nohost
[info] 2020-11-10T13:05:33.438559Z nonode@nohost <0.11.0> -------- Application ets_lru started on node nonode@nohost
[notice] 2020-11-10T13:05:33.456718Z nonode@nohost <0.250.0> -------- rexi_server : started servers
[notice] 2020-11-10T13:05:33.460052Z nonode@nohost <0.254.0> -------- rexi_buffer : started servers
[info] 2020-11-10T13:05:33.460249Z nonode@nohost <0.11.0> -------- Application rexi started on node nonode@nohost
[warning] 2020-11-10T13:05:33.496314Z nonode@nohost <0.262.0> -------- creating missing database: _nodes
[info] 2020-11-10T13:05:33.496376Z nonode@nohost <0.263.0> -------- open_result error {not_found,no_db_file} for _nodes
[warning] 2020-11-10T13:05:33.590652Z nonode@nohost <0.278.0> -------- creating missing database: _dbs
[warning] 2020-11-10T13:05:33.590694Z nonode@nohost <0.277.0> -------- creating missing database: _dbs
[info] 2020-11-10T13:05:33.590730Z nonode@nohost <0.280.0> -------- open_result error {not_found,no_db_file} for _dbs
[notice] 2020-11-10T13:05:33.618702Z nonode@nohost <0.292.0> -------- mem3_reshard_dbdoc start init()
[notice] 2020-11-10T13:05:33.624984Z nonode@nohost <0.294.0> -------- mem3_reshard start init()
[notice] 2020-11-10T13:05:33.625040Z nonode@nohost <0.295.0> -------- mem3_reshard db monitor <0.295.0> starting
[notice] 2020-11-10T13:05:33.628318Z nonode@nohost <0.294.0> -------- mem3_reshard starting reloading jobs
[notice] 2020-11-10T13:05:33.628361Z nonode@nohost <0.294.0> -------- mem3_reshard finished reloading jobs
[info] 2020-11-10T13:05:33.629079Z nonode@nohost <0.11.0> -------- Application mem3 started on node nonode@nohost
[info] 2020-11-10T13:05:33.629182Z nonode@nohost <0.11.0> -------- Application fabric started on node nonode@nohost
[info] 2020-11-10T13:05:33.662937Z nonode@nohost <0.11.0> -------- Application chttpd started on node nonode@nohost
[info] 2020-11-10T13:05:33.675574Z nonode@nohost <0.11.0> -------- Application couch_index started on node nonode@nohost
[info] 2020-11-10T13:05:33.675639Z nonode@nohost <0.11.0> -------- Application couch_mrview started on node nonode@nohost
[info] 2020-11-10T13:05:33.675743Z nonode@nohost <0.11.0> -------- Application couch_plugins started on node nonode@nohost
[notice] 2020-11-10T13:05:33.678567Z nonode@nohost <0.324.0> -------- chttpd_auth_cache changes listener died because the _users database does not exist. Create the database to silence this notice.
[error] 2020-11-10T13:05:33.678665Z nonode@nohost emulator -------- Error in process <0.325.0> with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,374}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,403}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,198}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,145}]}]}

[info] 2020-11-10T13:05:33.719995Z nonode@nohost <0.11.0> -------- Application couch_replicator started on node nonode@nohost
[info] 2020-11-10T13:05:33.750550Z nonode@nohost <0.11.0> -------- Application couch_peruser started on node nonode@nohost
[info] 2020-11-10T13:05:33.792142Z nonode@nohost <0.11.0> -------- Application ddoc_cache started on node nonode@nohost
[info] 2020-11-10T13:05:33.804017Z nonode@nohost <0.11.0> -------- Application dreyfus started on node nonode@nohost
[info] 2020-11-10T13:05:33.824547Z nonode@nohost <0.11.0> -------- Application global_changes started on node nonode@nohost
[info] 2020-11-10T13:05:33.824584Z nonode@nohost <0.11.0> -------- Application jiffy started on node nonode@nohost
[info] 2020-11-10T13:05:33.840372Z nonode@nohost <0.11.0> -------- Application jwtf started on node nonode@nohost
[info] 2020-11-10T13:05:33.869704Z nonode@nohost <0.11.0> -------- Application ken started on node nonode@nohost
[info] 2020-11-10T13:05:33.891545Z nonode@nohost <0.11.0> -------- Application mango started on node nonode@nohost
[info] 2020-11-10T13:05:33.900346Z nonode@nohost <0.11.0> -------- Application setup started on node nonode@nohost
[info] 2020-11-10T13:05:33.936550Z nonode@nohost <0.11.0> -------- Application smoosh started on node nonode@nohost
[info] 2020-11-10T13:05:33.936570Z nonode@nohost <0.11.0> -------- Application snappy started on node nonode@nohost
[info] 2020-11-10T13:05:33.936590Z nonode@nohost <0.11.0> -------- Application recon started on node nonode@nohost
[notice] 2020-11-10T13:05:38.687870Z nonode@nohost <0.324.0> -------- chttpd_auth_cache changes listener died because the _users database does not exist. Create the database to silence this notice.
[error] 2020-11-10T13:05:38.693616Z nonode@nohost emulator -------- Error in process <0.463.0> with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,374}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,403}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,198}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,145}]}]}

[notice] 2020-11-10T13:05:38.693711Z nonode@nohost <0.346.0> -------- couch_replicator_clustering : cluster stable
[notice] 2020-11-10T13:05:38.767402Z nonode@nohost <0.356.0> -------- Started replicator db changes listener <0.464.0>
[info] 2020-11-10T13:05:38.768057Z nonode@nohost <0.466.0> -------- open_result error {not_found,no_db_file} for _replicator
[notice] 2020-11-10T13:05:43.690260Z nonode@nohost <0.324.0> -------- chttpd_auth_cache changes listener died because the _users database does not exist. Create the database to silence this notice.
[error] 2020-11-10T13:05:43.690398Z nonode@nohost emulator -------- Error in process <0.518.0> with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,374}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,403}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,198}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,145}]}]}

[notice] 2020-11-10T13:05:48.457433Z nonode@nohost <0.250.0> -------- rexi_server : cluster stable
[notice] 2020-11-10T13:05:48.457475Z nonode@nohost <0.254.0> -------- rexi_buffer : cluster stable
[notice] 2020-11-10T13:05:48.691218Z nonode@nohost <0.324.0> -------- chttpd_auth_cache changes listener died because the _users database does not exist. Create the database to silence this notice.
[error] 2020-11-10T13:05:48.691335Z nonode@nohost emulator -------- Error in process <0.551.0> with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,374}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,403}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,198}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,145}]}]}

[notice] 2020-11-10T13:05:53.692250Z nonode@nohost <0.324.0> -------- chttpd_auth_cache changes listener died because the _users database does not exist. Create the database to silence this notice.
[error] 2020-11-10T13:05:53.692382Z nonode@nohost emulator -------- Error in process <0.600.0> with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,374}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,403}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,198}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,145}]}]}

[wohali]

I cannot reproduce this. With the two files provided in a temp directory:

$ cat local.ini
[couchdb]
single_node=true

[couch_peruser]

[chttpd]

[httpd]

[couch_httpd_auth]

[ssl]

[vhosts]

[admins]
$ cat mycustom.ini
[couchdb]
single_node=true

[httpd]
enable_cors = true

[chttpd]
port = 31986
$ cat Dockerfile
FROM couchdb:3.1
WORKDIR /src
COPY ./mycustom.ini .
COPY ./local.ini .

RUN cp mycustom.ini /opt/couchdb/etc/local.d/
RUN cp local.ini /opt/couchdb/etc/
$
$
$ docker build -t testy .
Sending build context to Docker daemon  4.096kB
Step 1/6 : FROM couchdb:3.1
3.1: Pulling from library/couchdb
bb79b6b2107f: Pull complete
64047dc3beae: Pull complete
1bf5219be029: Pull complete
fe133310c07e: Pull complete
605d5fefdc3d: Pull complete
38f02fa87302: Pull complete
f0d636474460: Pull complete
5bf601c5e86b: Pull complete
7540b73bba44: Pull complete
cb1d0befd4e1: Pull complete
9d632d0fb7a8: Pull complete
Digest: sha256:b5e6939af792a0e40218e405e8656d68c14a03220899f470b58611a00dc65a9a
Status: Downloaded newer image for couchdb:3.1
 ---> 015b6fbd87be
Step 2/6 : WORKDIR /src
 ---> Running in f5afabacceb3
Removing intermediate container f5afabacceb3
 ---> 8c139411f4ac
Step 3/6 : COPY ./mycustom.ini .
 ---> 690379b5120a
Step 4/6 : COPY ./local.ini .
 ---> 8e018ae2a62d
Step 5/6 : RUN cp mycustom.ini /opt/couchdb/etc/local.d/
 ---> Running in 8efee9b50378
Removing intermediate container 8efee9b50378
 ---> f541543d48c0
Step 6/6 : RUN cp local.ini /opt/couchdb/etc/
 ---> Running in 12c1718b78a7
Removing intermediate container 12c1718b78a7
 ---> 142e9cdd7b71
Successfully built 142e9cdd7b71
Successfully tagged testy:latest

I then launch the container using docker run -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password testy.

To be 100% sure I am talking to the container's CouchDB, I run a shell inside of it:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                          NAMES
e975a389d9cd        testy               "tini -- /docker-ent…"   27 seconds ago      Up 26 seconds       4369/tcp, 5984/tcp, 9100/tcp   modest_jones
$ docker exec -it e975a389d9cd /bin/bash
root@e975a389d9cd:/src# curl localhost:31986
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"e904db056debb1588ddf46a91080c86b","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
root@e975a389d9cd:/src# curl admin:password@localhost:31986/_all_dbs
["_replicator","_users"]

There is a slight delay in creation of those databases, but this is expected for CouchDB.

[willemodendaal]

That's strange... I recreated my cluster to make sure I started from a clean slate, but I had the same issue.

Then I tried copy-pasting your files as-is in a new temp directory, and it works perfectly. So there must be something wrong with my original files. No sure what, but will continue digging...

[willemodendaal]

OK, I got it working. I removed comments from the ini files when I shared them on this issue (local.ini has a lot of comments by default).

My mycustom.ini file really looked like this, and the inline comment was the problem:

[couchdb]
single_node=true  ; We set this so that system databases are created automatically on startup.

[httpd]
enable_cors = true

[chttpd]
port = 31986

The issue was that there was two spaces before the ";" character, so I imagine couchDb must have interpreted that value as "true " (with a space at the end).

Not sure if that's something that needs to be solved or not, or maybe just mentioned in the docs?

[kocolosk]

Hah, nice find! That's a bona fide bug and one that has been in the codebase for at least 9 years I imagine:

https://github.com/apache/couchdb-config/blob/main/src/config.erl#L413-L414

We'll allow a single trailing space, or a trailing tab, but we're not greedy about consuming the whitespace, and as you suspected the system fails to convert "true " to the atom true. I think we should add a test for this and fix it properly.

[willemodendaal]

Yay, I discovered a fossil!

[kocolosk]

Filed #3259 for this bug.

[guanfeix]

Filed #3259 for this bug.
@kocolosk
when i don't create these system db _users,_replicator, what will happens, can i use the couchdb normally.i notice the couchdb instance seems have a crush.