Invalid purge doc - how to remove? #3219

VasylSlobodyanyuk · 2020-10-20T08:13:55Z

VasylSlobodyanyuk
Oct 20, 2020

Hi Team,

I have a Cluster setup with q=8 & n=3.

After upgrade to CouchDB version 3.1.1, I constantly receive such errors:

[error] 2020-10-20T07:53:19.591169Z couchdb@ip-10-0-20-78.ec2.internal <0.17929.2391> -------- Invalid purge doc '_local/purge-mrview-cee5e6e4f79f51d1acefc9b6022978ed' on <<"shards/80000000-9fffffff/operational-data.1578561147">> with purge_seq '0'

In docs at https://docs.couchdb.org/en/latest/cluster/purging.html states that:
"If this type of log warning occurs, remove the local purge doc from a database."

The question is how would I remove those 'mrview' purge docs if they do not exist in the system (CentOS)?

I receive about 15k errors per node per day. The amount of errors is constant.
Appreciate for any suggestions.

Thanks,
Vasyl

Answered by jiangphcn

Oct 23, 2020

The following steps might help to delete local document in shard database using remsh.
Given we have db1 database with _local/purge-mrview-e17fdd1b24a0f0637d95e6546e6bfa97 document in shards/00000000-7fffffff/db1.1603437987.couch

Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe] [dtrace]

Eshell V10.4.4  (abort with ^G)
(node1@127.0.0.1)1> rr(mem3).
[btree,changes_args,doc,doc_info,extern_resp_args,
 fabric_changes_acc,full_doc_info,httpd,index_header,leaf,
 ordered_shard,proc,reduce_fold_helper_funs,rev_info,shard,
 size_info,user_ctx,view_fold_helper_funs]
(node1@127.0.0.1)2> DB1 = <<"shards/00000000-7fffffff/db1.1603437987">>.
<<"shards/00…

View full answer

jiangphcn · 2020-10-20T13:01:49Z

jiangphcn
Oct 20, 2020
Collaborator

The question is how would I remove those 'mrview' purge docs if they do not exist in the system (CentOS)?

Could you please check whether the document '_local/purge-mrview-cee5e6e4f79f51d1acefc9b6022978ed' exists on database <<"shards/80000000-9fffffff/operational-data.1578561147">>? If so, you can log on to remsh to use couch_db:delete_doc(Db, DocId, Revs) to delete document on specific shard database.

1 reply

VasylSlobodyanyuk Oct 20, 2020
Author

May I delete it just here via Fauxton?

http://10.0.20.78:5984/_utils/#database/operational-data/_local/purge-mrview-cee5e6e4f79f51d1acefc9b6022978ed

jiangphcn · 2020-10-21T03:19:18Z

jiangphcn
Oct 21, 2020
Collaborator

May I delete it just here via Fauxton?

Yes, you can delete from fauxton or use curl like command to delete it. To do so, this local document will be deleted from multiple shard databases.

1 reply

VasylSlobodyanyuk Oct 21, 2020
Author

I have deleted '_local/purge-mrview-cee5e6e4f79f51d1acefc9b6022978ed' via Fauxton and now it says that document does not exist

Same via CLI:
[root@ip-10-0-20-78 ~]# curl -u user:password -X GET http://10.0.20.78:5984/operational-data/_local/purge-mrview-cee5e6e4f79f51d1acefc9b6022978ed
{"error":"not_found","reason":"missing"}

and the same for each node.

However, I still receive these errors:

_[error] 2020-10-21T11:15:10.804302Z couchdb@ip-10-0-20-78.ec2.internal <0.28742.6279> -------- Invalid purge doc '_local/purge-mrview-cee5e6e4f79f51d1acefc9b6022978ed' on <<"shards/a0000000-bfffffff/operational-data.1578561147">> with purge_seq '0'
[error] 2020-10-21T11:15:10.805147Z couchdb@ip-10-0-20-78.ec2.internal <0.28742.6279> -------- Invalid purge doc '_local/purge-mrview-d29039366ea2f9fd90c159ddca220bdf' on <<"shards/a0000000-bfffffff/operational-data.1578561147">> with purge_seq '611839'
[error] 2020-10-21T10:41:12.899550Z couchdb@ip-10-0-20-78.ec2.internal <0.11015.6215> -------- Invalid purge doc 'local/purge-mrview-d29039366ea2f9fd90c159ddca220bdf' on <<"shards/80000000-9fffffff/operational-data.1578561147">> with purge_seq '610429'

When trying to get info on that new mrview - receiving same error:
[root@ip-10-0-20-78 ~]# curl -u user:password -X GET http://10.0.20.78:5984/operational-data/_local/purge-mrview-d29039366ea2f9fd90c159ddca220bdf
{"error":"not_found","reason":"missing"}

So why these errors still come up when the document doesn't exist in the database?
Please suggest how can I resolve those errors appearance.

Thanks,
Vasyl

jiangphcn · 2020-10-23T07:36:02Z

jiangphcn
Oct 23, 2020
Collaborator

The following steps might help to delete local document in shard database using remsh.
Given we have db1 database with _local/purge-mrview-e17fdd1b24a0f0637d95e6546e6bfa97 document in shards/00000000-7fffffff/db1.1603437987.couch

Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe] [dtrace]

Eshell V10.4.4  (abort with ^G)
(node1@127.0.0.1)1> rr(mem3).
[btree,changes_args,doc,doc_info,extern_resp_args,
 fabric_changes_acc,full_doc_info,httpd,index_header,leaf,
 ordered_shard,proc,reduce_fold_helper_funs,rev_info,shard,
 size_info,user_ctx,view_fold_helper_funs]
(node1@127.0.0.1)2> DB1 = <<"shards/00000000-7fffffff/db1.1603437987">>.
<<"shards/00000000-7fffffff/db1.1603437987">>
(node1@127.0.0.1)3> {ok, Db} = couch_db:open_int(DB1, []).
{ok,{db,1,<<"shards/00000000-7fffffff/db1.1603437987">>,
        "/Users/jiangph/couchdb/couchdb.0208/couchdb/dev/lib/node1/data/shards/00000000-7fffffff/db1.1603437987.couch",
        {couch_bt_engine,{st,"/Users/jiangph/couchdb/couchdb.0208/couchdb/dev/lib/node1/data/shards/00000000-7fffffff/db1.1603437987.couch",
                             <0.16296.0>,#Ref<0.2133496106.1599340545.157555>,undefined,
                             {db_header,8,3,0,
                                        {20802,{3,0,#size_info{active = 384,external = 209}},216},
                                        {21018,3,213},
                                        {12476,[],190},
                                        nil,nil,4253,1000,<<"eb47b2ec"...>>,
                                        [{...}],
                                        0,...},
                             false,
                             #btree{fd = <0.16296.0>,
                                    root = {20802,
                                            {3,0,#size_info{active = 384,external = 209}},
                                            216},
                                    extract_kv = fun couch_bt_engine:id_tree_split/1,
                                    assemble_kv = fun couch_bt_engine:id_tree_join/2,
                                    less = undefined,
                                    reduce = fun couch_bt_engine:id_tree_reduce/2,
                                    compression = snappy},
                             #btree{fd = <0.16296.0>,
                                    root = {21018,3,213},
                                    extract_kv = fun couch_bt_engine:seq_tree_split/1,
                                    assemble_kv = fun couch_bt_engine:seq_tree_join/2,
                                    less = undefined,
                                    reduce = fun couch_bt_engine:seq_tree_reduce/2,
                                    compression = snappy},
                             #btree{fd = <0.16296.0>,
                                    root = {12476,[],190},
                                    extract_kv = fun couch_bt_engine:local_tree_split/1,
                                    assemble_kv = fun couch_bt_engine:local_tree_join/2,
                                    less = undefined,reduce = nil,compression = snappy},
                             snappy,
                             #btree{fd = <0.16296.0>,root = nil,
                                    extract_kv = fun couch_bt_engine:purge_tree_split/1,
                                    assemble_kv = fun couch_bt_engine:purge_tree_join/2,
                                    less = undefined,
                                    reduce = fun couch_bt_engine:purge_tree_reduce/2,
                                    compression = snappy},
                             #btree{fd = <0.16296.0>,root = nil,
                                    extract_kv = fun couch_bt_engine:purge_seq_tree_split/1,
                                    assemble_kv = fun couch_bt_engine:purge_seq_tree_join/2,
                                    less = undefined,
                                    reduce = fun couch_bt_engine:purge_tree_reduce/2,...}}},
        <0.16295.0>,nil,3,<<"1603438383093613">>,
        #user_ctx{name = null,roles = [],handler = undefined},
        [{<<"members">>,{[{<<"roles">>,[<<"_admin">>]}]}},
         {<<"admins">>,{[{<<"roles">>,[<<"_admin">>]}]}}],
        undefined,nil,nil,undefined,
        [{default_security_object,[{<<"members">>,
                                    {[{<<"roles">>,[<<...>>]}]}},
                                   {<<"admins">>,{[{<<"role"...>>,[...]}]}}]},
         {props,[]}],
        undefined}}
(node1@127.0.0.1)4> Doc = couch_db:open_doc(Db, <<"_local/purge-mrview-e17fdd1b24a0f0637d95e6546e6bfa97">>).
{ok,#doc{id = <<"_local/purge-mrview-e17fdd1b24a0f0637d95e6546e6bfa97">>,
         revs = {0,[<<"1">>]},
         body = {[{<<"type">>,<<"mrview">>},
                  {<<"purge_seq">>,0},
                  {<<"updated_on">>,1603437987},
                  {<<"ddoc_id">>,<<"_design/bar">>},
                  {<<"signature">>,"e17fdd1b24a0f0637d95e6546e6bfa97"}]},
         atts = [],deleted = false,meta = []}}
(node1@127.0.0.1)6> {ok, DocInfo} = Doc.
{ok,#doc{id = <<"_local/purge-mrview-e17fdd1b24a0f0637d95e6546e6bfa97">>,
         revs = {0,[<<"1">>]},
         body = {[{<<"type">>,<<"mrview">>},
                  {<<"purge_seq">>,0},
                  {<<"updated_on">>,1603437987},
                  {<<"ddoc_id">>,<<"_design/bar">>},
                  {<<"signature">>,"e17fdd1b24a0f0637d95e6546e6bfa97"}]},
         atts = [],deleted = false,meta = []}}
(node1@127.0.0.1)7> Doc1 =DocInfo#doc{deleted=true}.
#doc{id = <<"_local/purge-mrview-e17fdd1b24a0f0637d95e6546e6bfa97">>,
     revs = {0,[<<"1">>]},
     body = {[{<<"type">>,<<"mrview">>},
              {<<"purge_seq">>,0},
              {<<"updated_on">>,1603437987},
              {<<"ddoc_id">>,<<"_design/bar">>},
              {<<"signature">>,"e17fdd1b24a0f0637d95e6546e6bfa97"}]},
     atts = [],deleted = true,meta = []}
(node1@127.0.0.1)8> couch_db:update_doc(Db, Doc1, []).
{ok,{0,<<"0">>}}

0 replies

VasylSlobodyanyuk · 2020-10-26T12:52:30Z

VasylSlobodyanyuk
Oct 26, 2020
Author

Thanks for the detailed example. I have followed it and do not see any more errors for a specific doc.
The next challenge is to mark about 800 docs in total as deleted=true.

0 replies

VasylSlobodyanyuk · 2020-10-26T19:32:14Z

VasylSlobodyanyuk
Oct 26, 2020
Author

Hi Peng Hui Jiang, do you see any way to automate the above procedure for hundred of docs in the list?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Invalid purge doc - how to remove? #3219

{{title}}

Replies: 5 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Invalid purge doc - how to remove? #3219

VasylSlobodyanyuk Oct 20, 2020

Replies: 5 comments · 2 replies

jiangphcn Oct 20, 2020 Collaborator

VasylSlobodyanyuk Oct 20, 2020 Author

jiangphcn Oct 21, 2020 Collaborator

VasylSlobodyanyuk Oct 21, 2020 Author

jiangphcn Oct 23, 2020 Collaborator

VasylSlobodyanyuk Oct 26, 2020 Author

VasylSlobodyanyuk Oct 26, 2020 Author

VasylSlobodyanyuk
Oct 20, 2020

Replies: 5 comments 2 replies

jiangphcn
Oct 20, 2020
Collaborator

VasylSlobodyanyuk Oct 20, 2020
Author

jiangphcn
Oct 21, 2020
Collaborator

VasylSlobodyanyuk Oct 21, 2020
Author

jiangphcn
Oct 23, 2020
Collaborator

VasylSlobodyanyuk
Oct 26, 2020
Author

VasylSlobodyanyuk
Oct 26, 2020
Author