Getting 404 on localhost:8080/example

[dertz]

Pulled down code and did docker compose. All containers running successfully. Tried localhost:8080/example. Entered some text into input text box and selected Send button. Got the following message on the screen - "Uh oh! 404 Client Error: Not Found for url: http://schema-registry:8081/subjects/test_provenance-key/versions/latest"

Before exception is thrown, seeing the follow messages in the schema-registry container log:

[2021-08-10 14:47:37,677] INFO Started @8964ms (org.eclipse.jetty.server.Server)
[2021-08-10 14:47:37,677] INFO Server started, listening for requests... (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)
[2021-08-10 15:44:21,466] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Group coordinator broker:29092 (id: 2147483646 rack: null) is unavailable or invalid due to cause: session timed out without receiving a heartbeat response.isDisconnected: false. Rediscovery will be attempted. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2021-08-10 15:44:21,499] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Discovered group coordinator broker:29092 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2021-08-10 15:44:21,500] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Discovered group coordinator broker:29092 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2021-08-10 15:44:21,501] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Group coordinator broker:29092 (id: 2147483646 rack: null) is unavailable or invalid due to cause: coordinator unavailable.isDisconnected: false. Rediscovery will be attempted. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2021-08-10 15:44:21,603] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Discovered group coordinator broker:29092 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

After these messages the following exception is being thrown and found in the in schema-registry container log -

[2021-08-10 20:54:51,917] ERROR Request Failed with exception (io.confluent.rest.exceptions.DebuggableExceptionMapper)
io.confluent.rest.exceptions.RestNotFoundException: Subject 'test_provenance-key' not found.
at io.confluent.kafka.schemaregistry.rest.exceptions.Errors.subjectNotFoundException(Errors.java:69)

Aug 10, 2021 2:47:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.kafka.schemaregistry.rest.resources.ModeResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.kafka.schemaregistry.rest.resources.ModeResource will be ignored.
Aug 10, 2021 2:47:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.kafka.schemaregistry.rest.resources.SchemasResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.kafka.schemaregistry.rest.resources.SchemasResource will be ignored.
Aug 10, 2021 2:47:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.kafka.schemaregistry.rest.resources.ConfigResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.kafka.schemaregistry.rest.resources.ConfigResource will be ignored.
Aug 10, 2021 2:47:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.kafka.schemaregistry.rest.resources.SubjectsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.kafka.schemaregistry.rest.resources.SubjectsResource will be ignored.
Aug 10, 2021 2:47:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.kafka.schemaregistry.rest.resources.ServerMetadataResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.kafka.schemaregistry.rest.resources.ServerMetadataResource will be ignored.
Aug 10, 2021 2:47:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource will be ignored.
Aug 10, 2021 2:47:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.kafka.schemaregistry.rest.resources.CompatibilityResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.kafka.schemaregistry.rest.resources.CompatibilityResource will be ignored.

at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.validateAndGetSchema(KafkaSchemaRegistry.java:959)
atio.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.getSchemaByVersion(SubjectVersionsResource.java:116)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:386)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:561)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:502)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:439)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:829)

[adam-earthscope]

It looks like the broker isn't running:

[2021-08-10 15:44:21,501] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Group coordinator broker:29092 (id: 2147483646 rack: null) is unavailable or invalid due to cause: coordinator unavailable.isDisconnected: false. Rediscovery will be attempted.

I think that takes down the registry, then the rest.

[dertz]

Looks like the archiver keeps failing. The last message in archiver log file is\: File "/es-example-django/kafka_interface.py", line 146, in __init__ r.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://schema-registry:8081/subjects/test_provenance-key/versions/latest

…

On Thu, Aug 12, 2021 at 11:12 AM Adam Clark ***@***.***> wrote: It looks like the broker isn't running: [2021-08-10 15:44:21,501] INFO [Schema registry clientId=sr-1, groupId=schema-registry] Group coordinator *broker:29092* (id: 2147483646 rack: null) *is unavailable* or invalid due to cause: coordinator unavailable.isDisconnected: false. Rediscovery will be attempted. I think that takes down the registry, then the rest. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIN2RICMZXQGR75Y7G4TNTT4P6IVANCNFSM5B45WZQQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

-- Douglas J. Ertz Project Manager III, UNAVCO 6350 Nautilus Drive, Suite B/C Boulder, CO 80301-5394 Voice: *303-381-7555* By cell: *720-466-0083* Fax: *303-381-7501* ***@***.*** Please acknowledge UNAVCO sponsors and awards: http://www.unavco.org/community/policies_forms/acknowledgment/acknowledgment.html

[adam-earthscope]

This is normal. (Well, "normal" :) The topic doesn't exist until you send a message on it, so the consumer will throw an error. It should keep retrying (although I'm not sure for how long) until a message actually appears, then it should pick it up and start normally.

[adam-earthscope]

Actually, I think there was something missing, can you try pulling the latest version?

[dertz]

Still getting 404 error. All containers appear to be up except archiver. Here's Docker desktop show them all: [image: Screen Shot 2021-08-12 at 12.52.13 PM.png] Seeing this in archiver log. Seeing this [2021-08-10 14:47:15,250] level=DEBUG name=urllib3.connectionpool Starting new HTTP connection (1): schema-registry:8081 [2021-08-10 14:47:15,256] level=ERROR name=es_common.utils Unretryable error: HTTPConnectionPool(host='schema-registry', port=8081): Max retries exceeded with url: /subjects/test_provenance-key/versions/latest (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f08a67fe640>: Failed to establish a new connection: [Errno 111] Connection refused')) [2021-08-10 14:47:15,256] level=DEBUG name=asyncio Using selector: EpollSelector Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn conn = connection.create_connection( File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused I would not expect a connection error, but maybe some other respnse for this scenario. Doug

…

On Thu, Aug 12, 2021 at 12:26 PM Adam Clark ***@***.***> wrote: This is normal. (Well, "normal" :) The topic doesn't exist until you send a message on it, so the consumer will throw an error. It should keep retrying (although I'm not sure for how long) until a message actually appears, then it should pick it up and start normally. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIN2RI4LHBFVQC23MQ5B6TT4QG5DANCNFSM5B45WZQQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

-- Douglas J. Ertz Project Manager III, UNAVCO 6350 Nautilus Drive, Suite B/C Boulder, CO 80301-5394 Voice: *303-381-7555* By cell: *720-466-0083* Fax: *303-381-7501* ***@***.*** Please acknowledge UNAVCO sponsors and awards: http://www.unavco.org/community/policies_forms/acknowledgment/acknowledgment.html

[dertz]

will do Doug.

…

On Thu, Aug 12, 2021 at 1:01 PM Adam Clark ***@***.***> wrote: Actually, I think there was something missing, can you try pulling the latest version? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIN2RJHCO2UYUAOSC2YZYDT4QLCDANCNFSM5B45WZQQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

-- Douglas J. Ertz Project Manager III, UNAVCO 6350 Nautilus Drive, Suite B/C Boulder, CO 80301-5394 Voice: *303-381-7555* By cell: *720-466-0083* Fax: *303-381-7501* ***@***.*** Please acknowledge UNAVCO sponsors and awards: http://www.unavco.org/community/policies_forms/acknowledgment/acknowledgment.html

[dertz]

Pulled latest version. Getting I think a little further. Producer now giving the following exception: https://docs.google.com/document/d/1mg8lsh1B_Af_QFydl0jaWYLP3XmyXjdE3bfteDKl3qQ/edit?usp=sharing The following message was displayed in the user interface: Uh oh! KafkaError{code=_KEY_SERIALIZATION,val=-162,str="HTTPConnectionPool(host='schema-registry', port=8081): Max retries exceeded with url: /subjects/test_provenance-key/versions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67be1d1190>: Failed to establish a new connection: [Errno 111] Connection refused'))"} Here's picture of screen: [image: Screen Shot 2021-08-12 at 2.40.53 PM.png]

…

On Thu, Aug 12, 2021 at 1:01 PM Adam Clark ***@***.***> wrote: Actually, I think there was something missing, can you try pulling the latest version? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIN2RJHCO2UYUAOSC2YZYDT4QLCDANCNFSM5B45WZQQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

-- Douglas J. Ertz Project Manager III, UNAVCO 6350 Nautilus Drive, Suite B/C Boulder, CO 80301-5394 Voice: *303-381-7555* By cell: *720-466-0083* Fax: *303-381-7501* ***@***.*** Please acknowledge UNAVCO sponsors and awards: http://www.unavco.org/community/policies_forms/acknowledgment/acknowledgment.html

[adam-earthscope]

Okay, I think I see some oddness in the bootstrapping process here. Just bringing the whole system down then back up might help:

docker compose down
docker compose up -d

I think the broker, registry, and application all tie into each other via the AVRO schema, and it sets up some complicated dependencies. I'm looking into unraveling them.

[dertz]

Did that and I think I got the expected output. [image: Screen Shot 2021-08-12 at 5.33.49 PM.png]

…

On Thu, Aug 12, 2021 at 5:20 PM Adam Clark ***@***.***> wrote: Okay, I think I see some oddness in the bootstrapping process here. Just bringing the whole system down then back up might help: docker compose down docker compose up -d I think the broker, registry, and application all tie into each other via the AVRO schema, and it sets up some complicated dependencies. I'm looking into unraveling them. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIN2RMBA7OJU4FWABUL5QTT4RJM7ANCNFSM5B45WZQQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

-- Douglas J. Ertz Project Manager III, UNAVCO 6350 Nautilus Drive, Suite B/C Boulder, CO 80301-5394 Voice: *303-381-7555* By cell: *720-466-0083* Fax: *303-381-7501* ***@***.*** Please acknowledge UNAVCO sponsors and awards: http://www.unavco.org/community/policies_forms/acknowledgment/acknowledgment.html