Kong's PG read only property

[RagunathInGIT]

We are running Kong in multiple DataCenters. (A and B)

DC - A -> Kong on K8s with PG DB. We run master (read and write) PG and replica (read only) instance here.
DC - B -> Kong on K8s listening to master PG on DC -A. We also have one more replica (read only) here.

Both DC's are active and to avoid latency in DC-B as it has to talk to master in DC-A, along with pg_host property, we configured pg_ro_host property in DC-B which points to DC-B's replica for read only calls. This helped us reduce the latency.
But for some reasons, if we bring down DC-B's replica, pg_ro_host goes down and eventually bringing down Kong itself even-though master in DC-A is active.

So question is, can Kong switch to use pg_host property if the host configured in pg_ro_host is not reachable?

[daltk]

pg_host does not failover automatically if pg_ro_host is not reachable. we can do a trick by pointing to multiple available instances for the pg_host cluster itself.

…

On Tue, Jul 27, 2021 at 8:33 PM RagunathInGIT ***@***.***> wrote: We are running Kong in multiple DataCenters. (A and B) DC - A -> Kong on K8s with PG DB. We run master (read and write) PG and replica (read only) instance here. DC - B -> Kong on K8s listening to master PG on DC -A. We also have one more replica (read only) here. Both DC's are active and to avoid latency in DC-B as it has to talk to master in DC-A, along with pg_host property, we configured pg_ro_host property in DC-B which points to DC-B's replica for read only calls. This helped us reduce the latency. But for some reasons, if we bring down DC-B's replica, pg_ro_host goes down and eventually bringing down Kong itself even-though master in DC-A is active. So question is, can Kong switch to use pg_host property if the host configured in pg_ro_host is not reachable? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7609>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFHL6EMHOB6BSHQPQI7JLTZ3DD7ANCNFSM5BCNZL7Q> .

[RagunathInGIT]

Thanks Raj for quick response.

If I understand correctly, there is no way that Kong can start using pg_host property for rw operations once hostname
configured in pg_ro_host property is not reachable.

One more setup that I did not cover in my initial comment is, we do have HAProxy configured for postgres instances to determine which is master and Kong's pg_host property listens to HAProxy service. So anytime, Kong will listen to PG master instance on pg_host property.

Only problem is in DC-B since we have only one instance of PG running, if that becomes master or if that goes down, then we loose pg_ro_host property in DC-B and that causes the issue.

[daltk]

Hi Raghu, Does your Kong's pg_host property listens to HAProxy service in DC-B? Thanks Raj

…

On Tue, Jul 27, 2021 at 9:21 PM RagunathInGIT ***@***.***> wrote: Thanks Raj for quick response. If I understand correctly, there is no way that Kong can start using pg_host property for rw operations once hostname configured in pg_ro_host property is not reachable. One more setup that I did not cover in my initial comment is, we do have HAProxy configured for postgres instances to determine which is master and Kong's pg_host property listens to HAProxy service. So anytime, Kong will listen to PG master instance on pg_host property. Only problem is in DC-B since we have only one instance of PG running, if that becomes master or if that goes down, then we loose pg_ro_host property in DC-B and that causes the issue. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7609 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFHLZT74C5VPKUMOE7UVTTZ3IZ3ANCNFSM5BCNZL7Q> .

[RagunathInGIT]

yes Raj. pg_host property in both DC's listens to HAProxy which forwards the connections to active master PG instance.
So we are good from pg_host property perspective.

Issue is with pg_ro_host property as we are not having HAProxy there. Because we have PG instances (2 in DC-A and 1 in DC-B) running in both DC's. We dont want read only property to listen across the DC's to avoid latency. Question is, since we have only 1 instance in DC-B, if that goes down, Kong is also not reachable as pg_ro_host property is down.

[daltk]

Yes I can see the problem please provision 2:2 instances in both DCs Logically state of PG load balancing is not maintained at kong level & kong never looks up for failover resilient nature of connection in pg_ro_host. Hence, just boot another PG instance in DC2 Hope this helps Respectfully Raj Mothe

…

On Tue, 27 Jul 2021 at 10:56 PM, RagunathInGIT ***@***.***> wrote: yes Raj. pg_host property in both DC's listens to HAProxy which forwards the connections to active master PG instance. So we are good from pg_host property perspective. Issue is with pg_ro_host property as we are not having HAProxy there. Because we have PG instances (2 in DC-A and 1 in DC-B) running in both DC's. We dont want read only property to listen across the DC's to avoid latency. Question is, since we have only 1 instance in DC-B, if that goes down, Kong is also not reachable as pg_ro_host property is down. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7609 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFHL23N4MOQG65EKEBEGLTZ3T4RANCNFSM5BCNZL7Q> .

[RagunathInGIT]

Thanks Raj. Yes this helps.