Facing service unavailable error while implementing load balanced architecture

[Akshaykale84]

KV Version : 22.2.13
Node driver Version : 5.3.4

I am using Oracle NoSQL Database for one of the project. I have configured the proxy server and the communication between DB and Application is established.

 

I created 4 instances of proxy to enable load balanced architecture and also configured the HAproxy to distribute load to the four proxy servers.

 

When I am trying to make NoSQL client with nodeJS driver it is throwing an error message stating "Service Unavailable"(HTTP_RESPONSE_CODE 503).

 

Kindly help in this regard.

 

And also suggest if there is any other way to implement load balanced architecture in Oracle NoSQL Database (on-prem).

[dario-vega]

I will review it tomorrow and provide you instructions to setup HA-proxy
@Akshaykale84 Is it a secure or non-secure NoSQL cluster?

[Akshaykale84]

I will review it tomorrow and provide you instructions to setup HA-proxy
@Akshaykale84 Is it a secure or non-secure NoSQL cluster?

Sure.

It's non-secured NoSQL cluster

[dario-vega]

Node.js application connection

        return new NoSQLClient({
            serviceType: ServiceType.KVSTORE,
            endpoint: process.env.NOSQL_ENDPOINT + ":" + process.env.NOSQL_PORT
	});

Deploy your cluster and start the HTTP proxy in the nodes.

Note: The Oracle NoSQL Database Proxy can run in one or multiple dedicated hosts. It can be hosted inside the nodes of the NoSQL Cluster.

e.g node1-nosql, node2-nosql, node3-nosql in my case, I am using the port 8080.

java -jar $KVHOME/lib/httpproxy.jar -helperHosts $KVHOST:5000 -storeName $KVSTORE -httpPort 8080 -verbose true

Test using curl for all nodes hosting a HTTP proxy node1-nosql, node2-nosql, node3-nosql

curl -v http://node1-nosql:8080 

Test using your Oracle NoSQL SDK for Node.js example for all nodes hosting a HTTP proxy node1-nosql, node2-nosql, node3-nosql

export NOSQL_ENDPOINT=node1-nosql
export NOSQL_PORT=8080 
node test.js 

Configuring the Load Balancer

You can use a load balancer as frontend which has a backend set of multiple NoSQL proxies on different hosts. In this case, you can setup HA proxy.

Install haproxy - in an external node to run the Load Balancer
e.g nosql-lb

sudo yum install haproxy

Add the following lines at the end of the file /etc/haproxy/haproxy.cfg

in my case, node1-nosql, node2-nosql, node3-nosql and using the port 8080 for all nodes including the load balancer

# Configure HAProxy to listen on port 80
frontend http_front
   bind *:8080
   stats uri /haproxy?stats
   default_backend http_back

# Configure HAProxy to route requests to Oracle NoSQL Database Proxy nodes on port 8080
backend http_back
   balance roundrobin
   server node1-nosql 10.0.0.56:8080 check
   server node2-nosql 10.0.0.167:8080 check
   server node3-nosql 10.0.0.143:8080 check

Restart HA proxy and validate the status

sudo systemctl stop haproxy.service
sudo systemctl start haproxy.service
sudo systemctl status haproxy.service

$ sudo systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-02-24 10:25:52 GMT; 2min 3s ago
  Process: 2205764 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -f $CFGDIR -c -q $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2205766 (haproxy)
    Tasks: 2 (limit: 100047)
   Memory: 4.5M
   CGroup: /system.slice/haproxy.service
           ├─2205766 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /run/haproxy.pid
           └─2205769 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /run/haproxy.pid

Test using curl

curl -v http://nosql-lb:8080 

Test using your Oracle NoSQL SDK for Node.js example

export NOSQL_ENDPOINT=nosql-lb
export NOSQL_PORT=8080 
node test.js 

Troubleshooting

The following error happens if:

• the load balancer cannot contact any of the IP addresses in backend http_back - you can use fqdn instead of IP addresses
• all the HTTP proxy are down

NoSQLServiceError: [SERVICE_ERROR] Unsuccessful HTTP response.  Status code: 503. Status message: Service Unavailable
    at HttpClient._handleResponse (/home/opc/demo-graphql-nosql/blogloader/node_modules/oracle-nosqldb/lib/http_client.js:93:33)
    at IncomingMessage.<anonymous> (/home/opc/demo-graphql-nosql/blogloader/node_modules/oracle-nosqldb/lib/http_client.js:150:22)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) 

Let me know if it works, Otherwise, please provide the output for the command documented here

[Akshaykale84]

Thanks @dario-vega for the detailed explanation.

I tried the same on my environment.

It's working till creating the client of DB.

return new NoSQLClient({ serviceType: ServiceType.KVSTORE, endpoint: process.env.NOSQL_ENDPOINT + ":" + process.env.NOSQL_PORT 	});

But when I try to put some data in the table using the client facing the following error:

client.put('tableName', {id:10, name:'akshay'}).then(res => console.log('inserted'));

Error Msg:

'[REQUEST_TIMEOUT] Operation timed out after 5000 ms and 5 retries; Caused by: [SERVI CE_ERROR] Unsuccessful HTTP response. Status code: 503. Status message: Service Unavailable'

The same code is working when I change the endpoint from load balancer host to proxy host.

Please assist here..

[dario-vega]

Provide the HA proxy version used.

haproxy -v

Here is an extract of my configuration file

# Configure HAProxy to route requests to Oracle NoSQL Database Proxy nodes on port 8080
backend http_back
   balance roundrobin
   server node1-nosql 10.0.0.56:8080 check
   server node2-nosql 10.0.0.167:8080 check
   server node3-nosql 10.0.0.143:8080 check

Please provide the output for the following commands (use your IPs and ports) for each line with the server directive.

ping 10.0.0.56
ping 10.0.0.167
ping 10.0.0.143
curl -v http://10.0.0.56:8080
curl -v http://10.0.0.167:8080
curl -v http://10.0.0.143:8080

You can also use DNS entries.

# Configure HAProxy to route requests to Oracle NoSQL Database Proxy nodes on port 8080
backend http_back
   balance roundrobin
   server node1-nosql node1-nosql:8080 check
   server node2-nosql node2-nosql:8080 check
   server node3-nosql node3-nosql:8080 check

FYI, my test code is creating a table and inserting data, not just making a connection

I am using

proxyVersion=5.9.1
kvclientVersion=22.2.13

[Akshaykale84]

I restarted the haproxy and proxy instances and now it's working properly.

Thanks @dario-vega for your help..

[dario-vega]

This is a really interesting question for our Q&A so I moved it to this repository as a GitHub Discussion.
BTW, I am in the process of adding examples of configuring HA proxy in this repository.

[dario-vega]

@Akshaykale84

It is for your information, 🆕 in Oracle NoSQL database 23.1

You can use a load balancer as frontend which has a backend set of multiple NoSQL proxies on different hosts as discussed. When configuring a Load Balancer, you can add now an HTTP health check. The Oracle NoSQL Database Proxy provides the following URI /V2/health endpoint to do it. An HTTP request to this URI will return a successful response 200 OK.

Here is an example

$ curl -vvv http://localhost:8080/V2/health
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /V2/health HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Content-Length: 34
< X-content-type-options: nosniff
< content-disposition: attachment; filename=api.json
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
{"beacon":"GREEN","info":"ALL OK"}

$ curl http://localhost:8080/V2/health
{"beacon":"GREEN","info":"ALL OK"}

[Akshaykale84]

Thanks @dario-vega for sharing this.