rpckeepalive: performance drops vs port freeze

Question

Hi,

is there any news on the rpc freeze problem?
https://www.multichain.com/qa/7006/node-rpc-completely-freezes-rpcssl?show=7178#a7178

 

What I still experience (v1.0.4):
Everything works fine using rpckeepalive=0 but there is another problem:

The performance (tx/s) decreases a lot when using rpckeepalive=0
I only get ~ 100 tx/s throughput when using rpckeepalive=0 (and rpcssl).

If I enable rpckeepalive I get ~800 tx/s (!)
But still the rpc api freezes after some time.

I think the problem only appears when using it together with rpcssl because every new requests has to open a new connection and has to do the ssl handshake all over again.

So:
keepalive=0 has a really bad performance (at least when using rpcssl?)
keepalive=1 freezes the node on high load after no more than 30 seconds.

Answer 1

Did you try keeping rpckeepalive=0 but increasing rpcthreads to a much larger number?

Comments

Yes, I started multichain with 64 rpc threads but tx/s stays completely the same at 100 tx/s.

---

OK, thanks. And if you don't use SSL, would you mind checking the tx rates like with and without keepalive?

---

Sorry, I had the wrong combination for the thread test.
I did not use keepalive=1

Now with keepalive=1 and rpcthreads=16 the throughput is much(!) better:
2000 rpc:"getinfo" per second and it does not seem to freeze anymore.

Maybe the problem is also a little bit in our setup:
We use a proxy application in front of all of our nodes. To scale we have some instances of this proxy (let's say 8) running. The requests to the proxy are load-balanced. The proxy now forwards the requests (with it's own ip) to the node.

That's why a single node has multiple connections open even from a single consumer. In a load test the node gets 8 connections (of 8 proxy instances).

But maybe this is also an advantage that we exactly know how many concurrent connections will be opened to a node at max.

I think it only depends on the number of proxy instances and on the go httpClient parameter:
"MaxIdleConnsPerHost"

So I will try to use rpcthreads=32 as our default and check if I can manage it to freeze the node again.

EDIT:
In the same moment I pressed enter, the mc node freezed. :(

Answer 2

Following setup works:

multichain: keepalive = 0 and ssl=off

nginx (on the machine where the node is running) in front of multichain does ssl-termination and forwards to multichain

Instead of 100 req/s I get 650 req/s. keepalive=0 between nginx and multichaind does not have a huge impact since theay are on the same machine.

 

I still use keepalive between my proxy and now nginx and there is no problem anymore. It really seems to be a strange problem with multichain keepalive and the only way to avoid freezing under high load conditions is to set it to 0. But this does not seem to be an optimal solution in my opinion.

Comments

Thanks for the update. We'll look into this to understand what is going on.