list stream keys (verbose: true) gives a "couldn't parse reply from server" error

Question

I am storing large hexadecimal data in keys on the chain. But when I do:
liststreamkeys 147-267-11492 * true 20
so basically:

list stream keys
for a stream with stream-ref 147-267-11492
list all of the keys
list 20 of them
it sometimes works and sometimes doesn't work. When it doesn't, it says

error: couldn't parse reply from server

I have this both in Postman, that I use to test my backend, as well as my terminal (windows). Sometimes I get an output, but most of the time not.

When I do get an output, the output is (according to this website: https://mothereff.in/byte-counter) 76,162 characters, totaling 76,162 bytes. Which is 6 keys, each containing 1 item

I'm using liststreamkeys as an index. Each key is a shipment, which is displayed in a table. The table shows a description and a in transfer or completed status

Any idea what's going on?

edit: I notice that it's running much smoother on my linux VM, I can fire it as often as fast as I want and it won't crash

Comments

Thanks for the question. I've forwarded this to the dev team to see what they suggest.

Answer 1

Thanks again for reporting this. Since this doesn't show up on Linux, our best guess is that it's some interprocess or network communication issue in your Windows version. We'll check it on our end, but could you please try some other API requests whose responses are of shorter lengths, to see if you can find the approximate threshold for response size where the problems show up? You can play with the number of keys for example. The most likely thresholds are powers of 2, e.g. 65536 bytes, 32768, 16384, etc...

Comments

Been testing a bit (unfortunately I don't have much time today), I tested with the first 5 keys, which are a response of 60,964 characters and bytes. When I put key #6 (a response of 15k characters and bytes) it becomes unstable.

I also tested this with other API calls, such as liststreamkeyitems. This one worked with a response of about 40k characters and bytes. Then I added 8 more keys of similar size. This one doesn't work anymore either :(

I tested liststreamkeyitems (stream) (*), so without verbose=true. This result was 65,057 chars and bytes. Then I added another item and reached that threshold again (both verbose=false and verbose=true). What I can do in my backend is try using the count parameter, to make sure I always stay below a 70kb response

---

Thanks. So this is almost definitely a 65,536 character (64K) buffer somewhere in the communication between multichain-cli and multichaind. We will look into it.

---

I found something really bizar. I tested a bit more with a colleague. I added a bunch more items. I about about 16 keys now.

response is about 70KB (verbose=true):
liststreamkeys doesn't work
liststreamkeyitems works

response is 194KB (verbose=true)
liststreamkeys works
liststreamkeyitems doesn't work

while typing this I wanted to double check, both methods worked.....and then they didn't anymore....I really don't know what to do anymore. I just stick to count = 4 or something

---

We weren't able to see a problem with 1 MB sized response payloads on our own Windows testing. Since the connection between multichain-cli and multichaind is a local TCP/IP request, is it possible there is some firewall or other software on this computer which could be limiting the sizes of the responses?

---

we're behind quite a thick security layer. My home desktop has no problems with 100kb+ sized response payploads, so it's probably device related.

Thanks for your time, I really appreciate it! keep up the good work.

---

Yes, if your API requests are not local but rather travelling over various routers, etc..., then this is undoubtedly the explanation.