liststreamkeyitems with gettxoutdata and low maxshowndata faster than liststreamkeyitems with high maxshowndata

Question

Hi,

I have the following observations:

1. publishmulti many stream items with JSON data into a stream e.g.1000

2. read all of them via liststreamkeyitems and save the time needed for that.

3. set max shown data to a very small value e.g. 10b.

4. Read the data again which will return with txids for the data and then read the real data with gettxoutdata for each item and save the time.

--> With liststreamkeyitems and then gettxoutdata I get all of the data in nearly half of the time!!!

I thought it should be slower because of more requests...

Is this normal? I found this here:
https://www.multichain.com/qa/11391/the-runtime-param-maxshowndata-has-default-size-16384-bytes

Thanks and best regards

Johannes

Answer 1

Yes, it's not surprising because of the time saved in building and manipulating the more complex JSON responses that include the data within the liststreamkeyitems response.

Comments

Thanks for your fast answer!
I don't understand why it is faster because gettxoutdata also contains the JSON data?!

-So would you suggest that when using JSON data in stream items we should set maxshowndata to e.g. 10 bytes and always use gettxoutdata to retrieve the data? But then we have more requests which is bad.
-Is there another option to get the data faster?

---

Here are some testresults:


maxshowndata: 16384b
small json data (~1kb)
Stream Items: 1000 each one has the same json data of 1kb
Time in ms: 2799
No of requests: 20 (listStreamKeyItems RPC batch requests (each RPC batch requests contains 50 requests))

maxshowndata: 10b
Items: 1000
Time in ms: 2637
No of requests: 40 (listStreamKeyItems + gettxoutdata RPC batch requests)

------------------------------------------------------------
bigger json data (~12kb):

maxshowndata: 16384b
Items: 1000
Time in ms: 13641
No of requests: 20

maxshowndata: 10b
Items: 1000
Time in ms: 6855
No of requests: 40


==> with small JSON payload the result is very similar but increases heavily with bigger JSON data

---

The reason for the speed difference is that building structured JSON responses requires some copying of data internally within MultiChain. So if you retrieve the data directly using gettxoutdata some of this copying is avoided.

There is certainly optimization work to be done in the JSON processing layer. For now you could also consider serializing/deserializing the JSON to and from binary in your application, and posting binary data in the stream. You may find that this performs better for you.