[Rock-dev] Questions Regarding Typelib

[Janosch Machowinski]

Am 08.06.2014 23:16, schrieb Sylvain Joyeux:
> Oh ... I would also need the size of each stream's sample (including 
> the size of vectors if there are any) ...
>
> About index loading: the way the index was marshalled needed to be 
> changed (but was not) after the change you made to indexing (i.e. 
> making indexes dense). A 3-line patch improves performance quite a lot 
> already. Alignment is already pretty good on my test file (~4s).
It gets worse with the number of streams. Try a testcase with ~60 
streams. There the performance really
drops, and this is the 'reality' test case...
>
> I've generated a 6G log file with 3 streams, ~650k samples in 1 minute 
> of logical time, reading the streams using the Ruby stream aligner 
> yields ~115x if I don't read the data sample, ~8x if I do. To me, this 
> definitely means that the performance problem is not pocolog's 
> low-level handling but both I/O (I have "images" of 1M, so I/O is 
> actually a big performance problem), typelib demarshalling (there is 
> one thing in this code path that sucks big time currently in the 
> typelib-ruby binding related to demarshalling, did not measure the 
> actual impact of it, though) and the fact that I hit the ruby GC 
> pretty hard by creating 650k samples and 650k intermediate buffers.
To summon it up, the c++ implementation still outperforms the ruby 
implementation by a factor of 4 in replay speed.
>
> The point is: as soon as you'll get ruby bindings, you'll have the 
> same issues (GC, typelib demarshalling and I/O will not go away). The 
> only thing that goes away is the creation of 650k intermediate 
> buffers, but that can be removed with very little work already.
So, what are you saying ? We should not go for ruby bindings any more ?
>
> Now, I do get 8x replay speed when demarshalling the data. That's very 
> far from 0.6. When you talk about bad performance, were you using the 
> replay UI ?
Replay scrips and UI. Using rock-replay is a pain right now.
>
> For reference:
>
> pocolog test.0.log < this is a 5.9G file
>
> Stream images [/images] < these are 1M samples
>   5994 samples from Sun 08/06/2014 22:36:42 to Sun 08/06/2014 22:46:41 
> [0:09:59.299]
> Stream motors [/motors] < these are 100 bytes samples
>   599708 samples from Sun 08/06/2014 22:36:42 to Sun 08/06/2014 
> 22:46:41 [0:09:59.706]
> Stream pose_samples [/pose_samples] < these are 512 bytes samples
>   59970 samples from Sun 08/06/2014 22:36:42 to Sun 08/06/2014 
> 22:46:41 [0:09:59.690]
>
> Sylvain
Do you got a real logset somewhere for tests ?
Should I upload one tomorrow ?
Greetings
     Janosch