[Rock-dev] updateHook hanging (blocked?) when started via syskit
Elmar Berghöfer
elmar.berghoefer at dfki.de
Tue Dec 17 10:00:37 CET 2013
Hi,
Just some additional informations on that topic:
Regarding the performance aspect Christian didn't mention some points
that might be important.
We also tested connecting the robot to one of the Laptops and just
connecting _one_ of the cameras
to its gigabit ethernet port. If you then start the camera and the robot
task within syskit the delays
also occur, and they not just stuck for several milliseconds but up to 8
seconds which causes the
driver delivering a time out on the connection to the robot (via serial
port) and shuts down the connection
completely. I think that it is not "just" a performance issue, the
Laptop has quite high performance hardware.
Also on the SeekurJr itself the problem occurs already with one camera
running. While the camera is running
with 10Fps and binning, the logger has to write ~1MB per Frame which is
just 10 MB/s. I did some benchmarking
on the harddrive (without system cache) which results in a write
performance between 50 - 60 MB/s constantly.
Btw. the aria task and the camera task are not in the same deployment.
So it still could be some performance issue, maybe in combination with
unfortunate configured tasks, especially
I am not sure about the camera task, as far as I understood the answer
from Alex Duda, I guess such a time out
is already used, but maybe Christian can check that better than me.
Regards
Elmar
Am 17.12.2013 08:36, schrieb Matthias Goldhoorn:
> On 16.12.2013 19:28, Christian Rauch wrote:
>> I set the priority by:
>>
>> task("seekur_drv","aria::Task").priority(99)
>>
>> which does not affect the result (still delays).
>>
>> The aria::Task is periodic (100ms) and the camera_ids::Task is
>> triggered in the updateHook after a new image arrives.
> I assume both in one deployment. You could (for testing reasons) try
> to separate the deployments.
>>
>> As the camera Logger is writing with approx. 10 MB/s (and CPU load is
>> below 20%) which is far away from the HDD limit (~60MB/s) and this
>> whole issue does not occur when running the cameras with logger in
>> syskit and the aria::Task outside syskit, I assume that this is not a
>> hardware related issue.
> did you take a look to the load AND the system CPU Usage (top lines
> from `top`)
>>
>> As this problem for now seems to occur only when running both tasks
>> (aria, camera_ids) within the bundle; what are the options to change
>> the scheduling by syskit? Does syskit change the behaviour of how
>> processes are scheduled by the OS in any way?
> Syskit does NOT influence the scheduling of tasks, this is defined at
> deployment time implicitly. The only differences when using syskit are:
> - Syskit takes also CPU Load
> - Syskit might use different connection types than you runscript ->
> take a look to `syskit instanciate ...` the connection policys are
> part of the graph
>
> I still assume some load-related problems on your machine, sinc'e you
> mention that if you stopping the logger everything works fine.
>>
>> Regards,
>> Christian
>>
>>
>> Am 16.12.2013 16:21, schrieb Matthias Goldhoorn:
>>> Several things could happen:
>>>
>>> The logger consumes too much time to get out the images, caused by a
>>> slow hdd.
>>> Try to increase the priority level of the pioneer task within your
>>> deployment to make this more urgent from the point of scheduling.
>>>
>>> Matthias
>>>
>>> P.S. i assume you task is periodic
>>>
>>> On 16.12.2013 16:17, Christian Rauch wrote:
>>>> Hi,
>>>>
>>>> I am using the aria task within syskit as a device which is
>>>> supposed to
>>>> be called every 100ms. The aria task uses an additional thread
>>>> provided by the aria driver to communicate with the robot.
>>>>
>>>> If I run that task from the syskit shell without additional sensors
>>>> the
>>>> updateHook gets called as expected. If I add one or more cameras
>>>> (camera_ids) with loggers it happens that the updateHook gets stuck
>>>> for some time:
>>>>
>>>> {{{
>>>> [DEBUG] - aria::updateHook at: 1386784751.580.629
>>>> [DEBUG] - aria::updateHook at: 1386784751.680.668
>>>> [DEBUG] - aria::updateHook at: 1386784751.780.542
>>>> ArRobotPacketReceiver::receivePacket: bad packet, bad checksum
>>>> Warning: Task 'Packet Handler' took 1213 ms to run (longer than the
>>>> 250
>>>> warning time)
>>>> Warning: ArRobot sync tasks too long at 1213 ms, (100 ms normal 250 ms
>>>> warning)
>>>> [DEBUG] - aria::updateHook at: 1386784753.673.374
>>>> [DEBUG] - aria::updateHook at: 1386784753.677.372
>>>> [DEBUG] - aria::updateHook at: 1386784753.681.379
>>>> [DEBUG] - aria::updateHook at: 1386784753.685.211
>>>> [DEBUG] - aria::updateHook at: 1386784753.689.226
>>>> [DEBUG] - aria::updateHook at: 1386784753.697.023
>>>> [DEBUG] - aria::updateHook at: 1386784753.701.405
>>>> [DEBUG] - aria::updateHook at: 1386784753.705.386
>>>> [DEBUG] - aria::updateHook at: 1386784753.709.383
>>>> [DEBUG] - aria::updateHook at: 1386784753.713.216
>>>> [DEBUG] - aria::updateHook at: 1386784753.717.292
>>>> [DEBUG] - aria::updateHook at: 1386784753.721.253
>>>> [DEBUG] - aria::updateHook at: 1386784753.725.392
>>>> [DEBUG] - aria::updateHook at: 1386784753.729.400
>>>> [DEBUG] - aria::updateHook at: 1386784753.733.392
>>>> ArRobotPacketReceiver::receivePacket: bad packet, bad checksum
>>>> [DEBUG] - aria::updateHook at: 1386784753.737.227
>>>> [DEBUG] - aria::updateHook at: 1386784753.741.226
>>>> [DEBUG] - aria::updateHook at: 1386784753.745.274
>>>> [DEBUG] - aria::updateHook at: 1386784753.749.305
>>>> [DEBUG] - aria::updateHook at: 1386784753.781.249
>>>> [DEBUG] - aria::updateHook at: 1386784753.881.453
>>>> [DEBUG] - aria::updateHook at: 1386784753.981.492
>>>> [DEBUG] - aria::updateHook at: 1386784754.081.523
>>>> [DEBUG] - aria::updateHook at: 1386784754.181.392
>>>> [DEBUG] - aria::updateHook at: 1386784754.281.428
>>>> [DEBUG] - aria::updateHook at: 1386784754.381.462
>>>> }}}
>>>>
>>>>
>>>> For some reason this problem does no longer occur as soon as the
>>>> camera
>>>> loggers are stopped. It also does not occur when the aria Task is
>>>> started outside of syskit by using the default deployment and the
>>>> TaskInspector or the orocos.rb API.
>>>>
>>>> It looks like the whole task is blocked somehow (the updateHook is not
>>>> really taking that much processing time).
>>>> Any ideas where this might come from and why this only occurs when
>>>> running both tasks with loggers in syskit?
>>>>
>>>> Regards,
>>>> Christian
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Rock-dev mailing list
>>>> Rock-dev at dfki.de
>>>> http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev
>>>
>>>
>>> --
>>> Dipl.-Inf. Matthias Goldhoorn
>>> Space and Underwater Robotic
>>>
>>> Universität Bremen
>>> FB 3 - Mathematik und Informatik
>>> AG Robotik
>>> Robert-Hooke-Straße 1
>>> 28359 Bremen, Germany
>>>
>>> Zentrale: +49 421 178 45-6611
>>>
>>> Besuchsadresse der Nebengeschäftstelle:
>>> Robert-Hooke-Straße 5
>>> 28359 Bremen, Germany
>>>
>>> Tel.: +49 421 178 45-4193
>>> Empfang: +49 421 178 45-6600
>>> Fax: +49 421 178 45-4150
>>> E-Mail:matthias.goldhoorn at informatik.uni-bremen.de
>>>
>>> Weitere Informationen:http://www.informatik.uni-bremen.de/robotik
>>>
>>>
>>>
>>> _______________________________________________
>>> Rock-dev mailing list
>>> Rock-dev at dfki.de
>>> http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev
>>>
>>
>>
>>
>> _______________________________________________
>> Rock-dev mailing list
>> Rock-dev at dfki.de
>> http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev
>
>
> --
> Dipl.-Inf. Matthias Goldhoorn
> Space and Underwater Robotic
>
> Universität Bremen
> FB 3 - Mathematik und Informatik
> AG Robotik
> Robert-Hooke-Straße 1
> 28359 Bremen, Germany
>
> Zentrale: +49 421 178 45-6611
>
> Besuchsadresse der Nebengeschäftstelle:
> Robert-Hooke-Straße 5
> 28359 Bremen, Germany
>
> Tel.: +49 421 178 45-4193
> Empfang: +49 421 178 45-6600
> Fax: +49 421 178 45-4150
> E-Mail:matthias.goldhoorn at informatik.uni-bremen.de
>
> Weitere Informationen:http://www.informatik.uni-bremen.de/robotik
>
>
> _______________________________________________
> Rock-dev mailing list
> Rock-dev at dfki.de
> http://www.dfki.de/mailman/cgi-bin/listinfo/rock-dev
--
Dipl.-Inform. Elmar Berghöfer
Junior Researcher
Cognitive Robotics
Besuchsadresse der Nebengeschäftstelle:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Straße 5
28359 Bremen, Germany
Postadresse der Hauptgeschäftsstelle Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Straße 1
28359 Bremen, Germany
Tel.: +49 421 178 45-4104
Zentrale: +49 421 178 45-0
Fax: +49 421 178 45-4150 (Faxe bitte namentlich kennzeichnen)
E-Mail: elmar.berghoefer at dfki.de
Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Straße 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.dfki.de/pipermail/rock-dev/attachments/20131217/4f7aef4f/attachment-0001.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4990 bytes
Desc: S/MIME Cryptographic Signature
Url : http://www.dfki.de/pipermail/rock-dev/attachments/20131217/4f7aef4f/attachment-0001.bin
More information about the Rock-dev
mailing list