tag:blogger.com,1999:blog-1920793513850944312018-11-02T16:50:58.239+01:00Jiri's allegedly HelenOS blogJiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.comBlogger42125tag:blogger.com,1999:blog-192079351385094431.post-62848718500047124042018-11-02T16:40:00.001+01:002018-11-02T16:50:58.216+01:00Replacing AVL tree in kernel with Ordered dictionary<p>I introduced the ordered dictionary (odict) module into HelenOS about two years ago (September 2016). I originally developed this module within another project (where I use it quite a lot) with the intent to use it in HelenOS, too. </p><p>It is a fairly versatile data structure based on Red-black trees. It never calls to malloc (all storage is provided by the user via odict_t and odlink_t structures). The key can be anything (the user provides a key comparison function). It also has some additional optimizations and such. </p><p>It didn't actually have any consumer in HelenOS, until SIF recently. In user space I assume it will see more use as the foundation for other data structures (such as the ID allocator, which I will talk about sometime else..). </p><p>In kernel, we currently have AVL trees and B+ trees. Jakub pointed out that we could replace B+ trees with odict, with the benefit that odict does not call malloc (and B+trees do). I realized that odict can replace AVL trees just as easily -- although the only clear benefit is that we will only have one tree implementation in the kernel -- which is a good reason in itself, right? </p><p>The only possible downfall is that, while red-black trees are slightly faster at insertion and deletion than AVL trees (at least in theory), our AVL tree has the advantage over odict that it does not need to call to the getkey/comparison functions (as the key is always an integer), which should bring some speed advantage. Therefore I was worried that replacing it with Odict could make the system slower -- although I thought the difference in real-time performace probably would not be measurable. </p><p>Therefore I decided to create a prototype implementation with Odict replacing AVL and benchmark it. For the measurement I first tried measuring some kernel tests with a stopwatch, but that obviously wasn't precise enough to show any difference. So I went for the only actual benchmark available in HelenOS, that is the IPC ping-pong benchmark. </p><p>I ran the benchmark 10 times and recorded the results, then calculated the averages and standard deviation using Gnumeric. I tested with AVL and Odict and also tested both in Qemu / KVM and QEMU without KVM. </p> In the following table all measurements were done with 10 samples. <table><tr><th>Test case</th><th>Average RTs / 10 s</th><th>Std. dev.</th></tr><tr><td>Odict / KVM</td><td>403320</td><td>4210</td></tr><tr><td>AVL / KVM</td><td>388700</td><td>4962</td></tr><tr><td>Odict / softmmu</td><td>25300</td><td>249</td></tr><tr><td>AVL / softmmu</td><td>23520</td><td>434</td></tr></table> <p>In the table above we can see Odict beating AVL by 4% in the KVM case and even a 7% improvement of Odict compared to AVL in the softmmu case, with standard deviation of the measurements being around 1 %. This is kinda surprising. </p><p>I then went and made some changes to the ping_pong benchmark. Instead or repeatedly doing 100 iterations until we measure that 10 seconds have expired, I changed it to determine the number of iterations (lowest power of two) needed to have the test run for at least 10 seconds, then measure the actual time taken (and divide number of iterations / duration). I hoped this would give me more accurate results and measure less of how getuptime() performs. I also had the ping_pong benchmark run 10 times and calculate the statistics for me. </p> Here's the results: Again all measurements are with 10 samples: <table><tr><th>Test case</th><th>Average RTs / s</th><th>Std. dev.</th></tr><tr><td>Odict / KVM</td><td>38988</td><td>241</td></tr><tr><td>AVL / KVM</td><td>38601</td><td>285</td></tr><tr><td>Odict / softmmu</td><td>2452</td><td>44</td></tr><tr><td>AVL / softmmu</td><td>2290</td><td>27</td></tr></table> <p>There still seems to be some improvement when switching from AVL to Odict, albeit it is much closer to the noise level of the measurement. </p> <p>AVL trees are used to hold the global lists of threads and tasks in the kernel. I am not sure if their performance ever comes to play in the ping_pong test. Any ideas how AVL/odict could affect results of the ping-pong test? How come we see any difference? Any suggestion for a benchmark that would touch better upon this area? (e.g. creating and destroying a thread repeatedly) </p>Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-40112004427589414342018-06-16T13:06:00.000+02:002018-06-16T13:06:18.315+02:00Roland MT-32 power up and testLet's power up and test the Roland MT-32. And listen to some game music, obviously. <p></p> <object width="360" height="288"><param name="movie" value="https://www.youtube.com/v/Ut0CXDqrIds?version=3&amp;autohide=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="https://www.youtube.com/v/Ut0CXDqrIds?version=3&amp;autohide=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="360" height="288"></embed></object>Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-91976471500359382622018-03-27T18:38:00.000+02:002018-03-27T18:38:33.850+02:00Beware of dragons!Dragons! This is Romania, Potter fans! Luckily we can protect ourselves with guns! And meteorites! <p></p> <a href="https://2.bp.blogspot.com/-rRLUmBiDvP4/WrpxrKviwuI/AAAAAAAAAFM/Vk2wsBlDrt0RvdFsOhih41ahPxZwUljWwCLcBGAs/s1600/dragon.jpg" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-rRLUmBiDvP4/WrpxrKviwuI/AAAAAAAAAFM/Vk2wsBlDrt0RvdFsOhih41ahPxZwUljWwCLcBGAs/s200/dragon.jpg" width="200" height="150" data-original-width="800" data-original-height="600" /></a><a href="https://1.bp.blogspot.com/-TULDqDw0CTA/WrpyadK8xlI/AAAAAAAAAFU/o8HvJKN_NKwSUoQKFnhkLO5ELJfgX87YACLcBGAs/s1600/cannon.jpg" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-TULDqDw0CTA/WrpyadK8xlI/AAAAAAAAAFU/o8HvJKN_NKwSUoQKFnhkLO5ELJfgX87YACLcBGAs/s200/cannon.jpg" width="200" height="150" data-original-width="800" data-original-height="600" /></a><a href="https://1.bp.blogspot.com/-ig3YT7O4drs/Wrpycvhr5eI/AAAAAAAAAFY/TYyMRC76JFIz7rEgtxhkmz0qescAUSUFwCLcBGAs/s1600/meteorite.jpg" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-ig3YT7O4drs/Wrpycvhr5eI/AAAAAAAAAFY/TYyMRC76JFIz7rEgtxhkmz0qescAUSUFwCLcBGAs/s200/meteorite.jpg" width="200" height="150" data-original-width="800" data-original-height="600" /></a>Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-11149256264117809712018-02-14T15:29:00.000+01:002018-02-14T15:29:14.086+01:00Bucharest in snow<a href="https://4.bp.blogspot.com/-W9C9HilCUvU/WoRGgRYqzqI/AAAAAAAAAE8/nlfOTIArfM4iO4_m6UC-a9OdmLoWkSnvwCPcBGAYYCw/s1600/bucharest.jpg" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-W9C9HilCUvU/WoRGgRYqzqI/AAAAAAAAAE8/nlfOTIArfM4iO4_m6UC-a9OdmLoWkSnvwCPcBGAYYCw/s320/bucharest.jpg" width="320" height="240" data-original-width="1024" data-original-height="768" /></a><p style="clear: left">Time for hot tea and donuts. </p>Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-52559728058907641552017-12-28T18:19:00.003+01:002017-12-28T18:19:42.616+01:00timrec now open source on GithubI've just <a href="https://github.com/jxsvoboda/timrec">published timrec</a>, one of my smaller projects, on Github under the permissive MIT license. This is a small Linux system service that records media from various sources based on a provided schedule. It can be used to record media such as internet radio or DVB (using a DVB tuner). I've been running it for the last 2+ years on a Raspberry Pi as a complete internet radio recording solution. (The DVB support is a fresh addition.)Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-84498025475679723072017-11-07T23:26:00.000+01:002017-11-07T23:26:50.290+01:00GZX open sourced!I published the source code to my personal ZX Spectrum emulator, GZX as free software. It is hosted <a href="https://bitbucket.org/jiri_svoboda/gzx/">on bitbucket.org</a> and published under a BSD-style license. Watch the video as I introduce GZX and its history. <br /><br /> Note that this is the kind of 'here's what I have, use it for whatever you want, I'll continue working on it for myself' thing. I am not soliciting code contributions and I cannot guarantee I would accept any since this is really a personal project. Of course, feedback is always welcome! Please keep in mind I am a busy person with several other projects and a life, so this is way down my priority list. <br /><br /> Still I hope to do more work in this interesting area. <br /><br /> <object width="360" height="288"><param name="movie" value="https://www.youtube.com/v/05ZU-JGY-G4?version=3&amp;autohide=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="https://www.youtube.com/v/05ZU-JGY-G4?version=3&amp;autohide=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="360" height="288"></embed></object>Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-73884569146346436342017-08-26T03:13:00.000+02:002017-08-26T03:13:23.876+02:00Simple HelenOS coastline package installationYou can now easily install coastline packages in HelenOS using the 'pkg install' command. It will automatically download and install the latest build of the package from ci.helenos.org (you need your HelenOS box to be connected to the Internet, obviously). <br /><br /> Previously you would have to download the package, uncompress it (into .tar format) and copy it to the overlay directory in your source tree. Then you would have to rebuild the boot CD image and restart the system. <br /><br /> <object width="360" height="288"><param name="movie" value="https://www.youtube.com/v/TIp61IjSQdM?version=3&amp;autohide=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="https://www.youtube.com/v/TIp61IjSQdM?version=3&amp;autohide=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="360" height="288"></embed></object>Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-60055950889528578702017-07-31T21:52:00.000+02:002017-07-31T21:52:00.422+02:00Roland MT-32 unboxingHey there, a while back when I finally got my Roland MT-32, I made this unboxing video. <br /><br /><object width="360" height="288"><param name="movie" value="https://www.youtube.com/v/XuARA-CfZ7E?version=3&amp;autohide=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="https://www.youtube.com/v/XuARA-CfZ7E?version=3&amp;autohide=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="360" height="288"></embed></object>Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-30758213290778756022017-07-29T21:00:00.000+02:002017-07-29T21:00:59.391+02:00Introduction to SILE typesetting systemI love this <a href="https://www.youtube.com/watch?v=5BIP_N9qQm4">SILE presentation video</a> from FOSDEM 2015 by Simon Cozens.Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-90394113502312122802016-11-04T14:17:00.000+01:002016-11-04T14:20:29.323+01:00Alive and kicking!Hey everyone, Fanatik is back!<br /><br /><object width="360" height="288"><param name="movie" value="https://www.youtube.com/v/0GDtrfpzqNM?version=3&amp;autohide=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="https://www.youtube.com/v/0GDtrfpzqNM?version=3&amp;autohide=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="360" height="288"></embed></object>Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-30450225273469487482015-03-24T23:02:00.000+01:002015-03-24T23:02:39.292+01:00Awesome presentation on good API design videoGo watch the video <a href="https://www.youtube.com/watch?v=aAb7hSCtvGw">How To Design A Good API and Why it Matters</a>. It's awesome. Well worth the 1 hour of your time. Boy that guy is good. Also, he must be reading my mind or something.Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-24210348611064984242012-11-12T11:59:00.000+01:002012-11-12T11:59:21.839+01:00Modules in C/C++?It looks like there is <i>again</i> a proposal to standardize <a href="http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1611.pdf">modules in C++</a>. This would be very nice if successful. While I don't care that much about C++, it has been the trend lately that ISO C picks up useful C++ features (e.g. threads). So who knows, maybe, <i>maybe</i> we could finally get modules in C, after 40 years? Pretty please?Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-37322275400244905522012-05-17T13:56:00.000+02:002012-05-18T14:37:33.190+02:00On Limits<blockquote cite="http://arc.opensolaris.org/caselog/PSARC/2002/188/20020403_casper.dik">Past experience has shown that for any number N, and for almost any object O, the statement "we'll never need more than N of O" is invariably untrue. </blockquote><cite style="text-align: right; display: block;">Casper Dik</cite>Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-74345324732327003992012-05-10T21:00:00.000+02:002012-05-13T10:29:17.728+02:00Assessment of Stepan Henek's thesis Security containers and access rights in HelenOS<ul><li>no justification why kernel must be modified to cache the tickets being used by a task</li><li>does not properly analyse UNIX file permission model and how it allows delegation of administration - no discussion of who and when can change owner and mode, s{u|g}id, sticky bit</li><li>interestingly there is actually no proposal for security containers in HelenOS</li><li>many types of tickets - it's not clear whether they are useful for anything and how one would choose among them</li><li>no real analysis/discussion of how the file permissions should be implemented and how it would (dis)allow common tasks such as sharing files, giving away files, (not) enforce users to follow some administrator-given rules for setting file permissions, etc.</li><li>it claims administration can be delegated thanks to the permissions on the individual RBAC virtual folders, but that's not true, these are too low-level and any useful permission given away will necessarily lead to privilege escalation (or at least no (counter)example is presented)</li><li>in the implementation, environment variables are introduced and used (without thought) for the sake of determining the default OGID for newly created files</li></ul> some more thoughts/ideas: <ul><li>not only that the almighty administrator cannot delegate his privileges, an ordinary user cannot delegate his privileges either (just like in UNIX) - no hierarchical principals</li><li>no way to enforce or even suggest permissions for files/directories under a subtree (cf. default ACLs)</li><li>single OGID for a file might be inflexible, especially if object groups can only be created/permissions set by the almighty administrator</li></ul> In summary, the proposal: <ul><li>might make it slishtly easier for an admin to assign permissions to a user</li><li>does not seem to allow any better delegation than the standard UNIX model</li><li>does not make proper analysis of what the model actually allows/ensures w.r.t operations on files and directories</li><li>introduces a new type of kernel resource with no obvious return value</li><li>makes it necessary to make a system call which queries the tickets (i.e. communicates with) a foreign task for any authorization operation, making it a communication-intesive operation (despite that IPC as such is not used)</li></ul> Back to the drawing board, I say!Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com13tag:blogger.com,1999:blog-192079351385094431.post-23235117069277733732012-01-29T15:14:00.000+01:002012-01-29T15:14:28.261+01:00I'm going to FOSDEM<a href="http://www.fosdem.org"><img src="http://www.fosdem.org/promo/going-to" alt="I'm going to FOSDEM, the Free and Open Source Software Developers' European Meeting" /></a><br /><br />You can find me in the <a href="http://fosdem.org/2012/schedule/track/microkernel_os_devroom">Microkernel OS Devroom</a>.Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-65636359716383415372011-11-23T08:03:00.000+01:002011-11-23T13:05:54.060+01:00TCP... the truth will be revealedI have long before recognized that the current network stack in HelenOS is messed up beyond repair and trying to fix it anymore is a waste of time. Since it's been two years after the master thesis was defended and we still do not have something that would work, it might not be well received by others if I threw everything out and started from zero, though.<br /><br />So I started with a complete rewrite of TCP. This is arguably the only more complex / non-trivial module in the stack and the part that is obviously most broken.<br /><br />Despite lack of spare time I have now a new TCP module that is not 100% complete, but complete enough so that it could be test-run against itself over the wire. I implemented it as a completely independent server that has nothing in common with the current networking stack. I tested the functionality using internal loopback in the TCP module and internal clients.<br /><br />Yesterday I started hooking it into the current network stack, the idea is to create a minimal connector between the two components. After three hours of work there is still some way to go before I can successfully send/receive PDUs to/from IP (I haven't started with the socket interface yet).<br /><br />During this work I had the chance to fully 'appreciate' the quirks and complexities of the network stack. IP understands sockets and TCP pseudo headers. TCP knows about IP headers and network devices. But who cares about layers or separation of concerns, right?<br /><br />Since quite a few people claimed that the remainder of the network stack actually works -- to some degree -- I wonder whether I get something at least remotely useful when I plug my TCP implementation into the stack. The truth will be revealed. And then I can maybe throw the rest of the rubbish out.Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-65870388314295765342011-08-10T20:18:00.000+02:002011-08-10T20:18:35.831+02:00The first sentence countsOf the novels and short stories I've read there were two cases where not only they were works of sheer literary genius, but also written with a humour that sold them to me with the very first sentence.<br /><br />In the first case the short story Jupiter Five, my favourite by sir A. C. Clarke, partly due to the excellent and slightly unusual Czech translation.<br /><br /><blockquote>Profesor Forster měl tak drobnou postavu, že pro něho museli vyrobit speciální kosmoskafandr. (Profesor Forster's stature was so small they had to make a special spacesuit for him.)<br /></blockquote><br />In the second case this was the (perhaps more widely known) novel Harry Potter and the Philosopher's stone:<br /><br /><blockquote>Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much.<br /></blockquote><br />When a story starts with a sentence like this, you just know you're in for a treat :-)Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-53618531493946454442011-07-08T15:51:00.000+02:002011-07-08T15:51:31.442+02:00Will ARM architecture go 64-bit?For some time now I have been thinking about the increasing memory capacities of mobile devices and the fact that the ARM architecture, upon which they are built, is still 32-bit only. This was puzzling for me, so I decided to dig around a little. The results are more than interesting.<br /><br />Apparently, in November last year (2010) rumors started spreading that ARM holdings is working on a 64-bit ARM processor (see <a href="http://www.xbitlabs.com/news/cpu/display/20101124221552_64_Bit_ARM_Chip_Design_in_the_Works_Report.html">Xbit labs</a>, <a href="http://www.v3.co.uk/v3-uk/news/1946826/rumours-hint-imminent-arm-bit-processor">v3.co.uk</a>). These rumors were <a href="http://www.computerworld.com/s/article/9208339/ARM_CEO_says_no_rush_to_64_bit_server_chip">denied by ARM CEO Warren East</a> in February 2011.<br /><br />In an interesting turn of events, <i>before that</i> in January 2011 at the Consumer Electronics Show in Las Vegas <a href="http://blogs.nvidia.com/2011/01/project-denver-processor-to-usher-in-new-era-of-computing/">Nvidia announced project Denver</a> which aims to create a 64-bit ARM processor powerful enough to compete on the desktop and server market. At the same time Microsoft announced that Windows 8 will run on ARM as well as on x86. You can read a vey nice <a href="http://arstechnica.com/gadgets/news/2011/01/nvidias-project-denver-cpu-puts-the-nail-in-wintels-coffin.ars">analysis on Ars Technica</a> or the <a href="http://venturebeat.com/2011/03/04/qa-nvidia-chief-explains-his-strategy-for-winning-in-mobile-computing/">interview with Nvidia CEO</a>.<br /><br />Nvidia's plans are very bold. Who knows whether they will succeed. If they do, this could stir the stale waters of x86 deskop quite a bit. Spending the last HelenOS camp finding a bug in the ARM context save/restore routines might not have been a waste of time after all. Who'd have thought? ^_^Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com2tag:blogger.com,1999:blog-192079351385094431.post-85146031749337467612011-05-18T05:00:00.000+02:002011-05-17T17:01:57.494+02:00How to ride your magic carpetMagic carpets are, you know, pretty much commonplace stuff. Everybody rides one, including Prince of Persia and the Rescue Rangers. Now the thing which has always bugged me is the question: How do you control which direction the carpet should fly?<br /><br />Looks like another big mystery solved. And I didn't have to look far for the answer, too. It's all written in, I am not kidding you, the <a href="http://www.usb.org/developers/devclass_docs/Hut1_12.pdf">USB HID specifications</a>. Quoting:<br /><br /><blockquote>Magic Carpet Simulation Device<br /><br />CA – Allows a device to be generally classified as one that uses the standard control of a magic carpet. This control is a bar, grasped by both hands, that controls the Yaw, Pitch and Roll of the carpet.<br /><br />The bar, at which the pilot sits, may be pushed forward or pulled back to cause the carpet to dive or rise, respectively. In the zero position, the carpet is in level flight. Pushing forward on the bar causes the carpet to nose down and generates negative values. Pulling back on the bar causes the carpet to nose up and generates positive values.<br /><br />Turning the bar turns the carpet. In the zero position, the carpet travels straight ahead. Pulling back on the right side turns the carpet to the right and generates positive values. Pulling back on the left side turns the carpet to the left and generates negative values. Rotating the bar rolls the carpet. In the zero position, the carpet travels level.<br /><br />Rotating the bar in a clockwise direction rolls the carpet to the right and generates positive values. Rotating the bar in the counterclockwise direction rolls the carpet to the left and generates negative values.<br /></blockquote><br />Now fortunately HelenOS will soon get USB HID support. So that means I can run HelenOS on my magic carpet's on board computer and still use the <i>standard controls</i> of the magic carpet, right? Right?Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-91514212906300405552011-04-18T22:11:00.000+02:002011-04-18T22:12:50.796+02:00GranumophobiaGetting dynamic linking to work right is a never-ending series of dumped cores. Today I found two bugs which were actually quite generic (i.e. not architecture-specific) and, to be honest, quite silly.<br /><br />First a sizeof(ptr) instead of sizeof(*ptr) passed to memset() caused the dyn_info_t structure not to be properly zeroed out. If something in dyn_info_t is not zero, it is assumed to be valid entry present in the dynamic section. Hence mysterious occasional crashes. Luckily thanks to the new memory allocator which has debugging turned on there was a suspicious quantity of dead beef being thrown around which slightly helped me find the problem.<br /><br />Second, the relocation processing code was dying while trying to fix up references to libc variables in the text segment of the main program, which is stored in a memory area with R-X permissions. Huh? Well, the main program is loaded by loader, instead of dynload. Unlike dynload, loader does not make all loaded segments writable. Bummer.<br /><br />Dynamic linking takes patience. If you suffer from <a href="http://en.wikipedia.org/wiki/Nucleophilic">granumophobia</a>, then stay away.Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com2tag:blogger.com,1999:blog-192079351385094431.post-583758630334558092011-03-19T15:14:00.000+01:002011-03-19T15:14:48.854+01:00Base class constructors? Oh my!When a person with quite limited exposure to OO programming (like me) gets such a crazy idea like designing an OO programming language (like Sysel), they shouldn't be surprised when they forget about something important. Like invoking base class constructors.<br /><a name='more'></a><br />What happened was that while working on constructing the syntax tree in NNPS I was getting exceptions due to accessing nil reference (equivalent of null pointer dereference in C). The problem boils down to the following: we have two classes A and B, B derived from A. Both have constructors, the constructor for A initializes some reference in A. When a new object of class B was created, only constructor for B was called so the reference was not initialized.<br /><br />When a new object is created, constructors must be called for its class and all base classes. This is easy if the constructors do not have parameters (in this case it can be done implicitly). If the constructors do have parameters, the programmer must explicitly specify them (since the compiler does not know what they should be).<br /><br />Different languages solve the problem differently. In C++ (and similar languages) the arguments / call to base class constructor(s) appear to be part of the constructor's signature.<br /><pre>class A {<br /> public: A(int i, int j) { }<br />};<br />class B : public A {<br /> public: B(int i) : A(i,i+1) { }<br />};<br /></pre>Others, like PHP, require the programmer to call the base class constructor with a syntax similar to a static method invocation:<br /><pre>class A {<br /> function __construct(i, j) { }<br />}<br />class B extends A {<br /> function __construct() {<br /> parent::__construct(i, i+1);<br /> }<br />}<br /></pre>The strength of the first approach is that it makes it is syntactically enforcible, i.e.<br />if you forget about it, the compiler can easily tell. In the second case you can do more<br />complex processing before calling the parent constructor (than just evaluating an expression),<br />but it is difficult to tell at compile time whether all code paths contain the constructor<br />invocation (a problem similar to determining if all code paths in a function end in a return statement).<br /><br />I still need to decide how to implement this in Sysel. Or I could just make my base class constructors parameter-less for the time being.Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com2tag:blogger.com,1999:blog-192079351385094431.post-57963183865508620052011-03-19T14:36:00.000+01:002011-03-19T14:36:45.385+01:00GSoC - Great System on a Chip? No!It seems a little pointless repeating what has already been said, but, wow, is this something special! HelenOS has been accepted as a <a href="http://www.google-melange.com/gsoc/program/accepted_orgs/google/gsoc2011">mentoring organization for 2011 Google Summer of Code</a>.<br /><br />What more to say? It feels great to be in company of so many well-known open-source projects from areas of OS, desktop, applications, games, scientific software, etc., etc.<br /><br />Now we need some students!Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com1tag:blogger.com,1999:blog-192079351385094431.post-2007331071482842362011-01-03T22:38:00.000+01:002011-01-04T23:56:26.959+01:00Observations on HelenOS device drivers frameworkSome time ago we merged Lenka's new Device Drivers Framework (DDF henceforth) to HelenOS mainline. When I finally read through her thesis and had a closer look at the code I promised to do a writeup of my observations. Here it is. A bit of a long read though, be warned.<br /><br />Let's start with a quick overview. Lenka's contribution can be broken down into several parts. First there is the <tt>devman</tt> server which (a) manages the physical device topology, driver task life cycle, driver attaching and detaching, (b) maintains a 'logical' device topology (devices grouped by their class).<br /><br />Second, Lenka proposed that for any IPC communication between two drivers or an application and driver should be strictly defined in terms of function calls and the client- and server-side IPC glue code should be kept strictly separate in a library to be reused by all servers and clients using such code.<br /><br />Third, when mapping the function calls to IPC, Lenka reserved the first field of the message for an <em>interface id</em>. This allowed her to communicate with a driver using several different protocols, namely a protocol for talking to the driver as such (create new instance) and for talking to the device itself (e.g. read/write, etc.).<br /><br /><a name='more'></a><br /><br />As you can see Lenka took on quite a challenge and tried to cover area that extends quite far past mere device management. I will not try to cover it all here, if you are interested, go ahead and read her thesis. I will only focus here on parts that I found particularly interesting, where I found room for improvement or that inspired me somehow.<br /><br />Summary of my observations. I have some practical concerns about the DDF, as it is implemented now:<br /><ul><li>Problematic support of pseudo-devices (e.g. <tt>file_bd</tt>)</li><li>Problematic support of 'multi-function' devices (e.g. <tt>i8042</tt>)</li></ul><br /><br />I believe these need to be addressed before we can convert our current drivers to the new driver model. Also, what I don't like is:<br /><ul><li>Making devices 'special' and making them use different communication than the rest of the system.</li></ul><br /><br />There are also two topics which I haven't seen mentioned in the thesis although they are highly relevant:<br /><ul><li>parallelism in device enumeration</li><li>persistent device naming</li></ul><br /><br />I have some ideas what could be handled better or differently, which I will outline below.<br /><br /><h2>IPC communication suggestions</h2><br /><br /><h3>The problem</h3><br /><br />Lenka faced a practical problem when communicating to a driver. Sometimes she needed to control the driver as such using a generic driver protocol (e.g. add_device operation), sometimes she needed to talk to the driver using a device-specific protocol (e.g. char_read, char_write), etc.<br /><br />For this reason Lenka came up with the idea that a driver could implement multiple different protocols. She reserves the first field of an IPC message for the protocol ID, moving the method ID effectively to the second field of the message.<br /><br />Lenka envisaged another use of the multiple interface support, namely that a device could support multiple device-specific protocols and that the interfaces and their API stubs could be reused (in a duck-type manner) and combined, possibly for semantically completely different (but syntactically equivalent) sets of operations.<br /><br />The first thing that I don't like at all is that here devices drivers are made somehow special and they use a communication protocol completely different from the rest of the system services. There is no reason for that, it will just make communication between different types of tasks incompatible.<br /><br /><h3>Better server task model</h3><br /><br />The proposed mechanism looks useful and something <em>similar</em> could/should be used for <em>all</em> IPC communications, not just to the device drivers!<br /><br />Recently Martin Děcký pushed some changes where he started a change in order to embed the interface ID (in the same sense as Lenka uses) to the first IPC message field (along with method ID) to use it in general IPC communication.<br /><br />Great! This is way better for two reasons:<br /><ol><li>It is generic, applies to all IPC communication equally</li><li>It is more space-conscious, does not consume an entire argument</li></ol><br /><br />Still, I think we can do much better than that. (I already outlined this in the mailing list, but I'll repeat it here).<br /><br />First, embedding the interface ID in every message is wasteful, it stays the same for the duration of an exchange, at the minimum.<br /><br />Second, I think the problem being solved stems from the fact that we are trying to control different entities within the same server task. So, instead of trying to talk different protocols to the same entity, we are, in fact, trying to talk to different entities (each speaking different protocol, incidentally).<br /><br />The model of a server task can be described in terms of object-oriented programming:<br /><ul><li>Lenka says: A server task is an object that implements one or more interfaces.</li><li>Jiří says: A server task is a collection of objects, each belonging to (exactly one) class (classes form a true inheritance hierarchy).</li></ul><br /><br />If we consider the driver to be a <em>collection</em> of objects (driver control object, device control object(s)), it also solves the problem for the case of the device driver.<br /><br />Another way to look at this is that we add another level of addressing. We can imagine that we access the inside of the server task via different points of access. We can call them <em>ports</em> in analogy with TCP/IP ports (or RM ISO/OSI SAPs) which allow talking to different entities within the same IP interface (IP node, IP address).<br /><br />One of the benefits of ports is that when a server provides access to a set of resources of some type (e.g. file, device) we can map each individual resource to a port. Then we don't need a special identifier (file descriptor, device descriptor) since the individual resource is already identified by the port (port ID). So we save one message field plus the framework actually understands that this is an identifier.<br /><br />Port references can be optionally made unforgeable (which is a very useful property). Ports can be implemented in different ways and have numerous other benefits which I will not detail here if just not to scare you off ;-)<br /><br />Implementation note: To allow the async framework to embed some information (any of interface ID, port ID, virtual path ID) automatically into a request slight changes to the async API are needed (when making a request we should pass a pointer to an exchange instead of integer/phone ID). Nevertheless the conversion of applications and servers is straight-forward and with a split API it can be implemented gradually, if needed.<br /><br /><h3>Better use of message space</h3><br /><br />When talking to a port, we'll talk the same protocol the entire time. Therefore, we can establish that right upon connecting. Not only we eliminate redundancy, but we can tell that someone does not speak the expected protocol straight away (better than recognizing it upon the first operation).<br /><br />So rather than embedding interface ID in the message, we can embed the port ID (having established the protocol spoken by a port already).<br /><br />Still, an entire <em>exchange</em> (see <a href="http://trac.helenos.org/trac.fcgi/wiki/AsyncSessions">async sessions</a>) will always talk to just one port, so this still seems a little wasteful.<br /><br />We could instead embed something that would allow us to identify different <em>exchanges</em> and multiplex parallel operations onto a single physical IPC connection.<br /><br />A good way to implement that seems to embed a <em>virtual path ID</em>. Virtual paths would be set up and torn down by the async framework (session code) on the client side. Each virtual path would be assigned an ID. The server would create a fibril to service each virtual path (within each connection). The session management code would no longer create (kernel-based) IPC connections for concurrent communication, but instead it would create (userspace-based) virtual paths. This provides and alternative (although not necessarily better?) to the current implementation, which creates multiple physical connections.<br /><br />The session API combines with this in the following way:<br /><ul><li>A client initiates a session to a server port.</li><li>On the session the client performs exchanges as usual.</li><li>Internally the async framework creates as many parallel virtual paths (within the physical connection) as necessary</li><li>The async framework on the server side creates a handler fibril for each incoming virtual path</li></ul><br /><br /><h2>Device model suggestions</h2><br /><br /><h3>Current model</h3><br /><br />In the current model we have two types of device nodes: nexus nodes and leaf nodes. A nexus node ('bus device') consists of several objects:<br /><ul><li>1 driver control object (generic driver protocol)</li><li>1 nexus (bus) device control object (device-specific protocol)</li><li><em>n</em> device attachment points (bus-specific protocol)</li></ul><br /><br />A leaf node consists of:<br /><ul><li>1 driver control object (generic driver protocol)</li><li>1 leaf device control object (device-specific protocol)</li></ul><br /><br />Important to note is that the driver control object and device control objects always coexist (you can't have one without the other).<br /><br /><h3>Problems</h3><br /><br />The biggest problem I see in the current model is that it requires a 1:1 relationship between a physical device (something to which you can attach a driver) and its logical functions (services provided by the device), at least for leaf drivers (you can't have one without the other).<br /><br /><h4>Pseudo-driver</h4><br /><br />Let's call a pseudo-driver any driver whose life-cycle is not naturally controlled by bus enumeration. An example is <tt>file_bd</tt> whose instance is naturally created when the user types in some command.<br /><br />If we wanted to port file_bd to the current DDF we would need to also add some special pseudo nexus driver (possibly a special one for file_bd) which the application would tell to attach a new child. Then we would need to somehow communicate the parameters to the child and...aargh!<br /><br />While the plain devmap-style drivers can be modified to speak a protocol compatible to the devman-style drivers, they are not fully fledged devices. We can't add them to device classes, so if we obtain a list of devices in some class, these pseudo-devices won't be included.<br /><br /><h4>Multifunction device</h4><br /><br />Let's call a multifunction device any device which behind one physical interface provides access to multiple logical functions.<br /><br />Buses can be considered multifunction devices (they expose multiple device attachment points). Our concern however are multifunction leaf devices. A typical example is i8042 (which provides two character interfaces) or a (partitioned) disk (which provides multiple block interfaces).<br /><br />If we wanted to port these drivers to the current DDF, we would need to implement them in terms of a nexus device plus a leaf (connector) device. Even if we implemented both the nexus and connector device inside a single driver binary, it is just unnecessary complication. The connector device would do nothing but forward IPC communication to the nexus device. Why clutter device drivers with completely generic code?<br /><br /><h3>Proposed solution</h3><br /><br />I propose a more uniform device model, where each (nexus or leaf device) consists of:<br /><ul><li>1 driver object</li><li>1 (physical) device object</li><li><em>n</em> logical functions</li></ul><br /><br />Each logical function can be either a <em>external / leaf function</em> (exposed to clients outside of DDF) or <em>internal / non-leaf function</em> to which the DDF can attach more levels of the device hierarchy.<br /><br />The choice of whether a logical function is external or internal could be up to the (nexus) device driver or, possibly, it could be made to depend on other factors (such as whether a device driver could be attached). Mixed-mode (leaf and nexus) drivers would be possible, if desired.<br /><br />More importantly, multifunction leaf devices would be easy to implement (just change the flavor of the functions from internal to external and voila). No need for implementing connector device nodes in such driver.<br /><br />Also we can make it possible for any server to call the DDF and say 'hey, I'm a pseudo driver' and to create logical functions. This enables us to construct (leaf) pseudo-device drivers in a simple and straightforward manner. These don't need to be included in the physical device tree (equivalent of UN*X /devices). Only their logical functions are included in the logical device function name space (equivalent of UN*X /dev).<br /><br />An interesting problem is the existence of pseudo nexus devices. A pseudo nexus device <em>must</em> be included in the physical device hierarchy (because its children are not pseudo devices in the sense described above). If we choose to allow pseudo nexus drivers, we will need, in the end, some 'pseudo' node in the device tree as a root for them. Still, we will have the benefit of manual device lifetime management. I am not sure whether pseudo nexus devices are useful or not.<br /><br />As a side note, the operation <em>add logical function</em> is (conceptually) more generic than <em>add child</em> because it does not imply what gets connected to the attachment point (logical function). In a multi-pathed device scenario the device node is not attached directly to the parent node. Instead for some bus B there is a multiplexer which connects to each B host adapter node. All device nodes then attach to this multiplexer. (So their connection to the bus host adapter node is indirect).<br /><br /><h2>Providing device services</h2><br /><br />Currently devman manages <em>device classes</em>. A device driver will register a device in some class and then applications can e.g. list all devices in some class.<br /><br />This is not necessarily bad, although it might be an unnecessary complication. If we implement location services in the CORBA sense (naming and trading service), they will provide the same functionality for all IPC services.<br /><br />(Explanation of what I mean here by LS in the CORBA sense: Any server can register a (non-singleton) service, specifying its <em>name</em> (generated hierarchical unambiguous (string) identifier) and <em>type</em> (hierarchical name of a class/type of the service). Then we can list all services, services by class, etc.)<br /><br />Now it is perfectly possible to export services provided by some device via the location services instead of implementing a special registry in devman.<br /><br />So, in the scenario described above, there would seem to be little added value in using a special devman class database, instead of the more generic (and possibly smarter) location services.<br /><br /><h2>Corollary</h2><br /><br />I described what I see as important limitations of the current HelenOS DDF. Lenka's work inspired me with many interesting ideas, which I described. I proposed a number of improvement proposals either to the DDF itself or in IPC. The proposals form a coherent vision, but many of the individual ideas are to a degree independent and thus they can be implemented individually and gradually.<br /><br />Thanks for reading.Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0tag:blogger.com,1999:blog-192079351385094431.post-50864428812009894102010-12-28T19:54:00.000+01:002010-12-28T22:15:19.868+01:00Mister triple W. meets miss H.I think the following image speaks for itself.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_UVqyMZ9NOdM/TRoyodKet0I/AAAAAAAAABo/hBtHAY4OFHU/s1600/test.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 215px;" src="http://4.bp.blogspot.com/_UVqyMZ9NOdM/TRoyodKet0I/AAAAAAAAABo/hBtHAY4OFHU/s320/test.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5555808760919668546" /></a><br /><br />After my recent bug fixes, HelenOS is able to serve a web page from inside Qemu. TCP is still shaky, sometimes it works, sometimes it does not. But it's still progress :-). In the past weeks and months I got my hands dirty in the networking stack and I tell you, the code is <i>weird</i> throughout!<br /><br />The web server is mostly fake, but it does serve its purpose :-) I should be committing it shortly.Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com1tag:blogger.com,1999:blog-192079351385094431.post-52615034166399769442010-12-03T20:36:00.000+01:002010-12-03T11:29:27.276+01:00A moment to rememberIf you don't understand Czech, turn on CC (captions) in the player before starting the video.<br /><br /><object width="360" height="288"><param name="movie" value="http://www.youtube.com/v/oFPzVByyMtg?fs=1&amp;hl=en_US"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/oFPzVByyMtg?fs=1&amp;hl=en_US" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="360" height="288"></embed></object>Jiří Svobodahttp://www.blogger.com/profile/09911075768532401265noreply@blogger.com0