Blog on camlcity.org

Blog on camlcity.org http://blog.camlcity.org en Articles by Gerd Stolpmann about O'Caml Documentation fun http://blog.camlcity.org/blog/pxp121.html http://blog.camlcity.org/blog/pxp121.html <div> <b>PXP-1.2.1 with a new reference manual</b><br>  </div> <div> The last release of a stable PXP version happened 5 years ago. That's a long time. Actually, a lot of devlopment took place since then, only that it was difficult to bring PXP into a releasable shape. Now the last missing piece has been added, namely extensive documentation. So I can proudly announce the best XML parser that has ever been available for programming in O'Caml: Not only fast and feature-rich, but also easy to understand. </div> <div> <p> Writing documentation is something programmers do not like very much, and also in the case of PXP the code was far ahead of any description about it. There was a "User's Guide", but it took an oldish approach of explaining things I don't like anymore. Also, it was very incomplete. Last year, I got some funding from a company to improve the PXP documentation, so I faced the problem to reorganize it completely, and to add anything missing. </p><p> If you would like to take a look at the result, here it is: <a href="http://projects.camlcity.org/projects/dl/pxp-1.2.1/doc/manual/html/ref/index.html">The PXP Reference</a>. </p><h2>Switching from docbook to ocamldoc</h2> <p> The old "User's Guide" was written as docbook document. This is a good general-purpose text format that allows one to structure a large text into chapters, sections, etc., and to generate viewable and printable output from it (especially one can convert it into a bunch of HTML pages, and into PDF). However, there is one difficulty: It does not integrate well with ocamldoc-style interface references.</p> <p> The "User's Guide" predates ocamldoc - when I wrote the first version of PXP documentation I had no other chance than to use some third-party tool to process it. Now what to do? Stick with docbook, and include ocamldoc somehow into the processing chain? The docbook format has clearly more features for formatting text, e.g. one can easily include pictures. However, ocamldoc cannot output in a format that would be convertible to docbook with only little effort, and this made this way unfeasible. </p><p> I decided to switch completely to ocamldoc. Not only the module interfaces should be documented with it, but also the various introductory chapters explaining concepts spanning several modules. Since O'Caml 3.09, ocamldoc understands the file suffix *.txt and takes these input files as pure documentation. One can still use all formatting directives like <code>{2 headings}</code> or <code>{!Hyperlinks}</code> pointing to code elements. However, there was still the difficulty of missing features. </p><p> So I looked at developing a custom HTML generator (I am mostly interested in outputting HTML). It is possible to load an add-on into ocamldoc that modifies its behaviour. One just has to write a class that inherits from <code>Odoc_html.html</code>, and overrides its methods: </p><pre> class chtml = object(self) inherit Odoc_html.html as super method private html_of_<foo> ... = ... end let chtml = new chtml let _ = Odoc_args.set_doc_generator (Some chtml :> Odoc_args.doc_generator option) </pre> <p> Of course, it was still the question whether my features could be added this way (without rewriting half of the generator class). Yes, they can, and it only needed about 160 lines of code. I must admit it took quite a long time to develop this code, since I had to dig into internals of ocamldoc to understand it better. But anyway, ocamldoc turns out to be a customizable utility. </p><p> What I added in particular: </p><ul> <li>A <code>{picture}</code> tag for including pictures<br>  </li><li>The possibility to change the output for <code>include Module</code> in interfaces so that the included interface is directly shown instead of only the include statement as such. For clarity, the included interface is indented, and has grey background. This change can be turned on and off with a <code>{directinclude}</code> tag.<br>  </li><li>The <code>include</code> change requires another feature to be really looking good. All references (hyperlinks and plain occurrences) pointing to the included module should be rewritten so that they point to the including module instead. That means if module <code>N</code> uses <code>include M</code> we want that all references <code>M.x</code> are changed into <code>N.x</code>. The intention is that <code>M</code> is no longer referenced, and that the duplication of definitions in two modules cannot confuse readers (especially those that are unfamiliar with the module system). I added that feature for my specific case, and the ocamldoc tag <code>{fixpxpcoretypes}</code> enables that rewriting. (It changes <code>Pxp_core_types.[S|I]</code> into <code>Pxp_types</code>.) <br>  </li><li>A last change has also to do with the <code>include</code> feature. With <code>{knowntype}</code> and <code>{knownclass}</code> one can add identifiers to the lists of known types and classes, so that the generator will emit hyperlinks to them, although there is no such definition in reality. It turned out that many identifiers were already pointing to the including module, but because there is no definition in the mli file, ocamldoc does not make these identifiers clickable. With <code>{knowntype}</code> and <code>{knownclass}</code> one can change that on a case by case basis. </li></ul> <p> The full source code of the custom generator class can be studied here: <a href="https://godirepo.camlcity.org/svn/lib-pxp/tags/pxp-1.2.1/tools/src/odoc/chtml.ml">chtml.ml</a>. The module with the mentioned <code>include</code> directive is <code>Pxp_types</code>. <a href="http://projects.camlcity.org/projects/dl/pxp-1.2.1/doc/manual/html/ref/Pxp_types.html">Look here how nice the generated page is.</a> </p><h2>The need for conceptual introductions</h2> <p> XML is a cute and simple text format, right? Many people think like that, but given the fact that many XML parsers are either feature-rich and slow, or poor and fast, there must be some complexity in the XML definition. Recently, I read the article "XML fever" (by Erik Wilde and Robert J. Glushko, Communications of the ACM, issue 7, 2008), where the authors point out a number of deficiencies in the definition of XML that can lead to delusion about XML, and finally into "fever". After years of maintaining this XML parser, I can only second the authors. Clearly, there are problems even in the fundamental XML specification. </p><p> I do not want to complain about this - XML is widely used, and many of the standards are practically unfixable without breaking large numbers of programs. For me the problem arose how to explain all that. For example, there is the question what is to be considered as the root node of an XML tree. This is a conceptual question, and the explanation should not be hidden in an interface description of a PXP module. For that reason, I had to add a number of chapters to the manual that explain concepts and generally introduce into the PXP world. All the <code>Intro_*</code> chapters are like this. </p><p> The nice thing is now that I can add direct links from introductory chapters to interface references and vice versa, since all documentation is now processed with the same utility, ocamldoc. When some complicated issue arises in some function description, it is now possible to point to the section in the introduction where this issue is explained in detail, and conversely, I can point to the definition in the interface when a function or type is used in an intro chapter. </p><h2>What next?</h2> <p> I must admit that my interest in XML has not gained in the last years, to say it politely. XML is most often used as a base technology for HTML, or as a data exchange format. Many of the advanced XML standards like XSLT or XQuery have not found the way into the daily life of us programmers. The hype is over. </p><p> Nevertheless, I promise that I will still maintain PXP, and now and then add another feature. For example, there is a nice XPath evaluator in the development pipeline - again, I do not find time to finish it, but hey, there are still many years for doing it. (By the way, if you want to accelerate that and have some money, we will find a way to quickly finish XPath.) </p><p> In August 2009, PXP becomes 10 years old (counted from the first mentioning in the O'Caml mailing list). This is already a long time for a software library and an open source project. I am quite confident it will now also reach its 20th birthday! </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> <p><b>Links:</b> <ul> <li><a href="http://projects.camlcity.org/projects/pxp.html">PXP homepage</a>: Find here links to downloads and documentation </li> </ul> </p> </div> Parallelizing with Ocamlnet http://blog.camlcity.org/blog/parallelmm.html http://blog.camlcity.org/blog/parallelmm.html <div> <b>A scalable implementation of matrix multiplications in O'Caml</b><br>  </div> <div> O'Caml seems not be recognized as a programming language where it is easy to parallelize tasks. Recently, there was a heated discussion on caml-list about this subject, and people complained that the memory management of O'Caml does not work well for multi-threaded programs. Actually, the memory manager enforces that only one O'Caml thread can execute code at any time, so that there is no way to make efficient use of multi-core CPUs. Well, there are ways around this limitation, and in this article I'll show how to multiply matrices in a distributed system. We don't use multi-threading, but multi-processing which means that the threads of execution don't have easy access to shared memory. For communication and synchronization we use remote procedure calls which can basically do the same as shared memory, but at higher cost. Most important, however, is that the restriction of the O'Caml memory manager no longer applies: The processes are independent, and every process has its own manager. </div> <div> <p>In order to tackle problem we need a way to break down the algorithm so that the running processes don't have to synchronize with each other often. This is very easy for the matrix multiplication (but may be more complicated for other problems): Every cell of the result matrix is the scalar product of one column of the first input matrix with one row of the second input matrix. Thus the cells can be computed independently, and one only has to take care that all processes have access to the input matrices, and that the processes do not interfer with each other when writing the results. </p><h2>The tools</h2> In this solution of the problem we use the toolset provided by Ocamlnet. It includes a mature and quite efficient implementation of SunRPC we'll use for interprocess communication. Furthermore, there is Netplex for creating and managing multi-process servers. The plan is to create an RPC service for multiplying matrices that can take advantage of several cores and that can even run in a compute cluster. <p>The Netplex/SunRPC combination is quite powerful. Basically, a SunRPC server accepts TCP connections, and executes the incoming RPC calls. With our tools we can choose whether we want to have a single process that processes all connections, or whether we want to create new processes for every connection. Obviously, the latter is the way to go for doing the hard computation work, because O'Caml allows us only to take advantage of several cores by running the code in different processes. The single process choice is also useful, namely for controlling the computation, i.e. for synchronization. In this example we'll use both types of process management. </p><p>SunRPC is a quite aged standard for doing remote procedure calls. RPC is typed message passing, that means we cannot only pass strings from one process to the other, but values of all types of the interface definition language. In SunRPC we have ints, floats, strings, arrays, options, records ("structs"), and variants ("unions"). Furthermore, RPCs are usually "two-way" messages: Input messages are always replied with output messages. </p><h2>The components</h2> We have two components. One is called the "controller" and manages the work to do. Actually, the controller defines the outer loops of the algorithm. Furthermore, there is the "worker" component. The worker can be instantiated several times (and runs then in several processes). It implements the inner loop of the algorithm that runs in parallel. <p>The general idea is that the controller triggers the workers when a new multiplication is to be done, and that the workers are themselves repsonsible for getting the input data, and for getting the tasks to execute. In the SunRPC IDL this looks like: </p><blockquote><small> <pre> program Controller { version V1 { void ping(void) = 0; dim get_dim(which) = 1; /* Workers can call this proc to get the dimension of the matrix */ row get_row(which,int) = 2; /* Workers can call this proc to get a row of the matrix. The int is the row number, 0..rows-1 */ jobs pull_jobs(int) = 3; /* The controller maintains a queue of jobs. This proc pulls a list of jobs from this queue. The int is the requested number. */ void put_results(results) = 4; /* Put results into the result matrix. */ } = 1; } = 2; program Worker { version V1 { void ping(void) = 0; void run(void) = 1; /* The controller calls this proc to initiate the action in the worker. When it returns, it is assumed that the worker is completely finished with everything. */ } = 1; } = 3; </pre> </small></blockquote> (There is also a program Multiplier with program number 1 we'll show later.) <p>These are the procedures the two components expose. When a multplication is to be done, the order of actions is as follows: </p><ol> <li>We assume the controller has the input data. The controller calls the "run" RPC of all workers to trigger them. </li><li>The workers now fetch the input data from the controller by calling "get_dim" and then "get_row" until they have all input values. </li><li>The workers now run jobs from the controller by repeatedly calling "pull_jobs", then executing the jobs, and finally transmitting the results back to the controller by invoking "put_results". The workers can pull several jobs at one, and they can put several results at once. This reduces the relative costs of the RPC overhead per job. </li><li>The workers are done when there are no more jobs, i.e. the list of jobs pulled from the controller is empty. Now they can pass back the result message of "run" to the controller. </li></ol> Note that the controller must be able to do several things in an overlapped way: After it has sent the input messages to the "run" RPC of the workers it must be responsive and accept incoming RPC calls from the workers. Finally it must wait until all result messages from the "run" RPCs have arrived. Ocamlnet's RPC implementation supports such overlapped RPC execution. (Note that Sun's original implementation does not support this!) <p>The Multiplier program is just another interface of the controller. It is the public interface used by the client of the multiplier: </p><blockquote><small> <pre> program Multiplier { version V1 { void ping(void) = 0; void test_multiply(int,int,int) = 1; /* Creates a test matrix with random values and multiplies them. Args are: (l_rows, r_cols, l_cols = r_rows) */ } = 1; } = 1; </pre> </small></blockquote> <h2>The implementation</h2> The complete source code is available here: <ul> <li><a href="https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/examples/rpc/matrixmult/">code/examples/rpc/matrixmult</a> </li></ul> It will also be available in future releases of Ocamlnet, but for now it is only accessible via Subversion. <p>Of course, using a distributed approach creates a lot of coding overhead. Compare with the direct implementation in simple.ml (also in this directory): The multiplication algorithm counts there only 9 lines of code. Compared with that we had to develop a lot of additional stuff: 389 lines for the distributed implementation. This is the price we have to pay in terms of coding effort. </p><p>Netplex requires a configuration file, here mm_server.cfg. Basically, the file lists the components that are supposed to accept incoming connections. Here, it looks like: </p><blockquote><small> <pre> netplex { ... service { name = "mm_controller"; protocol { ... }; processor { type = "mm_controller"; worker { host = "localhost"; port = 2022 }; worker { host = "localhost"; port = 2022 }; }; workload_manager { ... }; }; service { name = "mm_worker"; protocol { ... }; processor { type = "mm_worker"; controller_host = "localhost"; controller_port = 2021; }; workload_manager { ... }; }; } </pre> </small></blockquote> We have omitted boring parts ("..."). We configure the mm_controller that connects with two workers, both reachable over TCP port localhost:2022, and the mm_workers talking to the controller over port localhost:2021. By including the same worker port twice in the list of workers, two independent connections are created, and because of the worker configuration two processes are started. (This is configured in the omitted workload_manager block.) <p>Netplex starts exactly the components it finds in the configuration file, even if more components are implemented in the code. This is quite useful, because the same binary can be used for different setups. In this example, we could run the multiplication server on host X with both controller and workers, and on host Y with only the workers. </p><h2>Some numbers</h2> I've done some performance tests on a Opteron 2212 system with 2 CPUs and each CPU has 2 cores (i.e. 4 cores in total). The tests were done in 64 bit mode and ocamlopt-compiled programs. The interesting question is not the total performance, but how much overhead we have by the message passing, and how scalable the system is. <p>The test is to square an NxN matrix of random numbers, for N=1000, N=2000, and N=3000. This has been done with the non-distributed version of the multiplation (simple.ml), and with the distributed version and W workers, for W=1, W=2, and W=4. </p><p>What we expect is that for larger input matrices the relative cost of message passing drops, because we have to transfer the input matrix only once for every worker, and the result matrix only once in total, but the computation time for the multiplication is O(N<sup><small>3</small></sup>) compared to O(N<sup><small>2</small></sup>) for message passing. </p><p>Here are the runtimes, as table, and as diagram: </p><table cellspacing="5" border="1"> <tr> <th>Test</th> <th>N</th> <th>Runtime (seconds)</th> </tr> <tr> <td>Non-distributed</td> <td>1000</td> <td>38.61</td> </tr> <tr> <td>Non-distributed</td> <td>2000</td> <td>359.765</td> </tr> <tr> <td>Non-distributed</td> <td>3000</td> <td>1483.759</td> </tr> <tr> <td>Distributed, W=1</td> <td>1000</td> <td>45.174</td> </tr> <tr> <td>Distributed, W=1</td> <td>2000</td> <td>382.874</td> </tr> <tr> <td>Distributed, W=1</td> <td>3000</td> <td>1847.225</td> </tr> <tr> <td>Distributed, W=2</td> <td>1000</td> <td>24.407</td> </tr> <tr> <td>Distributed, W=2</td> <td>2000</td> <td>206.426</td> </tr> <tr> <td>Distributed, W=2</td> <td>3000</td> <td>849.089</td> </tr> <tr> <td>Distributed, W=4</td> <td>1000</td> <td>20.842</td> </tr> <tr> <td>Distributed, W=4</td> <td>2000</td> <td>144.718</td> </tr> <tr> <td>Distributed, W=4</td> <td>3000</td> <td>576.02</td> </tr> </table> <div> <img src="/files/img/parallelmm_runtime.png" width="450" height="588"> </div> <p>Not bad, right? The time spent for message passing seems to be acceptable. If you compare the non-distributed time for N=3000 with the time of the distributed system and one worker, the overhead is approximately 25% of the total runtime. </p><p>For larger W the overhead for the initial transfer of the input matrices increases - they have to be copied to every worker. The algorithm can be improved here, because the controller process becomes soon the bottleneck as it has to copy the matrices for every worker. Alternatively, the matrices could be spread among the workers by the workers themselves in order to get a better degree of parallelization during the transfer phase. </p><p>As last, fun test I started workers on three nodes so that in total 10 cores were available (I couldn't do more because I did not find more idle systems in Wink's cluster). I got a runtime of 349 seconds for N=3000. As the cores were not identical, this does not tell us much except that we really can do it over the network, and that this multiplier really takes advantage of a cluster. </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> <p><b>Links:</b> <ul> <li><a href="http://projects.camlcity.org/projects/ocamlnet.html">Ocamlnet</a>: Project page </li> <li><a href="http://caml.inria.fr/pub/ml-archives/caml-list/2008/05/6e74a918c2903f1e11add5714a72e352.en.html">Thread about parallelization on caml-list</a>: Sometimes it sucks, sometimes it rocks </li> </ul> </p> </div> About LambdaRank http://blog.camlcity.org/blog/lambdarank.html http://blog.camlcity.org/blog/lambdarank.html <div> <b>The ranking algorithm behind GODI Search</b><br>  </div> <div> The question has always been: Are the results better if a search engine really understands the text it indexes? You can view my latest project, <a href="http://docs.camlcity.org">GODI Search</a>, as an attempt to answer this question for a very limited set of documents, namely the code and its documentation of <a href="http://godi.camlcity.org">GODI</a>, the source code O'Caml distribution. Actually, understanding text allows it to rank the results better, so that the more important occurrences of the query words are shown more at the beginning of the result list. Because ranking really bases on text interpretation, and the text is here a variant of lambda code, I call this method LambdaRank. </div> <div> <p>As every search engine, GODI Search consists roughly of two main components, namely the indexer and the searcher. The indexer iterates over the set of documents, and populates a database with the words it extracts. The searcher is the simpler part of the game, as it finds everything well-prepared in this database, and when a query comes in, it only has to pull the matching documents out of the database, sort it according to the ranking score, and show it to the user. So the tricky part is the indexing. </p><p>If you don't understand the text at all, you cannot do much about ranking but count the occurrences of a word in a document. The idea is that a text that speaks about a certain subject also mentions the subject more often than other texts, and thus the number of occurrences is a good measure for ranking. In a document set where the texts are connected with hyperlinks one can furthermore look at the relationships between the documents. Google's PageRank is based on this approach (but, as rumors say, is now heavily modified from its original design). </p><p>Fortunately, we know a lot about the document set GODI Search analyzes. Most documents are like this: </p><ul> <li><a href="http://docs.camlcity.org/docs/godipkg/3.10/godi-ocaml/lib/ocaml/std-lib/list.mli">O'Caml module interfaces</a> </li><li><a href="http://docs.camlcity.org/docs/godipkg/3.10/godi-ocaml/lib/ocaml/std-lib/list.ml">O'Caml module implementations</a> </li><li><a href="http://docs.camlcity.org/docs/godipkg/3.10/godi-omake/doc/godi-omake/html/omake-doc.html">Technical manuals</a> </li><li><a href="http://docs.camlcity.org/docs/godipkg/3.10/apps-coq/share/emacs/site-lisp/coq.el">Code in programming languages other than O'Caml</a> </li><li><a href="http://docs.camlcity.org/docs/godipkg/3.10/apps-coq/lib/coq/ide/utf8.v">Code in exoctic languages</a> </li></ul> O'Caml code files and closely corresponding manuals dominate the corpus. GODI Search tries to make the best of this by analyzing O'Caml code in detail. <p>After looking at many examples I had some ideas which occurrences must be ranked higher than others. </p><p><b>Idea 1. Definitions of identifiers are more important than uses.</b> This sounds natural, but actually not every definition is important. GODI Search also looks at the scope of the definition: A local definition is restricted to a surrounding function, and scores the least. Then follow definitions on module level, and in top-level modules. The highest score is given to exported identifiers that occur in top-level module interfaces. </p><p>Only let-bound identifiers are ranked this way. Function arguments, variables in pattern matchings, and fun-bound identifiers are ignored. This is a bit arbitrary, but my feeling is that these identifiers are usually not important in the coding styles I'm aware of. </p><p>For types, exceptions, and module names similar scoring techniques exist. </p><p><b>Idea 2. Values and types are rated separately.</b> The namespace of all identifiers can be roughly divided into two big zones: Values and types. Of course, there are more kinds of identifiers (modules, classes, labels, file names,...), but my impression is that the typical programmer has a mind set that is dominated by only these two classes of symbols. In this sense, a "value" names an executable thing, and a "type" names meta data. Both have little to do with each other, and thus a document that contains many words "list" as type has little importance in a search for "list" as value. </p><p><b>Idea 3. Keywords are stopwords.</b> Keywords occur in practically every code file in big number, and thus say nothing about it. GODI Search simply ignores keywords. </p><p><b>Idea 4. Qualified identifiers are hyperlinks.</b> If you search for "mem" then you will get a list of top-level definitions of this function. The question is which occurrence is shown first. GODI Search implements the PageRank idea of scoring hyperlinks pointing to a document by looking at qualified identifiers. So if there are more "Hashtbl.find" than "List.find" in code anywhere in the corpus, the module "Hashtbl" scores higher than "List". (Actually, it is the other way round if you also take other references into account.) </p><p><b>Idea 5. Code and non-code are rated separately.</b> Of course, the above applies only to text sections that are O'Caml code. Other languages and non-code cannot be rated this way. For this reason, GODI Search puts a lot of effort into separating both types of text. Currently, this is done on a per-paragraph basis, i.e. every paragraph is first analyzed in order to know whether it is O'Caml code or not. Also, comments and string literals in code files are considered as non-code. </p><p>So far about the ideas behind LambdaRank. The results of the implementation look promising: If the user types in "fold_left" he or she will be taken to really relevant occurrences. And user experience is what counts, finally. </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> <p><b>Links:</b> <ul> <li><a href="http://docs.camlcity.org">GODI Search</a>: Try out GODI Search here </li> <li><a href="http://godi.camlcity.org">GODI Homepage</a>: About GODI in general </li> </ul> </p> </div> Cross-Language Cluster Computing http://blog.camlcity.org/blog/hydrostory.html http://blog.camlcity.org/blog/hydrostory.html <div> <b>The Story Behind Hydro</b><br>  </div> <div> On the surface a search engine looks like a very simple web site, but actually most things happen in the backend, and are hidden from the user. A search engine consists of 10 to 20 different types of servers, and many of them are instantiated several times in a cluster configuration. No single programming language is well suited for the entire implementation. In addition, some language environments may have compelling libraries that are lacking in other languages. It is, however, still difficult to let servers written in different languages communicate with each other. At Wink, we decided to go with ICE, and to develop the missing O'Caml implementation ourselves. </div> <div> <p>For two years now I have done consulting work for <a href="http://wink.com">Wink Technologies</a> who started out as a user-powered search engine, but switched later to people search. Actually, people search began as an experiment, and was originally developed in O'Caml with some Java standard components. It turned out that O'Caml was very well suited for crawling and parsing, and that our solution was convincing enough so we could go on with O'Caml as implementation language. It was also a big plus that it was already possible to develop server clusters in O'Caml with Ocamlnet's SunRPC implementation - we only had to add a highly available directory and configuration service. However, the problem of SunRPC is that there is no good C++ implementation (interestingly, there is an acceptable one for Java, RemoteTea). Too bad, since SunRPC is simple and robust, and its type system matches the one of O'Caml quite well (there are records, arrays, variants, and option types). </p><h2>ICE: An RPC Middleware</h2> <p>Looking for an alternative, somebody came up with ICE ("Internet Communications Engine"). This is a commercial product by <a href="http://zeroc.com">ZeroC</a> which is dual-licensed under the GPL (like e.g. MySQL). There are implementations for a number of languages, including C++, Java, Python, and PHP, but unfortunately not for O'Caml. Well, this is no surprise, but at least the other company languages are covered. So we looked closer at ICE. Does it match our needs? </p><p>ICE follows the object-oriented paradigm. For an RPC middleware this means that a remote call is seen as sending a message to a remote object. Of course, it is possible to have several such objects of the same type, and creating instances is made possible by a class construct. Such a design is very acceptable, but unfortunately object orientation often meant in the past that the rest of the type system was crippled. To some degree, this also happened to ICE - especially, there are no variants and no option types. Well, not optimal, but there are at least clean "design patterns" how to emulate these missing features with classes, and for pure OO languages like Java the ICE approach simplifies the language mapping. </p><p>Unlike CORBA, there is a fixed protocol the components have to use to talk to each other. That means a client in language X can directly contact a server in language Y, and there is no need for an intermediate instance to translate the protocol. Basically, this means you can use ICE without any infrastructure - no central server you are dependent upon. For developing massively parallel cluster services this is an essential requirement, because such central servers don't scale well enough, and are single points of failure. </p><p>For using ICE in a cluster context, there is the IceGrid add-on. Basically, this is a highly available directory and configuration service, and serves for a similar purpose like the service we had developed for SunRPC before. Clients ask IceGrid where to find their servers in the network, and IceGrid replies with a suggestion of TCP ports. This can be used for load-balancing and for high availability. </p><h2>Hydro: Implementing ICE for O'Caml</h2> <p>After ICE was found to be good enough, we needed an implementation for O'Caml. Well, this was my field - I already developed the SunRPC support in Ocamlnet years ago, and this made me an expert for this type of work. It took only about 3 weeks until it was possible to generate client code, and about another week until server support was ready. However, it was still challenging work, because the ICE type system needed to be mapped to O'Caml's type system. Furthermore, the ICE reference manual was full of errors, and everything had to be checked against ZeroC's implementation. </p><p>The difficulty of the type mapping is that ICE demands that objects and exceptions can be downcasted. O'Caml, however, does not support this operation, because there is no efficient implementation of downcasting for a type system like O'Caml's that includes structural subtyping. Nevertheless, downcasting is a reasonable operation in the context of RPC, and it is hardly possible to get around it. </p><p>Maybe an example demonstrates this best. In Slice, the IDL for ICE, one can easily define a hierarchy of classes (the syntax resembles Java's): </p><blockquote lang="x-slice"> <code><pre> class SearchResult { string url; string title; } class PeopleSearchResult extends SearchResult { string name; } class BandSearchResult extends SearchResult { string bandName; stringSeq bandMembers; } </pre></code></blockquote> <p>When the search engine returns a <code>SearchResult</code> item, it can also be one of the descendants of this class. Of course, a client of the search engine that simply wants to display the result, needs to know all details, and thus downcasts <code>SearchResult</code> to the real subclass. </p><p>In a normal OO program one can get rid of this downcast by adding an operation for displaying the result: </p><blockquote lang="x-slice"> <code><pre> class SearchResult { string url; string title; string display(); } </pre></code></blockquote> In an RPC context such an addition might be difficult, however, or may break some other principle of the RPC design. Basically, RPC is about marshalling data, and that means getting data out of the context of one server and forcing them into the context of another server. The "unity of data and operations", one of the OO principles, is intentionally given up. <p>Note that ICE allows to define operations for classes, and operations are always executed in the context of the data. In this example, it would be in deed possible to define <code>display</code> in a reasonable way, and to avoid the downcast. However, <code>display</code> then becomes part of the protocol, although it is rather a detail of the client. Anyway, one quickly faces the situation where downcasting is unavoidable. </p><p>In the O'Caml mapping generated by Hydro, these three classes would appear like </p><blockquote lang="x-ocaml"> <code><pre> class type o_SearchResult = object inherit o_Ice_Object method url : string ref method title : string ref end class type o_PeopleSearchResult = object inherit o_SearchResult method name : string ref end class type o_BandSearchResult = object inherit o_SearchResult method bandName : string ref method bandMembers : string array ref end val as_SearchResult : #Hydro_lm.object_base -> o_SearchResult val as_PeopleSearchResult : #Hydro_lm.object_base -> o_PeopleSearchResult val as_BandSearchResult : #Hydro_lm.object_base -> o_BandSearchResult </pre></code></blockquote> This is a bit simplified, but shows the idea. The ICE classes are mapped to O'Caml classes with some hidden machinery. The data members appear as O'Caml methods returning references - the most direct translation of this concept. The class hierarchy corresponds to the hierarchy in ICE, so the O'Caml operator for upcasting, :>, can be directly used. The hidden machinery comes into play by inheriting from <code>o_Ice_Object</code>, the root of the ICE hierarchy, and by using <code>object_base</code>, an even smaller antecedent that defines the marshalling core. <p>The downcast operation is emulated by defining conversion functions for every class type: <code>as_PeopleSearchResult</code> checks whether the argument is a <code>PeopleSearchResult</code> in reality, and if so, casts it to this class type. If not, an exception is raised. </p><p>Of course, this emulation is a bit inconvenient, but this is mostly a problem of generating good code. From a user's perspecitve, there is not much difference between calling a generated conversion function, or using a built-in language operation. It makes, however, the whole generated code a lot more difficult to understand. </p><h2>The Story Continues</h2> Missing support for RPC middleware is one of biggest concerns when using a new language in enterprises. In a startup company like Wink it is possible to address these concerns, because such companies are open for unconventional solutions (and ICE is unconventional - the industry standards are CORBA and DCOM in the LAN, and HTTP-based protocols like SOAP in the Internet). Finally, by integrating several languages into the system it was possible to deliver some components quicker and with better quality because we could choose the best language for every component. <p>In the people searcher context we use now both SunRPC and ICE. The former is arguably better when only O'Caml components have to talk with each other, and the latter is for crossing the language boundaries. </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> <p><b>Links:</b> <ul> <li><a href="http://wink.com">Wink Technologies</a>: Company homepage </li> <li><a href="http://www.zeroc.com">ZeroC</a>: Company homepage, ICE documentation, ICE implementation for a number of languages </li> <li><a href="http://remotetea.sourceforge.net">RemoteTea</a>: SunRPC for Java </li> <li><a href="http://oss.wink.com/hydro">Hydro</a>: ICE for O'Caml </li> <li><a href="http://projects.camlcity.org/projects/ocamlnet.html">Ocamlnet</a>: Internet library for O'Caml with SunRPC support </li> </ul> </p> </div> Mixing Apples And Pears http://blog.camlcity.org/blog/polyvariants.html http://blog.camlcity.org/blog/polyvariants.html <div> <b>Using Polymorphic Variants</b><br>  </div> <div> It is one of the coolest language constructs, but its conception leads sometimes to confusion. O'Caml allows it to form ad-hoc unions of tagged values, the so-called polymorphic variants. They are the free-style counterpart of the "normal" variant types. We want to shed some light on this construction in this article, and encourage programmers to try it out. </div> <div> <p> The most baffling property of the polyvariants is that one can mix tags that come from different pieces of code. We'll give an example of that later in the text, but first let's explain some foundations. Syntactically, the tags are written with a leading reverse apostrophe, so for example <code>`Apple</code> is a (value-less) tag. Like the normal variants the tags can have attached values, so for instance <code>`Pear "it's sweet man"</code> is a tag with a string value. It is not required to declare polyvariant types, one can simply start creating such tagged values in the code. </p><h2>Data Analysis Step-By-Step</h2> <p>Imagine we want to analyze a string. In a first step, we would like to classify every character of the string, and determine whether it is a letter, a digit, or something else. Using polyvariants, this function does the job: </p><blockquote lang="x-ocaml"> <code><pre> let classify_chars s = let rec classify_chars_at p = if p < String.length s then let c = s.[p] in let cls = match c with | '0'..'9' -> `Digit c | 'A'..'Z' | 'a'..'z' -> `Letter c | _ -> `Other c in cls :: classify_chars_at (p+1) else [] in classify_chars_at 0 </pre></code> </blockquote> <p>So this function would return this list for the input string "a56*": </p><blockquote lang="x-ocaml"> <code><pre>[ `Letter 'a'; `Digit '5'; `Digit '6'; `Other '*' ] </pre></code> </blockquote> <p>Note that there is no type declaration! If you enter this function into the O'Caml toploop, you see that its type is inferred like this: </p><blockquote lang="x-ocaml"> <code><pre> val classify_chars : string -> [> `Digit of char | `Letter of char | `Other of char ] list = <fun> </pre></code> </blockquote> <p>Read this as: The function returns tagged values with tags <code>`Digit</code>, <code>`Letter</code>, or <code>`Other</code>, and every tag has an attached character. Note the "greater than" sign at the beginning of the tag list. It means that this tag list is compatible with being mixed with completely unrelated tags. There are also polyvariant types where this sign is reversed (like in <code>[<...]</code>), or completely missing. We'll come back to that later. </p><p>Back to our string analysis example. We are now interested in recognizing integer numbers in the list of classified characters. We assume our input is a list that contains <code>`Digit</code> tags, but also other tags. In the output list, sequences of <code>`Digit</code> are replaced by <code>`Number</code>: </p><blockquote lang="x-ocaml"> <code><pre> let recognize_numbers l = let rec recognize_at m acc = match m with | `Digit d :: m' -> let d_v = Char.code d - Char.code '0' in let acc' = match acc with | Some v -> Some(10*v + d_v) | None -> Some d_v in recognize_at m' acc' | x :: m' -> ( match acc with | None -> x :: recognize_at m' None | Some v -> (`Number v) :: x :: recognize_at m' None ) | [] -> ( match acc with | None -> [] | Some v -> (`Number v) :: [] ) in recognize_at l None </pre></code> </blockquote> <p>The inferred type of this function is now really strange: </p><blockquote lang="x-ocaml"> <code><pre> val recognize_numbers : ([> `Digit of char | `Number of int ] as 'a) list -> 'a list = <fun> </fun></pre></code> </blockquote> <p>Basically, it says that there is a tagged list as input, and that the output list has the same tags. Furthermore, the ">" sign again signals extensibility, so we can not only use the tags mentioned in the function, but any other tag as well. Especially, we are free to pass <code>`Letter</code> and <code>`Other</code> tags in: </p><blockquote lang="x-ocaml"> <code><pre> recognize_numbers [ `Digit '1' ; `Digit '3'; `Letter 'a'; `Digit '2'; `Other '*' ] </pre></code> yields <code><pre> [ `Number 13; `Letter 'a'; `Number 2; `Other '*'] </pre></code> </blockquote> <p>Note that the type of the <code>recognize_numbers</code> function does not reflect all what we could know about the function. We can be sure that the function will never return a <code>`Digit</code> tag, but this is not expressed in the function type. We have run into one of the cases where the O'Caml type system is not powerful enough to find this out, or even to write this knowledge down. In practice, this is no real limitation - the types are usually a bit weaker than necessary, but it is unlikely that weaker types cause problems. </p><p>The really great thing about the polymorphic variants is that it is possible to mix tags that come from different contexts. So in our example we can combine the two functions, and apply one after the other: </p><blockquote lang="x-ocaml"> <code><pre> let analyze s = recognize_numbers (classify_chars s) </pre></code> </blockquote> <p>It is no problem that <code>classify_chars</code> emits tags that are completely unknown to <code>recognize_numbers</code>. And both functions can use the same tag, <code>`Digit</code>, without having to declare in some way that they are meaning the same. It is sufficient that the tag is the same, and that the attached value has the same type. </p><p>This may have big advantages for structuring programs. Of course, our example of string analysis already benefits from the loose type correspondence the polyvariants make possible. The problem can now be divided into several steps, and every step needs only to know the tags it operates on. There is no global type all steps have to agree upon, rather every step sees only the fraction of type information that is needed for the task. </p><h2>Limiting Tags</h2> <p>Compare these two functions: </p><blockquote lang="x-ocaml"> <code><pre> let number_value1 t = match t with | `Number n -> n | `Digit d -> Char.code d - Char.code '0' let number_value2 t = match t with | `Number n -> n | `Digit d -> Char.code d - Char.code '0' | _ -> failwith "This is not a number" </pre></code> </blockquote> <p>The difference is that the second version explicitly catches the case of "any other tag" whereas the first version leaves this unspecified. This leads to different typings: </p><blockquote lang="x-ocaml"> <code><pre> val number_value1 : [< `Digit of char | `Number of int ] -> int val number_value2 : [> `Digit of char | `Number of int ] -> int </pre></code> </blockquote> <p>So in the first version we have a "<" sign! This sign usually only appears for input arguments, and means that the function can only process these tags (or less tags), but no other tags. The type checker prevents that any other tag can be passed in: </p><blockquote lang="x-ocaml"> <code><pre> # number_value1 (`Letter 'a');; This expression has type [> `Letter of char ] but is here used with type [< `Digit of char | `Number of int ] </pre></code> </blockquote> <p>In the second version of the function, this case is handled at runtime. From a typing perspective, the ">" sign signals that the function can also accept other tags than the mentioned ones. However, this function actually raises an exception. </p><blockquote lang="x-ocaml"> <code><pre> # number_value2 (`Letter 'a');; Exception: Failure "This is not a number". </pre></code> </blockquote> <p>So the "<" sign is a way to limit the number of tags a function can process. The programmer gets it by not adding a "catch all" case to pattern matchings. This kind of polyvariant is useful to enforce some strictness in programming, and the type checker catches the cases that would otherwise only be handed at runtime. </p><h2>Giving Polyvariants Names</h2> <p> Although the mantra of this article is that we don't need declarations, it is of course possible to define named polymorphic variants. For example, we could introduce these named types: </p><blockquote lang="x-ocaml"> <code><pre> type classified_char = [ `Digit of char | `Letter of char | `Other of char ] type number_token = [ `Digit of char | `Number of int ] </pre></code> </blockquote> <p>Note that there is no ">" or "<" sign in such definitions - it would not make sense to say something about whether more or less tags are possible than given, because the context is missing. </p><p>Using these names, one could simplify the typings of our functions: </p><blockquote lang="x-ocaml"> <code><pre> val classify_chars : string -> [> classified_char ] val recognize_numbers : ( [> number_token ] as 'a) list -> 'a list val number_value1 : [< number_token ] -> int val number_value2 : [> number_token ] -> int </pre></code> </blockquote> <p>As you see, the ">" or "<" sign can still appear in function types using these names. Actually, there is a special syntax behind this notation. If you just say <code>number_token</code> in a type expression, exactly the definition applies (without sign). But you can also construct new polyvariants from existing ones. For example, </p><blockquote lang="x-ocaml"> <code><pre>[ classified_char | number_token ] </pre></code> </blockquote> <p>would mean a type that combines the tags of both types, i.e. this is the same as if all four tags were enumerated. The syntax <code>[<number_token ]</code> is only a special case of this type constructor, where the new type also gets a sign. </p><h2>Matching Variants</h2> <p>Try to compile this piece of code that ought to sum up all <code>`Digit</code> and <code>`Number</code> tags of a list of any tags: </p><blockquote lang="x-ocaml"> <code><pre> let rec sum l = match l with | x :: l' -> ( match x with | `Digit _ | `Number _ -> number_value1 x + sum l' | _ -> sum l' ) | [] -> 0 </pre></code> </blockquote> <p>It compiles, but there is a little surprise. The compiler emits a warning that the <code>_ -> sum l'</code> case of the matching is unused, and the inferred type is just </p><blockquote lang="x-ocaml"> <code><pre> val sum : [ `Digit of char | `Number of int ] list -> int </pre></code> </blockquote> i.e. the ">" sign is missing that would allow us to pass any tags in. What's wrong? <p>This is a pitfall one quickly runs into when using polyvariants. The type checker assumes that the <code>x</code> in <code>number_value1 x</code> has the same type as the <code>x</code> that is matched again. It is not sufficient to use a matched variable in the expression of the matched case to restrict its type. Fortunately, there is a special syntax for that (look at the bold "as" clause): </p><blockquote lang="x-ocaml"> <code><pre> let rec sum l = match l with | x :: l' -> ( match x with | (`Digit _ | `Number _) <b>as y</b> -> number_value1 <b>y</b> + sum l' | _ -> sum l' ) | [] -> 0 </pre></code> </blockquote> <p>Here, <code>y</code> is the value <code>x</code> for the case that the match applies, so <code>y</code> can have a stricter type than <code>x</code>. </p><p>Alternatively, the match condition could also have been written as: </p><blockquote lang="x-ocaml"> <code><pre> match x with | #number_token as y -> ... </pre></code> </blockquote> i.e. one can use named polyvariants in matchings. This is just a convenience notation for the former. <h2>Behind The Scene</h2> <p>The polyvariants might be cool, but many programmers suspect that the performance of their programs suffer when they use them. Well, although there are some runtime cost, these are very small, and not noticeable for many programs. </p><p>Internally, the tags are represented by hash values of the names of the tags. So the tags are simply reduced to integers at runtime. Compared with the normal variant types, there is some additional overhead for tags with values. In particular, for storing <code>`X value</code> one extra word is needed in comparison with the normal variant <code>X value</code>. </p><p>It is possible that the hash values of variants collide, e.g. </p><blockquote lang="x-ocaml"> <code><pre> # type x = [ `jagJhn | `oZshTt ];; Variant tags `jagJhn and `oZshTt have same hash value. Change one of them. </pre></code> </blockquote> As you see, the compiler checks for this rare case. I've never seen it in practice. <p>Despite rumours, there is nothing special done at link time. The tags are already cut down to integers at this point of compiling. </p><h2>Conclusion</h2> <p>I hope I've shown how elegant code looks that uses polyvariants to represent data cases. But there is some more to say, especially if you look at other programming languages. </p><p>The author of this article thinks that polymorphic variants are one of the features that makes O'Caml so different in comparison with mainstream languages like Java. In particular, there are competing approaches for representing data cases, and one of the radical ideas of object orientation has always been that combining data and program cases into a single class construct is the best way to deal with the problem. However, I think something has been overlooked - data and algorithms do not always walk hand in hand, and classes are inflexible if only a loose correlation between both is needed. In contrast, polyvariants give the programmer the maximum of freedom in this respect. Thus it is believed that polyvariants are a serious alternative for representing data cases. </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> <p><b>Links:</b> <ul> <li><a href="http://caml.inria.fr/pub/docs/manual-ocaml/manual006.html#htoc41">Polymorphic variants</a>: Chapter in the O'Caml manual </li> </ul> </p> </div>