Blog on camlcity.org

Blog on camlcity.org http://blog.camlcity.org en Articles by Gerd Stolpmann about O'Caml Ocamlnet 3 finally released http://blog.camlcity.org/blog/ocamlnet3_release.html http://blog.camlcity.org/blog/ocamlnet3_release.html <div> <b>What's new in Ocamlnet 3</b><br/>  </div> <div> So, finally it is there: <a href="http://projects.camlcity.org/projects/ocamlnet.html">Ocamlnet 3.0.0</a>. After almost 3 years of development, many parts of Ocamlnet have been touched and extended while keeping most of the existing APIs. It is not immediately visible what the striking new features are, so a bit of explanation is necessary. </div> <div> <p> When renovating a building, it is common to do this floor by floor. In this sense, Ocamlnet 3.0.0 focused on the foundation and the first floor. Also, the renovation is not yet finished - many features still need to be added, like supporting SSL for more protocols. This is now easier thanks to some new basic APIs that have been introduced in the first step. </p><h2>Netsys</h2> <p> One of the parts that got most attention is <code>Netsys</code>, the library adding the missing links to the operating system (OS). One of the driving forces was the port to Win32. This lead to the introduction of generalized versions of <code>Unix.read</code> and <code>Unix.write</code> calls (defined in <code>Netsys</code>): </p><pre> val gread : fd_style -> Unix.file_descr -> string -> int -> int -> int val gwrite : fd_style -> Unix.file_descr -> string -> int -> int -> int </pre> For getting some Win32-specific emulations right, it is sometimes required to call other functions instead of <code>Unix.read</code> and <code>Unix.write</code>, e.g. <code>Netsys_win32.pipe_read</code> and <code>Netsys_win32.pipe_write</code>. In order to avoid that such case distinctions are scattered over the whole library, the idea of defining these generic functions was born. In <code>fd_style</code> the user passes in how to handle the descriptor. Usually <code>fd_style</code> is automatically determined by another function <code>get_fd_style</code> (this requires a few system calls and is factored out because of this). Although targeting mostly at Win32, there are already some benefits for POSIX systems, e.g. the <code>fd_style</code> already encodes whether a descriptor is a socket, and whether it is connected, which is sometimes quite useful information. In the future, this system will be extended: <ul> <li><p>Seekable files are currently not well supported by the asynchronous I/O layer. The reason is that the <code>select</code> and <code>poll</code> system calls cannot predict whether I/O would be blocking or non-blocking (and thus always say non-blocking). This can be improved by using special AIO calls of the OS. Of course, files for which AIO is to be used need to be flagged specially, and a new <code>fd_style</code> could do so. </p></li><li><p>There are also some ideas for labeling SSL sockets by a special <code>fd_style</code>. This would make it a bit easier to support SSL thoughout the library. This is a bit more work than just calling <code>Ssl.read</code> and <code>Ssl.write</code>, though, because the SSL protocol allows renegotiations at any time, and a read may also require writes on the socket level, and vice versa. </p></li></ul> Another new idea on the <code>Netsys</code> level is a little object definition called <code>pollset</code>: <pre> class type pollset = object method find : Unix.file_descr -> Netsys_posix.poll_req_events method add : Unix.file_descr -> Netsys_posix.poll_req_events -> unit method remove : Unix.file_descr -> unit method wait : float -> ( Unix.file_descr * Netsys_posix.poll_req_events * Netsys_posix.poll_act_events ) list method dispose : unit -> unit method cancel_wait : bool -> unit end </pre> A <code>pollset</code> represents a set of file descriptor events one wants to poll. Again, this data structure was originally required for the Win32 port (because Win32 is very different in this respect), but there are also advantages for Unix systems. Nowadays, there are various improved APIs for polling such as Linux epoll or BSD kqueue. The <code>pollset</code> abstraction will make it very easy to support these - the user simply selects one of the advanced implementations of <code>pollset</code>, and thanks to dynamic binding of object methods it is automatically used everywhere. (One of the next versions of Ocamlnet will allow this.) <p>Another word about polling. The Ocaml runtime only provides <code>select</code>. Although not as bad as claimed by some people, it imposes artificial limitations, especially about the number of supported file descriptors. Because of this, <code>Netsys_posix</code> includes now a binding of the <code>poll</code> system call which is not suffering from this disease. Of course, <code>poll</code> is now the only poll API used throughout Ocamlnet (and, as noted, even better APIs will be supported in one of the next releases). </p><p>Other additions on the OS level for Unix systems: </p><ul> <li><code>Netsys_posix.spawn</code> is a new way of starting subprograms, with special support for monitoring the subprocesses asynchronously </li><li>There are now bindings for syslog in <code>Netsys_posix</code> </li><li>The system calls <code>fsync</code> and <code>fdatasync</code> are supported </li><li>If the OS provides this call, <code>fadvise</code> can be invoked to control the page cache </li><li>There is also <code>fallocate</code> to allocate disk space, so far the OS provides it </li><li>POSIX semaphores are supported, so far the OS provides the complete interface (i.e. named semaphores for synchronization between unrelated processes) </li><li>There is a coordinator module for signals, <code>Netsys_signal</code>, so that various users of signals do not mutually override their handlers </li></ul> <p> For all systems, <code>Netsys</code> implements: </p><ul> <li>Wrappers for multicasting system calls on sockets </li><li>In <code>Netsys_mem</code> there is now special support for using bigarrays of chars as efficient I/O buffers. Such bigarray-backed buffers are called <code>memory</code> (reminding us to the fact that these buffers are not relocatable like strings, but bound to fixed memory addresses). There are functions for allocating page-aligned or cache-line-aligned <code>memory</code> buffers. Also, there is experimental support for copying Ocaml values into buffers (used by the Camlbox module, see below). Finally, there are also versions of <code>read</code>, <code>write</code>, <code>recv</code> and <code>send</code> operating on memory buffers rather than strings. These versions open the door to zero-copy network I/O (if supported by the OS). </li><li>For better support of multi-threading there is now a version of the thread API that even exists when the thread library is not linked in, so that especially critical sections are emulated as no-ops in the single-threaded case. It is hoped that more functions can be made thread-safe by this new feature (in <code>Netsys_oothr</code>). </li><li>The exception registry <code>Netexn</code> is now almost outdated, because the Ocaml standard library recently introduced a similar feature (yes, sometimes feature wishes are honoured :-). </li></ul> <h2>Equeue</h2> <p> As <code>Netsys</code> uses now pollsets to manage polling, <code>Equeue</code> had to be rewritten to take advantage of this. In particular, there is now <code>Unixqueue_pollset</code> which is a port of the old <code>Unixqueue</code> API around pollsets. For the user, there is absolutely no difference. </p><p> What's more important is the extension of the engine API. Ocamlnet 2 introduced engines as a way of expressing a suspended I/O possibility, but there was only limited support for it in the library. This has now changed - engines are now a first class member of Ocamlnet. In particular, there are now much more synchronization primitives (e.g. <code>stream_seq_engine</code> for executing an open number of engines in sequence, or <code>msync_engine</code> for waiting for the completion of multiple engines). This development was mostly driven by another project of mine: Plasma (see other blog articles on this site). Plasma uses engines for all kinds of concurrent execution of I/O code, and while I was developing Plasma, I extended the Ocamlnet engine API step by step. </p><p> There is also now a way to call RPC procedures with an engine: <code>Rpc_proxy.ManagedClient.rpc_engine</code>. This function has originally also been developed for the Plasma project. </p><p> For simpler I/O needs, I added <code>Uq_io</code>. It contains "engineered" versions of simple I/O functions like <code>input</code>, <code>input_line</code> or <code>flush</code>. <code>Uq_io</code> is not limited to file descriptors, but works also on top of a number of other I/O devices (including virtual ones). </p><p> The operators <code>++</code> and <code>>></code> have been introduced as abbreviations for sequential execution, and result mapping of engines, respectively. For example, the synchronous code </p><pre> let line1 = input_line ch_in in let line2 = input_line ch_in in output_string ch_out (line1 ^ line2 ^ "\n") </pre> would now look in "engineered" code: <pre> Uq_io.input_line_e d_in ++ (fun line1 -> Uq_io.input_line_e d_in ++ (fun line2 -> Uq_io.output_string_e d_out (line1 ^ line2 ^ "\n") ) ) </pre> Not bad, if you compare with the previous solution (hand-coding a scanner for lines, writing the event handler routines, etc., adding up to 100-200 lines of code). <h2>Netplex</h2> The development in the <code>Netplex</code> area was focused on easing multi-processing. With <code>Netplex</code> it is very easy to run code in several worker processes, e.g. for network servers. What was missing up to now, however, was an easy way to manage the collaboration of the processes. <p> <code>Netplex</code> worker processes got now a number of ways to talk to each other: </p><ul> <li>It is now possible to store variables in a common place, so that each process can get and set these (<code>Netplex_sharedvar</code>). Of course, this mechanism is typed. </li><li>There are mutexes and semaphores for synchronization (<code>Netplex_mutex</code> and <code>Netplex_semaphore</code>) </li><li>Each process can be directly contacted via a private channel, the so-called container socket. This is also an RPC mechanism, but unlike normal RPC servers the caller directly addresses a process (and not only a service in general, and the Netplex machinery automatically selects the destination process). There is also a directory so that processes can see which other processes exist. </li></ul> <p> The implementation of these mechanisms is not yet optimal, but the APIs are defined and backed by simple but robust modules. It is expected that in the future more sophisticated implementations will become available, e.g. the <code>Netplex_sharedvar</code> code use a shared memory object if the OS supports that. </p><p> Another addition are "levers". This kind of handle exists within the Netplex master process, but can be activated from the child processes. It is a kind of little RPC function for a special purpose: Sometimes the process model requires that certain functionality must be done within the scope of the master process. An example would be the start of another child process. By doing that via a lever, this action can also be triggered from any child process. </p><p> Besides that there are numerous smaller enhancements. Especially the module <code>Netplex_cenv</code> has been extended, e.g. there are now timers that can be attached to the Netplex event queue. </p><h2>RPC</h2> The development went into two directions: First, it was aimed at a more powerful RPC client implementation, and second, performance performance performance. <p> The improved client is called <code>Rpc_proxy</code>. All experience went in that I made at my Ocaml job at Mylife.com - lots of RPC calls in an unreliable environment (if you have hundreds of machines, one box is always down). Clients can now be recycled, they can react better on errors, and even load balancing and fail-over to alternate endpoints are now supported. (See the other blog posting, "The next server, please!".) </p><p> Performance improvements were achieved by two means: First, the XDR encoding and decoding was optimized. This has not yet come to an end yet, but certain XDR types like arrays of strings are now processed a lot faster. The other strategy was to replace many string buffers by bigarrays of char (see under "memory" above). This allows it to get rid of a number of copy operations, especially when large strings are transmitted via RPC. This new string representation is even accessible by user code via a new XDR type <code>_managed string</code>. This may avoid even more copies. </p><h2>Shell</h2> The API of <code>Shell</code> is mostly the same - only a few suspicious functions have been removed. The implementation, however, has changed a lot. <p> <code>Shell</code> now uses the new <code>Netsys</code> functions for starting subprocesses. As these functions are written in C, one gets some immediate benefits: <code>Shell</code> is now officially supported for multi-threaded programs because it is possible to do the signal handling right in C (but still, this is notoriously difficult). Also, there is now no risk anymore that the Ocaml garbage collector wants to clean up in the worst moment, namely between fork and exec. </p><p> Another benefit is that <code>Shell</code> works now also under Win32. The C part is completely different, though. </p><h2>Netcgi</h2> Not much has changed here, only that the old version <code>Netcgi1</code> is gone now. <h2>Camlboxes</h2> An exciting but still experimental addition are Camlboxes. They are designed as a fast way of sending messages between unrelated processes. Camlboxes use shared memory for communication. <p> This works as follows: If process 1 want to send process 2 a message, both have to map the same memory pages into their address space. The message is orignally an Ocaml value somewhere in the private memory of process 1. With the help of Camlbox this value is now copied to shared memory so that, and this is the pivotal point, process 2 can directly access the value without additional decoding step. This reduces greatly the overhead of message sending - actually only a relatively fast value copy is done, bypassing any kernel-controlled I/O devices. </p><p> For passing a short message, this takes now only a few microseconds. Most of that time is spent for synchronization, of course, not for copying. (On the hardware level, the synchronization is mostly done by moving cache lines from one CPU core to the other, so this is some kind of hidden copying. It is worth noting that Camlboxes are way faster on single-core machines than on multi-cores because this low-level synchronization is not required then.) </p><p> Camlboxes have one downside, though. They are not perfectly integrated into the garbage collecting machinery, and because of this, one has to follow some programming rules. In particular, there is no way to recognize that a message (or part of it) is no longer referenced, so messages are manually deleted, and there is of course the danger that bad code keeps references to (or into) deleted messages. For fixing this, we would need more help by the Ocaml GC. </p><p> Another problem is missing integration with <code>Equeue</code>. Camlboxes are synchronous by design - that's the price for their speed. </p><h2>Where to get Ocamlnet 3</h2> Look at the <a href="http://projects.camlcity.org/projects/ocamlnet.html"> project page</a> for the newest version and links to the manual, mailing list, etc. <br/> <br/> <img src="/files/img/blog/ocamlnet3_release_bug.gif" width="1" height="1"/> </div> <div> Gerd Stolpmann works as O'Caml consultant. <a href="http://www.gerd-stolpmann.de/buero/work_ocaml_search.html.en"> He is accepting new customers!</a> </div> <div> </div> Plasma build simplified http://blog.camlcity.org/blog/plasma2.html http://blog.camlcity.org/blog/plasma2.html <div> <b>Plasma MapReduce and PlasmaFS</b><br/>  </div> <div> There is now a script that simplifies the Plasma build: <a href="http://download.camlcity.org/download/plasma_install.sh"> plasma_install.sh</a>. It just bootstraps GODI and puts a complete file tree into /opt/plasma - including ocaml and all required libraries. </div> <div> <p> The script has even built-in knowledge about certain Linux distributions, and installs missing system packages. Nobody should stumple over libpq-dev. It can also configure PostgreSQL to some extent. </p><p> I hope this script is especially for non-ocamlers a big help, because one cannot expect knowledge about how ocaml software is usually installed. And Plasma is still missing in distros. </p><p>More information about Plasma can be found on the <a href="http://projects.camlcity.org/projects/plasma.html">project page</a>. <img src="/files/img/blog/plasma2_bug.gif" width="1" height="1"/> </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> </div> Plasma: Map/Reduce for Ocaml http://blog.camlcity.org/blog/plasma1.html http://blog.camlcity.org/blog/plasma1.html <div> <b>Plasma MapReduce and PlasmaFS</b><br/>  </div> <div> I'm very proud to announce the public availability of Plasma MapReduce, a map/reduce compute framework, and PlasmaFS, the underlying distributed filesystem. All of this is written in Ocaml and makes it now possible to develop map/reduce programs in a functional programming language. </div> <div> <p> Plasma MapReduce is a distributed implementation of the map/reduce algorithm scheme. In a sentence, map/reduce performs a parallel List.map on an input file, sorts and splits the output by some criterion into partitions, and runs a List.fold_left on each partition. Only that it does not do that sequentially, but in a distributed way, and chunk by chunk. Because of this Plasma MapReduce can process very large files, and if run on enough computers, this also will work in reasonable time. Of course, map and reduce are Ocaml functions here. </p><p> This all works on top of a distributed filesystem, PlasmaFS. This is a user-space filesystem that is primarily accessed over RPC (but it is also mountable as NFS volume). Actually, most of the effort went here. PlasmaFS focuses on reliability and speed for big blocksizes. To get this, it implements ACID transactions, replicates data and metadata with two-phase commit, uses a shared memory data channel if possible, and monitors itself. Unlike other filesystems for map/reduce, PlasmaFS implements the complete set of usual file operations, including random reads and writes. It can also be used as unspecialized global filesystem. </p><p>Both pieces of software are bundled together in one download. Here is the <a href="http://projects.camlcity.org/projects/plasma.html">project page</a>. </p><p> This is an early alpha release. A lot of things work already, and you can already run map/reduce jobs. However, it is in no way complete. <img src="/files/img/blog/plasma1_bug.gif" width="1" height="1"/> </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> </div> Cluster Computing at Mylife.com http://blog.camlcity.org/blog/omeeting2010.html http://blog.camlcity.org/blog/omeeting2010.html <div> <b>Slides of the talk in Paris</b><br/>  </div> <div> For anybody who is interested, here are the slides of the talk I gave in Paris yesterday. </div> <div> <a href="/download/omeeting2010.pdf">Cluster Computing at Mylife.com</a> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> </div> Ocamlnet-3.0test2 http://blog.camlcity.org/blog/ocamlnet3_test2.html http://blog.camlcity.org/blog/ocamlnet3_test2.html <div> <b>Second testing version</b><br/>  </div> <div> A second testing version of Ocamlnet 3 has been released: <a href="http://download.camlcity.org/download/ocamlnet-3.0test2.tar.gz">Ocamlnet-3.0test2</a>. </div> <div> <p> This version mainly includes a large number of bug fixes compared to the first version, but also a few additions: </p><ul> <li>netcamlbox: a fast ipc mechanism for sending ocaml values to another process. Netcamlbox is shared-memory based, and works well on multi-cores (see <a href="http://projects.camlcity.org/projects/dl/ocamlnet-3.0test2/doc/html-main/Netcamlbox.html">Netcamlbox.html</a> for doc) </li><li>netplex adds per-process sockets, so one can send messages to individual containers, and not only to the whole service </li><li>wrappers for POSIX semaphores </li><li>wrappers for syslog </li><li>performance optimizations (serialization, page-aligned I/O) </li><li>updated documentation </li></ul> <p> Already in the first test version: </p><ul> <li>Port to Win32 (as outlined in <a href="http://blog.camlcity.org/blog/ocamlnet3_win32.html">Stranger an a strange land</a>) </li><li>The new <code>Rpc_proxy</code> layer (as described in <a href="http://blog.camlcity.org/blog/ocamlnet3_ha.html">The next server, please!</a>) </li><li>Extensions of Netplex (see especially <a href="http://blog.camlcity.org/blog/ocamlnet3_mp.html">Mastering Multi-processing</a>) </li><li>New implementation of the Shell library for starting subprocesses </li><li>Uniform debugging with <code>Netlog.Debug</code> </li><li>Exception printers (<code>Netexn</code>) </li><li>Introduction of pollsets (<code>Netsys_pollset</code>); removal of <code>Unix.select</code> (i.e. more than 1024 file descriptors) </li><li>The <code>netcgi1</code> library has been dropped in favor of <code>netcgi2</code> </li></ul> <p> I've quickly checked that the library builds on linux, freebsd-7.2, open solaris, and Win32 (MinGW). Nevertheless, testers are especially encouraged to check whether Ocamlnet 3 still works on all platforms, because a lot of new platform-specific code has been added. </p><p>Download etc: </p><ul> <li><a href="http://projects.camlcity.org/projects/ocamlnet.html">Homepage</a> </li><li><a href="http://download.camlcity.org/download/ocamlnet-3.0test2.tar.gz">Source</a> </li><li><a href="http://projects.camlcity.org/projects/dl/ocamlnet-3.0test2/doc/html-main/index.html">Manual</a> </li><li><a href="https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/TODO">My "scratch pad" describing changes, plans, etc </a></li><li><a href="https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/">Subversion</a> </li></ul> <p> There is a GODI package, but you have to enable a special repository to get it: Add </p><blockquote> GODI_BUILD_SITES+=http://www.ocaml-programming.de/godi-build/ocamlnet3/ </blockquote> to godi.conf to see the new packages in godi_console. This works first after the bootstrap is finished (godi_console cannot be built with ocamlnet3 yet). Keep in mind that this is development code, and there is no easy way to downgrade to ocamlnet2. Best is you do this only for new GODI installations. <p>Special thanks to everybody who helped me to produce this new version - by reporting bugs, or even sending fixes, or by maintaining subtrees (Christophe Troestler). </p><p> More blog postings will follow describing the highlights. </p><p> Please report results to gerd@gerd-stolpmann.de </p> <img src="/files/img/blog/ocamlnet3_test2_bug.gif" width="1" height="1"/> </div> <div> Gerd Stolpmann works as O'Caml consultant. <a href="http://www.gerd-stolpmann.de/buero/work_ocaml_search.html.en"> He is accepting new customers!</a> </div> <div> </div> Mastering Multi-processing http://blog.camlcity.org/blog/ocamlnet3_mp.html http://blog.camlcity.org/blog/ocamlnet3_mp.html <div> <b>What's new in Ocamlnet 3: Synchronization primitives in Netplex</b><br/>  </div> <div> Ocamlnet is being renovated, and there is already a first testing version of Ocamlnet 3. The author, Gerd Stolpmann, explains in a series of articles what is new, and why Ocamlnet is the best networking platform ever seen. When it comes to parallelism, many people react skeptical on multi-processing - mostly because this is not the way of concurrent programming the "big players" prefer. However, with the help of a good multi-processing framework like Netplex, this style can be easily mastered, and delivers stable results in a more quicker way than other styles such as multi-threading. Ocamlnet 3 adds important synchronization primitives like semaphores. </div> <div> <p> Netplex is a framework for creating network services. In this model, the most important role of processes is to serve as containers for protocol interpreters that accept and deal with network connections ("worker processes"). In the traditional multi-processing world (like the <code>inetd</code> master server in Unix) this is taken to the extent that each connection is served by its own process. Netplex, fortunately, relaxes this 1:1 relationship - a process can handle more than one connection, either serially or even concurrently. Netplex was introduced in Ocamlnet 2, and got a lot of attention since then. What remained unanswered, however, is the question how to deal with tasks that are not immediately network-driven, but nevertheless have to run in parallel with the existing processes. In this article, I present a way to control a global timer for a network service, so one can run a certain task periodically - as an example how to use the new synchronization primitives. </p> <p> Before going into detail, let me clarify on one important thing. These synchronization primitives are not designed to be fast. They are designed to be always available, and to behave well even under abnormal operating circumstances (like crashing processes). Usually, the operating system provides better implementations, but often these implementations require certain arrangements (like knowing global names, or certain process relationships). For many uses, it is more important to have an immediately available mechanism than to have the fastest one. Also, such a basic mechanism can be helpful for negotiating faster ones between processes. </p><h2>The example: Competing for running the timer</h2> <p> In the Netplex process model one of the processes has a special role: The master process is the first process, and the one that serves as parent of all the other processes. The master process is not involved in CPU-intensive tasks, and is usually very responsive to incoming requests. Because of this, it is the ideal instance to perform helper tasks. Ocamlnet 3 adds the concept of plugins to the controller object that runs in the master process. Plugins are little RPC servers that can be attached to the main server the controller object provides. The synchronization primitives are implemented as such plugins. Before using a plugin, it needs to be added to the controller <code>ctrl</code>, such as in </p><pre> ctrl # add_plugin Netplex_semaphore.plugin </pre> <p> This is best done early in the program. Fortunately, a good moment for doing so is provided as a service hook by Netplex. All these hooks are bundled together as a hook object, and this object configures some aspects of a Netplex network service: </p><pre> class type processor_hook = object method post_add_hook : socket_service -> controller -> unit method post_rm_hook : socket_service -> controller -> unit method pre_start_hook : socket_service -> controller -> container_id -> unit method post_start_hook : container -> unit method pre_finish_hook : container -> unit method post_finish_hook : socket_service -> controller -> container_id -> unit method receive_message : container -> string -> string array -> unit method receive_admin_message : container -> string -> string array -> unit method system_shutdown : unit -> unit method shutdown : unit -> unit method global_exception_handler : exn -> bool end </pre> <p> Such a hook object can always be passed as an additional parameter to the protocol interpreter implementation (no matter which: http, the various CGI variants, Sun RPC, ICE as provided by Hydro). The <code>add_plugin</code> call is best done in the <code>post_add_hook</code> hook which is executed once after the socket service is configured in the controller. We will also use other hooks for our timer - even more radical, we will only use hooks to attach the timer to the network service. This means that we don't assume much about the service itself, and because of this the solution is quite generic. </p><p> A semaphore is a resource counter. The counter can be atomically increased and decreased, but can never fall below zero. An attempt to do so either fails immediately, or the execution of the process blocks until the counter is positive again. The idea behind that can be illustrated by a queue supplying a number of processes with tasks. The counter reflects the length of the queue. When a process needs another task, it checks the counter, and tries to decrease it. Either this works immediately, which means that the process gets the task, and has the right to fetch it from the queue. Or the queue is empty, and the process has to wait until a new task is again posted to the queue. Of course, the latter is the interesting case, because it can happen that several processes want to get new tasks from the empty queue. The semaphore then works as an arbitration mechanism, as it selects which process will be considered first. </p><p> In our example, the resource is the timer, or better the possibility of running the timer. We have a number of processes, but the timer should only run in one of them ("it is owned by one process only"). The counter value 1 means that the timer is not (!) running (i.e. 1 = the timer can be started once more), and 0 means that the timer is already running (0 = the timer cannot be started for an additional time). </p><p> This function tries to get the ownership of the timer, and returns true if that could be achieved: </p><pre> let sem_name = "some_name" let own_timer = ref false let acquire_timer() = (* precondition: the semaphore value can be 1 or 0 *) !own_timer || ( let v = Netplex_semaphore.decrement sem_name in (* By convention, v=-1 means that the value was already 0. It is not further decreased, however. v=0 means that it was decreased from 1 to 0, and we own the semaphore now. *) own_timer := (v = 0); !own_timer ) </pre> <p> Note that we don't pass <code>~wait:true</code> to the <code>decrement</code> call. This causes the operation to return -1 when the counter is already 0 rather than to wait for becoming positive. Also, semaphores have Netplex-wide global names. Instead of "some_name" one should pass something more intelligent, e.g. a name derived from the service name ("service.semaphore0"). </p><p> Although semaphores are automatically created at the time of the first use, they are often initialized wrong (counter value of 0). In our example, the initial value must be 1 - meaning that the timer is unacquired: </p><pre> let create_sem() = let success = Netplex_semaphore.create ~protected:true sem_name 1L in () </pre> <p>This function creates the semaphore if it does not exist, and sets the initial value to 1. Also, it is a so-called protected semaphore: If the process terminates in an abnormal way, Netplex ensures that the increment and decrement operations called by the process are reverted, i.e. if the process happens to own the timer, and the counter is 0, it will be automatically set back to 1, so another process has the chance to take over the ownership. </p><p> When all processes call <code>create_sem</code> when they start up it can be ensured that the semaphore exists. </p><p> The timer itself is started like this: </p><pre> let timer_running = ref None let start_timer() = timer_running := Some(Netplex_cenv.create_timer (fun timer -> ...; true) 60.0 ) </pre> <p>The function body is run every 60 seconds (the result value of true means that the timer is restarted after each timeout). Such timers are entered into the main event loop of the Netplex process container. This means it is not guaranteed that they are run as often as demanded - if some regular computation takes too long and the event loop cannot run then the timer activation will be deferred. For many timers, especially when runnning only infrequently, this way of activation is reliable enough. Timers are automatically stopped at shutdown time. We can, however, also enfore an earlier stop: </p><pre> let stop_timer() = match !timer_running with | None -> () | Some timer -> Netplex_cenv.cancel_timer timer; timer_running := None </pre> <p> Finally, there is the case that the process terminates and has to give up the ownership of the timer. This is easily done by incrementing the counter. However, we have to do more in this example, because the decrement operation does not wait: We have to notify other processes so that they again try to acquire the now free semaphore. Netplex has a simple built-in messaging system that we can use here: </p><pre> let release_timer() = if !own_timer then ( let v = Netplex_semaphore.increment sem_name in let cont = Netplex_cenv.self_cont() in cont # send_message "*" "release_semaphore" [| sem_name |] ) </pre> <p> (The destination address of "*" means that the message is broadcasted to all receivers. This could be optimized by sending only to the processes of the same service.) </p><p> Now, let's put everything together: When the Netplex system starts up, we add the plugin. When a process starts, we first try to create the semaphore, and then try to acquire it. If this is successful, we start the timer. For getting the shutdown right, we stop the timer and release the ownership before the process exits. Also, we listen on messages, and if the right one arrives, we look whether the semaphore is orphaned: </p><pre> class our_hooks() = object(self) inherit Netplex_kit.empty_processor_hooks() method post_add_hook _ ctrl = ctrl # add_plugin Netplex_semaphore.plugin method post_start_hook cont = create_sem(); if acquire_timer() then start_timer() method pre_finish_hook cont = stop_timer(); release_timer() method receive_message cont msg_name msg_args = if msg_name = "release_sempahore" && msg_args = [| sem_name |] then ( if acquire_timer() then start_timer() ) end </pre> <p> There is still a possible improvement. As pointed out, a protected semaphore automatically adjusts the counter value when the process crashes. But this is only the first action of <code>release_timer</code>. The second is to notify the other processes of the vacant timer ownership. This could be done from the master process (which is complicated), or by simply starting another timer that continuously tries to get the semaphore: </p><pre> method post_start_hook cont = create_sem(); if acquire_timer() then start_timer(); ignore(Netplex_cenv.create_timer (fun _ -> if acquire_timer() then start_timer(); true ) 60.0) </pre> <h2>Other IPC primitives</h2> So far, Netplex does not only have semaphores and message broadcasting as primitives, but also: <ul> <li>Shared variables: The plugin <code>Netplex_sharedvar</code> allows the processes to access variables in the master process. This is a very limited mechanism, though: Accesses are quite expensive, and the variables should not become big (because they would have to be copied at <code>fork</code> time). Nevertheless, this is a useful mechanism to spread dynamic information in a Netplex process system. </li><li>Mutexes: The plugin <code>Netplex_mutex</code> can be used to protect critical code sections (e.g. when accessing shared variables). Again, this mechanism is slow. </li><li>Accessing Unix Domain sockets: It is possible to look up the address of Unix Domain sockets that are served by other services in the same Netplex system: the <code>lookup</code> method of containers. This can be used to establish a fast IPC channel between processes. </li></ul> <p>This list is going to be extended (e.g. message queues are on my list). Also, it is tried to speed up the mechanisms when Netplex finds out that the OS has native support for these primitives (e.g. use POSIX semaphores for implementing the <code>Netplex_semaphore</code> plugin if available). </p><h2>Why using processes?</h2> <p>Some final words on why multi-processing is an interesting option for implementing network servers. This is important to explain because there are a number of myths about multi-processing (e.g. it is slow, hard to master, not that flexible), and especially non-tech people see it as a technology of the past. There are also big companies who are interested in spreading these myths, e.g. Microsoft, because their OS is a bad choice when it comes to multi-processing, or Sun, because Java is heavily optimized for multi-threading and JIT is not really compatible with multi-processing. The truth is that the big players simply did not invest in this technology. Actually, there are a number of advantages, and some make multi-processing superior: </p><p> First, there is the <b>stability</b> of the resulting programs. Even if a process crashes, the remaining processes of the system can continue to run. Netplex is designed with crash resilience in mind (crashed processes are restarted). </p><p> Second, multi-processing systems can more easily benefit from <b>multicores</b> because shared memory only plays a subordinate role for them which leads to more effective caching of RAM. Multi-processing systems often use uni- and bidirectional messaging instead. For programs written in Ocaml this is even more the case (but not for other languages) because Ocaml does not allow multi-threaded programs to profit from multicores at all. </p><p> Third, multi-processing systems can be more easily extended to <b>clusters</b> (i.e. they are running on several computers). The messaging mechanisms can often be easily implemented as network protocols. </p><p> Finally, many synchronization primitives are also available for the multi-processing case (as demonstrated). More or less, it is only shared memory which is really harder to master - e.g. Ocaml does not provide any form of memory management for explicitly allocated shared memory, so one could put Ocaml values directly into such memory blocks for easy and quick access. (However, I'm working on partial solutions for this.) </p><h2>Where to get Ocamlnet 3</h2> <p>There is no final release yet. The current testing version is <a href="http://download.camlcity.org/download/ocamlnet-3.0test1.tar.gz">Ocamlnet-3.0test1</a>. Look at <a href="http://blog.camlcity.org/blog/ocamlnet3_test1.html">this article for a list of changes</a>. </p><p>Alternately, one can also check out <a href="https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/">the Subversion repository</a> (use the <code>svn</code> command on this URL, or click on it to view it with your web browser - most of the discussed code lives in <code>src/netplex</code>). </p> <img src="/files/img/blog/ocamlnet3_mp_bug.gif" width="1" height="1"/> </div> <div> Gerd Stolpmann works as O'Caml consultant. <a href="http://www.gerd-stolpmann.de/buero/work_ocaml_search.html.en"> He is accepting new customers!</a> </div> <div> </div> Ocamlnet-3.0test1 http://blog.camlcity.org/blog/ocamlnet3_test1.html http://blog.camlcity.org/blog/ocamlnet3_test1.html <div> <b>First testing version</b><br/>  </div> <div> A first testing version of Ocamlnet 3 has been released: <a href="http://download.camlcity.org/download/ocamlnet-3.0test1.tar.gz">Ocamlnet-3.0test1</a>. </div> <div> <p> The idea of this release is to make this version available to a larger audience for testing, and to allow everybody to check whether code using this library still works. It is not yet ready for production environments. </p><p> List of major changes: </p><ul> <li>Port to Win32 (as outlined in <a href="http://blog.camlcity.org/blog/ocamlnet3_win32.html">Stranger an a strange land</a>) </li><li>The new <code>Rpc_proxy</code> layer (as described in <a href="http://blog.camlcity.org/blog/ocamlnet3_ha.html">The next server, please!</a>) </li><li>Extensions of Netplex </li><li>New implementation of the Shell library for starting subprocesses </li><li>Uniform debugging with <code>Netlog.Debug</code> </li><li>Exception printers (<code>Netexn</code>) </li><li>Introduction of pollsets (<code>Netsys_pollset</code>); removal of <code>Unix.select</code> (i.e. more than 1024 file descriptors) </li><li>The <code>netcgi1</code> library has been dropped in favor of <code>netcgi2</code> </li></ul> <p> There are also a lot of minor changes. Some of the changes are incompatible with code written for Ocamlnet 2. </p><p> Testers are especially encouraged to check whether Ocamlnet 3 still works on all platforms, because a lot of new platform-specific code has been added. I mainly tested with Linux and the MinGW port for Win32. </p><p> The library is not yet available via GODI. I'm working on this. </p><p> More blog postings will follow describing the highlights. </p><p> Please report results to gerd@gerd-stolpmann.de </p> <img src="/files/img/blog/ocamlnet3_test1_bug.gif" width="1" height="1"/> </div> <div> Gerd Stolpmann works as O'Caml consultant. <a href="http://www.gerd-stolpmann.de/buero/work_ocaml_search.html.en"> He is accepting new customers!</a> </div> <div> </div> The next server, please! http://blog.camlcity.org/blog/ocamlnet3_ha.html http://blog.camlcity.org/blog/ocamlnet3_ha.html <div> <b>What's new in Ocamlnet 3: Highly-available RPC</b><br/>  </div> <div> Ocamlnet is being renovated, and there will be soon a first testing version of Ocamlnet 3. The author, Gerd Stolpmann, explains in a series of articles what is new, and why Ocamlnet is the best networking platform ever seen. Experience with Hydro, another RPC implementation for Ocaml using the ICE protocol, has shown that it is advantageous to provide a layer that automatically manages the reaction of the RPC system in case of machine failures. Ocamlnet 3 adds such a layer for SunRPC clients. </div> <div> <p> The SunRPC client implementation included in past versions of Ocamlnet does not deal in any way with connection failures. The socket error was simply passed through to the caller, and the RPC client had then to be shut down. It was up to the caller how to react on such incidents. For example, one could indicate to the end user that the operation cannot be carried out, or one could retry the call. Programming practice showed that it is cumbersome to develop anew the right reaction for every type of RPC connection used in a larger system. The upcoming Ocamlnet 3 release tries to be better here. On top of the existing <code>Rpc_client</code> module the programmer may now install an <code>Rpc_proxy</code> module that handles socket errors much more nicely. </p><p> However, before going into detail here, let us first step back, and look at distributed systems that try to handle machine failures. The assumption here is that a socket error is caused by a malfunctioning (usually dead) machine, and that a simple repetition of the RPC call to the same machine is not the most prospective reaction. Distributed systems that can cope with failures of one or several nodes are called <b>highly available</b> (HA). The development of software that has built-in HA capabilities is known be very difficult, and it is especially useful when there is library support for dealing with machine failures. </p><h2>Examples of fail-over scenarios</h2> <img src="/files/img/blog/ocamlnet3_ha_1.gif" width="389" height="326" align="right" hspace="10" vspace="10"/> A frequent case is the simple fail-over from one server to an alternate server (see picture). Ideally, the alternate server is a copy of the primary server and operates on exactly the same data set. It is beyond the scope of this article how this can be achieved - possible solutions include hardware-based approaches like switching disks from one machine to the other, or software approaches like replicating all data modifications by means of commit protocols. For Ocamlnet it is only important that the client needs a criterion for when to switch to the other server, and instructions how to switch. <p>When to switch: Basically, we have the case that the client can autonomously decide whether to contact the secondary server instead of the primary one, and the case that the client needs external help. For example, the client could look at socket errors, and if the number of such errors for a certain server exceeds a maximum, the server is declared to be dead, and the client switches autonomously to the alternate server. This is convenient, but it assumes that the alternate server is at any time a replacement for the primary one. Especially when the fail-over is hardware-based this is not valid - before the alternate server is ready a special fail-over procedure must be run that disconnects the disks from the old server and attaches them to the replacement machine. Also, the old server needs to be isolated (i.e. "zombie machines" are ensured to be unreachable from the rest of the network). There is another reason for relying on external information about when to switch: Often, it is resaonable to monitor the cluster machines by special monitoring daemons. These are in a better position to collect and aggregate information about the liveliness of the cluster machines. For example, such services can contiuously ping the nodes of the cluster and also check which network ports are actually reachable. So it is often advisable to factor out the decision whether a node is alive or dead even if the client could do this on its own. </p><p>How to switch: RPC calls that can be simply repeated without changing the meaning are called idempotent. For example, if the client just wants to get the value of a remote variable, the RPC call doing so can be simply repeated as often as necessary to obtain the result. In contrast to that an RPC procedure that modifies a remote variable can often not be repeated (e.g. imagine an integer variable is increased by 1). This case is way more complicated to handle, and one ends often up with a so-called commit protocol. This means that the modification is encapsulated as a transaction that is either completely executed or aborted meaning that the effects are completely nullified. The commit protocol ensures that both the client and the server know whether the transaction is finally done or not done. Also, the commit protcol makes it possible to keep data sets synchronized that are stored on several servers. For Ocamlnet this means that the right reaction is either trivial (redirection of the failed idempotent call to the alternate server), or so complex that generic library support is impossible. </p><h3>A HA scheme for partitioning data</h3> <p> <img src="/files/img/blog/ocamlnet3_ha_2.gif" width="446" height="282" align="right" hspace="10" vspace="10"/> Often, HA is not the only reason for using a cluster of computers. Another motivation is to increase the total capacity of the system by spreading the load over several systems. In the picture there is an example where the load is shared by four servers. For pure load-balancing there is no need to check for and manage connection failures - before doing an RPC call, the right destination machine is simply selected, and the call is directed to it. </p><p>However, load-balancing can be combined with HA. The depicted system allows the client to read-access a dataset that is split into four partitions - the read, blue, gree, and brown partition. Each server has one of these partitions as its primary dataset. For example, when the client needs to read a data item from the blue partition it can go to server 2 because this server has all "blue data" (shown as blue square). In addition to that, all servers also store replicated data from other servers to provide a back-up in the case that a node fails. An easy way to do this would be to store the replica for server k on server k'=(k+1) mod N. However, this leads to the problem that the server k' gets double load when server k fails, foiling the increase of capacity by sharing load. The system in the picture implements a better scheme. Here, the partitions are further subdivided so that equal fractions of any partition are replicated on all machines. For example, the blue partition is split into three smaller parts which are then replicated on the three alternate nodes. When server 2 fails, the client can still find all the "blue data" on the remaining three machines, and because all remaining machines get an equal part of the fail-over load the overall capacity of the system shrinks only by 25%. </p><p>This scheme has to be considered for the Ocamlnet library in so far it is no longer always the same node that replaces a failing node. It depends now on the data item which server is the right alternate machine if the primary server fails. </p><h2>Detecting node outages</h2> When a node goes down, we assume that any pending TCP connection to it hangs, and also that attempts for creating new connections are not responded. After some time, the router usually detects that the node is unavailable, and emits special ICMP packets notifying machines in the network about the outage. However, this takes usually a few minutes. Until then, the cluster system needs other mechanims to detect the unavailablility of the node. <p>In the literature (including this text) a machine is usually either up or down, and there is nothing in between. In practice, however, the problem of zombie machines really exists, and is not as rare as many think. Often, the cause for hovering between life and death are bad disks - disk accesses still work, but take a multitude of the usual time to complete. Of course, such bad nodes are unusable, and should be counted to the dead nodes, but the problem is that the network ports are still responsive. This does not render the network connectivity tests useless, but one should keep in mind that these tests do not cover all reasons for an outage. </p><p>The most basic mechanism for recognizing failed nodes are timeouts. Due to the nature of timeouts it takes some time until such a test can indicate a failure, and this limits the usefulness of timeouts. Nevertheless, timeouts are important even when there are better criteria because they are often the only way to interrupt already established but now hanging TCP connections. We have to distinguish between two kinds of timeouts: Firstly, there are timeouts on individual send and receive operations. In many implementations of network protocols these are the only kind of timeouts. Essentially, these timeouts specify a lower bound on the bandwidth of the connection, and they work independently of the length of the exchanged messages. Secondly, one can also set an upper bound on the total time for the RPC call (including the time for both the request and the response). As RPC is usually run on fast LAN's, and specifications of RPC systems often require upper limits on the latency as seen by the end user (something like "95% of the user requests must be responded within 0.5 seconds"), this second interpretation is the more interesting one. Actually, Ocamlnet only implements this second notion of timeout. </p><p>The question is whether there are faster ways of detecting failed nodes. One method is surprisingly simple: Each node running an RPC client pings the nodes with the RPC servers the clients are connected to. A repeatedly failed ping is considered as a node outage. Ocamlnet 3 does not (yet) implement such a component, but the new <code>Rpc_proxy</code> layer is prepared to take external information about outages into account. (For an implementation look at <a href="http://oss.wink.com/hydro/hydro-0.7.1/doc/html/Hydrodoc_hydromon.html">Hydromon</a>. Although written for ICE instead of SunRPC it is straight-forward to port Hydromon to Ocamlnet's SunRPC implementation.) Even on busy servers this method is able to detect bad nodes within a few seconds. </p><p>Another idea is to share information about good and bad nodes as much as possible. If one connection runs into a timeout and the node is considered as failed, this information should be made available to all threads, and if possible even to all processes of the node. This avoids further attempts to connect to the failed node, and one can even cancel pending calls to this node. </p><h2>Proxies: Configurable connection management</h2> After this lengthy foreword, let's look at the actual implementation of connection management in <code>Rpc_proxy</code>. The name, "proxy", reminds us of the ideal of RPC: A remote call should be as simple and safe to conduct as a local procedure call, and the "proxy" is the local agent dealing with most of the complexities of remote calls, so that they look as much as possible like local calls. <p>This module is divided into three parts: </p><ul> <li><code>Rpc_proxy.ReliabilityCache</code> remembers which nodes and which network ports failed in the past, and decides on this information which nodes are considered to be down. </li><li><code>Rpc_proxy.ManagedClient</code> is an encapsulation of the simpler <code>Rpc_client</code> that is able to reconnect to a service. Effectively, a <code>ManagedClient</code> is a single TCP connection to a remote service that can be reestablished after an error. </li><li><code>Rpc_proxy.ManagedSet</code> is a set of <code>ManagedClient</code>s for the same RPC service that controls how many RPC calls are routed over each managed client, and that also implements the logic for skipping bad clients. </li></ul> <p>The generator for the language mapping, <code>ocamlrpcgen</code>, has also been extended. It emits now a functorized version of the client wrappers. For example, for the RPC program <code>P</code> the generator would output a functor </p><pre> module Make'P : Rpc_client.USE_CLIENT -> sig ... end </pre> where the returned module includes for every procedure <code>p</code> a call wrapper <pre> val p : C.t -> argument -> result </pre> Here, <code>C</code> is the input module <code>Rpc_client.USE_CLIENT</code>. This means one can inject any client implementation that at least implements <code>Rpc_client.USE_CLIENT</code>, and is not restricted to the default implementation <code>Rpc_client</code>. Of course, the enhanced client implementation <code>Rpc_proxy</code> also provides the <code>Rpc_client.USE_CLIENT</code> functionality, so it can be substituted here: <pre> module P = Make'P(Rpc_proxy.ManagedClient) </pre> The user is free, however, to pass any module as input here that meets the formal requirements - for example, one could pass further improved or derived versions of <code>Rpc_proxy.ManagedClient</code>. (Of course, the output generated by the new <code>ocamlrpcgen</code> is compatible with what earlier versions emitted. For users it is not required to switch to the functorized version.) <h3>Comparing basic clients and managed clients</h3> Basic clients as provided by <code>Rpc_client</code> live as long as the underlying TCP connection exists. This means the connection is created at the beginning of the client's lifetime, and the client becomes unsuable when the connection ends (either programmatically, or because of socket errors). In contrast to this, a managed client can establish the connection again when needed - but only to the same host and port. (Fail-overs to other hosts/ports are implemented in <code>ManagedSet</code>, see below.) <p>The basic clients are configured after being created - by invoking configuration functions like <code>configure</code>, <code>set_exception_handler</code> or <code>set_auth_methods</code>. This has the advantage that the configuration can be changed during the lifetime of the client. For managed clients, however, we require that the configuration must be fully given at creation time. The point is here that the managed client internally creates a series of basic clients, and immediately needs to know how these clients are to be configured. </p><p>For example, this piece of code creates a managed client: </p><pre> let mconfig = Rpc_proxy.ManagedClient.create_mclient_config ~programs:[ P._program ] ~msg_timeout: 10.0 () let mclient = Rpc_proxy.ManagedClient.create_mclient (Rpc_client.Internet(Unix.inet_addr_of_string "10.3.2.55", 9005)) (esys : Unixqueue.event_system) </pre> <p>Note that the TCP connection is not immediately created, but only when the first remote procedure is called. This is on line with the automatic reconnection feature: When the connection is closed or crashes, the next procedure call triggers that the connection is established again. </p><p>In this example, we have only configured a message timeout - the client will signal a timeout when the response takes more than 10 seconds to arrive. Normally, the timeout does not mean that the connection is considered as broken. We have to explicitly configure this feature - the proxy layer cannot know by itself what possible reasons for timeouts are. This is just another argument of <code>create_mclient_config</code>: </p><pre> Rpc_proxy.ManagedClient.create_mclient_config ... ~msg_timeout_is_fatal:true ... () </pre> <p>A fatal error is recorded, and may, depending on the configuration of the <code>ReliabilityCache</code>, disable the server host and/or server port for some time. The <code>ManagedSet</code> layer will recognize the error situation, and take this information into account when selecting good server endpoints for future RPC calls - but more on that below. On the level of <code>ManagedClient</code> it is only important to define which malfunctions are considered as fatal errors that may trigger the fail-over to other server nodes. As a <code>ManagedClient</code> is always only connected with one server node it is pointless to decide what to do in case of node failures. </p><p>There are two more connection management features: </p><ul> <li>One can set an idle timeout. Unused TCP connections are closed after the idle timeout period is over. The idea is to save server resources by closing unused connections - which often also means that the thread or process in the server can be released. Also, some network configurations do not tolerate that connections remain unused for longer periods of time. </li><li>One can demand to ping the service before sending the first RPC request. The background is that servers usually accept TCP connections before knowing whether there are really enough resources to handle them (i.e. free threads or processes). So it may happen that a TCP connection can be established, but it turns out to be immediately dead. By requesting one ping before really using the connection one can detect this problem. </li></ul> <p>Once the <code>ManagedClient</code> is configured and created, it can be used in a similar way as the older <code>Rpc_client</code>. For example, to call <code>do_something</code> we would write </p><pre> let result = P.do_something mclient argument </pre> (given that <code>P</code> is the applied functor from above). The asynchronous version, <code>P.do_something'async</code> is also available. <h3>The <code>ReliabilityCache</code></h3> The reliability cache is the instance that decides whether a server host or server port is available or not. The cache implements a logic for disabling servers when RPC clients recently reported fatal errors. Higher RPC levels such as <code>ManagedSet</code> can then interpret this information and fail-over to alternate servers. In the bigger picture outlined in the first half of this article the cache plays primarily the role of the autonomous fail-over trigger. In addition to that, there is also a hook for plugging in external sources for recognizing dead nodes. <p>The reliability cache is notified by the managed clients whether RPC calls lead to fatal errors or not. The cache disables the host or port of the server when a certain number of fatal errors occur in sequence. As we don't have a better idea for how long to disable, the cache simply implements a time-based logic where the length of the time period the server is assumed to be down depends on the number of fatal errors: Following fatal errors double the duration of the assumed unavailability until a maximum duration is reached. (A justification for this logic are server overloads: When a server cannot be contacted because of too much load, it is reasonable to delay the addition of load, and to consider the delay as a function of past slowness.) </p><p>There is usually only one reliability cache per process (the "global" cache). This is reasonable because the information about failures should be spread as far as possible. Nevertheless, each <code>ManagedClient</code> can configure its own criteria when to disable servers. The function </p><pre> val derive_rcache : rcache -> rcache_config -> rcache </pre> makes this possible: A new cache is created with a new configuration, but the cache shares error data with a parent cache. For example, this could be used to set the hook for getting external liveliness information: <pre> let rconfig = Rpc_proxy.ReliabilityCache.create_rcache_config ~availability:(fun rcache sockaddr -> ...) () let rc = Rpc_proxy.ReliabilityCache.derive_rcache (Rpc_proxy.ReliabilityCache.global_rcache()) rconfig </pre> <p>Right now, there is no mechanism to make the mentioned error counters accessible across process boundaries. This may be added in future versions of Ocamlnet. </p><h3>The <code>ManagedSet</code> of RPC clients</h3> Finally, we come to the layer that represents possible connections to several server nodes, and that decides what to do when one of the servers dies. <p>A managed set of clients is configured by specifying an array of remote addresses (usually corresponding to the available server nodes). For each of the addresses the set may create a managed client. Basically, there are two scenarios: Either the addresses are seen as alternate server endpoints (when the first one fails try the second etc.), or they are seen as equivalent, and it is tried to distribute the load over all server endpoints that are alive. This translates into the two policies managed sets can operate under: <code>`Failover</code> and <code>`Balance_load</code>. </p><p>For example, this managed set specifies a fail-over policy: When 10.3.2.55 fails, it tries to contact 10.3.3.55 instead. </p><pre> let sconfig = Rpc_proxy.ManagedSet.create_mset_config ~policy:`Failover () let mset = Rpc_proxy.ManagedSet.create_mset sconfig [| Rpc_client.Inet(Unix.inet_addr_of_string "10.3.2.55", 9009), 100; Rpc_client.Inet(Unix.inet_addr_of_string "10.3.3.55", 9009), 100; |] (esys : Unixqueue.event_system) </pre> <p>The number 100 is the maximum number of simultaneous connections to this endpoint. The managed set automatically increases the number of connections to the selected endpoint when the load grows (where the load is the number of simultaneously submitted RPC calls - remember this is an asynchronous RPC implementation, and it is able to handle several RPC calls at the same time). When the maximum is reached, further connections attempts are not rejected, but the alternate endpoint is then used instead. (So <code>`Failover</code> does not prevent that the alternate endpoint is used, it only specifies the preference to use the first endpoint as long as possible.) </p><p>If we had passed <code>~policy:`Balance_load</code> instead, the managed set would create connections to both servers, and try to achieve that the number RPC calls directed to each of the servers is roughly the same. </p><p>The user code calls <code>mset_pick</code> to get the next managed client according to these selection rules: </p><pre> let mclient, index = Rpc_proxy.ManagedSet.mset_pick mset </pre> There is also the optional <code>from</code> argument allowing to restrict the endpoints from which the client must be chosen. E.g. <pre> let mclient, index = Rpc_proxy.ManagedSet.mset_pick ~from:[2;3] mset </pre> would take either the endpoint at index 2 or the endpoint at index 3 from the endpoint array, and return the actually selected index. This is useful for fail-over scenarios where the node is a function of the input data - like in the presented partitioning example. <p>Managed sets do not repeat failed RPC calls by themselves. They only record failures in the <code>rcache</code>, and if enough failures happened, the problematic server is no longer considered as alive when submitting new calls. For idempotent calls, however, there is some support for automatic repetition. But first let's have a closer look at how the clever partitioning scheme from the above picture could be implemented. </p><h3>Sketch how to implement partitioned data sets</h3> In this example, we had four servers: <pre> let server1 = Rpc_client.Inet(Unix.inet_addr_of_string "10.0.0.1", 7654) let server2 = Rpc_client.Inet(Unix.inet_addr_of_string "10.0.0.2", 7654) let server3 = Rpc_client.Inet(Unix.inet_addr_of_string "10.0.0.3", 7654) let server4 = Rpc_client.Inet(Unix.inet_addr_of_string "10.0.0.4", 7654) </pre> In order to access the data item with key <code>k : key</code> we define two hash functions, one for the primary partitioning scheme, and one for the secondary scheme: <pre> val h_primary : key -> int (* actually: key -> {0..3} *) val h_secondary : key -> int (* actually: key -> {0..3} *) </pre> For example, one could define these as: <pre> let h_primary (k : string) = (Char.code (Digest.string k).[0]) land 3 let h_secondary (k : string) = let h1 = h_primary k in let rec attempt j = let k' = k ^ string_of_int j in let h2 = (Char.code (Digest.string k').[0]) land 3 in if h2 = h1 then ( if j = 4 then (h1+1) mod 4 (* ensure that the loop terminates *) else attempt (j+1) ) else h2 in attempt 0 </pre> Now, we create our managed set: <pre> let sconfig = Rpc_proxy.ManagedSet.create_mset_config ~policy:`Failover () let mset = Rpc_proxy.ManagedSet.create_mset sconfig [| server1, 100; server2, 100; server3, 100; server4, 100; |] (esys : Unixqueue.event_system) </pre> When trying to get the record for key <code>k</code>, we first compute the two nodes that store this record: <pre> let n1 = h_primary k let n2 = h_secondary k </pre> Finally, we pick a managed client from the managed set. We have to restrict the set of available servers to those where the paritioning functions say they store the record. So we get <pre> let mclient, index = Rpc_proxy.ManagedSet.mset_pick ~from:[n1;n2] mset </pre> <p>The <code>mclient</code> can then be used for calling a remote procedure. Remember that managed clients/sets do not automatically repeat failed calls, so we still can get socket errors etc. The error is only recorded, and when the server looks too faulty, it is finally disabled, and the managed set will no longer pick it. Such a call could look like: </p><pre> let result = P.get_file mclient file_id </pre> <p>Idempotent calls can be repeated without changing the semantics. For this reason, managed sets have a little helper to do so: </p><pre> let result = Rpc_proxy.ManagedSet.idempotent_sync_call ~from:[n1;n2] mset P.get_file'async file_id </pre> <p>This would repeat the <code>get_file</code> call in case of errors (up to a configurable maximum). Note that we have to pass the asynchronous version of <code>get_file</code> as argument although the resulting call is synchronous. </p><p>Summarized, the proxy module achieves that the actual fail-over logic is implemented within <code>ManagedSet</code> and the layers below it. The user code can rather focus, as shown, on describing which server to chose, and does not need to think in actions ("if this fails then try that"). </p><h2>Where to get Ocamlnet 3</h2> <p>There is no official release yet, not even an alpha release for developers. In order to get it, one has to check out <a href="https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/">the Subversion repository</a> (use the <code>svn</code> command on this URL, or click on it to view it with your web browser - most of the discussed code lives in <code>src/rpc</code>). </p> </div> <div> Gerd Stolpmann works as O'Caml consultant. <a href="http://www.gerd-stolpmann.de/buero/work_ocaml_search.html.en"> He is accepting new customers!</a> </div> <div> </div> Stranger in a strange land http://blog.camlcity.org/blog/ocamlnet3_win32.html http://blog.camlcity.org/blog/ocamlnet3_win32.html <div> <b>What's new in Ocamlnet 3: The Win32 port</b><br/>  </div> <div> Ocamlnet is being renovated, and there will be soon a first testing version of Ocamlnet 3. The author, Gerd Stolpmann, explains in a series of articles what is new, and why Ocamlnet is the best networking platform ever seen. One of the fundamental and intriguing improvements is the port to Win32 - after all, Win32 isn't well-known as good platform for asynchronous programming, but Ocamlnet is based on this programming style. A number of hard problems had to be tackled. </div> <div> <p> In the POSIX world (let me use "POSIX" as general term for Unix/Linux/BSD) the <code>select()</code> system call is known as the linchpin for dispatching file descriptor events. Generally, a program using <code>select()</code> looks like </p><pre> (* Event loop: *) while <something to do> do <find out interesting descriptors> select(); <interpret events> done </pre> and the crucial point is that all kinds of descriptors can be passed as input to <code>select()</code>. This makes it possible to wait simultaneously for very different events, like socket events, pipeline events, or events for devices. There is no such universal <code>select()</code> in Win32. The <code>select()</code> call provided by Win32 only works for sockets. There are other kinds of mechanisms for event handling, though, but they are less systematic, and one faces the problem that different ways of watching for events need to be integrated into the single event loop. <p> Since version 3.11, Ocaml includes a "fancy" version of <code>select()</code> in its standard library, which actually tries to emulate the POSIX semantics by combining several Win32 event-handling approaches in a quite tricky way. The reader might ask what the whole point for Ocamlnet is then - one could simply have relied on this emulation. However, there are a number of drawbacks. First, some of the used emulation techniques are incredibly simplistic. For example, if one of the descriptors references the output side of a pipe, the implementation falls back to a form of busy waiting (wasting CPU time). The input side of a pipe always signals that the pipe has space for new data (which may block the program). There is no special support for named pipes, although Win32 supports them much better than anoynmous pipes. The second problem is that the emulation is limited to 64 descriptors only - far too less for servers [please see update below]. For all these reasons, Ocamlnet does not make use of this <code>select()</code> emulation, but follows its own, more ambitious approach. Actually, the basic problem of this emulation is that <code>select()</code> is not the right level of abstraction for combining multiple ways of waiting for events into a single operation. </p><p> <b>Update:</b> I've got a message from Sylvain Le Gall, the implementor of this emulation code. He explained that actually 4096 handles can be waited for (which is true, sorry for my mistake). Also, he points out that he does not know how to handle anonymous pipes better, and that even Cygwin uses the same technique as his code. The reason for not treating named pipes specially is that the Ocaml standard library does not support them anyway. - I think his implementation decisions are perfectly rational for a general-purpose and drop-in <code>select()</code> emulation, and the standard library is now way better than before his contribution. However, Ocamlnet has special needs (like using named pipes as substitute for Unix Domain sockets), and by switching to pollsets (see below) as the API for handling events the system resources can be better managed. </p><h2>Overlapped I/O</h2> If you asked Microsoft whether Windows supported asynchronous programming well, they would point you proudly to overlapped I/O. Actually, overlapped I/O is a form of asynchronous I/O, but it is nevertheless very different from <code>select()</code>-style I/O. It works a lot like a non-blocking TCP connect: One has to start the I/O operation, and it is signaled to the caller when the operation is completed. The signal can be a callback (APC), or it can be an event variable that is set to signaled state. The difference to <code>select()</code> is that the latter indicates <b>in advance</b> whether an I/O operation would be possible (and non-blocking) before the operation is started whereas overlapped I/O requires that the operation is actually initiated, and one can only wait asynchronously for its completion. (Look <a href="http://msdn.microsoft.com/en-us/library/aa365683(VS.85).aspx"> here</a> for how Microsoft explains overlapped I/O.) <p>This article is not about which style is better, but how to port programs using Ocamlnet that assume <code>select()</code> loops as their basic construction principle. So the question here is: Can we get some kind of emulation of <code>select()</code> for overlapped I/O? </p><p>Before answering, let us look at for what we would need overlapped I/O. Essentially, one can use it for files, sockets, and a few IPC mechanisms. Ocamlnet does not have support for reading or writing files asychronously anyway, and for sockets it is easy to use the Win32 call <code>WSAWaitForMultipleEvents()</code> in order to combine waiting for sockets with other events. Actually, Ocamlnet is mostly interested in overlapped I/O for named pipes, because named pipes are a good replacement for the otherwise missing Unix Domain sockets and socketpairs. (Note that Win32 named pipes are a different IPC mechanism than POSIX named pipes, and support connection multiplexing like TCP sockets.) </p><p>The Win32 functions <code>ReadFile()</code> and <code>WriteFile()</code> can be used to start an overlapped I/O request by passing an <code>OVERLAPPED</code> struct to them (for a non-overlapped request the argument would remain NULL). This causes that the function returns immediately, and that the completion of the operation is signaled to the caller by an IPC mechanism. In Ocamlnet Win32 events are used for that purpose. A Win32 event is a synchronization primitive that works similar to a condition variable: It can be in unsignaled or in signaled state, and it is possible to suspend the program until it enters signaled state. Win32 events have the big advantage that they provide the required level of genericity: Many Win32 objects are actually either subtypes of Win32 events, and implement the event interface directly, or they can be connected to Win32 events. For example, a process handle is also an event, and it is signaled when the referenced process is terminated - so waiting for the process handle as event means to wait until the process is finished. It is also possible to wait until one of several events enter signaled state (<code>WaitForMultipleObjects()</code>). By putting an event into the <code>OVERLAPPED</code> struct Windows notifies the user about the completion of the operation by signalling the event. </p><p>Collecting events and waiting until one of them is signaled sounds already a lot like <code>select()</code>. Still, the problem remains that one has to start the operation before one can wait for it. There is no way around it on the Win32 level. The only chance for porting Ocamlnet was to change the level of abstraction the user code sees. Actually, it was possible to provide an emulation, but the price is that the user code must no longer invoke the generic read/write operations, but special wrappers that do the required impedance transformation. So <code>Unix.read</code> and <code>Unix.write</code> are forbidden when dealing with named pipes, and instead the special wrapper functions <code>Netsys_win32.pipe_read</code> and <code>Netsys_win32.pipe_write</code> have to be called. Ocamlnet provides buffers for input and output, so that <code>pipe_read</code> only reads from the input buffer (and raises <code>EWOULDBLOCK</code> if the buffer is empty), and that <code>pipe_write</code> only writes to the output buffer (and raises <code>EWOULDBLOCK</code> if the buffer is full). In addition to that, Ocamlnet organizes that overlapped I/O operations are started in the background when data needs to be pumped from the named pipe to the input buffer, or from the output buffer to the named pipe. This way, the overlapped operation is hidden from the user, and Ocamlnet can provide a view so that an event signals when a <code>pipe_read</code> or a <code>pipe_write</code> can actually process data. </p><p>Here is a small subset of the named pipe API provided by Ocamlnet in the module <a href="https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/src/netsys/netsys_win32.mli"><code>Netsys_win32</code></a>: </p><pre> type w32_event (* Win32 event objects *) type w32_pipe (* A pipe endpoint *) type pipe_mode = Pipe_in | Pipe_out | Pipe_duplex val pipe_pair : pipe_mode -> w32_pipe * w32_pipe (* like socketpair *) val pipe_read : w32_pipe -> string -> int -> int -> int (* like Unix.read *) val pipe_write : w32_pipe -> string -> int -> int -> int (* like Unix.write *) val pipe_shutdown : w32_pipe -> unit (* like Unix.shutdown *) val pipe_rd_event : w32_pipe -> w32_event val pipe_wr_event : w32_pipe -> w32_event (* get the events notifying about read/write possibility *) val wsa_wait_for_multiple_events : w32_event array -> int -> int option (* wait for a number of events, or until a timer times out *) </pre> <p>For instance, this code reads from two named pipes p1 and p2 simultaneously, and outputs the data to stdout: </p><pre> let s = String.create 1024 let try_read p = try let n = Netsys_win32.pipe_read p s 0 1024 in if n=0 then raise Exit; (* deal somehow with eof *) print_string (String.sub s 0 n) with Unix.Unix_error(Unix.EWOULDBLOCK,_,_) -> () let loop() = try let e1 = Netsys_win32.pipe_rd_event p1 in let e2 = Netsys_win32.pipe_rd_event p2 in while true do match Netsys_win32.wsa_wait_for_multiple_events [| e1; e2 |] (-1) with | None -> () | Some _ -> try_read p1; (* always try both for simplicity of the example *) try_read p2 done with | Exit -> () </pre> <p>Note that p1 and p2 have type <code>Netsys_win32.w32_pipe</code>, and <b>not</b> <code>Unix.file_descr</code>. </p><p>This example has the shape of the <code>select()</code> loop outlined at the beginning of the article. There are still differences, though, to the POSIX way of doing it: The file handle provided by the OS is hidden by the <code>Netsys_win32</code> layer, and cannot be directly used by the program (because this could break the abstraction). Also, one first has to create event objects (here by calling <code>pipe_rd_event</code>) in order to set up waiting. Last but not least the emulation itself is also not free of subtle artefacts introduced by the <code>Netsys_win32</code> layer. In particular, there is no way of cancelling the overlapped I/O operations performed under the hood of the emulation (one can only close/disconnect the pipe to stop them). This can be an issue when the file descriptor is passed on to other processes. (N.B. Windows Vista promises to solve the cancellation issue, but I had not yet a chance to test it.) </p><p>Of course, this approach only works when the watched Win32 file object implements overlapped I/O. If not, one can only read and write synchronously, and Ocamlnet provides special helper threads for dealing with this issue. This is discussed in more detail below. First lets look how to generalize <code>select()</code> so it can also be backed by the Win32 call <code>WSAWaitForMultipleEvents()</code>. </p><h2>The pollset class type</h2> The <code>select()</code> call is reflected by the Ocaml standard library as a function <pre> val Unix.select : file_descr list -> file_descr list -> file_descr list -> float -> file_descr list * file_descr list * file_descr list </pre> <p> This interface has a few disadvantages. First, in every round of waiting one has to pass all descriptors to <code>select()</code>. This is time-consuming, and the reason for the bad reputation of <code>select()</code> with regards to performance (although in reality is not as bad as some bloggers pretend). Second, there is no way to cancel an already started <code>select()</code> from a different thread. This is important for multi-threaded programs, because a second thread may want to change the list of descriptors the first thread is watching. </p><p> Ocamlnet uses now a different interface for polling descriptors, so-called <a href="https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/src/netsys/netsys_pollset.mli">pollsets</a>: </p><pre> class type pollset = object method find : Unix.file_descr -> poll_req_events method add : Unix.file_descr -> poll_req_events -> unit method remove : Unix.file_descr -> unit method wait : float -> ( Unix.file_descr * poll_req_events * poll_act_events ) list method dispose : unit -> unit method cancel_wait : bool -> unit end </pre> <p> These sets are used in this way: Descriptors may be added and removed from the set, and for each descriptor one can specify which events to watch for (reading or writing). When the set is ready, the user can invoke <code>wait</code> to start waiting for the specified events. The function returns the events that are actually signalled by the OS. It is possible to cancel waiting at any time by calling <code>cancel_wait true</code>. </p><p> I had not only the Win32 port in mind when designing the pollsets, but also POSIX-type OS. For example on Linux there is the epoll API that operates on a similar data structure, and that can easily back a pollset implementation. </p><p> The Win32 implementation of pollsets is done in two layers. The basic class <code>Netsys_pollset_win32.pollset</code> already supports all kinds of descriptors Ocamlnet needs, but is restricted to watch at most 64 Win32 event objects (corresponding to 63 sockets, or 31 named pipes). This restriction is abandoned by <code>Netsys_pollset_win32.threaded_pollset</code>. However, the latter class requires that the program is multi-threaded. </p><p> Essentially, the implementation works by inspecting the descriptors to be watched, and by looking up the required helper objects (like calling <code>Netsys_win32.pipe_rd_event</code> to get the Win32 event object reflecting the read status of a pipe). After that, <code>WSAWaitForMultipleEvents()</code> is invoked to start waiting, and when events happen, they are mapped back from the signaled event objects to the connected file descriptors. The <code>cancel_wait</code> feature is supported by always adding an additional Win32 event object to the set of watched events which is set to signaled state when <code>cancel_wait</code> is called. </p><p> Of course, this is only a rough sketch of the algorithm. It is quite complicated which helper objects are actually needed, and how they affect the central <code>WSAWaitForMultipleEvents()</code> call. Of course, this depends very much on the type of the descriptors put into the pollset, and it would go too far to fully present these details in this article. </p><p> However, one thing should not remain "magic" to the reader: In the above paragraphs, I pointed out that the representation of Win32 objects like named pipes is complex (e.g. it includes buffers, <code>OVERLAPPED</code> structs, and Win32 event objects), and that an opaque type like <code>Netsys_win32.w32_pipe</code> needs to hide the details of the representation from the user. Also, I mentioned that using the <code>Unix.file_descr</code> of the named pipe handle would break the abstraction, and that the handle is made unavailable to user code for this reason. However, pollsets nevertheless use file descriptors for passing system objects around. How does this fit together? </p><p> Ocamlnet does not give up on <code>Unix.file_descr</code> as the central type for referencing system objects - switching to a different type for this purpose would break tons of user code. Instead, a tricky mechanism has been added allowing us to keep <code>Unix.file_descr</code> but also to attach further management objects to such descriptors. This is explained in detail below. The crucial idea is that Ocamlnet introduces artificial descriptors that are only used for identifying system objects but that cannot be used for actually performing I/O. So the descriptor handed out to user code for a named pipe is not the Win32 handle for the named pipe (which would allow to do I/O and to break abstractions), but it is an additionally allocated handle that only exists for the purpose of identifying the system object. This handle, now called proxy descriptor, is the value passed to pollsets and other interfaces assuming <code>Unix.file_descr</code> as the type for referencing system objects. </p><h2>Handling I/O with helper threads</h2> Before looking at proxy descriptors, I should briefly present how Ocamlnet deals with other file handles than named pipes. <p> For <b>sockets</b> everything is very easy. As mentioned, the pollset implementation is based upon <code>WSAWaitForMultipleEvents()</code> which is actually a Winsock function. It supports sockets directly - no tricky emulation layers are required. </p><p> Win32 distinguishes <b>anonymous pipes</b> as returned by <code>Unix.pipe</code> from named pipes. Anonymous pipes do not support overlapped I/O. As this kind of pipes is important for starting subprocesses, Ocamlnet nevertheless tries to provide an asynchronous API for them. Because only synchronous I/O is possible helper threads need to be created which implement buffers in much the same way than it is done for overlapped I/O: The helper threads pump data from the buffer to the pipe, or from the pipe to the buffer (depending on the direction of I/O). The user code only accesses the buffer in a non-blocking way, and Win32 event objects are used to signal the state of the buffer (empty or full). The resulting API looks very much like the API for named pipes, and it is also required that the special <code>read</code> and <code>write</code> functions of the API are called by user code instead of <code>Unix.read</code> and <code>Unix.write</code>, and there are also proxy descriptors. As the implementation is done by helper threads, there is the difficulty how to stop these threads when there is no more interest in watching the descriptors. Unfortunately, this is not possible in the general case - when the pipe "hangs" the thread will also hang, and there is no means to interrupt it (there are no signals (software interrupts) in Win32, and thread cancellation is a hot issue). As anonymous pipes are mostly used for driving external processes this seems to be acceptable (there is always the fallback solution to kill the process). </p><p> The Win32 <b>consoles</b> are supported in the same way as anonymous pipes. </p><p> Even <b>processes</b> can be waited for. Although there is no direct data flow (neither <code>read</code> nor <code>write</code> make sense in any way), processes are referenced by means of file handles. When the handle is set to signaled state, this means that the process has terminated. So process handles can be added to pollsets, and this makes it easy to wait for the termination of a subprocess in parallel to managing the I/O over the pipes that are connected with the process. </p><p> For other types of file handles there is no good support yet (except one creates the mentioned helper threads). Of course, adding support would be easy for all handles where Win32 allows overlapped I/O. However, this seems not to be urgent. </p><h2>Proxy descriptors</h2> Back to the trick Ocamlnet uses to keep <code>Unix.file_descr</code> while having complex management objects for controlling asynchronous I/O. For example, one can get a proxy descriptor for a named pipe by calling: <pre> val pipe_descr : w32_pipe -> Unix.file_descr </pre> The returned descriptor cannot be used for anything except for looking up the attached named pipe up: <pre> val lookup_pipe : Unix.file_descr -> w32_pipe </pre> (There is also a slightly more general lookup function that can be used for any type of Win32 object using proxy descriptors.) <p> The proxy descriptors are backed by real file handles (otherwise it could happen that the next <code>open()</code> returns the same handle, and the proxy descriptor would no longer be identifiable as such), but a cheap kind of handle was chosen to avoid too much resource consumption. There is a hidden global table that maps proxy descriptors to the referenced complex objects, and by GC trickery it is ensured that the table shrinks when proxy descriptors are freed by GC runs (note that <code>Unix.file_descr</code> is a heap-allocated value for Win32, so we can add finalisers). </p><p> Of course, user code has to close the proxy descriptors when they are no longer needed (but only when they were actually requested). This means they have the same "lifetime" as normal file descriptors which also need to be closed after use. </p><h2>Some kind of generic API</h2> The consequence of the chosen emulation approach is that for each kind of file object a different set of I/O functions need to be called. This may be acceptable for special operations like <code>connect</code> where a generic approach is hard to get right, but is totally impractical for simple reading and writing. It would be required to call different functions that have very similar signatures, e.g. (read case) <code>pipe_read</code> for named pipes, <code>input_thread_read</code> for objects managed by helper threads, and of course the well-known <code>Unix.recv</code> for sockets and <code>Unix.read</code> for normal files. <p> In the <a href="https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/src/netsys/netsys.mli"><code>Netsys</code></a> module a simple generic approach of handling read and writes is available. There is a function inspecting the kind of file descriptor, and a set of generic functions for actually performing read/write: </p><pre> type fd_style (* indicates the kind of descriptor (details omitted here) *) val get_fd_style : Unix.file_descr -> fd_style (* get the file descriptor style *) val gread : fd_style -> Unix.file_descr -> string -> int -> int -> int (* generic read: call the right implementation function depending on the fd style *) val blocking_gread : fd_style -> Unix.file_descr -> string -> int -> int -> int (* similar to gread, but it is blocked until at least one byte can be read *) val really_gread : fd_style -> Unix.file_descr -> string -> int -> int -> unit (* similar to gread, but it is blocked until exactly the passed number of bytes are read *) (* similar functions are available for writing, for shutting down, and for closing *) </pre> If the descriptors are proxy descriptors, these functions automatically look up the underlying complex management object and invoke the right I/O function. If the descriptors are sockets, they call socket functions like <code>Unix.recv</code>. Otherwise, they fall back to <code>Unix.read</code> or <code>Unix.write</code>. <p> Large parts of Ocamlnet have been ported so they use this generic layer instead of directly calling <code>Unix.read</code> or <code>Unix.write</code>. For example, the class <code>Netchannels.input_descr</code> wraps a netchannel object around a file descriptor, and it has been changed so it can now also deal with all kinds of descriptors supported by <code>gread</code>. </p><h2>High-level I/O</h2> Many users don't want to see all these implementation details I have reported so far. They want to just use the high-level I/O functions like <code>Http_client</code>. The question is what can be supported on Win32. <p> Fortunately, the answer is - thanks to dealing with these details carefully - that almost everything works! Although not every module has been fully tested yet, the difficult modules could be ported, and there is now the conviction that the simpler ones are not in any way problematic. </p><p> The most difficult case was <b>Netplex</b>. Of course, there is no way to support multi-processing as there is no <code>fork()</code> equivalent in Win32. However, multi-threading works well. The socketpairs connecting the containers with the controller have been replaced by pairs of connected named pipes. For Unix Domain sockets there is the possibility of using either named pipes, or Internet sockets bound to localhost. </p><p> As Netplex uses <b>SunRPC</b> as base library, it was of course also possible to port this Ocamlnet feature. SunRPC cannot only be used on sockets, but also on named pipes. </p><p> Another difficult beast was the <b>Shell</b> library for starting and managing external processes. It is now as easy to create complex pipelines of interconnected subprocesses for Win32 as it used to be for POSIX. </p><p> The <b>Nethttpd</b> web server library could also be verified to be working, even in conjunction with Netplex. </p><h2>Where to get Ocamlnet 3</h2> There is no official release yet, not even an alpha release for developers. In order to get it, one has to check out <a href="https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/">the Subversion repository</a> (use the <code>svn</code> command on this URL, or click on it to view it with your web browser - most of the discussed code lives in <code>src/netsys</code>). <p> The Win32 port of Ocamlnet requires the MinGW port of Ocaml. Also, the same set of base libraries are needed as for POSIX, especially PCRE. The simplest way to install that is to use <a href="http://godi.camlcity.org">GODI</a> which also supports MinGW. </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> </div> Documentation fun http://blog.camlcity.org/blog/pxp121.html http://blog.camlcity.org/blog/pxp121.html <div> <b>PXP-1.2.1 with a new reference manual</b><br/>  </div> <div> The last release of a stable PXP version happened 5 years ago. That's a long time. Actually, a lot of devlopment took place since then, only that it was difficult to bring PXP into a releasable shape. Now the last missing piece has been added, namely extensive documentation. So I can proudly announce the best XML parser that has ever been available for programming in O'Caml: Not only fast and feature-rich, but also easy to understand. </div> <div> <p> Writing documentation is something programmers do not like very much, and also in the case of PXP the code was far ahead of any description about it. There was a "User's Guide", but it took an oldish approach of explaining things I don't like anymore. Also, it was very incomplete. Last year, I got some funding from a company to improve the PXP documentation, so I faced the problem to reorganize it completely, and to add anything missing. </p><p> If you would like to take a look at the result, here it is: <a href="http://projects.camlcity.org/projects/dl/pxp-1.2.1/doc/manual/html/ref/index.html">The PXP Reference</a>. </p><h2>Switching from docbook to ocamldoc</h2> <p> The old "User's Guide" was written as docbook document. This is a good general-purpose text format that allows one to structure a large text into chapters, sections, etc., and to generate viewable and printable output from it (especially one can convert it into a bunch of HTML pages, and into PDF). However, there is one difficulty: It does not integrate well with ocamldoc-style interface references.</p> <p> The "User's Guide" predates ocamldoc - when I wrote the first version of PXP documentation I had no other chance than to use some third-party tool to process it. Now what to do? Stick with docbook, and include ocamldoc somehow into the processing chain? The docbook format has clearly more features for formatting text, e.g. one can easily include pictures. However, ocamldoc cannot output in a format that would be convertible to docbook with only little effort, and this made this way unfeasible. </p><p> I decided to switch completely to ocamldoc. Not only the module interfaces should be documented with it, but also the various introductory chapters explaining concepts spanning several modules. Since O'Caml 3.09, ocamldoc understands the file suffix *.txt and takes these input files as pure documentation. One can still use all formatting directives like <code>{2 headings}</code> or <code>{!Hyperlinks}</code> pointing to code elements. However, there was still the difficulty of missing features. </p><p> So I looked at developing a custom HTML generator (I am mostly interested in outputting HTML). It is possible to load an add-on into ocamldoc that modifies its behaviour. One just has to write a class that inherits from <code>Odoc_html.html</code>, and overrides its methods: </p><pre> class chtml = object(self) inherit Odoc_html.html as super method private html_of_<foo> ... = ... end let chtml = new chtml let _ = Odoc_args.set_doc_generator (Some chtml :> Odoc_args.doc_generator option) </pre> <p> Of course, it was still the question whether my features could be added this way (without rewriting half of the generator class). Yes, they can, and it only needed about 160 lines of code. I must admit it took quite a long time to develop this code, since I had to dig into internals of ocamldoc to understand it better. But anyway, ocamldoc turns out to be a customizable utility. </p><p> What I added in particular: </p><ul> <li>A <code>{picture}</code> tag for including pictures<br/>  </li><li>The possibility to change the output for <code>include Module</code> in interfaces so that the included interface is directly shown instead of only the include statement as such. For clarity, the included interface is indented, and has grey background. This change can be turned on and off with a <code>{directinclude}</code> tag.<br/>  </li><li>The <code>include</code> change requires another feature to be really looking good. All references (hyperlinks and plain occurrences) pointing to the included module should be rewritten so that they point to the including module instead. That means if module <code>N</code> uses <code>include M</code> we want that all references <code>M.x</code> are changed into <code>N.x</code>. The intention is that <code>M</code> is no longer referenced, and that the duplication of definitions in two modules cannot confuse readers (especially those that are unfamiliar with the module system). I added that feature for my specific case, and the ocamldoc tag <code>{fixpxpcoretypes}</code> enables that rewriting. (It changes <code>Pxp_core_types.[S|I]</code> into <code>Pxp_types</code>.) <br/>  </li><li>A last change has also to do with the <code>include</code> feature. With <code>{knowntype}</code> and <code>{knownclass}</code> one can add identifiers to the lists of known types and classes, so that the generator will emit hyperlinks to them, although there is no such definition in reality. It turned out that many identifiers were already pointing to the including module, but because there is no definition in the mli file, ocamldoc does not make these identifiers clickable. With <code>{knowntype}</code> and <code>{knownclass}</code> one can change that on a case by case basis. </li></ul> <p> The full source code of the custom generator class can be studied here: <a href="https://godirepo.camlcity.org/svn/lib-pxp/tags/pxp-1.2.1/tools/src/odoc/chtml.ml">chtml.ml</a>. The module with the mentioned <code>include</code> directive is <code>Pxp_types</code>. <a href="http://projects.camlcity.org/projects/dl/pxp-1.2.1/doc/manual/html/ref/Pxp_types.html">Look here how nice the generated page is.</a> </p><h2>The need for conceptual introductions</h2> <p> XML is a cute and simple text format, right? Many people think like that, but given the fact that many XML parsers are either feature-rich and slow, or poor and fast, there must be some complexity in the XML definition. Recently, I read the article "XML fever" (by Erik Wilde and Robert J. Glushko, Communications of the ACM, issue 7, 2008), where the authors point out a number of deficiencies in the definition of XML that can lead to delusion about XML, and finally into "fever". After years of maintaining this XML parser, I can only second the authors. Clearly, there are problems even in the fundamental XML specification. </p><p> I do not want to complain about this - XML is widely used, and many of the standards are practically unfixable without breaking large numbers of programs. For me the problem arose how to explain all that. For example, there is the question what is to be considered as the root node of an XML tree. This is a conceptual question, and the explanation should not be hidden in an interface description of a PXP module. For that reason, I had to add a number of chapters to the manual that explain concepts and generally introduce into the PXP world. All the <code>Intro_*</code> chapters are like this. </p><p> The nice thing is now that I can add direct links from introductory chapters to interface references and vice versa, since all documentation is now processed with the same utility, ocamldoc. When some complicated issue arises in some function description, it is now possible to point to the section in the introduction where this issue is explained in detail, and conversely, I can point to the definition in the interface when a function or type is used in an intro chapter. </p><h2>What next?</h2> <p> I must admit that my interest in XML has not gained in the last years, to say it politely. XML is most often used as a base technology for HTML, or as a data exchange format. Many of the advanced XML standards like XSLT or XQuery have not found the way into the daily life of us programmers. The hype is over. </p><p> Nevertheless, I promise that I will still maintain PXP, and now and then add another feature. For example, there is a nice XPath evaluator in the development pipeline - again, I do not find time to finish it, but hey, there are still many years for doing it. (By the way, if you want to accelerate that and have some money, we will find a way to quickly finish XPath.) </p><p> In August 2009, PXP becomes 10 years old (counted from the first mentioning in the O'Caml mailing list). This is already a long time for a software library and an open source project. I am quite confident it will now also reach its 20th birthday! </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> <p><b>Links:</b> <ul> <li><a href="http://projects.camlcity.org/projects/pxp.html">PXP homepage</a>: Find here links to downloads and documentation </li> </ul> </p> </div> Parallelizing with Ocamlnet http://blog.camlcity.org/blog/parallelmm.html http://blog.camlcity.org/blog/parallelmm.html <div> <b>A scalable implementation of matrix multiplications in O'Caml</b><br/>  </div> <div> O'Caml seems not be recognized as a programming language where it is easy to parallelize tasks. Recently, there was a heated discussion on caml-list about this subject, and people complained that the memory management of O'Caml does not work well for multi-threaded programs. Actually, the memory manager enforces that only one O'Caml thread can execute code at any time, so that there is no way to make efficient use of multi-core CPUs. Well, there are ways around this limitation, and in this article I'll show how to multiply matrices in a distributed system. We don't use multi-threading, but multi-processing which means that the threads of execution don't have easy access to shared memory. For communication and synchronization we use remote procedure calls which can basically do the same as shared memory, but at higher cost. Most important, however, is that the restriction of the O'Caml memory manager no longer applies: The processes are independent, and every process has its own manager. </div> <div> <p>In order to tackle problem we need a way to break down the algorithm so that the running processes don't have to synchronize with each other often. This is very easy for the matrix multiplication (but may be more complicated for other problems): Every cell of the result matrix is the scalar product of one column of the first input matrix with one row of the second input matrix. Thus the cells can be computed independently, and one only has to take care that all processes have access to the input matrices, and that the processes do not interfer with each other when writing the results. </p><h2>The tools</h2> In this solution of the problem we use the toolset provided by Ocamlnet. It includes a mature and quite efficient implementation of SunRPC we'll use for interprocess communication. Furthermore, there is Netplex for creating and managing multi-process servers. The plan is to create an RPC service for multiplying matrices that can take advantage of several cores and that can even run in a compute cluster. <p>The Netplex/SunRPC combination is quite powerful. Basically, a SunRPC server accepts TCP connections, and executes the incoming RPC calls. With our tools we can choose whether we want to have a single process that processes all connections, or whether we want to create new processes for every connection. Obviously, the latter is the way to go for doing the hard computation work, because O'Caml allows us only to take advantage of several cores by running the code in different processes. The single process choice is also useful, namely for controlling the computation, i.e. for synchronization. In this example we'll use both types of process management. </p><p>SunRPC is a quite aged standard for doing remote procedure calls. RPC is typed message passing, that means we cannot only pass strings from one process to the other, but values of all types of the interface definition language. In SunRPC we have ints, floats, strings, arrays, options, records ("structs"), and variants ("unions"). Furthermore, RPCs are usually "two-way" messages: Input messages are always replied with output messages. </p><h2>The components</h2> We have two components. One is called the "controller" and manages the work to do. Actually, the controller defines the outer loops of the algorithm. Furthermore, there is the "worker" component. The worker can be instantiated several times (and runs then in several processes). It implements the inner loop of the algorithm that runs in parallel. <p>The general idea is that the controller triggers the workers when a new multiplication is to be done, and that the workers are themselves repsonsible for getting the input data, and for getting the tasks to execute. In the SunRPC IDL this looks like: </p><blockquote><small> <pre> program Controller { version V1 { void ping(void) = 0; dim get_dim(which) = 1; /* Workers can call this proc to get the dimension of the matrix */ row get_row(which,int) = 2; /* Workers can call this proc to get a row of the matrix. The int is the row number, 0..rows-1 */ jobs pull_jobs(int) = 3; /* The controller maintains a queue of jobs. This proc pulls a list of jobs from this queue. The int is the requested number. */ void put_results(results) = 4; /* Put results into the result matrix. */ } = 1; } = 2; program Worker { version V1 { void ping(void) = 0; void run(void) = 1; /* The controller calls this proc to initiate the action in the worker. When it returns, it is assumed that the worker is completely finished with everything. */ } = 1; } = 3; </pre> </small></blockquote> (There is also a program Multiplier with program number 1 we'll show later.) <p>These are the procedures the two components expose. When a multplication is to be done, the order of actions is as follows: </p><ol> <li>We assume the controller has the input data. The controller calls the "run" RPC of all workers to trigger them. </li><li>The workers now fetch the input data from the controller by calling "get_dim" and then "get_row" until they have all input values. </li><li>The workers now run jobs from the controller by repeatedly calling "pull_jobs", then executing the jobs, and finally transmitting the results back to the controller by invoking "put_results". The workers can pull several jobs at one, and they can put several results at once. This reduces the relative costs of the RPC overhead per job. </li><li>The workers are done when there are no more jobs, i.e. the list of jobs pulled from the controller is empty. Now they can pass back the result message of "run" to the controller. </li></ol> Note that the controller must be able to do several things in an overlapped way: After it has sent the input messages to the "run" RPC of the workers it must be responsive and accept incoming RPC calls from the workers. Finally it must wait until all result messages from the "run" RPCs have arrived. Ocamlnet's RPC implementation supports such overlapped RPC execution. (Note that Sun's original implementation does not support this!) <p>The Multiplier program is just another interface of the controller. It is the public interface used by the client of the multiplier: </p><blockquote><small> <pre> program Multiplier { version V1 { void ping(void) = 0; void test_multiply(int,int,int) = 1; /* Creates a test matrix with random values and multiplies them. Args are: (l_rows, r_cols, l_cols = r_rows) */ } = 1; } = 1; </pre> </small></blockquote> <h2>The implementation</h2> The complete source code is available here: <ul> <li><a href="https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/examples/rpc/matrixmult/">code/examples/rpc/matrixmult</a> </li></ul> It will also be available in future releases of Ocamlnet, but for now it is only accessible via Subversion. <p>Of course, using a distributed approach creates a lot of coding overhead. Compare with the direct implementation in simple.ml (also in this directory): The multiplication algorithm counts there only 9 lines of code. Compared with that we had to develop a lot of additional stuff: 389 lines for the distributed implementation. This is the price we have to pay in terms of coding effort. </p><p>Netplex requires a configuration file, here mm_server.cfg. Basically, the file lists the components that are supposed to accept incoming connections. Here, it looks like: </p><blockquote><small> <pre> netplex { ... service { name = "mm_controller"; protocol { ... }; processor { type = "mm_controller"; worker { host = "localhost"; port = 2022 }; worker { host = "localhost"; port = 2022 }; }; workload_manager { ... }; }; service { name = "mm_worker"; protocol { ... }; processor { type = "mm_worker"; controller_host = "localhost"; controller_port = 2021; }; workload_manager { ... }; }; } </pre> </small></blockquote> We have omitted boring parts ("..."). We configure the mm_controller that connects with two workers, both reachable over TCP port localhost:2022, and the mm_workers talking to the controller over port localhost:2021. By including the same worker port twice in the list of workers, two independent connections are created, and because of the worker configuration two processes are started. (This is configured in the omitted workload_manager block.) <p>Netplex starts exactly the components it finds in the configuration file, even if more components are implemented in the code. This is quite useful, because the same binary can be used for different setups. In this example, we could run the multiplication server on host X with both controller and workers, and on host Y with only the workers. </p><h2>Some numbers</h2> I've done some performance tests on a Opteron 2212 system with 2 CPUs and each CPU has 2 cores (i.e. 4 cores in total). The tests were done in 64 bit mode and ocamlopt-compiled programs. The interesting question is not the total performance, but how much overhead we have by the message passing, and how scalable the system is. <p>The test is to square an NxN matrix of random numbers, for N=1000, N=2000, and N=3000. This has been done with the non-distributed version of the multiplation (simple.ml), and with the distributed version and W workers, for W=1, W=2, and W=4. </p><p>What we expect is that for larger input matrices the relative cost of message passing drops, because we have to transfer the input matrix only once for every worker, and the result matrix only once in total, but the computation time for the multiplication is O(N<sup><small>3</small></sup>) compared to O(N<sup><small>2</small></sup>) for message passing. </p><p>Here are the runtimes, as table, and as diagram: </p><table cellspacing="5" border="1"> <tr> <th>Test</th> <th>N</th> <th>Runtime (seconds)</th> </tr> <tr> <td>Non-distributed</td> <td>1000</td> <td>38.61</td> </tr> <tr> <td>Non-distributed</td> <td>2000</td> <td>359.765</td> </tr> <tr> <td>Non-distributed</td> <td>3000</td> <td>1483.759</td> </tr> <tr> <td>Distributed, W=1</td> <td>1000</td> <td>45.174</td> </tr> <tr> <td>Distributed, W=1</td> <td>2000</td> <td>382.874</td> </tr> <tr> <td>Distributed, W=1</td> <td>3000</td> <td>1847.225</td> </tr> <tr> <td>Distributed, W=2</td> <td>1000</td> <td>24.407</td> </tr> <tr> <td>Distributed, W=2</td> <td>2000</td> <td>206.426</td> </tr> <tr> <td>Distributed, W=2</td> <td>3000</td> <td>849.089</td> </tr> <tr> <td>Distributed, W=4</td> <td>1000</td> <td>20.842</td> </tr> <tr> <td>Distributed, W=4</td> <td>2000</td> <td>144.718</td> </tr> <tr> <td>Distributed, W=4</td> <td>3000</td> <td>576.02</td> </tr> </table> <div> <img src="/files/img/parallelmm_runtime.png" width="450" height="588"/> </div> <p>Not bad, right? The time spent for message passing seems to be acceptable. If you compare the non-distributed time for N=3000 with the time of the distributed system and one worker, the overhead is approximately 25% of the total runtime. </p><p>For larger W the overhead for the initial transfer of the input matrices increases - they have to be copied to every worker. The algorithm can be improved here, because the controller process becomes soon the bottleneck as it has to copy the matrices for every worker. Alternatively, the matrices could be spread among the workers by the workers themselves in order to get a better degree of parallelization during the transfer phase. </p><p>As last, fun test I started workers on three nodes so that in total 10 cores were available (I couldn't do more because I did not find more idle systems in Wink's cluster). I got a runtime of 349 seconds for N=3000. As the cores were not identical, this does not tell us much except that we really can do it over the network, and that this multiplier really takes advantage of a cluster. </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> <p><b>Links:</b> <ul> <li><a href="http://projects.camlcity.org/projects/ocamlnet.html">Ocamlnet</a>: Project page </li> <li><a href="http://caml.inria.fr/pub/ml-archives/caml-list/2008/05/6e74a918c2903f1e11add5714a72e352.en.html">Thread about parallelization on caml-list</a>: Sometimes it sucks, sometimes it rocks </li> </ul> </p> </div> About LambdaRank http://blog.camlcity.org/blog/lambdarank.html http://blog.camlcity.org/blog/lambdarank.html <div> <b>The ranking algorithm behind GODI Search</b><br/>  </div> <div> The question has always been: Are the results better if a search engine really understands the text it indexes? You can view my latest project, <a href="http://docs.camlcity.org">GODI Search</a>, as an attempt to answer this question for a very limited set of documents, namely the code and its documentation of <a href="http://godi.camlcity.org">GODI</a>, the source code O'Caml distribution. Actually, understanding text allows it to rank the results better, so that the more important occurrences of the query words are shown more at the beginning of the result list. Because ranking really bases on text interpretation, and the text is here a variant of lambda code, I call this method LambdaRank. </div> <div> <p>As every search engine, GODI Search consists roughly of two main components, namely the indexer and the searcher. The indexer iterates over the set of documents, and populates a database with the words it extracts. The searcher is the simpler part of the game, as it finds everything well-prepared in this database, and when a query comes in, it only has to pull the matching documents out of the database, sort it according to the ranking score, and show it to the user. So the tricky part is the indexing. </p><p>If you don't understand the text at all, you cannot do much about ranking but count the occurrences of a word in a document. The idea is that a text that speaks about a certain subject also mentions the subject more often than other texts, and thus the number of occurrences is a good measure for ranking. In a document set where the texts are connected with hyperlinks one can furthermore look at the relationships between the documents. Google's PageRank is based on this approach (but, as rumors say, is now heavily modified from its original design). </p><p>Fortunately, we know a lot about the document set GODI Search analyzes. Most documents are like this: </p><ul> <li><a href="http://docs.camlcity.org/docs/godipkg/3.10/godi-ocaml/lib/ocaml/std-lib/list.mli">O'Caml module interfaces</a> </li><li><a href="http://docs.camlcity.org/docs/godipkg/3.10/godi-ocaml/lib/ocaml/std-lib/list.ml">O'Caml module implementations</a> </li><li><a href="http://docs.camlcity.org/docs/godipkg/3.10/godi-omake/doc/godi-omake/html/omake-doc.html">Technical manuals</a> </li><li><a href="http://docs.camlcity.org/docs/godipkg/3.10/apps-coq/share/emacs/site-lisp/coq.el">Code in programming languages other than O'Caml</a> </li><li><a href="http://docs.camlcity.org/docs/godipkg/3.10/apps-coq/lib/coq/ide/utf8.v">Code in exoctic languages</a> </li></ul> O'Caml code files and closely corresponding manuals dominate the corpus. GODI Search tries to make the best of this by analyzing O'Caml code in detail. <p>After looking at many examples I had some ideas which occurrences must be ranked higher than others. </p><p><b>Idea 1. Definitions of identifiers are more important than uses.</b> This sounds natural, but actually not every definition is important. GODI Search also looks at the scope of the definition: A local definition is restricted to a surrounding function, and scores the least. Then follow definitions on module level, and in top-level modules. The highest score is given to exported identifiers that occur in top-level module interfaces. </p><p>Only let-bound identifiers are ranked this way. Function arguments, variables in pattern matchings, and fun-bound identifiers are ignored. This is a bit arbitrary, but my feeling is that these identifiers are usually not important in the coding styles I'm aware of. </p><p>For types, exceptions, and module names similar scoring techniques exist. </p><p><b>Idea 2. Values and types are rated separately.</b> The namespace of all identifiers can be roughly divided into two big zones: Values and types. Of course, there are more kinds of identifiers (modules, classes, labels, file names,...), but my impression is that the typical programmer has a mind set that is dominated by only these two classes of symbols. In this sense, a "value" names an executable thing, and a "type" names meta data. Both have little to do with each other, and thus a document that contains many words "list" as type has little importance in a search for "list" as value. </p><p><b>Idea 3. Keywords are stopwords.</b> Keywords occur in practically every code file in big number, and thus say nothing about it. GODI Search simply ignores keywords. </p><p><b>Idea 4. Qualified identifiers are hyperlinks.</b> If you search for "mem" then you will get a list of top-level definitions of this function. The question is which occurrence is shown first. GODI Search implements the PageRank idea of scoring hyperlinks pointing to a document by looking at qualified identifiers. So if there are more "Hashtbl.find" than "List.find" in code anywhere in the corpus, the module "Hashtbl" scores higher than "List". (Actually, it is the other way round if you also take other references into account.) </p><p><b>Idea 5. Code and non-code are rated separately.</b> Of course, the above applies only to text sections that are O'Caml code. Other languages and non-code cannot be rated this way. For this reason, GODI Search puts a lot of effort into separating both types of text. Currently, this is done on a per-paragraph basis, i.e. every paragraph is first analyzed in order to know whether it is O'Caml code or not. Also, comments and string literals in code files are considered as non-code. </p><p>So far about the ideas behind LambdaRank. The results of the implementation look promising: If the user types in "fold_left" he or she will be taken to really relevant occurrences. And user experience is what counts, finally. </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> <p><b>Links:</b> <ul> <li><a href="http://docs.camlcity.org">GODI Search</a>: Try out GODI Search here </li> <li><a href="http://godi.camlcity.org">GODI Homepage</a>: About GODI in general </li> </ul> </p> </div> Cross-Language Cluster Computing http://blog.camlcity.org/blog/hydrostory.html http://blog.camlcity.org/blog/hydrostory.html <div> <b>The Story Behind Hydro</b><br/>  </div> <div> On the surface a search engine looks like a very simple web site, but actually most things happen in the backend, and are hidden from the user. A search engine consists of 10 to 20 different types of servers, and many of them are instantiated several times in a cluster configuration. No single programming language is well suited for the entire implementation. In addition, some language environments may have compelling libraries that are lacking in other languages. It is, however, still difficult to let servers written in different languages communicate with each other. At Wink, we decided to go with ICE, and to develop the missing O'Caml implementation ourselves. </div> <div> <p>For two years now I have done consulting work for <a href="http://wink.com">Wink Technologies</a> who started out as a user-powered search engine, but switched later to people search. Actually, people search began as an experiment, and was originally developed in O'Caml with some Java standard components. It turned out that O'Caml was very well suited for crawling and parsing, and that our solution was convincing enough so we could go on with O'Caml as implementation language. It was also a big plus that it was already possible to develop server clusters in O'Caml with Ocamlnet's SunRPC implementation - we only had to add a highly available directory and configuration service. However, the problem of SunRPC is that there is no good C++ implementation (interestingly, there is an acceptable one for Java, RemoteTea). Too bad, since SunRPC is simple and robust, and its type system matches the one of O'Caml quite well (there are records, arrays, variants, and option types). </p><h2>ICE: An RPC Middleware</h2> <p>Looking for an alternative, somebody came up with ICE ("Internet Communications Engine"). This is a commercial product by <a href="http://zeroc.com">ZeroC</a> which is dual-licensed under the GPL (like e.g. MySQL). There are implementations for a number of languages, including C++, Java, Python, and PHP, but unfortunately not for O'Caml. Well, this is no surprise, but at least the other company languages are covered. So we looked closer at ICE. Does it match our needs? </p><p>ICE follows the object-oriented paradigm. For an RPC middleware this means that a remote call is seen as sending a message to a remote object. Of course, it is possible to have several such objects of the same type, and creating instances is made possible by a class construct. Such a design is very acceptable, but unfortunately object orientation often meant in the past that the rest of the type system was crippled. To some degree, this also happened to ICE - especially, there are no variants and no option types. Well, not optimal, but there are at least clean "design patterns" how to emulate these missing features with classes, and for pure OO languages like Java the ICE approach simplifies the language mapping. </p><p>Unlike CORBA, there is a fixed protocol the components have to use to talk to each other. That means a client in language X can directly contact a server in language Y, and there is no need for an intermediate instance to translate the protocol. Basically, this means you can use ICE without any infrastructure - no central server you are dependent upon. For developing massively parallel cluster services this is an essential requirement, because such central servers don't scale well enough, and are single points of failure. </p><p>For using ICE in a cluster context, there is the IceGrid add-on. Basically, this is a highly available directory and configuration service, and serves for a similar purpose like the service we had developed for SunRPC before. Clients ask IceGrid where to find their servers in the network, and IceGrid replies with a suggestion of TCP ports. This can be used for load-balancing and for high availability. </p><h2>Hydro: Implementing ICE for O'Caml</h2> <p>After ICE was found to be good enough, we needed an implementation for O'Caml. Well, this was my field - I already developed the SunRPC support in Ocamlnet years ago, and this made me an expert for this type of work. It took only about 3 weeks until it was possible to generate client code, and about another week until server support was ready. However, it was still challenging work, because the ICE type system needed to be mapped to O'Caml's type system. Furthermore, the ICE reference manual was full of errors, and everything had to be checked against ZeroC's implementation. </p><p>The difficulty of the type mapping is that ICE demands that objects and exceptions can be downcasted. O'Caml, however, does not support this operation, because there is no efficient implementation of downcasting for a type system like O'Caml's that includes structural subtyping. Nevertheless, downcasting is a reasonable operation in the context of RPC, and it is hardly possible to get around it. </p><p>Maybe an example demonstrates this best. In Slice, the IDL for ICE, one can easily define a hierarchy of classes (the syntax resembles Java's): </p><blockquote lang="x-slice"> <code><pre> class SearchResult { string url; string title; } class PeopleSearchResult extends SearchResult { string name; } class BandSearchResult extends SearchResult { string bandName; stringSeq bandMembers; } </pre></code></blockquote> <p>When the search engine returns a <code>SearchResult</code> item, it can also be one of the descendants of this class. Of course, a client of the search engine that simply wants to display the result, needs to know all details, and thus downcasts <code>SearchResult</code> to the real subclass. </p><p>In a normal OO program one can get rid of this downcast by adding an operation for displaying the result: </p><blockquote lang="x-slice"> <code><pre> class SearchResult { string url; string title; string display(); } </pre></code></blockquote> In an RPC context such an addition might be difficult, however, or may break some other principle of the RPC design. Basically, RPC is about marshalling data, and that means getting data out of the context of one server and forcing them into the context of another server. The "unity of data and operations", one of the OO principles, is intentionally given up. <p>Note that ICE allows to define operations for classes, and operations are always executed in the context of the data. In this example, it would be in deed possible to define <code>display</code> in a reasonable way, and to avoid the downcast. However, <code>display</code> then becomes part of the protocol, although it is rather a detail of the client. Anyway, one quickly faces the situation where downcasting is unavoidable. </p><p>In the O'Caml mapping generated by Hydro, these three classes would appear like </p><blockquote lang="x-ocaml"> <code><pre> class type o_SearchResult = object inherit o_Ice_Object method url : string ref method title : string ref end class type o_PeopleSearchResult = object inherit o_SearchResult method name : string ref end class type o_BandSearchResult = object inherit o_SearchResult method bandName : string ref method bandMembers : string array ref end val as_SearchResult : #Hydro_lm.object_base -> o_SearchResult val as_PeopleSearchResult : #Hydro_lm.object_base -> o_PeopleSearchResult val as_BandSearchResult : #Hydro_lm.object_base -> o_BandSearchResult </pre></code></blockquote> This is a bit simplified, but shows the idea. The ICE classes are mapped to O'Caml classes with some hidden machinery. The data members appear as O'Caml methods returning references - the most direct translation of this concept. The class hierarchy corresponds to the hierarchy in ICE, so the O'Caml operator for upcasting, :>, can be directly used. The hidden machinery comes into play by inheriting from <code>o_Ice_Object</code>, the root of the ICE hierarchy, and by using <code>object_base</code>, an even smaller antecedent that defines the marshalling core. <p>The downcast operation is emulated by defining conversion functions for every class type: <code>as_PeopleSearchResult</code> checks whether the argument is a <code>PeopleSearchResult</code> in reality, and if so, casts it to this class type. If not, an exception is raised. </p><p>Of course, this emulation is a bit inconvenient, but this is mostly a problem of generating good code. From a user's perspecitve, there is not much difference between calling a generated conversion function, or using a built-in language operation. It makes, however, the whole generated code a lot more difficult to understand. </p><h2>The Story Continues</h2> Missing support for RPC middleware is one of biggest concerns when using a new language in enterprises. In a startup company like Wink it is possible to address these concerns, because such companies are open for unconventional solutions (and ICE is unconventional - the industry standards are CORBA and DCOM in the LAN, and HTTP-based protocols like SOAP in the Internet). Finally, by integrating several languages into the system it was possible to deliver some components quicker and with better quality because we could choose the best language for every component. <p>In the people searcher context we use now both SunRPC and ICE. The former is arguably better when only O'Caml components have to talk with each other, and the latter is for crossing the language boundaries. </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> <p><b>Links:</b> <ul> <li><a href="http://wink.com">Wink Technologies</a>: Company homepage </li> <li><a href="http://www.zeroc.com">ZeroC</a>: Company homepage, ICE documentation, ICE implementation for a number of languages </li> <li><a href="http://remotetea.sourceforge.net">RemoteTea</a>: SunRPC for Java </li> <li><a href="http://oss.wink.com/hydro">Hydro</a>: ICE for O'Caml </li> <li><a href="http://projects.camlcity.org/projects/ocamlnet.html">Ocamlnet</a>: Internet library for O'Caml with SunRPC support </li> </ul> </p> </div> Mixing Apples And Pears http://blog.camlcity.org/blog/polyvariants.html http://blog.camlcity.org/blog/polyvariants.html <div> <b>Using Polymorphic Variants</b><br/>  </div> <div> It is one of the coolest language constructs, but its conception leads sometimes to confusion. O'Caml allows it to form ad-hoc unions of tagged values, the so-called polymorphic variants. They are the free-style counterpart of the "normal" variant types. We want to shed some light on this construction in this article, and encourage programmers to try it out. </div> <div> <p> The most baffling property of the polyvariants is that one can mix tags that come from different pieces of code. We'll give an example of that later in the text, but first let's explain some foundations. Syntactically, the tags are written with a leading reverse apostrophe, so for example <code>`Apple</code> is a (value-less) tag. Like the normal variants the tags can have attached values, so for instance <code>`Pear "it's sweet man"</code> is a tag with a string value. It is not required to declare polyvariant types, one can simply start creating such tagged values in the code. </p><h2>Data Analysis Step-By-Step</h2> <p>Imagine we want to analyze a string. In a first step, we would like to classify every character of the string, and determine whether it is a letter, a digit, or something else. Using polyvariants, this function does the job: </p><blockquote lang="x-ocaml"> <code><pre> let classify_chars s = let rec classify_chars_at p = if p < String.length s then let c = s.[p] in let cls = match c with | '0'..'9' -> `Digit c | 'A'..'Z' | 'a'..'z' -> `Letter c | _ -> `Other c in cls :: classify_chars_at (p+1) else [] in classify_chars_at 0 </pre></code> </blockquote> <p>So this function would return this list for the input string "a56*": </p><blockquote lang="x-ocaml"> <code><pre>[ `Letter 'a'; `Digit '5'; `Digit '6'; `Other '*' ] </pre></code> </blockquote> <p>Note that there is no type declaration! If you enter this function into the O'Caml toploop, you see that its type is inferred like this: </p><blockquote lang="x-ocaml"> <code><pre> val classify_chars : string -> [> `Digit of char | `Letter of char | `Other of char ] list = <fun> </pre></code> </blockquote> <p>Read this as: The function returns tagged values with tags <code>`Digit</code>, <code>`Letter</code>, or <code>`Other</code>, and every tag has an attached character. Note the "greater than" sign at the beginning of the tag list. It means that this tag list is compatible with being mixed with completely unrelated tags. There are also polyvariant types where this sign is reversed (like in <code>[<...]</code>), or completely missing. We'll come back to that later. </p><p>Back to our string analysis example. We are now interested in recognizing integer numbers in the list of classified characters. We assume our input is a list that contains <code>`Digit</code> tags, but also other tags. In the output list, sequences of <code>`Digit</code> are replaced by <code>`Number</code>: </p><blockquote lang="x-ocaml"> <code><pre> let recognize_numbers l = let rec recognize_at m acc = match m with | `Digit d :: m' -> let d_v = Char.code d - Char.code '0' in let acc' = match acc with | Some v -> Some(10*v + d_v) | None -> Some d_v in recognize_at m' acc' | x :: m' -> ( match acc with | None -> x :: recognize_at m' None | Some v -> (`Number v) :: x :: recognize_at m' None ) | [] -> ( match acc with | None -> [] | Some v -> (`Number v) :: [] ) in recognize_at l None </pre></code> </blockquote> <p>The inferred type of this function is now really strange: </p><blockquote lang="x-ocaml"> <code><pre> val recognize_numbers : ([> `Digit of char | `Number of int ] as 'a) list -> 'a list = <fun> </fun></pre></code> </blockquote> <p>Basically, it says that there is a tagged list as input, and that the output list has the same tags. Furthermore, the ">" sign again signals extensibility, so we can not only use the tags mentioned in the function, but any other tag as well. Especially, we are free to pass <code>`Letter</code> and <code>`Other</code> tags in: </p><blockquote lang="x-ocaml"> <code><pre> recognize_numbers [ `Digit '1' ; `Digit '3'; `Letter 'a'; `Digit '2'; `Other '*' ] </pre></code> yields <code><pre> [ `Number 13; `Letter 'a'; `Number 2; `Other '*'] </pre></code> </blockquote> <p>Note that the type of the <code>recognize_numbers</code> function does not reflect all what we could know about the function. We can be sure that the function will never return a <code>`Digit</code> tag, but this is not expressed in the function type. We have run into one of the cases where the O'Caml type system is not powerful enough to find this out, or even to write this knowledge down. In practice, this is no real limitation - the types are usually a bit weaker than necessary, but it is unlikely that weaker types cause problems. </p><p>The really great thing about the polymorphic variants is that it is possible to mix tags that come from different contexts. So in our example we can combine the two functions, and apply one after the other: </p><blockquote lang="x-ocaml"> <code><pre> let analyze s = recognize_numbers (classify_chars s) </pre></code> </blockquote> <p>It is no problem that <code>classify_chars</code> emits tags that are completely unknown to <code>recognize_numbers</code>. And both functions can use the same tag, <code>`Digit</code>, without having to declare in some way that they are meaning the same. It is sufficient that the tag is the same, and that the attached value has the same type. </p><p>This may have big advantages for structuring programs. Of course, our example of string analysis already benefits from the loose type correspondence the polyvariants make possible. The problem can now be divided into several steps, and every step needs only to know the tags it operates on. There is no global type all steps have to agree upon, rather every step sees only the fraction of type information that is needed for the task. </p><h2>Limiting Tags</h2> <p>Compare these two functions: </p><blockquote lang="x-ocaml"> <code><pre> let number_value1 t = match t with | `Number n -> n | `Digit d -> Char.code d - Char.code '0' let number_value2 t = match t with | `Number n -> n | `Digit d -> Char.code d - Char.code '0' | _ -> failwith "This is not a number" </pre></code> </blockquote> <p>The difference is that the second version explicitly catches the case of "any other tag" whereas the first version leaves this unspecified. This leads to different typings: </p><blockquote lang="x-ocaml"> <code><pre> val number_value1 : [< `Digit of char | `Number of int ] -> int val number_value2 : [> `Digit of char | `Number of int ] -> int </pre></code> </blockquote> <p>So in the first version we have a "<" sign! This sign usually only appears for input arguments, and means that the function can only process these tags (or less tags), but no other tags. The type checker prevents that any other tag can be passed in: </p><blockquote lang="x-ocaml"> <code><pre> # number_value1 (`Letter 'a');; This expression has type [> `Letter of char ] but is here used with type [< `Digit of char | `Number of int ] </pre></code> </blockquote> <p>In the second version of the function, this case is handled at runtime. From a typing perspective, the ">" sign signals that the function can also accept other tags than the mentioned ones. However, this function actually raises an exception. </p><blockquote lang="x-ocaml"> <code><pre> # number_value2 (`Letter 'a');; Exception: Failure "This is not a number". </pre></code> </blockquote> <p>So the "<" sign is a way to limit the number of tags a function can process. The programmer gets it by not adding a "catch all" case to pattern matchings. This kind of polyvariant is useful to enforce some strictness in programming, and the type checker catches the cases that would otherwise only be handed at runtime. </p><h2>Giving Polyvariants Names</h2> <p> Although the mantra of this article is that we don't need declarations, it is of course possible to define named polymorphic variants. For example, we could introduce these named types: </p><blockquote lang="x-ocaml"> <code><pre> type classified_char = [ `Digit of char | `Letter of char | `Other of char ] type number_token = [ `Digit of char | `Number of int ] </pre></code> </blockquote> <p>Note that there is no ">" or "<" sign in such definitions - it would not make sense to say something about whether more or less tags are possible than given, because the context is missing. </p><p>Using these names, one could simplify the typings of our functions: </p><blockquote lang="x-ocaml"> <code><pre> val classify_chars : string -> [> classified_char ] val recognize_numbers : ( [> number_token ] as 'a) list -> 'a list val number_value1 : [< number_token ] -> int val number_value2 : [> number_token ] -> int </pre></code> </blockquote> <p>As you see, the ">" or "<" sign can still appear in function types using these names. Actually, there is a special syntax behind this notation. If you just say <code>number_token</code> in a type expression, exactly the definition applies (without sign). But you can also construct new polyvariants from existing ones. For example, </p><blockquote lang="x-ocaml"> <code><pre>[ classified_char | number_token ] </pre></code> </blockquote> <p>would mean a type that combines the tags of both types, i.e. this is the same as if all four tags were enumerated. The syntax <code>[<number_token ]</code> is only a special case of this type constructor, where the new type also gets a sign. </p><h2>Matching Variants</h2> <p>Try to compile this piece of code that ought to sum up all <code>`Digit</code> and <code>`Number</code> tags of a list of any tags: </p><blockquote lang="x-ocaml"> <code><pre> let rec sum l = match l with | x :: l' -> ( match x with | `Digit _ | `Number _ -> number_value1 x + sum l' | _ -> sum l' ) | [] -> 0 </pre></code> </blockquote> <p>It compiles, but there is a little surprise. The compiler emits a warning that the <code>_ -> sum l'</code> case of the matching is unused, and the inferred type is just </p><blockquote lang="x-ocaml"> <code><pre> val sum : [ `Digit of char | `Number of int ] list -> int </pre></code> </blockquote> i.e. the ">" sign is missing that would allow us to pass any tags in. What's wrong? <p>This is a pitfall one quickly runs into when using polyvariants. The type checker assumes that the <code>x</code> in <code>number_value1 x</code> has the same type as the <code>x</code> that is matched again. It is not sufficient to use a matched variable in the expression of the matched case to restrict its type. Fortunately, there is a special syntax for that (look at the bold "as" clause): </p><blockquote lang="x-ocaml"> <code><pre> let rec sum l = match l with | x :: l' -> ( match x with | (`Digit _ | `Number _) <b>as y</b> -> number_value1 <b>y</b> + sum l' | _ -> sum l' ) | [] -> 0 </pre></code> </blockquote> <p>Here, <code>y</code> is the value <code>x</code> for the case that the match applies, so <code>y</code> can have a stricter type than <code>x</code>. </p><p>Alternatively, the match condition could also have been written as: </p><blockquote lang="x-ocaml"> <code><pre> match x with | #number_token as y -> ... </pre></code> </blockquote> i.e. one can use named polyvariants in matchings. This is just a convenience notation for the former. <h2>Behind The Scene</h2> <p>The polyvariants might be cool, but many programmers suspect that the performance of their programs suffer when they use them. Well, although there are some runtime cost, these are very small, and not noticeable for many programs. </p><p>Internally, the tags are represented by hash values of the names of the tags. So the tags are simply reduced to integers at runtime. Compared with the normal variant types, there is some additional overhead for tags with values. In particular, for storing <code>`X value</code> one extra word is needed in comparison with the normal variant <code>X value</code>. </p><p>It is possible that the hash values of variants collide, e.g. </p><blockquote lang="x-ocaml"> <code><pre> # type x = [ `jagJhn | `oZshTt ];; Variant tags `jagJhn and `oZshTt have same hash value. Change one of them. </pre></code> </blockquote> As you see, the compiler checks for this rare case. I've never seen it in practice. <p>Despite rumours, there is nothing special done at link time. The tags are already cut down to integers at this point of compiling. </p><h2>Conclusion</h2> <p>I hope I've shown how elegant code looks that uses polyvariants to represent data cases. But there is some more to say, especially if you look at other programming languages. </p><p>The author of this article thinks that polymorphic variants are one of the features that makes O'Caml so different in comparison with mainstream languages like Java. In particular, there are competing approaches for representing data cases, and one of the radical ideas of object orientation has always been that combining data and program cases into a single class construct is the best way to deal with the problem. However, I think something has been overlooked - data and algorithms do not always walk hand in hand, and classes are inflexible if only a loose correlation between both is needed. In contrast, polyvariants give the programmer the maximum of freedom in this respect. Thus it is believed that polyvariants are a serious alternative for representing data cases. </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> <p><b>Links:</b> <ul> <li><a href="http://caml.inria.fr/pub/docs/manual-ocaml/manual006.html#htoc41">Polymorphic variants</a>: Chapter in the O'Caml manual </li> </ul> </p> </div>