A Key Point: Scalability

Maintaining Causality

In the context of a distributed simulation, should no special mechanism be used, the propagation of simulated events would depend on the propagation of messages in the actual network of computing nodes, which offers no guarantee in terms of respective timing and therefore order of reception.

As the minimal example below [5] shows, a simulation cannot work correctly as such:

There are three simulation actors here, which are supposed to be instantiated each on a different computing node. Thus, when they communicate, they exchange messages over the network on which the distributed simulator is running.

Actor #1 is a rocket launcher that fires to actor #2, which is a tank. Thus actor #1 sends a message, M1, to all simulation actors that could be interested by this fact (including actor #2 and actor #3), to notify them of the rocket launch.

Here, in this technical context (computers and network), actor #2 (the tank) happens to receive M1 before actor #3 (the observer).

According to its model, the tank, when hit by a rocket, must explode. Therefore it sends a message (M2) to relevant actors (among which there is the observer), to notify them it exploded and that the corresponding technical actor is removed from the simulation.

The problem lies in the point of view of actor #3. Indeed in that case the observer received:

1. M2, which told it that a tank exploded for no reason (unexpected behaviour)
2. then M1, which tells it that a rocket was fired to a simulation actor that actually does not even exist

This situation makes absolutely no sense for this actor #3. At best, the model of the observer should detect the inconsistency and stop the simulation. At worse, the actor received incorrect inputs, and in turn injected incorrect outputs in a simulation that should not be trusted anymore.

The root of the problem is that here there is no guarantee that received messages will respect their timely constraints - whereas (at least in synchronous approaches) no return to the past shall be considered, however tempting.

This faulty behaviour would be all the more unfortunate that the incorrect outputs are likely to be indistinguishable from correct ones (i.e. they can go unnoticed in the simulation), distorting the results invisibly, a bit like a pocket calculator which would silently ignore parentheses, and would nevertheless output results that look correct, but are not.

Maintaining Reproducibility

Let's suppose for now we somehow managed to preserve causality. This does not imply that reproducibility is ensured.

Using the same example where actor #1 launches a rocket (sending the M1 message), actor #3 can in the meantime develop its own behaviour, which may imply this observer detected the tank. This can lead the observer notifying the tank, thus to its sending the M3 message.

The point here is that there is no direct nor causal relationship between M1 and M3. These are truly concurrent events, they may actually happen in any order. Therefore concurrent events are not expected to be reordered by the mechanism used to maintain causality, since situations A and B are equally correct.

However, when the user runs twice exactly the same simulation, she most probably expects to obtain the same result [6]: here M1 should be received by actor #2 always before M3, or M3 always before M1, and the implicit race condition should not exist in that context.

In that case, causality is not enough, some additional measures have to be taken to obtain reproducibility as well.

With some time management approaches, once causality is restored, ensuring reproducibility is only a matter of enforcing an arbitrary order (i.e. which depends only on these messages, not in any way on the context) on concurrent events.

Allowing For Ergodicity

The context-free message reordering allows to recreate the arbitrary order we need to ensure reproducibility.

However the simulator should offer the possibility to go beyond this mechanism, otherwise "ergodicity" (a term we chose in reference to Monte-Carlo computations) cannot be achieved: in some cases we want all combinations of concurrent events to be able to occur, not only the ones that correspond to the arbitrary order we enforced.

The best solution we know here is, in a time-stepped context, to let the reproducibility mechanism activated, but, in addition to the sorting into an arbitrary order, to perform then an uniform random shuffle: then we are able not only to recreate all licit combinations of events during a given simulation tick at the level of each actor, but also to ensure that all these combinations have exactly the same probability of showing up.

Approach A: use of a centralised queue of events

A unique centralised queue of simulation events is maintained, events are sorted chronologically, and processed one after the other.

Pros:

• purely sequential, incredibly simple to implement

Cons:

• splendid bottleneck, not able to scale at all, no concurrent processing generally feasible, distribution not really usable there; would be painfully slow on most platforms as soon as more than a few hundreds of models are involved

Approach B: use a time-stepped approach

The simulation time is chopped in intervals short enough to be able to consider the system as a whole as constant during a time step, and the simulator iterates through time-steps.

Pros:

• still relatively simple to understand and implement
• may allow for a massive, yet rather effective, parallelization of the evaluation of model instances
• the simulation engine may be able to automatically jump over ticks that are identified as being necessarily idle, gaining most of the advantages of event-driven simulations
• the resulting simulator can work in batch mode or in interactive mode with very small effort, and with no real loss regarding best achievable performance

Cons:

• not strictly as simple as one could think, but doable (e.g. reordering of events must be managed, management of stochastic values must be properly thought of, induced latency may either add some constraints to the implementation of models or require a more complex approach to time management)
• a value of the time step must be chosen appropriately (although we could imagine that advanced engines could determine it automatically, based on the needs expressed by each model)

Approach C: use a conservative event-driven approach

The simulation time will not advance until all model instances know for sure they will never receive a message from the past.

Pros:

• efficient for systems with only few events occurring over long periods
• there must be other advantages (other than the fact it is still a field of actual academic research) that I overlooked or that do not apply to the case discussed here

Cons:

• complex algorithms are needed: it is proven that this mechanism, in the general case, leads to deadlocks. Thus a mechanism to detect them, and another one to overcome them, must be implemented
• deadlock management and attempts of avoidance induce a lot of null (empty) messages to be exchanged to ensure that timestamps make progress, and this generally implies a significant waste of bandwidth (thus slowing down the whole simulation)

Approach D: use an optimistic event-driven approach

For each model instance (actor), simulation time will advance carelessly, i.e. disregarding the fact that other model instances might not have reached that point in time yet.

Obviously it may lead to desynchronised times across actors, but when such an actor receives a message from its past, it will rewind its logic and state in time and restart from that past. The problem is that it will likely have sent messages to other actors in-between, so it will have to send anti-messages that will lead to cascading rewinds and anti-messages...

Pros:

• efficient in some very specific situations where actors tend to nevertheless advance spontaneously at the same pace, thus minimising the number of messages in the past received (not the case here, I think)
• there must be other advantages (other than the fact it is still a field of actual academic research) that I overlooked or that do not apply to the case discussed here

Cons:

• overcoming messages in the past implies developing a generic algorithm allowing for distributed roll-backs over a graph of model instances. This is one of the most complex algorithm I know and for sure I would not dare to implement and validate it, except maybe in a research-oriented project
• even when correctly implemented, each simulation roll-back is likely to be a real performance killer

General Principles

Simplified Mode of Operation

A time step will be generally mentioned here as a simulation tick.

Sim-Diasca uses a special technical component - a process with a specific role - which is called the Time Manager and acts as the simulation scheduler.

It will be the sole controller of the overall simulation time. Once created, it is notably given:

• a simulation start time, for example: Thursday, November 13, 2008 at 3:19:00 PM, from which the initial simulation tick will be deduced
• an operating frequency, for example: 50 Hz, which means each virtual second will be split in 50 periods, with therefore a (constant) simulation tick whose duration - in virtual time - will be 1000/50 = 20 ms; this time step must be chosen appropriately, depending on the system to simulate [8]
• an operating mode, i.e. batch or interactive

In batch mode, the simulation will run as fast as possible, whereas in interactive mode, the simulator will be kept on par with the user (wall-clock) time. If the simulation fails to do so (i.e. if it cannot run fast enough), the user is notified of the scheduling failure, and the simulator tries to keep on track, on a best-effort basis.

Not depending on the operating mode, when started the Time Manager will always follow the same algorithm, shown below:

At the beginning of a new tick, the Time Manager will notify all subscribed simulation actors that a new tick began, thanks to a top message.

Each actor will then process all the actor messages it received during the last tick, reordered appropriately, as explained in the Maintaining Causality and Maintaining Reproducibility sections. This deferred message processing ensures the simulation time always progresses forward, which is a property that simplifies considerably the time management.

Processing these actor messages may imply state changes in that actor and/or the sending of actor messages to other actors.

Once all past messages have been processed, the actor will go on, and act according to its spontaneous behaviour. Then again, this may imply state changes in that actor and/or the sending of actor messages to other actors.

Finally, the actor reports to the time manager that it finished its simulation tick, thanks to a done message.

The key point is that all actors can go through these operations concurrently, i.e. there is no limit on the number of actors that can process their tick simultaneously.

Therefore each simulation tick will not last longer than needed, since the time manager will determine that the tick is over as soon as the last actor reported it has finished processing its tick.

More precisely, here each simulation tick will last no longer than the duration took by the actor needing the most time to process its tick, compared to a centralised approach where it would last as long as the sum of all the durations needed by each actor. This is a tremendous speed-up indeed.

Then the time manager will determine that the time for the next tick has come.

Actual Mode of Operation

For the sake of clarity, the previous description relied on quite a few simplifications, that are detailed here.

Actual Fine-Grain Scheduling

The simulation time is discretised into fundamental time steps (ticks, which are positive, unbounded integers) of equal duration in virtual time (e.g. 10ms, for a simulation running at 100 Hz) that are increased monotonically.

From the user-specified simulation start date (e.g. Monday, March 10, 2014 at 3:15:36 PM), a simulation initial tick Tinitial is defined (e.g. Tinitial = 6311390400000).

Tick offsets can be used instead of absolute ticks; these offsets are defined as a difference between two ticks, and represent a duration (e.g. at 100Hz, Toffset=15000 will correspond to a duration of 2 minutes and 30 seconds in virtual time).

Internally, actors use mostly tick offsets defined relatively to the simulation initial tick.

During a tick T, any number of logical moments (diascas, which are positive, unbounded integers) can elapse. Each tick starts at diasca D=0, and as many increasing diascas as needed are created to solve causality.

All diascas of a tick occur at the same simulation timestamp (which is this tick), they solely represent logical moments into this tick, linked by an "happen before" relationship: if two events E1 and E2 happen respectively at D1 and D2 (both at the tick T), and if D1 < D2, then D1 happened before D2.

So the full timestamp for an event is a Tick-Diasca pair, typically noted as {T,D}.

Diascas allows to manage causality despite parallelism: effects will always happen strictly later than their cause, i.e. at the very least on the diasca immediately following the one of the cause, if not in later diascas or even ticks, depending on the intended timing of the causal mechanism: causes can follow effects either immediately or after any specified duration in virtual time [9].

[9]

Durations shall been specified by modellers regardless of a simulation frequency, in absolute terms (e.g. "6 minutes and 20 seconds"), rather than directly as a number of ticks: the engine is indeed able to convert the former to the latter at runtime, and to stop automatically if the conversion resulted in a rounding error higher than a threshold (either the default one, or a user-specified one for that duration). As much as possible, models should be uncoupled from the simulation frequency.

This is what happens when an actor A1 sends a message to an actor A2 at tick T, diasca D (i.e. at {T,D}). A2 will process this message at {T,D+1}. If needing to send back an answer, it may do it directly (while still at {T,D+1}), and A1 will be able to process it at {T,D+2}.

This allows immediate exchanges in virtual time (we are still at tick T - and A2 could have similarly triggered any number of third parties before answering to A1, simply resulting in an increase of the current diasca), while still being massively parallel and preserving causality and other expected simulation properties. Of course non-immediate exchanges are also easily done, since A2 may wait for any number of ticks before sending its answer to A1.

Simulation currently stalled at tick #3168318240271, waiting for following actor(s): [<0.50.0>,<0.57.0>].
Current tick not ended yet because:
 + actor <0.50.0> is waiting for an acknowledgement from [<0.1.0>]
 + actor <0.57.0> is waiting for an acknowledgement from [<0.1.0>,<0.38.0>]

case CurrentTick of

  ActionTick ->

      do_action();
      ...

Implementing an Actor

Each model must inherit, directly or not, from the actor class (class_Actor). As a consequence, its constructor has to call the one of at least one mother class.

Each constructor should start by calling the constructors of each direct parent class, preferably in the order in which they were specified; a good practice is to place the model-specific code of the constructor after the call to these constructors (not before, not between them).

An actor determines its own scheduling by calling oneways [12] helper functions offered by the engine (they are defined in the class_Actor module):

• addSpontaneousTick/2 and addSpontaneousTicks/2, to declare additional ticks at which this instance requires to develop a future spontaneous behaviour (at their diasca 0)
• withdrawnSpontaneousTick/2 and withdrawnSpontaneousTicks/2, to withdraw ticks that were previously declared for a spontaneous behaviour but are not wanted anymore
• declareTermination/1, to trigger the termination of this actor

An actor is to call these helper functions from its actSpontaneous/1 oneway or any of its actor oneways. This includes its onFirstDiasca/2 actor oneway, which is called as soon as this actor joined the simulation, so that it can define at start-up what it intends to do next, possibly directly at this very diasca (no need for example to wait for the first next tick).

Even if actors are evaluated in parallel, the code of each actor is purely sequential (as any other Erlang process). Hence writing a behaviour of an actor is usually quite straightforward, as it is mostly a matter of:

• updating the internal state of that actor, based on the changes operated on the value of its attributes (which is an immediate operation)
• sending actor message(s) to other actors (whose processing will happen at the next diasca)

On each tick the engine will automatically instantiate as many diascas as needed, based on the past sending of actor messages and on the management of the life cycle of the instances.

So the model writer should consider diascas to be opaque values that just represent the "happened before" relationship, to account for causality; thanks to these logical moments which occur during the same slot of simulated time, effects always happen strictly after their causes.

As a consequence, the model writer should not base a behaviour onto a specific diasca (e.g. "at diasca 7, do this"); the model should send information to other instances or request updates from them (in both cases thanks to other messages) instead.

So, for example, if an actor asks another for a piece for information, it should just expect that, in a later diasca (or even tick, depending on the timeliness of the interaction), the target actor will send it a message back with this information.

The point is that if the model-level protocol implies that a target actor is expected to send back an answer, it must do so, but at any later, unspecified time; not necessarily exactly two diascas after the request was sent: we are in the context of asynchronous message passing.

This allows for example an actor to forward the request to another, to fetch information from other actors, or simply to wait the duration needed (in virtual time) to account for any modelled processing time for that request (e.g. "travelling from A to B shall last for 17 minutes").

When actor decides it is to leave the simulation and be deleted, it has to ensure that:

• it has withdrawn all the future spontaneous ticks it may have already declared
• it calls its declareTermination/{1,2} oneway (or the class_Actor:declare_termination/{1,2} helper function)

The actor must ensure that no other actor will ever try to interact with it once it will have terminated, possibly using its deferred termination procedure to notify these actors that from now they should "forget" it.

Please refer to the Sim-Diasca Developer Guide for more information.

Latest Enhancements

These evolutions have been implemented for the 2.0.x versions of Sim-Diasca, starting from 2009.

Distributed Time Manager

The Time Manager was described as a centralised actor, but actually, for increased performances, the time management service is fully distributed, thanks to a hierarchy of time managers.

By default there is exactly one time manager per computing host, federating in an Erlang node all cores of all its processors and evaluating all the actors that are local to this host (and only them).

One of these time managers is selected (by the deployment manager) as the root time manager, to be the one in charge of the control of the virtual time. The other time managers are then its direct children (so the height of the scheduling tree is by default equal to 1).

Other kinds of trees could be chosen: they might be unbalanced, have a different heights (e.g. to account for multiple clusters/processors/cores), etc., as shown in the physical diagram below:

The overall objective is to better make use of the computing architecture and also to minimize the induced latency, which is of paramount importance for synchronous simulations (we want to be able to go through potentially short and numerous time-steps as fast as possible).

Two protocols are involved in terms of scheduling exchanges, as shown in the logical diagram:

• one for higher-level synchronisation, between time managers
• another for lower-level actor scheduling, between a local time manager and the actors it drives

Of course there will be many more actors than displayed on the diagram created on each computing node (typically dozens of thousands of them), therefore a lot of scheduling messages will be exchanged between these actors and their local time manager instance.

The point is that these (potentially numerous) messages will incur as little overhead as possible, since they will be exchanged inside the same computing node: only very few scheduling messages will have to cross the node boundaries, i.e. to be conveyed by the bandwidth-constrained network. We trade the number of messages (more numerous then) for their network cost, which is certainly a good operation.

The load balancer has to be an actor as well (the only special one, created at start-up), since, when the simulation is running, it must be able to enforce a consistent order in the actor creations, which, inside a time step, implies the use of the same message reordering as for other actors.

In the special case of the simulation set-up, during which the initial state of the target system is to be defined, the initial actors have to be created, before the simulation clock is started. Only one process (generally, directly the one corresponding to the simulation case being run; otherwise the one of a scenario) is expected to create these initial actors. Therefore there is no reordering issues here [13].

[13]	However race conditions must be avoided there (between creations and also with the simulation start), this is why all initial creations are by design synchronous.

Virtual Time Versus Real Time

In batch mode, the time of the simulation (a.k.a. virtual time) is fully decorrelated from the user, wall-clock time: the engine will run as fast as possible, but will take as long as needed to fully evaluate simulated events. As a consequence, depending on the computer resources that are used, the resulting simulation might be faster or slower than real-time.

In interactive mode, provided that the hardware is able to run the simulation faster than the real time (possibly once a user-specified scaling factor has been applied), the engine will perform the necessary waiting so that the virtual time stays on par with the real time, allowing for possible third-party interactions with the simulation (actual devices, humans in the loop, etc.).

Quantification & Timings

The virtual time is quantised, i.e. chunked into slices of equal durations (this is why Sim-Diasca is a discrete synchronous simulation engine). These periods are called simulation ticks (often shorten as ticks here).

For example, if the user specifies a tick duration of 20ms, then all durations will be multiples of 20ms (possibly 0ms - instant actions are supported) and the simulation will run at 50Hz.

Note that, for the models to be as much as possible uncoupled from the simulation frequency, they should express their timings in terms of actual "absolute" durations (e.g. 100ms), rather than in terms of a number of ticks (e.g. 5 ticks, at 50Hz).

This way, changing the simulation frequency (e.g. setting it to 100Hz) will not imply that all models need to have their internal timing settings updated accordingly; indeed the engine is able to convert at runtime such actual durations into the relevant number of ticks, and will automatically ensure that the relative quantification error that is then induced stays below a threshold (either the default one or a user-defined one). If the quantification error is too high, the simulation will just report it and stop.

Spontaneous Behaviours & Triggered Actions

At its creation (whether initial or in the course of the simulation), each actor is scheduled once and is able to tell the engine at which tick (if any) it plans to develop its next spontaneous behaviour.

The engine (actually, the root time manager) is then able to know for each new tick whether it should be scheduled (this will happen iff at least one actor planned a spontaneous action for that tick). This knowledge will then allow the simulation to automatically jump over ticks that are known to be idle (i.e. with no actor to schedule). This feautre allows such a synchronous engine to provide features that are quite similar to the ones provided by asynchronous engines.

However, such a simulation would be useless if the actors could not interact with other actors: during a tick, any actor is able to trigger actions on any other actor.

Causality & Diasca

To ensure, even in a parallel and/or distributed setting, that no cause can happen after its effects, the model instances can only communicate through the exchange of actor messages.

These messages can be seen as method calls which are properly intercepted, reordered and actually triggered by the engine.

For that the notion of diasca is introduced: each tick is evaluated as a series of diascas (starting at diasca D=0), each of them being a logical moment inside that tick.

The purpose of diascas is to split each tick so that the consequence of an event (an actor message is sent at diasca D) happens at the next diasca (the actor message is processed by its recipient at diasca D+1).

These triggered actions may in turn result in new actor messages being sent, resulting in as many diascas as needed (D+2, D+3, etc.) being instantiated. The current tick will terminate only when no more diascas are requested, i.e. when there is no more immediate action declared. Then the next tick will be scheduled, and will start from diasca 0 again.

As such, diascas do not correspond to any specific duration within a tick (which is by design the finest observable simulated duration): their purpose is just to allow to determine that some events happen before others, i.e. to maintain causality.

Next Steps: Time Generalization

We plan to support the following time-related generalizations:

• to provide more flexibility and granularity, ticks could be of a parametrised type, i.e. a user-decided, simulation-specific type (e.g. they could be floats instead of being integers), provided that they respect a (simple) contract specified by the engine; see class_TimeManager:tick() for more details
• to allow for recursive, imbricated/nested actors (actors containing, or being made of actors) and time refinement, diascas could be tuples (e.g. {17,5,0}, {17,5,1}, etc. being between {17,5} and {17,6}, themselves being between 17 and 18); see class_TimeManager:diasca() for more details

Uniform Law

This "white noise" generator will draw values into a user-parameterised range, all samples having an equal probability of being chosen.

When placing a Sim-Diasca probe at the output of a RandomManager set to deliver a uniform law, we have the following result:

By setting the appropriate flag, Sim-Diasca can also be configured to use a uniform generator of superior quality, which was designed for cryptography.

This uniform distribution is often the basis to generate in turn more complex distributions.

Exponential Law

This distribution, defined by a single parameter, lambda, leads to the following result:

This distribution is directly generated by Sim-Diasca from the previous white noise source.

Gaussian Law

Two parameters, mu, the mean value, et sigma, the variance (whose positive square root is the standard deviation), define the Gaussian law [15].

This results in the following curve:

The Sim-Diasca white noise generator is used to generate this Gaussian law as well.

Extra Reliability-Oriented Laws

A total of about 20 random laws are supported (uniform, exponential, gaussian, Gamma, Weibull, Gumbell, etc.; this includes all distributions defined from here), some with a few variations (with integer samples rather than floating-point ones, yiedling only positive values, etc.), each defined based on a set of parameters.

Even if some laws (uniform, exponential, gaussian) may be obtained by a direct use of the per-actor embedded random generator, most are computed through inverse transform sampling (precisely the alias method, once the PDF has been discretised).

These laws are typically considered by the Python Reliability library as candidate blueprints that may fit actual data: such libraries will attempt to determine the best choice in terms of function and associated parameters to match as closely as possible the input data.

Random Generators

At the core of most implementations, one relies on a random generator, which usually outputs floating-point values uniformly distributed between 0.0 and 1.0.

For a better stochastic management, the engine does not rely anymore on the basic random module of Erlang; it may operate instead with the crypto module (if available and enabled), otherwise it will default on the newer rand module, which offers various algorithms, including:

• exsplus: Xorshift116+, 58 bits precision and period of 2^116-1 (state uses 320 bytes on 64-bit platforms)
• exs64: Xorshift64*, 64 bits precision and a period of 2^64-1 (state of 336 bytes on 64-bit platforms)
• exs1024: Xorshift1024*, 64 bits precision and a period of 2^1024-1 (state of 856 bytes on 64-bit platforms)

Unless overridden (see random_utils.erl), the algorithm used by the engine is exsplus.

Based on a uniform random value in [0.0,1.0], one can generate uniform values in any other range, and values that respect all kinds of non-uniform laws (like the Gaussian one).

However a question still remains: how many instances of random generators should we use?

Mode Of Operation

Random generators usually have a state, which is initialised with a seed - either set by default, or specifically given.

From a seed a series of random numbers can be generated, and as such it can be reproduced identically, as long as the same seed is used.

The trouble comes from the fact that, during any given logical moment (diasca), multiple simulation actors may require - and therefore request from a random generator - any number of values complying to any number of probabilistic laws, each parameterised as wished, in any order. And of course we do not want to loose the reproducibility of simulations because of that.

Initially the engine was relying on a limited number of centralised random manager instances (possibly even just one), each used by a (potentially large) set of model instances.

Each of these model instances would then interact with its random manager(s) in a consistent manner, through actor messages to preserve simulation properties.

Such an approach induces many constraints, like the additional synchronisation and diasca creation involved (hence significant runtime overhead), a more complex model-level logic to request and wait for these values, etc.

How to Create Initial Instances Programmatically

This is the easiest, yet most limited, approach. No external identifier needs to be involved here.

If wanting to create two instances that have to know each other, one may use, possibly directly from the simulation case:

A = class_Actor:create_initial_actor( class_Foo, [ Xa, Ya ] ),
B = class_Actor:create_initial_actor( class_Bar, [ Xb, Yb, Zb ],
   "My placement hint" ),

A ! { declareBar, B, self() },
B ! { declareFoo, A, self() },

bar_declared = test_receive(),
foo_declared = test_receive(),

[...]

A few remarks here:

• A and B are PIDs
• this mutual recognition requires a specific method to be added in both classes (namely declare{Foo,Bar}/2)
• one can guess that the declare{Foo,Bar}/2 methods exposed here are requests (even if no particular result had to be returned), for synchronicity reasons (otherwise there could be a race condition between these messages and the ones related to the start of the simulation); this is why we have to receive the *_declared messages from the simulation case
• we can see that here A will be created using default placement policy, when B will be created by the load balancer according to the specified hint: when created programmatically, all instances specifying the same placement hint will be created on the same computing node (as chosen by the load balancer)

How to Use Initialisation Data to Create Initial Actors

{class_Foo,["Hello world!",1.4]}.

class_Actor:create_initial_actor(class_Foo,["Hello world!",1.4])

{class_Foo,["Hello world!",1.4],my_placement_hint}.

class_Actor:create_initial_placed_actor(class_Foo,
  ["Hello world!",1.4],my_placement_hint)

"My first instance" <- {class_Foo,["Hello world!",1.4]}.

{class_Bar,[an_atom,3,{user_id,"My first instance"},7]}.

{class_Dalek,[true,{user_id,"EXTERMINATE"}],"Milky Way"}.

"John" <- {class_Beatle,[{user_id,"Paul"},{user_id,"George"}]}.

General Mode of Operation

Sim-Diasca offers a fully concurrent support for simulation results (from their generation to their retrieval), so that they are efficiently produced (only the relevant ones, and in a massively parallel way) and automatically returned back to the user in its launching directory, knowing that on a distributed context the output data is by design scattered across the various networked computing nodes.

For that, if and if only the simulation finishes successfully, a directory specific to that simulation run (based on simulation name, time and date, user and a rather unique identifier [20]) is created in the current directory (i.e. the directory from which make My_Case_run was executed) of the user node; this result directory bears a name like:

Sim-Diasca_My_Case-on-2015-12-10-at-10h-05m-59s-by-boudevil-1f793a6ba507

Results of interest are automatically transferred there (otherwise one would have to log in on each and every computing node [21] to select and retrieve these results by hand).

Results are handled by:

• the overall Result Manager (a class_ResultManager singleton, automatically created at deployment time on the user node), which keeps track of results and drive them
• Result producers (all class_ResultProducer instances, like probes, either basic or datalogger-based), which provide support to declare, aggregate and send their results

Following mode of operation allows to handle results:

1. the simulation user specifies initially, in the simulation case, what are the results he is interested in, thanks to a Result Specification (detailed below)
2. the result manager checks this specification, and precomputes what will be needed to discriminate between wanted and unwanted simulation results
3. when the creation of result producers (typically probes instantiated from models or scenarios) is considered (either initially or at simulation-time), it is declared automatically to the result manager, which is then able to tell whether a given result producer is wanted or not (then only the necessary producers will be created and fed); this allows a given model to support any number of probes, and to enable only the relevant ones on a per-simulation case basis
4. in the course of the simulation, result producers gather sample data, performing immediate or deferred writes (their volume usually exceeds the capacities of the overall distributed RAM), potentially terminating at any time and then performing operations on these data (e.g. generating plots from them)
5. when (if) the simulation ends successfully, the result manager automatically requests the relevant results from all relevant producers, and copy them in the result directory (more precisely, it generates them only if appropriate, like in the case of plots, and send them in a compressed from over the network, to the user node)

Result Specification

Which results are wanted is generally specified directly from the simulation case, among the simulation settings.

The selection is based on the names of the result producers to enable, since it is the only way the user can refer to them a priori (e.g. of course no PID can be statically anticipated in the simulation case).

In the result specification, a simulation case may require:

• a binary selection: either all results are wanted or none of them
• a type-based selection: either the results corresponding to plain probes or to virtual ones (i.e. based on the data-logger) are wanted
• a finer pattern-based selection: only results whose name matches at least one targeted pattern, and none of the blacklisted patterns, are to be selected (most general case)

Patterns are to be expressed according to the Perl Compatible Regular Expressions conventions, or PCRE for short.

For more information on the pattern format, see the re module.

The detailed supported syntax is specified in the sim-diasca/src/core/src/scheduling/class_TimeManager.hrl header file; see the result_specification field of the simulation_settings record. Examples can be found in the cases available in the mock-simulators directory.

Relying on these simulation settings allows to define which results are expected statically, which is fine for most uses. However, under some circumstances, it may be convenient to set or modify the result specification dynamically (e.g. if it is difficult to anticipate on the name of a probe or whether it is actually wanted).

Thus result specification can be also modified at simulation-time, thanks to method calls (see the {add,remove,set}{Targeted,Blacklisted}Pattern*/2 methods of class_ResultManager).

Early Disabling of Results

All results could be generated in all cases, and only be retrieved if requested.

However a better approach could be to collect data samples and process them (e.g. in graphical plots) only if needed.

A still better approach is needed: as the result manager is able to tell directly whether a result is wanted, it will be able to disable unwanted results from the start, i.e. reject any attempt of creating a result producer (e.g. a probe) whose results are not wanted by the user.

As a consequence, a classical, model-level probe may be created thanks to the class_Probe:declare_result_probe/6 static method, which will return either the PID of this newly created probe (if the name of that probe is acknowledged as a wanted result by the result manager), or the non_wanted_probe atom.

Then the class_Probe:send_data/3 method can be called as often as needed by the model in order to potentially feed that probe with relevant sample data (the fact that this probe may not be enabled being then transparently managed).

Identifying Reasons For Observed Phenomena

Finding actual causes is seldom straightforward:

Having Reasonable Expectations

A simulation is not the silver bullet that will ask the right questions on the user's behalf and answer them with infinite accuracy:

Simulation being a rather expensive and time-consuming mode of evaluation, it should be used on carefully selected cases that cannot be solved satisfactorily thanks to other methods, like comparison with actual systems, expert assessments, coarse spreadsheet-based studies, etc.

Even in that case, a few well-selected metrics must be defined, that must be both helpful to the user and solvable by the simulation.

Extrapolating Results, Really?

Unless it has been proven separately, one cannot arbitrarily reduce the problem size and expect that a small-scale experiment will still provide reliable insights about a real-sized system: reductionism cannot be applied blindly.

This is why the scalability of a simulation engine is a key property: whenever smaller-scale experiments cannot be safely attempted (the general case), it offers a better chance of capturing the reality.

Indeed extrapolating becomes too often a wild guess:

In most cases, approaches based on extrapolations are hardly sustainable:

Sharing the Findings With the Intended Audience

The lessons learned thanks to the simulation must be synthesised appropriately, with proper wording for the targeted public, so that the conclusions are sufficiently emphasized to be well-understood:

Concerns must be correctly shared among the people involved, with appropriate common metrics and goals:

Making Good Use of the New Knowledge

It is certainly out of the scope of this document, but simulations may generate new knowledge, which must be carefully leveraged, lest it worsens the situation:

Preparing for any later recovery

Let's suppose that N computing hosts are assigned to a simulation having to exhibit a k-crash resiliency.

This resilience service is implemented internally by first establishing a "k-map", which determines, for each of the N hosts, which of the k other hosts it backs-up and, reciprocally, which hosts back it up.

For example, if N=6, hosts may be [a,b,c,d,e,f], and the k-map could then be, for a requested resilience level of k=5:

For a resilience level of 5, result is: k-map for 6 nodes:
+ for node a:
 - securing nodes [b,c,d,e,f]
 - being secured by nodes [f,e,d,c,b]

+ for node b:
 - securing nodes [c,d,e,f,a]
 - being secured by nodes [f,e,d,c,a]

+ for node c:
 - securing nodes [d,e,f,a,b]
 - being secured by nodes [f,e,d,b,a]

+ for node d:
 - securing nodes [e,f,a,b,c]
 - being secured by nodes [f,e,c,b,a]

+ for node e:
 - securing nodes [f,a,b,c,d]
 - being secured by nodes [f,d,c,b,a]

+ for node f:
 - securing nodes [a,b,c,d,e]
 - being secured by nodes [e,d,c,b,a]

This example corresponds to, graphically (see class_Resilience_test.erl):

Of course this resilience feature is typically to be used with a far larger number of nodes; even with a slight increase, like in:

we see that any central point in the process would become very quickly a massive bottleneck.

This is why the actual work (both for serialisation and deserialisation tasks) is done in a purely distributed way, and exchanges are done in a peer-to-peer fashion, using the fastest available I/O for that [26], while the bulk of the data-intensive local work is mostly done in parallel (taking advantages of all local cores).

To ensure a balanced load, each computing host is in charge of exactly k other hosts, while reciprocally k other hosts are in charge of this host. After failures, the k-map is recomputed accordingly, and all relevant instances are restored, both in terms of state and connectivity (yet, in the general case, on a different computing host), based on the serialisation work done during the last simulation milestone.

Actual Course of Action

Setting up the resilience service is a part of the deployment phase of the engine. Then the simulation is started and, whenever a serialisation wall-clock time milestone is reached, each computing host disables the simulation watchdog, collects and transforms the state of its simulation agents, actors and result producers (including their currently written data files), and creates a compressed, binary archive from that.

Typically, such an archive would be a serialisation-2503-17-from-tesla.bin file, for a host named tesla.foobar.org, for a serialisation happening at the end of tick offset 2503, diasca 17. It would be written in the resilience-snapshots sub-directory of the local temporary simulation (for example in the default /tmp/sim-diasca-<CASE NAME>-<USER>-<TIMESTAMP>-<ID>/ directory).

This archive is then directly sent to the k other hosts (as specified by the current version of the k-map), while receiving reciprocally the same type of information from k other hosts. One should note that this operation, which is distributed by nature, is also intensely done in parallel (i.e. on all hosts, all cores are used to transform the state of local instances into a serialised form, and the two-way transfers themselves are made in parallel).

Then, as long as up to k hosts fail, the simulation can still rely on a snapshot for the last met milestone, and restart from it (provided the remaining hosts are powerful enough to support the whole simulation by themselves).

The states then collected require more than a mere serialisation, as some elements are technical information that must be specifically handled.

This is notably the case for the PIDs that are stored in the state of an instance (i.e. in the value of an attribute, just by itself or possibly as a part of an arbitrarily complex data-structure).

Either such a PID belongs to a lower layer (Myriad, WOOPER or Traces), or it is related directly to Sim-Diasca, corresponding typically to a simulation agent of a distributed service (e.g. a local time manager, data exchanger or instance tracker), to a model instance (an actor) or to a result producer (a probe).

As PIDs are technical, contextual, non-reproducible identifiers (somewhat akin to pointers), they must be translated into a more abstract form prior to serialisation, to allow for a later proper deserialisation; otherwise these "pointers" would not mean anything for the deserialising mechanism:

• Lower layers are special-cased (we have mostly to deal with the WOOPER class manager and the trace aggregator)
• Simulation agents are identified by agent_ref (specifying the service they implement and the node on which they used to operate)
• Model instances are identified by their AAI (Abstract Actor Identifier), a context-free actor identifier we already need to rely upon for reproducibility purposes, at the level of the message-reordering system
• Probes are identified based on their producer name (as a binary string); the data-logger service is currently not managed by the resilience mechanisms

In the case of the probes, beyond their internal states, the engine has to take care also of the data and command files they may have already written on disk.

The result of this full state conversion could be stored on the k nodes either in RAM (with an obvious constraint in terms of memory footprint), but storing these information instead in dedicated files offers more advantages (but then a two-way serialisation service is needed).

For that we defined a simple file format, based on a header (specifying the version of that format) and a series of serialised entries, each of them being made of a type information (i.e. serialisation for a model instance, a probe instance or an agent instance) and a content, whose actual format depends on that type. The full specification of the format is documented in class_ResilienceAgent.erl.

Multiple steps of this procedure are instrumented thanks to WOOPER; notably:

• once, with the help of the time manager, the resilience manager determined that a serialisation shall occur, it requests all its distributed resilience agents to take care of the node they are running on
• to do so, each of them retrieves references (PID) of all local actors (from the corresponding local time manager), local simulation agents and local probes; then each of these instances is requested to serialise itself
• such a serialisation involves transforming its current state, notably replacing PID (that are transient) by higher-level, reproducible identifiers (the conversion being performed by a distributed instance tracking service); for that, the underlying data-structure of each attribute value (e.g. nested records in lists of tuples containing in some positions PID) is discovered at runtime, and recursively traversed and translated with much help from nested higher-order functions and closures; it results finally into a compact, binary representation of the state of each instance
• on each node (thus, in a distributed way), these serialisations are driven by worker processes (i.e. in parallel, to take also advantage of all local cores), and the resulting serialised content is sent to a local writer process (in charge of writing the serialisation file), tailored not to be a bottleneck; reciprocally, the deserialisation is based on as many parallel processes (for reading, recreating and relinking instances) as there are serialisation files to read locally

A few additional technical concerns had to be dealt with this resilience feature, like:

• The proper starting of Erlang VMs, so that the crash of a subset of them could be first detected, then overcome (initial versions crashed in turn; using now run_erl/to_erl)
• The redeployment of the engine services onto the surviving hosts; for example, the loss of nodes used to result in reducing accordingly the number of time managers, and thus merging their serialised state; however this mode of operation has not been kept, as the random state of these managers cannot be merged satisfactorily (to preserve reproducibility, models but also time managers need to rely on the same separate, independent random series as initially, notwithstanding the simulation rollbacks)
• Special cases must be accounted for, as crashes may happen while performing a serialisation snapshot or while being already in the course of recovering from previous crashes

Currently, when recovering from a crash, by design there is at least one extra set of agent states to consider (corresponding to at least one crashed node). Either these information are merged in the state of agents running on surviving nodes, or more than one agent of a given kind is created on the same computing node.

The latter solution raises issues, as up to one agent of a kind can register locally, and multiplying agents that way may hurt the performances.

So we preferred the former solution, even if the agents have then to be merged, and also if it leads to having rollbacks break reproducibility: indeed, whenever a computing node has to manage more than one serialisation file, its time manager will inheritmore than one random seed, and it will not be able to reproduce the two different random series that existed before the crash.

Erlang

Basically, Erlang provides a full environment particularly suitable for multi-agent applications.

At this lower level of the architecture, we are dealing with Erlang processes, which are lightweight objects:

• having each their own execution thread (their mode of operation is inherently concurrent)
• communicating between them only thanks to messages (pure asynchronous message passing, no memory is shared)

Erlang processes are not mapped to any scheduling object provided by the operating system or by the general execution environment. Thus Erlang processes are not system processes nor threads.

The Erlang virtual machine schedules itself all the Erlang processes it is hosting [29]. This is why the number of concurrent processes within the Erlang virtual machine can strongly outperform the number that could be achieved with any system-level thread of execution, as the overhead that Erlang processes induce is considerably smaller.

Erlang processes execute functions that are gathered in modules. Being Erlang code, they are implemented in a declarative, functional way, with single-assignment, absence of side-effects, pattern-matching and recursive behaviour:

Although a bit disconcerting at first glance, these kinds of languages are very suitable to develop complex algorithms and, more generally, scalable and reliable software.

Notably they offer higher-level constructs that allow Erlang programs written for one node to be distributed on a set of machines seamlessly, i.e. with little or no changes in the code, since Erlang processes will in both cases only exchange messages, regardless of their actual location (either on the same node or in networked nodes).

Erlang is an open source software, and has been used for more than 20 years. Its future is bright: now that physical limits are almost reached, microprocessors are less and less able to increase their operating frequencies, therefore the only solution that remains is to multiply the number of cores and try to use them effectively.

However traditional sequential languages cannot cope gracefully with such a parallel context, as they cannot deal with concurrent accesses to shared memory without error-prone and resource-consuming solutions, like mutex and semaphores. Therefore concurrency-based languages like Erlang should be increasingly needed in the years to come.

Myriad

Myriad (precisely, Ceylan-Myriad) is a generic toolbox built on top of Erlang, offering many low-level services used by all the upper layers involved here.

WOOPER

Erlang offers very suitable base services for our requirements, but when developing simulations one extra property would help a lot: the ability to model and implement simulated elements according to the Object-Oriented Paradigm.

Indeed, the modelling of numerous and complex behaviours is easier if being able to use the classical concepts of OOP like components, classes, instances, remote method invocations, inheritance, polymorphism, state management, etc.

Implementation is itself easier if the language supports some way of relying on a direct mapping between the OOP modelling concepts and the nuts and bolts offered by that language.

As classical Erlang is process-based and declarative, the OOP constructs have to added to the language. This is the task of WOOPER, which stands for Wrapper for OOP in Erlang.

WOOPER (precisely, Ceylan-Myriad) is a very lightweight layer that adds some code and conventions to Erlang so that a full-blown OOP approach can be applied directly to the language, at the expense of very little additional developing efforts and only a small run-time overhead.

Therefore, from that level on, we will not speak in terms of Erlang processes any more, we will mostly be dealing with instances of WOOPER classes.

WOOPER is an open source software (LGPL license).

Sim-Diasca

Such WOOPER instances are however not simulation actors yet: the support for the already mentioned mechanisms required in the context of a distributed simulation must be added, otherwise causality, reproducibility, etc. would not be ensured.

This is the task of the core component of the Sim-Diasca simulation engine: it provides the required technical components (like the TimeManager and the RandomManager) and the counterpart behaviours that all simulation actors should develop to interact properly with these technical components.

More precisely, Sim-Diasca Core provides the Actor class, from which all Sim-Diasca models should inherit (directly or not). Then they will automatically embed all the necessary logic to interact with the TimeManager, which includes managing top messages, tracking transparently acknowledgements of sent actor messages, dealing with errors and appropriate ends of ticks, etc.

Actors making use of random variables have also to interact correctly with the RandomManager. This is done similarly, just by inheriting from the StochasticActor class, which itself is a child class of the Actor class. Then all the mechanisms to find and use the RandomManager will be readily available, like for example the algorithm to maintain automatically a buffer of cached random values.

Finally, thanks to these inheritances, the development of models will mostly consist on specifying the business-specific state changes and message exchanges supported by each type of simulated element. Most technical issues are hidden to the model developer, who will only have to define:

• how an actor will be initialised (i.e. the constructor of its class)
• how an actor will be deleted (i.e. the destructor of its class)
• how an actor will behave spontaneously at each tick (i.e. its act method)
• any other behaviours that could be triggered by notifications received from other actors (i.e. the methods other simulation actors might call, thanks to actor messages)

These are totally model-specific, no simulation mechanism can provide them, only the model developer can know which code is relevant here.

Sim-Diasca Core provides as well useful technical components, like a full distribute trace system to be used by simulations.

LogMX

LogMX is a simple yet quite powerful tool to view logs. In the context of Sim-Diasca it is the main part of the supervisor of simulation traces.

As stated earlier, a simulation-specific format for traces is needed, and of course LogMX cannot know it a priori. Therefore a small LogMX-compliant trace parser, written in Java, has been developed, which integrates to LogMX. This is the only bit of Java involved in Sim-Diasca.

LogMX is a rather inexpensive tool (at most $29 per user), and Sim-Diasca can make use of its evaluation version as well.

gnuplot

gnuplot is a very well-known portable data and function plotting utility.

It is notably used by Sim-Diasca probes when they are requested to output a graphical view of their state: they automatically generate the appropriate command and data files, then call gnuplot to have it render the corresponding curves in a graphic file that might be displayed if wanted.

gnuplot is freely distributed.

Graphviz Dot

Graphviz is another quite widespread tool, which is a graph visualisation software.

Sim-Diasca uses it to generate graphical views of meshes: a Mesh (directed graph) is able to output a suitable description of this vertices and edges so that the dot program can generate from it a graphic file that might be displayed if wanted.

Various models make use of such meshes, like the LowVoltageMesh.

Graphviz (including dot) is an open source software.

Image Viewer

When Sim-Diasca needs to display a graphic file, it can drive various tools to do so, including the eog viewer, which is open source.

Mplayer/Mencoder

Mplayer is an open source software package that allows, among other things, to generate movies from a set of image files (with mencoder), and to display them (with mplayer).

Sim-Diasca uses them to aggregate a set of time stamped frames (each frame corresponding to one simulation tick) into a movie, so that the changes over time of graphical simulation results can be better monitored by the user.

For example, if, for a mesh, the generation of a time-based description of its state has been requested, a corresponding movie can be generated.

GNU make

The GNU make utility, which determines automatically which pieces of a large program need to be recompiled, and issues the commands to recompile them, is intensively used by Sim-Diasca, to build and run the simulator itself but also to post-process some of its results.

GNU make is open source.

Version Control: Git

Git, is a free and open source distributed version control system. It allows to keep track of the changes of the Sim-Diasca source code, and to share it among developers.

Docutils

Docutils is a set of open source documentation utilities. It operates on text files respecting the reStructuredText mark-up syntax, and is able to generate from it various formats, including LateX (hence PDF) and HTML.

This document has been generated thanks to Docutils.

Principles

Traces (a.k.a. simulation logs) are a way of recording for later use any event of interest, of any sort (technical or domain-specific), happening during a simulation. Traces allow to monitor selectively an execution, without recording each and every event that occurred.

Traces are not simulation results per se, their purpose is to make the technical troubleshooting easier, notably in order to help developing and debugging models.

Trace facilities are gathered in a separate layer, the Traces one (in the traces directory), which is used, among others, by Sim-Diasca. The Traces service depends only on WOOPER and on Myriad. Refer to the Ceylan-Traces official website for most information.

Please refer to the Sim-Diasca Developer Guide for information about the actual (practical) use of traces.

Defining a trace format allows to uncouple the execution of a simulation from the interpretation of what happened during its course of action (we can either monitor the simulation "live", or study it "post-mortem", i.e. after its termination).

If moreover the format is designed to be "universal" (in the context of discrete-time simulations), in order to be independent from a particular domain and from any trace generator or supervisor, then the post-processing toolchain of traces might be shared between several simulators.

Architecture

With Sim-Diasca, the management of distributed traces is split into the following roles:

• Each simulated object (model instance, i.e. actor), but also each technical agent or simulation case, may be led to send traces regarding its operation. The vast majority of them are TrameEmitter instances (they inherit from that class, directly or not), other are not instances thereof but nevertheless need to be able to output traces (e.g. test cases)
• The distributed traces have to be collected and aggregated, this is done by a TraceAggregator instance, which supports various output formats. Then overall analysis and search for event correlations are possible
• The simulation user might want to monitor the system, based on the emitted traces. This can be done either at execution-time (in real time) or post-mortem (once the simulation is over), both thanks to a TraceSupervisor, which can run from a remote host; moreover TraceListener instances can be created, so that from any host one can connect to the simulation while it is running, catching up automatically (past traces being sent as a whole initially, in a compressed form, next ones being sent directly, so that none is lacking)

How to manage the trace-induced overhead

Sending, collecting and aggregating traces proves very useful, but this feature demands resources, in terms of processing, memory and bandwidth: most of our models and agents by default emit many traces (they are intentionally very talkative), to help troubleshooting. As this is a distributed trace system, traces are emitted concurrently but at the end must be aggregated sequentially, in order to be consulted as a whole, in a single location. Therefore, beyond some simulation scale - and despite careful design - the trace aggregator is bound to become a bottleneck because of the massive message sending.

As a consequence, the point has been, for non-critical channels, to be able to disable trace sending as a whole, without any performance penalty compared to the same code in which there would not be any trace sending at all.

To disable trace sending and incur no runtime overhead, one should ensure that the compile-time make variable ENABLE_TRACES is set to false [30]. To do so, one should either specify it directly on the command-line (e.g. make clean all ENABLE_TRACES=false), or update directly the make files (typically by setting ENABLE_TRACES := false in traces/GNUmakevars.inc). Then everything above WOOPER (i.e. the Traces layer, the Sim-Diasca layer, and the code using it) should be rebuilt, as it is a compile-time (as opposed to runtime) option.

Finally, one should know that, if going for higher simulation scales, traces shall not be the only feature to be appropriately managed. Please refer to the How Should I run larger simulations? section for a description of the relevant measures that shall be taken for that (this includes trace disabling).

Trace Channels

There are eight built-in trace channels, of increasing severity:

• debug
• info
• notice
• warning
• error
• critical
• alert
• emergency

Depending on the nature of its message, a trace emitter can select in which channel its current trace should be output.

The five most critical - yet least used - trace channels (warning, error, critical, alert and emergency) will always echo their messages on the console as well (to ensure none can remain unnoticed), and will not be disabled even if the trace system is deactivated (i.e. regardless of the ENABLE_TRACES settings).

Depending on the needs of the simulation user, various types of trace outputs can be selected:

• traces integrated to an interactive graphical tool (LogMX)
• traces in raw text (to be browsed thanks to any text editor)
• PDF traces (any PDF viewer can then be used)

The type of traces is by default based on LogMX.

This can be changed (TraceType being then either log_mx_traces or {text_traces,T} where T is either text_only or pdf):

• one time for all, by editing the TraceType define in traces/src/traces.hrl (and then rebuilding all)

• on a per-case basis, still at compilation-time, by using the ?case_start(TraceType) macro (e.g. see soda_stochastic_integration_test.erl)

• at runtime, by specifying the --trace-type TraceSpecType command-line option, where TraceSpecType is logmx, text or pdf, like in:

  $ make foobar_run CMD_LINE_OPT="--trace-type text --batch"
  

Traces For LogMX

The resulting interface when browsing the traces (written in a *.traces file) with default aggregator output corresponds to:

Using the LogMX trace supervisor is surely the most useful way of monitoring the traces, in real-time or post-mortem. As such it is the trace supervisor that is by far the most frequently used.

LogMX offers rich features that allow to:

• browse conveniently even large sets of traces
• select the minimum level of detail for the traces to be displayed
• search traces for specific patterns, or by emitter, date, location, etc.

The only drawbacks of this trace mode are that:

• it requires a specific (commercial) tool (LogMX), properly obtained, installed and configured with the Sim-Diasca parser
• the results cannot be directly integrated into a report

LogMX-based traces are the default output type. Should the trace output type have been changed, it can be restored by setting TraceType to log_mx_traces, in traces/src/traces.hrl.

Text Based Traces

Although it is generally less convenient, simulation traces can be inspected directly from any text viewer (hence without LogMX).

If such a reading could be directly performed on the trace file (typically named MY_CASE-by-MY_USER-MY_SII.traces) that is generated by default for LogMX, its formatting is designed for that latter tool rather than for humans, hence is a bit difficult to read in its raw form. A better option is in this case to select a trace format whose outputs are meant to be read directly as they are, as free text.

To do so, in traces/src/traces.hrl, just define TraceType to {text_traces,text_only} instead of log_mx_traces, and rebuild everything above WOOPER (i.e. the Traces package, the Sim-Diasca package, and the code using it).

As a result, simulation traces output in the aforementioned trace file will be in plain text and encoded for human readability (within a nice ASCII-art array); they can thus be then read as are thanks to any text viewer.

By default gedit will be automatically triggered as a trace supervisor, and the user will be requested by the editor to reload that document as newer traces are appended.

If wanting to use another text viewer, just update accordingly the executable_utils:get_default_wide_text_viewer/1 function in the Myriad package (in myriad/src/utils/executable_utils.erl): gedit might be replaced for example by nedit or by xemacs.

Note that only the most interesting trace fields are kept here (a well-chosen subset of the full, actual ones), for the sake of readability.

PDF Trace Report

When wanting to generate a trace report once the simulation is over - rather than monitoring the traces during the simulation, a PDF output can be retained.

To do so, in traces/src/traces.hrl, just define TraceType to {text_traces,pdf} instead of the default log_mx_traces, and rebuild everything above WOOPER (i.e. the Traces package, the Sim-Diasca package, and the code using it).

Then, as soon as the simulation will have stopped, the traces will be properly aggregated and formatted from Sim-Diasca, a PDF being generated and then automatically displayed. This PDF can be useful to share simulation results asynchronously (e.g. with remote colleagues).

Only the most interesting trace fields are kept here (a well-chosen subset of the full, actual ones), for the sake of readability.

Note that this feature implies of course that you have already the proper documentation toolchain installed, which includes the RST converters (python-docutils package) and a PDF viewer (by default, evince will be used).

In the future, we could imagine an enhancement that would allow to convert a trace file generated for LogMX into a PDF, so that the user can generate a report without having to run again the simulation with different trace settings.

However, as simulations are expected to be totally reproducible, it would just a matter of saving some runtime, so the priority of this enhancement is low.

If wanting to use another PDF reader than evince, just update accordingly the executable_utils:get_default_pdf_viewer/0 function in the Myriad package (in myriad/src/utils/executable_utils.erl): evince might be replaced for example by mupdf, otherwise by acroread or xpdf.

Then recompiling this module should be enough.

Generic Probe

A generic probe can be dynamically configured to gather any number of time series, and represent them with as many curves whose abscissa axis corresponds to the simulation time, like in:

Samples may or may not be sent to their probe in chronological order (indeed, depending on their settings, probes may write directly their data to disk, yet timestamping them allows their renderer to reorder them if needed).

An example of the output rendering is:

The generic probe is defined in class_Probe.erl, and tested in probe_rendering_test.erl (unit rendering test) and class_Probe_test.erl (integration test).

By default:

• a probe will create a command file only when requested to generate a report (i.e. not when itself being created, in order to be able to take into account any later change in the curve declarations and rendering options before rendering is requested)
• the sample data that it will receive will be written to disk on-the-fly (i.e. it will not be cached in memory, in order to minimise the memory footprint of the probes)

Both behaviours can be changed, thanks to construction-time options (see, respectively, the create_command_file_initially and deferred_data_writes options).

As shown in the probe test, new curves can be dynamically declared, so that samples of increasing sizes can be sent over time (in the example, the fourth curve, the purple one, is dynamically added). The corresponding curves will be appropriately rendered the next time a report generation is requested. Existing curves can be renamed as well on-the-fly.

Full time-steps can be skipped, i.e. a new sample might be associated to a tick more than one tick after the previous one, and curve rendering will respect that time lapse (in the example, no sample at all was sent for ticks #4 and #9).

Partial samples can be sent as well, when no data is available for a given curve at a given tick (in the example, the second curve, the green one, had no relevant data for tick #3).

Finally, probes can also support the generation of histograms like this one:

Specialised Probes

Reliability Probes

Once associated with an equipment, a reliability probe, still based on a time series, records at each simulation tick the status of this equipment (functional or in failure), and can generate the chronology of status changes as shown here:

The reliability probe is defined in class_ReliabilityProbe.hrl, and tested in class_ReliabilityProbe.erl.

Probe Troubleshooting

Here are some general troubleshooting hints about probes, when they seem to fail to produce their expected graph renderings.

First, no error nor warning should be output in the terminal or in the simulation traces; otherwise the user code must be fixed accordingly. It can happen for example when no data at all was sent to a probe, whereas it was requested to generate a report.

For each probe that was created, fed with data and requested to be output, a report (a graphical view of its data) should be made available to the user.

By default, a basic probe will perform immediate (non-deferred) writes: otherwise it would have to keep its sample in-memory, potentially exhausting the RAM if the simulation was long enough.

As a consequence, by default, each probe uses one opened file descriptor. This limits by default the maximum number of simultaneously existing basic probes per computing node to roughly one thousand on most systems; having too many of such probes created results in the emfile error being reported - possibly as soon as probe creation, when each of them is to check that its prerequisites are met (typically regarding the gnuplot version available locally).

As this limit is just a shell setting, this can be overcome; for example ulimit -n 20000 will raise the maximum number of open file descriptors from the usual default 1024 to 20000, which is likely to be sufficient for most simulations.

This is nevertheless a transient setting that will be reset at each new shell. This limit is not raised automatically by Sim-Diasca at start-up, as on most systems this would require root privileges - at least, in terms of file descriptor count, above some threshold. For example ulimit -n 2000 may be accepted, whereas ulimit -n 20000 may not, returning then:

ulimit: open files: cannot modify limit: Operation not permitted

A more permanent solution (requiring root privileges) is to use nofile entries in /etc/security/limits.conf. For example one may add following entries to the end of this file (or edit them if already existing):

root soft  nofile 20000
root hard  nofile 40000

As larger numbers of probes are unlikely to generate diagrams that will be actually viewed, just generating and storing the corresponding data samples (no image being then produced) might suffice, in which case the result specifications (as declared in the simulation case) may opt for the data_only producer option (rather than rendering_only or data_and_rendering; refer to class_ResultProducer:producer_option/0).

An even safer option (scalability-wise) is, if relevant, to also tell directly to the probes themselves that no rendering is to be prepared (see the rendering_enabled field of the probe_options record for that).

gnuplot Test_probe.dat

Objectives

More precisely, the data-exchanger fulfills two technical purposes:

• to allow for a more efficient, yet safe, sharing of data, based on an automatic distribution that allows all read requests (potentially very numerous) to be performed locally only (i.e. directly on the computing node on which each reading actor is running - thus with very low overhead)
• to provide a way of propagating data conveniently and as quickly as possible in simulation time, latency-wise: an update made at tick T will be visible to all readers during the full T+1 tick, right from its start, and thus can then be accessed with zero-latency, through any number of direct reading messages per actor (without having to be synchronized thanks to actor messages that would be reordered and executed later); actor-wise, this allows to perform series of any number of (potentially conditional) read requests during the same tick, with potentially arbitrarily complex read patterns, at virtually no performance cost nor developing effort

Each data that is shared according to following conventions:

• data is defined as key/value pairs, respectively an atom and any Erlang term
• data can be static, i.e. be defined initially and thus be permanently available (e.g. through the built-in support for reading from configuration files) and/or be introduced in the course of the simulation, i.e. be dynamic
• can be modified over time once defined (non-const, to be defined with the mutable qualifier) or not (immutable, to be defined with the const qualifier)

Note that static/dynamic and mutable/const are orthogonal properties, all combinations of which making sense.

So, basically, compared to inter-actor messages, the data-exchanger offers an alternate, radically different paradigm to manage data, that can be seen as a DSM (i.e. a Distributed Shared Memory), somewhat a dual approach to the in-tick reordered actor messages. The point is that this service still allows to preserve simulation properties, and that the different trade-offs it offers may, depending on the situation, be more suitable for some needs of information sharings, notably in the one writer/many readers case.

These services had to be offered with care, as, for the model developer, unwanted side-effects or race conditions should not be made easier to perform, and for sure these sharing services must not become a means of bypassing the simulation mechanisms (e.g. if instantaneous reads and writes were allowed, then the induced possible immediate inter-actor communication would break at least reproducibility).

Note that the only standard way of exchanging information and synchronising between actors (as opposed to the sharing of information) remains the sending of actor messages.

Sharing Constant Data

One of the notable use cases of the data exchange service is the sharing of simulation-specific configuration settings: then all actors can have access to the same static (const) data sets, which will be available from the start, before the simulation is started, even before the first initial actor is created. It is then functionally equivalent to having each actor read these information from a common configuration file, except that the reading is done once for all actors on each node, instead of once per actor, avoiding the massive reading and parsing, etc. that would be incurred otherwise.

These configuration files must respect the Erlang term syntax, with a {Key,Value} pair or a {Key,Value,Qualifier} triplet per logical line of the file, ending with a dot. Qualifier is either mutable or const; not specifying a qualifier implies const [33]

Such configuration files are to be declared in the enable_data_exchanger field of the deployment settings (see the detailed syntax in class_DeploymentManager.hrl). They will be then automatically deployed [34] and their corresponding data defined. As a convention, their recommended extension is .cfg (e.g. my_simulation_parameters.cfg). See the valid_example_configuration_file.cfg and invalid_example_configuration_file.cfg as examples.

Of course the data-exchange distributed repository can also be fed at runtime, as opposed to statically (i.e. through configuration files). And this can be done initially (before the simulation is started), through method calls (see the define_initial_data/{1,2,3,4} static methods to define new data entries, and the modify_initial_data/{1,2,3,4} static methods to update them once defined) or from an actor (see the class_Actor:define_data/{2,3,4} and class_Actor:modify_data/{2,3,4} counterparts helper functions), thus either initially as well, or in the course of the simulation.

For an actor to be able to use the data-exchange service, it must first update its state with the class_Actor:enable_data_exchange/1 helper function. See class_DataExchangeTestActor.erl for a complete example. Note then that its subsequent reads (and commits) may happen whether the simulation is already running or not (the latter case applies to initial actors).

So, during any given tick, each actor can freely perform from the data exchanger immediate, direct (zero-latency, as not based on an actor message) read operations of any number of data-sets, and the developer can be confident that this will be done in the most convenient, safe and reliable, performance-efficient, scalable way.

Reciprocally, instead of being read, new data can be defined, or existing data can be modified, provided of course it was declared as non-const (i.e. with the mutable qualifier), as discussed in the next section.

Modifying Data

Any already-defined data, whose current qualifier is mutable, can be modified, either initially (in this case the change will be immediate) or in the course of the simulation (in this case the change will occur only starting from the next tick).

This is done thanks to calls to the modify_data/{3,4,5} helper functions, which on each tick are synchronous operations resulting in the root data exchanger recording all commits that happen during the current tick, and checking them on the fly (e.g. no change in const data, only up to one commit per data key per tick, etc.). A commit request is implemented as a direct message sending, from an actor to the root data exchanger (thus not going through the data-exchange tree).

Once this tick is over, should at least one commit have been requested, the root time manager notifies the root data exchanger that we are in-between two ticks and that it can thus propagate safely its commit(s), down the exchanger hierarchy (tree-based propagation).

Once done (i.e. fully acknowledged by a reverse, leaves-to-root, traversal), the root exchanger notifies the root time manager that the next scheduled tick can be triggered.

That way, at the expense of a minimal (and conditional [35]) inter-tick wall-clock latency [36], all subsequent readings then done can be performed locally (on an exact local clone of the root exchanger) and instantly (with a zero-tick latency in virtual time): indeed, during a tick, from the point of view of actors, no data can change; their update will happen only in-between ticks, so all reads can be done

Note also that setting a data involves either defining a new key or modifying a value associated to an already-defined key, possibly also updating the qualifier of a key/value pair, provided these operations are licit (e.g. a mutable qualifier may be set to const, not the other way round; a data can be defined only once, and modified any number of times once first defined).

More precisely, data modification, during the simulation, is to be done from actors, and is either:

• defining new data, with class_Actor:define_data/{2,3,4}
• or modifying pre-existing data, with class_Actor:modify_data/{2,3,4}

The point is that, once the simulation is started:

• a modification done at tick T will be visible (thanks to read operations) only at the next scheduled tick (e.g. T+1)
• a given data (i.e. a value associated to a key), provided it was declared as mutable, can be modified during the simulation up to once per tick, otherwise the data exchanger will detect the mismatch (commit inconsistency, up to one writer per key and per tick) and crash on purpose the simulation with a relevant exception

So if the data exchanger holds at tick T a {my_data,AnyTerm} mutable entry, and if during this tick an actor commits a {my_data,AnotherTerm} entry, then during T all read requests of my_data will return AnyTerm, and during T+1 they will all return AnotherTerm. If more than one actor attempts to commit at T a change to my_data, then the simulation will fail. Even if they try to commit the exactly the same value.

Practical Use

The data-exchange service is activated by default, as its runtime overhead in case it is not actually used is fairly low.

It can be explicitly enabled or disabled through the deployment settings, see the enable_data_exchanger field of the deployment_settings record, defined in class_DeploymentManager.hrl. It is also the place where the list of configuration files to be used for static information (if any) can be specified.

Data can be defined, then modified and/or read.

When a data is to be read, only its key is to be specified.

When a data entry (key and at least value) is specified either for definition or for modification, if no qualifier is specified, then the default one (const) will be implied. This means that requesting a modification with a data entry {my_data,AnyTerm}`` implies notably that:

• my_data has already been defined
• its previous qualifier was mutable (thus was explicitly set as such), otherwise the modification would be bound to fail
• its new qualifier will be const (none was specified in the modification request), thus no further modification will be done on that data

These data operations can be made either from the simulation case (thus, before the simulation is started) and/or from actors (before or during the simulation).

Implementation Details

There is one data exchanger agent per computing node, and they form a tree (the exchanger hierarchy), whose root is the root data-exchanger, very much like the one of time managers.

That way commit consistency is verified by the root data exchanger, which holds the sole reference copy of the data repository, which is replicated on all non-root (i.e. local) data exchangers. As updates are made between ticks, during a tick the data is stable, and each actor will be able to read them locally, and at will.

Data definitions and modifications will be recorded during a tick, and applied once that tick is over, before the next scheduled one.

Data can also be defined and modified before the simulation is started. Definitions can be done through requests or through the reading of any number of configuration files.

For actors, data readings are just plain, direct, non-reordered, WOOPER requests that are made locally (on the node of the reading actor), thanks to the local exchanger that was automatically deployed there. Therefore readings should be considered as being by design fast and inexpensive.

From the simulation case, the readings are managed a bit differently internally, depending on whether the user host is included or not in the simulation.

If included, then on that same user host there is a computing node with its own local data-exchanger. As a consequence, to avoid a data duplication of the exchange repository on the user host, the user node will interact with the data-exchanger local to the computing node on the same host.

If the user node is not included in the simulation, then there is not computing host to rely upon, and a local data-exchanger dedicated to the user node is simply created and integrated in the data-exchange hierarchy.

Overview

Currently only a support for 2D environments is provided - of course the user might define any number of other environments if wanted.

The spatial environment is embodied by an instance (a singleton) of class_TwoDimensionalEnvironment.

The support for low-level spatial computations is provided by the linear_2D module (in myriad/src/maths).

Distances are expressed in meters, speeds in meters per second.

The simulated world has an origin ({0.0,0.0}) and spreads in both dimensions. Cartesian coordinates are used.

By default the world is a torus, i.e. the coordinates wrap around. They range from the origin to {XMax,YMax}.

Spatialised Elements

A simulation element that is spatialized has a position in the simulation world, and should be (directly or not) an instance of class_SpatializedEntity or, more precisely here, a class_TwoDimensionalSpatializedActor.

The constructors of each of these classes take, as first actual parameter, the PID of the environment. An upper bound of the speed of each instance may be specified (it allows to optimise the mode of operation of the environment).

This class_TwoDimensionalSpatializedActor class defines an attribute position whose type is linear_2D:point() and a -spec getPosition(wooper_state(),pid()) actor oneway which triggers back on the caller, at the next diasca, the -spec notifyPosition(wooper_state(),linear_2D:point(),pid()) actor oneway.

Interaction with the Environment

In the general case, no instance is able to know the whole simulation world. An instance is only able to perceive a subset of it, through its perception.

Such a neighborhood can only be obtained thanks to a specific actor, the environment.

An instance can indeed call the actor oneway defined, here in class_TwoDimensionalEnvironment, as -spec getEntitiesWithin(point(),radius(),pid()): it requests that the entities within a disc whose center and radius are specified are determined by the environment.

This center of perception must be by convention the current location of the perceiving instance, as:

• a given instance should only perceive its own neighborhood
• the environment will take advantage of this call to update its knowledge of the location of the perceiver

Requirements & Functional Coverage

The current version provides the following features:

• trace the following memory consumptions on the user node and on computing nodes along with the simulation duration or with the wall-clock time:
  • the available memory (which is: free memory + buffers + cache)
  • the memory used by the other applications (i.e. all non-Erlang applications)
  • the memory allocated to the Erlang emulator (i.e. the memory currently used by the simulator), plus any other Erlang program being executed at the same time
  • the used swap
• trace the overall number of Erlang processes (on a per-node basis) during the simulation in real (wall-clock) time and/or in simulation time (tick)

In the future, it will allow to trace also:

• overall number of instances of a given class; for example: how many classical or virtual probes, or class_Foobar instances on each node
• the wall-clock duration of each simulation tick, on each node

The output of the performance tracker is:

• time series (.dat files)
• and/or the corresponding graphs (.png)

Its reports are available post-mortem (not live), among the simulation results.

Implementation

The performance tracker and its integration test are located in sim-diasca/src/core/src/data-management/monitoring/performance.

The performance tracker is integrated at the Sim-Diasca level (i.e. not only at the WOOPER-level), and relies on various services of the engine (e.g. probe, deployment manager, result manager).

The performance tracker is implemented as an optional service, disabled by default (to avoid any unnecessary runtime overhead), which can be enabled on a per-need basis.

It defines following classical probes:

• a global probe is created in order to track the overall number of Erlang processes on each node: it includes as many curves as there are nodes (user node and computing nodes); each curve is named according to its corresponding node
• a probe per node is also created in order to track the resource consumptions on each node

In terms of architecture, currently the performance tracker is centralized (a singleton) and is created on the same node with the Depployment Manager in order to enable the performance information available in all circumstances, such as, simulation fail because of a timeout or a computing node crash.

In case of performance tracker enabled, the data for the probes is retrieved in a predefined interval during simulation, by default, this interval is defined as 100ms and it can be modified by sending a "setTickerPeriod" message with new interval to the performance tracker.

In the future and in particular for the scalability test when numerous computing nodes are engaged, there will be a performance monitor per node besides this centralized tracker, in order to reduce the exchanges over the network. Contrariwise, the distributed monitoring way can lead to lose some node performance information when a node is crashed during the simulation and relevant data saved on it.

Performance Tracker Activation and Creation

The activation of the performance tracker is to be requested from the simulation case, through its deployment settings: the enable_performance_tracker field of the DeploymentSettings record (defined in class_DeploymentManager.hrl) must be set to true.

When the performance tracker is enabled, the user can customise it according to her needs, notably in order to set the wall-clock period, expressed in milliseconds, between two measures (performance tracker samples); see its setTickerPeriod/2 method.

To do so, the PID of the tracker must be obtained, thanks to the following static method:

MyPerformanceTrackerPid = class_PerformanceTracker:get_tracker()

Performance Tracker Report Example

Example 1: a single-host simulation

These reports are generated based on the results of the performance tracker test (class_Performance_Tracker_test.erl) with following test configuration:

• only one initial test actor (class_PerformanceTracker_TestActor)
• each test actor creates a new test actor every 4 simulation tick
• the total simulation duration is 36 simulation ticks
• the simulation runs on only one machine, that means: the user node and the computing node are on the same machine

One can have a global view of the simulation, first in wall-clock time:

The same global view can also be shown in simulation time:

More detailed information can be collected, on a per node basis. Here first is the report specific to the user node, in wall-clock time, then the same report for a computing node:

Example 2: a simulation distributed over 4 nodes

These reports are generated based on the results of the performance tracker test (class_Performance_Tracker_test.erl) with following test configuration:

• only one initial test actor (class_PerformanceTracker_TestActor)
• each test actor creates a new test actor every 4 simulation tick
• the total simulation duration is 36 simulation ticks
• the simulation runs on 4 machines, that means: the user node and one computing node are on one machine, the three other computing nodes are on three different machines

One can have a global view of the simulation, first in wall-clock time:

The same global view can also be shown in simulation time:

More detailed memory consumption information can be collected. Here first is the report specific to the user node, in wall-clock time, then the same report for computing nodes:

Time-Series Analyzer

Time-series are typically produced by simulation probes (either basic or virtual), and are stored in *.dat files, which gathers the values of a series of curves at different ticks.

Such files can be inputs to the Sim-Diasca analyzer tool for time series, which can then apply them various algorithms, some related to signal processings.

For a time-series file, identified processings of interest are:

• select a subset of curves in a time-series file
• remove empty ones and/or constant ones
• generate a basic gnuplot command file corresponding to a .dat file

For a curve, identified processings of interest are:

• find the extrema (minimum and maximum values), and when they occur (list of matching ticks)
• compute the average value of the curve (which is not necessarily the average of the values, due to potentially non-consecutive ticks)
• apply a convolution filter on the curve samples
• compute the curve for the moving average (a specific case of convolution), which is a way of obtaining a smoother curve, making its interpretation easier by highlighting trends
• perform a linear interpolation between samples
• perform decimation, to reduce the number of samples
• change scale (e.g. switch to logarithmic, reshape to fit into a 0..1 interval, etc.)

General Principles

Each plugin is able to:

• specify requests for configuration changes (e.g. like the number of sequencers on each computing node)
• know runtime information, like the list of the computing nodes that have been actually elected by the engine
• be notified of all main events (engine-related or even case-defined) regarding the simulation, which for example allows tools to monitor resources during only some phases of the simulation - typically when models are effectively evaluated (skipping deployment, preparation, initialisation phases, as well as the ones after the simulation)

More In-Depth Description

A simulation case can specify a list of directories that will be scanned for plugins at start-up by the engine.

This is done thanks to the plugin_directories field of the deployment_settings record, which can be set to a list of directory paths (either absolute or relative to the directory where the case was launched). See city_benchmarking_test.erl (in mock-simulators/city-example/src) for such an example case.

Each BEAM file found in any of these directories is expected to comply with the interface described in the sim_diasca_plugin behaviour [37] ).

This boils down to implementing a callback for all simulation events:

• on_simulator_start: when the simulator is started (or almost, as basic services, including the trace one, are already up)
• on_deployment_start: when the deployment phase starts
• on_deployment_stop: when the deployment phase stops
• on_technical_settings_available: when the simulation technical settings are available, notably once the deployment phase is over
• on_case_initialisation_start: when the simulation case starts the creation of the initial state of the simulation
• on_case_initialisation_stop: when the simulation case finished the creation of the initial state of the simulation
• on_simulation_start: when the simulation is started (first tick, first diasca)
• on_simulation_bootstrap_start: when the simulation is just started and must evaluate the first diasca of all initial actors
• on_simulation_bootstrap_stop: when the evaluation of the first diasca of all initial actors is over
• on_simulation_wallclock_milestone_met: when a wallclock milestone is met (i.e. when a given duration in real time elapsed)
• on_simulation_tick_milestone_met: when a tick milestone is met (i.e. when a given number of ticks have been evaluated)
• on_simulation_stop: when the simulation is stopped (an ending criterion was just met; only called on successful ending)
• on_result_gathering_start: when the results start being gathered, after simulation termination
• on_result_gathering_stop: when the results have been gathered
• on_simulator_stop: when the simulator execution stopped under normal circumstances (i.e. not crashing)
• on_case_specific_event: triggered iff a case decided to notify the plugins of a specific event

Most callbacks just take one parameters, the current plugin state, and return one value, the new plugin state.

This allows stateful plugins to be easily implemented (instead of, for example, spawning a dedicated and registering it), the engine keeping track of their state between callbacks. Of course using that feature is optional and, for example, all callbacks can ignore the engine-specified state and only return the undefined atom as new state.

Some callbacks are a little more complex:

• on_simulator_start/2 is given also the current configuration changes (as possibly already updated by other plugins - the first of them receiving a blank configuration_changes record), so that the currently executed plugin can update it
• on_technical_settings_available/2 is given also a technical_settings/2 record, describing the actual settings then enforced by the engine, notably on the basis of the previously consolidated plugin configuration requests
• on_simulation_wallclock_milestone_met/2 and on_simulation_tick_milestone_met are respectively also given the current simulation timestamp (respectively in real and virtual time), whenever a milestone is met
• on_case_specific_event/3 is given the name of the corresponding case-specific event (as an atom) and its associated data; see class_CityGenerator.erl for an example of use

Implementation Notes

All services of the lower layers are available to plugins. As a result, they can rely upon the facilities of the Myriad layer, WOOPER classes may be used as plugins, and the distributed trace system can be used by the plugins as well.

The plugin service is mostly located in sim-diasca/src/core/src/plugins.

An example of a full, complete plugin is available in plugins/tests/my_plugin_example.erl.

We might consider supporting even distributed plugins (with one instance of them, code and data, being deployed and run on each computing node), if such use case was found interesting.

Actors & Models

Expressivity

Of course, Turing machines can be simulated, but more generally very few restrictions apply to models, which can be arbitrarily complex.

The most difficult issue to deal with comes probably from the latency (in virtual time) which is induced by the simulation mechanisms: an information (i.e. an actor message) sent at tick T cannot be processed by its recipient sooner than tick T+1.

This one-tick latency can be alleviated by increasing the simulation frequency. It will then just be a matter of choosing the preferred trade-off between latency and wall-clock simulation duration.

Time-Stepped Simulation

Models and actors have to comply with the operating conventions of the Sim-Diasca simulations.

In that context, the main convention is that these simulations are time-stepped. This means that the simulation time is sliced into chunks of constant durations.

Therefore we can define at which frequency a simulation should run: a simulation scheduled at 50Hz will rely on a fundamental period of 1/50 = 20 ms, thus all simulation time-steps (also known as ticks) will correspond to 20 ms of virtual time.

Uncoupling From Virtual Time

Actors are only aware of the passing of this simulation virtual time: the actual time that we, users, experience must not matter to them, otherwise the simulation would loose most of its expected properties.

For example, we do not want the same simulation to output different results depending on the processing power that happens to be available for it.

Thus, for an actor, whether the computing of a given virtual 20ms time-step actually lasts for 1 ms or for two minutes of our time shall be irrelevant; each of these 20 ms time-steps will last in our time exactly as long as needed to compute it appropriately, and the processing of two successive time-steps might require vastly different actual durations, in user time.

Formalisation

Modelling allows to bridge the gap between our knowledge of the system and its implementation in the context of a simulation. Generally, increasingly detailed models are specified: first as full text, then with more formal languages.

One can certainly use flowcharts:

But one may preferably rely on a few UML diagrams, including:

• use case diagram
• activity diagram
• class diagram
• sequence diagram

Other interesting diagrams might be:

• communication diagram
• state machine diagram

This is one of the most delicate steps, as often the domain experts are not able to write by their own their corresponding models: they generally make use of domain-specific languages, which are tailored for their needs but quite often are, simulation-wise, not standard.

So a translation step to the simulation language must generally take place, and usually domain experts cannot perform that work [40].

The best practice we recommend is to adopt a unified language to specify all models first, and to practise pair-work (one domain expert sitting on the side of a computer scientist/model developer) to ensure that the translation is correct. Indeed, mistakes can easily be made:

Actor Messages

Actors can communicate only thanks to the passing of specific messages, named actor messages.

During one tick, any actor can send any number of actor messages to any number of actors.

When an actor A sends during tick N an actor message to actor B, B will process it only during the next tick, N+1 [41].

A corollary is then that if A requests an information from B during tick N, A will process that information during tick N+2.

Moreover during a tick an actor may receive multiple messages from multiple actors. No assumption should be made on their processing order within that tick, as the simulation algorithm will have reordered them to ensure the respect of the simulation properties (e.g. reproducibility or ergodicity).

Actor Life-Cycle

Scheduling Sequence

At each fundamental simulation tick, each actor may or may not be triggered by the time manager.

If an actor message was sent to it during the last tick, then the actor will automatically process that message, regardless of its scheduling policy.

Then, the time manager determines whether the scheduling policy of this actor implies that it should develop its spontaneous behaviour during this tick.

If yes, the actor will be notified of the current tick and be requested to act according to its planned behaviour.

If no, the actor will not be specifically contacted?

Stochastic Variables

An actor may rely on any number of stochastic variables, each of which following any probability density function.

Sim-Diasca provides three of the most usual stochastic laws: uniform, Gaussian and exponential. User-specified laws can be added quite easily.

Once synchronised, an actor can draw any number of stochastic variables for any law during one tick, immediately (i.e. with a zero-tick latency).

Fundamental Frequency

As discussed previously, the root time manager in charge of a simulation will maintain its virtual time based on the fundamental overall frequency the simulation user specified: this frequency (e.g. 50 Hz) directly dictates the duration in virtual time between two successive engine ticks (e.g. 20 ms).

Therefore once this root time manager will have determined that all the actors to be scheduled this tick (the ones having to process actor message(s) and/or having to develop their behaviour on that tick) have reported that they have finished taking that current tick into account, it will then just increment the simulation tick, since the corresponding virtual 20 ms will have elapsed, and then declare that a new tick just has just begun.

Relying on a fundamental simulation frequency does not imply however that each and every simulation actor will have to be scheduled according to that exact overall frequency, i.e. at each tick. Each model is able to pick a scheduling policy that match best its needs.

Once all models have been established, the overall frequency of the simulation can be determined: it should be chosen at least equal to the highest frequency of the models involved.

Policies in Terms of Actor Scheduling

Generally the fundamental frequency will have been chosen so that the most reactive actors can be scheduled at the exact pace they require, but usually there will be also many other actors whose behaviour does not need to be evaluated as frequently.

Therefore, to ease the implementation of models and to preserve performances, the Sim-Diasca simulation engine allows models to request a scheduling more flexible than "every actor is triggered at each simulation tick".

Scheduling-wise, the three most common types of actors are:

• periodical actors: an actor requesting a scheduling period of N would be triggered by the time manager one time step every N elapsed; therefore an actor could run, in virtual time, at a frequency of 10 Hz even if the fundamental frequency of the simulation was set for example to 50Hz
• step-by-step actors: when such actors finish a time step, they may specify the next tick at which they should be triggered again (look-ahead), unless they receive an actor message in-between, in which case they may withdraw their already planned activation and set a new one, earlier or later
• purely passive actors: these actors have no spontaneous behaviour, they are triggered only when they receive a message from another actor during the previous tick

These scheduling policies - and many others - can be implemented with Sim-Diasca thanks to the definition of future actions: each actor, during its triggered and spontaneous behaviours, is able to specify, if needed, a future tick at which it should be scheduled for a spontaneous behaviour.

Thus periodical actors will just define at the end of their spontaneous behaviour one future action which is to take place a fixed duration (in simulation time) after the current tick, step-by-step actors will define arbitrary future actions, and passive actors will never define any specific future action.

Frequency-Independent Timings

In the context of a model, durations (in virtual time) are encouraged to be defined explicitly, absolutely, rather than directly as a given number of simulation ticks, so that models remain as much as possible independent from the actual frequency a simulation is running at.

For example, when a simulated device starts a new task and thus has to determine when a priori it will have finished the corresponding work and be available again, its model may evaluate the corresponding duration to "1400 ms" (in virtual time) rather than directly to an hard-coded "28 fundamental ticks".

Then only (i.e. at run-time), that duration, depending on the actual settings of the current simulation, will be converted to the appropriate number of ticks, so that a change in the simulation fundamental frequency will not impact models.

Nature Of The Model

Should all concepts to be ultimately simulated be represented by models of their own? Sometimes using a simple data structure owned by another actor is the most appropriate approach.

If a concept:

• is used in multiple different contexts
• and/or is used by multiple actors
• and/or has a complex state and/or behaviour
• and/or is not tightly coupled to any model

then most probably this concept should be mapped to a specific model, i.e. a dedicated class inheriting from class_Actor.

Model Temporality And Reactivity

Supposing we determined that the model was to be implemented as a class, we must then establish whether this model is able to perform spontaneous actions.

If yes (i.e. its instances are able to trigger actions not directly related to the receiving of a message received from other actors), then this is an active actor that will have a spontaneous behaviour, possibly periodical or erratic (step-by-step), etc.

The model is then able to specify with a total freedom its spontaneous scheduling, using notably addSpontaneousTick/2, addSpontaneousTicks/2, withdrawnSpontaneousTick/2 and withdrawnSpontaneousTicks/2, from its actSpontaneous/1 oneway or any of ithe actor oneways it defined.

Please refer to the Sim-Diasca Developer Guide for further details.

State And Behaviour Of The Model

These are very model-specific, but general rules still apply.

Processing an actor message and acting spontaneously both boil down to writing an appropriate method, which may send actor messages and/or return any updated state and/or trigger the removal of that actor.

Initial Actors

Initial actors are to be created directly from the simulation case, and their creation must be synchronous, otherwise there could be a race condition between the moment they are all up and ready and the moment at which the simulation starts.

There must be at least one initial actor, as otherwise the simulation will stop as soon as started, since it will detect that no event at all can possibly happen anymore.

...
VendingMachinePid = class_Actor:create_initial_actor(
  class_SodaVendingMachine, [ _Name="My machine", _CanCount=15 ] ),
...
% Now simulation can be started.

...
LoadBalancerPid = class_LoadBalancer:get_balancer(),
...

FirstVendingMachinePid = class_Actor:create_initial_actor(
      class_SodaVendingMachine, [ _Name="My first machine",
         _FirstCanCount=15 ],
      LoadBalancerPid ),
...
SecondVendingMachinePid = class_Actor:create_initial_actor(
      class_SodaVendingMachine, [ "My second machine",
         _SecondCanCount=8 ],
      LoadBalancerPid ),
...
% Now simulation can be started.

...
FirstVendingMachinePid = class_Actor:create_initial_placed_actor(
   class_SodaVendingMachine, [ "My first machine", _CanCount=15 ]
   my_placement_hint_a ),
...
% Using now the variation with an explicit load balancer:
% (only available in the distributed case)
LoadBalancerPid = class_LoadBalancer:get_balancer(),
...

SecondVendingMachinePid = class_Actor:create_initial_placed_actor(
      class_SodaVendingMachine, [ "My second machine",
        _SecondCanCount=0 ],
      LoadBalancerPid, my_placement_hint_a ),
...
ThirdVendingMachinePid = class_Actor:create_initial_actor(
      class_SodaVendingMachine, [ "My third machine",
        _ThirdCanCount=8 ],
      LoadBalancerPid, my_placement_hint_b ),
...
% Now simulation can be started.

Simulation-Time Actors

These actors are created in the course of the simulation.

Such actors can only be created by other (pre-existing) actors, otherwise the uncoupling of real time and simulated times would be jeopardised. Thus once the simulation is started it is the only way of introducing new actors.

As before, actors can be created with or without placement hints.

...
CreatedState = class_Actor:create_actor(
       _CreatedClassname=class_PinkFlamingo,
       [_Name="Ringo",_Age=34], CurrentState ),
...

onActorCreated( State, CreatedActorPid,
               ActorClassName=class_PinkFlamingo,
               ActorConstructionParameters=[ "Ringo", 34 ],
               LoadBalancerPid ) ->
% Of course this oneway is usually overridden, at least
% to record the PID of the created actor and/or to start
% interacting with it.

Testing in a Simple Case

A good first step is to ensure that, in the target hardware and software setting, the intended simulation is already able to run in a small scale from begin to end, first on a single host then, if targeted, on an increasing number of hosts (see the distributed gotchas for that).

Ensure that only the safest choices are made (e.g. have a properly-configured DNS, fix permissions, rely on a normal user - not root, etc.). Investigate any network-related issue with such checkings.

Enabling the production Execution Target

If, for a given simulation, more than a few nodes are needed, then various preventive measures shall be taken in order to be ready to go to further scales (typically disabling most simulation traces, extending key time-outs, etc.).

For that the EXECUTION_TARGET compile-time overall flag has been defined. Its default value is development (simulations will not be really scalable, but a good troubleshooting support will be provided), but if you set it to production, then all settings for larger simulations will be applied.

It is a compile-time option, hence it must be applied when building Sim-Diasca and the layers above; thus one may run, from the root:

$ make clean all EXECUTION_TARGET=production

to prepare for any demanding run.

One may instead set EXECUTION_TARGET=production once for all, typically in myriad/GNUmakevars.inc, however most users prefer to go back and forth between the execution target settings (as traces, shorter time-outs etc. are very useful for developing and troubleshooting), using the command-line to switch.

It is even possible to compile everything in production mode, touch the source files that one wants to be still talkative (touch some_module.erl) and run just make all for the root: all touched module (and only them) will be then recompiled with the default execution target, by default the development one.

Circumventing any System Limit

Many GNU/Linux operating systems enforce various limits onto the resources that one application may use (RAM, file descriptors, cores, etc.).

Notably, UNIX processes that are considered using "too much" RAM might be killed by the operating system far before exhausting this memory.

The ulimit command and the configuration of the Linux kernel capabilities may be of interest then.

Containerization / OS-level virtualization (e.g. Docker, Singularity) may also have an impact.

Increasing the Maximum Number of Erlang Processes

The Erlang default limit is only 32768 processes, but Sim-Diasca relies on the myriad/src/scripts/launch-erl.sh script to launch its user VM, and this script enforces a larger limit (of a few thousands; refer to its max_process_count variable).

One may set the MAX_PROCESS_COUNT make variable (defined in myriad/GNUmakevars.inc) to set that process limit to any value of interest.

Selecting Whether any File-based Creation of Initial Actors shall be Concurrent

Currently, by default the simdiasca_allow_reproducible_nested_initial_creations token is defined (see the SIM_DIASCA_SETTINGS_FLAGS make variable in sim-diasca/GNUmakevars.inc), and thus the creation of the initial actors from an initialisation file is done by a single process.

This allows the nested creations (creating an initial actor from the constructor of another one) to be fully reproducible, including regarding the allocation of the AAI of these actors.

However, if not relying on such nested creations, or if totally-reproducible initial AAIs are not necessary, then it may useful to not define that token in one's settings; a pool of actor creators will then be spawned (one per core of the user host), resulting in a considerable speed-up of the initialisation of larger file-based simulations.

Monitoring Resources

Any tool to track resource usage (at least CPU, RAM, swap) on the target host(s), at the level of the operating system will certainly be of use.

Regarding Erlang itself (notably its VM), the observer application provides also invaluable runtime information.

Monitoring Traces and Logs

The Sim-Diasca traces are an invaluable means of tracking the course of a given simulation; error-like severities will always be enabled (even in production mode).

In case of a runtime problem, one should investigate also the main log files of the operating system (typically thanks to journalctl), as many events can happen (OOM - Out of Memory, the Sim-Diasca main process being killed process due to some limit being reached, a container enforcing some constraint, etc.).

Issue #1: undefined parse transform 'myriad_parse_transform'

When a build recipe fails that way, this usually means that one attempted to compile elements of a layer depending on the Myriad layer whereas this latter layer is itself not already compiled. One should preferably run make from the root, to ensure that the full code-base is rebuilt (layers will then be compiled in a relevant order).

Why is it so? Because a parse-transform (Erlang code modifying how Erlang code is compiled) defined in the Myriad layer is necessary to compile most of the BEAM files of all layers, hence it must be available prior to any of these builds.

Issue #2: Protocol: register error

If running a simulation and being notified of a strange error like:

{error_logger,{{2008,11,25},{15,25,6}}, "Protocol: ~p: register
error: ~p~n", ["inet_tcp",{{badmatch,{error,duplicate_name}},
[{inet_tcp_dist,listen,1},{net_kernel,start_protos,4},
{net_kernel,start_protos,3},{net_kernel,init_node,2},
{net_kernel,init,1},{gen_server,init_it,6}, {proc_lib,init_p,5}]}]}
[...]

then it is the symptom that a similarly-named Erlang virtual machine is already running on the same host. Either it is an idle forgotten simulation still opened on another terminal [50] (in that case shutting down the corresponding virtual machine will correct the situation), or the user really wants to have two instances of the same simulation run simultaneously. In that case, the two virtual machines should be named differently, knowing that, with the default Sim-Diasca Make rules, the launched virtual machines are named according to the case that is run. For example, to run the simulation case defined in simulationAndScenario_test.erl from the host myhost.foobar.org, one may just issue make simulationAndScenario_run. The corresponding Erlang virtual machine will be then named simulationAndScenario_run@myhost.foobar.org.

Issue #3: Can't set long node name! Please check your configuration

Such a problem may be reproduced simply by running on a given host:

$ erl -name my_test

Instead of running the expected VM, like in:

Erlang/OTP 21 [erts-10.3] [source] [64-bit] [...]

Eshell V10.3  (abort with ^G)
(my_test@hurricane.foobar.org)1>

the VM launcher may report that no long node naming can be used.

This may happen whenever the network configuration of the local host is not consistent, at least from the point of view of the Erlang virtual machine. More specifically, it can happen if in the /etc/hosts file the first name to appear for the local host is not the expected proper FQDN (Fully-Qualified Domain Name) and/or when the domain is not correctly specified.

For example, supposing that in /etc/resolv.conf a domain is specified as domain localdomain and that the local hostname is foo, then a line in /etc/hosts like:

127.0.0.1 localhost.localdomain localhost foo.bar.org foo

should be corrected into:

127.0.0.1 foo.bar.org foo localhost.localdomain localhost

Typically, one of the simplest /etc/hosts could be in this context:

127.0.0.1 localhost.localdomain localhost
::1       localhost
127.0.1.1 foo.localdomain foo

Ping your local host, use hostname, hostname -f and/or hostnamectl to check that the name resolution is correctly set. See also the related note about Domain configuration in the Sim-Diasca Installation Guide.

If you have for example a laptop making use of DHCP servers that assign over time different host/domain names and you find it impractical, you may reintroduce a stable naming (to be used at least by Sim-Diasca) by adding at the end of your /etc/hosts a line like:

127.0.2.1 a_host_name.a_domain_name a_host_name

(where a_host_name and a_domain_name can be any network names of your choice)

Then, in myriad/GNUmakevars.inc, the FQDN information shall be set statically, accordingly by editing the corresponding section with:

FQDN := a_host_name.a_domain_name

(before ifdef FQDN [...])

Issue #4: The Deployment of a Sim-Diasca Module apparently failed

The corresponding symptom is an exception being thrown during deployment and including:

{module_deployment_failed,SOME_MODULE,...

This may happen when running distributed simulations whereas hostname resolution is somehow failing.

For example, we encountered sometimes faulty network configurations (e.g. w.r.t. to a stale domain name) where a host contacted as foo.bar.org was responding as foo.other.org, and thus was never reported as available.

In other cases, a computing host was designated (either in a host file or directly in the simulation case) not, as expected, by its name (preferably FQDN) but, incorrectly, by its IP address (which is disallowed, see the computing_hosts field of the deployment_settings record).

Issue #5: Execution seems to be blocked right after having been triggered.

This may happen (albeit now on very rare cases; or, possibly, never anymore) if using a virtualized environment (e.g. VMWare or VirtualBox). Indeed there used to be, with some unspecified configurations, a general problem related to timers and message receiving, and apparently Sim-Diasca was not the culprit here (as unrelated applications were affected similarly). Erlang was maybe not guilty either, as possibly related issues were reported on the VMWare side.

Anyway, because of these problems and of the incurred performance penalty, the use of virtualized environments should be avoided here; at least one should develop and test one's simulation on a real hardware before considering running it in a virtualized form.

Another cause of a launched computing node not being found and resulting in a time-out might be an inconsistent name resolution (see issue #3).

For example, beware of specifying in /etc/resolv.conf a wrong domain in the domain entry (e.g. bar.org instead of foo.org) . Otherwise your user node may try to reach A_COMPUTING_NODE_NAME@HOST.foo.org whereas this one will believe its own name actually is A_COMPUTING_NODE_NAME@HOST.bar.org and thus will not respond - leading to Sim-Diasca freezing at start-up before automatically timing-out. If in doubt and having the relevant permissions, one may comment-out the domain information, at least for a first troubleshooting.

Issue #6: At least one computing node times-out because it did not receive on time (from the user node) the deployment archive.

The default deployment time-out is supposedly sufficient for most configuration settings.

If for example relying on very slow hard-disks and/or having defined extra simulation data to deploy whose size exceeds a few dozens megabytes, then maybe indeed you may need to increase your deployment time-out, at least for this simulation case.

For that, see the maximum_allowed_deployment_duration field of the deployment_settings record (defined in class_DeploymentManager.hrl, in the sim-diasca/src/core/src/deployment directory).

Such larger simulation archives may also result from user-level errors. A typical mistake was to run the Erlang installation script install-erlang.sh directly from its location (in myriad/conf): then the full build tree of Erlang/OTP could still reside in this latter directory. In this case, the deployment manager, when scanning the Myriad package, would also detect the BEAM files of Erlang/OTP and include them in the simulation archive. Note that a specific checking has been since then introduced so that the specific case of a local build of the Erlang/OTP runtime should be correctly detected, but this issue may arise for other codebases as well.

Of course including such duplicated BEAMs (as they shall be already available on the computing hosts) is not desirable at all, and results in larger simulation packages bound to trigger a deployment time-out.

So: just remove then, from the overall Sim-Diasca codebase, all build trees that do not belong there!

Issue #7: At start-up, the rebuild of the simulator codebase fails, although the code is correct.

This may happen if at least one source file (e.g. myFile.erl) is being edited without having been saved yet: some editors then create a temporary file like ~myFile.erl or .#myFile.erl in the same directory. The make system will try to rebuild that file, but the compilation will fail necessarily, as this filename will not match the module name. A proper error message should have been sent in the simulation traces.

Issue #8: A noconnection error is triggered in the course of the execution.

This usually means that at least one of the involved computing nodes unexpectedly crashed. The most likely reason is that its host was exceedingly loaded. This happens typically in the course of the creation of the initial actors: a too large simulation may then result on the exhaustion of the RAM (and, possibly, swap) of at least one computing host, crashing the whole simulation.

Solution: opt for a less demanding simulation and/or use more hosts, ensuring they have roughly the same level of free resources (knowing that the load balancer tends to even the resource demands across the available hosts).

Issue #9: Apparently my newer code does not seem to be taken into account!

More precisely, some changes to the source code have been made, yet the newer executions seem to correspond to the code that existed before the change rather than to the updated one. Or, more generally, the executed code does not seem to correspond to the specified one.

This could happen when multiple BEAM versions of the same module can be found from the deployment root. For example, from some subdirectory in the sources, one may have issued cp -r foo_directory foo_directory-hidden, to save temporarily its content while experimenting in-place in foo_directory.

The problem is that the deployment manager will scan for all BEAMs from the deployment root, and include them in the deployment archive. As a result, on each computing node, any BEAM found in foo_directory-hidden will be deployed as well and, depending on the code path, foo_directory-hidden/a_module.beam may be found before foo_directory/a_module.beam (unfortunately this tends to be often the case). As a consequence, the previous version of the code (the hidden one) would be wrongly executed.

The solution is to avoid to perform back-ups directly in the source tree (e.g. use git stash) or, at the very least, to copy them once all BEAMs have been removed, to avoid that they silently collide.

Another possible cause of not seeing a change when running Sim-Diasca (at least, not the first time it is then run) is to modify a source file without recompiling it afterwards: Sim-Diasca, during its deployment, will then recompile the whole (thus updating any BEAM file that requires it), yet the previous version of the BEAM may have already been loaded by the user node (and possibly sent over the network to other nodes). These changes would be visible only from the second run, not the first one. To avoid that, one should recompile a module when having modified it - anyway after a change we have to check that the module still compiles, isn't it?

Issue #10: My simulation seems to be finished, however it does not return to the shell, and it is still eating a lot of resources for quite long. What's happening?

It may happen whenever a simulation is executed for a long time and/or with numerous actors, whereas the intensity of trace sendings has not been lowered: although all trace modes write down a trace directly as soon as possible once received, and none, except the PDF mode, incurs long processings at shutdown, nevertheless all trace modes can significantly delay this shutdown phase.

The reason is that the trace aggregation process (see class_TraceAggregator) could not cope with the speed at which traces are sent by the various emitters, including actors. Thus traces accumulate in the aggregator mailbox, and time is needed for them to be formatted and flushed on disk. Sending too many traces regarding the aggregator speed should be avoided, as accumulating messages in the mailbox may result in a huge RAM consumption, delayed shutdown, and risk that a simulation crash happens whereas the corresponding traces are not written yet.

Issue #11: At runtime, an exception like {unexpected_ack_from,APid,PidList,ATick,ActorPid} is thrown.

Although it looks as if the engine was faulty, the cause must lie in the code of the class corresponding to the instance ActorPid refers to: most probably that an updated state was not taken into account into one of its methods, from where an actor message was sent (directly or not, like in the case of the creation of another actor) to the process corresponding to APid.

Indeed an actor message must have been sent, returning an updated state tracking that sending, whereas a previous state, unaware of that sending, was instead returned to WOOPER by that method. Thus when that actor received the acknowledgement corresponding to the actor message it sent, it does not correspond to any recorded sending, leading to the unexpected_ack_from exception to be triggered.

Issue #12: Simulation runs, but is slow.

This is a difficult issue to tackle generically. Some slowness are more acceptable than others:

Most efficient solutions to increase speed are:

• increase your computing resources (more nodes, more powerful, better network, etc.); check that you are never hitting the swap and, more generally, try to ensure that computing nodes stay well below a high load (performances in that case degrade swiftly)
• make (a better) use of advanced scheduling (models seldom require all the same evaluation frequency)
• selectively tune your models (e.g. use etop and the traces to spot the most-demanding ones)
• switch to more "exotic" solutions, like native compilation or the use of NIFs (i.e. Native Implemented Functions)
• ultimately, if at all possible, reduce your problem size
• improve your algorithms (e.g. choose better data-structures):

Issue #13: Simulation seems to freeze, or to be surprisingly slow, or more generally does not behave as expected, and I do not want to stick io:format calls everywhere to understand what is happening.

If not using the simulation traces either to figure out what is happening, then a good approach could be to connect to the busiest computing nodes (use simply top on each host) to determine what they are doing; to do so, track in the console the line which reminds the user of the names of the computing nodes and of the simulation cookie, like in:

To connect to computing nodes [
 'Scheduling_scalability_test-boudevil@server1',
 'Scheduling_scalability_test-boudevil@server2',
 'Scheduling_scalability_test-boudevil@server3'], use cookie
 '1f793a6ba507-d389-2e11-5bd1-2f759320'.

Then run a new node, connect to the computing node and run etop to inspect it, like in (maybe exporting DISPLAY and/or increasing the net tick time can help):

erl -epmd_port 4506 -setcookie '1f793a6ba507-d389-2e11-5bd1-2f759320' -sname inspector
(inspector@tesla)1> net_adm:ping(
'Scheduling_scalability_test-boudevil@server2').
 pong

Then hit CTRL-G and enter:

--> r 'Scheduling_scalability_test-boudevil@server2'
--> j
   1  {shell,start,[init]}
   2* {'Scheduling_scalability_test-boudevil@server2',shell,start,[]}
--> c 2
     (Scheduling_scalability_test-boudevil@server2)1> etop:start().

(note that the ping is not necessary, just issuing r 'Scheduling_scalability_test-boudevil@server2' then c would suffice)

Then you are able to see something like:

You can also run observer instead:

(Scheduling_scalability_test-boudevil@server2)1> observer:start().

And then we have:

Issue #14: Simulation runs, but result generation fails.

If the error message mentions unknown or ambiguous terminal type, this means that gnuplot (used by probes to generate graphical outputs) is (surprisingly enough) not able to generate PNG files. Either rebuild it accordingly, or select a gnuplot package in your distribution whose PNG support has been enabled beforehand.

Issue #15: At start-up, no available computing node is found, each candidate node being apparently successfully launched, but not responding.

This may happen if a previous simulation crashed and thus could not reach its clean-up phase: then pending Erlang nodes, spawned by the previous run, may linger for up to 10 minutes before their automatic shutdown, should the node cleaner script have been unable to remove them, for any reason (which must be very uncommon).

Indeed their node name will be correct, so no attempt to launch them will be made, but the automatic authentication system of the engine, based on security cookies generated from a unique UUID, will prevent the connection to these preexisting nodes. They will thus be deemed unavailable and the simulation will stop, short of being able to rely on any computing node. The solution is then either to remove these pending nodes manually (one effective yet rough means of doing so being killall -9 ssh beam beam.smp, to be run on all computing nodes) or to set the perform_initial_node_cleanup field in the deployment_settings record to true (see class_DeploymentManager.hrl) and recompile, in which case any lingering node would be removed when colliding with a newer run; as this latter setting is now the default, this issue should not happen frequently anymore, or at all.

Issue #16: Simulation runs and fails with no specific error message in the traces.

Of course this never happens usually, as it is precisely what we want to avoid.

Such a behaviour may sum up to a message like:

--- diasca {2200,2} still in progress at 2021/1/12 10:29:21 ---

being issued, then:

<----------------
[emergency] The 'Sim-Diasca-XXX-YYY-128694-computing-node@foobar.org'
node disconnected, performing an emergency shutdown.
---------------->

<----------------
[emergency] EXIT message received for <11029.94.0>, whose exit
reason was: noconnection, terminating now.
---------------->

The only case when such a behaviour was reported happened when a model developer created by mistake an infinite recursion [51]; the induced RAM consumption resulted in instantly having the VM killed by the operating system.

So chances are that this corresponds to a user implementation error.

Issue #17 [now unlikely to happen, as run_erl not used by default anymore]: A simulation case is launched, yet it freezes just after the line telling the trace aggregator has been created, and stays unresponsive until CTRL-C is entered.

This typically happens after a first failed launch: a virtual machine bearing the same name is already running on the background, thus preventing another one to be launched. The solution may be as simple as a brutal, yet efficient, killall -9 beam.smp.

This issue used to occur more frequently when the default launching mode was set to rely on run_erl (rather than a direct start from the command-line). No more {error_logger,T,"Protocol: ~tp: the name X@Ya seems to be in use by another Erlang node",["inet_tcp"]} was reported by the VM (as discussed in issue #1) yet, strangely enough, the issue discussed here could happen during the mass running of tests (e.g. when executing make test from the root). run_erl was suspected here.

Issue #18 Simulation is not reproducible.

One may run, in reproducible mode, a simulation twice, and unfortunately realize that results happen to differ.

Whether or not the technical setting changed (e.g. local run versus a distributed one), it is abnormal and surely disturbing - moreover it tends to be among the issues that are the most difficult to investigate.

Of course the engine might be the culprit, yet, for the moment at least, every time that reproducibility was lost, the cause was found to lie in the simulation itself, not in the engine.

The actual culprit could be the simulation case (e.g. see Randomness Pitfalls) or the models. For example the implementor must remind that simulations are executed so that they are reproducible, while PIDs are expected to change from one run to another (a bit like pointers). Hence no operation, except equality testing, shall be performed on them. For reliable, stable actor identifiers, one must use AAIs instead.

However this function happens to internally sort the elements of that list; as a consequence, removing duplicates from a list of non-reproducible PIDs resulted in a non-reproducible ordering, and the whole simulation started to behave differently from a run to the next...

To considerably increase the chances of spotting that different outcomes stem from a simulation (without even looking at the results), now the total number of diascas elapsed and of instance schedulings is displayed on the console. As soon as at least one of them differ from a run to another, the simulation is known to introduce non-reproducible elements, and must be fixed.

Issue #19 Problem when rebuilding the documentation.

In some cases the generated documentation encountered problems, typically the table of contents of the technical manual was empty.

This may come from some tools that insert Unicode characters (typically U+FEFF) that are invisible in most editors (e.g. emacs) yet that are not supported by the documentation generators (based on docutils and the RST syntax).

A solution is to check the output of the documentation tools (e.g. rubber) or to use editors like nedit, which displays these characters that shall be removed.

Traces are part of simulation results

This is not what we promote: we see the distributed traces as a way of monitoring technically a simulation run. Results are typically probe reports. Moreover, for actual large-scale runs, we generally prefer to disable traces.

The Performance Tracker is the one responsible for the progress information output on the terminal

No, the culprit is the console tracker, which is a live lightweight Sim-Diasca built-in, whereas the performance tracker is an unrelated, optional, more complex post-mortem feature.

Thanks to parallelism, I will have all my cores 100% busy

Unfortunately, the simulations of complex systems are in the general case not embarrassingly parallel, as their various components (actors) are bound to interact (there lies the main interest of these simulations); model instances have therefore to synchronise, and it certainly comes at a cost. If having many lightweight, intensely-interacting models, the engine may spend a large part of its time just enforcing the proper scheduling and communication of the corresponding instances.

Such a synchronisation is a "hard" problem that all the concurrent, discrete-time simulations experience; to the best of our knowledge, in terms of algorithms no silver bullet exists (one just has to pick one's poison). Sim-Diasca is based on trade-offs that we deem relevant for most cases. Depending on the problem to simulate at hand, other approaches may perform better (or worse).

Harnessing a larger set of resources thanks to distribution will boost the simulation performances

In the sense of:

• completing a given simulation sooner, the opposite is very likely: many simulations of complex systems are very sensitive to latency, and of course synchronising over a network is bound to be a lot slower than doing so inside a single local RAM; so a significant overhead shall be expected as soon as more than one computing host is used
• running simulations that could not even run on a single host, certainly

In terms of speedup, in link with this question and the previous ones, Amdahl's law gives an upper bound to one's expectations in the cases where the problem size is fixed.

Actor Creation

An actor can be created either from a simulation launcher (e.g. a test case), most often before the simulation started, or by another actor, usually in the course of simulation.

In both cases, these creations should lead to a totally reproducible simulation, moreover with no regard for the technical context (like the sets of computers being used).

To do so, an intermediate agent is used, the load balancer, which is a simulation actor and therefore can rely on the mechanisms for reproducibility.

Therefore the only simulation agent that will effectively create (spawn) a new simulation actor will be the load balancer.

Load Balancing

The load balancer, which is a singleton, will create actors remotely, on one of the available nodes, based on its balancing policy.

A priori the load balancer does not strictly need to create actors synchronously (as it will serialize their AAI assignment), but it needs anyway to return to the caller a notification that the created actors are ready, so that the caller can continue its operations without race conditions (e.g. for a creating actor, it means finishing its own tick). Otherwise one could not be sure for example that the created actor managed to properly subscribe to its time manager during the same tick.

As actors may have to be created either before or in the course of a simulation, there are two ways of telling the load balancer to spawn them:

• before the simulation started, the scenario or test case will call its bootstrap_actor request (not a oneway so that the caller can know for sure that the creation is over and, for example, can start the simulation from a proper and reproducible initial situation)
• in the course of the simulation: then the creator will be one of the current actor, and the creation will be based on an actor message sent to the load balancer, spawn_actor

Some more advanced load-balancing policies are interesting to provide, for example: the more instances interact, the closer they should be placed, the ideal situation being to have them run on the same computing node; the mere possibility of providing a hint to the load-balancer helps already a lot.

Indeed, this is just a matter of passing to the load balancer the same extra parameter, any Erlang term (e.g. an atom like house_111452), to the load-balancer when creating all actors corresponding to a given home (here the #111452). If having N candidate computing nodes, in that case the load balancer would bypass its default placement policy and always choose the N1 = hash(house_111452) rem N node (i.e. the hash value of the specified placement hint, modulo the number of nodes), ensuring all devices of that home - expected to be tightly linked - will be placed on the same node, N1. This could lead to a rather uniform distribution across nodes, while minimising very effectively the network traffic.

Actor Identifiers

As in all cases - reproducible or ergodic - messages will be reordered based on various information including the identity of the sender, the simulation engine must rely on actor identifiers which are themselves reproducible.

The Sim-Diasca 0.1.x versions relied for that on Erlang process identifiers (PID), which where indeed reproducible in the context of a given computing architecture, but which would offer no guarantee should, for example, the number of computers change. Therefore in this case a given actor could be identified by different Pid, which in turn would lead to different reorderings, and therefore different simulation outcomes.

With the Sim-Diasca 0.1.x versions, actors use abstract identifiers, not technical ones like the Pid. In the course of a simulation, there is a one-to-one relationship (a bijection) between an Abstract Actor Identifier (AAI) and a Pid. But the same simulation run twice will exhibit actors bearing the same AAI with presumably different Pid.

AAI are managed by the load balancer, which simply maintains a counter of actors as it creates them, and keeps track of their Pid to be able to answer to look-ups.

The AAI of an actor is assigned when it is created, directly as the first construction parameter.

To avoid listing such an additional constructor parameter that we would have preferred invisible, the load balancer could have sent a message instead, that the actor would have waited in its constructor. However it is more complex, requires to send more networked messages, and finally might maybe be hidden with an appropriate parse transform.

The two-way resolving between AAI and Pid should probably be taken in charge preferably by a dedicated agent.

Actor Start-up

The fundamental frequency of the simulation should be automatically specified to any created actor, so that it can adjust its timings or report an unability to comply with the simulation frequency.

After the start-up phase, each actor should know at least:

• the Pid of its time manager
• the fundamental frequency of the simulation or, more precisely, the duration in simulation time of a elementary time step
• its own random seeding for reproducibility and all
• its Abstract Actor Identifier (AAI)
• its scheduling policy

Actor Scheduling

Now the spontaneous and triggered behaviours of an actor do not have to happen at each tick: an actor can freely choose at which future simulation tick it should be scheduled next by the timer manager, whereas the actor messages sent to it will dictate when it will process them.

So an actor can behave spontaneously either as a periodical actor (being scheduled 1 tick every N), as a step-by-step actor (determining, while being scheduled, when it should be scheduled next), or as a purely passive actor (being only triggered by incoming actor messages, not having any spontaneous behaviour).

Actors will have also to be able to withdraw or change a previously selected tick. This can be useful when an actor receives an actor message between two spontaneous schedulings and based on that decides to change the planned one.

To do so, at the beginning of a tick when an actor is expected to develop some behaviour, it will be triggered by its time manager:

• either by a spontaneous_top message, meaning this actor had planned to develop its spontaneous behaviour at this tick, and implicitly meaning that it has no actor message to process
• or a triggered_top message, meaning at least there is at least one pending actor message to be processed, and implicitly meaning that it has no spontaneous behaviour to develop at this tick
• or a twofold_top message, meaning there is at least one pending actor message to be processed and that this actor had planned to develop its spontaneous behaviour at this tick ; they will be managed in that order

With each of these top messages, the current simulation tick will be passed by the time manager.

Once the actor will have managed the top message it received, and once it will have successfully waited for the pending acknowledgement of any actor messages it sent this tick, it will notify its time manager its tick is finished by one of the following messages:

• {done,N} where N is a (strictly positive) number of ticks before this actor should be next scheduled for a spontaneous behaviour; for example, if during the tick 100 an actor returned {done,2}, then it will be scheduled for its spontaneous behaviour only at tick 102, possibly jumping over tick 101 if it did not receive any actor message at tick 100 ; having each actor specify explicitly its next spontaneous tick is the most flexible possible policy ; for example periodical schedulings or purely passive ones are just special cases
• {done,none} if this actor intends to remain purely passive (i.e. only triggered by message, with no spontaneous behaviour), at least until the first next receiving of an actor message
• terminating if this actor plans its removal at the next tick ; it will then receive a termination_top at the tick, and then nothing more

Although an actor may send directly these messages, they can be automatically handled by manage_end_of_tick, depending on the scheduling policy declared by the actor, in passive, {periodical,P} and custom.

Each time manager will maintain an ordered list of the next ticks to schedule. If no event is planned in the simulated system for a period of virtual time, then the simulation will automatically jump directly over that period (i.e. no resource will be wasted examining idle ticks).

For reliability and testing purposes, the current tick of the actor can be appended to each of these messages, so that the time manager can check whether times are properly synchronised.

An actor can send at any time during its tick a {withdraw_spontaneous,Tick} message telling its time manager it does not want any more to be scheduled for spontaneous behaviour on the specified (absolute) tick.

Inter-Actor Communication

When an actor A1 needs to communicate an information to an actor A2, A1 will send an actor message to A2.

This will actually involve the sending of three messages:

1. A1 sends the actual actor message to A2
2. upon reception, A2 sends:
   • an acknowledgment message to A1, so that A1 can finish its tick
   • a schedule_trigger message to its time manager, so that this manager schedules it back on the next tick in order for this message to be processed by A2

If A2 already knows that it will be triggered next tick in order to process actor messages (i.e. regardless on any spontaneous scheduling), it may choose not to notify again its time manager.

We could have imagined that, instead of A2, A1 could have contacted the time manager so that A2 is triggered on the next tick. However, in a distributed context, A1 and A2 may depend on different time managers, and we want to notify the one of A2, which handles A2, not the one, potentially different, of A1.

This is why it is the task of A2 to send adequately the schedule_trigger message. Not only A2 knows which time manager to notify whereas A1 does not, but also it allows to use only one potentially non-local (networked) message instead of two.

Inter-Time Manager Synchronisation

A time manager can have zero or one parent time manager (a time manager cannot be its own parent and no child manager should be set as a parent), any number of child time managers, and any number of actors to manage directly.

Therefore the time managers respect a hierarchical structure. As in each simulation any two time managers must be, directly or not, ancestor and heir (they must belong to the same graph), the structure is actually a tree, whose root corresponds to the time manager directly in touch with the user, and whose leaves are either time managers or, more probably, actors (an actor cannot be placed elsewhere than on a leaf).

When a tick is finished, all time managers, from bottom to top, reports the first next tick they have to schedule, and the next simulation tick will be the one that will happen sooner.

So each time manager will determine, based on its own actors (if any) and on its direct child time managers (if any), what is the next tick T it would schedule (the soonest of the reported next ticks), then sends to its parent time manager (if any) a {next_tick,T} message.

Then when these messages reach the overall time manager (the root one, the only one having no parent time manager), the smaller tick of all is known, the consensus is found and sent down recursively in the scheduling tree of time managers with a {begin_tick,T} message. Each time manager will in turn translate it with the proper top messages for the actors they drive.

Granularity of Synchronisation

The scheduling tree can be of any depth, and we could imagine having one time-manager per core, per processor, per computer, or per simulation.

The trade-off we currently prefer is to let the Erlang SMP interpreter spread as much as possible in a computing node, i.e. across processors and cores.

For example, with a computer relying on two processors with four cores each, we could have imagined 8 time managers (one per core), or 2 (one per processor), however just having one of them is possible and probably better, performance-wise. So we would have here one Erlang node making use of eight run queues. The optimal number of such queues might be further optimised.

Reproducibility and Ergodicity

The simulation user can request the engine to work according to one of the following schemes:

• reproducible, with or without a user-specified random seed
• ergodic

In reproducible mode, running twice the same simulation (same scenario, with possibly different computing contexts) should output exactly the same results.

In ergodic mode, each simulation execution will follow a specific possible trajectory of the system, knowing that statistically, over a large number of executions, the exploration of the possible states should be fair, i.e. all possible situations allowed by the models should be able to show up, and moreover they should occur with respect to their theoretical probabilities, as dictated by the models.

In practical, the reproducible mode without a user-specified random seed will just result in each actor reordering its messages according to their hash.

Thus the actor will be seeded (for any need in terms of generation of stochastic values they could have) but will not perform any additional message permutations [52]. A default seed will then be used.

On the contrary, the other modes will rely on the actor-specific seed to perform an additional permutation of the messages.

More precisely, that seed will be the user-specified one if reproducible, or a seed automatically determined from current time if ergodic.

As the seed used in an ergodic context is recorded in the simulation traces, any ergodic execution can be later run again at will by the user, simply by specifying that seed in a new simulation execution, this time in reproducible mode.

As a consequence of these settings, in the context of a simulation the time managers, which are created by the load balancer, will:

• all be given a seed, and will generate a specific seed for each actor they manage
• request their actors either to reorder their messages based on hash only, or with an additional permutation

Reordering Of Actor Messages

Depending on the simulator settings, the reordering of the actor messages received for a given tick will be performed either so that reproductivity is ensured (i.e. messages are sorted according to a constant arbitrary order, the default one or one depending on a user-defined seed), or so that "ergodicity" is ensured, i.e. so that all possible reordering of events (messages) have a uniform probability of showing up.

In all cases a basic constant arbitrary order is obtained, based on keysort, which sorts the actor messages according to the natural order defined over Erlang terms [53].

Let's suppose for example that an actor has, for the current tick, the following message list, whose elements are triplets like {SenderActorPid,SenderActorAai,ActorMessage}:

L = [{pa,5,5},
     {pb,4,5},
     {pc,6,5},
     {pd,1,7},
     {pe,10,5},
     {pf,2,5},
     {pg,3,5},
     {ph,7,5},
     {ph,7,1},
     {pa,5,8},
     {pa,5,1}]

The constant arbitrary order is obtained thanks to lists:keysort(3,lists:keysort(2,L)) [54].

This means we sort first on the AAI (which is likely to be quite quick), like in:

lists:keysort(2,L).
[{pd,1,7},
 {pf,2,5},
 {pg,3,5},
 {pb,4,5},
 {pa,5,5},
 {pa,5,8},
 {pa,5,1},
 {pc,6,5},
 {ph,7,5},
 {ph,7,1},
 {pe,10,5}]

Once the entries are sorted in increasing AAI order (element #2), knowing that an actor may have sent multiple messages to that same actor, then we sort these entries based on their messages [55] (element #3):

lists:keysort(3,lists:keysort(2,L)).
[{pa,5,1},
 {ph,7,1},
 {pf,2,5},
 {pg,3,5},
 {pb,4,5},
 {pa,5,5},
 {pc,6,5},
 {ph,7,5},
 {pe,10,5},
 {pd,1,7},
 {pa,5,8}]

So, at the end, the reordering ensured that messages are always sorted by increasing AAI and, when multiple messages share the same AAI (i.e. they were sent by the same actor), these messages are always sorted identically (i.e. according to an increasing message order).

At this point a basic reproducible order, totally independent from the technical context, is ensured.

Then, depending on whether reproducibility or ergodicity are targeted, further reorderings are performed over that constant base.

If the user selected reproducibility, the list of actor messages obtained from the basic reordering are then uniformly permuted, according to the simulation seed, which is either the default one or a user-defined one.

If the user selected ergodicity, a fair exploration of all possible simulation outcomes is obtained by operating exactly like for the reproducible case, except that the random seed is not user-specified, it is itself automatically drawn at random, based on user time.

Then each simulation will explore its own way one of the possible trajectories of the system, knowing that any of these trajectories is fully determined by the drawn ergodic seed.

As a consequence, whenever such an ergodic trajectory is deemed interesting, it can be replayed at will simply by feeding the simulator with the same seed, this time in the context of a reproducible execution based on that user-defined seed.

Simulation Deployment

From the simulation scenario or from the test case, the load balancer must be created with the relevant simulation settings, including the list of candidate computing nodes.

The load balancer will then select the eligible computing nodes, which are the subset in the candidates nodes that can be connected:

• the corresponding host must be up and running
• it must be available from the network (ping)
• a properly configured and named Erlang VM either can be launched on that node (with a password-less SSH connection) or is already launched
• a two-way connection must be established with it (e.g. the security cookie must match)

Performances

One major goal of the Sim-Diasca 2.x versions is to increase the performances in a distributed context.

However some less demanding simulations will still be run in a local (non-distributed) context. So another requirement is to ensure that the new distributed mode of operation does not result in a loss of performances in a local context.

Example of Use

An example of such interaction could be:

% Here instances are created on each calculator in turn:
BalancerPid = class_LoadBalancer:new_link( round_robin,
    [ host_a, host_b, host_b ] ),

% The load balancer creates on each calculator as many local time
% managers as there are available nodes.

% Replaces class_PLCNetwork:remote_new_link(MyHost,35,4,rural):
BalancerPid ! {instantiate_link,[class_PLCNetwork,[35,4,rural]],self()},

PLCNetworkPid = receive

    {wooper_result,{instantiated,Pid,_Computer}} ->
        Pid

end,
[..]

Load Balancing Approaches

Instead of an hardcoded placement, a load balancer can perform:

• either a static balancing, i.e. actors will be created regardless of the actual machine loads, with a priori rules (e.g. round-robin)
• or a dynamic one, i.e. thanks to heuristics the load balancer will try to dispatch the induced load as evenly as possible among the computing nodes, based on the measurement of their actual load over time

In both cases, using a load balancer will lead in most cases to break the reproducibility of the association between a given actor instance and a Pid: a static balancing over a varying number of computing nodes or a dynamic balancing in all contexts will result in a given actor to bear different Pid from a simulation to another [56].

As explained below, this is not what we want, as we aim to uncouple totally the results of the simulations from the technical environments that support them.

On a side note, once the user code is able to rely on a load balancer, it will not depend on any particular type of load balancer, since all balancers will all be given creation requests and will all return the Pid of the corresponding created instances.

Therefore one can start with a very basic load balancer (like a round-robin based one), knowing that the integration of a more advanced ones (say, a dynamic one using advanced heuristics) should not imply any model to be modified.

Another interesting feature would be to have a load balancer which would take into account the tightness of the coupling between a set of actors. Then, the more actors would interact, the stronger the tendency to instantiate them on the same node would be.

If such a guessing about coupling intensity seems difficult to achieve for a load balancer, the simulation user could hint it, for example by designating a group by an atom and specifying that atom at each creation of one of its member. Then the load balancer would just have to try to place all actors bearing that atom on the same node, whatever it is.

Actor Creation

In the course of the simulation, an actor may need to create another actor [57]. In this case it has to request the creation to the load balancer.

In the future, we could imagine following enhancements:

• the creating actor could be able to specify a placement hint, which could be any Erlang term (generally, an atom), to increase the probability that coupled actors are created on the same node; so, for example, an anthouse A, itself created with a placement hint anthouse-a, could specify the same hint whenever requesting the creation of an ant. Then the load balancer would compute the hash value of that hint and select always the same node based on that, provide this does not lead to a too unbalanced dispatching of actors onto nodes
• the creating actor could be able to specify a request identifier, which would help it tracking which actors were created by the load balancer on its behalf; indeed, if an actor requests at the same tick the creation of an instance of two different classes, then by default, when it will be notified by the load balancer of these creations at the next tick, it will not be able to tell which returned PID corresponds to which instance, knowing that the load balancer had all its requests reordered