Distributed Mode

Note

This feature is currently in alpha stage. While it's fully functional and ready for use, it's not massively tested in production yet. Bugs, performance issues and API changes should be expected. Welcome to have a try and build together with us!

Distributed mode enables you to create actors at remote nodes. When calling a remote actor, all arguments will be serialized and send to the remote node through network, after execution the return value will be deserialized and send back to the caller.

To make your actor class be able to be created remotely, you need to use EXA_REMOTE macro to register the class, e.g.:

class YourClass  {
 public:
  void Method1() {};
  void Method2() {};
};

YourClass FactoryCreate() { return YourClass(); }
EXA_REMOTE(&FactoryCreate, &YourClass::Method1, &YourClass::Method2);

In the EXA_REMOTE macro, the first argument is a factory function to create your class, and the rest are the methods you want to call remotely.

This is used to generate a serialization schema for network communication.

Then instead of calling ex_actor::Spawn<YourClass>(), you need to call ex_actor::Spawn<YourClass, &FactoryCreate>(). Or an exception will be thrown.

✨ BTW, this can be simplified in C++26 using reflection, I'll implement it when C++26 is released, stay tuned! ✨

Example

#include <cassert>
#include <cstdlib>
#include "ex_actor/api.h"

class PingWorker {
 public:
  explicit PingWorker(std::string name) : name_(std::move(name)) {}

  // You can also put this outside the class if you don't want to modify your class
  static PingWorker FactoryCreate(std::string name) { return PingWorker(std::move(name)); }

  std::string Ping(const std::string& message) { return "ack from " + name_ + ", msg got: " + message; }

 private:
  std::string name_;
};

// 1. Register the class & methods using EXA_REMOTE
EXA_REMOTE(&PingWorker::FactoryCreate, &PingWorker::Ping);

exec::task<void> MainCoroutine(uint32_t this_node_id, size_t total_nodes) {
  uint32_t remote_node_id = (this_node_id + 1) % total_nodes;

  // 2. Specify the factory function in ex_actor::Spawn
  auto ping_worker = co_await ex_actor::Spawn<PingWorker, &PingWorker::FactoryCreate>(
      ex_actor::ActorConfig {.node_id = remote_node_id}, /*name=*/"Alice");
  std::string ping_res = co_await ping_worker.Send<&PingWorker::Ping>("hello");
  assert(ping_res == "ack from Alice, msg got: hello");
}

int main(int argc, char** argv) {
  uint32_t this_node_id = std::atoi(argv[1]);
  std::vector<ex_actor::NodeInfo> cluster_node_info = {{.node_id = 0, .address = "tcp://127.0.0.1:5301"},
                                                       {.node_id = 1, .address = "tcp://127.0.0.1:5302"}};
  ex_actor::Init(/*thread_pool_size=*/4, this_node_id, cluster_node_info);

  stdexec::sync_wait(MainCoroutine(this_node_id, cluster_node_info.size()));

  ex_actor::Shutdown();
}

Compile this program into a binary, let's say distributed_node.

In one shell, run: ./distributed_node 0, in another shell, run: ./distributed_node 1. Both processes should exit normally.

Serialization

We choose reflect-cpp as the serialization library. It is a reflection-based C++20 serialization library, which can serialize basic structs automatically, avoid a lot of boilerplate code.

As for the protocol, since we have a fixed schema by nature(all nodes use the same binary, so the types are the same across all nodes), we can take advantage of a schemafull protocol. From this reason, we choose Cap'n Proto from reflect-cpp's supported protocols.

Network

We choose ZeroMQ, it's a well-known and sophisticated message passing library.

The topology is a full mesh. Each node holds one receive DEALER socket bound to local and several send DEALER sockets connected to other nodes.

While full mesh is simple and efficient in small clusters, it has a potential scalability issue, because the number of connections is O(n^2). It fits my current use case, so I have no plan yet to optimize further. If you encounter scalability issues, you can try to use a different topology, e.g. star topology. With ZeroMQ you can easily implement it by adding a central broker. Welcome to contribute!