Flex Classic Core Architecture

Dalet Flex 'Classic' allows you to deploy the application in a traditional way, running on a set of distributed physical or virtual servers. While powered by containerized micro-services based asoftware architecture, Flex's workload is being spread against multiple machines and follows a classical n-tiers approach:

  • Public access to Flex is powered by a set of application load-balancers, the only services exposed to public Internet (when required), secured by a classical L3 firewall (L7 web application firewall can be associated, if required).
  • Traffic is then routed back to either Web application frontends or backend API servers, provided proper user authentication and authorization.
  • Backend servers hold the fort, with distributed redunded data bases and persistent data storage.

Flex Infrastructure

Flex application workload is consequently distributed amongst several kind of servers (which can be optional or mutualized as well).

Reference Blueprint

Reference blueprint would be:

  • Load Balancer nodes can consist of either Cloud 'as-a-service' resources or servers which purpose is to receive public traffic, filter it and route it back to application backends. There's usually 2 instances of them (primary + secondary), for active-passive failover support.
  • Master nodes provide core API and job & workflow scheduler.
  • Job nodes where job and workflows get executed.
  • App nodes where most of backend application micro-services sit.
  • Database nodes can consist of either Cloud 'as-a-service' resources or servers which purpose is to provide MariaDB database. There's usually 2 instances of them (primary + secondary), for active-passive failover support.
  • Services nodes always run as a set of 3-servers cluster, provided various database backend services, in a fully distributed and reliable way.
  • FSP nodes are optional and meant for Flex Stream Processor feature. They usually consist of a possibly large farm of servers used to process and transcode media assets, requiring plenty of processing horsepower.
  • Monitor node is a special machine providing various monitoring and observability utilities (logs and metrics) and possibly acting an internal deployment gateway to other nodes.

On Flex systems, multiple application and storage backends are used to keep track of persistent data.

Databases

Databases servers play a vital role in application resilience and performance and can quite often become bottlenecks if undersized or poorly managed.

As a rule fo thumb, the more physical memory you can provide your databases instances with, the better performances will be. Databases use memory for caching and any request hitting the cache does not need to be made against (slower) disk. Always favor memory over CPU for databases if you had to pick between the two.

Physical storage should be backed by fastest as possible disks, SATA SSDs is the bare minimal, while NVMe SSDs is strongly recommended.

Dalet Flex consists of different types of databases:

  • MariaDB, persistent relational DB, used for core asset model, and other relational data structures including job management, resources, and permissions.
  • ArangoDB, persitent graph DB, used for metadata document storage, graph data structures such as taxonomies, thesaurus.
  • MongoDB, persistent non-relational DB, used for application audit events.
  • Redis, volatile in-memory queue, used for distributed locks, caching, application local queues, other transient distributed application data.
  • OpenSearch, persistent search engine, used for data indexing and searching.
  • RabbitMQ, persitent queue, used for internal inter-service communications.
  • Consul, service networking platform, used for internal inter-service discovery and environment specific configuration settings storage.

All of them are being used in their open-source form, free of charge, but with commuity-restricted support.

Persistent Storage

Dalet Flex may rely on different type of physical media for data storage:

  • Local disks, meant to store operating system data, application code and non-shared, non-critical data. Usually consists of SATA (NVME recommended) SSDs.
  • Remote network storage, meant to store various data or media assets which are to be shared accross server instances. Shared storage is usually provided by a resilient, durable and scalable RAID-compatible NAS/SAN endpoint, over 10+ Gbps private network. Logical storage access is usually provided through NFS (recommended) or CIFS protocols.
  • Remote object-storage, when available, is recommended to storage large media assets. It is cheaper than distributed filesystems and provides stateless access connectivity, making systems more robust to errors and network incidents. Logical access is provided through S3-compatible protocol.

Dalet Flex has been designed to be infrastructure vendor-agnostic, as much as can be, making it virtually compatible with any kind of deployments.

When deployed on-premises, it usually runs as virtual machines (VMware, Nutanix, Proxmox, QEMU/KVM, Xen) or even metal servers, with no specific technology stickiness.

When deployed on-cloude, it usually runs as 'as-a-service' virtual instances, possibly extended by 'as-a-service' databases, load-balancers, network gateways, storage ...

Flex Infrastructure Overview 1/2 Flex Infrastructure Overview 2/2

High-Availability

Whenever possible, Dalet recommends spreading Flex deployment into 3+ availability-zones (AZ).

Availability-zones is a concept of having dedicated datacenter (or at least rooms) isolated from each other, while keeping a reasonable network latency (< 1ms), which are autonomous and self-sufficient in terms of power supply, cooling, network access, Internet connectivity and such.

This ensures that, when correctly distributed amongst AZs, Flex survives a zone failure and remain active.