Resource Plugins

Capacity

Folder

A Folder plugin represents a folder for the storage of assets. A folder resource references one or more storage resources that is actually used for managing the storage of the underlying files. (A folder can span multiple storage resources). A folder is closer in concept to an object storage “bucket” as it does not support nesting.

Folder Properties
Folder Configuration

Hot Folder Resource

A Hot Folder is a special type of folder resource, designed for the processing and ingesting of files. This is a very versatile plugin that supports a range of features.

Variables

You can view a list of variables that are injected by Hot Folders here.

File Name Limitations in Hot Folders

The following characters are allowed in filenames in hot folders: !, @, $, #, &, _ (underscore), - (hyphen), ^, `, ‘, ~, ;, [], (), {}

The following special characters are not allowed in filenames in hot folders: %, Æ, Ø, Å, æ, ø, å, 电脑
(Any Chinese or Japanese characters), Umlauts (e.g. Ã¥ ä ö ü ß Ä Ö Ãœ), < >, :, “, /, , |, ?, *

Note: If a file name contains any of these disallowed characters, the system cnnot pick up that file from the hot folder.

Housekeeping Period

The Housekeeping Period option determines how often to clean up (delete) partial media file uploads, old MD5 checksum files and old manifest files from Dalet Flex.

Inactivity Timeout

This option specifies how much time should pass since the file was last modified before starting to process the file.

If a file is transferred without a manifest or MD5 checksum, it is impossible to know whether the transfer is completed. In this case, Dalet Flex assumes that the transfer completed since the last modified date, and begin ingest.

If an MD5 checksum fails, Dalet Flex fires an event. So it is easy to monitor for failed MD5 checksums. In fact, a range of Events related to transfer processing can be monitored.

Storage Resources

Storage resources are used in this context to point to storage mounts that the Hot Folder uses to process incoming files.

Match Pattern

Match pattern is a regular expression used to match file names for processing.

Ignore Pattern

Ignore pattern is a regular expression used to specify the files to ignore.

MD5 Checksums

MD5 checksum algorithms are used by Dalet Flex to check the integrity of files once ingested. Dalet Flex uses these algorithms to verify the integrity of the media file once delivered.

Workflow

Workflow specifies a workflow definition to run once the file has been processed.

A workflow should contain an Ingest node to ensure the file gets converted to an asset for further processing.

Owner

Owner specifies the owner (user) of the imported files.

Workspace

Workspace is the name of the Workspace to associated the newly ingested asset with.

Sidecar

Sidecar specifies a sidecar file (this is injected into the workflow as a variable).

Hot Folder Resource Properties
Hot Folder Resource Configuration

Hot Folder

The Hot Folder plugin can scan subdirectories recursively. This Hot Folder plugin does not need to be backed by a storage resource. Hot folder resource configuration itself supports configuring multiple VFS locations, which must be scanned for any changes.

Points to be Considered Before Using the Hot Folder Plugin

  • The current implementation of the Hot Folder service does not currently support all features of the previous Hot Folder Resource, such as Md5 support, Side car xml support, and Housekeeping.
  • A segmented asset must be copied as a package (even though there is a single file) inside the hot folder.
  • Only the master manifest file is tracked in Core. A segmented asset in Core points to the master m3u8 file only. However, the copy and move actions are modified to consider the segmented asset as a package, so that whenever a copy or move action is run against a segmented asset, it copies or moves the master manifest with all the other reference files.
  • The Hot Folder plugin does not read the m3u8 file, so cannot decide which is a master manifest and which are sub, or child manifest files. Pattern matching must be configured on the hot folder to help the hot folder to identify the master manifest.
  • Segmented assets created as part of hot folder scanning are marked as live files. Once the file is no longer live, it currently needs to be set to live = false explicitly by hitting the asset REST endpoint. In future, this will be taken care of automatically by the Hot Folder plugin.
  • Resource items are created in Core for every file scanned or processed (after applying match and ignore patterns). However, if there is more than one file with the same name (in different sub-directories) then there will be more than one resource item with the same name in Core. A tool-tip is displayed when you hover on the item name, showing a path to the specific file item location.

Hot Folder Properties
Hot Folder Configuration

Inbox

An inbox resource is a special type of folder resource, designed for the delivery of incoming files. This is a very versatile plugin that supports a range of features, including delivery from the FlexMOVE and standard FTP clients. The inbox V2 resource currently supports HTTP upload only. Later versions will incorporate functionality from the older inbox resource, which will be deprecated, and will support all types of remote transfer.

Manifest Files

When a file is transferred from the FlexMOVE, a manifest file is first sent to the inbox. This file comprises:

  • Folder name: The remote Folder being delivered to.
  • Md5 Checksum Data: Information for verifying that the file has been successfully transferred and its integrity is intact.
  • File Size Info: A file size value for verifying that the entire file has been delivered.
  • User: user details to identify the Dalet Flex user that transferred the file.
  • Workspace: The chosen workspace that the user would like the ingested asset associated with.
  • File name: The original file name (with timestamp prepended if this is enabled).
  • Content Metadata: Metadata values to associated with the ingested asset (assuming a metadata schema has been enabled).

Using a manifest file makes ingest more robust and also allows for the combination of files and “sidecar” metadata. When using the Transfer Agent, Manifest files are created and transferred automatically behind the scenes. Manifest files can also be created by other systems and transferred first, if you wish to use an FTP client.

If you intend to deliver to a specific workspace, or require Metadata entry on upload, you must enable manifests in your inbox.

Timestamps

If a user transfers a file with the same name as a file that is already in the inbox, then the transfer fails. This can be avoided by enabling the removal of timestamps.

If the Timestamp option is enabled, the Transfer Agent adds a timestamp at the start of the file. The timestamp is then removed prior to ingest into Dalet Flex. If this option is enabled and you are using an FTP client, you must add a timestamp yourself. The format is:

13-09-11-14-57-43_FILEname

where FILEname is the original name of your file.

MD5 Checksums

MD5 checksum algorithms are used by Dalet Flex to check the integrity of files once ingested. If you use a manifest file, Dalet Flex expects to find the checksum inside it. If you are not using a manifest, then the md5 checksum file must be sent first. Dalet Flex uses this file to verify the integrity of the media file once delivered.

If you use manifests, you can ignore the MD5 checksum option, as the MD5 checksum is always included in the manifest and therefore always used.

Housekeeping Period

The Housekeeping Period option is used to determine how often to clean up (delete) partial media file uploads, old MD5 checksum files and old manifest files from Dalet Flex.

Inactivity Timeout

This option is used to specify how much time should pass since the file was last modified before starting to process the file. If there is a manifest file, then Dalet Flex can check the file size to be sure that it completed transfer. If there is an MD5 file, Dalet Flex can check the integrity of the file. If the file transfer is complete and the file integrity is correct, the ingest process begins.

If a file is transferred without a manifest or MD5 checksum, it is impossible to know whether the transfer is completed. In this case, Dalet Flex assumes that the transfer has completed since the last modified date, and begins ingest.

If an MD5 checksum fails, Dalet Flex fires an event. So it is easy to monitor for failed MD5 checksums. In fact, a range of events related to transfer processing can be monitored.

Storage Resources

Storage resources are used in this context to point to storage mounts on the Dalet Flex Transfer servers. Normally, more than one of these entries exist, as Dalet Flex Transfer servers are typically deployed in clusters. The inbox polls these locations for new files to process and ingest.

Workflow

This option is used to specify a workflow defintion to run once the file has been processed.

A workflow should contain an Ingest node to ensure the file gets converted to an asset for further processing.

Validation

This option is used to specify an optional validation profile to use to validate the file before the Transfer Agent commences upload. This option is only relevant if you intend to use the Transfer Agent for transferring files.

Variant

This option is used to convert the ingest file into a specific asset . You can also specify what metadata schema to associate with the variant.

Inbox Properties
Inbox Configuration

DIVArchive Resource

This resource provides access to an archive resource. Configuration options are governed by the specific vendor solution being used, so in this case, the configuration options are related to DIVArchive.

DIVArchive Resource Properties
DIVArchive Resource Configuration

Shared Folder

Shared Folder resource is a special type of folder resource, designed for the sharing files residing in Dalet Flex. This is a very versatile plugin that supports a range of features, including transfer from the FlexMOVE and standard FTP clients.

Files uploaded to a shared folder can be downloaded using the FlexMOVE and using FTP clients. Shared files are never ingested into Dalet Flex.

MD5 Checksums

MD5 checksum algorithms are used by Dalet Flex to check the integrity of files once ingested.

Housekeeping Period

The Housekeeping Period option is used to determine how often to clean up (delete) partial media file uploads, old MD5 checksum files and old manifest files from Dalet Flex.

Inactivity Timeout

This option is used to specify how much time should pass since the file was last modified before starting to process the file. if there is a manifest file, then Dalet Flex can check the file size to be sure that it completed transfer. If there is an MD5 file, Dalet Flex can check the file’s integrity. If the file transfer is complete and the file integrity is correct, the ingest process will begin.

If a file is transferred without a manifest or MD5 checksum, it is impossible to know whether the transfer is completed. So in this case, Dalet Flex will assume the the transfer is complete by the time since the last modified date, and begin ingest.

If an MD5 checksum fails, Dalet Flex will fire an Event. So it’s easy to monitor for failed MD5 checksums. In fact there are a range of Events related to transfer processing that can be monitored.

Storage resources

Storage resources are used in this context to point to storage mounts on the Dalet Flex Transfer servers. You will normally see more than one of these entries as Dalet Flex Transfer servers are typically deployed in clusters. The inbox will poll these locations for new files to process and ingest.

If you just configure storage resources they should be on the Dalet Flex Transfer Servers. The file will be uploaded to Transfer Servers and will also be downloadable from FlexMOVE.

Incoming Storage Resources

If you configure incoming resources, these should be on the transfer servers and the storage resources should be on the internal network. Then you need to setup an Event Handler to move the files from “Incoming resource” to “Storage resource” - the idea being that you could virus scan etc. the incoming files before putting them in a Shared location. In this case, only those that get shifted to “storage resource” would be visible to download in FlexMOVE.

Shared Folder Properties
Shared Folder Configuration

Storage

CDN Storage

This plugin is a specialisation of the VFS storage resource. It provides a means of managing a CDN service as a storage resource in Dalet Flex. This can be extremely useful if, within a workflow, you wish to copy an asset from local storage directly to a CDN location for external access. It can also be useful if you wish to make an asset globally stream-able or downloadable outside of the Dalet Flex Console and yet still manageable as a Dalet Flex storage resource.

Dalet Flex can be configured to play back proxies from CDN storage resources within the Player framework.

The CDN storage resource requires two separate VFS mappings:

  • Read VFS Location: How the file on the CDN can be read by Dalet Flex file-based jobs. This may differ from the write protocol. For example the read protocol could be HTTPS.
  • Write VFS Location: How the file on the CDN can be written by Dalet Flex file-based jobs. This may differ from the read protocol. For example the write protocol could be FTPS.

CDN Storage Properties
CDN Storage Configuration

VFS Storage

This plugin acts as a wholesale replacement for the older storage plugin, which is supported but will be deprecated in future releases. The purpose of the VFS resource is to:

  • Provide access to different types of concrete file systems, managed by Dalet Flex, in a uniform way.
  • Support file-based processes between different file systems at the job level while hiding the underlying complexity.
  • Offer a larger degree of configuration and security options.
  • Make managing and integrating diverse storage devices more simple and less erroneous.

The virtual file system (VFS) is an abstraction layer on top of a more concrete file system. Within Dalet Flex, the concrete file system can be made up of:

  • object storage devices
  • File storage devices
  • Cloud storage devices

VFS Protocols

The following Protocols exist for the VFS storage resource:

  • SMB
  • SFTP
  • FTPS
  • FTP
  • WEBDAV
  • HTTP
  • HTTPS
  • FILE
  • S3
VFS Storage Properties
VFS Storage Configuration

Process

Extract

The extract resource is used for extracting technical metadata and keyframes from media files. It is used by extract plugins.

Extract jobs will not run unless an extract resource is available.

Extract Properties

Inject

The inject resource is used for injecting metadata into media files. It is used by inject plugins.

Inject jobs will not run unless an inject resource is available.

Inject Properties

VFS QC Resource

This resource provides access to networked QC resources. Configuration options are governed by the specific vendor solution being used. The QC Engine specifies which vendor solution should be used.

This resource relies on the QC Resource Adapter. Please consult this Adapter for information on which QC vendor solutions are supported.

VFS QC Resource Properties
VFS QC Resource Configuration

Azure Transcode Resource

This resource gives you access cloud based encoding features of the Standard Media Encoder within Azure Media Services, as another transcoding resource for Dalet Flex users.

Azure Transcode Resource Properties
Azure Transcode Resource Configuration

VFS Transcode

This resource provides access to networked Transcode resources. Configuration options are governed by the specific vendor solution being used. The Transcode Engine specifies which vendor solution should be used.

This resource relies on the Transcode resource Adapter. Please consult this Adapter for information on which Transcode vendor solutions are supported.

VFS Transcode Properties
VFS Transcode Configuration

I/O

Dalet Flex FTP Transfer

This plugin is used for managing a Dalet Flex Transfer Server resource. This resource can be configured to listen on multiple ports (FTP Services) as well as to support plain FTP or UDP-accelerated FTP. Some of the more important configuration options are discussed below.

Callback End Point

This field is used to instruct the Dalet Flex Transfer Server what URL to use when communicating with Dalet Flex’s Manager nodes.

FTP Service

This option allows for the configuration or one or more FTP Services. Each service listens for connections on its own port range.

Accelerated

This option allows for choosing between plain TCP or accelerated UDP. Accelerated UDP allows the Transfer Agent to transfer files more rapidly than with plain FTP.

A service set up with acceleration enabled will not work for normal FTP software.

Secure Transfer

These options allow for specification of SSL details to secure transfer.

Stopping this resource type will disconnect all remote connections.

Dalet Flex FTP Transfer Properties
Dalet Flex FTP Transfer Configuration

HTTP Transfer

This plugin is used for managing an HTTP-based Dalet Flex Transfer Server resource. This resource can be configured to listen on multiple ports (HTTP Service) and HTTP-based file transfer.

This resource supports large file transfer and advanced features such as pause and resume. It also support file chunking within the Dalet Flex Transfer Server.

It is recommended that you use secure transfer with HTTPS.

HTTP Transfer Properties
HTTP Transfer Configuration

Aspera Transfer

Aspera Transfer Properties
Aspera Transfer Configuration

Outbound Transfer Resource

Outbound Transfer Service is a Flex External resource Framework Service responsible for handling transfers between Dalet Flex and a target External VFS location. This resource is used in conjunction with the External Copy action, which supports the copying of a file from a source VFS location (Dalet Flex) to a target VFS location (External Systems). |

Outbound Transfer Resource Properties
Outbound Transfer Resource Configuration

Workflow

Vantage Resource Workflow

This version of the Vantage resource Workflow supports Vantage 7.0.

Note: When the Vantage resource is configured with FTP locations, a Copy node must be added after the Receive node in the Vantage workflow, to copy the FTP asset into the local path.

Vantage Resource Workflow Properties
Vantage Resource Workflow Configuration