Analyse
The Media plugin uses Microsoft’s Video Indexer to analyze media assets for faces, sentiments, and annotations. The Media Plugin is also able to auto generate subtitles for a media asset.
Media
The Media plugin uses Microsoft’s Video Indexer to analyze media assets. The Video Indexer analyzes media assets for the following things:
- Temporal Metadata: Transcripts, Faces, OCRs, and Visual Content Moderation (captured in Dalet Flex using asset metadata timelines).
- Timeline Annotations: Topics, sentiments, audio effects, annotations, participants.
- Asset Metadata Instance: Language models, participants, and face information.
Notes:
- Upgrading to 7.1.0: Microsoft has deprecated version 1 of their API. This will affect the running of this plugin. You must upgrade to Dalet Flex 7.1.0 in order for the Video Indexer integration to function correctly. When you have upgraded, you must create new type (variant) with new asset metadata and new timelines. Instructions for this can be found below, in the “Configuring the Analysis Structure for a New Metadata Definition” and the “Setting Up Timelines” sections. If you have existing metadata and timelines associated with an asset, which contains the analysis from the old version of Video Indexer, you can keep these results. However, if you wish to run the new video indexer which references version 2 of Microsoft’s API, you must associate the asset with a brand new type (variant), with brand new asset metadata and timelines. and then run the Analyse action against the asset again. When you do this, you will lose any analysis results that were previously gathered.
- Microsoft has set a hard limit of 10GB when using FILE or S3.
- When uploading using FILE, the network can only upload a 2GB file within the space of 5 mins. After which the Microsoft server times out.
Configuring the Analysis Structure for a New Metadata Definition
The following steps are necessary in order for the analysis results to be returned in the form of a metadata instance.
-
In the Metadata Designer, create a type (variant). See the Types section for more information.
-
Create a metadata definition and associate it with the type (variant) that you created in step one.
-
Add the mandatory Video Indexer analysis fields to your metadata definition inside the
definitionsarray using the following PUT request:api/metadataDefinitions/{definitionId}/definition. The fields that should be added inside thedefinitionsarray are as follows:
Note: You must replace ${DEFINITION_ID} with the definition ID of your metadata definition created in step 1.
{
"definitionId": ${DEFINITION_ID},
"name": "root",
"definition": [
{
"id": 101707,
"name": "video-analysis",
"displayName": "Video Analysis",
"description": null,
"type": "complex",
"searchable": false,
"editable": false,
"required": false,
"formType": null,
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"children": [
{
"id": 101708,
"name": "id",
"displayName": "id",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101709,
"name": "created",
"displayName": "created",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101710,
"name": "faces",
"displayName": "faces",
"description": null,
"type": "complex",
"multiplicity": "0..*",
"searchable": false,
"editable": false,
"required": false,
"formType": null,
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"children": [
{
"id": 101711,
"name": "face-id",
"displayName": "Id",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101712,
"name": "face-flex-asset-id",
"displayName": "Flex Asset Id",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101713,
"name": "face-name",
"displayName": "Name",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101714,
"name": "face-description",
"displayName": "Description",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101715,
"name": "face-title",
"displayName": "Title",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101716,
"name": "face-thumbnail-id",
"displayName": "Thumbnail Id",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101717,
"name": "face-confidence",
"displayName": "Confidence",
"description": null,
"type": "double",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101718,
"name": "face-seen-duration",
"displayName": "Seen Duration",
"description": null,
"type": "double",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101719,
"name": "face-seen-duration-ratio",
"displayName": "Seen Duration Ratio",
"description": null,
"type": "double",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
}
]
},
{
"id": 101720,
"name": "participants",
"displayName": "participants",
"description": null,
"type": "complex",
"multiplicity": "0..*",
"searchable": false,
"editable": false,
"required": false,
"formType": null,
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"children": [
{
"id": 101721,
"name": "participant-id",
"displayName": "Id",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101722,
"name": "participant-name",
"displayName": "Name",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
}
]
},
{
"id": 101723,
"name": "language-models",
"displayName": "language Models",
"description": null,
"type": "complex",
"multiplicity": "0..*",
"searchable": false,
"editable": false,
"required": false,
"formType": null,
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"children": [
{
"id": 101724,
"name": "language-model-source-language",
"displayName": "Source Language",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101725,
"name": "language-model-language",
"displayName": "Language",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101726,
"name": "language-model-indexing-present",
"displayName": "Indexing Present",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101727,
"name": "language-model-linguistic-model-id",
"displayName": "Linguistic Model Id",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
}
]
}
]
}
],
"rules": []
}
Here you can see how these fields look as a metadata instance in Core:
-
Navigate to the definition in the Metadata Designer and you will see the fields.
-
Configure the Transcript, Faces, OCR, and Visual Content Moderation Timelines. For more information, please see the Setting Up Timelines section below.
-
Run the Analyse Media action against a media asset.
-
When the job has completed, click the Metadata tab to view the analysis results in the form of a metadata instance.
Note: If the type (variant) is changed after you have run the Analyse Media plugin, you must run the Analyse Media again.
Configuring the Analysis Structure for an Existing Metadata Definition Structure
- Run the following GET request against an existing metadata definition:
api/metadataDefinitions/{definitionId}/definition
This GET request retrieves the current structure of an existing metadata definition. In the response that is returned, you will see an array field called definitions. You will need to add the mandatory Video Indexer Analysis fields inside this array.
Below is an example of a metadata definition that has been returned prior to adding the mandatory Video Indexer analysis fields. In our example the metadata definition fields are basic strings: cust1, cust2, cust 3. In step 3 we will use a PUT request to add some mandatory fields to this existing metadata definition, so that the Analyse Media action has somewhere to place the results of the analysis.
{
"definitionId":144607,
"name":"root",
"definition":[
{
"id":144608,
"name":"cust1",
"displayName":"cust1",
"description":null,
"type":"string",
"searchable":false,
"editable":true,
"required":false,
"formType":"text",
"format":null,
"formatDescription":null,
"validation":null,
"maxLength":-1,
"expressionEnabled":false,
"validationDescription":null,
"validationHandler":null,
"valueGeneratorType":null,
"unitString":null,
"commentable":false,
"isVisible":true,
"preProcessors":[
]
},
{
"id":144609,
"name":"cust2",
"displayName":"cust2",
"description":null,
"type":"string",
"searchable":false,
"editable":true,
"required":false,
"formType":"text",
"format":null,
"formatDescription":null,
"validation":null,
"maxLength":-1,
"expressionEnabled":false,
"validationDescription":null,
"validationHandler":null,
"valueGeneratorType":null,
"unitString":null,
"commentable":false,
"isVisible":true,
"preProcessors":[
]
},
{
"id":144610,
"name":"cust3",
"displayName":"cust3",
"description":null,
"type":"string",
"searchable":false,
"editable":true,
"required":false,
"formType":"text",
"format":null,
"formatDescription":null,
"validation":null,
"maxLength":-1,
"expressionEnabled":false,
"validationDescription":null,
"validationHandler":null,
"valueGeneratorType":null,
"unitString":null,
"commentable":false,
"isVisible":true,
"preProcessors":[
]
}
],
"rules":[
]
}
- Add the mandatory Video Indexer analysis fields to your existing metadata definition inside the
definitionarray using the following PUT request:
api/metadataDefinitions/{definitionId}/definition
The fields that should be added inside the definition array are as follows:
{
"definitionId": 144607,
"name": "root",
"definition": [
{
"id": 144608,
"name": "cust1",
"displayName": "cust1",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 144609,
"name": "cust2",
"displayName": "cust2",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 144610,
"name": "cust3",
"displayName": "cust3",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102984,
"name": "video-analysis",
"displayName": "videoAnalysis",
"description": null,
"type": "complex",
"searchable": false,
"editable": false,
"required": false,
"formType": null,
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"children": [
{
"id": 102985,
"name": "faces",
"displayName": "faces",
"description": null,
"type": "complex",
"multiplicity": "0..*",
"searchable": false,
"editable": false,
"required": false,
"formType": null,
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"children": [
{
"id": 102986,
"name": "face-id",
"displayName": "id",
"description": null,
"type": "long",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102987,
"name": "face-bing-id",
"displayName": "bingId",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102988,
"name": "face-name",
"displayName": "name",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102989,
"name": "face-description",
"displayName": "description",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102990,
"name": "face-title",
"displayName": "title",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102991,
"name": "creationtime",
"displayName": "creationTime",
"description": null,
"type": "date",
"searchable": false,
"editable": true,
"required": false,
"formType": "date",
"format": "dd-MMM-yyyy",
"formatDescription": "dd-MMM-yyyy",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"timeZone": "UTC",
"locale": "en_GB"
},
{
"id": 102992,
"name": "face-thumbnail-full-url",
"displayName": "thumbnailFullUrl",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102993,
"name": "face-confidence",
"displayName": "confidence",
"description": null,
"type": "float",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102994,
"name": "face-seen-duration",
"displayName": "seenDuration",
"description": null,
"type": "double",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102995,
"name": "face-seen-duration-ratio",
"displayName": "seenDurationRatio",
"description": null,
"type": "float",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102996,
"name": "face-short-id",
"displayName": "shortid",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
}
]
},
{
"id": 102997,
"name": "participants",
"displayName": "participants",
"description": null,
"type": "complex",
"multiplicity": "0..*",
"searchable": false,
"editable": false,
"required": false,
"formType": null,
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"children": [
{
"id": 102998,
"name": "participant-id",
"displayName": "id",
"description": null,
"type": "long",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 102999,
"name": "participant-name",
"displayName": "name",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 103000,
"name": "participant-picture-url",
"displayName": "picture url",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
}
]
},
{
"id": 103001,
"name": "content-moderations",
"displayName": "Content Moderation",
"description": null,
"type": "complex",
"multiplicity": "0..*",
"searchable": false,
"editable": false,
"required": false,
"formType": null,
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"children": [
{
"id": 103002,
"name": "content-moderation-adult-classifier-value",
"displayName": "adultClassifierValue",
"description": null,
"type": "double",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 103003,
"name": "content-moderation-racy-classifier-value",
"displayName": "racyClassifierValue",
"description": null,
"type": "double",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 103004,
"name": "content-moderation-banned-words-count",
"displayName": "bannedWordsCount",
"description": null,
"type": "long",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 103005,
"name": "content-moderation-banned-words-ratio",
"displayName": "bannedWordsRatio",
"description": null,
"type": "double",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 103006,
"name": "content-moderation-review-recommended",
"displayName": "reviewRecommended",
"description": null,
"type": "boolean",
"searchable": false,
"editable": true,
"required": false,
"formType": "radio",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"options": [
{
"id": 103007,
"name": "true",
"displayName": "true",
"default": false,
"value": "true"
},
{
"id": 103008,
"name": "false",
"displayName": "false",
"default": false,
"value": "false"
}
]
},
{
"id": 103009,
"name": "content-moderation-is-adult",
"displayName": "isAdult",
"description": null,
"type": "boolean",
"searchable": false,
"editable": true,
"required": false,
"formType": "radio",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"options": [
{
"id": 103010,
"name": "true",
"displayName": "true",
"default": false,
"value": "true"
},
{
"id": 103011,
"name": "false",
"displayName": "false",
"default": false,
"value": "false"
}
]
}
]
},
{
"id": 103012,
"name": "audio-effects-categories",
"displayName": "Audio Effects Categories",
"description": null,
"type": "complex",
"multiplicity": "0..*",
"searchable": false,
"editable": false,
"required": false,
"formType": null,
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"children": [
{
"id": 103013,
"name": "audio-effect-category-type",
"displayName": "type",
"description": null,
"type": "long",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 103014,
"name": "audio-effect-category-key",
"displayName": "key",
"description": null,
"type": "text",
"searchable": false,
"editable": true,
"required": false,
"formType": "textarea",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
}
]
},
{
"id": 103015,
"name": "language-models",
"displayName": "languageModel",
"description": null,
"type": "complex",
"multiplicity": "0..*",
"searchable": false,
"editable": false,
"required": false,
"formType": null,
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": [],
"children": [
{
"id": 103016,
"name": "language-model-source-language",
"displayName": "sourceLanguage",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 103017,
"name": "language-model-language",
"displayName": "language",
"description": null,
"type": "text",
"searchable": false,
"editable": true,
"required": false,
"formType": "textarea",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 103018,
"name": "language-model-indexing-present",
"displayName": "indexingPresent",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 103019,
"name": "language-model-linguistic-model-id",
"displayName": "linguisticModelId",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
}
]
}
]
}
],
"rules": []
}
Once you have run the PUT request, the fields will be added to your metadata definition.
Note: The mandatory Video Indexer analysis fields must be added to your metadata definition in order for the analysis results to be returned as a metadata instance.
-
Navigate to the definition in the Metadata Designer and you will see the newly added fields.
-
Configure the Transcripts, Faces, OCRs, and Visual Content Moderation timelines. For more information, please see the Setting Up Timelines section below.
-
Run the Analyse Media action against an asset.
-
When the job has completed, click the Metadata tab to view the analysis results in the form of a metadata instance.
Note: If the type (variant) is changed after you have run the Analyse Media plugin, you must run the Analyse Media again.
Setting up Timelines
Below, you can see how to set up the various timeline tracks. The tracks are as follows:
- Transcripts
- Faces
- Optical Character Recognition (OCRs)
- Visual Content Moderations
Setting Up a Transcripts Timeline
The Microsoft Video Indexer can extract speech and convert it into text. This is represented as a track on the timeline below the player in MAM.
Note: The name of this timeline must be “Transcripts”.
To configure this, follow these steps:
- In the Metadata Designer, navigate to the type (variant) you have created.
- Create a timeline with the name “Transcripts”. For more information about how to create timelines, please see the Timeline Configuration guide.
- Create a metadata definition and then associate it with the Transcript timeline. This metadata definition must have the following fields (all of which must be strings):
- id
- text
- confidence
- participentid
{
"definitionId": ${DEFINITION_ID},
"name": "root",
"definition": [
{
"id": 101733,
"name": "id",
"displayName": "id",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101734,
"name": "text",
"displayName": "text",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101735,
"name": "confidence",
"displayName": "confidence",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101736,
"name": "participantid",
"displayName": "participantid",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
}
],
"rules": []
}
Setting Up an Optical Character Recognition (OCR) Timeline
The Video Indexer supports optional character recognition in a video. The Video Indexer can recognise text that appears in a video. For example: text on street signs.
Note: The name of this timeline must be “OCRs”.
To configure this, follow these steps:
- In the Metadata Designer, navigate to the type (variant) you have created.
- Create a timeline with the name “Faces”. For more information about how to create timelines, please see the Timeline Configuration guide.
- You must create a metadata definition and then associate it with the OCR timeline. This metadata definition must have the following fields (all of which must be strings):
- id
- top
- width
- height
- left
- language
- textdata
- confidence
{
"definitionId": ${DEFINITION_ID},
"name": "root",
"definition": [
{
"id": 101729,
"name": "id",
"displayName": "id",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101730,
"name": "name",
"displayName": "name",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 101731,
"name": "confidence",
"displayName": "confidence",
"description": null,
"type": "double",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
}
],
"rules": []
}
Setting up a Visual Content Moderation Timeline
The Video Indexer supports Visual Content Moderation. If certain content is deemed to only be appropriate for certain audiences or age ranges, this content will be put into a separate timeline which shows how appropriate or inappropriate the content is.
Note: The name of this timeline must be “VisualContentModerations”.
To configure this, follow these steps:
-
In the Metadata Designer, navigate to the type (variant) you have created.
-
Create a timeline with the name “Faces”. For more information about how to create timelines, please see the Timeline Configuration guide.
-
You must create a metadata definition and then associate it with the Visual Content Moderation timeline. This metadata definition must have the following fields (all of which must be strings):
-
id
-
adult-score
-
racy-score
{
"definitionId":${DEFINITION_ID},
"name": "root",
"definition": [
{
"id": 324526,
"name": "id",
"displayName": "id",
"description": null,
"type": "string",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": null,
"formatDescription": null,
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 324527,
"name": "adult-score",
"displayName": "Adult Score",
"description": null,
"type": "double",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
},
{
"id": 324528,
"name": "racy-score",
"displayName": "Racy Score",
"description": null,
"type": "double",
"searchable": false,
"editable": true,
"required": false,
"formType": "text",
"format": "#,##0.###",
"formatDescription": "#,##0.###",
"validation": null,
"maxLength": -1,
"expressionEnabled": false,
"validationDescription": null,
"validationHandler": null,
"valueGeneratorType": null,
"unitString": null,
"commentable": false,
"isVisible": true,
"preProcessors": []
}
],
"rules": []
}