MyChannels Documentation

VAS Upload

Editors can upload videos in VMS, so they can use them in the VAAS. We're using the Video Service (or VAS), made by De Videotheek, to handle these uploads.

When the VAS receives the file we've uploaded, it sends the file to their CDN (currently Ooyala) and performs certain operations on the video, like encoding it for different video formats, extracting screenshots, etc. After this is done, we can retrieve the URLs to the playable content and these screenshots and use them in our VMS.

Building blocks

Vas::API

This is a small generic wrapper around the VAS API and implements the default HTTP verbs like GET, POST, PUT and DELETE. It uses RestClient to make requests to the API. The API only accepts JSON.

Vas::Video

This is a class which implements all VAS API endpoints that we use in order to create, update and delete video assets, and some other things. It's build on top of Vas::API. It also contains a few convenience methods to determine the current upload state of the video asset.

Vas::CreateVideo

Before we can start an upload, we need to create an asset first. This is a requirement from the VAS API. This class does exactly that, you call it by invoking the #call method and pass in some parameters. The ones in DEFAULT_OPTIONS are required by the VAS API, and in addition to that you need to have a title, which should be supplied by the user and contained in video_params.

It then returns an instance of Vas::Video. You can then use #upload_url to retrieve the actual URL where you need to upload the video file to.

Vas::ReplaceVideo

This class can be used to retrieve a replace_url from the VAS API. With this URL, you can replace the contents of an already existing asset with a new video file. For instance, when you've fixed an error in an existing video file.

This works similar to Vas::CreateVideo, but instead of creating a new asset, you pass in the vas_content_id of an existing video asset. It will then return a Hash which contains the replace_url, an URL where you can upload the new video file to. The VAS API will then take care of switching the old file with the new file.

Although nice in theory we've ended up not using this, because there can be a large lag between the new video being uploaded and processed, and when it's actually available due to caching in the CDN. Since VmsUsers usually want the new video to be available immediately, we've stopped using this class and instead just delete the old asset and create a new one.

RemoveVasVideoWorker

When a VmsUser uploads a new version of an existing asset, this worker is added as a background job to Sidekiq, so it can delete the video file from the VAS. We do this in order to prevent the VAS filling up with files that are not being used anymore.

In some instances it can happen that the deleting of an asset fails, because the API says the asset has the property ready_for_use: true, and it cannot delete such assets. We do not set this property for productions we upload, so it's a bit weird why this happens. Luckily it happens only rarely. When it does, we usually just delete the background job from the Sidekiq queue.

UpdateProductionFromVas + UpdateProductionFromVasWorker

This is the main task that checks if a video has been processed correctly by the VAS. When it has, we grab all computed properties of the video asset from the API and add them to the Production. We then set the vas_content_status of the Production to done, indicating it can be published in the VMS.

When the API tells us the upload has failed, we also mark the Production as such, and save it. In this case, please check with De Videotheek what could have happened.

When the VAS API returns any other status, the job will be rescheduled one minute later, until either a done or failed state has been reached.

According to the VAS API docs, we should mark an asset as "ready" when it has been succesfully uploaded. We used to have a call to Vas::Video#mark_as_ready in the UpdateProductionFromVas class, but we've removed this as setting this property prevents us from deleting the asset. Also, whether this value if set to true or false is irrelevent for our use case.

In some cases it can happen that an upload never reaches either done or failed. In that case, contact De Videotheek to see if they have any issues on their side.

Workflow

The basic upload flow we've implemented using the VAS API is as following:

At the time of writing an issue has been scheduled to slightly change this workflow. Please check if this issue has been finished and update the docs accordingly.

  1. Select video file to upload for new or existing Production
  2. Create an asset in the VAS API
  3. Retrieve upload_url for the asset from VAS API
  4. Upload video file to upload_url
  5. Save Production as draft
  6. Start background job to check if video has been processed
  7. Mark Production as ready to be published in VMS
  8. User can publish the Production in VMS

1. Select video file to upload for new or existing Production

A video file can be uploaded for either a new or an existing Production. Valid files are .mov and .mp4 and cannot exceed 5GB.

2. Create an asset in the VAS API

When the upload is initiated, we first need to create an asset in the VAS API. This is all handled by vas-video-uploader.js, which does calls to the VasVideoController, which does the actual calls to the VAS API (so we don't leak our API credentials to external users and we can keep all the business logic server side).

3. Retrieve upload_url for the asset from VAS API

With the ID of the created asset we can now fetch a URL to which we need to upload the actual video file. This is also handled by vas-video-uploader.js

4. Upload video file to upload_url

This sends the actual video file to a URL (which is a S3 bucket on AWS, maintained by De Videotheek). This URL contains a time-limited signature, so it can only be used once and within a limited amount of time.

5. Save Production as draft

When the upload has finished, the Production can be saved as draft. This is required as we will need an ID from our database in order to start the UpdateProductionFromVasWorker that checks if the video has been processed and then updates the status of the Production.

After saving as draft the vas_content_status of the Production will be processing. This means the Production cannot be published and we show a few indicators on both the form as the index page to show this to the user.

6. Start background job to see if video has been processed

When the Production has been saved, two workers can be added to the job queue:

  1. UpdateProductionFromVasWorker
  2. RemoveVasVideoWorker

The UpdateProductionFromVasWorker is always added to the queue, and will check whether the upload has finished processing and if so, update the Production accordingly.

The RemoveVasVideoWorker is only added to the queue when the Production already had a non-empty vas_content_id before saving (meaning a video file was uploaded previously). This worker will remove the previously uploaded file from the VAS.

7. Mark Production as ready to be published in VMS

When the UpdateProductionFromVasWorker finds the processing in VAS has finished successfully, it will update the Production with the following data from VAS:

  • video_time
  • adaptive_hls_video_url
  • screenshot_imgix_src
  • ooyala_content_id (optional)
  • Set vas_content_status to done

See commit [ddfc9079](https://github.com/my-channels/mychannels-dpg-api/commit/ddfc9079f69d1be3ec3838f026a671e12d76ebd8) to explain why we sometimes set the video_time to 1 when the VAS reports it's actually 0.

8. User can publish the Production in VMS

When vas_content_status is done, a VmsUser can publish the Production and all indicators of the Production being in progress are removed.

When for an existing published Production new content is uploaded, the Production can remain to be published, as there's still content to serve while the upload has not been processed yet (the adaptive_hls_video_url has not changed yet). This way, you don't need to re-publish an already published Production when you upload a new video file.

Common errors in VAS calls

These are some common errors we've seen while running this functionality. In case any of these errors has been fixed permanently, please feel free to remove them from here.

1. RestClient::TooManyRequests

Incidents

Cause

We're sending too many request to the VAS API.

Possible solution

These incidents are all from actions that are being executed independently from each other. Therefor it could be hard to fix this issue, as we can't simply add a sleep to the tasks to prevent less requests being sent to the server.

Also, I don't think we're sending a lot of requests anyway and it will be hard to reduce the number of requests we're currently sending, as all of them are required.

The best solution might be to check with De Videotheek if they can increase our current rate-limit.

2. URI::InvalidURIError

Incidents

Cause

This happens when a VmsUser uses the "Import from VAS" function and instead of entering a valid content_id like 95212c84-86b7-4277-85e9-0c20cf4ad2e6 enters a random string (e.g. Lenny en Jolien ontmoeten elkaar voor het eerst).

Possible solution

A cool solution would be to use the search endpoint in the VAS API to find the content_id that matches with the string the user entered (which is usually the title of the video they want to upload). This might not always be feasible though, as the search endpoint might return more than one result for a given title.

A quick solution would be to add a regex-validation to the input field to check if the input matches the format of a valid content_id and to show an error message hinting what a correct content_id should look like if it doesn't.

3. RestClient::InternalServerError

Incidents

Cause

Usually this is an issue with the VAS API itself. Unfortunately the API sometimes throws 500 errors which cannot be explained by us, let alone prevented. If you look at some samples of these incidents, you can see most of them are temporary issues, which were fixed automatically over time.

There exist a few similar incidents like RestClient::GatewayTimeout, that are also caused by VAS API itself, and not by us.

Possible solution

Can't really solve this. But when there are a lot of 500 errors in a short timeframe, it's a good idea to check with De Videotheek to see if they are experiencing any errors.

4. RestClient::BadRequest

Incidents

Cause

This happens when we try to delete a video that has been marked in the VAS as ready_for_use: true. Videos which are marked as such cannot be deleted.

The VAS will then respond with the following message:

{
    "message": "The video could not be deleted as it is ready for use."
}

Possible solution

This is tricky to prevent, as we don't know if a video has been marked as ready_for_use: true in the VAS. You can set this value through the video/:id/mark_as_ready endpoint in the API, but we're not using this anymore. (We used to, but that's a long time ago.) Also, there's no endpoint to change the value back to false again.

This error keeps occuring every now and then, so the best solution is to check with De Videotheek why a particular video has been marked as such. I usually just remove the failing RemoveVasVideoWorker job from the queue. This means the video will not be removed, but that's not a big deal anyway.

Check the documentation of Vas::ReplaceVideo above to read the rationale about removing videos from the VAS

Edit this page on GitHub
Last Updated: 3/28/2019, 10:30:15 AM

←