Unverified Commit 0305db28 authored by Jelle Besseling's avatar Jelle Besseling Committed by GitHub
Browse files

Add support for saving video files to object storage (#4290)

* Add support for saving video files to object storage

* Add support for custom url generation on s3 stored files

Uses two config keys to support url generation that doesn't directly go
to (compatible s3). Can be used to generate urls to any cache server or
CDN.

* Upload files to s3 concurrently and delete originals afterwards

* Only publish after move to object storage is complete

* Use base url instead of url template

* Fix mistyped config field

* Add rudenmentary way to download before transcode

* Implement Chocobozzz suggestions

https://github.com/Chocobozzz/PeerTube/pull/4290#issuecomment-891670478

The remarks in question:
    Try to use objectStorage prefix instead of s3 prefix for your function/variables/config names
    Prefer to use a tree for the config: s3.streaming_playlists_bucket -> object_storage.streaming_playlists.bucket
    Use uppercase for config: S3.STREAMING_PLAYLISTS_BUCKETINFO.bucket -> OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET (maybe BUCKET_NAME instead of BUCKET)
    I suggest to rename moveJobsRunning to pendingMovingJobs (or better, create a dedicated videoJobInfo table with a pendingMove & videoId columns so we could also use this table to track pending transcoding jobs)
    https://github.com/Chocobozzz/PeerTube/pull/4290/files#diff-3e26d41ca4bda1de8e1747af70ca2af642abcc1e9e0bfb94239ff2165acfbde5R19

 uses a string instead of an integer
    I think we should store the origin object storage URL in fileUrl, without base_url injection. Instead, inject the base_url at "runtime" so admins can easily change this configuration without running a script to update DB URLs

* Import correct function

* Support multipart upload

* Remove import of node 15.0 module stream/promises

* Extend maximum upload job length

Using the same value as for redundancy downloading seems logical

* Use dynamic part size for really large uploads

Also adds very small part size for local testing

* Fix decreasePendingMove query

* Resolve various PR comments

* Move to object storage after optimize

* Make upload size configurable and increase default

* Prune webtorrent files that are stored in object storage

* Move files after transcoding jobs

* Fix federation

* Add video path manager

* Support move to external storage job in client

* Fix live object storage tests
Co-authored-by: Chocobozzz's avatarChocobozzz <me@florianbigard.com>
parent f88ae8f5
......@@ -31,6 +31,11 @@ jobs:
ports:
- 10389:10389
s3ninja:
image: scireum/s3-ninja
ports:
- 9444:9000
strategy:
fail-fast: false
matrix:
......@@ -40,6 +45,7 @@ jobs:
PGUSER: peertube
PGHOST: localhost
NODE_PENDING_JOB_WAIT: 250
ENABLE_OBJECT_STORAGE_TESTS: true
steps:
- uses: actions/checkout@v2
......
......@@ -36,7 +36,8 @@ export class JobsComponent extends RestTable implements OnInit {
'video-live-ending',
'video-redundancy',
'video-transcoding',
'videos-views'
'videos-views',
'move-to-object-storage'
]
jobs: Job[] = []
......
......@@ -6,6 +6,10 @@
The video is being transcoded, it may not work properly.
</div>
<div i18n class="alert alert-warning" *ngIf="isVideoToMoveToExternalStorage()">
The video is being moved to an external server, it may not work properly.
</div>
<div i18n class="alert alert-info" *ngIf="hasVideoScheduledPublication()">
This video will be published on {{ video.scheduledUpdate.updateAt | date: 'full' }}.
</div>
......
......@@ -18,6 +18,10 @@ export class VideoAlertComponent {
return this.video && this.video.state.id === VideoState.TO_IMPORT
}
isVideoToMoveToExternalStorage () {
return this.video && this.video.state.id === VideoState.TO_MOVE_TO_EXTERNAL_STORAGE
}
hasVideoScheduledPublication () {
return this.video && this.video.scheduledUpdate !== undefined
}
......
......@@ -95,6 +95,39 @@ storage:
# If not, peertube will fallback to the default fil
client_overrides: 'storage/client-overrides/'
object_storage:
enabled: false
# Without protocol, will default to HTTPS
endpoint: '' # 's3.amazonaws.com' or 's3.fr-par.scw.cloud' for example
region: 'us-east-1'
credentials:
# You can also use AWS_ACCESS_KEY_ID env variable
access_key_id: ''
# You can also use AWS_SECRET_ACCESS_KEY env variable
secret_access_key: ''
# Maximum amount to upload in one request to object storage
max_upload_part: 2GB
streaming_playlists:
bucket_name: 'streaming-playlists'
# Allows setting all buckets to the same value but with a different prefix
prefix: '' # Example: 'streaming-playlists:'
# Base url for object URL generation, scheme and host will be replaced by this URL
# Useful when you want to use a CDN/external proxy
base_url: '' # Example: 'https://mirror.example.com'
# Same settings but for webtorrent videos
videos:
bucket_name: 'videos'
prefix: ''
base_url: ''
log:
level: 'info' # 'debug' | 'info' | 'warn' | 'error'
rotation:
......
......@@ -93,6 +93,39 @@ storage:
# If not, peertube will fallback to the default file
client_overrides: '/var/www/peertube/storage/client-overrides/'
object_storage:
enabled: false
# Without protocol, will default to HTTPS
endpoint: '' # 's3.amazonaws.com' or 's3.fr-par.scw.cloud' for example
region: 'us-east-1'
credentials:
# You can also use AWS_ACCESS_KEY_ID env variable
access_key_id: ''
# You can also use AWS_SECRET_ACCESS_KEY env variable
secret_access_key: ''
# Maximum amount to upload in one request to object storage
max_upload_part: 2GB
streaming_playlists:
bucket_name: 'streaming-playlists'
# Allows setting all buckets to the same value but with a different prefix
prefix: '' # Example: 'streaming-playlists:'
# Base url for object URL generation, scheme and host will be replaced by this URL
# Useful when you want to use a CDN/external proxy
base_url: '' # Example: 'https://mirror.example.com'
# Same settings but for webtorrent videos
videos:
bucket_name: 'videos'
prefix: ''
base_url: ''
log:
level: 'info' # 'debug' | 'info' | 'warn' | 'error'
rotation:
......
......@@ -73,6 +73,7 @@
"swagger-cli": "swagger-cli"
},
"dependencies": {
"@aws-sdk/client-s3": "^3.23.0",
"@uploadx/core": "^4.4.0",
"async": "^3.0.1",
"async-lru": "^1.1.1",
......
......@@ -89,9 +89,10 @@ elif [ "$1" = "api-4" ]; then
moderationFiles=$(findTestFiles ./dist/server/tests/api/moderation)
redundancyFiles=$(findTestFiles ./dist/server/tests/api/redundancy)
objectStorageFiles=$(findTestFiles ./dist/server/tests/api/object-storage)
activitypubFiles=$(findTestFiles ./dist/server/tests/api/activitypub)
MOCHA_PARALLEL=true TS_NODE_FILES=true runTest "$1" 2 $moderationFiles $redundancyFiles $activitypubFiles
MOCHA_PARALLEL=true TS_NODE_FILES=true runTest "$1" 2 $moderationFiles $redundancyFiles $activitypubFiles $objectStorageFiles
elif [ "$1" = "external-plugins" ]; then
npm run build:server
......
......@@ -6,9 +6,10 @@ import { VideoModel } from '../server/models/video/video'
import { initDatabaseModels } from '../server/initializers/database'
import { JobQueue } from '../server/lib/job-queue'
import { computeResolutionsToTranscode } from '@server/helpers/ffprobe-utils'
import { VideoTranscodingPayload } from '@shared/models'
import { VideoState, VideoTranscodingPayload } from '@shared/models'
import { CONFIG } from '@server/initializers/config'
import { isUUIDValid } from '@server/helpers/custom-validators/misc'
import { addTranscodingJob } from '@server/lib/video'
program
.option('-v, --video [videoUUID]', 'Video UUID')
......@@ -47,7 +48,7 @@ async function run () {
if (!video) throw new Error('Video not found.')
const dataInput: VideoTranscodingPayload[] = []
const { resolution } = await video.getMaxQualityResolution()
const resolution = video.getMaxQualityFile().resolution
// Generate HLS files
if (options.generateHls || CONFIG.TRANSCODING.WEBTORRENT.ENABLED === false) {
......@@ -62,6 +63,7 @@ async function run () {
resolution,
isPortraitMode: false,
copyCodecs: false,
isNewVideo: false,
isMaxQuality: false
})
}
......@@ -87,10 +89,13 @@ async function run () {
}
}
await JobQueue.Instance.init()
JobQueue.Instance.init()
video.state = VideoState.TO_TRANSCODE
await video.save()
for (const d of dataInput) {
await JobQueue.Instance.createJobWithPromise({ type: 'video-transcoding', payload: d })
await addTranscodingJob(d, {})
console.log('Transcoding job for video %s created.', video.uuid)
}
}
import { registerTSPaths } from '../server/helpers/register-ts-paths'
registerTSPaths()
import { getDurationFromVideoFile, getVideoFileBitrate, getVideoFileFPS, getVideoFileResolution } from '../server/helpers/ffprobe-utils'
import { VideoModel } from '../server/models/video/video'
import { optimizeOriginalVideofile } from '../server/lib/transcoding/video-transcoding'
import { initDatabaseModels } from '../server/initializers/database'
import { basename, dirname } from 'path'
import { copy, move, remove } from 'fs-extra'
import { basename, dirname } from 'path'
import { createTorrentAndSetInfoHash } from '@server/helpers/webtorrent'
import { getVideoFilePath } from '@server/lib/video-paths'
import { CONFIG } from '@server/initializers/config'
import { processMoveToObjectStorage } from '@server/lib/job-queue/handlers/move-to-object-storage'
import { VideoPathManager } from '@server/lib/video-path-manager'
import { getMaxBitrate } from '@shared/core-utils'
import { MoveObjectStoragePayload } from '@shared/models'
import { getDurationFromVideoFile, getVideoFileBitrate, getVideoFileFPS, getVideoFileResolution } from '../server/helpers/ffprobe-utils'
import { initDatabaseModels } from '../server/initializers/database'
import { optimizeOriginalVideofile } from '../server/lib/transcoding/video-transcoding'
import { VideoModel } from '../server/models/video/video'
run()
.then(() => process.exit(0))
......@@ -39,43 +42,49 @@ async function run () {
currentVideoId = video.id
for (const file of video.VideoFiles) {
currentFilePath = getVideoFilePath(video, file)
const [ videoBitrate, fps, dataResolution ] = await Promise.all([
getVideoFileBitrate(currentFilePath),
getVideoFileFPS(currentFilePath),
getVideoFileResolution(currentFilePath)
])
const maxBitrate = getMaxBitrate({ ...dataResolution, fps })
const isMaxBitrateExceeded = videoBitrate > maxBitrate
if (isMaxBitrateExceeded) {
console.log(
'Optimizing video file %s with bitrate %s kbps (max: %s kbps)',
basename(currentFilePath), videoBitrate / 1000, maxBitrate / 1000
)
const backupFile = `${currentFilePath}_backup`
await copy(currentFilePath, backupFile)
await optimizeOriginalVideofile(video, file)
// Update file path, the video filename changed
currentFilePath = getVideoFilePath(video, file)
const originalDuration = await getDurationFromVideoFile(backupFile)
const newDuration = await getDurationFromVideoFile(currentFilePath)
if (originalDuration === newDuration) {
console.log('Finished optimizing %s', basename(currentFilePath))
await remove(backupFile)
continue
await VideoPathManager.Instance.makeAvailableVideoFile(video, file, async path => {
currentFilePath = path
const [ videoBitrate, fps, dataResolution ] = await Promise.all([
getVideoFileBitrate(currentFilePath),
getVideoFileFPS(currentFilePath),
getVideoFileResolution(currentFilePath)
])
const maxBitrate = getMaxBitrate({ ...dataResolution, fps })
const isMaxBitrateExceeded = videoBitrate > maxBitrate
if (isMaxBitrateExceeded) {
console.log(
'Optimizing video file %s with bitrate %s kbps (max: %s kbps)',
basename(currentFilePath), videoBitrate / 1000, maxBitrate / 1000
)
const backupFile = `${currentFilePath}_backup`
await copy(currentFilePath, backupFile)
await optimizeOriginalVideofile(video, file)
// Update file path, the video filename changed
currentFilePath = VideoPathManager.Instance.getFSVideoFileOutputPath(video, file)
const originalDuration = await getDurationFromVideoFile(backupFile)
const newDuration = await getDurationFromVideoFile(currentFilePath)
if (originalDuration === newDuration) {
console.log('Finished optimizing %s', basename(currentFilePath))
await remove(backupFile)
return
}
console.log('Failed to optimize %s, restoring original', basename(currentFilePath))
await move(backupFile, currentFilePath, { overwrite: true })
await createTorrentAndSetInfoHash(video, file)
await file.save()
}
})
}
console.log('Failed to optimize %s, restoring original', basename(currentFilePath))
await move(backupFile, currentFilePath, { overwrite: true })
await createTorrentAndSetInfoHash(video, file)
await file.save()
}
if (CONFIG.OBJECT_STORAGE.ENABLED === true) {
await processMoveToObjectStorage({ data: { videoUUID: video.uuid } as MoveObjectStoragePayload } as any)
}
}
......
import * as express from 'express'
import { move } from 'fs-extra'
import { basename } from 'path'
import { getLowercaseExtension } from '@server/helpers/core-utils'
import { deleteResumableUploadMetaFile, getResumableUploadPath } from '@server/helpers/upload'
import { uuidToShort } from '@server/helpers/uuid'
import { createTorrentAndSetInfoHash } from '@server/helpers/webtorrent'
import { getLocalVideoActivityPubUrl } from '@server/lib/activitypub/url'
import { addOptimizeOrMergeAudioJob, buildLocalVideoFromReq, buildVideoThumbnailsFromReq, setVideoTags } from '@server/lib/video'
import { generateWebTorrentVideoFilename, getVideoFilePath } from '@server/lib/video-paths'
import { generateWebTorrentVideoFilename } from '@server/lib/paths'
import {
addMoveToObjectStorageJob,
addOptimizeOrMergeAudioJob,
buildLocalVideoFromReq,
buildVideoThumbnailsFromReq,
setVideoTags
} from '@server/lib/video'
import { VideoPathManager } from '@server/lib/video-path-manager'
import { buildNextVideoState } from '@server/lib/video-state'
import { openapiOperationDoc } from '@server/middlewares/doc'
import { MVideo, MVideoFile, MVideoFullLight } from '@server/types/models'
import { uploadx } from '@uploadx/core'
......@@ -139,23 +148,20 @@ async function addVideo (options: {
const videoData = buildLocalVideoFromReq(videoInfo, videoChannel.id)
videoData.state = CONFIG.TRANSCODING.ENABLED
? VideoState.TO_TRANSCODE
: VideoState.PUBLISHED
videoData.state = buildNextVideoState()
videoData.duration = videoPhysicalFile.duration // duration was added by a previous middleware
const video = new VideoModel(videoData) as MVideoFullLight
video.VideoChannel = videoChannel
video.url = getLocalVideoActivityPubUrl(video) // We use the UUID, so set the URL after building the object
const videoFile = await buildNewFile(video, videoPhysicalFile)
const videoFile = await buildNewFile(videoPhysicalFile)
// Move physical file
const destination = getVideoFilePath(video, videoFile)
const destination = VideoPathManager.Instance.getFSVideoFileOutputPath(video, videoFile)
await move(videoPhysicalFile.path, destination)
// This is important in case if there is another attempt in the retry process
videoPhysicalFile.filename = getVideoFilePath(video, videoFile)
videoPhysicalFile.filename = basename(destination)
videoPhysicalFile.path = destination
const [ thumbnailModel, previewModel ] = await buildVideoThumbnailsFromReq({
......@@ -210,9 +216,13 @@ async function addVideo (options: {
createTorrentFederate(video, videoFile)
.then(() => {
if (video.state !== VideoState.TO_TRANSCODE) return
if (video.state === VideoState.TO_MOVE_TO_EXTERNAL_STORAGE) {
return addMoveToObjectStorageJob(video)
}
return addOptimizeOrMergeAudioJob(videoCreated, videoFile, user)
if (video.state === VideoState.TO_TRANSCODE) {
return addOptimizeOrMergeAudioJob(videoCreated, videoFile, user)
}
})
.catch(err => logger.error('Cannot add optimize/merge audio job for %s.', videoCreated.uuid, { err, ...lTags(videoCreated.uuid) }))
......@@ -227,7 +237,7 @@ async function addVideo (options: {
})
}
async function buildNewFile (video: MVideo, videoPhysicalFile: express.VideoUploadFile) {
async function buildNewFile (videoPhysicalFile: express.VideoUploadFile) {
const videoFile = new VideoFileModel({
extname: getLowercaseExtension(videoPhysicalFile.filename),
size: videoPhysicalFile.size,
......
......@@ -3,9 +3,9 @@ import * as express from 'express'
import { logger } from '@server/helpers/logger'
import { VideosTorrentCache } from '@server/lib/files-cache/videos-torrent-cache'
import { Hooks } from '@server/lib/plugins/hooks'
import { getVideoFilePath } from '@server/lib/video-paths'
import { VideoPathManager } from '@server/lib/video-path-manager'
import { MStreamingPlaylist, MVideo, MVideoFile, MVideoFullLight } from '@server/types/models'
import { HttpStatusCode, VideoStreamingPlaylistType } from '@shared/models'
import { HttpStatusCode, VideoStorage, VideoStreamingPlaylistType } from '@shared/models'
import { STATIC_DOWNLOAD_PATHS } from '../initializers/constants'
import { asyncMiddleware, videosDownloadValidator } from '../middlewares'
......@@ -81,7 +81,15 @@ async function downloadVideoFile (req: express.Request, res: express.Response) {
if (!checkAllowResult(res, allowParameters, allowedResult)) return
return res.download(getVideoFilePath(video, videoFile), `${video.name}-${videoFile.resolution}p${videoFile.extname}`)
if (videoFile.storage === VideoStorage.OBJECT_STORAGE) {
return res.redirect(videoFile.getObjectStorageUrl())
}
await VideoPathManager.Instance.makeAvailableVideoFile(video, videoFile, path => {
const filename = `${video.name}-${videoFile.resolution}p${videoFile.extname}`
return res.download(path, filename)
})
}
async function downloadHLSVideoFile (req: express.Request, res: express.Response) {
......@@ -107,8 +115,15 @@ async function downloadHLSVideoFile (req: express.Request, res: express.Response
if (!checkAllowResult(res, allowParameters, allowedResult)) return
const filename = `${video.name}-${videoFile.resolution}p-${streamingPlaylist.getStringType()}${videoFile.extname}`
return res.download(getVideoFilePath(streamingPlaylist, videoFile), filename)
if (videoFile.storage === VideoStorage.OBJECT_STORAGE) {
return res.redirect(videoFile.getObjectStorageUrl())
}
await VideoPathManager.Instance.makeAvailableVideoFile(streamingPlaylist, videoFile, path => {
const filename = `${video.name}-${videoFile.resolution}p-${streamingPlaylist.getStringType()}${videoFile.extname}`
return res.download(path, filename)
})
}
function getVideoFile (req: express.Request, files: MVideoFile[]) {
......
......@@ -6,7 +6,8 @@ import { dirname, join } from 'path'
import * as WebTorrent from 'webtorrent'
import { isArray } from '@server/helpers/custom-validators/misc'
import { WEBSERVER } from '@server/initializers/constants'
import { generateTorrentFileName, getVideoFilePath } from '@server/lib/video-paths'
import { generateTorrentFileName } from '@server/lib/paths'
import { VideoPathManager } from '@server/lib/video-path-manager'
import { MVideo } from '@server/types/models/video/video'
import { MVideoFile, MVideoFileRedundanciesOpt } from '@server/types/models/video/video-file'
import { MStreamingPlaylistVideo } from '@server/types/models/video/video-streaming-playlist'
......@@ -78,7 +79,7 @@ async function downloadWebTorrentVideo (target: { magnetUri: string, torrentName
})
}
async function createTorrentAndSetInfoHash (
function createTorrentAndSetInfoHash (
videoOrPlaylist: MVideo | MStreamingPlaylistVideo,
videoFile: MVideoFile
) {
......@@ -95,22 +96,24 @@ async function createTorrentAndSetInfoHash (
urlList: [ videoFile.getFileUrl(video) ]
}
const torrent = await createTorrentPromise(getVideoFilePath(videoOrPlaylist, videoFile), options)
return VideoPathManager.Instance.makeAvailableVideoFile(videoOrPlaylist, videoFile, async videoPath => {
const torrent = await createTorrentPromise(videoPath, options)
const torrentFilename = generateTorrentFileName(videoOrPlaylist, videoFile.resolution)
const torrentPath = join(CONFIG.STORAGE.TORRENTS_DIR, torrentFilename)
logger.info('Creating torrent %s.', torrentPath)
const torrentFilename = generateTorrentFileName(videoOrPlaylist, videoFile.resolution)
const torrentPath = join(CONFIG.STORAGE.TORRENTS_DIR, torrentFilename)
logger.info('Creating torrent %s.', torrentPath)
await writeFile(torrentPath, torrent)
await writeFile(torrentPath, torrent)
// Remove old torrent file if it existed
if (videoFile.hasTorrent()) {
await remove(join(CONFIG.STORAGE.TORRENTS_DIR, videoFile.torrentFilename))
}
// Remove old torrent file if it existed
if (videoFile.hasTorrent()) {
await remove(join(CONFIG.STORAGE.TORRENTS_DIR, videoFile.torrentFilename))
}
const parsedTorrent = parseTorrent(torrent)
videoFile.infoHash = parsedTorrent.infoHash
videoFile.torrentFilename = torrentFilename
const parsedTorrent = parseTorrent(torrent)
videoFile.infoHash = parsedTorrent.infoHash
videoFile.torrentFilename = torrentFilename
})
}
function generateMagnetUri (
......
......@@ -153,6 +153,29 @@ function checkConfig () {
}
}
// Object storage
if (CONFIG.OBJECT_STORAGE.ENABLED === true) {
if (!CONFIG.OBJECT_STORAGE.VIDEOS.BUCKET_NAME) {
return 'videos_bucket should be set when object storage support is enabled.'
}
if (!CONFIG.OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET_NAME) {
return 'streaming_playlists_bucket should be set when object storage support is enabled.'
}
if (
CONFIG.OBJECT_STORAGE.VIDEOS.BUCKET_NAME === CONFIG.OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET_NAME &&
CONFIG.OBJECT_STORAGE.VIDEOS.PREFIX === CONFIG.OBJECT_STORAGE.STREAMING_PLAYLISTS.PREFIX
) {
if (CONFIG.OBJECT_STORAGE.VIDEOS.PREFIX === '') {
return 'Object storage bucket prefixes should be set when the same bucket is used for both types of video.'
} else {
return 'Object storage bucket prefixes should be set to different values when the same bucket is used for both types of video.'
}
}
}
return null
}
......
......@@ -73,6 +73,26 @@ const CONFIG = {
PLUGINS_DIR: buildPath(config.get<string>('storage.plugins')),
CLIENT_OVERRIDES_DIR: buildPath(config.get<string>('storage.client_overrides'))
},
OBJECT_STORAGE: {
ENABLED: config.get<boolean>('object_storage.enabled'),
MAX_UPLOAD_PART: bytes.parse(config.get<string>('object_storage.max_upload_part')),
ENDPOINT: config.get<string>('object_storage.endpoint'),
REGION: config.get<string>('object_storage.region'),
CREDENTIALS: {
ACCESS_KEY_ID: config.get<string>('object_storage.credentials.access_key_id'),
SECRET_ACCESS_KEY: config.get<string>('object_storage.credentials.secret_access_key')
},
VIDEOS: {
BUCKET_NAME: config.get<string>('object_storage.videos.bucket_name'),
PREFIX: config.get<string>('object_storage.videos.prefix'),
BASE_URL: config.get<string>('object_storage.videos.base_url')
},
STREAMING_PLAYLISTS: {
BUCKET_NAME: config.get<string>('object_storage.streaming_playlists.bucket_name'),
PREFIX: config.get<string>('object_storage.streaming_playlists.prefix'),
BASE_URL: config.get<string>('object_storage.streaming_playlists.base_url')
}
},
WEBSERVER: {
SCHEME: config.get<boolean>('webserver.https') === true ? 'https' : 'http',
WS: config.get<boolean>('webserver.https') === true ? 'wss' : 'ws',
......
......@@ -24,7 +24,7 @@ import { CONFIG, registerConfigChangedHandler } from './config'
// ---------------------------------------------------------------------------
const LAST_MIGRATION_VERSION = 655
const LAST_MIGRATION_VERSION = 660
// ---------------------------------------------------------------------------
......@@ -147,7 +147,8 @@ const JOB_ATTEMPTS: { [id in JobType]: number } = {
'videos-views': 1,
'activitypub-refresher': 1,
'video-redundancy': 1,
'video-live-ending': 1
'video-live-ending': 1,
'move-to-object-storage': 3
}
// Excluded keys are jobs that can be configured by admins
const JOB_CONCURRENCY: { [id in Exclude<JobType, 'video-transcoding' | 'video-import'>]: number } = {
......@@ -162,7 +163,8 @@ const JOB_CONCURRENCY: { [id in Exclude<JobType, 'video-transcoding' | 'video-im
'videos-views': 1,
'activitypub-refresher': 1,
'video-redundancy': 1,
'video-live-ending': 10
'video-live-ending': 10,
'move-to-object-storage': 1
}
const JOB_TTL: { [id in JobType]: number } = {
'activitypub-http-broadcast': 60000 * 10, // 10 minutes
......@@ -178,7 +180,8 @@ const JOB_TTL: { [id in JobType]: number } = {
'videos-views': undefined, // Unlimited
'activitypub-refresher': 60000 * 10, // 10 minutes
'video-redundancy': 1000 * 3600 * 3, // 3 hours
'video-live-ending': 1000 * 60 * 10 // 10 minutes
'video-live-ending': 1000 * 60 * 10, // 10 minutes
'move-to-object-storage': 1000 * 60 * 60 * 3 // 3 hours
}
const REPEAT_JOBS: { [ id: string ]: EveryRepeatOptions | CronRepeatOptions } = {
'videos-views': {
......@@ -412,7 +415,8 @@ const VIDEO_STATES: { [ id in VideoState ]: string } = {
[VideoState.TO_TRANSCODE]: 'To transcode',
[VideoState.TO_IMPORT]: 'To import',
[VideoState.WAITING_FOR_LIVE]: 'Waiting for livestream',
[VideoState.LIVE_ENDED]: 'Livestream ended'
[VideoState.LIVE_ENDED]: 'Livestream ended',
[VideoState.TO_MOVE_TO_EXTERNAL_STORAGE]: 'To move to an external storage'