Add support for video captions #915

skjnldsv · 2021-05-28T07:35:10Z

fix #239
(add subtitles for video feature #239)

Generate HTML5 subtitle tracks dynamically based on the contents of the folder
where the video is located.

For example: given a movie 'video.mkv', the player will look for VTT subtitle
files named 'video.XX.vtt' and '.video.XX.vtt', where XX is the two-letter
code for the language.

Signed-off-by: Frederic Ruget [email protected]

douzebis · 2021-05-29T05:51:28Z

fix #239
(add subtitles for video feature #239)

Generate HTML5 subtitle tracks dynamically based on the contents of the folder
where the video is located.

For example: given a movie 'video.mkv', the player will look for VTT subtitle
files named 'video.XX.vtt' and '.video.XX.vtt', where XX is the two-letter
code for the language.

Signed-off-by: Frederic Ruget [email protected]

Thanks a lot @skjnldsv (and @beardhatcode)!

I now have setup a dedicated test environment with a dedicated server separate from my production instance.
Also I have rebuild all from source (server and viewer directly github).

--> I was able to check that the modifications to Videos.vue do not break master.

I hope to make some progress this weekend.

douzebis · 2021-05-30T05:48:06Z

[ Intermediate status report ]

Hi @skjnldsv and @beardhatcode. I believe I have gathered most of the pieces I will need to implement the requested changes and I am now confident that I will be able to get it to completion.

I have installed xdebug so I can breakpoint/watch the server-side code from vscode, plus I use chrome's builtin javascript debugger for the vue code on client side.

I use apps/forms's ApiController.php and SubmissionService.php for inspiration.

Currently I have three new/modified files:

apps/viewer/
- appinfo/routes.php
- lib/Controller/VideoController.php
- src/components/Videos.vue

On client side, I use generateUrl() and axios() to call the server, and I package the filename as a query parameter to the url.

On server side, I use OCP\Files\File to get access to the filesystem utility functions, and I use OCP\IL10N to retrieve factory.this.availableLanguages.

So far I have successfully implemented roundtrip communication between client and server, passing parameters to the server and retrieving the response on client side.

douzebis · 2021-05-30T05:54:20Z

One question still:

From OCP\IL10N I can retrieve a list of abbreviated language names (e.g.: en, fr).
However I was not able to retrieve the unabbreviated language names (e.g.: English, French).
Is there another variable where this information is present?
Or a function I could call from the php controller?

skjnldsv · 2021-05-30T08:14:12Z

Currently I have three new/modified files:

You can push them, so we can review and comment :)

skjnldsv · 2021-05-30T09:00:41Z

From OCP\IL10N I can retrieve a list of abbreviated language names (e.g.: en, fr).

You have those in OCP\L10N\IFactory
https://nextcloud-server.netlify.app/classes/ocp-l10n-ifactory

douzebis · 2021-05-30T09:32:14Z

Currently I have three new/modified files:

You can push them, so we can review and comment :)

OK. I'll try and push later today.

skjnldsv · 2021-05-30T10:09:36Z

OK. I'll try and push later today.

It's sunday, don't pressure yourself! 🤗

douzebis · 2021-05-30T20:04:48Z

I have just pushed to feat/video-caption.
The code is refactored and working :)

A couple of questions as comments in 'VideoController.php' -- about @nnotations (I don't understand what they do).

Looking forward to studying your review.

beardhatcode · 2021-05-30T21:02:08Z

lib/Controller/VideoController.php

+		$rootName = pathinfo($video->getFileInfo()->getPath())['filename'];
+		$rootLen = strlen($rootName);
+		// eg: $rootName is 'movie', $rooLen is 5
+		$files = $video->getParent()->getDirectoryListing();


What does this line return when only the video file is shared? (And not its folder)

Good question.
I made a test with the following setup: Alice shares 'movie.mkv' with John, but does not share 'movie.en.vtt'.
When john open the video for viewing, 'getDirectoryListing()' returns a shorter list of files, including 'movie.mkv', but not 'movie.en.vtt'. As a result, John does not have access to captions.

Does this answer the question? or maybe you had a different setup in mind?

I had the public share setup in mind. I don't know what getParent returns if the video would be shared with a link

Good call for the public side of this feature

@beardhatcode I did a second test today.

Sharing 'movie.mkv' (and only this file) using federation sharing -- john@https://sandbox.nc.com (remote)
There was no problem for john viewing the movie (without captions of course).
Single stepping the php controller,getParent() seems to return the "local" parent folder, not the "remote" parent folder where the file is actually hosted.

Btw, is there a specification of the expected behavior for getParent(), including exception cases and/or error return values?

See the code: https://github.com/nextcloud/server/blob/master/lib/public/Files/Node.php#L234 you can click on getParrent to find all the implementations of it for various types of files

The "normal" one is here: https://github.com/nextcloud/server/blob/f2101487a0a32cd5be5068329a2c7cd6d8a1583f/lib/private/Files/Node/Node.php#L282-L288 , it appears that the root folder is returned whenever the dirname cannot be obtained. I can't figure out what happens when te file is shared, in the share20 folder "setParent" is marked as deprecated ... (@skjnldsv do you know how this works?)

OK. I think it is OK then.
The path is always from the point of view of the local user file space.
Regarding the "normal" implementation, I understand it maps "" (empty path), ".", and "/" to the root directory of the local user. (It does not necessarily mean that the dirname could not be obtained.)

@beardhatcode I believe I took most of the review comments into account.
Is there more to be done about the getParent issue?

src/components/Videos.vue

lib/Controller/VideoController.php

appinfo/routes.php

douzebis · 2021-06-02T18:47:39Z

@beardhatcode @skjnldsv I suppose we want some non-regression test to be added to viewer's automated test suite?

Maybe we can setup a mini video folder, have viewer "render" the video and parse the resulting "Elements" (as in chrome -> inspect). E.g.:

test-video/
- video.mkv (minimal in size)
- video.en.vtt (can be empty)
- .video.fr.vtt (dotted)
- video.someotherextension

-> Parse the resulting HTML5 for correctness: checking video.en.vtt and .video.fr.vtt are present as tracks and nothing else.

Is there a description of NC's test framework?
How can we get viewer to render the video page from within the test?
What other tests would be needed? (if any)

douzebis · 2021-06-02T18:49:38Z

@beardhatcode @skjnldsv what about user documentation?

Should the feature be mentioned in the Github's README for the viewer?

beardhatcode · 2021-06-02T19:55:08Z

@beardhatcode @skjnldsv I suppose we want some non-regression test to be added to viewer's automated test suite?

Yes

Maybe we can setup a mini video folder, have viewer "render" the video and parse the resulting "Elements" (as in chrome -> inspect).

I think we should do this by querying the captions of the plyr object. I think that is the best way to test this.

1. Is there a description of NC's test framework?

It is different for every app, but since you added PHP code, we will also need to add some PHP test (something like this). We'll also need to add video.mkv.spec.js.

2. How can we get viewer to render the video page from within the test?

Yes, see the video.mkv.spec.js

3. What other tests would be needed? (if any)

We should have PHP and cypress tests

beardhatcode · 2021-06-02T20:04:27Z

Oh and you should also test video's and captions with odd names like:

this 🎥 has 🗨️🗨️🗨️ subtitles🕵️‍♀️ %20.mp4
this 🎥 has 🗨️🗨️🗨️ subtitles🕵️‍♀️ %20.en.vtt

lib/Controller/VideoController.php

appinfo/routes.php

beardhatcode · 2022-01-28T20:10:15Z

I have been thinking, wouldn't it be better to use the fileid in the API (no escaping issues).

And, with @CarlSchwan search suggestion we can try all the locales :). I was also wondering what:

		$video = $this->rootFolder
			->getUserFolder($this->userSession->getUser()->getUID())

gives on public shares.

I might continue working on this over the next month or so (if not, and you are reading this in 2 months, feel free to continue the work 😅)

Generate HTML5 subtitle tracks dynamically based on the contents of the folder where the video is located. For example: given a movie 'video.mkv', the player will look for VTT subtitle files named 'video.XX.vtt' and '.video.XX.vtt', where XX is the two-letter code for the language. Signed-off-by: Frederic Ruget <[email protected]>

Signed-off-by: Frederic Ruget <[email protected]>

Instead fetchTracks() can be called from options(). Signed-off-by: Frederic Ruget <[email protected]>

Signed-off-by: Frederic Ruget <[email protected]>

Property track was not properly declared, and sometimes was not reactive. E.g. closing and re-opening the player would make subtitles disappear. Fixing entails declaring tracks in the data() function associated with the Videos vue. Signed-off-by: Frederic Ruget <[email protected]>

(Templates are not made to call methods.) Signed-off-by: Frederic Ruget <[email protected]>

@skjnldsv

Move folder-analysis and mapping-of-locales-to-languages code to server-side. Big thanks to @skjnldsv and @beardhatcode for their help and guidance. Signed-off-by: Frederic Ruget <[email protected]>

Signed-off-by: Frederic Ruget <[email protected]>

If there are 10k+ files in the folder, we prefer not to have to "manually" iterate through all or them. Prefiltering by locale allows to limit the search to approximately 400 files. Signed-off-by: Frederic Ruget <[email protected]>

Signed-off-by: Frederic Ruget <[email protected]>

Signed-off-by: Carl Schwan <[email protected]>

MrRinkana · 2023-01-09T03:10:09Z

I think it would be awesome if you could add support for looking for these .vtt files in hidden directories of the same name as the video file.

I.e. :

"aVideoName.mp4"
".aVideoName" (a hidden folder that contains the "aVideoName.en.vtt" etc.)

I believe that there is benefit to having auxillary files to the videos (captions, thumbnail and such) all be hidden and grouped, the first to not have extra things visible in the files view, the second to not create chaos when people choose to view hidden files to add or change the auxillary files.

Imagine 6 translations per video + thumbnail etc. - unless every video is in its own directory, which I don't think many have, this will be hard to navigate.

I believe supporting this subdirectory approach is the way to go, also for thumbnails (and different qualities if that gets implemented) but that's not relevant to this pull req.

skjnldsv mentioned this pull request May 28, 2021

Add support for video captions #910

Closed

skjnldsv added 2. developing Work in progress enhancement New feature or request labels May 28, 2021

skjnldsv added this to the Nextcloud 23 milestone May 28, 2021

beardhatcode reviewed May 30, 2021

View reviewed changes