Mailing List Archive

I have added some videos and do a rescan. Some videos are not seen or added to the list. I can rename those videos and they are still not seen on a rescan. I can put them in a different subdirectory and they are still not seen.

On 08/08/17 21:41, Mark Goldberg wrote: > I have added some videos and do a rescan. Some videos are not seen or > added to the list. I can rename those videos and they are still not seen > on a rescan. I can put them in a different subdirectory and they are > still not seen. > > This is on mythtv 0.28.1-3 from rpmfusion on Fedora 26.r > > Any suggestions how to troubleshoot this? > > Thanks, > > Mark

Have you tried scanning an empty videos folder first? AIUI a video is not added if its hash is already present.

On 08/08/17 22:35, John Pilkington wrote: > On 08/08/17 21:41, Mark Goldberg wrote: >> I have added some videos and do a rescan. Some videos are not seen or >> added to the list. I can rename those videos and they are still not >> seen on a rescan. I can put them in a different subdirectory and they >> are still not seen. >> >> This is on mythtv 0.28.1-3 from rpmfusion on Fedora 26.r >> >> Any suggestions how to troubleshoot this? >> >> Thanks, >> >> Mark > > Have you tried scanning an empty videos folder first? AIUI a video is > not added if its hash is already present.

I'm not sure that fits what you say, though. The Video path can be set up either via Storage Groups in mythtvsetup or in the frontend via Setup > Media Settings. A mismatch?

On 8/8/17 4:41 PM, Mark Goldberg wrote: > I have added some videos and do a rescan. Some videos are not seen or > added to the list. I can rename those videos and they are still not > seen on a rescan. I can put them in a different subdirectory and they > are still not seen. One other thing to look at: do you use "parental levels" for videos? If so, the videos might have already been added, but are hidden because your default parental level is lower than that of the missing videos. _______________________________________________ mythtv-users mailing list mythtv-users@mythtv.org http://lists.mythtv.org/mailman/listinfo/mythtv-usershttp://wiki.mythtv.org/Mailing_List_etiquetteMythTV Forums: https://forum.mythtv.org

On Wed, Aug 9, 2017 at 5:55 AM, Dan Wilga wrote: > > On 8/8/17 4:41 PM, Mark Goldberg wrote: >> >> I have added some videos and do a rescan. Some videos are not seen or added to the list. I can rename those videos and they are still not seen on a rescan. I can put them in a different subdirectory and they are still not seen. > > One other thing to look at: do you use "parental levels" for videos? If so, the videos might have already been added, but are hidden because your default parental level is lower than that of the missing videos.

On 08/09/17, John Pilkington wrote: >On 08/08/17 22:35, John Pilkington wrote: >> On 08/08/17 21:41, Mark Goldberg wrote: >>> I have added some videos and do a rescan. Some videos are not seen or >>> added to the list. I can rename those videos and they are still not >>> seen on a rescan. I can put them in a different subdirectory and they >>> are still not seen. >>> >>> This is on mythtv 0.28.1-3 from rpmfusion on Fedora 26.r >>> >>> Any suggestions how to troubleshoot this? >>> >>> Thanks, >>> >>> Mark >> >> Have you tried scanning an empty videos folder first? AIUI a video is >> not added if its hash is already present. > >I'm not sure that fits what you say, though. The Video path can be set >up either via Storage Groups in mythtvsetup or in the frontend via Setup > Media Settings. A mismatch?

I have videos set up in mythsetup for years. They are in one directory, /archives/videos. In a subdirectory, /archives/videos/hdv, I am creating h.264 / PCM files with names like MVI_6007_5994_18D.MOV MVI_6007_5994_18E.MOV MVI_6007_5994_18F.MOV MVI_6007_5994_18G.MOV with ffmpeg, trying different encoding parameters.

I do a rescan, and I can see the first two files but not the second two. All are original files, not downloaded from anywhere and have no matching metadata.

Mediainfo for the files seen and not seen is the same except for the filename.

I can do a "touch junk.mov" in that subdirectory, and that is seen after a scan, but it obviously complains when trying to play it as it is empty.

I hope this explains things better.

I am not sure how to scan an empty video folder. There is just one option, rescan the whole directory.

On 09/08/17 14:25, Mark Goldberg wrote: > On Wed, Aug 9, 2017 at 5:55 AM, Dan Wilga wrote: > > > > On 8/8/17 4:41 PM, Mark Goldberg wrote: > >> > >> I have added some videos and do a rescan. Some videos are not seen > or added to the list. I can rename those videos and they are still not > seen on a rescan. I can put them in a different subdirectory and they > are still not seen. > > > > One other thing to look at: do you use "parental levels" for videos? > If so, the videos might have already been added, but are hidden because > your default parental level is lower than that of the missing videos. > > > On 08/09/17, John Pilkington wrote: > >On 08/08/17 22:35, John Pilkington wrote: > >> On 08/08/17 21:41, Mark Goldberg wrote: > >>> I have added some videos and do a rescan. Some videos are not seen or > >>> added to the list. I can rename those videos and they are still not > >>> seen on a rescan. I can put them in a different subdirectory and they > >>> are still not seen. > >>> > >>> This is on mythtv 0.28.1-3 from rpmfusion on Fedora 26.r > >>> > >>> Any suggestions how to troubleshoot this? > >>> > >>> Thanks, > >>> > >>> Mark > >> > >> Have you tried scanning an empty videos folder first? AIUI a video is > >> not added if its hash is already present. > > > >I'm not sure that fits what you say, though. The Video path can be set > >up either via Storage Groups in mythtvsetup or in the frontend via Setup > > Media Settings. A mismatch? > > I have videos set up in mythsetup for years. They are in one > directory, /archives/videos. > In a subdirectory, /archives/videos/hdv, I am creating h.264 / PCM files > with names like > MVI_6007_5994_18D.MOV > MVI_6007_5994_18E.MOV > MVI_6007_5994_18F.MOV > MVI_6007_5994_18G.MOV > with ffmpeg, trying different encoding parameters. > > I do a rescan, and I can see the first two files but not the second two. > All are original files, not downloaded from anywhere and have no > matching metadata. > > Mediainfo for the files seen and not seen is the same except for the > filename. > > I can do a "touch junk.mov" in that subdirectory, and that is seen after > a scan, but it obviously complains when trying to play it as it is empty. > > I hope this explains things better. > > I am not sure how to scan an empty video folder. There is just one > option, rescan the whole directory. > > I am not using parental levels. > > Mark

Create an empty directory, define it as your video source in mythtvsetup-or-media-files, and scan it. Then go back to your original configuration and rescan. I don't think this has changed recently, but Ticket #10500 was perhaps similar. Or search 'mythtv video hash'

On 09/08/17 15:04, John Pilkington wrote: > On 09/08/17 14:25, Mark Goldberg wrote: >> On Wed, Aug 9, 2017 at 5:55 AM, Dan Wilga wrote: >> > >> > On 8/8/17 4:41 PM, Mark Goldberg wrote: >> >> >> >> I have added some videos and do a rescan. Some videos are not seen or added to the list. I can >> rename those videos and they are still not seen on a rescan. I can put them in a different >> subdirectory and they are still not seen. >> > >> > One other thing to look at: do you use "parental levels" for videos? If so, the videos might >> have already been added, but are hidden because your default parental level is lower than that of >> the missing videos. >> >> >> On 08/09/17, John Pilkington wrote: >> >On 08/08/17 22:35, John Pilkington wrote: >> >> On 08/08/17 21:41, Mark Goldberg wrote: >> >>> I have added some videos and do a rescan. Some videos are not seen or >> >>> added to the list. I can rename those videos and they are still not >> >>> seen on a rescan. I can put them in a different subdirectory and they >> >>> are still not seen. >> >>> >> >>> This is on mythtv 0.28.1-3 from rpmfusion on Fedora 26.r >> >>> >> >>> Any suggestions how to troubleshoot this? >> >>> >> >>> Thanks, >> >>> >> >>> Mark >> >> >> >> Have you tried scanning an empty videos folder first? AIUI a video is >> >> not added if its hash is already present. >> > >> >I'm not sure that fits what you say, though. The Video path can be set >> >up either via Storage Groups in mythtvsetup or in the frontend via Setup >> > Media Settings. A mismatch? >> >> I have videos set up in mythsetup for years. They are in one directory, /archives/videos. >> In a subdirectory, /archives/videos/hdv, I am creating h.264 / PCM files with names like >> MVI_6007_5994_18D.MOV >> MVI_6007_5994_18E.MOV >> MVI_6007_5994_18F.MOV >> MVI_6007_5994_18G.MOV >> with ffmpeg, trying different encoding parameters. >> >> I do a rescan, and I can see the first two files but not the second two. All are original files, >> not downloaded from anywhere and have no matching metadata. >> >> Mediainfo for the files seen and not seen is the same except for the filename. >> >> I can do a "touch junk.mov" in that subdirectory, and that is seen after a scan, but it obviously >> complains when trying to play it as it is empty. >> >> I hope this explains things better. >> >> I am not sure how to scan an empty video folder. There is just one option, rescan the whole >> directory. >> >> I am not using parental levels. >> >> Mark > > Create an empty directory, define it as your video source in mythtvsetup-or-media-files, and scan > it. Then go back to your original configuration and rescan. I don't think this has changed > recently, but > Ticket #10500 was perhaps similar. Or search 'mythtv video hash' > > Perhaps your ffmpeg processes gave identical results. > Clarify: "define it as your *only* video source".

Yes, this is most likely the issue. The hash that's created to identify unique videos is done with only a small part of the file at the beginning and the end (because reading multiple gigabytes per file from disk and running the entire video through a hash algorithm would take ages to complete). It's likely your test files are just too close to the same, so are being considered the same and only showed once (one or the other is shown and the one not shown is ignored).

There's no way to "fix" this as it's working as designed (and typical use cases don't involve multiple almost identical files). So, you can rotate them in and out to see them all. Note, too, that the title of the video shown is not necessarily the name of the file being played--you'd have to look at the filename in the detailed metadata screen to know for sure which you're playing--and if you swap files the new file may show with the same title, even though it would now play the other file.

You may be able to work around this by creating test encodes of differing lengths/portions of the video.

> On 08/09/2017 10:04 AM, John Pilkington wrote: >> Or search 'mythtv video hash' >> >> Perhaps your ffmpeg processes gave identical results. > > Yes, this is most likely the issue. The hash that's created to identify > unique videos is done with only a small part of the file at the > beginning and the end (because reading multiple gigabytes per file from > disk and running the entire video through a hash algorithm would take > ages to complete). It's likely your test files are just too close to > the same, so are being considered the same and only showed once (one or > the other is shown and the one not shown is ignored). > > There's no way to "fix" this as it's working as designed (and typical > use cases don't involve multiple almost identical files).

I wonder if the hash algorithm could be updated to include the file creation date or something else similar to help mitigate this type of overlap?

> So, you can > rotate them in and out to see them all. Note, too, that the title of > the video shown is not necessarily the name of the file being > played--you'd have to look at the filename in the detailed metadata > screen to know for sure which you're playing--and if you swap files the > new file may show with the same title, even though it would now play the > other file. > > You may be able to work around this by creating test encodes of > differing lengths/portions of the video. > > Mike >

On 08/09/2017 05:05 PM, Mark Perkins wrote: > On 10 August 2017 12:01:01 am "Michael T. Dean" wrote: > >> On 08/09/2017 10:04 AM, John Pilkington wrote: >>> Or search 'mythtv video hash' >>> >>> Perhaps your ffmpeg processes gave identical results. >> Yes, this is most likely the issue. The hash that's created to identify >> unique videos is done with only a small part of the file at the >> beginning and the end (because reading multiple gigabytes per file from >> disk and running the entire video through a hash algorithm would take >> ages to complete). It's likely your test files are just too close to >> the same, so are being considered the same and only showed once (one or >> the other is shown and the one not shown is ignored). >> >> There's no way to "fix" this as it's working as designed (and typical >> use cases don't involve multiple almost identical files). > I wonder if the hash algorithm could be updated to include the file > creation date or something else similar to help mitigate this type of overlap?

> I wonder if the hash algorithm could be updated to include the file > creation date or something else similar to help mitigate this type of overlap?

Probably a bad idea, since those who use "cp" instead of "cp -p" to copy a file will reset that date.

Since this is such a specialized case (possibly-identical results from manual runs of ffmpeg while testing something), I wonder if there might be some other workaround that can be used in that workflow, rather than by changing mythtv. ffmpeg and avconv both have the -metadata flag, for example; perhaps that can be used to embed unique metadata at the start of the file in a place that the hash will see. The easiest metadata to use is probably seconds or nanoseconds since the epoch.

On Aug 10, 2017, "Michael T. Dean" wrote: > > On Wed, Aug 9, 2017 at 5:55 AM, Dan Wilga wrote: > On 08/09/2017 05:05 PM, Mark Perkins wrote: > > On 10 August 2017 12:01:01 am "Michael T. Dean" wrote: > > > >> On 08/09/2017 10:04 AM, John Pilkington wrote: > >>> Or search 'mythtv video hash' > >>> > >>> Perhaps your ffmpeg processes gave identical results. > >> Yes, this is most likely the issue. The hash that's created to identify > >> unique videos is done with only a small part of the file at the > >> beginning and the end (because reading multiple gigabytes per file from > >> disk and running the entire video through a hash algorithm would take > >> ages to complete). It's likely your test files are just too close to > >> the same, so are being considered the same and only showed once (one or > >> the other is shown and the one not shown is ignored). > >> > >> There's no way to "fix" this as it's working as designed (and typical > >> use cases don't involve multiple almost identical files). > > I wonder if the hash algorithm could be updated to include the file > > creation date or something else similar to help mitigate this type of overlap? > > We're using the same hash algorithm used by other projects, the > OpenSubtitles hashing algorithm, which is based on Media Player Classic. > http://trac.opensubtitles.org/projects/opensubtitles/wiki/HashSourceCodes> > Any changes make ours a custom hash, with custom bugs and custom testing > and custom maintenance required. It's also quite possible that some > people are using the hash with other programs. > > >

I checked the files and they are, in fact, identical, despite using different command options to ffmpeg. So, I suppose there is not really a big issue. Thanks for pointing out the hash algorithm, at least it is not a mystery to me any longer.

Mark, pls make sure that this not seen movie hasn’t parental control level higher that currently selected to see videos. When it will be higher - You will not see movie in the list…

> Wiadomo?? napisana przez Mark Goldberg <marklgoldberg@gmail.com> w dniu 08.08.2017, o godz. 22:41: > > I have added some videos and do a rescan. Some videos are not seen or added to the list. I can rename those videos and they are still not seen on a rescan. I can put them in a different subdirectory and they are still not seen. > > This is on mythtv 0.28.1-3 from rpmfusion on Fedora 26.r > > Any suggestions how to troubleshoot this? > > Thanks, > > Mark > > _______________________________________________ > mythtv-users mailing list > mythtv-users@mythtv.org > http://lists.mythtv.org/mailman/listinfo/mythtv-users> http://wiki.mythtv.org/Mailing_List_etiquette> MythTV Forums: https://forum.mythtv.org