Transcode the audio into a encoding compatible with regular music players including MythMusic. For this example you would need to install ffmpeg. This is just an example as there are multiple tools you can use to extract and re-encode or copy the audio track from the mkv file.

Transcode the audio into a encoding compatible with regular music players including MythMusic. For this example you would need to install ffmpeg. This is just an example as there are multiple tools you can use to extract and re-encode or copy the audio track from the mkv file.

Revision as of 16:11, 20 November 2012

Note: Currently Lossless Cut scripts are available as an Beta release. The intent is to have MythTV users qualify their recorded videos to determine the script's compatibility with a wide variety of recording devices. So far these are the known Supported Recording Devices. Hopefully with the assistance of the MythTV community we can identify other recording devices which work and/or have the script improved to include additional devices and their encoding types.

By popular demand changed the KeyFrame Adjust automatic cut list generation to be an option on the command line "-g". The default is changed back to NOT alter the cut list when it does not exist. See the KeyFrame Adjust docs for instructions on how to invoke automatic generation of a cut list when it is empty and there is a skip list.

v0.1.8 Added small feature:

KeyFrame Adjust now automatically generates a cut list when it is empty and there is a skip list. Saves the manually effort of loading the skip list through the MythTV edit UI.

v0.1.7 Bug Fixes:

Fixed how the SQL starttime and progstart UTC values were determine. On some MythTV 0.26 installs the values were not correct which caused Lossless Cut to be unable to find essential markup record data like FPS and therefore gracefully abort

Fixed a bug where the bug report sample starttime was not properly converting hours into seconds

With a concert cut that does not have a subtitle, remove the subtitle from the file name format string

v0.1.7 Changes:

Added check for the minimum MythTV install which is v0.24+fixes

Added check for pre MythTV release versions that cannot be supported.

This change directly effects Gentoo users who are using the ebuilds for their MythTV v0.26 installs. The current ebuilds are using a pre-release of v0.26 that does not include fixes that Lossless Cut requires. You will need to upgrade your install manually to at least v0.26+fixes

Added a check for the UTCTZ datetime class with MythTV v0.26 and higher

Change the method to get fps, first and last frame from a SQL call to use the python bindings which avoids issues with UTC date conversions in MythTV v0.26 and higher

Removed several SQL calls and use the MythTV python bindings instead which reduced the need to deal with UTC start and end times changes in MythTV v0.26 and higher

Removed the python-dateutil dependency

Added a new configuration file section called "mkvmerge_user_settings". This section allows a user to customize the command line for either/or the mkvmerge cut or merge processing. This is a "use at your own risk" set of variables.

Made a common routine to create a lossless_cut.cfg file and removed redundant code in lossless_cut.py, keyframe_adjust.py and ll_report.py

With a "-r" replace lossless_cut.py renames the original recording with an added ".old" rather than ".Old" to be consistent with mythtranscode's similar function

When replacing a recording reset the commflagged indicator to be consistent with mythtranscode

GOING FORWARD: Assuming the v0.1.9 release has no major issues the project will move into maintenance mode in two weeks Wed, Nov 21st.

UPDATE: (Wed, 07 Nov 2012 10:14) for MythTV v0.24 and higher this bug as been resolved. NOTE: Scott Harris discovered a MythWeb related issue with Lossless Cut mkv video files when using the "-r" replace option: "One thing I discovered today is Lossless Cut breaks MythWeb's direct download links and streaming ability. If you navigate to the recordings screen in MythWeb and click on the direct download link for a recording that has been "lossless cut" you'll get an error about an unknown file."
NOTE: Ubuntu 11.04 Natty users, it appears that Launchpad will no longer allow creating debs for your installed Ubuntu version. This should not be a problem as you can download any of the any of latest debs and install it via the command line. The package dependencies have not changed at all or is there any 32bit or 64bit dependencies:

sudo dpkg -i "/path to downloaded deb/losslesscut.deb"

If you have problems with Lossless Cut please see Bug Reporting for an easy way to provide the developer with a bug report and if required a video sample file

A Debian PPA is available along with source. The PPA has the benefit of always keeping you up to date with the latest enhancements and bug fixes. The source tarball can be easily downloaded and installed on non-Debian based distros. Check Lossless Cut install from PPA or Lossless Cut install from source for details.

Individual files generated for each cut segment. This option is primarily used for Concert and comedy recordings where you want to save the specific segments instead of merge them all togther. A very flexible method to automatically name each segment is enabled with a "segment.cfg" file. See the link above for how-to details.

Full integration with the MythTV metadata grabbers including a fall back to EPG metadata when the program cannot be found by the grabbers, using the grabbers as set in MythTV settings including the language code

When exported or moved the mkv file is renamed to a customizable format e.g. "Sanctuary - S02E01 - End of Nights (1).mkv"

When exported to MythVideo any missing sub directories will be automatically created according to a customizable path and naming convention

The MythVideo Video storage group does not have to be on the same backend as the recording backend. Transfer of the cut mkv file is handled by the MythTV backend.

The command line and arguments used during each step of loss less cut processing is added to a Log file, generated with every userjob. The log file is automatically deleted if the job successfully completes. This is essential information for problem analysis.

Apply a video track positive or negative adjustment in millseconds to the Lossless Cut mkv file with the "-D" command line option

The lossless_cut.py script requires read/write access to the directories containing the recorded videos

Installation Instructions

Installing Dependencies

You must use recent version of the MKVToolnix and Mediainfo or Lossless Cut will not work. Lossless Cut is not compatible with earlier versions of "mkvmerge". Even recent distributions such as Ubuntu 12.04 do not come with the required version of MKVToolNix so use the PPA or install from source.
HP-mini on the user list suggested that if you are using older versions of Ubuntu and need to compile MKVToolNix or MediaInfo from source then you need to update your toolchain. This can be done using ubuntu devs toolchain ppa. Here are the PPA packages.

Mediainfo

To add the Mediainfo PPA as a 3rd Party source and install Mediainfo, type the following in a terminal session. You will need root access:

For Debian add the Mediainfo PPA as a 3rd Party source and install the deb, by typing the following in a terminal session. You will need root access:
The example below is specific to Debian 7 "Wheezy" but you can change the two instances of "precise" to the Ubuntu distribution that is closest to what your Debian install reflects.

MKVToolNix

MKVToolNix supports many Linux distributions and multiple releases of those distributions. Installation instructionsand alternately Source downloads
NOTE: You only need to install the "mkvtoolnix" package but NOT the "mkvtoolnix-gui" package.

Lossless Cut install from PPA

NOTE: Ubuntu 11.04 Natty users, it appears that Launchpad will no longer allow added debs for your installed Ubuntu version. This should not be a problem as you can download any of the v0.1.5 debs and install it via the command line. The package dependencies have not changed at all or is there any 32bit or 64bit dependencies:

sudo dpkg -i "/path to downloaded deb/losslesscut.deb"

Do not install Lossless Cut until you have successfully installed both Mediainfo and MKVToolNix.
The Lossless CutLaunchpad PPA

To add the Lossless Cut PPA as a 3rd Party source and install the deb, type the following in a terminal session. You will need root access:

For Debian add the Lossless Cut PPA as a 3rd Party source and install the deb, by typing the following in a terminal session. You will need root access:
The example below is specific to Debian 7 "Wheezy" but as long as you have satisifed the listed dependencies the deb is release/architecture agnostic.

Verify that you are ready to use Lossless Cut

NOTE: DO NOT RUN THIS STEP AS ROOT! A few users did and the permissions of the configuration file and the working directory cause issues when you run Lossless Cut as a userjob. This can be very confusing to analyse.

If you run your MythTV backend under an account like "mythtv" then run the following to avoid permission issues with the workpath and/or configuration file:

sudo -umythtv ./lossless_cut.py -f "/path to a/recorded.mpg" --test

Otherwise run the following command in a terminal session.

./lossless_cut.py -f "/path to a/recorded.mpg" --test

This command will verify that your MythTV backend has all the required prerequisites. Running this test will also automatically create the Lossless Cut configuration file "~/.mythtv/lossless_cut.cfg" or added new sections to a pre-existing cfg file.

If you see the "Congratulations!" message at the bottom of the default settings display you are good to go:

Congratulations! All script dependencies have been satisfied.
You are ready to perform loss less cuts on MythTV recorded videos.

Customize lossless_cut.cfg

NOTE: It seems a number of users are neglecting to examine the "~/.mythtv/lossless_cut.cfg" file. This file can save you using command line arguments as most of them can be set directly in the cfg file. Also the target directory location format for MythVideo export and filename format for moving and exporting are highly configurable. I encourage any user to at least view the options contained in the cfg file.

Assuming you successfully verified that Lossless Cut will run on your installation you should review and alter as required the configuration file "~/.mythtv/lossless_cut.cfg" with a text editor. There is nothing wrong with just using the default configuration. The configuration file contains extensive documentation.

MythTV User Jobs

Keyframe Adjustment of Cut and Skip Lists

This user job is optional but will help speed up setting accurate cut points after you have commflagged a recording. When you schedule a recording or create a recording rule, enable auto flag commercials and this Keyframe Adjust user job.
If you installed Lossless Cut from the PPA:

When this job runs every skip or cut frame will be adjusted to its nearest keyframe according to the recorded file's seek frames.

The start cut frames are adjusted to the nearest keyframe less than or equal to the start skip or cut frame number

The end cut frames are adjusted to the nearest keyframe greater than or equal to the end skip or cut frame number

To have a cutlist automatically generated when there is a skip list add the "-g" command line option to the User Job Command, for example:

User Job command: /usr/bin/keyframe_adjust -f "%DIR%/%FILE%" -g

Loss Less Cut user jobs

There are three different options for Lossless Cut user Jobs. The first two are non-destructive, which means that the original recorded video is left untouched.
NOTE: The "-D" delay option changes when the video track starts playing by a positive or negative number of milliseconds. When your recording already has audio and subtitle track(s) with a delay then the delay you specify with the "-D" option does not replace the delay in your recorded file. It will add or subtracted from what is already there. For example if your recoding's audio track has a delay of 3ms and you specify "-D -300" then that audio track will end up with a delay of "-297ms" before it starts in the Lossless Cut mkv file. This is just how mkvmerge works. In general this option should rarely be used. This option will not correct a Lossless Cut video that exhibits audio drift overtime.

Cut and Replace a Recording

Be warned: After the successful creation of the mkv file the old recording will be replaced in the "Watched Recordings" screen and the original mpg file will be deleted. Any cut and/or skip list will be removed. The MKV file will have the same name as the original recording but with a "mkv" file extension.
If you installed Lossless Cut from the PPA:

Watch the cut video. Assuming you did not replace the original recording you can always go back and fine tune your cuts as described in the previous step. Remember to delete the previous exported or saved cut video file before running the Lossless Cut job again.

Assuming you did not replace the original recording and you are satisfied with the cuts delete the original recording

NOTE: If you perform a second cut and "-m" move but had NOT deleted the previous cut video, a second video file will be created with a date and time tagged on to the end of the file name. In this way you can never accidentally overwrite an existing file with the same name.E.g. "Boss (2011) - S02E02 - Mania - Thur Sep 20th 12:00:02.mkv
A second cut on a MythVideo "-e" exported video will fail. You need to use the MythVideo UI to delete the video. This cleans up both the video file and the MythVideo database record.

Concert Cuts

When specified this option creates individual mkv files from each segment specified in the cut list. The Concert Cuts option is not intended to be used as a userjob but should be run from a terminal session. This is a powerful option for creating music tracks from a Concert recording or saving your favourite comedy skits from a show like SNL (Saturday Night Live).
You can specify a naming format and/or individually name each track including the subdirectories to save the segment files by creating a "segment.cfg" file before you run this Lossless Cut option. See details below. Any missing subdirectories are automatically created either when exporting to MythVideo or moving the cut segments to a directory.

Concert Cuts Workflow

Copy the "example_segment.cfg" file and rename it to something relevant to the recording (e.g. /tmp/Seether_One_Cold_Night.cfg). You will likely only use this file once. The example file is located in the "lossless_cut/importcode/example_segments.cfg" subdirectory. If you installed via the PPA it would be located in "/usr/share/lossless_cut/importcode/example_segments.cfg"

Use a text editor to specify the naming format for all of the segments or specific segment file names or a mixture of both. The ability to name individual tracks can be a powerful tool for a Concert recordings where multiple artist performed and you want to properly identify each artist, album, song and even specifically where to save the segment. The "example_segment.cfg" file contains extensive documentation on all if these options.

Run the Concert cuts from a terminal session on the MythTV backend where the recording resides

Transcode the audio into a encoding compatible with regular music players including MythMusic. For this example you would need to install ffmpeg. This is just an example as there are multiple tools you can use to extract and re-encode or copy the audio track from the mkv file.

DVB Subtitles

DVB Subtitles are supported with the ProjectX v0.91.0 java applet. Although this recent version of ProjectX is included with the Lossless Cut PPA and source the required Java runtime is not. Some users may not want Java installed or need DVB Subtitle support. If you want to have DVB Subtitle support follow these three simple steps:

Install either the open or Sun Java runtimes for example with a Debian distro use one of the following commands:

Edit your "~/.mythtv/lossless_cut.cfg" file and change the following configuration variable from "false" to true":

include_dvb_subtitles=true

Optionally modify any of the other variables in the "dvb_subtitles" configuration section.

Auto Remove Recording

Sometimes you may want to export or move recordings and have their original recordings automatically deleted. These would be recordings with no skip or cut lists, recordings you consider low risk or generally do not care if they are not Lossless Cut accurately. For those type of recordings you can specify configuration rules to have them deleted after a successful export or move.

Edit your "~/.mythtv/lossless_cut.cfg" file and read the ample documentation in the "mkvmerge_user_settings" section to add your own mkvmerge command line options

The two variables in this section allow you to add custom options to either/or the mkvmerge cut or merge processing. You need to understand the mkvmerge options to use these variables effectively and not mess up the Lossless Cut processing.

One person uses these variables to override a mkvmerge default action. Mkvmerge sets Audio Track 1 as the default audio track. This switch stops it setting any track as default. The player (mythtv in this case) then sets the audio track based on its own logic. Here is that setting as it should appear in the lossless_cut.cfg file:

mkvmerge_merge_addon=--default-track -1:0

In general it is unlikely that you will need to add your own variables but just in case here is a link to the mkvmerge CLI documentation.

Reports

As it is very difficult to anticipate all the issues that may occur while processing recordings produced by the wide variety of MythTV devices. This is the reason that the bug reporting script ll_report.py has been included. There are four variations of reports that can be produced.

Create a text file report that includes detailed information about the OS, the relevant utility versions in your environment, the recording device used and the specifics of the recorded video file (fps, resolutions, etc ...).See: Bug Reporting Option: -b

Create an archive of the text file mentioned above, the associated log file plus a 25Mg video file containing the first ~30 seconds of the recorded video file. The archive (tar.bz2) should be uploaded to an Internet location or service like mediafire or dropbox so that the developer can download and analyze the issues.See: Bug Reporting Option: -B

Create an archive of the text file mentioned above, the associated log file plus a 25Mg video file containing ~30 seconds starting at a specific start time into the the recorded video file. The time is expressed in "HH:MM:SS" format. The "HH" and "MM" are optional. The archive (tar.bz2) should be uploaded to an Internet location or service like mediafire or dropbox so that the developer can download and analyze the issues.See: Bug Reporting Option: -B -s HH:MM:SS

Despite the formats MKVToolNIX supports, Lossless Cut can only accurately cut a recording which has valid keyframe seek records in the MythTV database. This means it is unlikely that videos which are manually added to MythTV will produce quality results.

How to check what your install supports

You can check what your version of MKVToolNix supports by typing the following in a terminal session:

mkvmerge -l

The containers, video, audio encoding and subtitle formats listed below are specific to "mkvmerge v5.8.0 ('No Sleep / Pillow') built on Sep 2 2012 14:35:40". Use the command above to see what your installed version supports. New capabilities are always being added to MKVToolNix.

Recorded Video Input Containers

Note: All Lossless Cut videos will be output in a MKV container.

Matroska audio/video files [mka mks mkv mk3d webm webmv webma]

QuickTime audio/video files [mov]

Ogg/OGM audio/video files [ogg ogm ogv]

RealMedia audio/video files [ra ram rm rmvb rv]

AVI (Audio/Video Interleaved) [avi]

WebM audio/video files [webm webmv webma]

Video Input encoding

AVC/h.264 elementary streams [264 avc h264 x264]

Dirac [drc]

IVF with VP8 video files [ivf]

MP4 audio/video files [mp4 m4v]

MPEG program streams [mpg mpeg m2v mpv evo evob vob]

MPEG transport streams [ts m2ts mts]

MPEG video elementary streams [m1v m2v mpv]

VC1 elementary streams [vc1]

Audio Input encoding

A/52 (aka AC3) [ac3 eac3]

AAC (Advanced Audio Coding) [aac m4a mp4]

ALAC (Apple Lossless Audio Codec) [caf m4a mp4]

Dolby TrueHD [thd thd+ac3 truehd true-hd]

DTS/DTS-HD (Digital Theater System) [dts dtshd dts-hd]

FLAC (Free Lossless Audio Codec) [flac ogg]

MPEG audio files [mp2 mp3]

TTA (The lossless True Audio codec) [tta]

WAVE (uncompressed PCM audio) [wav]

WAVPACK v4 audio [wv]

Subtitle Input Formats

NOTE: As of v0.1.5 Lossless Cut supports the extraction of DVB Subtitles and teletext format subtitles. Neither or these subtitle formats are supported directly by mkvmerge but Lossless Cut uses the following included utilities:

With mythccextractor (MythTV v0.25 and higher) and/or CCExtractor v0.64-alpha7, teletext subtitles are extracted and converted to SRT then muxed back into the MKV file

ProjectX v0.91.0 extracts DVB Subtitle tracks and converts them into idx/sub files which are muxed into the MKV file as a VobSub. See DVB Subtitles for details.

The subtitle tracks that mkvmerge supports natively:

PGS/SUP subtitles [sup]

SRT text subtitles [srt]

SSA/ASS text subtitles [ass ssa]

USF text subtitles [usf xml]

VobSub subtitles [idx]

Current list of MediaInfo supported formats

MediaInfo is used to identify what tracks (Video, Audio and Subtitles) are included in a MythTV recording. This information is used during Lossless Cut processing. See: Supported Formats

Supported Recording Devices

Please add your table row in "Manufacturer" and "Model" alphabetical order.
Before adding your device to this table please check that it is not already listed.

1 - The convention in the UK is for SD channels to be MPEG2 encoded, and HD channels to be h.264 encoded (and for all channels on DVB-T/DVB-S to be MPEG2). The entries in this table relate to the video encoding standard, and the SD/HD indicator is just a guide based on UK convention.

T - Teletext Subtitles present and display in mythtv but lost after cutting

Unsupported Recording Devices

Please add your table row in "Manufacturer" and "Model" alphabetical order.
Before adding your device to this table please post a bug report to the MythTV mailing list. Follow the Bug Reporting instructions. Also if the script works, most of the time, with the recordings from a particular device then it is likely that the recording itself is the issue. See the first bullet of Known Limitations. Only post a table row for an Unsupported device when Lossless Cut consistently fails.

Known Limitations

If the original recording is corrupted then Lossless Cut is unlikely to work. Audio sync and faulty edits have been observed. Likely these type of recordings would suffer the same fate when edited in video editors like Avidemux. MythTV can often play back the original recordings and pass through the area(s) of corruption but you will notice that is where Lossless Cut video runs into problems.

Specifically for h.264 encoded videos, any rebuild of the recording's seektable (mythcommflag --rebuild or --video) does not produce valid keyframes. This means that the Lossless Cut script will not likely cut a video accurately even when setting the keyframe cut points manually with the frontend edit UI. The lossless cut results are a hit and miss proposition, which will not change until ticket #11144is fixed. Most of the time this limitation will not effect anyone unless you try to manually add a recording to MythTV and then rebuilding the seek table. Restoring the original seek records from a database backup will work as the keyframes would be accurate. Unless you know what you are doing it is not recommended that you replace/restore specific records in a database.

mythccextractor will not be used until ticket #11081is fixed. Currently CCExtractor (included with Lossless Cut) is exclusively used for subtitle extraction and conversion to SRT format.

Recorded videos without MythTV seek table records cannot be edited

Recorded videos that have been replaced by a loss less cut mkv video cannot be re-edited

A user has reported that Lossless Cut works as expected with their HDHomerun device for all but a few specific channels (e.g. ABC channel 13 WZZM in Grand Rapids). Lossless Cut videos from that channel produces only an audio track. While on a different channel (NBC Grand Rapids) the merge of cuts segments consistently does not include the very last cut segement. More than one user has noted inconsistencies in how mkvmerge treats HDHomerun recordings. Hopefully MKVToolNix future releases will address these issues.

UPDATE: (Wed, 07 Nov 2012 10:14) for MythTV v0.24 and higher this bug as been resolved. For Lossless Cut mkv video files when the "-r" replace option was used, clicking MythWeb's direct download and streaming links resulted in an "unkown type" error. Patches have been provided to the MythWeb developer which fixes this issue.

Credits

The developers at Bunkus.orgfor the MKVToolNix suite of utilities. Without those utilities Lossless Cut would not exist.

Ian Thiele for his h264cut.sh scriptwhich demonstrated the versatility and maturity of the MKVToolNix "mkvmerge" utility. Also Lossless Cut follows the same general process flow for video cut and merging as in his h264cut.sh script.

Raymond Wagner for developing the Python bindings including adding new bindings when requested. Also for putting up with my questions on how to use the python bindings to their best effect.

Carlos Fernandez for his versatile subtitle extractor utility CCExtractorand his excellent support