Mailing List Archive

On Mon, 2018-04-30 at 11:52 -0400, Brian J. Murrell wrote: > 2. subtree includes a copy of FFmpeg's repo in our repo. > > Correct. Although I don't think it's the whole "repo", where a repo is > every object ever created in a project and so includes all branches, > tags, etc. Rather I think git subtree (and subrepo) are just taking > the content of a single branch and copying it to your subdir.

AIUI technically what it does (without --squash) is take the tree-sha of the tree (the datastructure which is the git equivalent to a directory) pointed to by an upstream commit (perhaps/probably one on a branch) and grafts it in directly as an entry in a sub-tree (i.e. a subdirectory) of the tree of the downstream repo.

On 04/30/2018 11:52 AM, Brian J. Murrell wrote: >> I believe that using subtree or subrepo requires that we use the >> master >> branch of FFmpeg. > I'm not convinced of that. At least with subrepo, you can choose which > branch you clone and track into the subdir. If/when you want to change > the upstream tracking branch, worst case scenario (assuming the tool > doesn't have a way to change) is you do as above. Create your patch > vs. upstream, blow the subdir away, recreate the subdir using > {subtree|submodule} and then apply your patch, fixing any conflicts, > etc. This is essentially what we do at the moment. We delete the source code, copy in the new source, and then work at re-applying all of our changes. I am trying to improve on that process, which is rather time-consuming.

On Mon, 2018-04-30 at 12:37 -0400, Peter Bennett wrote: > > This is essentially what we do at the moment. We delete the source > code, > copy in the new source, and then work at re-applying all of our > changes. > I am trying to improve on that process, which is rather time- > consuming.

Sure, but this is worst-case scenario. I have not tried to switch branches with git subrepo (nor git subtree as I have not even used that). Maybe it's easier than that. But it's also only something you would ever have to do (again, in worst-case) when switching upstream tracking (i.e. release) branches, so I think you are still better off.

At least you have git a workflow and sub{repo|tree} bookkeeping your updating for you. Overall it should make keeping your ffmpeg subdir up-to-date with upstream trivial, and the more frequently you refresh (git subrepo pull) it the smaller the potential conflicts are.

If it were me, I'd set up a job daily or weekly just to refresh from upstream to keep the potential for conflicts as entirely small as possible.

Cheers, b.

Attached Files:

On 04/30/2018 01:52 PM, Brian J. Murrell wrote: > Sure, but this is worst-case scenario. I have not tried to switch > branches with git subrepo (nor git subtree as I have not even used > that). Maybe it's easier than that. But it's also only something you > would ever have to do (again, in worst-case) when switching upstream > tracking (i.e. release) branches, so I think you are still better off. Currently the only time we do any updating at all is when switching upstream branches. So, no improvement.

> At least you have git a workflow and sub{repo|tree} bookkeeping your > updating for you. Overall it should make keeping your ffmpeg subdir > up-to-date with upstream trivial, and the more frequently you refresh > (git subrepo pull) it the smaller the potential conflicts are.

> If it were me, I'd set up a job daily or weekly just to refresh from > upstream to keep the potential for conflicts as entirely small as > possible. We need to decide a strategy. Currently we only refresh FFmpeg once a year or so. I would not want to do any refresh shortly before a new MythTV release because we would not want to possibly introduce unexpected problems at the last minute.

> On 30 Apr 2018, at 7:52 pm, Brian J. Murrell <brian@interlinx.bc.ca> wrote: > > On Mon, 2018-04-30 at 12:37 -0400, Peter Bennett wrote: >> >> This is essentially what we do at the moment. We delete the source >> code, >> copy in the new source, and then work at re-applying all of our >> changes. >> I am trying to improve on that process, which is rather time- >> consuming. > > Sure, but this is worst-case scenario. I have not tried to switch > branches with git subrepo (nor git subtree as I have not even used > that). Maybe it's easier than that. But it's also only something you > would ever have to do (again, in worst-case) when switching upstream > tracking (i.e. release) branches, so I think you are still better off.

I’ve never seen a case where ffmpeg didn’t break API between version.

Updating automatically ffmpeg from a subrepo is just not going to work, there’s just no project that does what you’re stating. Every single project checkout a particular version of ffmpeg and stick to that version until the next stable release. It’s the one it’s tested against, and patched if necessary.

The only exception to this I can think of is mplayer, and ever attempted to checkout an older version of mplayer (which always pull the latest version of ffmpeg) ? Good luck. mplayer current checkout will work with ffmpeg of the day, never day to perform a bisect or attempt to lookup for a regression.

So not having to continually rebase our changes and particularly configure is one thing.

On Mon, 2018-04-30 at 20:08 +0200, Jean-Yves Avenard wrote: > > but all this talk about automated, git subtree vs subrepo and so > forth, I’m sorry to say, clearly show you’ve really have never worked > with ffmpeg before.

No I haven't. I never claimed to have and I find your condescending and dismissing tone about offensive. But I work with lots of other upstream projects in my real-life job even so I am not completely talking out of my ass here.

I was providing input about git sub* tool experience. But if that experience is not really appreciated and is going to be met with such an attitude I will just keep it to myself, OK?

b.

Attached Files:

There seems to be a little confusion concerning how git subtree works. It doesn't do anything weird to the subdirectory it's applied to: the only difference anyone would notice in the mythtv repo, if using git subtree, is that the updates to the ffmpeg subdir would appear as merges and some of the commits in the branches merged would mysteriously refer to files within the ffmpeg subdir without the "external/FFmpeg" prefix. Using git subtree would not make the mythtv repo rely on the ffmpeg repo. It would just make them share some sha-1 hashes in a useful way.

Before git subtree existed, it was possible to have a local repo for one project include remotes from an entirely different project. Without git subtree it wasn't useful to do so because if you checked out on a branch from one project all the files from the other project would disappear. git subtree is just a useful way of using remotes from different projects in the case that one project represents the files of a subdirectory of another project. _______________________________________________ mythtv-dev mailing list mythtv-dev@mythtv.org http://lists.mythtv.org/mailman/listinfo/mythtv-devhttp://wiki.mythtv.org/Mailing_List_etiquetteMythTV Forums: https://forum.mythtv.org

In my own copy of MythTV repository, I tried using a subtree to include FFmpeg. I hit up against a few problems -

I had wanted to try importing a particular commit from FFmpeg, but I was unable to do that. I could only import the latest version of a branch. Also I could see no way of cherry-picking FFmpeg commits into our copy.

I do not want to be forever constrained to only updating to the latest commit of a branch in FFmpeg.

I have now created my own copy of FFmpeg repository and applied our changes to that. Perhaps this is a better solution. We could maintain an ffmpeg repository in MythTV and either copy from it as needed after merging from the main FFmpeg, or else use a subtree based on our own FFmpeg. We could create a mythtv-master branch in the private FFmpeg for the subtree, and merge or cherry-pick to that as needed. This way we would a better workfolw and accurate tracking of who contributed changes in future. Currently the tracking of contributions to MythTV's copy of FFmpeg is inaccurate, incorrectly showing most things having been contributed by the various people who have done the FFmpeg resync.

We could also make a submodule instead of a subtree from our private FFmpeg, but that means that people who access MythTV source would have to run an explicit command to pull in the submodule data and to pull in updates from it when needed.

On the current merge: I am using FFmpeg master as of yesterday. The recent fixes to mediacodec are not in the release branch. After applying the MythTV modifications (5680 lines of patches) there were 12 files with conflicts. Most were easy to fix, but FFmpeg has redesigned the code for registering demuxers so I had to make some changes to the code for registering ff_mythtv_mpegts_demuxer and ff_mythtv_mpegtsraw_demuxer.

It now has compile errors in the mythtv-mpegts demuxer code caused by undeclared functions and macros, presumably deprecated things removed from FFmpeg. I will have to fix these. The mythtv-mpegts demuxer code also uses a lot of deprecated functions.

On Fri, May 11, 2018 at 01:51:26PM -0400, Peter Bennett wrote: > In my own copy of MythTV repository, I tried using a subtree to include > FFmpeg. I hit up against a few problems - > > I had wanted to try importing a particular commit from FFmpeg, but I was > unable to do that. I could only import the latest version of a branch. Also > I could see no way of cherry-picking FFmpeg commits into our copy. > > git subtree add -P mythtv/external/FFmpeg > https://github.com/FFmpeg/FFmpeg.git 7e3a070 --squash > fatal: Couldn't find remote ref 7e3a070 > > I tried adding it as a remote, with the same result > > git remote add ffmpeg https://github.com/FFmpeg/FFmpeg.git> git fetch ffmpeg > git subtree add -P mythtv/external/FFmpeg ffmpeg 7e3a070 --squash > fatal: Couldn't find remote ref 7e3a070 > > If you replace the commit id with a branch name it works > > git subtree add -P mythtv/external/FFmpeg > https://github.com/FFmpeg/FFmpeg.git release/4.0 --squash > worked > > I do not want to be forever constrained to only updating to the latest > commit of a branch in FFmpeg.

I have one idea, but it's not without its own share of warts. That is to treat the ffmpeg-master branch (and possibly others) like tracking branches. It could go something like this.

One wart is that anytime ffmpeg adds new files in their root, we would have to notice them when we pull and git mv them to external/FFmpeg.

Another wart is we would have to do the same with any other ffmpeg branches that we might want to cherry-pick from.

> I have now created my own copy of FFmpeg repository and applied our changes > to that. Perhaps this is a better solution. We could maintain an ffmpeg > repository in MythTV and either copy from it as needed after merging from > the main FFmpeg, or else use a subtree based on our own FFmpeg. We could > create a mythtv-master branch in the private FFmpeg for the subtree, and > merge or cherry-pick to that as needed. This way we would a better workfolw > and accurate tracking of who contributed changes in future. Currently the > tracking of contributions to MythTV's copy of FFmpeg is inaccurate, > incorrectly showing most things having been contributed by the various > people who have done the FFmpeg resync. > > We could also make a submodule instead of a subtree from our private FFmpeg, > but that means that people who access MythTV source would have to run an > explicit command to pull in the submodule data and to pull in updates from > it when needed. > > On the current merge: > I am using FFmpeg master as of yesterday. The recent fixes to mediacodec are > not in the release branch. After applying the MythTV modifications (5680 > lines of patches) there were 12 files with conflicts. Most were easy to fix, > but FFmpeg has redesigned the code for registering demuxers so I had to make > some changes to the code for registering ff_mythtv_mpegts_demuxer and > ff_mythtv_mpegtsraw_demuxer. > > It now has compile errors in the mythtv-mpegts demuxer code caused by > undeclared functions and macros, presumably deprecated things removed from > FFmpeg. I will have to fix these. The mythtv-mpegts demuxer code also uses a > lot of deprecated functions.

FFmpeg really is a moving target, isn't it! :( Is it worth trying again to get some of our changes accepted upstream?

On Fri, 2018-05-11 at 13:51 -0400, Peter Bennett wrote: > In my own copy of MythTV repository, I tried using a subtree to include > FFmpeg. I hit up against a few problems - > > I had wanted to try importing a particular commit from FFmpeg, but I was > unable to do that. I could only import the latest version of a branch. > Also I could see no way of cherry-picking FFmpeg commits into our copy. > > git subtree add -P mythtv/external/FFmpeg > https://github.com/FFmpeg/FFmpeg.git 7e3a070 --squash > fatal: Couldn't find remote ref 7e3a070 > > I tried adding it as a remote, with the same result > > git remote add ffmpeg https://github.com/FFmpeg/FFmpeg.git> git fetch ffmpeg > git subtree add -P mythtv/external/FFmpeg ffmpeg 7e3a070 --squash > fatal: Couldn't find remote ref 7e3a070

(I fudged the prefix because it already exists, but I guess you have already done "git rm -r" to get rid of the old copy in your tree, I used -n so I wouldn't get all the ffmpeg tags in my local repo). You could avoid the remote by just doing:

but I guess this is the one you want going forward after the initial add.

For cherry-picking there is a process given in https://stackoverflow.com/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree (second half of first answer) which looks pretty plausible to me, but it does rely on not using `--squash` for the regular merges (although possibly it could be adapted).

TBH I'd recommend not using squash anyway, it'll give a more accurate picture of the history which is useful for people doing archaeology which crosses into the ffmpeg tree (it shows the actual upstream commits and authors instead of the subtree-merger) and I think git does a better job of merging etc if it has more granular history to look at (e.g. I think it can spot when the same changes appear in two commits in different branches, perhaps due to cherry-picking, which helps merge do the right thing more often).

On 05/12/2018 07:30 AM, Ian Campbell wrote: > On Fri, 2018-05-11 at 13:51 -0400, Peter Bennett wrote: >> In my own copy of MythTV repository, I tried using a subtree to include >> FFmpeg. I hit up against a few problems - >> >> I had wanted to try importing a particular commit from FFmpeg, but I was >> unable to do that. I could only import the latest version of a branch. >> Also I could see no way of cherry-picking FFmpeg commits into our copy. >> >> git subtree add -P mythtv/external/FFmpeg >> https://github.com/FFmpeg/FFmpeg.git 7e3a070 --squash >> fatal: Couldn't find remote ref 7e3a070 >> >> I tried adding it as a remote, with the same result >> >> git remote add ffmpeg https://github.com/FFmpeg/FFmpeg.git>> git fetch ffmpeg >> git subtree add -P mythtv/external/FFmpeg ffmpeg 7e3a070 --squash >> fatal: Couldn't find remote ref 7e3a070 > There are two forms of `git subtree add`: >> git subtree add -P <prefix> <commit> >> git subtree add -P <prefix> <repository> <ref> > You've tried the second one but I think you need the first, in a slight > modification to your second attempt with a remote: > > ijc@dagon:mythtv-subtree-test$ git remote add ffmpeg https://github.com/FFmpeg/FFmpeg.git> ijc@dagon:mythtv-subtree-test$ git fetch -n ffmpeg > warning: no common commits > [...] > ijc@dagon:mythtv-subtree-test$ git subtree add -P mythtv/external/FFmpeg-test 7e3a070 --squash > Added dir 'mythtv/external/FFmpeg-test' > ijc@dagon:mythtv-subtree-test$ ls mythtv/external/FFmpeg-test > Changelog COPYING.GPLv2 CREDITS INSTALL.md libavformat/ libswresample/ Makefile tests/ > compat/ COPYING.GPLv3 doc/ libavcodec/ libavresample/ libswscale/ presets/ tools/ > configure* COPYING.LGPLv2.1 ffbuild/ libavdevice/ libavutil/ LICENSE.md README.md > CONTRIBUTING.md COPYING.LGPLv3 fftools/ libavfilter/ libpostproc/ MAINTAINERS RELEASE > > (I fudged the prefix because it already exists, but I guess you have > already done "git rm -r" to get rid of the old copy in your tree, I > used -n so I wouldn't get all the ffmpeg tags in my local repo). You > could avoid the remote by just doing: > > git fetch https://github.com/FFmpeg/FFmpeg.git <branch|commit> > > instead of the `git remote add foo` + `git fetch -n foo`. > > I've not looked at how this differs from the subtree merge command: > >> git subtree merge -P <prefix> <commit> > but I guess this is the one you want going forward after the initial > add. Thank you for the insight. This gives a better way of adding a subtree. > > For cherry-picking there is a process given in https://stackoverflow.co> m/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree > (second half of first answer) which looks pretty plausible to me, but > it does rely on not using `--squash` for the regular merges (although > possibly it could be adapted). It is a bit messy - 5 step process. > TBH I'd recommend not using squash anyway, it'll give a more accurate > picture of the history which is useful for people doing archaeology > which crosses into the ffmpeg tree (it shows the actual upstream > commits and authors instead of the subtree-merger) and I think git does > a better job of merging etc if it has more granular history to look at > (e.g. I think it can spot when the same changes appear in two commits > in different branches, perhaps due to cherry-picking, which helps merge > do the right thing more often). The thing I was concerned about with that is that there are so many commits in FFmpeg compared with MythTV, that they would overwhelm the history. There are 91000 commits in the master branch of FFmpeg. I feared that the day I pulled in the subtree without squash, it may become very difficult to find anything useful pertaining to MythTV in the log. Perhaps that would only affect certain views of the log.

I am currently trying an approach of having a private copy of the FFmpeg repository. Then I can use the simpler option at the bottom of the stackoverflow article, that says "Another more simple option, if you have access to the original subtree repository, is to make the cherry pick there in a branch and then just git subtree pull that specific branch.".

On 05/11/2018 05:13 PM, David Engel wrote: > I have one idea, but it's not without its own share of warts. That is > to treat the ffmpeg-master branch (and possibly others) like tracking > branches. It could go something like this. > > # Remove the existing external/FFmpeg from the mythtv repo. > git rm -r external/FFmpeg > > # Create an empty ffmpeg-master branch in the mythtv repo. > git co ce7a5f62 (the epoch commit with an empty repo> > git branch ffmpeg-master > git co ffmpeg-master > > # Pull in ffmpeg master branch and move it to the right place. > git pull --allow-unrelated-historieshttps://github.com/FFmpeg/FFmpeg.git master > mkdir external/FFmpeg > git mv <list of files> external/FFmpeg > > # Merge the ffmpeg-master branch into mythtv master. > git co master > git merge ffmpeg-master > > One wart is that anytime ffmpeg adds new files in their root, we would > have to notice them when we pull and git mv them to external/FFmpeg. > > Another wart is we would have to do the same with any other ffmpeg > branches that we might want to cherry-pick from. I originally thought of this option before I found subtree. It is still rather complicated. I don't know how it would handle files that are moved from one ffmpeg directory to another (that was the case with a bunch of files between 3.2 and 3.4).

On Sat, 2018-05-12 at 12:29 -0400, Peter Bennett wrote: > > For cherry-picking there is a process given in https://stackoverflow.co> > m/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree > > (second half of first answer) which looks pretty plausible to me, but > > it does rely on not using `--squash` for the regular merges (although > > possibly it could be adapted). > It is a bit messy - 5 step process.

Yeah, it might be possible to script a bunch of it in a nice way. Perhaps it is worth dropping a line to the git devs with a wishlist request?

> > TBH I'd recommend not using squash anyway, it'll give a more accurate > > picture of the history which is useful for people doing archaeology > > which crosses into the ffmpeg tree (it shows the actual upstream > > commits and authors instead of the subtree-merger) and I think git does > > a better job of merging etc if it has more granular history to look at > > (e.g. I think it can spot when the same changes appear in two commits > > in different branches, perhaps due to cherry-picking, which helps merge > > do the right thing more often). > The thing I was concerned about with that is that there are so many > commits in FFmpeg compared with MythTV, that they would overwhelm the > history. There are 91000 commits in the master branch of FFmpeg. I > feared that the day I pulled in the subtree without squash, it may > become very difficult to find anything useful pertaining to MythTV in > the log. Perhaps that would only affect certain views of the log.

`git log` orders things by commit date, so if the 91,000 ffmpeg commits were largely historical then it wouldn't be so bad. But I suspect that the ongoing flow of patches into ffmpeg probably outstrips mythtv by a fair bit, in which case your concerns would be true by default.

You can use `git log --not ?branch|commit|etc?` to say "don't mention things which are in here (i.e. `git log --not ffmpeg/master`), but that's a bit tedious to remember to do.

> I am currently trying an approach of having a private copy of the FFmpeg > repository. Then I can use the simpler option at the bottom of the > stackoverflow article, that says "Another more simple option, if you > have access to the original subtree repository, is to make the cherry > pick there in a branch and then just git subtree pull that specific > branch.".

That's effectively what the 5 step cherry-pick process above is doing, it's just doing it all in the one repo using branches and bare checkouts. It's certainly less things to keep straight in ones head to do it in a separate repo though.

A separate repo presumably makes it easier to keep track of where you've gotten up to (i.e. most recently merged) and lets you have branches for fixes/30 etc and still keep cherry picking onto those independently of master.

On 05/13/2018 03:32 AM, Ian Campbell wrote: > On Sat, 2018-05-12 at 12:29 -0400, Peter Bennett wrote: >>> For cherry-picking there is a process given in https://stackoverflow.co>>> m/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree >>> (second half of first answer) which looks pretty plausible to me, but >>> it does rely on not using `--squash` for the regular merges (although >>> possibly it could be adapted). >> It is a bit messy - 5 step process. > Yeah, it might be possible to script a bunch of it in a nice way. > Perhaps it is worth dropping a line to the git devs with a wishlist > request? > >>> TBH I'd recommend not using squash anyway, it'll give a more accurate >>> picture of the history which is useful for people doing archaeology >>> which crosses into the ffmpeg tree (it shows the actual upstream >>> commits and authors instead of the subtree-merger) and I think git does >>> a better job of merging etc if it has more granular history to look at >>> (e.g. I think it can spot when the same changes appear in two commits >>> in different branches, perhaps due to cherry-picking, which helps merge >>> do the right thing more often). >> The thing I was concerned about with that is that there are so many >> commits in FFmpeg compared with MythTV, that they would overwhelm the >> history. There are 91000 commits in the master branch of FFmpeg. I >> feared that the day I pulled in the subtree without squash, it may >> become very difficult to find anything useful pertaining to MythTV in >> the log. Perhaps that would only affect certain views of the log. > `git log` orders things by commit date, so if the 91,000 ffmpeg commits > were largely historical then it wouldn't be so bad. But I suspect that > the ongoing flow of patches into ffmpeg probably outstrips mythtv by a > fair bit, in which case your concerns would be true by default. > > You can use `git log --not «branch|commit|etc»` to say "don't mention > things which are in here (i.e. `git log --not ffmpeg/master`), but > that's a bit tedious to remember to do. > >> I am currently trying an approach of having a private copy of the FFmpeg >> repository. Then I can use the simpler option at the bottom of the >> stackoverflow article, that says "Another more simple option, if you >> have access to the original subtree repository, is to make the cherry >> pick there in a branch and then just git subtree pull that specific >> branch.". > That's effectively what the 5 step cherry-pick process above is doing, > it's just doing it all in the one repo using branches and bare > checkouts. It's certainly less things to keep straight in ones head to > do it in a separate repo though. > > A separate repo presumably makes it easier to keep track of where > you've gotten up to (i.e. most recently merged) and lets you have > branches for fixes/30 etc and still keep cherry picking onto those > independently of master. > > Ian. > In my private MythTV repository I added a subtree for FFmpeg (with squash) from my private FFmpeg repository (with patches committed) and got everything working. Then I pulled new ffmpeg commits into the private ffmpeg repo and tried to pull the changes into the private MythTV. That was a disaster. Every change, even 1 line changes, was marked as a conflict. I think that is because with the squash it cannot find a base for the merge. So that is a bust.

Now I have deleted the FFmpeg tree from my priivate MythTV and copied all of the files from the private FFmpeg and committed that. This is similar to our traditional approach, with the difference that I am copying files from my own FFmpeg repo (with MythTV patches included) instead of copying them from the main FFmpeg and re-applying patches.

I think maybe the best is to set up a FFmpeg repository under MythTV and do it that way - all FFmpeg changes that we make should be first applied to the FFmpeg repository and then copy the source over to MythTV.

I have the new FFmpeg working and very lightly tested with the master MythTV. Playback works on VAAPI, VDPAU, XVIDEO. LiveTV works and LiveTV subtitles work. That is all I have tested.

If you want to take a look it is in github under bennettpeter - MythTV and FFmpeg repositories, master branches.

On Mon, May 14, 2018 at 10:44:19AM -0400, Peter Bennett wrote: > > > On 05/13/2018 03:32 AM, Ian Campbell wrote: > > On Sat, 2018-05-12 at 12:29 -0400, Peter Bennett wrote: > > > > For cherry-picking there is a process given in https://stackoverflow.co> > > > m/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree > > > > (second half of first answer) which looks pretty plausible to me, but > > > > it does rely on not using `--squash` for the regular merges (although > > > > possibly it could be adapted). > > > It is a bit messy - 5 step process. > > Yeah, it might be possible to script a bunch of it in a nice way. > > Perhaps it is worth dropping a line to the git devs with a wishlist > > request? > > > > > > TBH I'd recommend not using squash anyway, it'll give a more accurate > > > > picture of the history which is useful for people doing archaeology > > > > which crosses into the ffmpeg tree (it shows the actual upstream > > > > commits and authors instead of the subtree-merger) and I think git does > > > > a better job of merging etc if it has more granular history to look at > > > > (e.g. I think it can spot when the same changes appear in two commits > > > > in different branches, perhaps due to cherry-picking, which helps merge > > > > do the right thing more often). > > > The thing I was concerned about with that is that there are so many > > > commits in FFmpeg compared with MythTV, that they would overwhelm the > > > history. There are 91000 commits in the master branch of FFmpeg. I > > > feared that the day I pulled in the subtree without squash, it may > > > become very difficult to find anything useful pertaining to MythTV in > > > the log. Perhaps that would only affect certain views of the log. > > `git log` orders things by commit date, so if the 91,000 ffmpeg commits > > were largely historical then it wouldn't be so bad. But I suspect that > > the ongoing flow of patches into ffmpeg probably outstrips mythtv by a > > fair bit, in which case your concerns would be true by default. > > > > You can use `git log --not ?branch|commit|etc?` to say "don't mention > > things which are in here (i.e. `git log --not ffmpeg/master`), but > > that's a bit tedious to remember to do. > > > > > I am currently trying an approach of having a private copy of the FFmpeg > > > repository. Then I can use the simpler option at the bottom of the > > > stackoverflow article, that says "Another more simple option, if you > > > have access to the original subtree repository, is to make the cherry > > > pick there in a branch and then just git subtree pull that specific > > > branch.". > > That's effectively what the 5 step cherry-pick process above is doing, > > it's just doing it all in the one repo using branches and bare > > checkouts. It's certainly less things to keep straight in ones head to > > do it in a separate repo though. > > > > A separate repo presumably makes it easier to keep track of where > > you've gotten up to (i.e. most recently merged) and lets you have > > branches for fixes/30 etc and still keep cherry picking onto those > > independently of master. > > > > Ian. > > > In my private MythTV repository I added a subtree for FFmpeg (with squash) > from my private FFmpeg repository (with patches committed) and got > everything working. Then I pulled new ffmpeg commits into the private ffmpeg > repo and tried to pull the changes into the private MythTV. That was a > disaster. Every change, even 1 line changes, was marked as a conflict. I > think that is because with the squash it cannot find a base for the merge. > So that is a bust.

Did you try without squash? I know whatever we do will be tedious, but what you describe below with our own, second repository sound more so.

David

> Now I have deleted the FFmpeg tree from my priivate MythTV and copied all of > the files from the private FFmpeg and committed that. This is similar to our > traditional approach, with the difference that I am copying files from my > own FFmpeg repo (with MythTV patches included) instead of copying them from > the main FFmpeg and re-applying patches. > > I think maybe the best is to set up a FFmpeg repository under MythTV and do > it that way - all FFmpeg changes that we make should be first applied to the > FFmpeg repository and then copy the source over to MythTV. > > I have the new FFmpeg working and very lightly tested with the master > MythTV. Playback works on VAAPI, VDPAU, XVIDEO. LiveTV works and LiveTV > subtitles work. That is all I have tested. > > If you want to take a look it is in github under bennettpeter - MythTV and > FFmpeg repositories, master branches. > > Peter

On 05/14/2018 02:11 PM, David Engel wrote: > Did you try without squash? I know whatever we do will be tedious, > but what you describe below with our own, second repository sound more > so. > > David > I did not. I will create another branch in my repository and try that.

On 05/14/2018 04:04 PM, Peter Bennett wrote: > > > On 05/14/2018 02:11 PM, David Engel wrote: >> Did you try without squash? I know whatever we do will be tedious, >> but what you describe below with our own, second repository sound more >> so. >> >> David >> > I did not. I will create another branch in my repository and try that. > > Peter

The log is full of ffmpeg entries. We will need some filtering on "git log" to make it useful. I have not looked into that. Look at https://github.com/bennettpeter/mythtv and select the "ffmpeg-subtree" branch. Notice it has 125000 commits, as opposed to the master branch, which has 34000.

I think this is a good solution, as long as other developers are not too unhappy about the log situation.

On Mon, May 14, 2018 at 07:44:11PM -0400, Peter Bennett wrote: > > > On 05/14/2018 04:04 PM, Peter Bennett wrote: > > > > > > On 05/14/2018 02:11 PM, David Engel wrote: > > > Did you try without squash?? I know whatever we do will be tedious, > > > but what you describe below with our own, second repository sound more > > > so. > > > > > > David > > > > > I did not. I will create another branch in my repository and try that. > > > > Peter > > Tested Without squash and without private repo: > > It works fine I am able to merge updates from ffmpeg wiothout conflicts as > well as cherry pick a commit without problem, using the procedure in https://stackoverflow.com/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree> > The log is full of ffmpeg entries. We will need some filtering on "git log" > to make it useful. I have not looked into that. Look at > https://github.com/bennettpeter/mythtv and select the "ffmpeg-subtree" > branch. Notice it has 125000 commits, as opposed to the master branch, which > has 34000. > > I think this is a good solution, as long as other developers are not too > unhappy about the log situation.

The log bloat is unfortunate, but I think this solution is better than the other alternatives so far. Thanks for testing all of this.

On 05/15/2018 11:51 AM, David Engel wrote: > The log bloat is unfortunate, but I think this solution is better than > the other alternatives so far. Thanks for testing all of this. > > David I did some testing of git log and I found that git seems to have a bug/restriction which actually helps.

If you cd to the base directory of the repo and run "git log -- mythtv" you only see the mythtv and mythtv subdirectory changes that were made in this repository but not any of the ffmpeg subtree changes that were imported via the subtree. So we see the ffmpeg customizations that we applied here but none of those imported via the subtree. This is exactly what I would want to see.

I cannot find a way of seeing a log of subtree imported commits using a path. Using mythtv/external/FFmpeg finds nothing, probably because when they were committed the path was just libavformat/ and not mythtv/external/FFmpeg/libavformat. However searching on path libavformat also shows nothing. If you want to see those you either have to use the full "git log", search on a revision range, or look in the ffmpeg repository. This seems to be a bug but works in our favor.

On Tue, 2018-05-15 at 12:14 -0400, Peter Bennett wrote: > If you cd to the base directory of the repo and run "git log -- mythtv" > you only see the mythtv and mythtv subdirectory changes that were made > in this repository but not any of the ffmpeg subtree changes that were > imported via the subtree. So we see the ffmpeg customizations that we > applied here but none of those imported via the subtree. This is exactly > what I would want to see.

This is because git log isn't aware of subtrees, so as it walks over commits which are actually from ffmpeg it isn't seeing mythtv/external/ffmpeg/foo/bar.c (which would match your mythtv argument) it sees just foo/bar.c (which doesn't match the argument).

So it relies a bit on ffmpeg not having a top-level directory called "mythtv" but that seems like a safe bet.

It's not future proof against the git devs make "git log" aware of subtrees -- but you would hope/imagine that in doing so they would add useful options like --no-subtrees at the same time.

> > I cannot find a way of seeing a log of subtree imported commits using a > path. Using mythtv/external/FFmpeg finds nothing, probably because when > they were committed the path was just libavformat/ and not > mythtv/external/FFmpeg/libavformat.

Correct (I should have read ahead before writing the above! I'll leave it now I've typed it)

> On Mon, May 14, 2018 at 07:44:11PM -0400, Peter Bennett wrote: >> >> On 05/14/2018 04:04 PM, Peter Bennett wrote: >>> >>> On 05/14/2018 02:11 PM, David Engel wrote: >>>> Did you try without squash? I know whatever we do will be tedious, >>>> but what you describe below with our own, second repository sound more >>>> so. >>>> >>>> David >>>> >>> I did not. I will create another branch in my repository and try that. >>> >>> Peter >> Tested Without squash and without private repo: >> >> It works fine I am able to merge updates from ffmpeg wiothout conflicts as >> well as cherry pick a commit without problem, using the procedure in https://stackoverflow.com/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree>> >> The log is full of ffmpeg entries. We will need some filtering on "git log" >> to make it useful. I have not looked into that. Look at >> https://github.com/bennettpeter/mythtv and select the "ffmpeg-subtree" >> branch. Notice it has 125000 commits, as opposed to the master branch, which >> has 34000. >> >> I think this is a good solution, as long as other developers are not too >> unhappy about the log situation. > The log bloat is unfortunate, but I think this solution is better than > the other alternatives so far. Thanks for testing all of this. > > David >

Do we really want to see all the ffmpeg commits in our git logs? I hate the idea :(