Expiry problem

I have been running a recent 0.25 pre-release(*) and have run into problems with expiry. What happens is that the disk fills up, the expiry thread runs, deletes the entry from the database, but does not delete the actual files containing the recording. Since the disk is still full, it goes ahead and deletes the next show and in a short space of time there is a full disk, and all programs which allow auto expire have been deleted from the DB. I recover the DB from backup and manually delete files which have been recorded since the backup, but obviously it is annoying to have to do this.

Possibly relevant is that a) I have a separate master and slave backend and b) am using NFS for the storage. The logs don't show anything interesting. Mythtv *thinks* it is doing the right thing. There are messages about expiring recordings, but nothing to do with failing to delete a file.

I'd appreciate any clues about what could cause this or how to debug it.

* I've upgraded again but it is too early to tell if the problem is still there.

On 18/04/2012 9:10 AM, Ian Dall wrote: > Hi, > > I have been running a recent 0.25 pre-release(*) and have run into > problems with expiry. What happens is that the disk fills up, the expiry > thread runs, deletes the entry from the database, but does not delete > the actual files containing the recording. Since the disk is still full, > it goes ahead and deletes the next show and in a short space of time > there is a full disk, and all programs which allow auto expire have been > deleted from the DB. I recover the DB from backup and manually delete > files which have been recorded since the backup, but obviously it is > annoying to have to do this. > > Possibly relevant is that a) I have a separate master and slave backend > and b) am using NFS for the storage. The logs don't show anything > interesting. Mythtv *thinks* it is doing the right thing. There are > messages about expiring recordings, but nothing to do with failing to > delete a file. > > I'd appreciate any clues about what could cause this or how to debug it. > > > * I've upgraded again but it is too early to tell if the problem is > still there. > More a question for the mythtv-users list. But sounds like a permission problem.

On Wed, 2012-04-18 at 10:37 +1000, Michael Watson wrote: > On 18/04/2012 9:10 AM, Ian Dall wrote: > > Hi, > > > > I have been running a recent 0.25 pre-release(*) and have run into > > problems with expiry. What happens is that the disk fills up, the expiry > > thread runs, deletes the entry from the database, but does not delete > > the actual files containing the recording. Since the disk is still full, > > it goes ahead and deletes the next show and in a short space of time > > there is a full disk, and all programs which allow auto expire have been > > deleted from the DB. I recover the DB from backup and manually delete > > files which have been recorded since the backup, but obviously it is > > annoying to have to do this. > > > > Possibly relevant is that a) I have a separate master and slave backend > > and b) am using NFS for the storage. The logs don't show anything > > interesting. Mythtv *thinks* it is doing the right thing. There are > > messages about expiring recordings, but nothing to do with failing to > > delete a file. > > > > I'd appreciate any clues about what could cause this or how to debug it. > > > > > > * I've upgraded again but it is too early to tell if the problem is > > still there. > > > More a question for the mythtv-users list. > But sounds like a permission problem.

Thanks, but I don't think so. Firstly I have checked, secondly this started happening after an upgrade with no change in configuration and thirdly I know of no unix or nfs permissions which allow create but not delete privilege and clearly files are being created or my disk wouldn't fill up ;-(

As for users vs dev I was expecting to debug this myself. I'd assume that if others were affected, it would have been noticed!

On 18/04/2012 6:34 PM, Ian Dall wrote: > On Wed, 2012-04-18 at 10:37 +1000, Michael Watson wrote: >> On 18/04/2012 9:10 AM, Ian Dall wrote: >>> Hi, >>> >>> I have been running a recent 0.25 pre-release(*) and have run into >>> problems with expiry. What happens is that the disk fills up, the expiry >>> thread runs, deletes the entry from the database, but does not delete >>> the actual files containing the recording. Since the disk is still full, >>> it goes ahead and deletes the next show and in a short space of time >>> there is a full disk, and all programs which allow auto expire have been >>> deleted from the DB. I recover the DB from backup and manually delete >>> files which have been recorded since the backup, but obviously it is >>> annoying to have to do this. >>> >>> Possibly relevant is that a) I have a separate master and slave backend >>> and b) am using NFS for the storage. The logs don't show anything >>> interesting. Mythtv *thinks* it is doing the right thing. There are >>> messages about expiring recordings, but nothing to do with failing to >>> delete a file. >>> >>> I'd appreciate any clues about what could cause this or how to debug it. >>> >>> >>> * I've upgraded again but it is too early to tell if the problem is >>> still there. >>> >> More a question for the mythtv-users list. >> But sounds like a permission problem. > Thanks, but I don't think so. Firstly I have checked, secondly this > started happening after an upgrade with no change in configuration and > thirdly I know of no unix or nfs permissions which allow create but not > delete privilege and clearly files are being created or my disk wouldn't > fill up ;-( That depends on your NFS configuration, and which machine writes the file, and which machine is trying to delete the file. The permissions of a file that has not been deleted, are they the say when viewed from each machine?

> > As for users vs dev I was expecting to debug this myself. I'd assume > that if others were affected, it would have been noticed! >

On Wed, 2012-04-18 at 18:44 +1000, Michael Watson wrote: > On 18/04/2012 6:34 PM, Ian Dall wrote: > > On Wed, 2012-04-18 at 10:37 +1000, Michael Watson wrote: > >> On 18/04/2012 9:10 AM, Ian Dall wrote: > >>> Hi, > >>> > >>> I have been running a recent 0.25 pre-release(*) and have run into > >>> problems with expiry. What happens is that the disk fills up, the expiry > >>> thread runs, deletes the entry from the database, but does not delete > >>> the actual files containing the recording. Since the disk is still full, > >>> it goes ahead and deletes the next show and in a short space of time > >>> there is a full disk, and all programs which allow auto expire have been > >>> deleted from the DB. I recover the DB from backup and manually delete > >>> files which have been recorded since the backup, but obviously it is > >>> annoying to have to do this. > >>> > >>> Possibly relevant is that a) I have a separate master and slave backend > >>> and b) am using NFS for the storage. The logs don't show anything > >>> interesting. Mythtv *thinks* it is doing the right thing. There are > >>> messages about expiring recordings, but nothing to do with failing to > >>> delete a file. > >>> > >>> I'd appreciate any clues about what could cause this or how to debug it. > >>> > >>> > >>> * I've upgraded again but it is too early to tell if the problem is > >>> still there. > >>> > >> More a question for the mythtv-users list. > >> But sounds like a permission problem. > > Thanks, but I don't think so. Firstly I have checked, secondly this > > started happening after an upgrade with no change in configuration and > > thirdly I know of no unix or nfs permissions which allow create but not > > delete privilege and clearly files are being created or my disk wouldn't > > fill up ;-( > That depends on your NFS configuration, and which machine writes the > file, and which machine is trying to delete the file. > The permissions of a file that has not been deleted, are they the say > when viewed from each machine?

Good point, but no. On each machine I did su mythtv touch /var/media/myth/mythtv/foo rm /var/media/myth/mythtv/foo ps -fp $(cat /var/run/mythbackend.pid)

and everything checks out.

As I said, I've upgraded mythtv to v0.26-pre-25-something and turned on "file" logging. I've also rebooted the virtual machine which was running mythmaster. It had been up for 59 days and was low on free memory and swap (I don't recall why). So I'll now wait and see if it happens again.

On 04/17/2012 07:10 PM, Ian Dall wrote: > I have been running a recent 0.25 pre-release(*) and have run into > problems with expiry. What happens is that the disk fills up, the expiry > thread runs, deletes the entry from the database, but does not delete > the actual files containing the recording. Since the disk is still full, > it goes ahead and deletes the next show and in a short space of time > there is a full disk, and all programs which allow auto expire have been > deleted from the DB. I recover the DB from backup and manually delete > files which have been recorded since the backup, but obviously it is > annoying to have to do this. > > Possibly relevant is that a) I have a separate master and slave backend > and b) am using NFS for the storage. The logs don't show anything > interesting. Mythtv *thinks* it is doing the right thing. There are > messages about expiring recordings, but nothing to do with failing to > delete a file. > > I'd appreciate any clues about what could cause this or how to debug it. >

Are the expired shows in the Deleted recording group? Watch Recordings, MENU|Change Group Filter, then select "Deleted" and see if any that the backend logs say were expired are there.

On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote: > On 04/17/2012 07:10 PM, Ian Dall wrote: > > I have been running a recent 0.25 pre-release(*) and have run into > > problems with expiry. What happens is that the disk fills up, the expiry > > thread runs, deletes the entry from the database, but does not delete > > the actual files containing the recording. Since the disk is still full, > > it goes ahead and deletes the next show and in a short space of time > > there is a full disk, and all programs which allow auto expire have been > > deleted from the DB. I recover the DB from backup and manually delete > > files which have been recorded since the backup, but obviously it is > > annoying to have to do this. > > > > Possibly relevant is that a) I have a separate master and slave backend > > and b) am using NFS for the storage. The logs don't show anything > > interesting. Mythtv *thinks* it is doing the right thing. There are > > messages about expiring recordings, but nothing to do with failing to > > delete a file. > > > > I'd appreciate any clues about what could cause this or how to debug it. > > > > Are the expired shows in the Deleted recording group? Watch Recordings, > MENU|Change Group Filter, then select "Deleted" and see if any that the > backend logs say were expired are there.

Ah! It took a while, but I think I have caught the misbehaviour in the act. I had a recording from an iptv "tuner". This should have been about 1 hour long, but the file was still open (as reported by lsof) and growing 2 hours later, even though the show showed as finished in "Watch Recordings". The entire file was watchable.

So, I deleted the recording through "Watch Recordings",Menu,Delete. It then disappeared from the "Watch Recordings" list, but the file was still on disk and still growing! Checking the "Deleted" group showed it there. Restarting the master backend (which also has the iptv tuner on it) at least stopped the file growing, but it was still there in the Deleted group.

It did eventually get deleted, presumably when the delete thread was run.

So it all makes sense except for the recording which never terminated problem. If this happens it would explain my original symptoms. It would also explain another strangeness. After my recovering from the db from restore and cleaning up, I'd expect only about 10% free space (only one day of shows), but was getting more like 50% free space. Presumably there was one or a few bloated files taking up most of the space.

On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote: > On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote: > > On 04/17/2012 07:10 PM, Ian Dall wrote: > > > I have been running a recent 0.25 pre-release(*) and have run into > > > problems with expiry. What happens is that the disk fills up, the expiry > > > thread runs, deletes the entry from the database, but does not delete > > > the actual files containing the recording. Since the disk is still full, > > > it goes ahead and deletes the next show and in a short space of time > > > there is a full disk, and all programs which allow auto expire have been > > > deleted from the DB. I recover the DB from backup and manually delete > > > files which have been recorded since the backup, but obviously it is > > > annoying to have to do this. > > > > > > Possibly relevant is that a) I have a separate master and slave backend > > > and b) am using NFS for the storage. The logs don't show anything > > > interesting. Mythtv *thinks* it is doing the right thing. There are > > > messages about expiring recordings, but nothing to do with failing to > > > delete a file. > > > > > > I'd appreciate any clues about what could cause this or how to debug it. > > > > > > > Are the expired shows in the Deleted recording group? Watch Recordings, > > MENU|Change Group Filter, then select "Deleted" and see if any that the > > backend logs say were expired are there. > > Ah! It took a while, but I think I have caught the misbehaviour in the > act. I had a recording from an iptv "tuner". This should have been about > 1 hour long, but the file was still open (as reported by lsof) and > growing 2 hours later, even though the show showed as finished in "Watch > Recordings". The entire file was watchable.

And I should add, in the system status information the ipv "tuner" was shown as NOT recording...

On 04/22/2012 11:26 AM, Ian Dall wrote: > On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote: >> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote: >>> On 04/17/2012 07:10 PM, Ian Dall wrote: >>>> I have been running a recent 0.25 pre-release(*) and have run into >>>> problems with expiry. What happens is that the disk fills up, the expiry >>>> thread runs, deletes the entry from the database, but does not delete >>>> the actual files containing the recording. Since the disk is still full, >>>> it goes ahead and deletes the next show and in a short space of time >>>> there is a full disk, and all programs which allow auto expire have been >>>> deleted from the DB. I recover the DB from backup and manually delete >>>> files which have been recorded since the backup, but obviously it is >>>> annoying to have to do this. >>>> >>>> Possibly relevant is that a) I have a separate master and slave backend >>>> and b) am using NFS for the storage. The logs don't show anything >>>> interesting. Mythtv *thinks* it is doing the right thing. There are >>>> messages about expiring recordings, but nothing to do with failing to >>>> delete a file. >>>> >>>> I'd appreciate any clues about what could cause this or how to debug it. >>>> >>> Are the expired shows in the Deleted recording group? Watch Recordings, >>> MENU|Change Group Filter, then select "Deleted" and see if any that the >>> backend logs say were expired are there. >> Ah! It took a while, but I think I have caught the misbehaviour in the >> act. I had a recording from an iptv "tuner". This should have been about >> 1 hour long, but the file was still open (as reported by lsof) and >> growing 2 hours later, even though the show showed as finished in "Watch >> Recordings". The entire file was watchable. > And I should add, in the system status information the ipv "tuner" was > shown as NOT recording...

On Sun, 2012-04-22 at 18:37 -0400, Michael T. Dean wrote: > On 04/22/2012 11:26 AM, Ian Dall wrote: > > On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote: > >> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote: > >>> On 04/17/2012 07:10 PM, Ian Dall wrote: > >>>> I have been running a recent 0.25 pre-release(*) and have run into > >>>> problems with expiry. What happens is that the disk fills up, the expiry > >>>> thread runs, deletes the entry from the database, but does not delete > >>>> the actual files containing the recording. Since the disk is still full, > >>>> it goes ahead and deletes the next show and in a short space of time > >>>> there is a full disk, and all programs which allow auto expire have been > >>>> deleted from the DB. I recover the DB from backup and manually delete > >>>> files which have been recorded since the backup, but obviously it is > >>>> annoying to have to do this. > >>>> > >>>> Possibly relevant is that a) I have a separate master and slave backend > >>>> and b) am using NFS for the storage. The logs don't show anything > >>>> interesting. Mythtv *thinks* it is doing the right thing. There are > >>>> messages about expiring recordings, but nothing to do with failing to > >>>> delete a file. > >>>> > >>>> I'd appreciate any clues about what could cause this or how to debug it. > >>>> > >>> Are the expired shows in the Deleted recording group? Watch Recordings, > >>> MENU|Change Group Filter, then select "Deleted" and see if any that the > >>> backend logs say were expired are there. > >> [iptv problem] Unfortunately it seems that the iptv problem is not the cause of the fail to delete option.

So, the disk filled up again, and yes, it ended up with about 90 recordings in the "Deleted" recording group. Restarting the master backend causes all the files to be deleted. I notice in the logs that there are successful regular deletions, (signalled by "About to delete file" messages) up until about 2:30pm then nothing until after the backend has been restarted at 11pm (after the disk space is exhausted), when all the files in the "Deleted" recording group get deleted.

I notice that the delete thread does check that the file has gone (and would produce an error message if it isn't) but I don't see any of these error messages.

It looks like the DeleteThread is simply not running. A check with gdb doesn't seem to show any deadlocked threads, but it is not always easy to easy to identify which thread is which.

One other point. I notice error messages to do with the "recordedfile" table. The recordedfile table exists but is empty. A search of the sources shows SELECT and DELETE actions on this table, but no INSERT. I assume that this is a legacy table and these messages can be ignored?

On 04/22/2012 06:37 PM, Michael T. Dean wrote: > On 04/22/2012 11:26 AM, Ian Dall wrote: >> On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote: >>> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote: >>>> On 04/17/2012 07:10 PM, Ian Dall wrote: >>>>> I have been running a recent 0.25 pre-release(*) and have run into >>>>> problems with expiry. What happens is that the disk fills up, the >>>>> expiry >>>>> thread runs, deletes the entry from the database, but does not delete >>>>> the actual files containing the recording. Since the disk is still >>>>> full, >>>>> it goes ahead and deletes the next show and in a short space of time >>>>> there is a full disk, and all programs which allow auto expire >>>>> have been >>>>> deleted from the DB. I recover the DB from backup and manually delete >>>>> files which have been recorded since the backup, but obviously it is >>>>> annoying to have to do this. >>>>> >>>>> Possibly relevant is that a) I have a separate master and slave >>>>> backend >>>>> and b) am using NFS for the storage. The logs don't show anything >>>>> interesting. Mythtv *thinks* it is doing the right thing. There are >>>>> messages about expiring recordings, but nothing to do with failing to >>>>> delete a file. >>>>> >>>>> I'd appreciate any clues about what could cause this or how to >>>>> debug it. >>>>> >>>> Are the expired shows in the Deleted recording group? Watch >>>> Recordings, >>>> MENU|Change Group Filter, then select "Deleted" and see if any that >>>> the >>>> backend logs say were expired are there. >>> Ah! It took a while, but I think I have caught the misbehaviour in the >>> act. I had a recording from an iptv "tuner". This should have been >>> about >>> 1 hour long, but the file was still open (as reported by lsof) and >>> growing 2 hours later, even though the show showed as finished in >>> "Watch >>> Recordings". The entire file was watchable. >> And I should add, in the system status information the ipv "tuner" was >> shown as NOT recording... > > http://code.mythtv.org/trac/ticket/9670 (specifically > http://code.mythtv.org/trac/ticket/9670#comment:31 ) > and > http://code.mythtv.org/trac/ticket/10620

On 04/23/2012 06:55 PM, Ian Dall wrote: > On Sun, 2012-04-22 at 18:37 -0400, Michael T. Dean wrote: >> On 04/22/2012 11:26 AM, Ian Dall wrote: >>> On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote: >>>> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote: >>>>> On 04/17/2012 07:10 PM, Ian Dall wrote: >>>>>> I have been running a recent 0.25 pre-release(*) and have run into >>>>>> problems with expiry. What happens is that the disk fills up, the expiry >>>>>> thread runs, deletes the entry from the database, but does not delete >>>>>> the actual files containing the recording. Since the disk is still full, >>>>>> it goes ahead and deletes the next show and in a short space of time >>>>>> there is a full disk, and all programs which allow auto expire have been >>>>>> deleted from the DB. I recover the DB from backup and manually delete >>>>>> files which have been recorded since the backup, but obviously it is >>>>>> annoying to have to do this. >>>>>> >>>>>> Possibly relevant is that a) I have a separate master and slave backend >>>>>> and b) am using NFS for the storage. The logs don't show anything >>>>>> interesting. Mythtv *thinks* it is doing the right thing. There are >>>>>> messages about expiring recordings, but nothing to do with failing to >>>>>> delete a file. >>>>>> >>>>>> I'd appreciate any clues about what could cause this or how to debug it. >>>>>> >>>>> Are the expired shows in the Deleted recording group? Watch Recordings, >>>>> MENU|Change Group Filter, then select "Deleted" and see if any that the >>>>> backend logs say were expired are there. >>>> [iptv problem] > Unfortunately it seems that the iptv problem is not the cause of the > fail to delete option. > > So, the disk filled up again, and yes, it ended up with about 90 > recordings in the "Deleted" recording group. Restarting the master > backend causes all the files to be deleted. I notice in the logs that > there are successful regular deletions, (signalled by "About to delete > file" messages) up until about 2:30pm then nothing until after the > backend has been restarted at 11pm (after the disk space is exhausted), > when all the files in the "Deleted" recording group get deleted. > > I notice that the delete thread does check that the file has gone (and > would produce an error message if it isn't) but I don't see any of these > error messages. > > It looks like the DeleteThread is simply not running. A check with gdb > doesn't seem to show any deadlocked threads, but it is not always easy > to easy to identify which thread is which.

It sounds to me like MythTV starts an iptv recording and "stops" it, but the recording keeps going without MythTV's "conscious" knowledge. These recordings fill up the file system, and MythTV begins to expire recordings, but because they're still open/in use, they don't actually get removed and, perhaps, cause the delete thread to hang. Restarting mythbackend closes all open files and means that next time you restart it will resume deleting the files it had in the queue, which are no longer open, and will get removed.

> One other point. I notice error messages to do with the "recordedfile" > table. The recordedfile table exists but is empty. A search of the > sources shows SELECT and DELETE actions on this table, but no INSERT. I > assume that this is a legacy table and these messages can be ignored? >

It's nothing--and about to be replaced. We changed the plan for it, and it has never actually been in use for anything.

On Mon, 2012-04-23 at 21:50 -0400, Michael T. Dean wrote: > On 04/23/2012 06:55 PM, Ian Dall wrote: > > On Sun, 2012-04-22 at 18:37 -0400, Michael T. Dean wrote: > >> On 04/22/2012 11:26 AM, Ian Dall wrote: > >>> On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote: > >>>> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote: > >>>>> On 04/17/2012 07:10 PM, Ian Dall wrote: > >>>>>> I have been running a recent 0.25 pre-release(*) and have run into > >>>>>> problems with expiry. What happens is that the disk fills up, the expiry > >>>>>> thread runs, deletes the entry from the database, but does not delete > >>>>>> the actual files containing the recording. Since the disk is still full, > >>>>>> it goes ahead and deletes the next show and in a short space of time > >>>>>> there is a full disk, and all programs which allow auto expire have been > >>>>>> deleted from the DB. I recover the DB from backup and manually delete > >>>>>> files which have been recorded since the backup, but obviously it is > >>>>>> annoying to have to do this. > >>>>>> > >>>>>> Possibly relevant is that a) I have a separate master and slave backend > >>>>>> and b) am using NFS for the storage. The logs don't show anything > >>>>>> interesting. Mythtv *thinks* it is doing the right thing. There are > >>>>>> messages about expiring recordings, but nothing to do with failing to > >>>>>> delete a file. > >>>>>> > >>>>>> I'd appreciate any clues about what could cause this or how to debug it. > >>>>>> > >>>>> Are the expired shows in the Deleted recording group? Watch Recordings, > >>>>> MENU|Change Group Filter, then select "Deleted" and see if any that the > >>>>> backend logs say were expired are there. > >>>> [iptv problem] > > Unfortunately it seems that the iptv problem is not the cause of the > > fail to delete option. > > > > So, the disk filled up again, and yes, it ended up with about 90 > > recordings in the "Deleted" recording group. Restarting the master > > backend causes all the files to be deleted. I notice in the logs that > > there are successful regular deletions, (signalled by "About to delete > > file" messages) up until about 2:30pm then nothing until after the > > backend has been restarted at 11pm (after the disk space is exhausted), > > when all the files in the "Deleted" recording group get deleted. > > > > I notice that the delete thread does check that the file has gone (and > > would produce an error message if it isn't) but I don't see any of these > > error messages. > > > > It looks like the DeleteThread is simply not running. A check with gdb > > doesn't seem to show any deadlocked threads, but it is not always easy > > to easy to identify which thread is which. > > It sounds to me like MythTV starts an iptv recording and "stops" it, but > the recording keeps going without MythTV's "conscious" knowledge. These > recordings fill up the file system, and MythTV begins to expire > recordings, but because they're still open/in use, they don't actually > get removed and, perhaps, cause the delete thread to hang.

That's what I was thinking too, but I deleted the iptv "tuner" before this latest round of trouble, which is why I concluded this is a separate problem.

On Mon, 2012-04-23 at 21:50 -0400, Michael T. Dean wrote: > On 04/23/2012 06:55 PM, Ian Dall wrote: > > On Sun, 2012-04-22 at 18:37 -0400, Michael T. Dean wrote: > >> On 04/22/2012 11:26 AM, Ian Dall wrote: > >>> On Mon, 2012-04-23 at 00:44 +0930, Ian Dall wrote: > >>>> On Wed, 2012-04-18 at 21:32 -0400, Michael T. Dean wrote: > >>>>> On 04/17/2012 07:10 PM, Ian Dall wrote: > >>>>>> I have been running a recent 0.25 pre-release(*) and have run into > >>>>>> problems with expiry. What happens is that the disk fills up, the expiry > >>>>>> thread runs, deletes the entry from the database, but does not delete > >>>>>> the actual files containing the recording. Since the disk is still full, > >>>>>> it goes ahead and deletes the next show and in a short space of time > >>>>>> there is a full disk, and all programs which allow auto expire have been > >>>>>> deleted from the DB. I recover the DB from backup and manually delete > >>>>>> files which have been recorded since the backup, but obviously it is > >>>>>> annoying to have to do this. > >>>>>> > >>>>>> Possibly relevant is that a) I have a separate master and slave backend > >>>>>> and b) am using NFS for the storage. The logs don't show anything > >>>>>> interesting. Mythtv *thinks* it is doing the right thing. There are > >>>>>> messages about expiring recordings, but nothing to do with failing to > >>>>>> delete a file. > >>>>>> > >>>>>> I'd appreciate any clues about what could cause this or how to debug it. > >>>>>> > >>>>> Are the expired shows in the Deleted recording group? Watch Recordings, > >>>>> MENU|Change Group Filter, then select "Deleted" and see if any that the > >>>>> backend logs say were expired are there. > >>>> [iptv problem] > > Unfortunately it seems that the iptv problem is not the cause of the > > fail to delete option. > > > > So, the disk filled up again, and yes, it ended up with about 90 > > recordings in the "Deleted" recording group. Restarting the master > > backend causes all the files to be deleted. I notice in the logs that > > there are successful regular deletions, (signalled by "About to delete > > file" messages) up until about 2:30pm then nothing until after the > > backend has been restarted at 11pm (after the disk space is exhausted), > > when all the files in the "Deleted" recording group get deleted. > > > > I notice that the delete thread does check that the file has gone (and > > would produce an error message if it isn't) but I don't see any of these > > error messages. > > > > It looks like the DeleteThread is simply not running. A check with gdb > > doesn't seem to show any deadlocked threads, but it is not always easy > > to easy to identify which thread is which. > > It sounds to me like MythTV starts an iptv recording and "stops" it, but > the recording keeps going without MythTV's "conscious" knowledge. These > recordings fill up the file system, and MythTV begins to expire > recordings, but because they're still open/in use, they don't actually > get removed and, perhaps, cause the delete thread to hang. Restarting > mythbackend closes all open files and means that next time you restart > it will resume deleting the files it had in the queue, which are no > longer open, and will get removed.

As noted, I have ruled out iptv by deleting the tuner.

I have also been capturing the network traffic between master and slave. I have also set gdb breakpoints in the slave, and /still/ I don't understand what is going on ;-(

What I can say for sure is that the slave gets a DELETE_RECORDING command. The recording is, at this stage in the "Default" recording group so MainServer::DoHandleDeleteRecording (mainserver.cpp:2468) figures that "justexpire" is true, changes the recording group to "Deleted" and returns. The code to start the DeleteThread is after the return and is never executed (on the slave at least).

I'm having trouble figuring out which threads are supposed to persist, which get started on demand and which get started from timers. Particularly in the case of the DeleteThread, how is it supposed to be started? I also am having trouble figuring out why delete involves the slave at all. The recording is available to the master so why doesn't the master just delete it?