I was afraid to hear that. Is there a discussion somewhere that is working on getting the TS downloads working properly with kmttg, tivodecode, etc? I'd be interested in participating in getting it to be more stable

I was afraid to hear that. Is there a discussion somewhere that is working on getting the TS downloads working properly with kmttg, tivodecode, etc? I'd be interested in participating in getting it to be more stable

i.e. source code is available if someone wants to take a crack at fixing tivodecode. But that still doesn't help if TS downloads from Premiere units are not reliable - only TiVo can fix that and they have much bigger fish to fry when it comes to the Premiere...

Yea, I made sure I have everything updated to the newest version. I'm not sure why its downloading so slow. I restarted a TS download to see if there would be a difference. It started coming at about 50Mbps for maybe 5 seconds, and has since dropped to an average of 10Mbps. I'm not sure what is causing that - I know the network is fine, and if I initiate a PS download it comes a little faster.

Besides that, I'm going to see if redownloading the TS file works with tivodecode this time, otherwise I'll just be leaving TS off for now until things get fixed...

I have to agree. I'm not that excited about the Tivo Premiere. I thought, if nothing else, it would be faster to navigate through the menus and whatnot, but it definately is slower than my Tivo HD. And theres no menu option to initiate a restart?

Its currently at 14.4 version, and I cant figure out how to get it to update to 14.5, not that I actually expect that to fix any of my problems, but tivo.com says 14.5 is newest...

I have to agree. I'm not that excited about the Tivo Premiere. I thought, if nothing else, it would be faster to navigate through the menus and whatnot, but it definately is slower than my Tivo HD. And theres no menu option to initiate a restart?

I know it's off topic, but: If I had a Premiere I would just stick to SD menus. From all accounts that is stable and much faster than THD/S3 units. Also PS downloads are much faster than THD/S3 units as well so I would be content with that as well.

@moyekj: I agree on the Premiere HDUI. I wish they would just offer an option to widen the existing SD menus and use a smaller font. It would still run very fast on the new hardware, but skip all that Flash mess until some other time (or not.) They should even back-port that to the S3/THD as well. Seeing as they call that an HD unit, it seems like it should have menus that optionally support that format optimally.

@marbordom and others experiencing the stalled downloads in curl on Windows:

It appears I have found a solution to this issue on my system. I installed the Cygwin version of curl and set kmttg to use that instead of the version that is packaged with it. If you are not familiar, Cygwin is a *NIX-style command shell and tools emulator for Windows. They have ported most of the library code that the *NIX tools require and can compile them to run in Windows, and they actually have done a very good job with most of them. Cygwin is a bit unconventional to most Windows users, so most packages including curl have native Windows distributions that run as standalone apps. I think the advantage in this case is that curl for Cygwin is compiled in a unique way and runs through a different set of APIs compared to the native Windows curl distribution. This looks like it is a fluke that just happens to work in our favor here. Whatever is happening does not seem to effect the Cygwin version. The curl developers will still need to find the root cause of the problem, but this buys us some time by giving us a relatively easy way around the issue for now.

To do it, just run the Cygwin setup program and then be sure to enable the curl package to be installed. Then change the kmttg curl path to point at c:\Cygwin\bin\curl.exe, or wherever it is installed. The Cygwin curl will run fine even if you are not in the Cygwin shell. Now your log entries should show curl running from that new path and with any luck you will also find that your downloads are working right again.

Interesting find ferror. NOTE: You don't need a whole cygwin installation to get it to work. I pulled out curl.exe and all the necessary accompanying cygwin .dll files and made a self-contained package that runs without having cygwin installed. Those having this issue can try out this version of curl:http://kmttg.googlecode.com/files/cygwin_curl.zip
(Rename the kmttg "curl" folder and replace with the one in the above zip file).

Thanks. Yeah, sorry installing Cygwin is a bit extreme, was just a quick way to get the desired result. I was just thinking about looking at what files would be needed to just run curl and then saw your post. Putting this version of curl into the kmttg dist would actually solve the problem completely, at least for the time being. Then the curl developers fixing it in their windows version would just be an interesting bit of news for building the next bundle.

EDIT: I have one thing to add regarding the cygwin curl. If you happen to already have cygwin installed on your machine for any other reason, you should use the cygwin setup program to install/update curl and not use a separate version of curl. The reason is that the .dll files will probably get out of sync.

@moyekj: I agree on the Premiere HDUI. I wish they would just offer an option to widen the existing SD menus and use a smaller font. It would still run very fast on the new hardware, but skip all that Flash mess until some other time (or not.) They should even back-port that to the S3/THD as well. Seeing as they call that an HD unit, it seems like it should have menus that optionally support that format optimally.

@marbordom and others experiencing the stalled downloads in curl on Windows:

It appears I have found a solution to this issue on my system. I installed the Cygwin version of curl and set kmttg to use that instead of the version that is packaged with it. If you are not familiar, Cygwin is a *NIX-style command shell and tools emulator for Windows. They have ported most of the library code that the *NIX tools require and can compile them to run in Windows, and they actually have done a very good job with most of them. Cygwin is a bit unconventional to most Windows users, so most packages including curl have native Windows distributions that run as standalone apps. I think the advantage in this case is that curl for Cygwin is compiled in a unique way and runs through a different set of APIs compared to the native Windows curl distribution. This looks like it is a fluke that just happens to work in our favor here. Whatever is happening does not seem to effect the Cygwin version. The curl developers will still need to find the root cause of the problem, but this buys us some time by giving us a relatively easy way around the issue for now.

To do it, just run the Cygwin setup program and then be sure to enable the curl package to be installed. Then change the kmttg curl path to point at c:\Cygwin\bin\curl.exe, or wherever it is installed. The Cygwin curl will run fine even if you are not in the Cygwin shell. Now your log entries should show curl running from that new path and with any luck you will also find that your downloads are working right again.

That find is awesome. Yesterday I ran kmttg under Ubuntu and even there I found the curl problem - I had to restart the Tivo since resetting the web server did not solve that no connection could be made to the tivo's IP. Now I have to fix my Win 7 before I can trye your solution .. after shrinking the Win7 partition I can not boot into this or the recover partition ...

Interesting find ferror. NOTE: You don't need a whole cygwin installation to get it to work. I pulled out curl.exe and all the necessary accompanying cygwin .dll files and made a self-contained package that runs without having cygwin installed. Those having this issue can try out this version of curl:http://kmttg.googlecode.com/files/cygwin_curl.zip
(Rename the kmttg "curl" folder and replace with the one in the above zip file).

Hi, when I use the unzipped dir under kmttg the curl.exe can not find cygz.dll. Since I do not know which version this it copying one would just be another trial and error. Other than that I do have cygwin installed and the version installed on my Win7 is not running either (gets stalled)
[xxx@acer]$ /bin/curl --version
curl 7.20.1 (i686-pc-cygwin) libcurl/7.20.1 OpenSSL/0.9.8o zlib/1.2.5 libidn/1.1
8 libssh2/1.2.5
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s rtsp scp sftp smt
p smtps telnet tftp
Features: IDN IPv6 Largefile NTLM SSL libz

Hmm, sounds like maybe the zip needs another file, but if you are running cygwin now anyway, that should do the trick. You are running the exact same version I am running here. Mine actually did stall again once during the afternoon today, but I've seen that happen before for other reasons. Before I switched to the cygwin curl last night, the stalls were happening on almost every file at least once or twice. It was a very acute issue, not just an annoyance. The -y 60 -Y 100 parameters did help to make it eventually get the files downloaded. When I was testing the cygwin curl, I took those parameters out to see how it handles it without them. When it stalled this afternoon, I decided to go ahead and put them back as an additional measure against the issue. Looks like the stalls can still happen, but in my case the cygwin curl reduced those stalls to such a low amount that it took a day of continuous running before I saw it happen again. Sorry you aren't having as much luck with yours. I am not sure what the variables are in this equation, but I think we are close.

Hi, when I use the unzipped dir under kmttg the curl.exe can not find cygz.dll. Since I do not know which version this it copying one would just be another trial and error. Other than that I do have cygwin installed and the version installed on my Win7 is not running either (gets stalled)
[xxx@acer]$ /bin/curl --version
curl 7.20.1 (i686-pc-cygwin) libcurl/7.20.1 OpenSSL/0.9.8o zlib/1.2.5 libidn/1.1
8 libssh2/1.2.5
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s rtsp scp sftp smt
p smtps telnet tftp
Features: IDN IPv6 Largefile NTLM SSL libz

BTW, I've now downloaded over 40 different shows from my S3s to a Win 7 laptop and have not had a single freeze or incomplete download, so I cannot reproduce this issue at all.
Are these curl freezes when downloading from Premiere units only?

For me it happens on either of my TiVo HDs and on my Premiere (which I thought I was getting to replace one the the THDs, but never wound up selling it). It seems to happen more often on the Premiere, than the THDs, but I think this is just because the download speed is faster and so it is that much more likely to fail. Due to the faster transfer speed, it may also increase the likelihood of getting a stall. I have not tried it on anything other than win7 but marbodom said he saw it on Ubuntu as well, which was a big surprise to me. Maybe the issue goes a bit deeper than originally thought.

Thanks for the quick reply, but whew! Me eyes are too tired too feed all this to my brain right now. And that's a good thing because it couldn't process it all tonight anyway. I'll re-read later when I have time to digest and reply. My first thought was this may have to do with padding recordings (a minute or more early and/or late) and chunk1 vs. chunk2 of metadata where <vActualShowing> vs. <showing> has the data.

Main concern is getting seriesId in there, programId is secondary, but why leave it out just because it's not documented in pyTivo metadata Wiki, does it make something choke on it?.

Comment 14 by moy...@yahoo.com, Sep 28, 2009
episodeNumber twice I've fixed. Wrong seriesId I kind of expected. The problem is the
way I'm parsing the xml is very dumb. I don't keep track of what keys I'm currently
inside at more than 1 level deep, so don't know if inside <series> element when
parsing. To do that properly I'd need to replace with a proper full blown xml parser
that keeps track of element hierarchy. I'll see if there's some Java code out there
for xml parsing that I can borrow from. Other than seriesId though I think the rest
seems to work, so perhaps for now just leaving seriesId out is the next best thing
(until a proper xml parser is implemented).

I think I understand now (until a proper xml parser is implemented).

I can send you samples of metadata from recordings where I;

A) don't add anything to the scheduled guide recording time.
B) start recording a minute before the scheduled recording time.
C) stop recording a minute after the sheduled recording time.
D) start recording a minute before and stop recording a minute after the scheduled recording time.

For each instance of A to D above I can include;
1) recursive query to my TiVo's NPL
2) query to my TiVo for video details of the recording
3) output from tivodecode/tcat for that same recording from the .tivo file after transfer to my PC
4) output from kmttg after trying to create metadata from said .tivo file

So, if it would help I could try this and send you all 16 files (A1-4, B1-4, C1-4, D1-4).

i.e.

Code:

A) don't add anything to the scheduled guide recording time.
1) recursive query to my TiVo's NPL
2) query to my TiVo for video details of the recording
3) output from tivodecode/tcat for that same recording from the .tivo file after transfer to my PC
4) output from kmttg after trying to create metadata from said .tivo file
B) start recording a minute before the scheduled recording time.
1) recursive query to my TiVo's NPL
2) query to my TiVo for video details of the recording
3) output from tivodecode/tcat for that same recording from the .tivo file after transfer to my PC
4) output from kmttg after trying to create metadata from said .tivo file
C) stop recording a minute after the sheduled recording time.
1) recursive query to my TiVo's NPL
2) query to my TiVo for video details of the recording
3) output from tivodecode/tcat for that same recording from the .tivo file after transfer to my PC
4) output from kmttg after trying to create metadata from said .tivo file
D) start recording a minute before and stop recording a minute after the scheduled recording time.
1) recursive query to my TiVo's NPL
2) query to my TiVo for video details of the recording
3) output from tivodecode/tcat for that same recording from the .tivo file after transfer to my PC
4) output from kmttg after trying to create metadata from said .tivo file

Only thing is I can't help you parse XML files and am not interested in getting back into coding (past couple lives of that burnt me out;-)

To be honest my motivation is pretty low to implement proper xml parser just to address this seriesId issue. There are full xml parsers available out there already but that adds a pretty big bloat of a library to include in kmttg.jar so don't really want to go there. For video transfers to TiVos they end up without programId anyway so not sure what the point is to include that field in metadata file? For seriesId I can understand wanting it in there since it does affect grouping when transferring back to TiVos. I'll look into that more when time permits - right now my real job is taking up most of my time.

Still using this great program for a couple of years now. I recorded Redeye off ABC the other night and I was able to download the .tivo file, but on 2 different pc's when I try to decode the files I am presented with the following error:

I have no problem with other tivo recordings and all this is the first one I had issue with. Any ideas on what is happening? It errors out if I run straight download and decode and it errors out if I download and then add the file to be decoded..its erroring out during the decode. Its not an invalid MAK, its the same one I use for all my other shows and I have no problem with any other recorded show.

Any ideas here? I tried searching but all I found was where folks actually had bad MAK #'s.

To be honest my motivation is pretty low to implement proper xml parser just to address this seriesId issue. There are full xml parsers available out there already but that adds a pretty big bloat of a library to include in kmttg.jar so don't really want to go there. For video transfers to TiVos they end up without programId anyway so not sure what the point is to include that field in metadata file? For seriesId I can understand wanting it in there since it does affect grouping when transferring back to TiVos. I'll look into that more when time permits - right now my real job is taking up most of my time.

Thanks for the response

I downloaded and am using pyTivoMetaGen, which meets my needs for this purpose. I guess one could always set up a custom script to call it and not check the metadata checkbox in kmttg when using it to transfer .tivo files from the TiVo to PC.

Still using this great program for a couple of years now. I recorded Redeye off ABC the other night and I was able to download the .tivo file, but on 2 different pc's when I try to decode the files I am presented with the following error:

I have no problem with other tivo recordings and all this is the first one I had issue with. Any ideas on what is happening? It errors out if I run straight download and decode and it errors out if I download and then add the file to be decoded..its erroring out during the decode. Its not an invalid MAK, its the same one I use for all my other shows and I have no problem with any other recorded show.

Any ideas here? I tried searching but all I found was where folks actually had bad MAK #'s.

It's probably due to an incomplete download? Have you checked the source .TiVo file with Mediainfo to see if it is a full length recording? Alternatively if you have at least partial TiVo Desktop installed (i.e. TiVo directshow filter installed) make sure it plays all the way through with Windows Media Player. Would also compare the .TiVo file size to the .mpg file size to see how close they are.

BTW, I've now downloaded over 40 different shows from my S3s to a Win 7 laptop and have not had a single freeze or incomplete download, so I cannot reproduce this issue at all.
Are these curl freezes when downloading from Premiere units only?

With this combination curl gets going.
So my final thoughts here are that at least two issues were hitting:
1) some version or curl with some dll from windows causes a stall of the Tivo download (not in TS, cant say)
2) due to this stall a curl binary will still be running in taskmgr. This causes the Tivo http server to stay connected. Since there is a limit of one download per Tivo it is required to terminate that stalled curl and either wait or reboot the Tivo. Exiting kmttg does not terminate this curl process. Latter brute force approach ist not always necessary. Sometimes a wait will let the http interface come back to live . Otherwise you get "http server busy" message or "transient error" from curl.

Sure is that the root cause is not clear and there are workarounds which will work for some but not for all - very much what I expect from the Windows world.

Thanks for the quick reply, but whew! Me eyes are too tired too feed all this to my brain right now. And that's a good thing because it couldn't process it all tonight anyway. I'll re-read later when I have time to digest and reply. My first thought was this may have to do with padding recordings (a minute or more early and/or late) and chunk1 vs. chunk2 of metadata where <vActualShowing> vs. <showing> has the data.

Main concern is getting seriesId in there, programId is secondary, but why leave it out just because it's not documented in pyTivo metadata Wiki, does it make something choke on it?.

FWIW I did find a fairly simple & standard way using DOM for parsing xml files which is part of normal Java distribution which made it easy to parse get seriesId out of .TiVo metadata information. That is already checked in to source tree and will be part of next release. (Note that when starting from downloads instead of .TiVo files seriesId was already taken care of). If desired I can upload a beta kmttg.jar with that change, but since you already are using a different metadata generator perhaps it's a don't care at this point.

Some shows apparently have glitches in them that prevent downloads from completing, and in some cases a reboot might cure problems. I don't run into the issues so far with my S3s but some do. Easy way to check is paste the url from the command above into your browser to do the download (actually if you just click on the link in your post it should work for you). Username is "tivo", password is your 10 digit MAK #. If that still fails (you get an incomplete download) then it's a TiVo issue that you can't really get around - re-recording the show if it airs again is one possibility.

I've run into periodic download failures from my S3's too. There is no error given - data transfer just simply stops - the relatively new retry logic does not get it going again. I've even tried using curl directly and had the same issue. I haven't yet found if there are any command line options that would make the problem go away.

One further piece of information - I don't know if it's relevant or not, but I wrote a perl version of a kmttg-like program, but it does the download in native perl instead of using curl underneath. I had periodic problems with it where a transfer would simply stop because the socket interface returned EWOULDBLOCK as an error code. I changed the perl module (I didn't like doing this) to tolerate this condition by simply sleeping for a bit and then retrying. Since I made this change, I haven't had a failure.

I've seen something similar on a podcast downloader named juice. Transfers would halt because of EWOULDBLOCK.

Is it possible that this is the underlying problem people are encountering when using curl? I'm not sure, but I though I'd throw that out there.

I virtually never have any issues with downloads from both my S3s and I have literally done thousands of downloads by now. Once or twice I remember a condition where no downloads would start at all and a reboot was in order, but I've never had partial download issues. Also for recent posts about troubles with Win7 downloads and curl hangups I could not reproduce on Win7 laptop, so apparently I'm not a good test subject for debugging these problems.
However I've seen enough posts indicating that downloading with other methods typically have same/similar problems as with curl, so I would more likely suspect TiVo server side problems than the client side pulling shows.