I am having issues with the expect module. I have a program that telnet's into a UNIX box that passes me into a proprietary system. I then execute a cat like command and wait for the output to finish printing to the screen. I then call EXP_BEFORE and store off the data. This process has worked for me for years but now the EXP_BEFORE is dropping some data. I upgraded to expect 1.21, tried setting raw_pty and the max accumulator, I also make sure I clear the accumulator before. I still get the same issues. This is a random issue with files larger than 100,000 lines. I can repeated the process several times and get the data correctly, but the system is slow and this file can take up to 2 hours or more to cat.

Just an FYI I cannot FTP the data off due to system limitations on the box.

I am running this on a HP box with 16 processors and lots of memory so I do not think that the box is the issue.

You might get better results if you try grabbing the file in chunks and stitching them together. One method might be to build an expect loop around the less command, instead of trying to cat all 100k lines at a time.

Unfortunately I cannot control the far end system like that (it is a proprietary system and does not support this). All I can do is a full dump with no control. Is it possible in expect to call exp_before then clear_accum and keep looping this until the file is done or will I loss parts of the file?

No idea. Is it possible to get the byte count of the remote file? From your post, it seems like the dropouts are intermittent. If you know how big the file is, you can keep trying until you get it all. Kinda sucks, but then, this whole interface sounds like a "it's a miracle it works at all" lashup.

No I cannot get a bit count but I do know when the files are right or not. The system displays a line count. I do not want to go back and try again because the time it takes to dump the file. That is what I am doing now.

I also know that the far end system is not at fault. When I turn on logging I see the whole file in STD out without errors. I would hate to pipe STD out to a file and parse it from there.

If you can figure out how to transfer the file using one of the serial xfer protocols in an interactive telnet session, then you can probably script it. Any of these is likely to give you a better result than simply cat'ing the file to the session.