If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

// This reads in the file
BufferedReader inFile = new BufferedReader (new
FileReader ("Date.java"));
// Creates the stream for the file
StreamTokenizer streamT = new StreamTokenizer(inFile);
streamT.wordChars(33,47); //characters make up words .
streamT.wordChars(58,64); // 48-57 are numerals 58-64 are some punctuation stuff
streamT.wordChars(91,96); // 65-90 are already wordchars 91-96 are more punctuation
streamT.wordChars(123,126);// 97-121 are already wordchars 123-126 are more punctuation
streamT.parseNumbers(); // this says to recognise numbers as numbers.
streamT.eolIsSignificant(true); // the end of line is significant, and returns a different token, not just whitespace
// creates array to hold the seperate tokens from the program
ArrayList programTokens = new ArrayList();
int printIndex = 0; // this is here for debugging
// while the next token isn't the end of file..
while(streamT.nextToken() != StreamTokenizer.TT_EOF)
{
System.out.println(streamT.ttype); // this is here for debugging
// if the type of variable is a word, then add the word
if(streamT.ttype == StreamTokenizer.TT_WORD)
{
programTokens.add(streamT.sval);
}
// if the type of variable is a number, then add the number
else if(streamT.ttype == StreamTokenizer.TT_NUMBER)
{
// ArrayList stores objects, so I convert my number to a string
String stringInput = Double.toString(streamT.nval);
programTokens.add(stringInput);
}
// if the type of variable is EOL, then print EOL.
else if(streamT.ttype == StreamTokenizer.TT_EOL)
{
programTokens.add("ENDOFLINE");
}
System.out.println(programTokens.get(printIndex)); // this is here for debugging
printIndex++; // this is here for debugging
}

I don't have an answer without writing a test program. Could you do that?
Write the shortest, simplest program that demonstrates the problem. Use a StringReader so the input is in the program. Put System.out.println() for all the cases. Run the program and copy the screen with the results to an editor. Comment the lines where the problem is and post it.

I believe that hyphens are only treated as numbers when they are prefixed to other numbers. So -69 would be -69 but 69- would be parsed as 2 entities since , for most things at least, you should prefix the sign to the number. What is it that you mean "fix it"? If you want the hyphens and such in your ArrayList as strings then you should just be able to do something like this:

Code:

// while the next token isn't the end of file..
while(streamT.nextToken() != StreamTokenizer.TT_EOF)
{
System.out.println(streamT.ttype); // this is here for debugging
// if the type of variable is a word, then add the word
if(streamT.ttype == StreamTokenizer.TT_WORD)
{
programTokens.add(streamT.sval);
}
// if the type of variable is a number, then add the number
else if(streamT.ttype == StreamTokenizer.TT_NUMBER)
{
// ArrayList stores objects, so I convert my number to a string
String stringInput = Double.toString(streamT.nval);
programTokens.add(stringInput);
}
// if the type of variable is EOL, then print EOL.
else if(streamT.ttype == StreamTokenizer.TT_EOL)
{
programTokens.add("ENDOFLINE");
}
else { // It was special
programTokens.add(""+streamT.ttype);
}
System.out.println(programTokens.get(printIndex)); // this is here for debugging
printIndex++; // this is here for debugging
}