Use String#split(...) if you can instead of a StringTokenizer. That way you'll be able to create an array of the right size no matter what text is entered

Be sure your comments agree with your code!

Never check for null that way lest you want to have lots of NPE's thrown. You would do it as if (token[i] == null) {

Please be compulsive about using proper code indentation and consistent formatting, including curly brace formatting. Coding is an exercise in precision and there's no place in it for sloppiness. This will also cut down on the number of code errors you produce.

Re: Tokenizer

why not just read the entire text file into one string, call its split(String regex) method, and then iterate over the returned array in a reversed for-loop (start at a number then decrement to 0)? if you want to seperate each token by whitespace, then just call:
String[] tokens = text.split(" ");

Last edited by kennyman94; 10-21-2011 at 04:16 AM.
Reason: forgot to mention something

Re: Tokenizer

I would think it would be pretty hard to run out of memory with a String. the JFugue api uses Strings to represent notes, chords, durations, etc. for midi and it can use Miles of characters in a single String (from what i hear anyways)