java.util.StringTokenizer

The StringTokenizer class of java.util package, allows an application to break a string into tokens. Most programmers opt to use StringTokenizer in tokenizing a string than StreamTokenizer because the methods are much simpler than the latter. The StringTokenizer methods do not distinguish among identifiers, numbers, and quoted strings, nor do they recognize and skip comments.

The set of delimiters may be specified either at creation time or on a per-token basis. Note: we term a delimiter as the characters that separate tokens like for example we have a string “Ryan Salvador|18|M”. The character | is the delimiter while the tokens would be “Ryan Salvador”, “18”, and “M”. I hope this is clear enough, because we will be using these terms all throughout this tutorial.

An instance of StringTokenizer behaves in one of two ways, depending on whether it was created with the returnDelims flag having the value true or false:

  • If the flag is false, delimiter characters serve to separate tokens. A token is a maximal sequence of consecutive characters that are not delimiters.
  • If the flag is true, delimiter characters are themselves considered to be tokens. A token is thus either one delimiter character, or a maximal sequence of consecutive characters that are not delimiters.
  • A StringTokenizer object internally maintains a current position within the string to be tokenized. Some operations advance this current position past the characters processed.

A token is returned by taking a substring of the string that was used to create the StringTokenizer object.

StringTokenizer Class Syntax

public class StringTokenizer
extends Object
implements Enumeration

StringTokenizer Class Compatibility Version

StringTokenizer Class is available since Java 1.0

Basic Usage of StringTokenizer

The StringTokenizer class as part of the java.util package is one of the classes of the java api that is widely used because it provides mechanism in tokenizing a string. Well it could be argued that a Scanner can do a better job in parsing a String however don’t get confused of the similarity of these two classes, they are totally different monsters.

Make a note that StringTokenizer is a legacy code and still being retained for compatibility purposes however it’s uses are highly discouraged for new implementation. It is highly recommended to use the split() method of String class or the java.util.regex package.

Let’s discuss first how to instantiate a StringTokenizer object

StringTokenizer st = new StringTokenizer("this is a test String");

From the above code, we basically just create a new StringTokenizer object and make use of one of the class constructor to initialize the String to be tokenized. There are multiple constructors available for StringTokenizer, one of it accepts a delimiter apart from the input String. Since we didn’t specify the delimiter, automatically the default delimiters are ” tnrf”: the space character, the tab character, the newline character, the carriage-return character, and the form-feed character. Below are some examples on the implementation:

StringTokenizer st = new StringTokenizer("this is a test String");
while(st.hasmoreTokens()){
  System.out.println(st.nextToken());
}

The above example will give the following output:

this
is
a
test
string

The above example is pretty basic, first we have initialized a new StringTokenizer object which we already have discussed. And then we have a while loop with condition that check if there are still tokens available on the buffer and then we have printed the result of the nextToken() method. The nextToken() method actually return the current token and then point to the next token if there are still available on the buffer. It’s pretty easy right? If you have understand the code above, then you are good to go. Below are some examples on the use of the different methods available with StringTokenizer class.

Math Method Usage Examples

The following are the detailed list of StringTokenizer methods and descriptions. We have also provided links to examples of each method on the list.

Returns the absolute value of a long value.

Modifier and Type Method and Description