00:00

Help Us Improve

We're sorry this tutorial wasn't helpful. Please tell us what we can do better.

StringTokenizer in Java – Splitting Strings into Tokens

java.util.StringTokenizer breaks a string into tokens separated by delimiter characters. It has been part of Java since version 1.0 and is still found in older code and competitive programming. The Javadoc describes it as a legacy class kept for compatibility and recommends String.split() or the java.util.regex package for new code.

This tutorial covers its constructors and methods, custom delimiters, returning delimiters as tokens, how it differs from split(), and modern alternatives.

1. Constructors and Methods

StringTokenizer splitting a string into tokens
Constructor / methodDescription
StringTokenizer(String str)Default delimiters: space, \t, \n, \r, \f
StringTokenizer(String str, String delim)Each character of delim is a delimiter
StringTokenizer(String str, String delim, boolean returnDelims)If true, delimiters are returned as tokens
boolean hasMoreTokens()Whether more tokens remain
String nextToken()The next token; NoSuchElementException if none
String nextToken(String delim)Switch delimiters, then return the next token
int countTokens()Tokens remaining
hasMoreElements() / nextElement()The same, via the Enumeration interface

2. Using StringTokenizer

TokenizerBasics.java 
import java.util.StringTokenizer;

public class TokenizerBasics {
    public static void main(String[] args) {
        // 1. Default delimiters: space, tab, newline, carriage return, form feed
        StringTokenizer st = new StringTokenizer("Java   is\tsimple and\npowerful");
        System.out.println("countTokens() = " + st.countTokens());
        while (st.hasMoreTokens()) System.out.println("  " + st.nextToken());

        // 2. Custom delimiters: every character in the string is a delimiter
        StringTokenizer csv = new StringTokenizer("Asha,29;Pune|Developer", ",;|");
        StringBuilder sb = new StringBuilder();
        while (csv.hasMoreTokens()) sb.append('[').append(csv.nextToken()).append(']');
        System.out.println(sb);

        // 3. returnDelims = true: delimiters are returned as tokens too
        StringTokenizer expr = new StringTokenizer("12+7*3-4", "+-*", true);
        StringBuilder parts = new StringBuilder();
        while (expr.hasMoreTokens()) parts.append(expr.nextToken()).append(' ');
        System.out.println("expression tokens: " + parts.toString().trim());
    }
}
Output
countTokens() = 5
  Java
  is
  simple
  and
  powerful
[Asha][29][Pune][Developer]
expression tokens: 12 + 7 * 3 - 4
  • Consecutive delimiters are treated as one: the three spaces after "Java" produced no empty tokens.
  • The delimiter string is a set of single characters, not a pattern: ",;|" splits on any of them.
  • With returnDelims = true, operators in an expression are kept, which is handy for simple parsers.

3. StringTokenizer vs split()

TokenizerVsSplit.java 
import java.util.Arrays;
import java.util.StringTokenizer;

public class TokenizerVsSplit {
    public static void main(String[] args) {
        String line = "Asha,,Pune,";

        // StringTokenizer skips empty tokens
        StringTokenizer st = new StringTokenizer(line, ",");
        StringBuilder tokens = new StringBuilder();
        while (st.hasMoreTokens()) tokens.append('[').append(st.nextToken()).append(']');
        System.out.println("StringTokenizer : " + tokens);

        // split keeps empty tokens in the middle (and drops trailing ones by default)
        System.out.println("split(\",\")      : " + Arrays.toString(line.split(",")));
        System.out.println("split(\",\", -1)  : " + Arrays.toString(line.split(",", -1)));

        // split uses regular expressions; StringTokenizer uses single characters
        System.out.println("split(\"\\\\s*,\\\\s*\"): " + Arrays.toString("a , b,c ,d".split("\\s*,\\s*")));
    }
}
Output
StringTokenizer : [Asha][Pune]
split(",")      : [Asha, , Pune]
split(",", -1)  : [Asha, , Pune, ]
split("\\s*,\\s*"): [a, b, c, d]
AspectStringTokenizerString.split()
DelimiterSet of charactersRegular expression
Empty tokensSkippedKept (trailing ones dropped unless limit is negative)
ResultIterated one by oneString[] array
Return delimitersOptionalNo (use regex lookarounds)
StatusLegacyRecommended

Skipping empty tokens is a real difference for data such as CSV, where an empty field matters: StringTokenizer silently loses it.

4. Modern Alternatives

ModernAlternatives.java 
import java.util.List;
import java.util.Scanner;
import java.util.regex.Pattern;

public class ModernAlternatives {
    public static void main(String[] args) {
        String data = "java spring  kafka docker";

        // Pattern + stream
        List<String> words = Pattern.compile("\\s+").splitAsStream(data).toList();
        System.out.println("splitAsStream : " + words);

        // Scanner with typed tokens
        try (Scanner sc = new Scanner("10 20 x 30")) {
            int sum = 0;
            while (sc.hasNext()) {
                if (sc.hasNextInt()) sum += sc.nextInt(); else sc.next();
            }
            System.out.println("Scanner sum   : " + sum);
        }
    }
}
Output
splitAsStream : [java, spring, kafka, docker]
Scanner sum   : 60
  • String.split(regex) for simple splitting into an array.
  • Pattern.splitAsStream or String.lines() when a stream is convenient.
  • Scanner for reading typed tokens (nextInt, nextDouble) from strings, files or input.
  • A proper CSV library (such as OpenCSV or Apache Commons CSV) for real CSV with quotes and escapes.

5. Summary

  • StringTokenizer splits a string into tokens using a set of delimiter characters.
  • Iterate with hasMoreTokens() and nextToken(); countTokens() gives the remaining count.
  • It skips empty tokens and can optionally return delimiters.
  • It is a legacy class: prefer split(), regex or Scanner in new code.

Related Tutorials