String Tokenization Reversal

Description

Reversing the words of a sentence, checking that "dog cat cat dog" follows the pattern "abba", and normalizing a search query all become easy once the text is a list of words. The hard part is getting a clean list of words, and most bugs come from what split actually returns. This skill covers splitting reliably, reassembling with a builder, and using words as keys with a two-way map.

We'll go from simple to more technical:

1. The pattern: trim, split on runs of whitespace, then reverse and join.
2. A second flavor: tokens as keys, a one-to-one match with two maps.
3. Where tokenizing shows up in production (search queries, logs, CSV, commands).
4. What breaks it: regex separators, empty tokens, vanishing fields, one-way maps.

The approach

String reverseWords(String s) {
    String[] words = s.trim().split("\\s+");    // trim the ends, split on runs of whitespace
    StringBuilder sb = new StringBuilder();
    for (int i = words.length - 1; i >= 0; i--) {
        sb.append(words[i]);
        if (i > 0) sb.append(' ');
    }
    return sb.toString();
}

def reverse_words(s: str) -> str:
    return " ".join(reversed(s.split()))             # split() with no argument: runs of whitespace, no empty strings

Keep "what counts as a word" (the split) separate from "what to do with the words" (reverse and join). The details of split decide the first part:

• It takes a regex: "1.2.3".split(".") returns an empty array, because . matches every character - write "\\.". The same goes for |, + and *.
• A leading separator leaves a leading empty string: " hello world ".split("\\s+") is ["", "hello", "world"].
• Trailing empty strings are dropped: "a,b,,".split(",") has 2 elements; split(",", -1) keeps all 4.
• split(" ") turns every extra space into an empty token: "a b".split(" ") is ["a", "", "b"].

That's why the pattern trims first and splits on \\s+.

Python's str.split behaves differently. With no argument, s.split() does all of that at once: it splits on runs of whitespace and drops the empty strings at both ends, so reversing words is " ".join(reversed(s.split())). With an argument, s.split(sep) is a different tool: the separator is a literal string, not a regex ("1.2.3".split(".") works), and every empty string is kept, leading and trailing ("a,b,,".split(",") has 4 elements, and "a b".split(" ") is ["a", "", "b"]). re.split is the regex version.

A second flavor: tokens as keys

Once text is tokens, words can be compared and mapped like any other values. "Does s follow pattern?" asks for a one-to-one match between pattern letters and words, and that needs a map in each direction:

boolean wordPattern(String pattern, String s) {
    String[] words = s.split(" ");
    if (words.length != pattern.length()) return false;
    Map<Character, String> toWord = new HashMap<>();
    Map<String, Character> toLetter = new HashMap<>();
    for (int i = 0; i < words.length; i++) {
        char c = pattern.charAt(i);
        if (toWord.containsKey(c) && !toWord.get(c).equals(words[i])) return false;
        if (toLetter.containsKey(words[i]) && toLetter.get(words[i]) != c) return false;
        toWord.put(c, words[i]);
        toLetter.put(words[i], c);
    }
    return true;
}

def word_pattern(pattern: str, s: str) -> bool:
    words = s.split(" ")
    if len(words) != len(pattern):
        return False
    to_word, to_letter = {}, {}
    for c, w in zip(pattern, words):
        if c in to_word and to_word[c] != w:
            return False
        if w in to_letter and to_letter[w] != c:
            return False
        to_word[c] = w
        to_letter[w] = c
    return True

With only the letter-to-word map, "abba" against "dog dog dog dog" passes: a and b both map to "dog" without conflict. The reverse map catches two letters sharing a word. The same two-map check decides whether two strings are isomorphic ("moon"/"deer" yes, "door"/"deed" no).

Where you'll see it

Search boxes trim and collapse whitespace before matching; log processors split lines into fields; command-line tools split arguments. split(",") is not a CSV parser: a quoted field like "Smith, John" contains the separator, and trailing empty fields vanish unless you pass -1 - use a CSV library for real data (Python's csv module).

Which tool for which job?

Common mistakes

Check yourself

Try to answer before opening each one - recalling beats re-reading.

1. What does " hello world ".split("\\s+") return, and why does the pattern call trim() first?

2. Why does the word-pattern check need a map in each direction?

3. Why does "1.2.3".split(".") return an empty array?

Key takeaways

• Separate tokenizing from processing: split, work on the list, join.
• split takes a regex, keeps a leading empty string and drops trailing ones (limit -1 keeps them).
• Use trim() + split("\\s+") for clean words.
• One-to-one mappings need a map in each direction, plus a length check.
• In-place word reversal: reverse everything, then each word.