Java 21: Virtual Threads and Structured Concurrency

Description

Java 21 (September 2023) is the long-term-support release after 17, and its headline feature is virtual threads: the biggest change to Java concurrency since java.util.concurrent arrived in Java 5.

Here's the problem they solve. A typical service handles each request on a thread from a pool of, say, 200 threads. Each request spends most of its time waiting: for the database, for two HTTP calls to other services. At 200 concurrent requests the pool is used up, and new requests queue, while the CPU sits at 10%. The threads aren't working; they're waiting. Until now the fixes were a bigger pool (but each classic thread is an operating-system thread with its own memory, so thousands get expensive) or rewriting the code as callbacks and CompletableFuture chains (fast, but hard to read and debug).

Virtual threads let simple, blocking, one-thread-per-request code scale to many thousands of concurrent requests. This skill covers how to use them, the rules that keep them working well, and two related features that were previews in Java 21.

Here's the path, from easiest to most technical:

1. What a virtual thread is, and how to start one.
2. The rules: what they're good for, and why you never pool them.
3. Pinning: the one trap on Java 21.
4. Structured concurrency and scoped values (previews).

What a virtual thread is

A virtual thread is a real java.lang.Thread, but a very cheap one. The JVM runs it on one of a small number of carrier threads (one per CPU core, by default). Whenever the virtual thread waits for something (sleeping, reading from the network, waiting for a lock), the JVM sets it aside and the carrier runs another virtual thread. When the wait is over, it's picked up again.

Starting one looks like starting any thread:

try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    for (Request r : requests) {
        executor.submit(() -> handle(r));     // a brand-new virtual thread for each task
    }
}                                             // the end of the block waits for all of them

Thread t = Thread.ofVirtual().name("import-", 0).start(task);

The effect is dramatic for waiting-heavy work: in the course's sandbox, 10,000 tasks that each slept for one second all finished in under three seconds, on a single carrier thread.

The rules

1. They help with waiting, not with computing. Virtual threads let many tasks wait at the same time. They don't make CPU-heavy work faster: there are still only as many carriers as cores.

2. Never put them in a pool. Pools exist to ration expensive threads. Virtual threads are cheap, so create one per task and let it go. If you need to protect something, like a database that can handle only 50 queries at once, limit that explicitly with a Semaphore(50) or the connection pool, not by limiting threads.

3. The JVM doesn't wait for them at exit. Virtual threads are always daemon threads. Wait for them yourself, for example with the executor's try-with-resources block.

4. Be careful with ThreadLocal. It still works, but every virtual thread gets its own copy, so a "one cached object per thread" trick that made 200 copies on a pool now makes one per task, potentially millions.

Pinning: the one trap on Java 21

On Java 21, a virtual thread that waits while inside a synchronized block can't be set aside. It pins its carrier: the carrier sits there, occupied but idle, until the wait ends. With only a few carriers, a few pinned threads can stall everything.

In the sandbox, with one carrier, four virtual threads that each slept half a second inside synchronized took about 2 seconds, one after another. The same tasks using a ReentrantLock instead finished in about half a second, because waiting while holding a ReentrantLock doesn't pin.

So on Java 21:

• Don't wait (sleep, do I/O, call other services) inside synchronized. Use ReentrantLock around waiting code. Short synchronized sections that don't wait are fine.
• -Djdk.tracePinnedThreads=full prints a stack trace whenever a thread pins, which helps find the culprits.
• Java 24 removed this limitation (JEP 491), one more reason teams move to newer releases.

Structured concurrency and scoped values (previews)

Two related features shipped in Java 21 as previews (they compile only with --enable-preview).

Structured concurrency treats a group of subtasks like a block of code: they start inside a scope and must all finish before it ends.

try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
    Subtask<Profile> profile = scope.fork(() -> fetchProfile(id));
    Subtask<List<Order>> orders = scope.fork(() -> fetchOrders(id));
    scope.join().throwIfFailed();       // wait for both; rethrow the first failure
    return new Page(profile.get(), orders.get());
}

If one subtask fails, the others are cancelled straight away. In the sandbox, when one subtask failed immediately, the scope gave up in well under a second instead of waiting two seconds for the other. No thread is left running after the block, and no failure is forgotten.

Scoped values pass context, such as the current user, to everything a piece of code calls, including subtasks:

static final ScopedValue<String> USER = ScopedValue.newInstance();

ScopedValue.where(USER, "alice").run(() -> handle());   // USER.get() is "alice" inside, unset afterwards

Unlike a ThreadLocal, a scoped value can't be changed, is only set for the duration of run, and needs no cleanup. Scoped values became final in Java 25; structured concurrency was still being refined there.

Common mistakes

Check yourself

Try to answer before opening each one. Recalling beats re-reading.

1. A service's 200 threads are all busy, the CPU is at 10%, and each request mostly waits on the database and HTTP calls. Why would virtual threads help?

2. Why shouldn't you put virtual threads in a fixed-size pool?

3. What is pinning, and how do you avoid it on Java 21?

4. What does StructuredTaskScope.ShutdownOnFailure give you over submitting two tasks to an executor?

Key takeaways

• Virtual threads make blocking, one-thread-per-request code scale for work that mostly waits.
• Create one per task, never pool them, and limit concurrency with semaphores or connection pools.
• They don't speed up CPU-heavy work, and they're daemon threads.
• On Java 21, don't wait inside synchronized: use ReentrantLock.
• Structured concurrency and scoped values (previews in 21) fix leaked subtasks and ThreadLocal cleanup.