Java NIO Basics
What NIO Is
NIO ("New I/O") is a second set of I/O APIs that Java added alongside the original java.io streams. It arrived in two stages:
| Release | Package(s) | What it added |
|---|---|---|
| Java 1.4 (2002) — NIO | java.nio, java.nio.channels, java.nio.charset | Buffers, channels, charsets, memory-mapped files, file locking, non-blocking network I/O with selectors |
| Java 7 (2011) — NIO.2 | java.nio.file | Path, Files, directory walking, file attributes, symbolic links, WatchService |
The previous lessons already used NIO.2 (Path and Files). This lesson covers the original NIO building blocks underneath it: buffers and channels.
Why they exist: classic streams move data one byte or one array at a time, in one direction, blocking until the operation finishes. That model is simple, but it doesn't allow some things high-performance programs need:
- reading and writing at any position in a file without reopening it,
- reusing one block of memory for many operations, including memory outside the Java heap,
- mapping a file directly into memory,
- locking regions of a file against other processes,
- handling thousands of network connections with a few threads (non-blocking I/O).
Most application code never touches buffers or channels directly — Files.readString and BufferedReader are simpler. But frameworks and servers (Netty, Tomcat, Kafka, databases) are built on them, and you need them for random-access binary files, large file processing, and anything performance-critical.
Buffers
A buffer is a fixed-size block of memory for one primitive type — ByteBuffer, CharBuffer, IntBuffer, LongBuffer and so on. ByteBuffer is by far the most important, because channels read and write bytes.
A buffer tracks three positions:
| Property | Meaning |
|---|---|
| capacity | Total size, fixed when the buffer is created |
| position | Index of the next element to read or write |
| limit | The first index that must not be read or written |
remaining() is limit - position: how much can still be read or written. Every relative put or get advances the position. The key to using buffers is understanding how a few methods move these markers.
import java.nio.ByteBuffer;
public class Buf {
static void show(String step, ByteBuffer b) {
System.out.printf("%-22s position=%d limit=%d capacity=%d remaining=%d%n",
step, b.position(), b.limit(), b.capacity(), b.remaining());
}
public static void main(String[] args) {
ByteBuffer buf = ByteBuffer.allocate(8);
show("allocate(8)", buf);
buf.put((byte) 'J').put((byte) 'a').put((byte) 'v').put((byte) 'a');
show("put 4 bytes", buf);
buf.flip();
show("flip()", buf);
System.out.println("get: " + (char) buf.get() + (char) buf.get());
show("get 2 bytes", buf);
buf.compact();
show("compact()", buf);
buf.clear();
show("clear()", buf);
}
}
Output:
allocate(8) position=0 limit=8 capacity=8 remaining=8
put 4 bytes position=4 limit=8 capacity=8 remaining=4
flip() position=0 limit=4 capacity=8 remaining=4
get: Ja
get 2 bytes position=2 limit=4 capacity=8 remaining=2
compact() position=2 limit=8 capacity=8 remaining=6
clear() position=0 limit=8 capacity=8 remaining=8
Step by step:
allocate(8)creates an empty buffer in write mode: position 0, limit = capacity.putwrites 4 bytes; the position moves to 4.flip()switches to read mode: the limit is set to the current position (4 — the amount of data written), and the position goes back to 0. Nowgetreads exactly the data that was put in.getreads two bytes; the position moves to 2. Two bytes (va) remain unread.compact()moves the unread bytes to the start of the buffer and switches back to write mode, with the position after them (2). Use it when you've only partly consumed the data and want to add more after it.clear()resets to write mode with position 0 and limit = capacity. It doesn't erase the bytes; they're simply overwritten by the next writes.
The most common buffer bug is forgetting flip() after filling a buffer: the position stays at the end of the data, so a following read or channel write sees nothing (or garbage after the data). The cycle to remember: fill → flip → drain → clear (or compact) → fill again.
Other useful methods:
ByteBuffer.wrap(byte[])creates a buffer around an existing array, already in read mode (position 0, limit = array length).get(int index)andput(int index, byte b)are absolute operations that don't move the position.rewind()sets the position to 0 without changing the limit, to read the same data again.
Multi-byte values and byte order
ByteBuffer can read and write numbers, which makes it the standard tool for binary formats and protocols:
import java.nio.*;
public class Order {
public static void main(String[] args) {
ByteBuffer big = ByteBuffer.allocate(4).putInt(0x0A0B0C0D);
ByteBuffer little = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(0x0A0B0C0D);
System.out.printf("big-endian: %02X %02X %02X %02X%n", big.get(0), big.get(1), big.get(2), big.get(3));
System.out.printf("little-endian: %02X %02X %02X %02X%n", little.get(0), little.get(1), little.get(2), little.get(3));
System.out.println("native order: " + ByteOrder.nativeOrder());
ByteBuffer header = ByteBuffer.allocate(14).putShort((short) 7).putLong(1_794_635_670_123L).putInt(42);
header.flip();
System.out.println("read back: version=" + header.getShort() + " time=" + header.getLong() + " count=" + header.getInt());
}
}
Output (the native order is LITTLE_ENDIAN on x86 and most ARM machines):
big-endian: 0A 0B 0C 0D
little-endian: 0D 0C 0B 0A
native order: LITTLE_ENDIAN
read back: version=7 time=1794635670123 count=42
- A
ByteBufferis big-endian by default (most significant byte first), which matches network protocols and Java's ownDataOutputStream.order(ByteOrder.LITTLE_ENDIAN)switches to the order used by formats like BMP, ZIP and most CPU architectures. putShort,putInt,putLong,putDoubleand the matchinggetmethods convert numbers to and from 2, 4 and 8 bytes. The 14-byte header is 2 + 8 + 4 bytes, and reading it back in the same order with the same byte order returns the original values.
Heap and direct buffers
ByteBuffer.allocate(n) creates a buffer backed by a normal Java byte[] on the heap. ByteBuffer.allocateDirect(n) allocates memory outside the heap, which the operating system can read and write directly, avoiding an extra copy during I/O. Direct buffers are more expensive to create and are freed only when the garbage collector reclaims the buffer object, so they're best allocated once and reused — the pattern network servers use. For ordinary code, heap buffers are fine.
Channels
A channel is a connection to an I/O source — a file, a socket — that reads into and writes from buffers. Unlike streams, a single channel can often both read and write, and a FileChannel can work at any position in the file.
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.*;
import static java.nio.file.StandardOpenOption.*;
public class Chan {
public static void main(String[] args) throws IOException {
Path file = Path.of("channel.txt");
try (FileChannel ch = FileChannel.open(file, CREATE, TRUNCATE_EXISTING, WRITE)) {
ByteBuffer buf = ByteBuffer.wrap("Hello from FileChannel\n".getBytes(StandardCharsets.UTF_8));
int written = ch.write(buf);
System.out.println("written " + written + " bytes, channel size " + ch.size());
}
try (FileChannel ch = FileChannel.open(file, READ)) {
ByteBuffer buf = ByteBuffer.allocate(5);
StringBuilder text = new StringBuilder();
int reads = 0;
while (ch.read(buf) != -1) {
reads++;
buf.flip();
text.append(StandardCharsets.UTF_8.decode(buf));
buf.clear();
}
System.out.print("read in " + reads + " calls: " + text);
}
}
}
Output:
written 23 bytes, channel size 23
read in 5 calls: Hello from FileChannel
FileChannel.open(path, options...)opens a channel with the same open options asFilesmethods.FileInputStream,FileOutputStreamandRandomAccessFilealso offergetChannel().write(buffer)writes the buffer's remaining bytes and advances the buffer's position. AFileChannelwrites all of them in practice, but the generalWritableByteChannelcontract allows partial writes, so robust code loopswhile (buf.hasRemaining())— especially for network channels.read(buffer)fills the buffer from the channel and returns the number of bytes read, or-1at the end. With a deliberately tiny 5-byte buffer, the 23 bytes take 5 reads.- The read loop shows the buffer cycle in action: read (fill) →
flip()→ decode (drain) →clear(). - Decoding 5-byte chunks separately works here only because the text is ASCII. A multi-byte UTF-8 character could be split across two chunks; correct code then uses a
CharsetDecoderwithcompact()to carry the incomplete bytes over to the next read — or simply uses aReader, which handles this for you.
Random access: reading and writing at any position
FileChannel has read(buffer, position) and write(buffer, position), which work at an absolute file position. This makes fixed-size record files straightforward: record n starts at byte n × recordSize, so any record can be read or updated without touching the others.
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.*;
import static java.nio.file.StandardOpenOption.*;
public class Records {
static final int NAME_BYTES = 20;
static final int RECORD_SIZE = 4 + NAME_BYTES + 4; // id + name + stock = 28 bytes
static void write(FileChannel ch, int index, int id, String name, int stock) throws IOException {
ByteBuffer buf = ByteBuffer.allocate(RECORD_SIZE);
buf.putInt(id);
byte[] nameBytes = name.getBytes(StandardCharsets.UTF_8);
if (nameBytes.length > NAME_BYTES) throw new IllegalArgumentException("name too long: " + name);
buf.put(nameBytes).put(new byte[NAME_BYTES - nameBytes.length]);
buf.putInt(stock);
buf.flip();
ch.write(buf, (long) index * RECORD_SIZE);
}
static String read(FileChannel ch, int index) throws IOException {
ByteBuffer buf = ByteBuffer.allocate(RECORD_SIZE);
ch.read(buf, (long) index * RECORD_SIZE);
buf.flip();
int id = buf.getInt();
byte[] nameBytes = new byte[NAME_BYTES];
buf.get(nameBytes);
String name = new String(nameBytes, StandardCharsets.UTF_8).replace("\0", "");
int stock = buf.getInt();
return id + " " + name + " stock=" + stock;
}
public static void main(String[] args) throws IOException {
Path file = Path.of("inventory.dat");
try (FileChannel ch = FileChannel.open(file, CREATE, TRUNCATE_EXISTING, READ, WRITE)) {
write(ch, 0, 101, "Gel pen", 40);
write(ch, 1, 102, "Notebook", 25);
write(ch, 2, 103, "Laptop bag", 3);
System.out.println("file size: " + ch.size() + " bytes");
write(ch, 1, 102, "Notebook", 24); // update record 1 in place
for (int i = 0; i < ch.size() / RECORD_SIZE; i++) {
System.out.println("record " + i + ": " + read(ch, i));
}
}
}
}
Output:
file size: 84 bytes
record 0: 101 Gel pen stock=40
record 1: 102 Notebook stock=24
record 2: 103 Laptop bag stock=3
How it works:
- Each record is exactly 28 bytes: a 4-byte
intid, a 20-byte name field and a 4-byteintstock. Three records make an 84-byte file. - The name is padded with zero bytes to fill its 20-byte field; names longer than 20 bytes (in UTF-8) are rejected, because they'd overwrite the next field. When reading, the padding (
\0characters) is removed. - The position is calculated as
index * RECORD_SIZE, cast tolongfirst, because file positions arelongand anintmultiplication could overflow for large files. - The update rewrites only record 1's 28 bytes — the stock changes from 25 to 24 while the rest of the file is untouched. With text formats like CSV, changing one value means rewriting the whole file.
- The channel is opened with
READandWRITE, so one channel serves both purposes.
Databases and search engines store data in fixed-size pages in exactly this way, at a much larger scale. java.io.RandomAccessFile offers the same seek-and-read model with seek(position), readInt() and writeInt() methods, and is common in older code.
Fast copying with transferTo
try (FileChannel in = FileChannel.open(source, READ);
FileChannel out = FileChannel.open(target, CREATE, TRUNCATE_EXISTING, WRITE)) {
long position = 0, size = in.size();
while (position < size) {
position += in.transferTo(position, size - position, out);
}
}
transferTo asks the operating system to move bytes directly from one channel to another. On many systems this avoids copying the data into Java memory at all (zero-copy), which is how web servers send static files efficiently. It may transfer fewer bytes than requested in one call, hence the loop. Files.copy uses similar optimisations internally.
Memory-Mapped Files
A channel can map a region of a file into memory. The resulting MappedByteBuffer reads and writes the file as if it were an array: the operating system loads pages of the file on demand and writes changes back.
import java.io.IOException;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.*;
public class Mapped {
public static void main(String[] args) throws IOException {
Path log = Files.writeString(Path.of("access.log"), "GET /a 200\nGET /b 404\nPOST /c 200\nGET /d 500\n".repeat(25_000));
try (FileChannel ch = FileChannel.open(log)) {
MappedByteBuffer map = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size());
int lines = 0, errors = 0;
for (int i = 0; i < map.limit(); i++) {
if (map.get(i) != '\n') continue;
lines++;
byte statusClass = map.get(i - 3); // each line ends with a 3-digit status code
if (statusClass == '4' || statusClass == '5') errors++;
}
System.out.println(ch.size() + " bytes mapped, " + lines + " lines, " + errors + " error responses");
}
}
}
Output:
1125000 bytes mapped, 100000 lines, 50000 error responses
ch.map(READ_ONLY, 0, size)maps the whole file.FileChannel.open(path)with no options opens it for reading.- The loop scans the bytes with absolute
get(i)calls. At each newline, the byte three positions earlier is the first digit of the status code;4xxand5xxresponses are counted as errors (2 of every 4 lines). - No read calls or buffer management are needed, and the operating system handles caching, which makes mapping very fast for scanning large files or for random access to them.
Things to know:
- A single mapping is limited to
Integer.MAX_VALUEbytes (about 2 GB); larger files are mapped in several regions. (Java 22's Foreign Function & Memory API removes this limit.) - The mapping stays valid even after the channel is closed, until the buffer is garbage-collected. On Windows, a mapped file can't be deleted while mapped.
READ_WRITEmode writes changes back to the file;force()ensures they reach the disk.
File Locking
A FileLock coordinates access to a file between processes — for example, to make sure only one instance of a scheduled job runs at a time on a server.
import java.io.IOException;
import java.nio.channels.*;
import java.nio.file.*;
import static java.nio.file.StandardOpenOption.*;
public class Lock {
public static void main(String[] args) throws IOException {
Path lockFile = Path.of("nightly-report.lock");
try (FileChannel ch = FileChannel.open(lockFile, CREATE, WRITE);
FileLock lock = ch.tryLock()) {
if (lock == null) {
System.out.println("another process is generating the report; exiting");
return;
}
System.out.println("lock acquired, shared=" + lock.isShared() + ", valid=" + lock.isValid());
try (FileChannel second = FileChannel.open(lockFile, WRITE)) {
second.tryLock();
} catch (OverlappingFileLockException e) {
System.out.println("same JVM, second tryLock: OverlappingFileLockException");
}
}
System.out.println("lock released");
}
}
Output:
lock acquired, shared=false, valid=true
same JVM, second tryLock: OverlappingFileLockException
lock released
tryLock()acquires an exclusive lock on the whole file and returns immediately: aFileLockon success, ornullif another process holds a lock.lock()waits until the lock becomes available instead.FileLockisAutoCloseable; closing it (or the channel, or the process exiting) releases the lock. try-with-resources handlesnullresources, so thenullcase needs no special treatment.- Locks are held on behalf of the whole JVM, so they don't coordinate threads within one program: a second attempt in the same JVM throws
OverlappingFileLockException. Usejava.util.concurrentlocks between threads. - On most Unix systems file locks are advisory: they only affect programs that also use locking. A program that just opens and writes the file isn't stopped.
lock(position, size, shared)locks a region, andshared = truerequests a shared (read) lock that other processes can also hold.
Charsets, Encoders and Decoders
java.nio.charset is where character encodings live — the Charset objects you pass to readers, writers and Files methods come from here. You can also convert between characters and bytes directly:
import java.nio.*;
import java.nio.charset.*;
public class Cs {
public static void main(String[] args) throws CharacterCodingException {
Charset utf8 = StandardCharsets.UTF_8;
ByteBuffer bytes = utf8.encode("Chennai ₹500");
System.out.println("encoded to " + bytes.remaining() + " bytes");
CharBuffer chars = utf8.decode(bytes);
System.out.println("decoded: " + chars);
CharsetDecoder strict = utf8.newDecoder()
.onMalformedInput(CodingErrorAction.REPORT)
.onUnmappableCharacter(CodingErrorAction.REPORT);
try {
strict.decode(ByteBuffer.wrap(new byte[] {'o', 'k', (byte) 0xE9}));
} catch (MalformedInputException e) {
System.out.println("strict decoder: MalformedInputException, length " + e.getInputLength());
}
System.out.println("lenient decode: " + utf8.decode(ByteBuffer.wrap(new byte[] {'o', 'k', (byte) 0xE9})));
}
}
Output:
encoded to 14 bytes
decoded: Chennai ₹500
strict decoder: MalformedInputException, length 1
lenient decode: ok�
charset.encode(String)returns aByteBuffer(11 ASCII characters plus 3 bytes for₹= 14 bytes);charset.decode(ByteBuffer)returns aCharBuffer.- These convenience methods replace invalid input with the replacement character
�silently. ACharsetDecoderconfigured withCodingErrorAction.REPORTthrows instead, which is whatFiles.readStringdoes internally. Use the strict form whenever corrupted text must be detected rather than passed on.
Where NIO Fits
| Need | Use |
|---|---|
| Read or write a whole small file | Files.readString, Files.writeString, Files.readAllBytes |
| Process a large text file line by line | Files.newBufferedReader, Files.lines |
| Copy or move files | Files.copy, Files.move |
| Read/write at arbitrary positions, fixed-size records | FileChannel with positional read/write |
| Scan very large files fast, or random access over them | Memory-mapped MappedByteBuffer |
| Ensure only one process runs a job | FileChannel.tryLock() |
| Binary protocols and formats | ByteBuffer with getInt/putInt and an explicit byte order |
| Thousands of network connections | Non-blocking channels with a Selector (Networking section), or a framework such as Netty |
The rule of thumb: start with Files and streams; reach for buffers and channels when you need positional access, mapping, locking or maximum throughput.