Creating In-Memory File Systems with Google’s Jimfs

March 18th, 2015 by Micha Kops

Sometimes when writing an application we might consider using an in-memory file system to speed up data access or to create some kind of cache.

There are different libraries to help us here but one looks especially promising for me because it supports almost every functionality of the Java NIO File APIs added in Java 7 – from creating, reading, deleting files and directory to handling symbolic and hard links or watching directory changes with a WatchService.

In the following short tutorial, I’d like to demonstrate how to setup an in-memory file system within a few minutes and how to access directories and files stored in this file system.

Creating an in-memory file system with Jimfs

Dependencies
In-Memory File System and the NIO API
Tutorial Sources
Resources
Article Updates

Dependencies

Only one dependency needs to be added to our project: com.google.jimfs:jimfs:1.0

Using Gradle as build tool, this is my build.gradle:

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'application'
 
sourceCompatibility = 1.8
version = '1.0.0'
jar {
    manifest {
        attributes 'Implementation-Title': 'hasCode.com Jimfs Tutorial', 'Implementation-Version': version
    }
}
 
repositories {
    mavenCentral()
}
 
dependencies {
    compile 'com.google.jimfs:jimfs:1.0'
}
 
mainClassName = 'com.hascode.tutorial.Example'

In-Memory File System and the NIO API

In the following examples, we’re creating an in-memory file system and we’re accessing it using the NIO API added with Java 7.

Example 1: Creating and Reading Directories and Files

First we’re writing two new files to the file system, we’re copying one file from the classpath to the file system and finally we’re reading all files from the in-memory file system’s data directory and print their name and byte-length.

package com.hascode.tutorial;
 
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileSystem;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
 
import com.google.common.collect.ImmutableList;
import com.google.common.jimfs.Configuration;
import com.google.common.jimfs.Jimfs;
 
public class Example {
 
	public static void main(final String[] args) throws IOException {
		System.out.println("Example 1: Creating, copying, reading files and directories");
		FileSystem fs = Jimfs.newFileSystem(Configuration.unix());
		Path data = fs.getPath("/data");
		Files.createDirectory(data);
 
		Path hello = data.resolve("test.txt"); // /data/test.txt
		Files.write(hello, ImmutableList.of("hello world"), StandardCharsets.UTF_8);
 
		Path csv = data.resolve("data.csv"); // /data/data.csv
		Files.write(csv, ImmutableList.of("test1,test2\ntest3,test4"), StandardCharsets.UTF_8);
 
		InputStream istream = Example.class.getResourceAsStream("/book.xml");
		Path xml = data.resolve("book.xml"); // /data/book.xml
		Files.copy(istream, xml, StandardCopyOption.REPLACE_EXISTING);
 
		Files.list(data).forEach(file -> {
			try {
				System.out.println(String.format("%s (%db)", file, Files.readAllBytes(file).length));
			} catch (Exception e) {
				e.printStackTrace();
			}
		});
	}
}

When we’re running the code above using our IDE of choice or using Gradle we should get a similar output:

$ gradle run
[..]
Example 1: Creating, copying, reading files and directories
/data/book.xml (146b)
/data/data.csv (24b)
/data/test.txt (12b)
 
BUILD SUCCESSFUL
 
Total time: 3.898 secs

Example 2: Creating and Reading Symbolic Links

In the following example, we’re creating a symlink to the file test.txt:

package com.hascode.tutorial;
 
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileSystem;
import java.nio.file.Files;
import java.nio.file.Path;
 
import com.google.common.collect.ImmutableList;
import com.google.common.jimfs.Configuration;
import com.google.common.jimfs.Jimfs;
 
public class Example {
 
	public static void main(final String[] args) throws IOException {
		System.out.println("Example 2: Handling Symbolic Links");
		FileSystem fs = Jimfs.newFileSystem(Configuration.unix());
		Path data = fs.getPath("/data");
		Files.createDirectory(data);
 
		Path hello = data.resolve("test.txt"); // /data/test.txt
		Files.write(hello, ImmutableList.of("hello world"), StandardCharsets.UTF_8);
 
		Path linkToHello = data.resolve("test.txt.link");
		Files.createSymbolicLink(linkToHello, hello);
 
		Files.list(data).forEach(file -> {
			try {
				System.out.println(String.format("%s (%db)", file, Files.readAllBytes(file).length));
			} catch (Exception e) {
				e.printStackTrace();
			}
		});
	}
 
}

When we’re running the code above using our IDE of choice or using Gradle we should get a similar output:

$ gradle run
[..]
Example 2: Handling Symbolic Links
/data/test.txt (12b)
/data/test.txt.link (12b)
 
BUILD SUCCESSFUL
 
Total time: 3.594 secs

Example 3: Watching for Changes using a WatchService

In the third example, we’re going to configure a watch-service to monitor changes in our virtual in-memory file system.

package com.hascode.tutorial;
 
import static java.nio.file.StandardWatchEventKinds.ENTRY_CREATE;
import static java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY;
 
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileSystem;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
 
import com.google.common.collect.ImmutableList;
import com.google.common.jimfs.Configuration;
import com.google.common.jimfs.Jimfs;
 
public class Example {
 
	public static void main(final String[] args) throws IOException, InterruptedException {
		System.out.println("Example 3: Watching changes using a WatchService");
		FileSystem fs = Jimfs.newFileSystem(Configuration.unix());
		Path data = fs.getPath("/data");
		Files.createDirectory(data);
 
		WatchService watcher = data.getFileSystem().newWatchService();
		Thread watcherThread = new Thread(() -> {
			WatchKey key;
			try {
				key = watcher.take();
				while (key != null) {
					for (WatchEvent<?> event : key.pollEvents()) {
						System.out.printf("event of type: %s received for file: %s\n", event.kind(), event.context());
					}
					key.reset();
					key = watcher.take();
				}
			} catch (Exception e) {
				e.printStackTrace();
			}
		}, "CustomWatcher");
		watcherThread.start();
		data.register(watcher, ENTRY_CREATE, ENTRY_MODIFY);
 
		Path hello = data.resolve("test.txt"); // /data/test.txt
		Files.write(hello, ImmutableList.of("hello world"), StandardCharsets.UTF_8);
	}
}

Running the code above should produce a similar output:

$ gradle run
[..]
Example 3: Watching changes using a WatchService
event of type: ENTRY_CREATE received for file: test.txt

Tutorial Sources

Please feel free to download the tutorial sources from my Bitbucket repository, fork it there or clone it using Git:

git clone https://bitbucket.org/hascode/jimfs-tutorial.git

Resources

Article Updates

2015-03-20: Example using a WatchService added.

Tags: cache, filesystem, fs, google, inmemory, jimfs, nio, nio2, symlink, watchservice

This entry was posted on Wednesday, March 18th, 2015 at 10:42 am and is filed under Java. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

One Response to “Creating In-Memory File Systems with Google’s Jimfs”

Betafish Says:
April 26th, 2016 at 9:40 am
I have created a VFS using JIMFS.

FileSystem virtualFS = Jimfs.newFileSystem(Configuration.unix());
Path virtualPath = virtualFS.getPath(“resources/virtualFolder”);
Files.createDirectories(virtualPath);
Path refData = virtualPath.resolve(“refData.csv”);
System.out.println(refData);

Files.write(refData, ImmutableList.of(sData),StandardCharsets.UTF_8);
I am trying to read the file (refData.csv) in another method (Path is passed to the other method).

What I have tried until now are :

1: new FileDataModel(new FileInputStream(Files.lines(refData)));

2: new FileDataModel((File) Files.lines(refData));

3: new FileDataModel(new File(refData));
Unfortunately, none of these work as of now. I understand, I am mixing the default FS with the Virtual FS.

Error: Exception in thread “main” java.lang.UnsupportedOperationException
How to access the file created?.