I’ve recently been looking into the options for storing a relatively large number of user-uploaded files in a non-hierarchical storage “bucket” with a sufficient degree of abstraction to allow the files to reside either locally or on a server (e.g. WebDAV, SFTP) without any (or many) changes to the code. A virtual file system (VFS) is pretty well suited to this kind of task and, as it happens, there are a number of VFS implementations available for Java including Apache Commons VFS and TrueVFS.

For now, I’ve opted to use Apache Commons VFS with a WebDAV file provider. Overall it’s relatively straightforward to get the various moving parts up and running, but there were a few pitfalls in setting up the WebDAV server, ensuring that Maven is pulling in the right dependencies and copying file content to VFS.

Rationale for using Sardine

From the Commons VFS homepage: “Apache Commons VFS provides a single API for accessing various different file systems. It presents a uniform view of the files from various different sources, such as the files on local disk, on an HTTP server, or inside a Zip archive.” WebDAV is one of the sources supported by Commons VFS, but the default implementation is the same as that used by Apache JackRabbit. As of writing, the JackRabbit implementation relies on the deprecated Commons HttpClient for its HTTP support, which has long been superseded by Apache HttpComponents. Including a deprecated and unsupported HTTP client library isn’t very appealing from a security standpoint, neither is it desirable in terms of longer-term project maintenance.

Thankfully, Sardine comes to the rescue here. In brief, Sardine is a relatively modern, lightweight WebDAV client focussed on the most common use cases for WebDAV. In combination with Nicolas Delsaux’s excellent commons-vfs-webdav-sardine project, it’s relatively straightforward to use Sardine as a Commons VFS WebDAV provider.

Configuring a WebDAV server compatible with Sardine

One of most attractive aspects of WebDAV is that it’s extremely easy to configure, especially on machines that are already running Apache. However, even given the relative ease of configuration, there are usually a couple of file permission and directory index issues that needed to be resolved.

A basic WebDAV server can be set up by including (or preferably Including) the following in httpd.conf:

LoadModule mod_dav_fs

DavLockDB "/usr/webdav/DavLock"

Alias /uploads "/usr/uploads"

<Directory "/usr/uploads">
    Dav On

    AllowOverride None
    Options Indexes FollowSymLinks
    Order Allow,Deny
    Allow from all

    AuthType Basic
    AuthName WebDAV-Realm

    AuthUserFile "/usr/webdav.passwd"

    <LimitExcept GET OPTIONS>
        require user admin
    </LimitExcept>
</Directory>

The inclusion of the “Options Indexes FollowSymLinks” directive is important as it prevents 403 errors being returned in response to HEAD requests to test for the existence of files on the WebDAV share.

The directory containing the DavLockDB file and the WebDAV directory both need to be readable and writable by the current Apache user and group. The following commands should ensure that this is the case (given the above configuration in httpd.conf):

sudo mkdir /usr/webdav
sudo mkdir /usr/uploads
sudo chown www:www /usr/webdav /usr/uploads

With AuthType set to Basic, the only remaining task is to create the “admin” user referenced in the LimitExcept directive and restart Apache:

sudo htpasswd -c /usr/webdav.passwd admin
sudo apachectl graceful

On OS X, the server configuration can then be tested using “Connect to Server…” in the Finder (Cmd-K):

Authenticating with “admin” and the password set in the htpasswd stage above should mount a readable and writable WebDAV share in the Finder.

Using Sardine with Apache VFS

With a WebDAV server running locally, VFS and Sardine can be used to take a “local” file and upload it to the WebDAV server as follows:

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.FileSystemException;
import java.util.UUID;

import org.apache.commons.io.IOUtils;
import org.apache.commons.vfs2.FileObject;
import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
import org.apache.commons.vfs2.provider.local.DefaultLocalFileProvider;

// This import causes the Sardine WebdavFileProvider to be used rather than the default VFS provider
import fr.perigee.commonsvfs.webdav.WebdavFileProvider;

public String saveFileToDefaultFileSystem(File file, String baseURL) throws FileSystemException, IOException {
	
	String fileName;
	DefaultFileSystemManager fsManager;

	try {
		fsManager = new DefaultFileSystemManager();
		fsManager.addProvider("webdav", new WebdavFileProvider());
		fsManager.addProvider("file", new DefaultLocalFileProvider());
		fsManager.init();
	} catch (org.apache.commons.vfs2.FileSystemException e) {
		throw new FileSystemException("Exception initializing DefaultFileSystemManager: " + e.getMessage());
	}
	
	UUID uuid = UUID.randomUUID();
	fileName = uuid.toString();
	
	FileObject uploadedFile;
	FileObject destinationFile;
	
	try {
		uploadedFile = fsManager.toFileObject(file);
		destinationFile = fsManager.resolveFile(baseURL + fileName);
		destinationFile.createFile();
	} catch (org.apache.commons.vfs2.FileSystemException e) {
		fsManager.close();
		throw new FileSystemException("Exception resolving file in file store: " + e.getMessage());
	}
	
	try (InputStream in = uploadedFile.getContent().getInputStream();
		 OutputStream out = destinationFile.getContent().getOutputStream()) {
		
		IOUtils.copy(in, out);
		} catch (IOException e) {
		throw new IOException("Exception copying data: " + e.getMessage());
	} finally {
		fsManager.close();
	}
	
	return fileName;
		
}

The above saveFileToDefaultFileSystem() method:

• Takes a Java File and a baseURL as a String
• Sets up the VFS DefaultFileSystemManager adding providers for WebDAV and the local filesystem
• Generates a random filename (UUID) for the bucket store
• Resolves and creates the (heretofore non-existent) file in the WebDAV file system
• Converts the passed-in File to a VFS FileObject
• Uses a try-with-resources block to open an InputStream and an OutputStream from the source and destination FileObjects, respectively
• Uses the Apache Commons IOUtils to copy the InputStream to the OutputStream and closes everything down
• Returns the UUID as a String

The only other aspect not covered in the code is the form of the URL passed into the method, which must contain the full URL for the WebDAV server, including credentials as follows:

webdav://username:password@127.0.0.1:80/uploads/

VFS does support use of a UserAuthenticator rather than including credentials in the URL, but that’s probably overkill for getting something up and running quickly. There’s an example of how to use UserAuthenticator on the Commons VFS website, but as a middle ground, the password can be encrypted:

java -cp commons-vfs-2.0.jar org.apache.commons.vfs2.util.EncryptUtil encrypt mypassword

and the output enclosed in curly braces in the URL as follows:

webdav://username:{D7B82198B272F5C93790FEB38A73C7B8}@127.0.0.1:80/uploads/

Maven

Sardine, Apache Commons VFS2 and the Commons VFS Sardine WebDAV file provider will all be added as dependencies when the commons-vfs-webdav-sardine dependency is added to Maven.

Sardine has a dependency on the Apache Commons Codec library. Notably, it makes use of the public Base64(int lineLength) constructor for the Base64 class, which was only introduced in version 1.4. The current project also had a dependency on docx4j, which in turn depends on version 1.3 of the Apache Commons Codec library, resulting in NoSuchMethodExceptions being thrown when the constructor is used. The solution is to either ensure that the Sardine dependency is declared first in the POM (in Maven versions 2.0.8 and later) or that an Apache Commons Codec version >=1.4 dependency is added explicitly to the POM. From the Maven Introduction to the Dependency Mechanism:

You can always guarantee a version by declaring it explicitly in your project’s POM.

Conclusion

There’s obviously much, much more to using Commons VFS with Sardine and WebDAV, but the above should get the basic toolchain up and running, providing a good base on which more sophisticated file operations can be built.