Category Archives: Programming

Saving a file to an OS X WebDAV server using Apache VFS and Sardine

by

I’ve recently been looking into the options for storing a relatively large number of user-uploaded files in a non-hierarchical storage “bucket” with a sufficient degree of abstraction to allow the files to reside either locally or on a server (e.g. WebDAV, SFTP) without any (or many) changes to the code. A virtual file system (VFS) is pretty well suited to this kind of task and, as it happens, there are a number of VFS implementations available for Java including Apache Commons VFS and TrueVFS.

For now, I’ve opted to use Apache Commons VFS with a WebDAV file provider. Overall it’s relatively straightforward to get the various moving parts up and running, but there were a few pitfalls in setting up the WebDAV server, ensuring that Maven is pulling in the right dependencies and copying file content to VFS.

Rationale for using Sardine

From the Commons VFS homepage: “Apache Commons VFS provides a single API for accessing various different file systems. It presents a uniform view of the files from various different sources, such as the files on local disk, on an HTTP server, or inside a Zip archive.” WebDAV is one of the sources supported by Commons VFS, but the default implementation is the same as that used by Apache JackRabbit. As of writing, the JackRabbit implementation relies on the deprecated Commons HttpClient for its HTTP support, which has long been superseded by Apache HttpComponents. Including a deprecated and unsupported HTTP client library isn’t very appealing from a security standpoint, neither is it desirable in terms of longer-term project maintenance.

Thankfully, Sardine comes to the rescue here. In brief, Sardine is a relatively modern, lightweight WebDAV client focussed on the most common use cases for WebDAV. In combination with Nicolas Delsaux’s excellent commons-vfs-webdav-sardine project, it’s relatively straightforward to use Sardine as a Commons VFS WebDAV provider.

Configuring a WebDAV server compatible with Sardine

One of most attractive aspects of WebDAV is that it’s extremely easy to configure, especially on machines that are already running Apache. However, even given the relative ease of configuration, there are usually a couple of file permission and directory index issues that needed to be resolved.

A basic WebDAV server can be set up by including (or preferably Including) the following in httpd.conf:

LoadModule mod_dav_fs

DavLockDB "/usr/webdav/DavLock"

Alias /uploads "/usr/uploads"

<Directory "/usr/uploads">
    Dav On

    AllowOverride None
    Options Indexes FollowSymLinks
    Order Allow,Deny
    Allow from all

    AuthType Basic
    AuthName WebDAV-Realm

    AuthUserFile "/usr/webdav.passwd"

    <LimitExcept GET OPTIONS>
        require user admin
    </LimitExcept>
</Directory>

The inclusion of the “Options Indexes FollowSymLinks” directive is important as it prevents 403 errors being returned in response to HEAD requests to test for the existence of files on the WebDAV share.

The directory containing the DavLockDB file and the WebDAV directory both need to be readable and writable by the current Apache user and group. The following commands should ensure that this is the case (given the above configuration in httpd.conf):

sudo mkdir /usr/webdav
sudo mkdir /usr/uploads
sudo chown www:www /usr/webdav /usr/uploads

With AuthType set to Basic, the only remaining task is to create the “admin” user referenced in the LimitExcept directive and restart Apache:

sudo htpasswd -c /usr/webdav.passwd admin
sudo apachectl graceful

On OS X, the server configuration can then be tested using “Connect to Server…” in the Finder (Cmd-K):

WebDAV connection

Authenticating with “admin” and the password set in the htpasswd stage above should mount a readable and writable WebDAV share in the Finder.

Using Sardine with Apache VFS

With a WebDAV server running locally, VFS and Sardine can be used to take a “local” file and upload it to the WebDAV server as follows:

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.FileSystemException;
import java.util.UUID;

import org.apache.commons.io.IOUtils;
import org.apache.commons.vfs2.FileObject;
import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
import org.apache.commons.vfs2.provider.local.DefaultLocalFileProvider;

// This import causes the Sardine WebdavFileProvider to be used rather than the default VFS provider
import fr.perigee.commonsvfs.webdav.WebdavFileProvider;

public String saveFileToDefaultFileSystem(File file, String baseURL) throws FileSystemException, IOException {
	
	String fileName;
	DefaultFileSystemManager fsManager;

	try {
		fsManager = new DefaultFileSystemManager();
		fsManager.addProvider("webdav", new WebdavFileProvider());
		fsManager.addProvider("file", new DefaultLocalFileProvider());
		fsManager.init();
	} catch (org.apache.commons.vfs2.FileSystemException e) {
		throw new FileSystemException("Exception initializing DefaultFileSystemManager: " + e.getMessage());
	}
	
	UUID uuid = UUID.randomUUID();
	fileName = uuid.toString();
	
	FileObject uploadedFile;
	FileObject destinationFile;
	
	try {
		uploadedFile = fsManager.toFileObject(file);
		destinationFile = fsManager.resolveFile(baseURL + fileName);
		destinationFile.createFile();
	} catch (org.apache.commons.vfs2.FileSystemException e) {
		fsManager.close();
		throw new FileSystemException("Exception resolving file in file store: " + e.getMessage());
	}
	
	try (InputStream in = uploadedFile.getContent().getInputStream();
		 OutputStream out = destinationFile.getContent().getOutputStream()) {
		
		IOUtils.copy(in, out);
		} catch (IOException e) {
		throw new IOException("Exception copying data: " + e.getMessage());
	} finally {
		fsManager.close();
	}
	
	return fileName;
		
}

The above saveFileToDefaultFileSystem() method:

  • Takes a Java File and a baseURL as a String
  • Sets up the VFS DefaultFileSystemManager adding providers for WebDAV and the local filesystem
  • Generates a random filename (UUID) for the bucket store
  • Resolves and creates the (heretofore non-existent) file in the WebDAV file system
  • Converts the passed-in File to a VFS FileObject
  • Uses a try-with-resources block to open an InputStream and an OutputStream from the source and destination FileObjects, respectively
  • Uses the Apache Commons IOUtils to copy the InputStream to the OutputStream and closes everything down
  • Returns the UUID as a String

The only other aspect not covered in the code is the form of the URL passed into the method, which must contain the full URL for the WebDAV server, including credentials as follows:

webdav://username:password@127.0.0.1:80/uploads/

VFS does support use of a UserAuthenticator rather than including credentials in the URL, but that’s probably overkill for getting something up and running quickly. There’s an example of how to use UserAuthenticator on the Commons VFS website, but as a middle ground, the password can be encrypted:

java -cp commons-vfs-2.0.jar org.apache.commons.vfs2.util.EncryptUtil encrypt mypassword

and the output enclosed in curly braces in the URL as follows:

webdav://username:{D7B82198B272F5C93790FEB38A73C7B8}@127.0.0.1:80/uploads/

Maven

Sardine, Apache Commons VFS2 and the Commons VFS Sardine WebDAV file provider will all be added as dependencies when the commons-vfs-webdav-sardine dependency is added to Maven.

Sardine has a dependency on the Apache Commons Codec library. Notably, it makes use of the public Base64(int lineLength) constructor for the Base64 class, which was only introduced in version 1.4. The current project also had a dependency on docx4j, which in turn depends on version 1.3 of the Apache Commons Codec library, resulting in NoSuchMethodExceptions being thrown when the constructor is used. The solution is to either ensure that the Sardine dependency is declared first in the POM (in Maven versions 2.0.8 and later) or that an Apache Commons Codec version >=1.4 dependency is added explicitly to the POM. From the Maven Introduction to the Dependency Mechanism:

You can always guarantee a version by declaring it explicitly in your project’s POM.

Conclusion

There’s obviously much, much more to using Commons VFS with Sardine and WebDAV, but the above should get the basic toolchain up and running, providing a good base on which more sophisticated file operations can be built.

Generating a hyperlinked table of contents for an Excel workbook using VBA

by

If you’ve ever needed to generate a hyperlinked table of contents (TOC) for an Excel workbook with more than a few worksheets, you’ll appreciate that it can be quite a laborious task. The following snippet of VBA will iterate over the sheets in a workbook, generating a list of hyperlinks to each sheet in a new sheet at the start of the workbook called “TOC”. If a “TOC” worksheet already exists, the script will ask for a new worksheet name in which to place the table of contents. Typing the same name again will overwrite the contents of the sheet with the TOC. Simple as that.

The code’s fairly run-of-the-mill with one possible exception: the implementation of the SheetExists function. SheetExists takes a string and returns true or false depending on whether a sheet with the specified name already exists in the active workbook. The function uses VBA’s On Error Resume Next statement to achieve this. As per the MSDN documentation, this “specifies that when a run-time error occurs, control goes to the statement immediately following the statement where the error occurred where execution continues.”

As such, in the SheetExists, the call to Sheets(SheetName) would ordinarily throw a “Run-time error 9: Subscript out of range” if the sheet didn’t exist. But with On Error Resume Next the code continues uninterrupted and can check to establish whether the assignment to TestWorksheet was successful or not. If it was, the function return value is set to True, the TestWorksheet variable is set to Nothing (so it can be garbage collected) and On Error GoTo 0 is called to disable the On Error Resume Next. Technically this last step isn’t required as custom error handling is disabled automatically when exiting the Function, but it’s a good practice to get into to avoid leaving run-time errors going unchecked over vast swathes of code.

There’s a gist available here for clones, forks or comments.

Sub GenerateLinkedTOCFromWorkSheetNames()

    Dim ProposedTOCWorksheetName As String
    Dim NewTOCWorksheetName As String
    Dim CurrentWorksheet As Worksheet
    Dim Count As Integer
    
    ProposedTOCWorksheetName = "TOC"
    NewTOCWorksheetName = "TOC"
    RowCounter = 2

    Application.ScreenUpdating = False

    Do While SheetExists(NewTOCWorksheetName)
        NewTOCWorksheetName = Application.InputBox( _
            Prompt:="A sheet named '" & ProposedTOCWorksheetName & "' already exists. " & _
                "Enter a new sheet name or type '" & ProposedTOCWorksheetName & "' to overwrite.", _
            Type:=2)
        
        If NewTOCWorksheetName = ProposedTOCWorksheetName Then
            Exit Do
        End If

        ProposedTOCWorksheetName = NewTOCWorksheetName
    Loop
    
    If SheetExists(NewTOCWorksheetName) Then
        Sheets(NewTOCWorksheetName).Cells.Clear
    Else
        Sheets.Add Before:=Worksheets(1)
        Worksheets(1).Name = NewTOCWorksheetName
    End If

    For Each CurrentWorksheet In Worksheets
        If CurrentWorksheet.Name <> NewTOCWorksheetName Then
            Sheets(NewTOCWorksheetName).Range("B" & RowCounter).Value = CurrentWorksheet.Name
        
            Sheets(NewTOCWorksheetName).Hyperlinks.Add _
                Anchor:=Sheets(NewTOCWorksheetName).Range("B" & RowCounter), _
                Address:="", _
                SubAddress:="'" & CurrentWorksheet.Name & "'!A1", _
                TextToDisplay:=CurrentWorksheet.Name, _
                ScreenTip:=CurrentWorksheet.Name
        
            RowCounter = RowCounter + 1
        End If
    Next
    
    Application.ScreenUpdating = True

End Sub

Function SheetExists(SheetName As String) As Boolean
    
    Dim TestWorksheet As Worksheet
    SheetExists = False
    
    On Error Resume Next
    Set TestWorksheet = Sheets(SheetName)
    If Not TestWorksheet Is Nothing Then SheetExists = True
    Set TestWorksheet = Nothing
    On Error GoTo 0
    
End Function

Angular.js tabs directive with dynamic loading of partial templates and controllers

by

The Angular.js framework strongly encourages modular application design, particularly when it comes to separating application logic from DOM manipulation. One of the greatest “it-just-works” components of Angular.js has got to be the ngRoute module, which allows extremely straightforward mapping of URLs to partial templates that load into a region of the main application template marked up with an ngView directive. For instance, a couple of simple routings might be defined as follows:

app.config(["$routeProvider",
	function($routeProvider) {
		$routeProvider.
			when("/home", {
				controller: "Home",
				templateUrl: "partials/home.html"
			}).
			when("/products", {
				controller: "Products",
				templateUrl: "partials/products.html"
			}).otherwise({
				redirectTo:"/home"
			})
	}
])

This would cause the respective partial templates (specified by the templateUrl property) to be loaded into a <div ng-view> in the main template, with the specified controllers attached. Unfortunately, there can only be one ng-view in the main template, so routing can only load partial templates into a single part of the main template.

In the app I’m currently working on, the single ng-view is used to load partial templates into the “detail view” of a two-pane list-detail view, in which the list view is a sidebar menu and the detail view takes up the majority of the screen and shows the “detail” corresponding to the current sidebar selection:

two-pane-view

This works well with one key exception: one of the detail views needs to include a number of other detail views in a tabbed area like this:

two-pane-view-with-tabbed-main-view

If we’re to make this work, it would obviously be highly desirable to re-use the same partial template HTML and controller JavaScript regardless of whether the detail view is loaded into the full screen view or the tabbed view of another detail view. With only one supported ng-view, this isn’t currently possible if we use the ngRoute module. Before filling this gap in functionality, it’s worth noting that the AngularUI Router framework already supports this exact type of functionality. Unfortunately, as of writing, the framework still comes with a disclaimer that the API is in flux and that it shouldn’t be used in production unless you’re willing to keep track of the changelog. Similarly, a much more sophisticated router is planned for AngularJS 2.0, and will be backwards compatible with AngularJS 1.3. But, for the time being, there are no “stable” modules that seem to enable this kind of functionality.

Angular AJAX Tabs

Starting with the tabs example on the AngularJS homepage, support for controller and template attributes can easily be added to the pane element. All of the changes to the directive are then concentrated in the link function, which is worth dissecting further:

link: function(scope, element, attrs, tabsCtrl) {
	var templateCtrl, templateScope;

	if (attrs.template && attrs.controller) {
		scope.load = function() {
			$http.get(attrs.template, {cache: $templateCache})
			.then(function(response) {
				templateScope = scope.$new();
				templateScope.isTabbedPane = true;
				templateCtrl = $controller(attrs.controller, {$scope: templateScope});
				
				element.html(response.data);
				element.data('$ngControllerController', templateCtrl);
				element.children().data('$ngControllerController', templateCtrl);
				
				$compile(element.contents())(templateScope);
			});	
		};
	}

	tabsCtrl.addPane(scope);
}

The link function first checks for the template and controller attributes on the element. If they don’t exist, it falls back to the standard tabs behaviour of inlining the HTML into the pane and adding the pane to the panes array in the tab controller’s scope. If both attributes do exist, a load function is added to the passed in scope. (The load function will later be called from the select function in the directive.)

The load function first uses $http.get to retrieve the partial template from the URL specified in the template attribute, adding it to the templateCache if it’s not already there. Once the template’s been retrieved, a new scope is created for the template and a single isTabbedPane property is added.

A new controller is then instantiated based on the controller attribute string, with the new scope injected in.

The HTML from the template is then set as the innerHTML of the pane element using the .html() function from Angular’s built in jqLite library. Similarly, the jqLite .data() function is then used to associate the instantiated controller with all of the pane’s child elements under the “$ngControllerController” key. $ngControllerController is poorly documented and there is certainly some confusion as to its purpose, but from a quick look through the AngularJS source code, it looks like it’s used by the internal jqLiteController function to return the default controller for an element. (jqLiteController provides the functionality underpinning the public controller() method on angular.element)

Finally, the template is compiled with the new template scope and the pane is added to the array in the tab controller’s scope as it would be had we not specified a controller and template.

Elsewhere, there’s also one minor change to the tabs directive, which simply triggers the pane’s load method when the select method is called at the time of a tab switch. The full select method in the tabs directive controller scope is then as follows:

$scope.select = function(pane) {
	angular.forEach(panes, function(pane) {
    	pane.selected = false;
	});
	if (pane.load !== undefined) {
		pane.load();	
	}
	pane.selected = true;
};

With the full module in place, ng-repeats and some structured metadata can be used to generate both the sidebar menu and the tabs themselves:

var tabbedPaneMetaData = [{
	"name": "Pane 1",
	"path": "pane-1",
	"partial": "pane-1.html",
	"controller": "Pane1",
	"includedInTabView": true
}, {
	"name": "Pane 2",
	"path": "pane-2",
	"partial": "pane-2.html",
	"controller": "Pane2",
	"includedInTabView": true
}, {
	"name": "Pane 3",
	"path": "pane-3",
	"partial": "pane-3.html",
	"controller": "Pane3",
	"includedInTabView": false
}];

In the tabbed pane, the following HTML could be added to the tabbed detail view controller:

<tabs>
	<pane ng-repeat="pane in panes | filter:{includedInTabView:true}" 
		tab-title="{{pane.name}}" 
		tabcontroller="{{pane.controller}}" 
		template="{{pane.partial}}">
	</pane>
</tabs>

Et voila, (relatively) straightforward configuration of ng-view partial templates and controllers as both full ng-view citizens and as participants in a tabbed view within another ng-view. The full Angular AJAX Tabs module source code can be found on GitHub with a barebones demo of the full functionality live on Plnkr.

Finally, if you’re already familiar with the concepts behind AngularJS but want to learn more about directives, Josh Kurz has written an excellent book, Mastering AngularJS Directives, that does the deep dive on writing directives, including many of the aspects covered briefly in this post.