Cloud It Yourself

"Fog can be considered a type of low-lying cloud," says Wikipedia [1], and who are we to argue; this definition sort of sums up the ambivalent nature of the whole cloud hype and the related services. The success of cloud file sharing and synchronization services like Dropbox leaves room for alternative offerings and projects, which are popping up like the proverbial mushrooms.

Dropbox, but Self-Managed, Please

What all these services have in common is that users transfer their data to a third party for fiduciary management. Most cloud services today only share the means of transport and entice users with partially free, or at least extremely convenient, offers of access to data from anywhere, along with the ability to share data with others.

The realization that this is not always a good idea started to mature when risqué images of celebrities began to emerge from the cloud. Additionally, the passion of government intelligence services for collecting data is virtually infinite. The affirmation of "We're the good guys" is always relative to the speaker's intent.

Nine Clouds and Two BitTorrents Compared

Hosting your own cloud is a potential answer to diverse dependencies. In this article, we examine software designed for private users and corporations. The minimum requirement our test team stipulated was that the software offered basic file sync mechanisms and at least partly allowed the user to control the transfer of data – that is, anything under the "Filesync and Share" (FSS) umbrella.

The following comparison starts with ownCloud and then presents eight alternatives (in alphabetical order) that provide similar services for Linux users. It also looks at the peer-to-peer-based approaches of BitTorrent Sync and Syncthing. Table 1 shows the main features, and the box "Anecdotes from the Cloud" gives you three tips from the cloud admin's daily grind to help you manage your own cloud – no matter which software you choose.

ownCloud

ownCloud [2] (Figure 1), developed in 2010 by Frank Karlitschek, is one of the most popular representatives in the DIY cloud software field. The modular design of the software suite, which is based on PHP and SQL databases, offers much more than just file synchronization and sharing capabilities, with numerous apps extending the functionality.

This universality may have been one of the reasons that the developer resources were distributed across too many features and that a weakness (in the form of the initially often-criticized, poor FileSync performance with large amounts of data) went unresolved for many moons. Indirectly, the developers even aggravated this with nominal support for SQLite as the database back end.

Users also reported problems with the sync clients. The developers were well aware of this, which explains why they postponed additional features, such as sharing via the desktop client or binary delta sync – which enables an efficient transfer of very large files, to focus on improved core functionality.

The current version 1.7 of the client, which as of writing is still beta, does add the eagerly awaited overlay icons, for example, that help the user identify in the file manager which data was shared. One major benefit of ownCloud is the almost infinite number of additional applications, which stem to a great extent from the strong community. The administrator of the ownCloud instance decides whether to use them. You can thus put together a customized package for use in the enterprise. You can even create your own extensions, for example, to map workflows, using an app.

Flexible and Comprehensive

OwnCloud installs on the major Linux distributions and Windows as well as on NAS devices, Raspberry Pis, or OpenWrt routers. Additionally, the architecture supports operation on most web servers and hosting packages. If you are worried about the installation and maintenance overhead, you can look at a list on ownCloud.org to find a trusted provider.

Server-Side Encryption

OwnCloud offers server-side encryption of data – which should be enabled – if you integrate external storage that is not under your own control. The documentation lists data that remains unencrypted in the ownCloud instance (e.g., thumbnails or even information in the search index).

In addition to the web interface, you can access ownCloud via native desktop clients for Linux, Mac OS X, and Windows, as well as apps for Android and iOS-based devices. Users of other mobile platforms must rely on existing WebDAV clients or revert to web access for mobile browsers, which was optimized in version 7.

Community and Enterprise Versions

The Community version of ownCloud can be generally installed and used free of charge. OwnCloud Inc. additionally offers an Enterprise version for companies. This differs from the Community version in terms of additional applications, which are not available under the AGPL, unlike all the other components, but rather under a commercial license. The Enterprise plan includes support services. An overview can be found on the ownCloud website.

For small-to-medium-sized companies with up to 500 ownCloud users, there is also a special offer of the Univention Corporate Server (UCS), which is installable via the App Center and can be purchased via Univention partners.

Conclusions: OwnCloud is characterized by its comprehensive approach and impresses by offering more than just a simple Dropbox replacement, both in home and enterprise use.

Cozy

French startup Cozy Cloud offers Cozy [3], another client-server product with a focus on a fully configurable web interface (Figure 2). Written in Python, the suite uses the document-oriented NoSQL database CouchDB as its back end. Cozy can also be installed on less powerful hardware, for example, Raspberry Pi machines or systems with little RAM. However, you can expect some compromises in terms of installable services, which – as usual – are known as apps here.

In addition to the file applications for accessing online files, you can find applications for calendars, contacts, and photo management, as well as various apps supplied by the community. One thing worthy of note is that, besides installation and upgrade instructions, the website also offers detailed uninstall instructions, in case a user happens to find Cozy less than all that.

Not a Multiuser Cloud

In the strictest sense of the word, Cozy really is your own cloud; although you can manage and share your own data with external users, the solution currently only allows one user per Cozy. Additionally, no option currently exists for integrating other Cozies to create what are known as "Federations."

The user can access the data with WebDAV clients via the Cozy Sync App. An Android client can be found in the Playstore. Desktop clients are a work in progress according to the Cozy forum.

Even though many other elements are still in flux in this project, the development team has obviously done enough to convince affluent investors of the quality of its work and thus put the project on a solid financial basis [4].

Conclusions: The promising technical approach and obviously secure funding are reason enough to keep an eye on Cozy, even if its standards are still currently not in line with your requirements.

Filr

After handing over iFolder to the Kablink community, Novell has relied on Filr [5] for a secure exchange of data since the end of 2012. Differences of opinion exist regarding its participation in the test; the critical motivation for saying yes was ultimately that it runs on both Windows servers and Novell OES 11, which in turn is based on SUSE Linux Enterprise Server 11 (Figure 3).

Filr is tailored for use in the enterprise, which Novell apparently understands to be mainly customers with SLES, OES, or Windows infrastructure. The product is available as a virtualized appliance on VMware, Hyper-V, or the Xen implementation of SLES 11 as of SP2.

Although the documentation only refers to Windows Active Directory and Novell E-directory as the supported directory services (to match the target group), Open LDAP integration should be also possible.

High Hardware Requirements

The hardware requirements specified for operation are relatively high: 12GB of RAM and four CPU cores are required for a single-server installation as an appliance. However, this is quite normal for an application relying on Java and Tomcat. For larger, distributed installations, the requirements are leveled out to some extent.

Filr is mainly designed to provide access to existing internal filesystems; it currently lacks the ability to integrate external storage. Linux desktops are also not part of the target group: Novell offers Filr synchronization clients for Windows and Mac OS X. In terms of mobile clients, the selection is comprehensive, including iOS, Android, Windows Phone, and BlackBerry 10. The administrator decides whether or not to allow specific items on their own system.

Conclusions: Filr is an interesting and, in part, flexible FSS solution for Novell OES customers and companies with a strong focus on a Windows infrastructure.

Git-annex

Linux Magazine already featured the file synchronization software git-annex [6] just over a year ago [7].

The main advantage of git-annex compared with a plain vanilla Git solution, such as SparkleShare, is the way you retrieve data from the repository [8]. This is what really gives the user a useful way of storing large files or creating their own Git repositories. Users with little inclination to visit the command line will appreciate git-annex assistant, which became part of the project after a hugely successful Kickstarter campaign by the developer. It lets users synchronize their own files with git-annex through a web interface.

In the Repositories

The software is included in the standard of many popular Linux distributions, although sometimes in older versions. More recent releases – like the OS X port and the still beta Windows Installer – can be found on the project page. Also, the Android app is still in beta and thus, for now, is only available from the project website.

Conclusions: Git-based synchronization without the disadvantages of Git.

Grau DataSpace

Grau Data from Schwäbisch Gmünd, Germany, offers Grau DataSpace (GDS), a product that is difficult to compare with the other candidates in our test.

Grau Data has many years of experience in the field of hierarchical storage management (HSM), archiving, and data exchange, and it primarily addresses businesses and providers with its GDS [9]. In terms of architecture here, Grau DataSpace is more at home in the content management category than as a free cloud.

A master-master replication completes the client-server model in Grau. This means, for example, that you can achieve cross-site, comprehensive data replication, thus catering to even the most complex scenarios, including setting up emergency data centers in a hybrid cloud. GDS is therefore quite liberal when it comes to choosing the storage back end. Although Grau always manages the metadata in a SQL database, the data storage itself can reside in the database, any CIFS store, or in an ObjectStore (e.g., Open Stack Swift or Amazon S3).

More of a CMS than a Cloud

With regard to the client access methods, GDS takes different approaches in addition to the usual, but with an HTML5-optimized web interface. Support from the content management interoperability services (CMIS), from the enterprise content management (ECM) camp, allows an exchange of data with other CMIS-enabled applications, such as SAP or SharePoint, while supporting access via existing CMIS clients, such as CmisSync. CMIS is also the basis for native clients on mobile devices that are available for Windows Phone and BlackBerry in addition to Android and iOS. A web interface optimized for smartphones and tablets by the name of Touch UI completes the offer.

Another method of accessing the data stored in GDS is via a gateway, which ownCloud synchronization clients can connect to. On the local network, admins can also allow access via CIFS.

For the fourth quarter of 2014, Grau Data is planning to release version 2.4 of GDS. This comes with a heavily revised desktop synchronization architecture, multiclient capability for use in hosting a "server trashcan."

Conclusions: Grau DataSpace is a highly scalable platform for the exchange of data in the enterprise.

Pydio

This project [10] was launched in 2010 as AjaXplorer by French programmer Charles du Jeu, but it was renamed by the developers to Pydio (Put Your Data Into Orbit; Figure 4) in October 2013. Apparently, the date also marked the transition to an additional commercial offering by Abstrium SAS, whose website is still under construction. The server-side architecture shows many parallels with ownCloud; however, with the almost simultaneous launch of the two projects, the question "Who invented it?" is difficult to answer.

In both tools, a front end written in PHP uses a MySQL or SQLite database to manage the metadata and stores the actual data on external or local storage. Currently, access to data in Pydio is limited to the web front end and mobile apps for iOS and Android, as well as a WebDAV interface. The desktop client, written in Java, has been frozen in beta since January 2014 and is being completely revised.

AjaXplorer – The Thunderbird Plugin

In the Mozilla add-ons, Thunderbird users will find a provider for file links, dubbed AjaXplorer, that helps them upload files directly from the mail client and generate sharing links. Although the file management capabilities in the Pydio workspaces are very extensive in the web interface, you will look in vain for functions that go beyond this. However, Pydio provides core functionality with virtually no weak spots, aside from the lack of a stable desktop client.

The developers offer both commercial case support for the community version, which is licensed under the AGPL, and a subscription model for a manufacturer-certified edition.

Conclusions: Pydio is a good alternative to ownCloud and surprises with its Thunderbird plugin.

Seafile

Seafile [11] is backed by a small team from China, Vietnam, Canada, and Germany. The installation of the server component is described for Linux (Red Hat-based, Debian, and Raspberry Pi), as well as Windows. Setup, as well as upgrading the server (which is provided as a tar.gz file), does involve some manual work at the command line. However, given the good documentation, this seems manageable (Figure 5).

In the Seafile web interface, you will find a few basic components for collaborative work on projects, a discussion forum, and a wiki, in addition to the file management features. Viewers are available in the Community Edition for documents in Open Document Format (ODF). The Business Edition is required to preview Microsoft Office files. The latter also contains a full-text search, support for object storage, and clustering, as well as features for tracking activities in the team's data.

In addition to desktop clients for Windows, Mac OS X, and Linux, you can synchronize the data from your own Seafile cloud via a command-line-based client on Linux. The Filelink add-on for Thunderbird users is available via a GitHub project linked from the Seafile website.

Mobile users of Android and iOS devices will find synchronization software in the respective app stores. The synchronization mechanisms and stability of Seafile are generally praised. According to the roadmap, a new major version is scheduled for the fourth quarter of 2014.

Conclusions: If you are primarily looking for a client-server solution for robust file synchronization and sharing and are not afraid of the command line, you will want to put Seafile on your shortlist.

SparkleShare

The makers of SparkleShare [12] are open and honest enough to reveal on their homepage what SparkleShare is good at doing, and what it isn't. The main application for the tool is synchronizing small to medium-sized, files under continuous version control.

If you work with Git in some form, you will find it easy to get started; otherwise, the setup might already be too technically demanding. It is difficult to imagine typical users working with this service, even given good administrative preparation. ViewGit or other Git front ends will tend to be overly complex.

The restricted group of users could be the reason why – for two years – nobody has updated the Android and iOS clients for SparkleShare. However, based on the GitHub pulse, the project still seems to be alive, although with very little community involvement. The last major update is from the April 2014.

Conclusions: Although SparkleShare's range of application is limited, it is quite suitable for simple synchronization tasks.

Tonido and FileCloud

Texas-based CodeLathe offers two separate variants of its software [13]. Although Tonido is primarily aimed at home users and small businesses, FileCloud addresses business customers with 20 users or more. Both versions are packetized or available as archives for various Linux distributions, Mac OS X, and Windows.

Tonido is also available for the Raspberry Pi. The latter is free for typical private use (up to five shares and five guest accounts), but it can be beefed up with an annual subscription.

Although the data remains on your own systems, you need a Tonido ID (e.g., <user>@tonidoid.com) for access as a substitute for the otherwise necessary DynDNS service. Although this approach may be convenient for some, it is likely to cause raised eyebrows among self-hosting purists. Worse, Tonido implements its core functions – even on Linux – with closed source binaries.

Commercial License

The FileCloud enterprise solution is also completely under a commercial license. Major parts of the PHP scripts, which run against a MongoDB database, are also protected from scrutiny by ionCube. Any reverse engineering is prohibited in the FileCloud terms of services. The terms also include the condition that FileCloud calls on the provider to look for software updates, and the privacy rules were actually written for Tonido.

In terms of features, FileCloud leaves little to be desired as an FSS solution. In addition to Web UI, WebDAV access, and desktop synchronization, CodeLathe also provides its own Windows application named Cloud Drive that mounts the cloud directly as a drive.

An app is available for Windows Phone, alongside iOS and Android support for mobile access. Outlook users can use a special add-on to free their mail attachments and send sharing links instead. As you would expect for a commercial offering, the documentation is also extensive.

Conclusions: The combination of Tonido and FileCloud is an attractively priced alternative to file synchronization and sharing, assuming that you trust the claims and intentions of the provider.

BitTorrent Sync

Many people only know of Torrents [14] by hearsay in the context of illegal file-sharing activities. However, the peer-to-peer protocol is also suitable by definition for synchronizing your own files across multiple devices or for sharing with other users. The venture-funded BitTorrent Inc. offers software called BitTorrent Sync (Sync or BTSync for short) [15] in addition to the normal BitTorrent client, which offers precisely this functionality (Figure 6).

The free Sync client lets you synchronize data between different computers and systems on the LAN but also – thanks to NAT traversal and UPnP port mapping – on the Internet; this means you can also use it as a home cloud system.

Each instance of BitTorrent Sync optionally acts as a data transmitter (client) or receiver (server). If you have a Raspberry Pi at home running a server instance, it receives the data from the stationary and mobile clients running the client. It has two modes of operation: read-only and full-access mode. In both, BitTorrent Sync keeps the data in sync between the source directory (client) and the target directory (server).

The difference: In full-access mode, BTSync deletes files from the source directory as soon as they disappear from the target directory, but this does not happen in read-only mode. On request, BTSync first pushes deleted files into the .SyncArchive subfolder.

BitTorrent Sync encrypts the data stream with an AES key and a 20-byte security key, according to the manufacturer's details. However, you cannot determine how secure the encryption actually is, because the source code is missing. If you do not trust the encryption, you can also use BTSync purely locally.

The good thing is that BitTorrent Sync runs across platforms and in the background; it is easy to use and also supports mobile clients. Maintaining the software is somewhat time consuming, however, once you have it running on multiple computers. After you upgrade to a new version, you need to set up BTSync again wherever you use it. Additionally, some of the sync processes hung frequently in our ADMIN magazine lab, forcing the admin to set up the server side yet again.

In contrast to the architecture of other FSS solutions, Sync does not require a server. Data is exchanged exclusively between the devices running Sync. Users can share with other Sync users and notify them by email or QR code. Both full control and read-only rights are configurable.

The fact that Torrent handles large files well is known to at least the users of traditional BitTorrent clients. BitTorrent Inc. offers closed source clients for Linux, Windows, Mac OS X, and FreeBSD. For mobile devices, the range extends from Android through iOS to Windows Phone. Sync is also available for some NAS devices (e.g., QNAP and Netgear).

The lack of an option for accessing data without a Sync client somewhat limits the range of applications for the tool. Although users retain control over the systems under their own sovereignty and can also influence shares, they need to trust that the clients do not have any data leaks. The less than fully transparent business model of BitTorrent Inc. – at least in terms of Sync – will typically lead to users developing an alternative plan for their synchronization strategy.

Conclusions: Sync offers robust file synchronization without a server and – restricted – external data sharing but is unfortunately closed source.

Syncthing (Pulse)

Syncthing [16] seeks to be an open and secure alternative to BTSync. The current version is 0.10.1; in other words, the software is still in its infancy. It synchronizes files on the LAN and over the Internet and comes with web-based clients for Linux, Windows, and Mac OS X.

Syncthing is the open source version of BitTorrent Sync in terms of functionality. The project, which was initiated by Swedish developer Jakob Borg last year, has found quite a few supporters on GitHub already. Ports of the Syncthing desktop client include Linux (including ARM 5 and ARM 6 architecture), Windows, Mac OS X, FreeBSD, and Solaris.

In terms of mobile devices, Android is currently the only supported OS – via the source code, which is available on GitHub. Syncthing is currently mainly of interest to technically savvy users as an alternative to BitTorrent Sync because of its relatively early stage of development.

Ind.ie

Syncthing was renamed in October 2014 to Pulse and became part of the Ind.ie project [17]. Aral Balkan, the founder and lead developer of Ind.ie, seeks to gain more independence from Google, Apple, and others. This drive, financed by a Kickstarter campaign that was still in progress in November 2014, includes the Ind.ie operating system, the Heartbeat social network client Pulse, and the Ind.ie phone, which is just one more reason to keep an eye on the project.

Conclusions: Noteworthy open source alternative to BitTorrent Sync.