ownCloud Planet

Welcome to the ownCloud community update for the first half of July – covering a crazy month! We released ownCloud 8.1 and published many blogs and videos about what’s new and improved. In coding news, work on 8.2 has started with work towards moving the Files app to WebDAV, revival of the Bookmarks app and the merge of many feature branches.

If you have thoughts or contributions for a future news post, ping Jos with your input. You can read our previous report here.

Moving Files app to WebDAV

A long-time goal has seen recent progress– moving the Files app to use the WebDAV interface. Reducing the number of code paths for file management, this will make it easier to test and improve ownCloud file handling.

Currently there are two ways in which files are managed, WebDAV and private PHP AJAX end points in the Files app. By moving the Files app over to using the WebDAV API, we concentrate future efforts in one place. As the WebDAV API is used by our mobile and desktop clients as well as any other WebDAV tool, performance and stability improvements will benefit those as well.

A lot of work is required to make this work. The Files app has previews, icons, favorite and sharing information, chunked upload and much more. Some of this is already available in the WebDAV API, as the clients have had a need for it, but much will still has to be added. Some interesting challenges are related to performance, like batch file deletion or moving. Older ownCloud releases used to delete or move files one by one, creating a huge number of database, network and file system operations. Newer ownCloud releases bundle such operations, something currently not available for WebDAV.

However, fixing these issues will greatly benefit the clients as well, as they can also make use of the new capabilities to improve their own performance and feature set. Moreover, the WebDAV backend, being based on the well-known SabreDAV, has several capabilities and advantages over the custom AJAX API’s. One of those advantages is using ‘part files’ internally, which makes it easy to resume uploads or retry parts which went wrong. This also helps deal with concurrency issues where, for example, a mobile or desktop client starts to sync a file before it is completely uploaded in the web interface (example). Similar issues exist with regards to encryption.

Right now progress on moving the Files app to WebDAV has stalled due to a compatibility issue–Internet Explorer 8 is incapable of handling the needed WebDAV commands, creating issues for the new Files app. Unless IE 8 support gets dropped or another work around implemented, introduction of the new WebDAV based Files app might have to be delayed to a later release.The OCS logo

Improving the OCS API

Complementing WebDAV, the OCS API is the other half of the external interface ownCloud offers to third party developers to access data provided by ownCloud apps. It is documented here.

There is work going on to improve the OCS API. To support Smashbox testing, an OCS API was developed for sending, receiving, accepting, and declining remote shares and OCS Share API link shares now always have a URL which fixes issue 16488. As that issue explains, this makes sure that the URL is correct (since we use the URL generator to get the route) and not depend on a client generating the correct URL. It also ensures that if the URL would change in the future the client would directly generate the new URL. It would also make it easier to write an app for URL shortening.

Another part of the effort to improve the OCS Share API is that Javascript is now used to fetch mimetype icons. This is kind of a separation of concenrs. The javascript is resposible for getting the correct icon for a given mimetype (without additional server calls) making sure ownCloud does not have to pass the icon around everywhere. This change was implemented in the Files app, bringing more responsive performance as an added advantage. In other mimetype work, users can add their own mimetypes and icons can now be changed by the theme.

Using Public API Internally

As described above, the Files app uses private PHP AJAX functionality rather than using the public OCS and WebDAV APIs to bring data from an ownCloud Server to the browser. Within the ownCloud Server, a similar situation exists where many ownCloud server apps have been developed using non-public functionality from the Core to do their work (the OC_ namespaces). Often there are public APIs (\OCP namespaces) available and this duplication is also due for cleanup.

Work on this is tracked in issue 4774 and the developers are hopeful that most instances of API duplication can be dealt with during the ownCloud 8.2 release cycle. This “dogfooding” will have similar benefits overall as those in the Files app– reducing the number of code paths allows more focused testing, bug fixing and improvement efforts.

PHP 7 Compatibility

As you might know, the PHP community is preparing for the release of PHP 7 for the end of this year. This first major release in quite some years will bring significant performance improvements between cleanups and new features. ownCloud developers have been making some modifications to ensure that ownCloud will run on this new PHP, like this PR.

Reviving Bookmarks

Blizzzblogged about his decision to make the ownCloud bookmarks app “prosper and flourish” again. He went through the open issues and pull requests, closing many, cleaning others, merging some and fixing more. A beta release has been made and you can expect a final release with many improvements in the ownCloud app store soon.

If you want to get involved in coding on ownCloud – check out this page!

Other ownCloud Happenings

On Planet ownCloud, a post from ownCloud.com featured a video of the Datto announcement of ownCloud partnership. This article (forbes.com) explains that Datto, a company with over 5 million customers world-wide, now ships ownCloud as part of their storage solutions. The writer points out that, in the crowded Enterprise File Sync and Share market, solutions have to provide something unique and many companies are looking for ways to store their data on-premise, benefiting from the security advantages this brings. Watch the announcement video at dattocon 2015 on owncloud.com

Note that if you’re an ownCloud contributor, you should get your blog aggregated on owncloud.org/news! Ping Jos with your RSS feed.

We hope you enjoyed this fresh report from ownCloud development!

Thanks to Vincent Petry and others for the tireless catching of embarrassing mistakes

At the ownCloud Contributor Conference ownClouders will get together for a week of discussion and coding on ownCloud. The talk program will kick off on Saturday, August the 29th, with a keynote by Angela Richter. Today, we announce our exciting program of talks and workshops, which are informing, inspiring and fun.

The Conference from a Bird’s Eye View

For designers and coders new to ownCloud contribution, the conference is the place where you write your first ownCloud app, contribute your first feature or get your first bug fix in! Basic web development skills (PHP, CSS, HTML and Javascript) are enough to get you started. Be sure to bring your laptop – we’re a hands-on community and code has to be written!

With already over 250 registrations, you are likely to find ownCloud core developers you are looking for at the conference. The existing contributors find the event a great place to discuss what they are working on, share inspiration and ideas and to get work done. The 5 days of coding are complemented with a two-day program on Saturday and Sunday with inspirational morning sessions and learning opportunities with our in-depth workshops and technical talks in the afternoon.

This event isn’t like traditional conferences – ownCloud is about making the change happen. While we have talks in the morning during the weekend, the afternoon and the rest of the week is dedicated to workshops and coding in the hacking rooms.

The Program

Most days will roughly run from 10am to 6pm, however, especially during the hacking days, the closing time tends to be more like around midnight. Saturday and Sunday we’ll open doors and registration at 9am while the introduction and first keynote will kick off at 10am with the program ending around 6pm.

Saturday

Saturday will be kicked off by an inspiring keynote from theatre director Angela Richter (interview with her here), followed by Frank, who will have some exciting announcements and great insight on where ownCloud is going.
After a short break, we’ll continue with lightning talks, including:

The morning also features two ‘ownCloud in action’ talks, one from our atom-smashing friends at CERN and one from Sciebo, the team building a 500.000 student deployment in North Rhine-Westphalia, Germany.

Hacking Days

Following the two scheduled days, as well as the Friday preceding it, we have our hacking rooms open and expect over a hundred ownCloud contributors getting their hands dirty. The inspiration and things-learned from the two days with talks can and will be put into code. It is easy to participate – the atmosphere is entirely informal, there are no expectations, time tables or pressures; everybody is free to do and work on what they find to be most interesting. We will have easily identifiable people who are there if you have any questions, technical, social or process-wise. And of course, we provide food and drinks during these days.

The ownCloud Contributor Conference is coming to Berlin from August 28 to September 3. The 2 day talk program will kick off on August 29 with a keynote speech given by Angela Richter, house director at the Cologne National Theater and director of the interactive project “Supernerds,” on the topic of digital surveillance, whistle blowing and digital dissidents. She has worked with various digital activists and hactivists such as Julian Assange, Daniel Ellsberg, Edward Snowden, Thomas Drake and many others, with the goal of bringing their story to a wider audience. Read on for a short interview with her.

Who is Angela Richter?

We asked Angela to introduce herself.

I am currently one of the four house directors at the Cologne National Theatre “Schauspiel Köln.” In recent years I have worked with such digital activists as WikiLeaks founder Julian Assange, and in 2012 premiered my piece “Assassinate Assange,” which was shown in Berlin, Hamburg, Cologne and Vienna. Since then, I not only visited regularly with Julian Assange in London, but have engaged in panel discussions, conferences and in various print media including Der Spiegel, Monopoly, Interview Magazine, and Revue Magazine for the Next Society for internet activists and hacktivists. During the 2014/15 season, I wrote and directed the interactive project “Supernerds” (info about Supernerds in English, German review) on the topic of digital mass surveillance, whistleblowing and digital dissidents in collaboration with national television and radio WDR, Schauspiel Köln and the Producers Gebrueder Beetz, including an “Internet Sudden life Game.” The audiences in the theatre, on TV and internet were part of the story; they experienced examples of hacking and surveillance on their smart phones and laptops. Supernerds is seen as a unique example of Transmedia because it allowed audiences to explore the story in several medias such as internet, sudden life gaming, radio, TV and live on stage.

The play was based on interviews that I conducted in the last three years, with whistleblowers, human rights lawyers, hackers and internet activists, among them Julian Assange (Wikileaks), Daniel Ellsberg (Pentagon Papers), and the NSA Whistleblowers Bill Binney, Thomas Drake and Jessely Radack. I also visited Edward Snowden in Moscow in February 2015 (German interview with him here). I published a selection of the interviews in the Book “Supernerds – Conversations with Heroes” with Alexander Verlag Berlin, both in German and English.

What do you see as the biggest problem with regards to privacy in our society? And what should we do about it?

Not enough awareness in the mainstream population. Politicians rather smother it up and downplay it. Mainstream media doesn’t shed light on it.

We need both technical and social solutions. First enlightenment to invigorate awareness about it. And also tech solutions; encouragement to make people less dependent on big Internet companies, who incapacitate people and hold them in dependency. Encouragement of free software, open source etc.

Your work and what you did with Supernerds (and other initiatives) gives you a ‘different’ perspective. How do you see this influence the debate and why does it matter?

As I am an artist and not a techie, I think I can communicate these issues without being intimidating. Through a project like Supernerds I can make people understand it easily and in a playful way. If I can get it as an artist, everybody can. Also the difference between knowing and really understanding how sensitive we are, in terms of surveillance, is important

At the ownCloud Contributor Conference, Angela will share some of her experiences and insights in both the issues she discusses in her work as well as in the work itself.

The busy time leading up to the ownCloud release has made it hard to keep up with development. As the value of outdated reports is minimal, this report will merely summarize work done over the last three months, after which we’ll put renewed effort in keeping up by re-instating weekly reports. These will have a reduced area of coverage to make regular updates feasible.

During April, May and June developers were mostly concerned with getting ready for the ownCloud 8.1 release. Since our report on March there have been feature additions, but finishing functionality merged during March has been the focus.

Development

In the three months we are covering here there have been over 560 pull requests that were merged in core alone – with a total of about 1215. As much work outside of the core takes place by direct commit rather than through pull request. It isn’t farfetched to conclude that core might be central to ownCloud, but at least half the work that benefits ownCloud users takes place outside of this repository, in the many apps developed by the ownCloud community. And that is not even covering those apps developed outside of the ownCloud project!Check out the feature overview of ownCloud 8.1

New Features

Not all major new ownCloud 8.1 features were in before April 1st. These were merged between April and the end of June:

Work is progressing on making core apps use the public API where possible. It seems the end goal is in sight and the participating developers have set their eyes on finishing this cleanup for 8.2. Expect more information about this effort in a future report

There has been a lot more of these, one of the reasons why ownCloud 8.1 is significantly faster than previous releases.

Smashbox Testing

Another area where a lot of effort was put in over the covered time period was to set up Smashbox for testing ownCloud. Smashbox is an automated testing tool developed by our Atom-smashing friends at CERN. We covered this in two blogs introducing Smashbox and describing how it is put to use.

Locking Work

ownCloud 8.1 has seen quite some work on file locking, like the addition of a file lock during download. This is part of an effort to make sure ownCloud cannot, even in the weirdest of circumstances, overwrite or lose files (see the many items tagged with feature: locking in github). ownCloud already does file locking to ensure two users can’t access the same file at the same time, but this takes place at the file system level. This will stop one user from overwriting a file being written by another user – but in between separate file operations, such as each individual file being uploaded to a folder, other operations can still take place. If, for example, one user uploads a big file to a folder on external storage which gets renamed by another user at the same time, strange things could happen. The new, high-level locking will ensure that the user who started his/her endeavor last will receive an error message, while the first user will finish successfully. In case of the sync client, it will automatically retry later.

ownCloud 8.1 introduces high level file locking as an experimental feature which has to be turned on in config.php. As it is only relevant to very large installations, while being very invasive and recently developed, it was decided to keep it off by default.

More

making the download preview for text look better

This provides a security hardening improvement to ownCloud. It came from an issue filed here as result of an external security audit done on ownCloud (this was the only problem found!)

If you insist in your user name to have a @ sign, now you can without breaking Federated Cloud Sharing

this PR should speed up the upgrade process a bit while this PR makes sure that maintenance mode state persists after upgrade (if it was on, it’ll stay on – and if it wasn’t, it will be turned off again)

ownCloud offers the ability to upgrade an owncloud instance via command line. In the past this was a very quiet command and only listed the very rough steps that were executed. On instances with more user data this results in long times without any visible progress.

This is totally fine if everything works as it should. But there were a few drawbacks with this output. When an app gets updated this is only printed once it is done. It would be more helpful to see which app gets updated next, because then you exactly know where to look for problems.

Beside that there are repair steps executed before and after the DB update. It would be nice to also list these steps and know a bit more about what happens during and upgrade.

These two enhancements landed in ownCloud 8.1.1 and the upcoming 8.2.0 when you specify the command line option -v which is widely used in other command line tools to increase verbosity too.

Because it is more fine granular you also know more precise what the upgrade process is actually doing.

Upcoming enhancements

In the next version of ownCloud we also will add a timestamp to each line of output while the -v option is specified. Then the admin knows exactly at which time a step happened and can give rough estimates which steps took how long.