Concurrent Online WebGIS

Last year we were playing around with websockets and we:

“…figured that an interesting use-case would to have a multi-user GIS where you can actually see where the other guy is, what he is seeing and together edit the map; think google-docs for map editing.”

We showed a first version of our application ‘cow’ (concurrent online webgis). Since then we’ve been expanding the possibilities. An obvious one is being able to add, edit and delete objects on the map; symbols, lines and polygons. We created a version where you could mark where the nature fire started with a portable device, that location would be shared over websockets and the nature fire model would calculate the spread of the fire and return the resulting time-polygons to all connected devices.

Another important possibility we explored was connecting Phoenix with websockets. This way you can expand Phoenix’ ‘same place, same time’ multi-user capabilities with ‘different place, same time’ collaboration. This could mean that someone in the field can see the results of discussions around the table as they are drawn into Phoenix on his smartphone or that different groups of experts in different countries all can contribute on the same map using multiple instances of Phoenix. What you see below is a combination of Phoenix with cow’s webviewer on a Windows 8 all-in-one PC, an Android tablet and an Android smartphone.

YouTube Preview Image

Websocket based 3D viewer

Our French intern Nils created a nifty extension on Phoenix; the Google Earth connector. This allows an user to control a seemingly unlimited number of 3D views from within Phoenix. The user has a control showing the current position in the 3D space and the point in the middle of the 3D view (the viewpoint). The user can change the 3D view by either dragging the viewpoint, the control or the map. You also can load 3D models in Phoenix and position them on the map.

YouTube Preview Image

If you connect more than one 3D view they will be connected to the initial control. However dragging a screen away from the control will create a new control, allowing for multiple views with independent control. This way multiple users can control their own 3D view, or an object or area can be shown from different angles in a in intuitive way.

The setup works with a websocket server which receives the coordinates, models, etc from Phoenix and passes them on to the connected webclients. The clients run the google earth plugin, which is controlled by some javascript and the commands received through the websocket connection. If a new client connects or an existing one disconnects the server informs Phoenix.

Building a websocket based multi-user map

Websockets are the next big thing for full-duplex communication on the web. At least that is what they are saying for a while now. Actual implementations are hard to find and documentation is a bit sparse. However, the idea of full-duplex communication is intriguing, the most used example is a webchat client. No more polling, long-polling, keep-a-lives and other tricks. Since there are already an over abundance of webchat clients it is not really a strong use-case. We figured that an interesting use-case would to have a multi-user GIS where you can actually see where the other guy is, what he is seeing and together edit the map; think google-docs for map editing.

Websockets are really easy to start with, especially if you use node.js, on the client it is:

var url = ws://host/
var ws = new WebSocket(url, 'connect');
ws.onopen = onOpen;
ws.onmessage = onMessage;
ws.onclose = onClose;
ws.onerror = onError;

On the server I used this; basically you start a websocket server and define what it has to do when people connect, send messages and disconnect again. This worked like a charm. In no time we were sending view-extents, features and the like around on the local network.

The problem however is that websockets are really really new. There is not that much information about them and hardly any tools to work with them. Most of the hardware of the internet does not know how to deal with it. We already had a problem when the node.js server was moved onto a different network within the company. The gateway didn’t understand the websocket packages and dropped them. There is a nice article explaining what is happening and what you should do. Basically the way forward is not using websockets (ws://) but secure websockets (wss://). So while websockets are new and not too well documented, secure websockets are a complete disaster in that respect.

Setting up a secure websocket with node.js is also very easy; just use wss:// on the client side and on the serverside use:

var http = require('https');

var options = {
      key: fs.readFileSync( 'ssl/socket-key.pem'),
      cert: fs.readFileSync('ssl/socket-cert.pem')
};
var server = http.createServer(options, function(request, response) {
      response.writeHead(200);
      response.end();
});
....etc

Since I am using self-signed certificates I needed to get the user sign the certificate, otherwise the wss:// connection fails silently. So I added an empty HTTP 200 response so the browser would show an unsecure certificate warning. This worked nicely through the company gateways. However, it was still impossible to access the server from the outside, because non of the proxy server we use understand websocket or secure websocket. So while I made a little progress within the different networks of the company I still couldn’t access the socket server from outside the network.

It took me a while, but with the help of this article I figured out the trick. He uses stunnel to unencrypt the secure websocket, but I prefer a one stop shop so: You need to setup a proxy facing the internet using the latest haproxy (with SSL support). Configure it as such:

global
        log 127.0.0.1   local0 info
        maxconn 4096
        user haproxy
        group haproxy
        daemon
        #debug
        #quiet

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        option redispatch
        option http-server-close
        maxconn 2000
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000

backend ws
        balance roundrobin
        option forwardfor # This sets X-Forwarded-For
        timeout connect 86400000s
        timeout server 86400000s
        server ws1 localhost:8080 weight 1 maxconn 1024 check

backend ww
        balance roundrobin
        option forwardfor # This sets X-Forwarded-For
        timeout connect 10s
        timeout server 30s
        server ww1 localhost

frontend https_proxy
        bind *:443 ssl crt /etc/ssl/cert.key_pem
        mode http
        acl is_websocket hdr(Upgrade) -i WebSocket
        acl is_websocket hdr_beg(Host) -i ws
        use_backend ws if is_websocket
        timeout client 86400000s
        default_backend ww

What this does is providing an https ‘server’ on the internet and filtering the wss requests out, unencrypt them and send them to the node.js instance. http://afitnerd.com really did a great job explaining their setup, only two things weren’t that clear. The first is that the client is using the wss:// protocol, so your config looks like this:

var url = wss://host/
var ws = new WebSocket(url, 'connect'); 
ws.onopen = onOpen; 
ws.onmessage = onMessage; 
ws.onclose = onClose; 
ws.onerror = onError;

The second is that the server still provides a non secure websocket service. So you need to use var http = require('http'); and not var https = require('https');. Now we have secure websockets working through all proxies, gateways and routers between you and the ws server, so without further ado:

Please accept the self-signed certificate, use a websocket enabled browser and click here.

OpenJDK7 vs. Oracle JDK7 with Geoserver

Introduction:

After installing a new server with ubuntu 64 I noticed that only openjdk packages are available nowadays. A quick search learned me that, though it is still possible, installing Oracle’s JDK wasn’t advisable. It is hard to keep up to date and people even claim that Oracle’s JDK is a security risk in itself. However the Geoserver developers recommend Oracle (Sun) JDK over OpenJDK. This is a test they did in september 2010 with Sun JDK 1.6 and OpenJDK 6.

Test setup:

Development has continued since then and now Java 7 is available, both in Oracle and OpenJDK flavour. I asked on twitter whether or not it still mattered which version you use. Apparently it is not entirely clear, so I did a quick test with two virtual machines on an ESX-cluster[*] I had handy. One I installed the standard OpenJDK that comes with ubuntu (7u9-2.3.3-0ubuntu1~12.04.1), the other I installed the latest Oracle JDK and JRE (jdk7u9).

On the database server the dutch top10 topographic map was loaded and styled with the SLD from NLextract. I used geoserver’s preview layer function to get an idea how fast the two JDKs are and how the results look. I’ve first zoomed in step by step from the entire Netherlands to village level (8steps) and changed the mapsize to 1280×1024 and zoomed back up. Obviously I had atop running on all three servers to prevent accidentally starting a request before the server were ready.

Results:

This produced the following graph:

Interestingly oracle’s render time tops around 48seconds and openjdk gets a DNF at the lower zoomlevels.In general you can say that Oracle is faster in parsing the millions of features from the database. On more detailed zoom levels the differences disappear:

z0 z1 z2 z3 z4 z5 z6 z7 z8
openjdk 60 41.3 11.97 3.65 1.44 0.552 0.294 0.183
openjdk-2 60 41.35 11.98 3.92 1.35 0.598 0.272 0.213
oracle 48.49 48.32 26.06 8 2.49 0.9 0.363 0.228 0.197
oracle-2 48.32 48.44 27.25 7.99 2.44 0.701 0.384 0.484 0.197
openjdk-large 60 28.47 9.3 3.53 1.62 0.802
openjdk-large-2 28.59 9.22 3.18 1.4 0.784
oracle-large 48.25 48.53 48.84 31.58 11.81 3.75 1.83 1.01 0.721
oracle-large-2 48.48 48.13 48.83 31.43 12.21 3.78 1.85 1.1 0.76

The render quality was on par though, this is with openjdk:

and this with oracle:

On zoom level 2, openjdk:

and oracle:

Conclusion:

This quick and dirty test shows that a reasonably configured openjdk-based geoserver instance is slower than a oracle-jdk based one when rendering a lot of features out of a database. Since the image quality seems to be the same (no labels here) it shouldn’t be too much of a problem though if you use geoserver as a backend of a tiling server. The millions of tiles of the deep-zoomlevels contain less features.

[*] test machine specs, all run Ubuntu 12.04 64bit:
Database server ‘cumulus’:
16GB RAM, 4CPU 2.67GHZ XEON X5650, postgresql 9.1 postgis 2.0, 2GB shared_buffers

Geoserver machines:
8GB RAM, 4CPU 2.67GHZ XEON X5650, tomcat 7, geoserver 2.2.1

 

 

OpenLayers and XKCD

Today Randall of XKCD created a most astonishing comic, which is a giant side view of a world with stuff in the air and underground. Looking at the comic I realised that it was a tiled map, though with a slightly odd tilescheme, in stead of a simple x/y it used a more complex, n/s,e/w starting from the middle, so the centre tile is 1n1e one to the left is 1n2e and one to the right is 1n1w, similar with going up or down.

OpenLayers to the rescue: being open source it is very easy to extend and modify their tiling-schema to work with the slightly weird XKCD system, see it in action here: http://research.geodan.nl/sites/xkcd

The slightly horrible code:

OpenLayers.Layer.XKCD = OpenLayers.Class(OpenLayers.Layer.XYZ, {
	getXYZ: function(bounds) {
		var res = this.getServerResolution();
		var x = Math.round((bounds.left - this.maxExtent.left) /
			(res * this.tileSize.w));
		if(x >= 50)  x = (x - 49)+'e';
		else if (x <50) x = (50 - x)  + 'w';
		var y = Math.round((this.maxExtent.top - bounds.top) /
			(res * this.tileSize.h));
		if(y > 50) y = (y -50)+'s';
		else if (y <=50) y = (51 -y )+ 'n';
	        var z = this.getServerZoom();
		return {'x': x, 'y': y, 'z': z};
	}
})


 

Phoenix is finished

After several years of doing research with touch tables and GIS we’ve finally build an application which is so intuitive that even children can use it:

YouTube Preview Image

This application (Phoenix) is a spatial discussion platform where people can discuss issues while standing around an interactive map. Different ideas we have tried out in earlier prototypes have been polished into a consistent application which is extendible with plugins. I made a teaser movie for those who are not in the neighbourhood to play with it themselves:

YouTube Preview Image

 

 

Geolocation API without maps

HTML5 provides the geolocation api. This is commonly used to move the map to the position of the user. Quite a useful feature, but obviously it is not limited to this. I made a quick HTML5 site which will give the height of you location according to the AHN - the dutch height model.

It is very simple: the geolocation API will provide a coordinate and I use that coordinate to do a getfeatureinfo on the AHN-service of EduGIS. To make it slightly more interesting it will show a cow if you’re above sea level, a nice coral reef if below and a foggy picture if the app doesn’t know where you are, or doesn’t have a height for that location.

Obviously this only works for the Netherlands, since the AHN only provides heights for the Netherlands.

You can find your own height here: http://research.geodan.nl/sites/hoogte/

Timesliding canvas maplayer

After someone saw my BAG building data movie YouTube Preview Image he asked if it would be possible to create an interactive map of the entire Netherlands. This made me think, since creating the movie was a very time consuming action. The problem is that there are about 6 million buildings in the BAG database. This makes the data a bit unwieldy to use directly in the browser. The old fashioned way to do time series on maps involves creating a new layer for each time-moment (year in this case). This would mean that there would be over 150 layers to be loaded on the map and switching between those for the ‘time sliding’ effect. Apart from the hideous task to set up 150 almost the same layers, it would end up with too much images for a browser to handle.

However modern browsers have the <canvas> element. This element allows for the manipulation of single pixels within this element. So I figured if I could encode the building-dates in a PNG and use canvas to display only those pixels which represent a building older than the given date it should be possible to time-slide through the buildings. Fortunately the fancy new mapping library Leaflet.js has a canvas tile layer build in. The BAG data is already available through EduGIS, so I only needed it to encode the data differently in the PNGs.

The encoding is very simple: per pixel there are 4 values available: red, green, blue and alpha. Since I only needed to encode 200-odd values I used just the red value. The years before 1850 are encoded as groups, since the data is so sparse, after 1850 each year is individually encoded. This means that from 1850 onwards the red value increases with 1. The client retrieves the encoded PNGs as normal tiles and look like this:

The image appears grey because I kept the green and blue values of the PNG the same as the red. This image is loaded into canvas and the imageData is retrieved using ctx.getImageData(0, 0, 256, 256) and stored as a jQuery data object on the canvas. This is important, since for visual effect we will manipulate the imageData on the canvas and we want to keep track of the original values. Once the imageData is attached to the canvas the colors are being calculated. It will take the original values, compares them to the current year and will decide whether or not to show the pixel and in which color.

Since only the grey tile is needed, the actual sliding through time is really fast because it doesn’t need to retrieve more data. With 4 bands of 255 values each you can encode an insane amount of data into a PNG, readily available through canvas for direct manipulation. Apart from time sliding, is detailed representation of DEM data an obvious use case.

Dynamic pie-charts

Spatial data traditionally has been displayed using maps and tables. Maps, though good in showing the spatial extent, are not always optimal in showing aggregated data. The human mind can easily be tricked into believing that a bigger country has a bigger share of the pie. Tables show the individual records, but still lack the overview of aggregated data, also tables are notoriously hard to read.

To view the information in an aggregated form one has to build complex queries. These are often slow and do not scale well and the user either has to interpret the resulting map or compare numbers in a table. The first being not very precise the second not very intuitive.

Within EuroGeoSource, a cross European project that allows users to identify, access, use and reuse aggregated geographical information on geo energy and mineral resources, we have come up with a new way. The users are not typical GIS experts and do not have the knowledge to build custom spatial queries. Instead we have determined the most important types of aggregation (eg. by country, by deposit-type etc).

The user can search for one or more commodities. Using MapQuery and gRaphael, an SVG chart library, we than present these commodities aggregated in (pie) charts. He can use the charts as a selection method, for instance by clicking on a country he will see all the occurrences of the commodity in that country on the map.

The result is a fast and intuitive way to search for aggregated data. Providing overview on the distribution of data in multiple domains and still giving access to detailed data in a traditional GIS manner.

Spatial data traditionally has been displayed using maps and tables. Maps, though good in showing the spatial extent, are not always optimal in showing aggregated data. The human mind can easily be tricked into believing that a bigger country has a bigger share of the pie. Tables show the individual records, but still lack the overview of aggregated data, also tables are notoriously hard to read.

To view the information in an aggregated form one has to build complex queries. These are often slow and do not scale well and the user either has to interpret the resulting map or compare numbers in a table. The first being not very precise the second not very intuitive.

Within EuroGeoSource, a cross European project that allows users to identify, access, use and reuse aggregated geographical information on geo energy and mineral resources, we have come up with a new way. The users are not typical GIS experts and do not have the knowledge to build custom spatial queries. Instead we have determined the most important types of aggregation (eg. by country, by deposit-type etc).

The user can search for one or more commodities. Using MapQuery and gRaphael, an SVG chart library, we than present these commodities aggregated in (pie) charts. He can use the charts as a selection method, for instance by clicking on a country he will see all the occurrences of the commodity in that country on the map.

The result is a fast and intuitive way to search for aggregated data. Providing overview on the distribution of data in multiple domains and still giving access to detailed data in a traditional GIS manner.

Wiekwiek

Climate models have several scenario’s to calculate different futures, depending on some starting parameters. The outcomes of these scenario’s can be visualized with maps. Comparing maps of the same variable in different scenario’s is one way to quickly grasp the impact of the starting parameters.

We’ve build a tool for the Surface to browse these different maps with a flick of the wrist: the wiekwiek. You select a variable from the list and a map showing the current situation is shown. By turning the wiekwiek one can display the different scenario’s and visually compare the impact of a scenario.

YouTube Preview Image