Some updates on OIA requests currently underway for various geospatial datasets. Note the Official Information Act allows 20 working days to fulfil a request.
- Department of Conservation – national track database. 29 days since I filed my request. They responded on 27 March with a ”We’re processing your request” and I haven’t heard from them since. I sent a follow-up email today and received an automated reply that my DoC contact is out of the office until next week. That puts them well on the far side of 20 working days.
- Ministry of Education – school zones. Request filed 21 March. No communications received at all.
- Fire Service – suburbs. Request filed 21 March. I’ve received two responses so far, from two different employees. The first asked ‘What I intended to use the data for’ and specifically if I intended to upload the data to Koordinates, to which I replied the OIA does not require the requester to state a purpose. The manager (?) of the first employee then responded that I could have the data for free if I signed a license agreement with the Fire Service. I responded that the OIA doesn’t make any allowance for license agreements, and would they please point to which clauses of the Official Information Act they think apply. I have yet to receive a third response. Here is a copy of the License Agreement they wanted me to sign:
Their agreement isn’t as restrictive as most, but that’s not the point. Surely a Government agency believing it can require acceptance of an arbitrary license agreement before fulfilling an OIA request defeats the entire purpose of the Official Information Act.
So to date, the Ministry of Health is the only Government agency to fulfil an OIA request for geospatial data quickly and without any significant problems.
You fail describe the full details about the datasets and reasons for you wanting the information. Are you merely a smart ass trying to shit stir and promote yourself?
Can you rock up to Telecom and ask for their exchange details and areas that are covered by each exchange? Can you ask Vodafone for all their cell-sites and relay system for their network? If you can then that is really pushing the reason for the Official Information act to the extreme.
NZFS, which is not a Government Agency, has built the localities Dataset in partnership with two commercial organizations who have both invested significant time and effort into producing a dataset that suits the requirements of all parties involved and is done so with the best interests of the New Zealand public. This dataset is used by over 50 organizations who have happily signed the agreement. The intent of the agreement is to guarantee that only the latest versions are ever displayed to the public, which I see from your site is something that you have very little interest in achieving. Request for changes are received from any licensed user and most organizations have a mechanism for collating public feedback for changes. The boundaries can be viewed on many websites.
So all I can deduce from your request is that you are a money grabbing POS who wants to on-sell the public data you so religiously collect.
Charging over $1,000 for LINZ address points is daylight robbery, you can get the entire LINZ cadastre for a few hundred dollars.
Your Topographic Rasters are out-dated and ugly.
There are websites that supply the current scanned NZMS260 maps for free. Also anyone can access Version 15 of the topo data on-line for free – I note that you are still selling Version 14.
I believe you have a problem with the OIA and not these agencies, and if the OIA gives you have the right to waste peoples time like this – then I agree with you on something.
Karl, I’m not sure that Vodafone and Telecom are covered by the OIA so your analogy is slightly misleading. In saying that however there are many ways to work with the agencies covered by the OIA and maybe a slightly less controversial approach may have been better in this case? I’m not one to judge but I think there are lessons to be learned on both sides here. Coming from a person who consumes data it is sometimes hard to find out where to even start trying to gather the required data. Here are a couple of things that the NZFS could do to promote responsible use of the data:
- Make sure that http://www.fire.org.nz/data is a page which exists (it currently doesn’t).
- On this page outline the source data sets that are available from NZFS
- Allow users to download the data sets directly (for example the source material for reports such as http://www3.fire.org.nz/cms.php?page=18326)
- One of the outcomes from the SSC Open Data Workshop was that the use of licenses was restrictive but as you point out the main reason for the license is to make sure people are using the latest version. Having a single url for this data set that is available to all who want it would go a long way to making this possible. If as a data consumer I can hit a url and get a last updated time stamp then this would make it trivial to check for updates.
- If you must have a license agreement you could make this a clickwrap agreement on the NZFS website so that it can pass the The 3am Factor.
I would also like to make a few comments on other points you raised:
> This dataset is used by over 50 organizations who have happily signed the agreement.
Thats great – The NZFS suburb list is by far the most comprehensive and accurate. The more people using this set the better.
> The boundaries can be viewed on many websites.
Can you point to a website with the boundaries marked? I would be interested to see how this is being used outside of the NZFS.
> Charging over $1,000 for LINZ address points is daylight robbery, you can get the entire LINZ cadastre for a few hundred dollars.
I believe that data set is being sold by Oliver and Co if this is the one you are talking about?
And for clarification/disclaimer I’n not from Koordinates but I know them fairly well from the Auckland Web Meetup and the Auckland Start-up community.
As a closing point for any other organisations that would like to open up their data Zoodle is keen to work with other relevant agencies to help with best practice in getting this information online. Zoodle as a webiste is working hard to provide relevant information for property buyers but we can only do that if we can get reliable, accurate and open information from various sources.
Glen
All government departments cost money to run.
I pay rates to my local council. They use this revenue to provide MANY services to the ratepayer.
I do not believe that supplying data to you benefits me as a ratepayer in anyway.
I completely agree with Karl that you are choosing to interpret the OIA for your own financial gain.
And I completely sympathise with the poor sods that get landed with your vaguely emailed requests for information!
It may be easy to disseminate the information to you, but anyone worth their salt in this industry knows that CONTEXT is King.
And data is worse than worthless without CONTEXT. You have the potential to cause great harm by providing data without it for your own financial gain.
” A little bit of knowledge is a dangerous thing ” ~ Oscar Wilde
So they (Zenbu and Koordinates) expect councils and govt departments to spend huge amount of resources to create data for them (to their requirements!) so they can comercialise it… tell them they’re dreamin’.
If the market demands it the market will come up with the money… The free ride for Ed, Sam and the like is over… If you wan’t quality data expect to pay for it chaps. You need to take your bisiness model back to the drawing board.
Looking forward to your OIA request… you’ll get a very quick answer from us. Two words. one of which will be ‘off’.
Oh and before slagging of someones data as ‘poor quality’ actuallity learn a bit about what data quality means… it may be crap to you but that is what fitness for purpose means.
@Shelley
> I pay rates to my local council. They use this revenue to provide MANY services to the ratepayer.
And therin lies one of the issues. The councils/Goverment should be providing services that we can’t. They should let others create value out of the data. I strongly recommend reading and listening to the workshop on open data here:
http://blog.e.govt.nz/index.php/2009/03/09/perspectives-on-open-data-workshop-on-the-re-use-of-government-held-non-personal-data/
This workshop gave some great insights into open data.
> I do not believe that supplying data to you benefits me as a ratepayer in anyway.
I would disagree. Take our site zoodle.co.nz for example. We pull data from several sources and present this to the user to give them a coherent view of a property and its community. It saves you (The house buyer in this case) time and effort in going to several different websites.
> It may be easy to disseminate the information to you, but anyone worth their salt in this industry knows that CONTEXT is King.
Which is what we try and do at zoodle.
> And data is worse than worthless without CONTEXT. You have the potential to cause great harm by providing data without it for your own financial gain.
But who is the arbitrator of what is the right context? Put another way if a person mashed up two data sets which showed a department in a negative light is this not in context?
@fred
> So they (Zenbu and Koordinates) expect councils and govt departments to spend huge amount of resources to create data for them (to their requirements!) so they can comercialise it… tell them they’re dreamin’.
Actually the amount of resources should be less overall. If the departments provided the data in open formats then this can be used both internally and externally to build services and tools for users. The benefits for the community far outweigh the costs.
> If the market demands it the market will come up with the money… The free ride for Ed, Sam and the like is over… If you wan’t quality data expect to pay for it chaps. You need to take your bisiness model back to the drawing board.
Ah, I’m sorry to say but the US is moving towards a far more open data strategy. Data is only becoming more open.
> Looking forward to your OIA request… you’ll get a very quick answer from us. Two words. one of which will be ‘off’.
Mind putting a url behind the name? We all have an opinion and I think it is important to discuss this in an open manner. I personally differ from Ed’s approach in that I am trying to take a more partnership approach in working with Government departments in opening up the data.
I’m not sure what the issue is – I personally want to work with the departments to get access to the data so we can provide innovative, interesting services to our users but we can only do that if we get free access to government data (which our users have already paid for via taxes).
I’m interested to see what the problem is with providing open access?
I work for a not for profit organisation that does public good research. A large part of my job involves acquiring GIS data from governmental and other organisations. I absolutely dread this. Why? I can never be sure how my requests for data will be received. Many organisations are lovely to deal with and very helpful
and friendly. Sometimes it is even a quick and painless process to get data.
However, at other times I feel like I am being a pain and that they resent me for bothering them and that I would bother them less if a) I was important or b) I knew them personally or c) I had large sums of money to buy the data. I don’t mind paying what it costs for someone to supply the data to me, but sometimes the data costs a lot
more than that. I don’t have a budget to pay thousands of dollars for basic GIS data that I know already exists and was paid for by the rate/tax payer.
And sometimes I have to go through long painful drawn out processes and meetings trying to justify why I am deserving of the data. Sometimes it feels like an inquisition. Sometimes it feels like a test and that I will only get the data if I say just the right words in just the
right way. Sometimes it is just annoyingly time consuming. And a few times it has been very upsetting.
I have considered submitting OIA and LGOIMA requests but have never done it because I was scared I’d be blacklisted from the GIS community and people would be nasty and that I’d ruin any chances of obtaining data ever again. Before today I thought I was just being chicken for no good reason.
The reason I like Koordinates is that I don’t have to go through the time consuming traumatic hoop-jumping process required to acquire data. And, more importantly to me, I don’t have to risk feeling like crap just for trying to get access to GIS data, which after all, is not worth anyone getting their knickers in a twist over.
Both Zenbu and Koordinates do not commercially gain from data obtained under the OIA from government departments. On both sites it is (or will be once available) supplied to the public free of charge, without any need to sign a contract or wait 20 working days. When such information is readily and freely available the world can simply get on with doing more productive things, like Zoodle.
The requested data already exists.
Government has already spent the money and resources creating it.
How can it possibly benefit anyone if it is not available?
In addition, none of these requests should be onerous in any way, they are specifically worded and supplying a digital file or extract should be quick and simple; for the most part in my experience, it has been.
The purposes of the OIA are
(a) To increase progressively the availability of official information to the people of New Zealand in order—
(i) To enable their more effective participation in the making and administration of laws and policies; and
(ii) To promote the accountability of Ministers of the Crown and officials, and thereby to enhance respect for the law and to promote the good government of New Zealand:
This blog points out the lows (and highs) of working through OIA requests. Through this I hope we can promote accountability and encourage increased availability of information.
(P.S. The suggestion, from an implied (local?) government employee, that they would supply a two word “___ off” response to an OIA request is simply flabbergasting.)
As I was the one that appeared to start this little ‘shit storm in a blog post’ via my reply to the ESRI User Group list, allow me to make some points as well.
@those thinking that it is all about the money – if you look closely as Koordinates, you may notice that it is in fact a marketplace AND repository for spatial data. There is a pile of free data on there as well, and this is where the OIA-requested data has been going. Other commercial entities have submitted paid-for datasets. As far as I know the two are different, so the OIA datasets are NOT being used for commercial gain. Ed may like to clarify this.
re: commercial datasets – no you can’t rock up and get commercial datasets, but that is no reason that publicly-funded data sets shouldn’t be easily accessible (as per the Geospatial Strategy)
re: Localities dataset – I’m one of those 50-odd entities that has licensed the dataset. It is a restrictive licence, as in its current form, it does not allow the data set to be packaged and distributed where the info may ‘escape’ – hence it is only useful for web-based services that don’t publicly expose the data, or those that can ensure the data is updated in a timely manner. This is not always a realistic assumption. Fine for emergency services, but real users don’t always have to implement every data update. I talked about this with Mike Judd before he moved on and tried to encourage the NZFS to release the data under a less restrictive licence – clearly I’ll have to restart that conversation with his superior. It is also about creating low barriers to entry – yes, I’m saying the licensing barrier needs to be even lower. Make NZFS CreativeCommons v3 No-Derivatives/Attribution Required. Use a well recognised license rather than writing one that requires paid legal advice before commencing usage of the data.
@Karl – pull your head in and have another close look at Koordinates – can you see any of the OIA datasets actually being listed for sale? Or are the commercial datasets perhaps being sold by other data commercial data suppliers that are using Koordinates as a listing service?
@Shelley – I don’t believe the public sector should be competing with commercial vendors for data, so that rules out councils being able to charge for data. But, for an example of how open data can benefit ratepayers… some of the data obtained has been restaurants, and these have been added to the FREE and OPEN Zenbu POIs. Tourists download these points-of-information into their GPS before coming to New Zealand, and then use them to find restaurants to dine at whilst travelling. You own a restaurant, your business is a ratepayer, you benefit from making the information freely available, and making it easier for tourists to find and navigate to your restaurant. Of course, people like I will go out and collect the data ourselves, place it in Zenbu or OpenStreetMap and this will probably destroy any potential data income that a council hopes to receive anyway. At which point the councils may as well release the data anyway as they won’t recover costs anymore. So, we should just cut to the chase and start making the information easily accessible now.
@Fred – likewise, you should look again at how Zenbu and Koordinates operate. The OIA’ed data is freely available, and conceivably could be forked by the community and maintained elsewhere – as long as the appropriate licensing conditions are followed, which in most cases is just correctly attributing the sources of data.
re: the OIA – I think you folk have missed the point entirely. Have you ever considered why they are standing up and publicising the use of the OIA to obtain spatial data?
1. People like me (and others) have tried to obtain geospatial data using ‘nice’ requests, explaining what we’d like to use it for, and clearly explaining that it is not for commercial purposes.
2. We have tried to encourage positive change by promoting the benefits that will flow from increased public access to publicly-funded geospatial data.
3. The NZ Geospatial Office has not been able to get some early wins in getting more public data released, and more vocally championing open geodata.
These mostly failed – other than a few shining lights here and there. Even Ed highlights MOH that has done a stunning job in releasing their boundary dataset. It is a pity that we don’t have a few more far-sighted organisations like them that are willing to better contribute to the public geospatial commons in NZ.
And need I remind you of the irony of OIA requests? Most public sector organisations treat every request, no matter how simple, as an OIA request. So in some cases we cannot even ask nicely, or open a discussion. Organisations automatically treat a request as OIA and go on the defensive. So, like it or not, and whether we request it officially under the OIA, or just ask (without referencing the OIA), it will always be an Official Information Act request.
I’d love to see more organisations being proactive and making their geodata more available, and under commonly used and clearly understood licences. Get the SSC involved – they have some smart people that can make recommendations in this area. But don’t give use a slightly different licence for each agency we deal with.
Interesting debate – Fred and Karl you may have the wrong impression that geospatial data ‘owned’ by the crown is best left unavailable and out of reach to the public.
The project that I started several years ago produces GPS street maps of New Zealand – I wonder how many tourists we have helped, or local users with GPS maps that are totally free and more up to date than the commercial companies can provide. Approx 200,000 downloads to date. The data we produce the maps from has come from many sources (including Zenbu) and contributions, maintained by a core group of mappers. We don’t do it for the money, but for the the community.
Our ‘source’ data that we have gathered is freely available in a couple of different formats. We know the benefit of sharing geospatial data.
Yes I also pay lots of tax, rates and vote.
Graeme
I fall into the ‘we have already paid for the data as taxpayers’ category and I think the service that Koordinates provides (both for chargeable and free data) is absolutely fantastic!
As an ex-regional council employee, yes, I did get a bit annoyed sometimes at having to constantly burn/send/print/compile data for various customers (mostly customers who didn’t actually know what they wanted)…..so surely the answer should be: Don’t do it!
But here is where there are two options. 1) Don’t do it all/Make it extremely hard for people to get it (as is currently the case it seems) or 2) Let someone else do it!
If the various departments/councils that have this data and are getting swamped with requests for it, then it is logical that if you only have to supply the data ONCE (to a distributer) that is better than having to supply it multiple times (to each individual request)….not to mention needing to store the subset for a while incase the customer loses the data/the disc wasn’t burnt properly etc etc
If the various departments/councils that have this data are getting swamped trying to pre-process all the data for each request, then it is logical that if you only have to pre-process the data ONCE (i.e. to remove any sensitive attributes) and supply it, that is a better option- let the customer do the work!
Good work Koordinates- Finally I have somewhere to get good quality spatial data easily, free/cheaply and quickly….afterall, as a taxpayer/ratepayer I don’t think I should have to pay for it twice if I don’t have to.
Cheers
I’m going to go on the record and say that I intend to upload as much of Northland Regional Council’s data on Koordinates as possible. It seems a logical choice to me.
It benefits people like shoresuze because they won’t have to be afraid to ask for it.
It benefits me because I won’t have to do multiple data requests for the same dataset.
It benefits the ratepayer because they don’t have to pay me to extract and send out the data multiple times.
Data provided under the OIA may still have copyright (Crown or regular) so check before you share or re-publish.
On the other hand, if the data was provided by an agency with a CC licence, the process would be made much simpler.