Last October, Christopher Schmidt mentioned that he was working on a REST-ful Web Processing Server in Python. I don’t recall how push came to shove, but I wound up volunteering to provide vector generalization code for the project. After failing to find any usable Python implementations of the Douglas-Peucker line simplification algorithm on the Internets, I decided to port some C code that I found to Python. I’ve put the result, dp.py, complete with doctests, in the Mapping Hacks code directory, in the hopes of saving a little time for the next person who wants to do line simplification / vector generalization in pure Python. My version of the code is released in the public domain. You can also find a modified version in the WebProcessingServer code repository, and a working demo on Christopher’s website. (And cheers to David Bitner for reminding me about these links.)

Posted in services, software | No Comments »

Updated World Borders Dataset

May 2nd, 2008 by Schuyler

Another one from the “old data made new” file: Bjørn Sandvik of thematicmapping.org has been good enough to make some improvements to our freely available world borders shapefile. (He’s also doing some kick-ass choropleth maps in OpenLayers.) The new and improved world borders shapefile is hosted on his website; we have mirrored it in our data directory.

Here are Bjørn’s improvements…

Read the rest of this entry »

Posted in geodata, data | No Comments »

CivicSpace ZIP Code Database

April 28th, 2008 by Schuyler

About four years ago, Civic Space Labs commissioned me to provide them with a freely available database of US ZIP code centroids. For a while, it was hosted on their site, but at some point the link was broken. Since then, I’ve gotten about an email a month from people looking for the database.

For future reference, the free US ZIP code centroid database is now hosted here on the Mapping Hacks site. From the README:

The ZIP code database contained in ‘zipcode.csv’ contains 43204 ZIP
codes for the continental United States, Alaska, Hawaii, Puerto Rico,
and American Samoa. The database is in comma separated value format,
with columns for ZIP code, city, state, latitude, longitude, timezone
(offset from GMT), and daylight savings time flag (1 if DST is observed
in this ZIP code and 0 if not).

This database was composed using ZIP code gazetteers from the US Census
Bureau from 1999 and 2000, augmented with additional ZIP code information
from the Census Bureau’s TIGER/Line 2003 data set. Timezone information
was added using cartographic data sets from nationalatlas.gov. The
database is guaranteed to exclusively contain information gathered from
sources in the public domain, and thus be legal to redistribute.

The database is believed to contain over 98% of the ZIP Codes in current
use in the United States. The remaining ZIP Codes absent from this
database are entirely PO Box or Firm ZIP codes added in the last five
years, which are no longer published by the Census Bureau, but in any
event serve a very small minority of the population (probably on the
order of .1% or less). Although every attempt has been made to filter
them out, this data set may contain up to .5% false positives, that is,
ZIP codes that do not exist or are no longer in use but are included due
to erroneous data sources. The latitude and longitude given for each ZIP
code is typically (though not always) the geographic centroid of the ZIP
code; in any event, the location given can generally be expected to lie
somewhere within the ZIP code’s “boundaries”.

The database and this README are copyright 2004 CivicSpace Labs, Inc.,
and are published under a Creative Commons Attribution-ShareAlike
license, which requires that all updates must be released under the same
license. See http://creativecommons.org/licenses/by-sa/2.0/ for more
details. Please contact schuyler@geocoder.us if you are interested in
receiving updates to this database as they become available.

Please note that although the database is now nearly four years old, I believe it to still be the most accurate freely available ZIP code database out there. If you’re interested in providing funding for an updated release, please contact me.

Posted in geodata, data, services/geocoder, public geodata | 1 Comment »

On food pricing

April 12th, 2008 by Jo

Each time i read another sensationalising “food crisis” article, I grumble. What else is the blogosphere for, but to get things like this off one’s chest;

So, prices of all kinds of food staples are rising globally, rapidly. Some - like rice - are specifically singled out, and specific warnings actually serve to worsen price inflation by sending those markets panic signals.

The UN’s Food and Agriculture Organisation (FAO) serves as a kind of global food pricing watchdog and they have been making worried sounds for the last couple of years at least. Last month, FAO published in association with the European Bank for Reconstruction and Development (EBRD) a neat short paper on Fighting Food Inflation Through Sustainable Investment.

So far so great and for me this shed some light on why food inflation is happening. Their lightning summary of causes:

On the supply side:

Agricultural subsidies rendering more efficient production elsewhere, unprofitable
Lower stock levels due both to increasing cost and change in stock management practises
Rising fuel costs

On the demand side:

Increasing demand for, and shipping cost of, grain as animal feed in “emerging markets”
Impact of biofuels/agrofuels on grain supplies for food consumption
Speculation, reflecting futures in current prices.

Yet the main thrust of this document barely seems credible to me. It seems inarguable that government policy of tinkering with export quotas and price controls is worsening the macro problem. Certainly if other FAO assessments are to be read right, food inflation turns into a “crisis”-like situation at around the same time as policy measures start to drastically restrict exports in anticipation of a future “crisis” - a kind of society-level hoarding.
One minute, the EBRD offers an acknowledgement that large offshore “agroholdings” businesses owned by private funds are moving in on marginal agricultural land which has become profitable again with increased prices - in places like the Ukraine and Poland. Next minute, there’s talk of “farmer access to credit”, as if the grass-chewing family farmer is still a figure who exists in reality. What the EBRD proposes are government measures designed to indirectly subsidise, rather than directly subsidise, massive agribusiness by (re)building transport infrastructure for long-distance shipping and large-scale storage. Credit facilities for farmland-based mortgages that will collect the equity in the land into ever fewer hands. The response is essentially “throw more hardware at the problem”, cross fingers, and rely on regulation to keep papering over the cracks.

There are serious historical precedents for great infrastructural change simultaneously in both agricultural production, and transportation technology. These changes both corresponded to and helped cause large-scale social change. The late eighteenth to early nineteenth centuries, canalization and the railways came along with a change in land management, enclosure, irrigation practises - an Agricultural Revolution to complement the Industrial one.

I suppose there was a corresponding set of changes after the Second World War, with containerisation, motorway transport, and mobile refrigeration alongside combine harvesters, effective pesticides, vast landholdings and the rest of industrial agricultural practise. This was also called a Green Revolution, a term that doesn’t work so well today.
So it makes sense that we should be looking for this pattern of change to happen once more. Some radical change in agricultural practise seems to be the only way out of this series of interlocking causes of rapid and socially unsustainable global food price inflation. Look at the FAO/EBRD six causes again; one of those is rising fuel costs, and another three are directly caused by rising fuel costs.

The nicest explanation of this I have seen was produced by Caroline Lucas MEP back in 2006, when food price inflation seems to have become really noticeable to the specialists. Her report, Fuelling a Food Crisis, works on the theme of Achieving food security in an era of peak oil is an urgent political priority. This will involve a more towards more low-energy and low-inpit farming, and the development of more localised markets.

(Hooray for Caroline Lucas MEP! Years ago, when Mapping Hacks was still in production and before the INSPIRE Directive had its first reading, she was the only South-East MEP who wrote back to me with an opinion on INSPIRE. We sent a complimentary copy of the book to her office when it came out; I wonder whether she ever got to look at it.)

Whether or not one buys into the discourse of “peak oil”, higher energy prices are both here to stay and are in broad side-effects socially desirable, if energy price dependence of food prices can be addressed. Reformation of agricultural and transportation technology, hand-in-hand, seem required to fix the underlying problems of food pricing. Further movement along the path we’re now on, hoping the faltering state apparatus will continue to be able to pick up the pieces, is building a road to nowhere.

The only thing i knew about FAO up til now was that they support and develop the fine open source GIS project GeoNetwork, which began as a FAO project. It’s a java-based package for cataloguing and publishing collections of geographic data in a range of legally-mandated formats. Through the UN they promote GeoNetwork to organisations like CGIAR to publish increasing amounts of their fieldwork data in sustainable agriculture, nearer to real time.

FAO also collects a lot of interesting historical and comparative pricing and trading data at FAOSTAT. What is published here only goes up to 2006, when the first signs of global inflation across all food types started to kick in. FAO’s more recent reports have statistics up to 2008, and while the FAOSTAT interface will generate nice tables and flash maps, the Open Knowledge Foundation’s rallying cry comes to mind: “We want the data, we want it raw, and we want it now!”

Update… I’m told that the FAOSTAT core data is now undergoing substantial update and review, after which more recent data will appearing through their web service interface. There are substantial archives through which one can “drill down” to more report data going back to the early 60s’. I still suspect it to be the tip of the iceberg of what’s available in FAO archives, though.

Posted in data, book, economics, permacrisis | No Comments »

OSGB Business Model Bashed By Gov’t Sponsored Report

March 13th, 2008 by Schuyler

Well, “bashed” in the politest, most scholarly possible way, but bashed all the same:

A report commissioned by Her Majesty’s Treasury, entitled Models of Public Sector Information Provision via Trading Funds, has been published on the official website of the Department for Business, Enterprise, and Regulatory Reform, a UK government watchdog agency. In British government parlance, a “trading fund” is a government agency with a remit to operate on a “cost recovery” basis - meaning that the agency is expected to make back some or all of its operating costs through licensing fees and other charges. As most of you probably know, the Ordnance Survey of Great Britain, which collects and provides all of the public geographic data for the UK on a monopoly basis, is one of these.

The 154 page economic report, linked from the Open Knowledge Foundation’s weblog, was written by several Cambridge University professors, including Rufus Pollock, now the Mead Fellow in Economics. Its main thrust is summed up by this juicy quote from the executive summary:

This study has analyzed the impact of adopting different models for the provision of public sector information by trading funds…. Performing this comparison on the subset of products suitable for analysis, it was found that, in most cases, a marginal cost regime would be welfare improving – that is, the benefits to society of moving to a marginal cost regime outweighed the costs.

(The emphasis above is the original authors’.) For those of us unfamiliar with the term “marginal cost”, Wikipedia helpfully defines it as “the change in total cost that arises when the quantity produced changes by one unit” - which is, in this case, the cost of distribution, since what UK trading funds like the Ordnance Survey are really selling is information. As anyone familiar with Coase’s Penguin already knows, and as the authors of this paper conveniently remind us,

The digital nature of these products has implied marginal costs of approximately zero and hence the marginal and zero costs regimes were essentially identical.

Although many of the figures have been redacted as “CONFIDENTIAL” from the details of the report (why?), the analysis section on the OSGB still has this to say:

… a change from an average cost to a marginal cost regime would increase welfare. Specifically, gross benefits would be around £168m a year while net costs to government would be around £12m. Overall this implies an overall net benefit to society of 156m.

The practical upshot is that we finally have a compelling, reasoned, and - more to the point - government sponsored economic analysis which tells us what we suspected all along: the British economy will be much better off when the OSGB is funded by the Treasury to give away at least some of their “unrefined” data - better off, in this case, to the tune of over 100 million pounds annually. Unless I’m mistaken, this figure leaves out the intangible benefits of making this data available to non-profit public service organizations doing environmental work, health studies, poverty studies, community service work, and so on.

Question is, what’s the next step to making it actually happen?

Posted in public geodata, open knowledge, economics | No Comments »

A credit for a name

February 28th, 2008 by Jo

The GeoNames blog carries news of improved commercial service with performance guarantees. The flipside, of course, is a limit on requests that can be made to their free services.

The limit is set high enough that it should only affect those making heavy use of the GeoNames web services. They’ve come up with an interesting credit system to price and meter usage. Requests that are “cheap” in terms of the processing load on their systems cost a credit or less. More “expensive” requests such as reverse geocoding, the contents of a spatial envelope, or annotating RSS feeds with GeoRSS information, cost 4 of these credits. As an anonymous user, you get 50,000 credits per IP address per day, which will take some using up. Otherwise a pay-as-U-process scheme along a faintly GRID-like lines.

It’s high time GeoNames got an income from the service that it provides. Web-based applications like the happening Dopplr build up using the components that GeoNames, Flickr et al provide on a free basis, creating positive feedback. There’s your “rising tide that raises all boats”, but longer-term we can’t rely on very large companies to support the components as loss-leader. Instead smaller companies can trickle income through to each other, and I’m glad there’s a place for GeoNames in that picture.

The data underlying GeoNames is still available on a free-to-reuse basis with a CC-BY license - on sale are the smarts and the quality of service. A culture of open data serves to create business, not to undermine it. The GeoNames team haven’t gone all the way, as geocoder.us did, and released the full set of software components as open source, along with the data. That didn’t hurt the sale of commercial services, may even have helped promote them. But is open source essential to an open service? No-one knows yet, but GeoNames seem to be doing a decent job of walking the line.

Posted in geodata, services/geocoder | No Comments »

Using GDAL to make little images from big ones

February 19th, 2008 by Rich

A member of the Geowanking list asked for advice on pulling 700×700 element chunks out of a 50,000×50,000 element raster file.

And doing it in under 10 seconds, please. That brings up the magical GDAL tools. The Geodata Abstraction Library. This is a set of libraries and command line tools which let you do just about anything you want with most any Raster (and with the included OGR tools Vector) data format you might find.

I’ve been using the GDAL tools to manage the large TIFF images I’ve been generating with the Gigapan Panoramas which I’ve been taking.

The Gigapan project treats gigapixel and up images the same as mapping tiles, and uses a slightly modified version of the Flash Earth Browser by Paul Neave to let you pan and zoom in a large tiled panorama the way you navigate map tiles.

I’ve been spending a lot of time on Gigapans, and I’ll write more on them, and their intersections with mapping, later.

But back to the problem at hand…Using a 1.83 GHz Intel Core Duo Mac Mini with 1 GB RAM the gdal_translate utility grabbed a
700 x 700 window from a 63434 x 11679 tiff and wrote it out to a jpeg in about 5 seconds.

Example: to grab a 700×700 pixel window from near the middle of a tif image and write that window to jpeg:
gdal_translate -srcwin 31000 5000 700 700 -of jpeg p2.tif out.jpg

To shrink an image to 5% of original size (took about 13 seconds)
gdal_translate -outsize 5% 5% -of jpeg p2.tif out.jpg
GDAL tools are at: http://gdal.org. You can download them as part of the precompiled binary FWTools at http://fwtools.maptools.org/

Posted in Uncategorized | No Comments »

Free Map India, 2008

February 14th, 2008 by Schuyler

I have been perhaps a bit remiss in not mentioning this here sooner, but Mikel Maron and I are currently engaged in facilitating a series of mapping workshops all across India, informally known as Free Map India 2008. So far we have met with Indictrans in Pune, and organized two-day workshops focused on seeding OpenStreetMap communities in Mumbai and Trivandrum. Tomorrow, we leave for Bangalore for another workshop, and then, in rapid succession, we’ll be visiting Ludhiana, Delhi, and Calcutta before the end of February.
Read the rest of this entry »

Posted in collaborative mapping, events, talks, openstreetmap | No Comments »

Galileo: journey to where?

December 1st, 2007 by Jo

It sometimes seems as if I’ve been reading the same Galileo article on the BBC News website for the last three years; only the delivery date changes each year. I missed this summer’s news of the collapse of the “consortium of consortiums” that was selected to build, launch and run the European-backed alternative to the GPS system. Now, I read that Transport Ministers representing each nation in the Council, have rushed through approval for Galileo’s build and launch to be funded outright by the European Union, e.g. taxpayers in the member states - moving 1.7bn euros from other, underused EU budgets.

Meanwhile, the US’s higher-accuracy GPS III without selective availability is planned for 2013; the Chinese space research agencies are building yet another global positioning system, and India and Japan are working on regional ones. There’s serious doubt as to whether the costs of Galileo will ever be covered by charging for privileged kinds of access (higher availability, guaranteed uptime) when there are so many alternative systems planned. I learnt all this from the terrific report into the background of this Galileo funding decision by the Transport Select Committee in the UK Parliament. It talks about how the techno-political context has changed since the Galileo project was begun in 1999 with a projected launch date of 2008. There’s too much detail to summarise, but I’d heartily recommend it as reading material to those who want to understand what’s happened, and likely will happen, with the project. The report’s language is emotive at times, perhaps with reason as the conclusions are saddening.

“We fear that Galileo’s status as a flagship grand projet is clouding the judgement of some in relation to its true, realistic and proven merits… No amount of vague and euphoric anticipation of enormous economic benefits can make up for rigourous and balanced analysis of costs and benefits.”

A network of regional systems, or a UN-led global agreement to build and maintain a shared global system, would seem to make sense for a positioning network; and I wonder why these efforts aren’t visibly happening.

Posted in planning, services | 2 Comments »

gvSIG, below the water line

November 20th, 2007 by Schuyler

Last week I had the distinct pleasure of attending las 3es Jornadas gvSIG, the third annual gvSIG conference, in Valencia, Spain. gvSIG, as you may know, is a (primarily) desktop GIS system written in Java. The project was initiated in 2003 by the Conselleria d’Infraestructures i Transport for the Generalitat Valenciana, the provincial government of Valencia. The gvSIG project started incubation in OSGeo this September.

gvSIG is an interesting case in the OSGeo world as it is (to my knowledge) the only Open Source GIS project where the majority of the project development is done in Spanish, and not in English. This has created a situation for the majority Anglophone OSGeo community with regard to gvSIG that is not unlike observing the proverbial iceberg: Most of us English speakers have seen only the bit that sticks up above the linguistic water line, and are unaware of the vast amount of effort and collaboration that has gone on underneath.

Read the rest of this entry »

Posted in community, software, osgeo, software/gvSIG | 2 Comments »

« Previous Entries

The Authors

Subscribe

The Contributors

Links

Mapping Hacks

by Schuyler Erle, Rich Gibson and Jo Walsh

Douglas-Peucker Line Simplification in Python