Author Topic: Broken proj.4 datum conversions in GISVM (and all Ubuntu/Debian systems)  (Read 3411 times)

tvrusso

  • Newbie
  • *
  • Posts: 4
GISVM's a great idea, but it has one issue that users of NAD27 and NAD83 datums (among others) should know about.

The official Debian and Ubuntu package repositories have a broken package for the cartographic projections library that has been built without including the datum shift grids for US, Canadian, French and New Zealand datums.  This impacts conversion of GIS data between those datums severely.  In fact, if one attempts to use "cs2cs" to do coordinate transformations between NAD27 and NAD83 datums, one finds not only that there is an incorrect datum shift, but that there is no datum shift applied:

   cs2cs +proj=latlon +datum=NAD27 +to +proj=latlon +datum=NAD83

will demonstrate the issue --- type in any lat/lon (such as 106dW 35dN) and you'll find it does no datum shift at all.  Since proj.4 is used by just about every GIS program included in GISVM, none of them will do correct datum conversion between NAD27, NAD83, NZGD49, or other datums that require the datum shift grids.

This issue has been reported in the Ubuntu bug tracker as https://bugs.launchpad.net/ubuntu/+source/proj/+bug/368065 --- and has not been acted on in any version of Ubuntu yet.  This issue has been present for many releases of Ubuntu, and the bug report was opened in April of 2009.

There is an unofficial "PPA" (Personal Package Archive) that one can use to get a fixed Proj.4 package.  The path to this "UbuntuGIS" PPA is in comment #3 of the bug report. It is possible to update the proj.4 install in GISVM from this unofficial PPA.  Follow the link in comment 3 and view the installation instructions.  It is a bit tricky, and I found that following the instructions through synaptic package manager didn't work cleanly -- the Software Sources page didn't seem ever to accept the additional sources, so I had to edit /etc/apt/sources.list directly to get them in there, then follow the instructions from there to get the GPG key.  Once the key was in place and the new source lines added to the sources list, synaptic could find the upgradable proj package.

To summarize the instructions on the PPA web site:  you must add

deb http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu hardy main
deb-src http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu hardy main

to /etc/apt/sources.list, then

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys  123456677
where 123456677 is to be replaced by the actual key value that appears on the PPA page after you click "Technical details about this PPA"

Then use Synaptic Package Manager to reload package data, and you can click "Mark for Upgrade" next to "proj".  Then click "Apply" and the new package will be installed.  Since all the other packages use this as a shared library, all of them will magically be fixed --- QGIS, GRASS, etc. will do coordinate transformations between NAD27 and NAD83 correctly after this procedure is performed.

If you are in an area where datum shift grids are required for datum conversions (such as North America where NAD27 and NAD83 data are still common), using the GISVM with this broken proj.4 package will give incorrect results when attempting to convert data from one datum to the other.

Hope this helps someone.

I really wish the Ubuntu package maintainer would update the package and backport it to Hardy, but it's been months since this has been reported and there's still no movement on it.  Users of GISVM who need correct datum conversion in North America (and other locations such as New Zealand) are left to rely on the unofficial UbuntuGIS software source at this point.

It would probably be a Very Good Thing if the next  version of GISVM were distributed with the updated proj.4 library rather than the broken one.

admin

  • Administrator
  • Jr. Member
  • *****
  • Posts: 72
Re: Broken proj.4 datum conversions in GISVM (and all Ubuntu/Debian systems)
« Reply #1 on: January 06, 2010, 02:14:17 AM »
thank you tvrusso for your contribution!!!!!!!!!!
I've been using proj.4 on GISVM without problems, but I really don't use NAD!!!  ;)
In fact the current cs2cs included on GISVM is using an outdated FWtools version 2.0.6.
http://www.gisvm.com/wiki/index.php?title=GISVM_Desktop_-_R_20081101

Unfortunately Frank has decided to maintain only the Windows version of FWTools (don't understand why? he probably doesn't know about
GISVM ;) )

Since each GIS application installed on GISVM has its own version of proj4, I guess you will not find any trouble on other software.
But don't get it granted, please test it first!

Cheers,
Ricardo Pinho

tvrusso

  • Newbie
  • *
  • Posts: 4
Re: Broken proj.4 datum conversions in GISVM (and all Ubuntu/Debian systems)
« Reply #2 on: January 06, 2010, 06:02:59 PM »
Since each GIS application installed on GISVM has its own version of proj4, I guess you will not find any trouble on other software.
But don't get it granted, please test it first!

Hmmmm.  I'm a little confused by that statement.  The individual GIS applications usually link to the system-wide installation of proj --- at least that is the case for all the GRASS and QGIS builds I've ever done.  You didn't link them statically, did you?  That's the only way that I can think of that they'd each have "its own version of proj4" as you say.

The "proj" package from Ubuntu is installed, and therefore all the other GIS packages should be linking to that broken shared library.  The proj package installs cs2cs --- perhaps the FWTools version is overwriting that particular binary, but it's still using the shared library from the proj package.  When I updated the proj package per the PPA instructions, cs2cs stopped printing incorrect NAD27->NAD83 conversions.

I have not done a rigorous test of all the other GIS packages to see if they were broken before and fixed after my change, but I would bet they are --- when I use Ubuntu to build GIS applications, it has always been the case that datum conversions are broken unless I use something other than the stock Ubuntu "proj" package.  When I get home to the machine where I have GISVM and VMware installed, I'll try checking the binaries to see what shared library they're linked against, and report back.

Unless someone is actually transforming data between NAD27 and NAD83 datums, one wouldn't notice this issue.  Problems come when trying to fit together old data sources in NAD27 with modern data sources, though, so this is a serious issue for North America.

tvrusso

  • Newbie
  • *
  • Posts: 4
Re: Broken proj.4 datum conversions in GISVM (and all Ubuntu/Debian systems)
« Reply #3 on: January 09, 2010, 05:38:04 AM »
Since each GIS application installed on GISVM has its own version of proj4, I guess you will not find any trouble on other software.
But don't get it granted, please test it first!

(...)

I have not done a rigorous test of all the other GIS packages to see if they were broken before and fixed after my change, but I would bet they are --- when I use Ubuntu to build GIS applications, it has always been the case that datum conversions are broken unless I use something other than the stock Ubuntu "proj" package.  When I get home to the machine where I have GISVM and VMware installed, I'll try checking the binaries to see what shared library they're linked against, and report back.


Just checked.  Grass, gdal, qgis, and even the programs in FWTools-2.0.6/bin are dynamically linked, and therefore all use the proj.4 library installed in /usr/lib.  Only the programs in FWTools-2.0.6/bin_safe are statically linked.  These were statically linked with a version of proj.4 that *does* have datum transformation grids installed.  So in the stock GISVM virtual machine, the only versions of any libraries that work properly to transform between NAD datums are those small handful of FWTools "safe" binaries.  All other programs in GISVM outside of the FWTools directory (and the dynamically linked binaries in the FWTools-2.0.6/bin directory) are linked to the broken proj library from Ubuntu.

So North Americans (and New Zealanders) who need the ability to transform data from old datums *must* upgrade their proj shared libraries to the ones in UbuntuGIS and not use the stock Ubuntu libraries.  This is not limited to GISVM --- it is true of all Ubuntu installations, even recent ones using Karmic (Ubuntu 9.10).

admin

  • Administrator
  • Jr. Member
  • *****
  • Posts: 72
Re: Broken proj.4 datum conversions in GISVM (and all Ubuntu/Debian systems)
« Reply #4 on: January 12, 2010, 04:51:27 PM »
tvrusso,

You are correct!
At first I thought the reported error was related to the FWtools pack only. Sorry, my mistake!
As you stated, it is reflected on the Oficial Ubuntu Repository, so it will affect all the Software packages installed from that Repository, namely: grass, gdal, qgis, mapserver, etc and even on postgresql/postgis!

Technically I can't tell what is causing that error (lack of grid data?!) but I have confirmed with other software (namely: FWTools for windows) that the examples presented are giving different results on GISVM.

As you stated, users can test by themselves:

Just enter Terminal and type:

proj
(returns the installed proj version)
Rel. 4.6.0, 21 Dec 2007

Then:
cs2cs +proj=latlon +datum=NAD27 +to +proj=latlon +datum=NAD83
(enter)
106dW 35dN
(returns)
106dW   35dN 0.000
(should return)
106d0'2.079"W   35d0'0.181"N 0.000

And I have also verified the same result on PostGIS!
Just enter PgAdmin and on postgis database run the following script:

select (astext(transform(PointFromText('Point(-106 35)',4267),4269)));
(returns)
POINT(-106 35.00000000088)
(should return)
"POINT(-106.000577500019 35.0000502786653)"

This is a serious bug that must be corrected ASAP!!!
Applying the suggested APP repository will solve the problems identified before.

sudo su
echo "deb http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu hardy main" >> /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu hardy main" >> /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 089EBE08314DF160
apt-get update
apt-get --force-yes --yes install proj
exit



If everything went well you will get:
proj
(returns the installed proj version)
Rel. 4.7.1, 23 September 2009

Then test the previous examples:
cs2cs +proj=latlon +datum=NAD27 +to +proj=latlon +datum=NAD83
(enter)
106dW   35dN 0.000
(should return)
106d0'2.079"W   35d0'0.181"N 0.000

On PgAdmin:
select (astext(transform(PointFromText('Point(-106 35)',4267),4269)));
(should return)
"POINT(-106.000577500019 35.0000502786653)"


But there is still the need to test the rest of GISVM installed software (namly the JAVA ones): Kosmo, OpenJump, gvSIG and Geoserver, etc.
I would appreciate your and other users help to test this issue and point out solutions (I'm very time limited at the moment), so I can build a new GISVM release and replace the current images available for download!

I specifically ask your help for:
- Testing this erroneous behavior on all GISVM software.
- Point out install solutions to correct the error on the erroneous cases!
 
Thank you again for your precious collaboration!
Ricardo Pinho
« Last Edit: January 12, 2010, 04:57:44 PM by admin »

tvrusso

  • Newbie
  • *
  • Posts: 4
Re: Broken proj.4 datum conversions in GISVM (and all Ubuntu/Debian systems)
« Reply #5 on: January 12, 2010, 05:22:27 PM »

Technically I can't tell what is causing that error (lack of grid data?!) but I have confirmed with other software (namely: FWTools for windows) that the examples presented are giving different results on GISVM.


The reason is that the grid data and the proj source code are distributed separately.  To build proj with datum shift grids, one needs to unpack the proj source code, navigate down to the "nad" directory, then unpack the grid data.  Only then should one build proj libraries.  If one skips the unpacking of the grid data, one gets a proj library that cannot do datum shifts for datums that require gridded data (such as NAD27).  Clearly, whoever bundles the proj library for Debian and Ubuntu isn't doing that step.  The folks who do packages for the UbuntuGIS PPA do.

I posted a link to the launchpad bug report early in this thread.  You might want to go there and add your voice to mine, calling for the package to be fixed.  Right now it's uncategorized and unprioritized, although it has at least been confirmed.