Adventures with SD cards in the field

We got some variable results with SD cards in some recent fieldwork – cameras not taking photos as regularly as possible, transfer rates not as advertised – and some research turned up even more disturbing news about SD cards and what to watch out for.

SD cards are cheap because they are fundamentally made of cheap, error-prone memory, and rely on a microprocessor in each card that manages the data and thus what your computer sees. This has huge security consequences. Watch the youtube video in the link.

There are hacked SD cards that are labelled at high capacities, will report a higher capacity when plugged into your computer, but actually have a fraction of that capacity available.

This post will be updated with tools and code for testing SD cards; am experimenting with a few different tools.

Geotagging a folder of images with ExifTool

Sheep and goats from 1200 feet AGL

Cattle, sheep and goats from 1200 feet AGL. Nikon D3100, 35mm lens.

We’re about to start a livestock and wildlife census in the Serengeti ecosystem, and using a point photo count method (APS). This is the first time we’ll have used APS for anything other than wildebeest, and we’re working on managing a different (and much larger) set of data.

Post-flight processing includes geotagging the photos collected based on the tracklog. I’ve used Graphicconverter on the Mac but am trialling ExifTool which will work on Windows as well (though it’s command line).

Update 2016-08-27: see this Github Gist where I’m editing and keeping more notes.

Geotag

To tag a folder of images, open Terminal on Mac/ Linux and enter:

exiftool -geotag tracklog/trk-2016-08-08-PM.gpx digpix/

… slightly differnt for the DOS (command) window in Windows:

exiftool.<strong>exe</strong> -geotag tracklog/trk-2016-08-08-PM.gpx digpix/

Where:

  • exiftool (or exiftool.exe) = the command
  • tracklog/trk-2016-08-08-PM.gpx = tracklog;
  • digpix/ = folder of images.

Geotagged aerial image with bomaResults

Geotagging 1,000 images took about 25 seconds just now. Looks pretty good – Graphicconverter shows the Google satellite image with the photo centre point next to the image, right.

You can see the shape of the old boma (overgrown) and the road. Nice!

 

Customised embedded Google map with KML data

screenshot 2016-02-15 at 17.00.30

  1. Create a track in Google Earth or QGIS and save it as a KML file.
  2. Head to google.com/mymaps …
  3. Click on ‘create a new map’.
  4. Under the ‘untitled layer’ menu item on the left, click ‘import’ and select your KML file.
  5. Change the style of the imported data to suit.

Spatialite-gui for OS X for Spatialite 3.0.1

Spatialite logo

This is an update to a previous post on spatialite for OS X, with a link to the updated spatialite-gui.

GUI tools

There are two great tools for viewing and administering SQlite / spatialite databases – one is even a basic GIS package.

  • Spatialite-gui – create and manage sqlite databases with spatialite support, do queries.
  • Spatialite-gis – view data in the database (change basic appearance / turn layers off and on).

Note: Homebrew for spatialite_gui isn’t working on my OS 10.8 as of 2013-09-18.

Screenshot: Spatialite-gui v1.5 in action

Screenshot: Spatialite-gui v1.5 in action

GIS for Mac OSX

Once, the only serious way to do much GIS work on your Mac was to use ArcGIS (or MapInfo) in a Windows virtual machine … but not anymore.

QGIS

QGIS 2.0-dev on Mac OS X 10.8

QGIS 2.0-dev on OS X – fast & stable.

Quantum GIS can now really compete with ArcGIS in terms of functionality and map creation. I miss a few things – proportional symbols for one – and the map layout for printing is not as easy – but the workflow is much easier, especially when editing shapefiles.

  • Installers for the current stable version of QGIS are maintained at William Kyngesburye’s site including development builds from time to time, as well as the supporting frameworks required to run it.
  • Development builds (v 2.0 Alpha) are available from the Dakota Cartography site and updated nightly. These generally work really well – but many of the plugins no longer function.
  • You can run BOTH the stable and development versions side by side on your Mac – this is what I do, so far without a problem. One will be called QGIS.app and the other QGIS_2.0-dev.app; the only problem going back and forth between them may be compatibility between project file (.qgs) versions.

Installation of either the stable or dev versions requires installation of the supporting frameworks, then a standard drag-and-drop of the app itself:

  1. Download and install the GDAL Complete framework (double click and install the package) – this gives you some required frameworks (applications) including GDAL, GEOS and SQLite (which are all useful in their own right) giving you command line access to those apps as well;
  2. Download and install the GSL framework (double click and install the package);
  3. Download and install QGIS (open the .dmg and drag the app to the Applications folder).
He also has installers for GRASS and PostGRESQL for the mac. I use the GRASS installer but find the Postgres.app installer (see below) far easier.

GISLook & GISMeta

Use quicklook to browse for a spatial file and press space to see a preview of the geometry.

Browse for a spatial file and press space to see a preview of the geometry.

These are plugins for Quicklook and Spotlight for OSX, letting you preview shapefiles and other types of geographic data directly in the Finder or from QGIS while browsing for files.

This is invaluable.

With GISLook you can preview (press the spacebar while browsing) geometry or image data from shapefiles, .e00 interchange files, coverages, .asc grids, DEMs and others (see the home page for a complete list).

GISMeta while show the size of  a raster grid file in the “get info” window in the Finder.

PostGIS

The easiest way to get this going is with the Postgres.app installer, giving you PostgreSQL 9.2 and PostGIS 2.02. Download, drag-and-drop, done.

I have battled with installing PostgreSQL and PostGIS on the Mac with varying results – homebrew or the packages from Kyngchaos can sometimes get it installed, but then the various frameworks or QGIS plugins (Python) don’t play well together.

  1. Download and use the drag-and-drop PostgreSQL installer from Postgresapp.com.
  2. Install PGAdmin to administer PostgreSQL.

R

RStudio

R plays very well with mapping and spatial operations – not good for editing shapefiles, but great for analysis and basic presentation.

You’ll probably want the packages:

  • sp
  • maptools
  • spatstat
  • rgdal

The R Project for Statistical Computing has installers for R on the Mac. For a development environment (including syntax highlighting, version management and export of plots) try RStudio.

What gets funded on Kickstarter?

Kickstarter certainly seems interesting; I’ve been browsing their site (and, in doing so, found two excellent little projects that seemed worthwhile and put some cash towards them) and taking some notes on what seems to succeed and what doesn’t. With all the interest in drones and wildlife survey, I was thinking that it might be fun to get a drone project funded through Kickstarter.

In a later moment of creative procrastination (avoiding all the other projects I have to do), I found a database of all the Kickstarter projects from about a year ago – 45,000 of them or so. Data on category, subcategory, goals, and the funded percentages. Did a little exploring in R, hence the not-very-pretty graphs below.

Continue reading ‘What gets funded on Kickstarter?’ »

PowerPoint is evil. Power corrupts. PowerPoint corrupts absolutely

Is Powerpoint evil? Tufte certainly thinks so, and it’s worth paying attention to genius.

Convert decimal degrees to DMS in QGIS

I often need to provide coordinates to light aircraft pilots – and most of them still use degree-minute-second notation (DMS) for charts and GPS. If we can’t upload flight plans and waypoints directly to the GPS I need to provide tables of DMS points – while QGIS can display in DMS in the canvas, it doesn’t have built-in functions for calculating and filling attribute fields.

A GIS.Stackexchange.com user shared part of the code for this last year, and with QGIS 1.9 the floor() function in the field calculator lets it work properly.

  1. Add your latlong point file to QGIS (or save-as in latlong then open it);
  2. Open the attribute table and start editing, then open the field calculator;
  3. Create a new text field of length 16, named ‘lon_dms’, and enter the code below in the ‘Expression’ box;
  4. Calculate, then repeat for ‘lat_dms’ but change each $x to $y.
  5. Save.

Edit 2017-01-16 – thanks to Skippy for pointing out the southern hemisphere error in the DMS coordinates. Here’s a better version for more general usage, giving N and S labels for latitude:

(CASE WHEN $y < 0 THEN 'S' ELSE 'N' END) || floor (abs($y)) || '°' || floor(((abs($y)) - floor (abs($y))) * 60) ||'\'' || substr( (tostring((((abs($y)) - floor (abs($y))) * 60) - floor(((abs($y)) - floor (abs($y))) * 60)) * 60),1,5) || '"'

Original code:

floor ($x) || '°' || floor((($x) - floor ($x)) * 60) ||'\'' || substr( (tostring(((($x) - floor ($x)) * 60) - floor((($x) - floor ($x)) * 60)) * 60),1,5) || '"'

Input: 123.12345° 

Output: 123°7’24.42″

 

QGIS on an Android tablet

Running QGIS on an android phone worked just fine – but running QGIS on an Android tablet is going to be a lot more useful, with a larger screen. I summarised the steps of installing QGIS on a Galaxy Advance (I9070) in a previous post, under Gingerbread (2.3.6); since then, there have been some issues with QGIS breaking due to Ministro (supporting library) updates, so I went through the installation steps again, this time on an Asus TF300T running Jelly Bean 4.1.1.

Installation

  1. Enable “Allow installation of apps from unknown sources” – Settings|Security.
  2. Download the installer .apk (this download should  be fixed and up-to-date) and open it (on most Androids it will download to a “Downloads” directory, but may be elsewhere; you can just click to open the file from your file browser and it should start the installer).
  3. Run the installer and select ‘download and install’. 83 MB of data will get downloaded. Luckily, it is a resumable download – if it fails, you can start the process off again where you left off.
  4. Confirm that you want to install (standard Android dialogue) – takes about 2 minutes to finish this stage.
  5. Run QGIS: “Unpacking post-install data” … 10 seconds.
  6. QGIS requires a supporting service, available from the Play store “This application requires Ministro service. Would you like to install it?” Confirm and install Ministro II, only 523KB.
  7. Looks like it needs MORE libraries to run! “Qgis needs extra libraries to run. Do you want to download them now?” Confirm and install the QtCore libraries – a lot bigger at 31MB.

Does it work? Sort of …

Sure does … except for the Python plugins, which are not working, it does run and I’ve used it in the field a few times now. The version I’ve just installed, however, won’t let me select the panels to view, which means I can’t access the GPS connection panel to do live tracking. This really limits the utility, but one can pan, zoom and view maps.

Screenshot_2013-03-27-13-33-55Screenshot_2012-08-23-15-13-09 Screenshot_2012-08-23-15-22-14 Screenshot_2013-03-27-13-33-18

Tips and Tricks

  1. Export a QGIS project from your desktop into a single folder + .qgs file using the QConsolidate plugin, then transfer that consolidated folder + project file to the MicroSD card. This will gather all the rasters and vector files into one place for easy transport.
  2. Don’t work with high-resolution rasters unless you’re very, very patient; turn these off or remove them altogether. In the example above, the red ‘slope’ layer was a 25%-scale version of my original (which we just had to show us places we couldn’t fly low-level safely in the Cessna).

Acknowledgements

Thanks again to Marco Bernasocchi for porting and continuing to develop QGIS for Android – if it’s useful, you should really make a donation!

Clean install of QGIS in OSX

This is an update to a previous post of the same name, but with more info on actual reinstallation.

QGIS works really well on OSX – but I have had some odd behaviour after trying out dozens of plugins, or messing with Python or Homebrew installations in other parts of the system. The main problem I’ve found is that plugins will crash or not work as expected, rather than QGIS crashing itself. (I’m also running QGIS-2.0-dev and QGIS 1.8.0-2 side-by-side, which might not be the greatest idea which doesn’t seem to cause problems aside from SEXTANTE compatibility).

A. Remove all QGIS data

If you are experiencing weirdness with QGIS and plugins, try this step first, then restart QGIS. However, to make really sure of a clean start, go to step B.

This will remove all your plugins and preferences! I usually make a list of my plugins before deleting them so I can reinstall more quickly. In the terminal, type:

ls -l ~.qgis/python/plugins/ > ~/Desktop/QGIS_Plugins.txt

You’ll see a new file on your desktop which is a list of all the plugins (or at least their directory names) you have downloaded, whether turned on or not.

Remove all the QGIS data – caches, preferences, saved states and all other hidden application data. Copy and paste each of these lines into the terminal (no sudo needed!):

rm ~/Library/Caches/org.qgis.*
rm ~/Library/Preferences/org.qgis.*
rm ~/Library/Saved\ Application\ State/org.qgis.*
rm -r ~/.qgis/

Restart QGIS and reinstall plugins – keep careful track of which plugins you installed and when problems appear!

B. Reinstall

For a complete reinstall, follow the above steps and then reinstall the application itself.

  1. Follow the steps in A.
  2. Delete your old QGIS – drag it to the trash.
  3. Download the latest frameworks from Kyngchaos.
  4. Install the GSL and GDAL frameworks from the .dmg files you downloaded;
  5. Download the latest (or version of your choice) from Kyngchaos. If you want the dev version, try Dakotacarto.
  6. Install QGIS from the new .dmg.