Alex ZvoleffJekyll2019-10-04T23:53:43+00:00http://www.azvoleff.com/Alex Zvoleffhttp://www.azvoleff.com/azvoleff@conservation.orghttp://www.azvoleff.com/articles/glcm-rotation-invariant2014-10-21T00:00:00+00:002014-10-21T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>This post outlines how to use the <code class="highlighter-rouge">glcm</code> package to calculate image textures
that are direction invariant (calculated over “all directions”). This feature
is only available in <code class="highlighter-rouge">glcm</code> versions >= 1.0.</p>
<h2 id="getting-started">Getting started</h2>
<p>First use install the latest version of <code class="highlighter-rouge">glcm</code>, and the <code class="highlighter-rouge">raster</code> package that
is also needed for this example:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">install.packages</span><span class="p">(</span><span class="s2">"glcm"</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Installing package into 'C:/Users/azvoleff/R/win-library/3.1'
## (as 'lib' is unspecified)</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## package 'glcm' successfully unpacked and MD5 sums checked
##
## The downloaded binary packages are in
## C:\Users\azvoleff\AppData\Local\Temp\Rtmp2j8wNL\downloaded_packages</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">glcm</span><span class="p">)</span><span class="w">
</span><span class="n">library</span><span class="p">(</span><span class="n">raster</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: sp</code></pre></figure>
<h2 id="calculating-rotationally-invariant-textures">Calculating rotationally invariant textures</h2>
<p><code class="highlighter-rouge">glcm</code> supports calculating GLCMs using multiple shift values. If multiple
shifts are supplied, <code class="highlighter-rouge">glcm</code> will calculate each texture statistic using each of
the specified shifts, and return the mean value of the texture for each pixel.<br />
In general, I have not found large differences in calculated image textures
when comparing GLCM textures calculated using a single shift versus calculating
rotationally invariant textures. However this may not be the case for images
with strongly directional textures.</p>
<p>To compare for a sample cropped out of a Landsat scene, use the L5TSR_1986
sample image included in the <code class="highlighter-rouge">glcm</code> package. This is a section of a 1986
Landsat 5 image preprocessed to surface reflectance. The image is from the
<a href="http://www.teamnetwork.org/network/sites/volc%C3%A1n-barva">Volcán Barva TEAM
site</a>.</p>
<p>When <code class="highlighter-rouge">glcm</code> is run without specifing a shift, the default shift (1, 1) is used
(90 degrees), with a window size of 3 pixels x 3 pixels. Below is an example
from running <code class="highlighter-rouge">glcm</code> with the default parameters:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">test_rast</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">raster</span><span class="p">(</span><span class="n">L</span><span class="m">5</span><span class="n">TSR_1986</span><span class="p">,</span><span class="w"> </span><span class="n">layer</span><span class="o">=</span><span class="m">1</span><span class="p">)</span><span class="w">
</span><span class="n">tex_shift1</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">glcm</span><span class="p">(</span><span class="n">test_rast</span><span class="p">)</span><span class="w">
</span><span class="n">plot</span><span class="p">(</span><span class="n">tex_shift1</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-10-21-glcm-rotation-invariant/glcm_90deg_shift-1.png" alt="center" /></p>
<p>To calculate rotationally invariant GLCM textures (over “all directions” in the
terminology of commonly used remote sensing software), use: <code class="highlighter-rouge">shift=list(c(0,1),
c(1,1), c(1,0), c(1,-1))</code>. This will calculate the average GLCM texture using
shifts of 0 degrees, 45 degrees, 90 degrees, and 135 degrees:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">tex_all_dir</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">glcm</span><span class="p">(</span><span class="n">test_rast</span><span class="p">,</span><span class="w"> </span><span class="n">shift</span><span class="o">=</span><span class="nf">list</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><span class="m">0</span><span class="p">,</span><span class="m">1</span><span class="p">),</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">1</span><span class="p">),</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">0</span><span class="p">),</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">-1</span><span class="p">)))</span><span class="w">
</span><span class="n">plot</span><span class="p">(</span><span class="n">tex_all_dir</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-10-21-glcm-rotation-invariant/glcm_all_directions-1.png" alt="center" /></p>
<p>To compare the difference between these textures, subtract the textures
calculated with a 90 degree shift from those calculated using multiple shifts,
and plot the result:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">((</span><span class="n">tex_all_dir</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">tex_shift1</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">tex_all_dir</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-10-21-glcm-rotation-invariant/glcm_all_directions_vs_90deg-1.png" alt="center" /></p>
<h2 id="computation-time">Computation time</h2>
<p>First look at the time difference for calculating a GLCM with only one shift
versus calculating a rotationally invariant form:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">microbenchmark</span><span class="p">)</span><span class="w">
</span><span class="n">glcm_one_dir</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">glcm</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">glcm_all_dir</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">glcm</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">shift</span><span class="o">=</span><span class="nf">list</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><span class="m">0</span><span class="p">,</span><span class="m">1</span><span class="p">),</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">1</span><span class="p">),</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">0</span><span class="p">),</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">-1</span><span class="p">)))</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">microbenchmark</span><span class="p">(</span><span class="n">glcm_one_dir</span><span class="p">(</span><span class="n">test_rast</span><span class="p">),</span><span class="w"> </span><span class="n">glcm_all_dir</span><span class="p">(</span><span class="n">test_rast</span><span class="p">),</span><span class="w"> </span><span class="n">times</span><span class="o">=</span><span class="m">5</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Unit: seconds
## expr min lq mean median uq
## glcm_one_dir(test_rast) 1.090759 1.117674 1.141704 1.146656 1.154196
## glcm_all_dir(test_rast) 4.090347 4.108833 4.189145 4.116991 4.164241
## max neval
## 1.199236 5
## 4.465313 5</code></pre></figure>
<p>As seen in the above, there is a performance penalty for using a rotationally
invariant GLCM (not surprisingly, as more calculations are involved).</p>
<p>Prior to having the ability to use multiple shifts hardcoded in <code class="highlighter-rouge">glcm</code>, it was
still possible to calculate rotationally invariant textures using the <code class="highlighter-rouge">glcm</code>
function. However, the calculation had to be done manually, using an approach
similar to what I do below with <code class="highlighter-rouge">glcm_all_dir_manual</code>. How much faster is it
perform the averaging directly in <code class="highlighter-rouge">glcm</code>?</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">glcm_all_dir_manual</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">text_0deg</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">glcm</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">shift</span><span class="o">=</span><span class="nf">c</span><span class="p">(</span><span class="m">0</span><span class="p">,</span><span class="m">1</span><span class="p">))</span><span class="w">
</span><span class="n">text_45deg</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">glcm</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">shift</span><span class="o">=</span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">1</span><span class="p">))</span><span class="w">
</span><span class="n">text_90deg</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">glcm</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">shift</span><span class="o">=</span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">0</span><span class="p">))</span><span class="w">
</span><span class="n">text_135deg</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">glcm</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">shift</span><span class="o">=</span><span class="nf">c</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">-1</span><span class="p">))</span><span class="w">
</span><span class="n">overlay</span><span class="p">(</span><span class="n">text_0deg</span><span class="p">,</span><span class="w"> </span><span class="n">text_45deg</span><span class="p">,</span><span class="w"> </span><span class="n">text_90deg</span><span class="p">,</span><span class="w"> </span><span class="n">text_135deg</span><span class="p">,</span><span class="w">
</span><span class="n">fun</span><span class="o">=</span><span class="k">function</span><span class="p">(</span><span class="n">w</span><span class="p">,</span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="p">,</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nf">return</span><span class="p">((</span><span class="n">w</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="m">4</span><span class="p">)</span><span class="w">
</span><span class="p">})</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">tex_all_dir_manual</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">glcm_all_dir_manual</span><span class="p">(</span><span class="n">test_rast</span><span class="p">)</span><span class="w">
</span><span class="c1"># Check that the textures match</span><span class="w">
</span><span class="n">table</span><span class="p">(</span><span class="n">getValues</span><span class="p">(</span><span class="n">tex_all_dir_manual</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">getValues</span><span class="p">(</span><span class="n">tex_all_dir</span><span class="p">))</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">##
## TRUE
## 273488</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">microbenchmark</span><span class="p">(</span><span class="n">glcm_all_dir_manual</span><span class="p">(</span><span class="n">test_rast</span><span class="p">),</span><span class="w"> </span><span class="n">glcm_all_dir</span><span class="p">(</span><span class="n">test_rast</span><span class="p">),</span><span class="w">
</span><span class="n">times</span><span class="o">=</span><span class="m">5</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Unit: seconds
## expr min lq mean median
## glcm_all_dir_manual(test_rast) 4.493543 4.501502 4.678655 4.656803
## glcm_all_dir(test_rast) 4.134398 4.190528 4.267464 4.225021
## uq max neval
## 4.809615 4.931815 5
## 4.304919 4.482455 5</code></pre></figure>
<p>The time difference isn’t that great, but the need for repeated calls to <code class="highlighter-rouge">glcm</code>
(and the need for multiple read/writes to disk for large files) could lead to a
more substantial advantage for the direct approach with <code class="highlighter-rouge">glcm</code> than is apparent
in this simple example. Of course, the manual approach does give more
flexibility if you need to do other processing (or scaling, etc.) to the
textures.</p>
<p><a href="http://www.azvoleff.com/articles/glcm-rotation-invariant/">Calculating rotation invariant GLCM textures</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on October 21, 2014.</p>http://www.azvoleff.com/articles/glcm-1-0-released2014-09-26T00:00:00+00:002014-09-26T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>I have released to CRAN<a href="http://cran.r-project.org/web/packages/glcm">
version 1.0 of the “glcm” R package</a> for calculating image texture measures
from grey-level co-occurrence matrices (GLCMs). Type:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">install.packages</span><span class="p">(</span><span class="s2">"glcm"</span><span class="p">)</span></code></pre></figure>
<p>at your R command prompt to download the latest CRAN release. This version
contains several new features, most importantly the ability to calculate
<a href="/articles/glcm-rotation-invariant">rotation invariant textures</a>, and to
automatically handle images that cannot fit in memory (using features from the
excellent <a href="http://cran.r-project.org/web/packages/raster/index.html">raster</a>
package).</p>
<p><a href="http://www.azvoleff.com/articles/glcm-1-0-released/">glcm 1.0 released</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on September 26, 2014.</p>http://www.azvoleff.com/articles/glcm-0-3-2-released2014-07-31T00:00:00+00:002014-07-31T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>I just released to CRAN<a href="http://cran.r-project.org/web/packages/glcm"> a
new version of the “glcm” R package</a> for calculating image texture measures
from grey-level co-occurrence matrices (GLCMs).</p>
<p>Version 0.3.2 fixes a minor bug in the projection assigned to the test image
included in <code class="highlighter-rouge">glcm</code>. The 1.0 release of <code class="highlighter-rouge">glcm</code>, which will support parallel
computation of GLCMs and computation of GLCMs over all directions, will be
coming soon - stay tuned. Type</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">install.packages</span><span class="p">(</span><span class="s2">"glcm"</span><span class="p">)</span></code></pre></figure>
<p>at your R command prompt to download the latest CRAN release. See the
<a href="http://cran.r-project.org/web/packages/glcm/NEWS">NEWS</a> file for more details.</p>
<p><a href="http://www.azvoleff.com/articles/glcm-0-3-2-released/">glcm 0.3.2 released</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on July 31, 2014.</p>http://www.azvoleff.com/articles/filtering-landsat-with-teamlucc2014-05-05T00:00:00+00:002014-05-05T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<h2 id="overview">Overview</h2>
<p>This post outlines how to use the <code class="highlighter-rouge">teamlucc</code> package to filter the Landsat
imagery available in the archives for a particular study site. <code class="highlighter-rouge">teamlucc</code>
includes several functions to make plots to assist with selecting anniversary
dates (or near anniversary dates…) when multiple Landsat path/rows are needed
to cover a single site. The <code class="highlighter-rouge">teamlucc</code> package also has functions to output an
order text file in the proper format for the USGS ESPA system and to
automatically download and verify the completed portions of a USGS ESPA order.</p>
<h2 id="getting-started">Getting started</h2>
<p>First load the <code class="highlighter-rouge">devtools</code> package, used for installing <code class="highlighter-rouge">teamlucc</code>. Install the
<code class="highlighter-rouge">devtools</code> package if it is not already installed:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">require</span><span class="p">(</span><span class="n">devtools</span><span class="p">))</span><span class="w"> </span><span class="n">install.packages</span><span class="p">(</span><span class="s1">'devtools'</span><span class="p">)</span></code></pre></figure>
<p>Now load the teamlucc package, using <code class="highlighter-rouge">devtools</code> to install it from github if it
is not yet installed. Also load the <code class="highlighter-rouge">rgdal</code> package needed for reading/writing
shapefiles:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">require</span><span class="p">(</span><span class="n">teamlucc</span><span class="p">))</span><span class="w"> </span><span class="n">install_github</span><span class="p">(</span><span class="s1">'azvoleff/teamlucc'</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: teamlucc
## Loading required package: Rcpp
## Loading required package: raster
## Loading required package: sp</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: replacing previous import by 'raster::buffer' when loading
## 'teamlucc'</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: replacing previous import by 'raster::interpolate' when loading
## 'teamlucc'</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: replacing previous import by 'raster::rotated' when loading
## 'teamlucc'</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">rgdal</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## rgdal: version: 0.9-1, (SVN revision 518)
## Geospatial Data Abstraction Library extensions to R successfully loaded
## Loaded GDAL runtime: GDAL 1.11.0, released 2014/04/16
## Path to GDAL shared files: C:/Users/azvoleff/R/win-library/3.1/rgdal/gdal
## GDAL does not use iconv for recoding strings.
## Loaded PROJ.4 runtime: Rel. 4.8.0, 6 March 2012, [PJ_VERSION: 480]
## Path to PROJ.4 shared files: C:/Users/azvoleff/R/win-library/3.1/rgdal/proj</code></pre></figure>
<h2 id="downloading-list-of-available-landsat-scenes">Downloading list of available Landsat scenes</h2>
<p>Start by setting up an account on <a href="http://earthexplorer.usgs.gov">USGS Earth
Explorer</a>. After setting up an account, login to
the system, and upload a shapefile or draw a polygon to indicate your area of
interest. After doing so, click the “Data Sets > >” button on the lower left of
the screen. For this example, we will be using Landsat CDR Surface Reflectance
imagery. Click the “+” next to “Landsat CDR”, and then click the checkboxes for
both “Land Surface Reflectance - L7 ETM+” and also “Land Surface Reflectance -
L4-5 TM”:</p>
<p><img src="/content/2014-05-05-filtering-landsat-with-teamlucc/ee_cdr_checkboxes.png" alt="center" /></p>
<p>Now click the “Results > >” button.</p>
<p>You will now see results for one of the two (Landsat 4-5 and Landsat 7)
datasets you selected. Look under “Data Set” on the left side of the screen,
and it will say “Land Surface Reflectance - L7 ETM+” if it is displaying the
Landsat 7 CDR dataset. You will need to separately export the Landsat 7 and
Landsat 4-5 scene lists.</p>
<p>To export the first scene list: from the “Search Results” page, download a CSV
file of ALL available Landsat imagery for the search area. To do this, click
the “Click here to export your results > >” text near the top right of the
screen.</p>
<p><img src="/content/2014-05-05-filtering-landsat-with-teamlucc/ee_export_button.png" alt="center" /></p>
<p>In the “Metadata Export” box, choose “Non-Limited Results” for “Export Type”.
For “Format” choose “CSV”:</p>
<p><img src="/content/2014-05-05-filtering-landsat-with-teamlucc/ee_export_screen.png" alt="center" /></p>
<p>A window will come up saying “Your export file is being generated.” Click
“OK”. Repeat the same process for the other CDR dataset, by changing the data
set you have selected, and again clicking the “Click here to export your
results > >” text.</p>
<p>You will receive two emails at your USGS Earth Explorer registered email
address each with a link to one a zipfile containing one of the scene lists.
Download these zipfiles and use them for the next step (or use the below
example data instead)</p>
<h1 id="selecting-landsat-scenes-to-download">Selecting Landsat scenes to download</h1>
<p>To follow along with this analysis, <a href="/content/2014-05-05-filtering-landsat-with-teamlucc/NAK_data.zip">download this
zipfile</a>
with a shapefile of Zone of Interation (ZOI) of the TEAM site in <a href="http://www.teamnetwork.org/network/sites/nam-kading-0">Nam Kading
National Protected Area</a>
in Lao PDR. The zipfile also includes scene lists from EarthExplorer of all
available Landsat scenes (as of April 23, 2014) for this site.</p>
<p>First, read in the Landsat scene lists downloaded from USGS EarthExplorer,
using the <code class="highlighter-rouge">ee_read</code> function in <code class="highlighter-rouge">teamlucc</code>:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">l</span><span class="m">7</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">ee_read</span><span class="p">(</span><span class="s1">'NAK_L7_20140423_scenelist.csv'</span><span class="p">)</span><span class="w">
</span><span class="n">l</span><span class="m">45</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">ee_read</span><span class="p">(</span><span class="s1">'NAK_L4-5_20140423_scenelist.csv'</span><span class="p">)</span></code></pre></figure>
<p>Now, merge the Landsats 4-5 and Landsat 7 scene lists so they can be analyzed
together:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">l</span><span class="m">457</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">merge</span><span class="p">(</span><span class="n">l</span><span class="m">7</span><span class="p">,</span><span class="w"> </span><span class="n">l</span><span class="m">45</span><span class="p">,</span><span class="w"> </span><span class="n">all</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">)</span></code></pre></figure>
<p>Selecting Landsat images, particularly for an area covered by multiple Landsat
path/rows, can be difficult. The <code class="highlighter-rouge">wrspathrow</code> package (available on CRAN) is
helpful for producing a quick visualization of the number of Landsat scenes
(path/row(s)) needed to cover an area. For example, the below code reads in the
ZOI shapefile for Nam Kading, and plots the Landsat path/rows needed to cover
the ZOI. This plot uses the path/row polygons included in the wrspathrow
package, and includes text labels for each path and row:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">wrspathrow</span><span class="p">)</span><span class="w">
</span><span class="n">NAK_zoi</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">readOGR</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span><span class="w"> </span><span class="s1">'ZOI_NAK_2012_EEsimple'</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## OGR data source with driver: ESRI Shapefile
## Source: ".", layer: "ZOI_NAK_2012_EEsimple"
## with 1 features and 3 fields
## Feature type: wkbPolygon with 2 dimensions</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">NAK_pathrows</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">pathrow_num</span><span class="p">(</span><span class="n">NAK_zoi</span><span class="p">,</span><span class="w"> </span><span class="n">as_polys</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">)</span><span class="w">
</span><span class="n">plot</span><span class="p">(</span><span class="n">NAK_pathrows</span><span class="p">)</span><span class="w">
</span><span class="n">plot</span><span class="p">(</span><span class="n">NAK_zoi</span><span class="p">,</span><span class="w"> </span><span class="n">add</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">,</span><span class="w"> </span><span class="n">lty</span><span class="o">=</span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="n">col</span><span class="o">=</span><span class="s2">"#00ff0050"</span><span class="p">)</span><span class="w">
</span><span class="n">text</span><span class="p">(</span><span class="n">coordinates</span><span class="p">(</span><span class="n">NAK_pathrows</span><span class="p">),</span><span class="w"> </span><span class="n">labels</span><span class="o">=</span><span class="n">paste</span><span class="p">(</span><span class="n">NAK_pathrows</span><span class="o">$</span><span class="n">PATH</span><span class="p">,</span><span class="w">
</span><span class="n">NAK_pathrows</span><span class="o">$</span><span class="n">ROW</span><span class="p">,</span><span class="w"> </span><span class="n">sep</span><span class="o">=</span><span class="s1">', '</span><span class="p">))</span></code></pre></figure>
<p><img src="/content/2014-05-05-filtering-landsat-with-teamlucc/pathrow_versus_zoi-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>From the above plots, we can see that it takes scenes from three Landsat
path/rows to cover the entire Nam Kading ZOI. Suppose we need images from
within a particular period, covering the entire area of that ZOI. The
the <code class="highlighter-rouge">ee_plot</code> function in the <code class="highlighter-rouge">teamlucc</code> package can make a plot of the
available imagery within a given time period, color coded by sensor and percent
cloud cover. To use this function, first specify a start and end date:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">start_date</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">as.Date</span><span class="p">(</span><span class="s1">'1995/1/1'</span><span class="p">)</span><span class="w">
</span><span class="n">end_date</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">as.Date</span><span class="p">(</span><span class="s1">'2000/1/1'</span><span class="p">)</span></code></pre></figure>
<p>Now use <code class="highlighter-rouge">ee_plot</code> to plot the available imagery from within that time period:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">ee_plot</span><span class="p">(</span><span class="n">l</span><span class="m">457</span><span class="p">,</span><span class="w"> </span><span class="n">start_date</span><span class="p">,</span><span class="w"> </span><span class="n">end_date</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-05-05-filtering-landsat-with-teamlucc/ee_plot-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>Each box in this plot represents a single Landsat scene from the Landsat
archive. The color of each box indicates the path and row of that scene, while
the outline color of each box indicates the sensor (Landsat 5 versus Landsat
7 for example). The shading of a particular box (light or dark) box indicates
the cloud cover of that scene (lighter colors correspond to scenes with greater
cloud cover).</p>
<p>From this plot, we can see that within the 1995 to 2000 time
period, January 1996 is the only month in which images are available for all
three path/rows needed to cover the Nam Kading ZOI. Further, from the shading
of the boxes in that month, we can tell that these images are almost cloud
free, and are from Landsat 5 (indicated by the green outlines).</p>
<p>Note that <code class="highlighter-rouge">ee_plot</code> will by default only plot imagery where greater than 70% of
the image is unobscured by plots (i.e. less than 30% cloud cover). This default
can be changed by supplying the <code class="highlighter-rouge">min_clear</code> parameter to <code class="highlighter-rouge">ee_plot</code>.</p>
<p>When dealing with a large number of scenes, a different type of plot can be
helpful. The <code class="highlighter-rouge">normalize</code> argument to <code class="highlighter-rouge">ee_plot</code> tells <code class="highlighter-rouge">ee_plot</code> to calculate the
best (lowest cloud cover) image for each path and row for each month. Then,
<code class="highlighter-rouge">ee_plot</code> sums across all the path and rows, and plots the results:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">ee_plot</span><span class="p">(</span><span class="n">l</span><span class="m">457</span><span class="p">,</span><span class="w"> </span><span class="n">start_date</span><span class="p">,</span><span class="w"> </span><span class="n">end_date</span><span class="p">,</span><span class="w"> </span><span class="n">normalize</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-05-05-filtering-landsat-with-teamlucc/ee_plot_normalized-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>This type of plot is helpful in visualizing the periods in which the greatest
proportion of a site can be covered by cloud-free (or nearly cloud-free)
imagery.</p>
<h1 id="downloading-landsat-scenes-using-espa">Downloading Landsat scenes using ESPA</h1>
<p><code class="highlighter-rouge">teamlucc</code> also facilitates placing orders for imagery using the <a href="https://espa.cr.usgs.gov">USGS ESPA
system</a>. The ESPA system accepts scenes lists as a
text file. To output a scene list for upload to ESPA, use the <code class="highlighter-rouge">espa_scenelist</code>
function, specifying the start and end dates needed, and the name of the output
file:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">espa_scenelist</span><span class="p">(</span><span class="n">l</span><span class="m">457</span><span class="p">,</span><span class="w"> </span><span class="n">as.Date</span><span class="p">(</span><span class="s1">'1996/1/1'</span><span class="p">),</span><span class="w"> </span><span class="n">as.Date</span><span class="p">(</span><span class="s1">'1996/12/31'</span><span class="p">),</span><span class="w">
</span><span class="s1">'NAK_ESPA_scenelist_1986.txt'</span><span class="p">)</span></code></pre></figure>
<p>The above line of code will save a text file named
<code class="highlighter-rouge">NAK_ESPA_scenelist_1986.txt</code> in your current working directory. To place an
order, login to the ESPA system, and upload the text file.</p>
<p>After receiving an email from the ESPA system notifying you that your order is
processed, download the order from within R using the <code class="highlighter-rouge">espa_download</code> function
by specifying 1) the email address you used to place the order, 2) the ESPA
order ID number (included in the email from ESPA), and 3) the output folder in
which to save the downloaded images. <code class="highlighter-rouge">espa_download</code> will first check within
the specified output folder to see if each image already exists, and will not
re-download existing files unless the existing files do not match the files
available on ESPA.</p>
<p><strong>Note the below code is not working as of 7/1/2014 due to changes in the ESPA
download system. I will update this post when the code is working again.</strong></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">espa_download</span><span class="p">(</span><span class="s1">'azvoleff@example.com'</span><span class="p">,</span><span class="w"> </span><span class="s1">'272014-114611'</span><span class="p">,</span><span class="w"> </span><span class="s1">'D:/ESPA_Downloads'</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Error in espa_download("azvoleff@example.com", "272014-114611", "D:/ESPA_Downloads"): Due to changes in the ESPA system, espa_download is not working as of 7/1/2014</code></pre></figure>
<!-- Note that the above warning occurs since I supplied a non-existent output
folder for this example - if the function were run on a real order it would
print status messages showing the results of the download. -->
<p><a href="http://www.azvoleff.com/articles/filtering-landsat-with-teamlucc/">Filtering available Landsat scenes with teamlucc</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on May 05, 2014.</p>http://www.azvoleff.com/articles/glcm-0-3-1-released2014-04-24T00:00:00+00:002014-04-24T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>I just released to CRAN<a href="http://cran.r-project.org/web/packages/glcm"> a
new version of the “glcm” R package</a> for calculating image texture measures
from grey-level co-occurrence matrices (GLCMs). Type</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">install.packages</span><span class="p">(</span><span class="s2">"glcm"</span><span class="p">)</span></code></pre></figure>
<p>at your R command prompt to download the latest CRAN release.</p>
<p>This version fixes a bug in handling window sizes other than the default 3x3
window size, adds additional test cases, and performs more validation on user
input to the <code class="highlighter-rouge">glcm</code> function. See the
<a href="http://cran.r-project.org/web/packages/glcm/NEWS">NEWS</a> file for more details.</p>
<p><a href="http://www.azvoleff.com/articles/glcm-0-3-1-released/">glcm 0.3.1 released</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on April 24, 2014.</p>http://www.azvoleff.com/articles/preprocessing-imagery-with-teamlucc2014-11-13T00:00:00-00:002014-04-16T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<h2 id="overview">Overview</h2>
<p>This post outlines how to use the <code class="highlighter-rouge">teamlucc</code> package to preprocess imagery from
the <a href="http://landsat.usgs.gov/CDR_LSR.php">Landsat Surface Reflectance Climate Data Record
(CDR)</a> archive. The <code class="highlighter-rouge">teamlucc</code> package
supports a number of common preprocessing steps, including:</p>
<ul>
<li>Conversion of CDR files to any GDAL supported file format</li>
<li>Cropping Landsat tiles to a given area of interest (AOI)</li>
<li>Mosaicking and cropping of DEM tiles (such as ASTER or SRTM) to a given AOI
or Landsat path/row</li>
<li>Topographic correction of CDR scenes</li>
</ul>
<h2 id="getting-started">Getting started</h2>
<p>First load the <code class="highlighter-rouge">teamlucc</code> package, and the <code class="highlighter-rouge">rgdal</code> package:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">teamlucc</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: Rcpp
## Loading required package: raster
## Loading required package: sp</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: replacing previous import by 'raster::buffer' when loading
## 'teamlucc'</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: replacing previous import by 'raster::interpolate' when loading
## 'teamlucc'</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: replacing previous import by 'raster::rotated' when loading
## 'teamlucc'</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">rgdal</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## rgdal: version: 0.9-1, (SVN revision 518)
## Geospatial Data Abstraction Library extensions to R successfully loaded
## Loaded GDAL runtime: GDAL 1.11.0, released 2014/04/16
## Path to GDAL shared files: C:/Users/azvoleff/R/win-library/3.1/rgdal/gdal
## GDAL does not use iconv for recoding strings.
## Loaded PROJ.4 runtime: Rel. 4.8.0, 6 March 2012, [PJ_VERSION: 480]
## Path to PROJ.4 shared files: C:/Users/azvoleff/R/win-library/3.1/rgdal/proj</code></pre></figure>
<h2 id="dem-setup">DEM setup</h2>
<p>Before CDR surface reflectance images can be topographically corrected with
<code class="highlighter-rouge">teamlucc</code>, a digital elevation model (DEM) needs to be obtained that is in the
same resolution and coordinate system as the CDR imagery, and slope and aspect
need to be calculated. The <code class="highlighter-rouge">auto_setup_dem</code> function in the <code class="highlighter-rouge">teamlucc</code> package
facilitates this task. There are many options you can supply to
<code class="highlighter-rouge">auto_setup_dem</code> - see <code class="highlighter-rouge">?auto_setup_dem</code> for more information on these options.
If you do not want to topographically correct your imagery, you can skip this
step and move ahead to the next section.</p>
<p><code class="highlighter-rouge">auto_setup_dem</code> assembles a DEM to cover a given area of interest (AOI), and
can mosaic multiple DEM files if needed to cover an AOI. To use
<code class="highlighter-rouge">auto_setup_dem</code>, the user must first define the area of interest with an AOI
polygon. If you have a shapefile of an area of interest, load it into R using
the <code class="highlighter-rouge">readOGR</code> command. The <code class="highlighter-rouge">readOGR</code> command needs the folder the shapefile is
in (in this example the current working directory, specified by <code class="highlighter-rouge">.</code>) as the
first parameter, and the filename of the shapefile (without the “.shp”
extension) as the second parameter (<code class="highlighter-rouge">PA_VB</code>) in this example). This example
uses a shapefile of the boundary of the Braulio Carrillo National Park, in
which the <a href="http://www.teamnetwork.org/network/sites/volc%C3%A1n-barva">Volcán Barva TEAM
site</a> is located:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">aoi</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">readOGR</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span><span class="w"> </span><span class="s1">'PA_VB'</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## OGR data source with driver: ESRI Shapefile
## Source: ".", layer: "PA_VB"
## with 5 features and 8 fields
## Feature type: wkbPolygon with 2 dimensions</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">aoi</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-preprocessing-imagery-with-teamlucc/VB_aoi_original-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>As seen in the above plot, there are a number of adjoining polygons in this
shapefile. The functions in <code class="highlighter-rouge">teamlucc</code> expect the AOI to be of length one. So
calculate the convex hull of the AOI using the functions in the <code class="highlighter-rouge">rgeos</code>
package:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">rgeos</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## rgeos version: 0.3-8, (SVN revision 460)
## GEOS runtime version: 3.4.2-CAPI-1.8.2 r3921
## Polygon checking: TRUE</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">aoi</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">gConvexHull</span><span class="p">(</span><span class="n">aoi</span><span class="p">)</span></code></pre></figure>
<p>The AOI is now a single polygon:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">aoi</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-preprocessing-imagery-with-teamlucc/VB_aoi_convex_hull-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>If you do not have an AOI, but know the Landsat path and row you want to work
with, an alternative is to define the AOI based on the path and row, using the
<code class="highlighter-rouge">wrspathrow</code> R package, and supplying the desired path and row numbers. Here
<code class="highlighter-rouge">127</code> is the WRS-2 path number, and <code class="highlighter-rouge">47</code> is the WRS-2 row number for the
path/row at the center of the above AOI:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">wrspathrow</span><span class="p">)</span><span class="w">
</span><span class="n">aoi_from_pathrow</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">pathrow_poly</span><span class="p">(</span><span class="m">127</span><span class="p">,</span><span class="w"> </span><span class="m">47</span><span class="p">)</span></code></pre></figure>
<p>In addition to the AOI, <code class="highlighter-rouge">auto_setup_dem</code> needs to know the location and spatial
extents of the DEM files you have available on your machine. This list can be
assembled automatically using the <code class="highlighter-rouge">get_extent_polys</code> function in <code class="highlighter-rouge">teamlucc</code>.<br />
See <code class="highlighter-rouge">?get_extent_polys</code> for more information. <strong>The below code will fail on your
machine because you will not have the proper DEMs for this example. Download
the proper DEMs for the area in which you are working and place them in a
folder on your machine if you want to test this function.</strong></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">dem_files</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">dir</span><span class="p">(</span><span class="s1">'H:/Data/CGIAR_SRTM/Tiles'</span><span class="p">,</span><span class="w"> </span><span class="n">pattern</span><span class="o">=</span><span class="s1">'.tif$'</span><span class="p">,</span><span class="w"> </span><span class="n">full.names</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">)</span><span class="w">
</span><span class="n">dems</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">lapply</span><span class="p">(</span><span class="n">dem_files</span><span class="p">,</span><span class="w"> </span><span class="n">raster</span><span class="p">)</span><span class="w">
</span><span class="n">dem_extents</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">get_extent_polys</span><span class="p">(</span><span class="n">dems</span><span class="p">)</span></code></pre></figure>
<p>For flexibility, <code class="highlighter-rouge">auto_setup_dem</code> can optionally crop each output DEM to the
spatial extent of the supplied AOI. If <code class="highlighter-rouge">crop_to_aoi=TRUE</code>, then
<code class="highlighter-rouge">auto_setup_dem</code> will crop the DEMs to the spatial extent of the supplied AOI.
If <code class="highlighter-rouge">crop_to_aoi=FALSE</code>, then <code class="highlighter-rouge">auto_setup_dem</code> will crop the DEMs to the extent
of the Landsat path/rows needed to cover the AOI.</p>
<p>Lastly, <code class="highlighter-rouge">auto_setup_dem</code> needs to know where to save its output. For this
example, save the output to the current working directory <code class="highlighter-rouge">.</code>. Now that all of
the essential inputs are defined, <code class="highlighter-rouge">auto_setup_dem</code> can be called:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">auto_setup_dem</span><span class="p">(</span><span class="n">aoi</span><span class="p">,</span><span class="w"> </span><span class="s1">'.'</span><span class="p">,</span><span class="w"> </span><span class="n">dem_extents</span><span class="p">,</span><span class="w"> </span><span class="n">crop_to_aoi</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## [1] "2014-11-13 16:49:49: started \"Processing DEMS for 1 path/rows\""
## [1] "2014-11-13 16:49:50: started \"Processing 1 of 1: 015-053\""
## [1] "2014-11-13 16:50:00: finished \"Processing 1 of 1: 015-053\" (10.047s elapsed)"
## [1] "2014-11-13 16:50:00: finished \"Processing DEMS for 1 path/rows\" (10.285s elapsed)"</code></pre></figure>
<p>The result will be a mosaicked DEM, in the current working directory, that can
be used for topographically correcting imagery with the
<code class="highlighter-rouge">auto_preprocess_landsat</code> function.</p>
<h2 id="preprocessing">Preprocessing</h2>
<p>Images are delivered from the CDR archive in either ENVI format, GeoTIFF
format, or Hierarchical Data Format (HDF). The <code class="highlighter-rouge">teamlucc</code> package will, by
default, convert HDF or ENVI format images to a GeoTIFF format, as these image
files can be easily read in R and in most commonly used remote sensing and GIS
software packages. This example assumes you want GeoTIFF output - see the help
files for <code class="highlighter-rouge">teamlucc</code> for other output options.</p>
<p>First you will need to acquire a time series of CDR imagery for your site. The
<code class="highlighter-rouge">espa_download</code> function can facilitate downloading files from an ESPA order.
Also see the post on <a href="/articles/filtering-landsat-with-teamlucc">Filtering and downloading Landsat
scenes</a> for more details on how
<code class="highlighter-rouge">teamlucc</code> can help with image acquisition.</p>
<p>Once you have downloaded your imagery from ESPA, I recommend you put all of the
zip files from your download in a single folder. You can then use the
<code class="highlighter-rouge">teamlucc</code> <code class="highlighter-rouge">espa_extract</code> function to automate extracting your CDR image files,
including placing the extracted files in consistently named output folders.<br />
If, for example, your files are located in <code class="highlighter-rouge">espa_downloads</code> and you want to
extract them to <code class="highlighter-rouge">espa_extracts</code>, run the following (<strong>this block, and the next
block of code, will both fail on your computer since you do not have the
required imagery - this is only an example, download your own imagery to follow
along</strong>):</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">download_folder</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s1">'espa_downloads'</span><span class="w">
</span><span class="n">extract_folder</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s1">'espa_extracts'</span><span class="w">
</span><span class="n">espa_extract</span><span class="p">(</span><span class="n">download_folder</span><span class="p">,</span><span class="w"> </span><span class="n">extract_folder</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## 1 of 2. Extracting LT50150532000044-SC20140514195145.tar.gz to espa_extracts/015-053_2000-044_LT5
## 2 of 2. Extracting LT50150532001014-SC20140514195632.tar.gz to espa_extracts/015-053_2001-014_LT5</code></pre></figure>
<p>Now that the CDR format image files are extracted, you are ready to run
<code class="highlighter-rouge">auto_preprocess_landsat</code>. As with <code class="highlighter-rouge">auto_setup_dem</code>, there are many options you
can supply to <code class="highlighter-rouge">auto_preprocess_landsat</code> - see <code class="highlighter-rouge">?auto_preprocess_landsat</code>. The
below is a simple example of how to call <code class="highlighter-rouge">auto_preprocess_landsat</code>.</p>
<p>The <code class="highlighter-rouge">image_dirs</code> line below is just a fancy way of finding all the folders
located in <code class="highlighter-rouge">extract_folder</code> that contain CDR imagery. You could just as easily
specify these folders individually as as a list of strings, like: <code class="highlighter-rouge">image_dirs
<- c('C:/folder1', 'C:/folder2')</code> if you had two CDR Landsat scenes located in
<code class="highlighter-rouge">C:/folder1</code> and <code class="highlighter-rouge">C:/folder2</code>, respectively.</p>
<p>The <code class="highlighter-rouge">prefix</code> parameter specifies a string that will be used in naming files
output by <code class="highlighter-rouge">auto_preprocess_landsat</code>. For <code class="highlighter-rouge">prefix</code> I suggest you use a short (2
or 3 character) site name or site code that is meaningful to you.</p>
<p>There are two other options we provide below to <code class="highlighter-rouge">auto_preprocess_landsat</code>.<br />
<code class="highlighter-rouge">tc=TRUE</code> tells <code class="highlighter-rouge">auto_preprocess_landsat</code> to perform topographic correction.<br />
Because of this, we also need to specify <code class="highlighter-rouge">dem_path</code> (where the DEM files
preprocessed by <code class="highlighter-rouge">auto_setup_dem</code> are located), so that the DEM files for this
scene can be found. Here we set <code class="highlighter-rouge">dem_path='.'</code> as the DEM is in our current
working directory. We supply an AOI (same AOI as above) to crop the output
images. <code class="highlighter-rouge">verbose=TRUE</code> indicates that we want detailed progress messages to
print while the script is running. The output of <code class="highlighter-rouge">auto_preprocess_landsat</code> is a
<code class="highlighter-rouge">data.frame</code> with a list of the preprocessed files and their file formats.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">image_dirs</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">dir</span><span class="p">(</span><span class="s1">'espa_extracts'</span><span class="p">,</span><span class="w">
</span><span class="n">pattern</span><span class="o">=</span><span class="s1">'^[0-9]{3}-[0-9]{3}_[0-9]{4}-[0-9]{3}_((LT[45])|(LE7))$'</span><span class="p">,</span><span class="w">
</span><span class="n">full.names</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">)</span><span class="w">
</span><span class="n">filelist</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">auto_preprocess_landsat</span><span class="p">(</span><span class="n">image_dirs</span><span class="p">,</span><span class="w"> </span><span class="n">prefix</span><span class="o">=</span><span class="s1">'VB'</span><span class="p">,</span><span class="w"> </span><span class="n">tc</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">,</span><span class="w">
</span><span class="n">dem_path</span><span class="o">=</span><span class="s1">'.'</span><span class="p">,</span><span class="w"> </span><span class="n">aoi</span><span class="o">=</span><span class="n">aoi</span><span class="p">,</span><span class="w"> </span><span class="n">verbose</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: executing %dopar% sequentially: no parallel backend registered</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## [1] "2014-11-13 16:50:52: started \"Preprocessing 015-053_2000-044_L5TSR\""
## [1] "2014-11-13 16:50:52: started \"cropping and reprojecting\""</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning in build_mask_vrt(file_base, mask_vrt_file, file_format): Using
## "fmask_band" instead of newer "cfmask_band" band name</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## [1] "2014-11-13 16:51:03: finished \"cropping and reprojecting\" (10.787s elapsed)"
## [1] "2014-11-13 16:51:03: started \"topocorr\""
## [1] "2014-11-13 16:53:25: finished \"topocorr\" (142.071s (~2.37 minutes) elapsed)"
## [1] "2014-11-13 16:53:25: started \"writing data\""
## [1] "2014-11-13 16:53:29: finished \"writing data\" (4.052s elapsed)"
## [1] "2014-11-13 16:53:29: finished \"Preprocessing 015-053_2000-044_L5TSR\" (156.919s (~2.62 minutes) elapsed)"
## [1] "2014-11-13 16:53:30: started \"Preprocessing 015-053_2001-014_L5TSR\""
## [1] "2014-11-13 16:53:30: started \"cropping and reprojecting\""</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning in build_mask_vrt(file_base, mask_vrt_file, file_format): Using
## "fmask_band" instead of newer "cfmask_band" band name</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## [1] "2014-11-13 16:53:44: finished \"cropping and reprojecting\" (14.161s elapsed)"
## [1] "2014-11-13 16:53:44: started \"topocorr\""
## [1] "2014-11-13 16:55:55: finished \"topocorr\" (131.256s (~2.19 minutes) elapsed)"
## [1] "2014-11-13 16:55:55: started \"writing data\""
## [1] "2014-11-13 16:55:59: finished \"writing data\" (4.236s elapsed)"
## [1] "2014-11-13 16:55:59: finished \"Preprocessing 015-053_2001-014_L5TSR\" (149.662s (~2.49 minutes) elapsed)"</code></pre></figure>
<p>The result is two cropped, reprojected, and topographically corrected Landsat
images covering the specified AOI. One image from 2000:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">ls_2000</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">brick</span><span class="p">(</span><span class="s1">'espa_extracts/015-053_2000-044_LT5/VB_015-053_2000-044_L5TSR_tc.tif'</span><span class="p">)</span><span class="w">
</span><span class="n">ls_2000</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">linear_stretch</span><span class="p">(</span><span class="n">ls_2000</span><span class="p">,</span><span class="w"> </span><span class="n">pct</span><span class="o">=</span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="n">max_val</span><span class="o">=</span><span class="m">255</span><span class="p">)</span><span class="w">
</span><span class="n">browse_image</span><span class="p">(</span><span class="n">ls_2000</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="o">=</span><span class="m">4</span><span class="p">,</span><span class="w"> </span><span class="n">g</span><span class="o">=</span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="o">=</span><span class="m">2</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-preprocessing-imagery-with-teamlucc/plot_2000_landsat-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>And one image from 2001:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">ls_2001</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">brick</span><span class="p">(</span><span class="s1">'espa_extracts/015-053_2001-014_LT5/VB_015-053_2001-014_L5TSR_tc.tif'</span><span class="p">)</span><span class="w">
</span><span class="n">ls_2001</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">linear_stretch</span><span class="p">(</span><span class="n">ls_2001</span><span class="p">,</span><span class="w"> </span><span class="n">pct</span><span class="o">=</span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="n">max_val</span><span class="o">=</span><span class="m">255</span><span class="p">)</span><span class="w">
</span><span class="n">browse_image</span><span class="p">(</span><span class="n">ls_2001</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="o">=</span><span class="m">4</span><span class="p">,</span><span class="w"> </span><span class="n">g</span><span class="o">=</span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="o">=</span><span class="m">2</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-preprocessing-imagery-with-teamlucc/plot_2001_landsat-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>There is a fair amount of cloud cover in the 2000 image. See the post on <a href="/articles/cloud-removal-with-teamlucc">cloud
removal</a> for one means of addressing
this issue.</p>
<p><a href="http://www.azvoleff.com/articles/preprocessing-imagery-with-teamlucc/">Landsat Surface Reflectance CDR preprocessing with teamlucc</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on April 16, 2014.</p>http://www.azvoleff.com/articles/cloud-removal-with-teamlucc2014-05-09T00:00:00-00:002014-04-16T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<h2 id="overview">Overview</h2>
<p>This post outlines how to use the <code class="highlighter-rouge">teamlucc</code> package to remove thick clouds
from Landsat imagery using the Neighborhood Similar Pixel Interpolator (NSPI)
algorithm by <a href="http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=6095313">Zhu et
al.</a><sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>. <code class="highlighter-rouge">teamlucc</code>
includes the original (modified slightly to be called from R)
<a href="http://www.exelisvis.com/ProductsServices/IDL.aspx">IDL</a> code by <a href="http://geography.osu.edu/grads/xzhu/">Xiaolin
Zhu</a>, as well as a native R/C++
implementation of the NSPI algorithm. Thanks to Xiaolin for permission to
redistribute his code along with the <code class="highlighter-rouge">teamlucc</code> package.</p>
<h2 id="getting-started">Getting started</h2>
<p>First load the <code class="highlighter-rouge">teamlucc</code> package, and the <code class="highlighter-rouge">SDMTools</code> package we will use
later:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">teamlucc</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: Rcpp
## Loading required package: raster
## Loading required package: sp</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: replacing previous import by 'raster::buffer' when loading
## 'teamlucc'</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: replacing previous import by 'raster::interpolate' when loading
## 'teamlucc'</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: replacing previous import by 'raster::rotated' when loading
## 'teamlucc'</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">SDMTools</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">##
## Attaching package: 'SDMTools'
##
## The following object is masked from 'package:teamlucc':
##
## accuracy
##
## The following object is masked from 'package:raster':
##
## distance</code></pre></figure>
<p>If <code class="highlighter-rouge">teamlucc</code> is not installed, install it using <code class="highlighter-rouge">devtools</code>”</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">require</span><span class="p">(</span><span class="n">teamlucc</span><span class="p">))</span><span class="w"> </span><span class="n">install_github</span><span class="p">(</span><span class="s1">'azvoleff/teamlucc'</span><span class="p">)</span></code></pre></figure>
<p>First I will cover how to cloud fill a single clouded image using a single
clear (or partially clouded) image. Skip to the end to see how to automate the
cloud fill process using <code class="highlighter-rouge">teamlucc</code>.</p>
<h2 id="cloud-fill-a-single-clouded-image-with-a-single-clear-image">Cloud fill a single clouded image with a single clear image</h2>
<p>This example will use a portion of a 1986 Landsat 5 scene from Volcan Barva,
Costa Rica (a <a href="http://www.teamnetwork.org/network/sites/volc%C3%A1n-barva">TEAM
Network</a> monitoring
site). The scene is WRS-2 path 15, row 53. Particularly in the tropics, it can
sometimes be difficult to find a Landsat image that is cloud-free. Cloud
filling can offer a solution to this problem if there are multiple Landsat
scenes captured of an area of interest, that, taken together, offer a
cloud-free (or nearly cloud-free) view of an area. Throughout this post I will
refer to the “base” and the “fill” images. The “base” image is a cloudy image
that will be filled using images (the “fill images) of the same area that were
captured on different dates.</p>
<p>While it can sometimes be possible to find a cloud-free scene from a different
part of the year that can be used to fill in a cloudy scene from an earlier or
later base date, it is often the case that both the fill and base image will
have clouds. Therefore we must use cloud masks to mark areas in both the base
and the fill image. Without a cloud mask for the fill image we could otherwise
inadvertently fill clouded areas in one image with also cloudy pixels from
another image.</p>
<p>The base (cloudy) image for this example is from January 5, 1986, and the fill
image is from January 21, 1986. The images are surface reflectance images from
the <a href="http://landsat.usgs.gov/CDR_LSR.php">Landsat Surface Reflectance Climate Data Record
(CDR)</a>, that also include cloud masks
constructed with the <a href="https://code.google.com/p/fmask">Function of Mask (fmask)
algorithm</a><sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup>. Both of these images have
significant cloud cover, and some areas are cloudy in both images. This
example will show the ability of the cloud fill algorithms to function even in
difficult circumstances.</p>
<p>To follow along with this analysis, <a href="/content/2014-04-16-cloud-removal-with-teamlucc/2014-04-16-cloud-removal-with-teamlucc.zip">download these
files</a>.<br />
Note that the original CDR reflectance images have been rescaled to range
between 0 and 255 in the files supplied here (this rescaling is not required
prior to performing cloud fill - I just did it here to make the files sizes
smaller so they could be more easily hosted on this site).</p>
<h3 id="load-input-data">Load input data</h3>
<p>First load the base and fill images into R:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">base</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">brick</span><span class="p">(</span><span class="s1">'vb_1986_005_b234.tif'</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## rgdal: version: 0.9-1, (SVN revision 518)
## Geospatial Data Abstraction Library extensions to R successfully loaded
## Loaded GDAL runtime: GDAL 1.11.0, released 2014/04/16
## Path to GDAL shared files: C:/Users/azvoleff/R/win-library/3.1/rgdal/gdal
## GDAL does not use iconv for recoding strings.
## Loaded PROJ.4 runtime: Rel. 4.8.0, 6 March 2012, [PJ_VERSION: 480]
## Path to PROJ.4 shared files: C:/Users/azvoleff/R/win-library/3.1/rgdal/proj</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">fill</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">brick</span><span class="p">(</span><span class="s1">'vb_1986_021_b234.tif'</span><span class="p">)</span></code></pre></figure>
<p>Notice the cloud cover in the base image:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plotRGB</span><span class="p">(</span><span class="n">base</span><span class="p">,</span><span class="w"> </span><span class="n">stretch</span><span class="o">=</span><span class="s1">'lin'</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/base_image-1.png" title="Base image" alt="Base image" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>The fill image also has cloud cover, but less than the fill image - there are
areas of the fill that can be used to fill clouded pixels in the base image:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plotRGB</span><span class="p">(</span><span class="n">fill</span><span class="p">,</span><span class="w"> </span><span class="n">stretch</span><span class="o">=</span><span class="s1">'lin'</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/fill_image-1.png" title="Fill image" alt="Fill image" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<h3 id="topographic-correction">Topographic correction</h3>
<p>In mountainous areas, topographic correction should be performed prior to cloud
fill<sup id="fnref:1:1"><a href="#fn:1" class="footnote">1</a></sup>. <code class="highlighter-rouge">teamlucc</code> supports performing topographic correction using
algorithms derived from those in the <code class="highlighter-rouge">landsat</code>
package](http://cran.r-project.org/web/packages/landsat/index.html) by Sarah
Goslee<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup>.</p>
<p>To perform topographic correction, use the <code class="highlighter-rouge">topographic_corr</code> function in
<code class="highlighter-rouge">teamlucc</code>. First load the slope and aspect rasters:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">slp_asp</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">brick</span><span class="p">(</span><span class="s1">'vb_slp_asp.tif'</span><span class="p">)</span></code></pre></figure>
<p>Now call the <code class="highlighter-rouge">topographic_corr</code> function twice, to topographically correct both
the base and fill image. Note that the sun angle elevation and sun azimuth
(both in degrees) must be supplied - values for these parameters can be found
in the metadata accompanying your imagery. See <code class="highlighter-rouge">?topographic_corr</code> for more
information. <code class="highlighter-rouge">DN_min</code> and <code class="highlighter-rouge">DN_max</code> can be used to ensure that invalid values
are not generated by the topographic correction routine (which can sometimes be
a problem in very heavily shadowed areas, or in very bright areas, such as
clouds).</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">base_tc</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">topographic_corr</span><span class="p">(</span><span class="n">base</span><span class="p">,</span><span class="w"> </span><span class="n">slp_asp</span><span class="p">,</span><span class="w"> </span><span class="n">sunelev</span><span class="o">=</span><span class="m">90-47.34</span><span class="p">,</span><span class="w"> </span><span class="n">sunazimuth</span><span class="o">=</span><span class="m">134.04</span><span class="p">,</span><span class="w">
</span><span class="n">DN_min</span><span class="o">=</span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="n">DN_max</span><span class="o">=</span><span class="m">255</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning: executing %dopar% sequentially: no parallel backend registered</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">fill_tc</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">topographic_corr</span><span class="p">(</span><span class="n">fill</span><span class="p">,</span><span class="w"> </span><span class="n">slp_asp</span><span class="p">,</span><span class="w"> </span><span class="n">sunelev</span><span class="o">=</span><span class="m">90-46.80</span><span class="p">,</span><span class="w"> </span><span class="n">sunazimuth</span><span class="o">=</span><span class="m">129.88</span><span class="p">,</span><span class="w">
</span><span class="n">DN_min</span><span class="o">=</span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="n">DN_max</span><span class="o">=</span><span class="m">255</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plotRGB</span><span class="p">(</span><span class="n">base_tc</span><span class="p">,</span><span class="w"> </span><span class="n">stretch</span><span class="o">=</span><span class="s1">'lin'</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/base_tc-1.png" title="Base image after topographic correction" alt="Base image after topographic correction" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plotRGB</span><span class="p">(</span><span class="n">fill_tc</span><span class="p">,</span><span class="w"> </span><span class="n">stretch</span><span class="o">=</span><span class="s1">'lin'</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/fill_tc-1.png" title="Fill image after topographic correction" alt="Fill image after topographic correction" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<h3 id="construct-cloud-masks">Construct cloud masks</h3>
<p>The fmask band from the CDR imagery uses the following codes:</p>
<table>
<thead>
<tr>
<th>Pixel type</th>
<th style="text-align: center">Code</th>
</tr>
</thead>
<tbody>
<tr>
<td>Clear land</td>
<td style="text-align: center">0</td>
</tr>
<tr>
<td>Clear water</td>
<td style="text-align: center">1</td>
</tr>
<tr>
<td>Cloud shadow</td>
<td style="text-align: center">2</td>
</tr>
<tr>
<td>Snow</td>
<td style="text-align: center">3</td>
</tr>
<tr>
<td>Cloud</td>
<td style="text-align: center">4</td>
</tr>
<tr>
<td>No observation</td>
<td style="text-align: center">255</td>
</tr>
</tbody>
</table>
<p>We need to construct a mask of areas where all pixels that are cloud (code 4)
or cloud shadow (code 2) are equal to 1, and where pixels in all other areas
are equal to zero. This is easy using raster algebra from the R <code class="highlighter-rouge">raster</code>
package. First load the masks:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">base_fmask</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">raster</span><span class="p">(</span><span class="s1">'vb_1986_005_fmask.tif'</span><span class="p">)</span><span class="w">
</span><span class="n">fill_fmask</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">raster</span><span class="p">(</span><span class="s1">'vb_1986_021_fmask.tif'</span><span class="p">)</span></code></pre></figure>
<p>Now do the raster algebra, masking out clouds and cloud shadows, and setting
missing values in both images to NAs in the masks:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># Set mask to 1 in clouds and shadow areas</span><span class="w">
</span><span class="n">base_cloud_mask</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="p">(</span><span class="n">base_fmask</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">2</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">base_fmask</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">4</span><span class="p">)</span><span class="w">
</span><span class="n">fill_cloud_mask</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="p">(</span><span class="n">fill_fmask</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">2</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">fill_fmask</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">4</span><span class="p">)</span><span class="w">
</span><span class="c1"># Set mask to NA in background areas</span><span class="w">
</span><span class="n">base_cloud_mask</span><span class="p">[</span><span class="n">base_fmask</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">255</span><span class="p">]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NA</span><span class="w">
</span><span class="n">fill_cloud_mask</span><span class="p">[</span><span class="n">fill_fmask</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">255</span><span class="p">]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NA</span><span class="w">
</span><span class="c1"># Set mask to NA in other NA areas in imagery (NAs can result from topographic </span><span class="w">
</span><span class="c1"># correction, generally in very dark areas or areas of very high slope)</span><span class="w">
</span><span class="n">base_cloud_mask</span><span class="p">[</span><span class="nf">is.na</span><span class="p">(</span><span class="n">base_tc</span><span class="p">[[</span><span class="m">1</span><span class="p">]])]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NA</span><span class="w">
</span><span class="n">fill_cloud_mask</span><span class="p">[</span><span class="nf">is.na</span><span class="p">(</span><span class="n">fill_tc</span><span class="p">[[</span><span class="m">1</span><span class="p">]])]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NA</span></code></pre></figure>
<p>Plot these masks to double-check they align with the clouds in the images we
viewed earlier:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">base_cloud_mask</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/base_cloud_mask-1.png" title="Base image cloud mask" alt="Base image cloud mask" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">fill_cloud_mask</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/fill_cloud_mask-1.png" title="Fill image cloud mask" alt="Fill image cloud mask" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>Now use these two masks to mask out the clouds in the fill and base images, by
setting clouded areas to zero (as the <code class="highlighter-rouge">cloud_remove</code> code treats pixels with
zero values as “background”:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">base_tc</span><span class="p">[</span><span class="n">base_cloud_mask</span><span class="p">]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="m">0</span><span class="w">
</span><span class="n">fill_tc</span><span class="p">[</span><span class="n">fill_cloud_mask</span><span class="p">]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="m">0</span></code></pre></figure>
<p>The cloud mask for the base image must be constructed so that each cloud has
its own unique integer code, with codes starting from 1. This process can be
automated using the <code class="highlighter-rouge">ConnCompLabel</code> function from the <code class="highlighter-rouge">SDMTools</code> package.<br />
However, because there are clouds in our fill image as well as in our base
image, we need to modify the <code class="highlighter-rouge">base_cloud_mask</code> slightly to account for this.
First, code all pixels in <code class="highlighter-rouge">base_cloud_mask</code> that are clouded in
<code class="highlighter-rouge">fill_cloud_mask</code> with <code class="highlighter-rouge">NA</code>s. This will tell the <code class="highlighter-rouge">ConnCompLabel</code> function not
to label these pixels as clouds (because they are also clouded in the fill
image, we cannot perform cloud fill on these pixels).</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># Set clouds in fill image to NA in base mask:</span><span class="w">
</span><span class="n">base_cloud_mask</span><span class="p">[</span><span class="n">fill_cloud_mask</span><span class="p">]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NA</span><span class="w">
</span><span class="c1"># Set missing values in fill image to NA in base mask:</span><span class="w">
</span><span class="n">base_cloud_mask</span><span class="p">[</span><span class="nf">is.na</span><span class="p">(</span><span class="n">fill_cloud_mask</span><span class="p">)]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NA</span></code></pre></figure>
<p>Now run <code class="highlighter-rouge">ConnCompLabel</code>, and set the output datatype to <code class="highlighter-rouge">INT2S</code> (meaning the
data in <code class="highlighter-rouge">base_cloud_mask</code> can range from -32768 - 32767). That said, please
don’t try to run cloud fill with 32,767 clouds in your image :).</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">base_cloud_mask</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">ConnCompLabel</span><span class="p">(</span><span class="n">base_cloud_mask</span><span class="p">)</span></code></pre></figure>
<p>The final <code class="highlighter-rouge">base_cloud_mask</code> is now coded as:</p>
<table>
<thead>
<tr>
<th>Pixel type</th>
<th style="text-align: center">Code</th>
</tr>
</thead>
<tbody>
<tr>
<td>Background in <code class="highlighter-rouge">fill</code> or <code class="highlighter-rouge">base</code></td>
<td style="text-align: center">NA</td>
</tr>
<tr>
<td>Clouded in <code class="highlighter-rouge">fill</code></td>
<td style="text-align: center">-1</td>
</tr>
<tr>
<td>Clear in <code class="highlighter-rouge">base</code>, clear in <code class="highlighter-rouge">fill</code></td>
<td style="text-align: center">0</td>
</tr>
<tr>
<td>Clouded in <code class="highlighter-rouge">base</code></td>
<td style="text-align: center">1 … n</td>
</tr>
</tbody>
</table>
<p>where n is the number of clouds in the image:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">base_cloud_mask</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/base_cloud_mask_recoded-1.png" title="Final base image cloud mask" alt="Final base image cloud mask" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<h3 id="fill-clouds">Fill clouds</h3>
<p>For this simple example, we will directly use the <code class="highlighter-rouge">cloud_remove</code> function in
<code class="highlighter-rouge">teamlucc</code>. This function has a number of input parameters that can be supplied
(see <code class="highlighter-rouge">?cloud_remove</code>). Two important ones to note are <code class="highlighter-rouge">DN_min</code> and <code class="highlighter-rouge">DN_max</code>.<br />
These are the minimum and maximum valid values, respectively, that a pixel in
the image can take on. These limits are used to ignore unrealistic predictions
that may arise in the cloud fill routine. For the base and fill images we are
working with here, these values are 0 and 255, for max and min, respectively.<br />
Set these parameters to appropriate values as necessary for the images you are
working with.</p>
<p>There are three different cloud fill algorithms that can be used from
<code class="highlighter-rouge">teamlucc</code>. Two require an IDL installation, while the third uses a cloud fill
algorithm that is native to R (though it is coded in C++ for speed reasons).<br />
The R-based algorithm is a bit more flexible than the IDL algorithms, and is
designed to handle images in which both the base and fill image have clouds.
Based on the options supplied to <code class="highlighter-rouge">cloud_remove</code>, <code class="highlighter-rouge">teamlucc</code> will select one of
the fourt algorithms to run. The <code class="highlighter-rouge">algorithm</code> parameter to <code class="highlighter-rouge">cloud_remove</code>
determine which cloud fill algorithm is used:</p>
<table>
<thead>
<tr>
<th style="text-align: center">Algorithm</th>
<th style="text-align: center">Requires IDL license?</th>
<th style="text-align: center">Algorithm used by <code class="highlighter-rouge">cloud_remove</code></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><code class="highlighter-rouge">CLOUD_REMOVE</code></td>
<td style="text-align: center">Yes</td>
<td style="text-align: center">CLOUD_REMOVE<sup id="fnref:1:2"><a href="#fn:1" class="footnote">1</a></sup></td>
</tr>
<tr>
<td style="text-align: center"><code class="highlighter-rouge">CLOUD_REMOVE_FAST</code></td>
<td style="text-align: center">Yes</td>
<td style="text-align: center">CLOUD_REMOVE_FAST<sup id="fnref:1:3"><a href="#fn:1" class="footnote">1</a></sup></td>
</tr>
<tr>
<td style="text-align: center"><code class="highlighter-rouge">teamlucc</code></td>
<td style="text-align: center">No</td>
<td style="text-align: center"><code class="highlighter-rouge">teamlucc</code> fill algorithm</td>
</tr>
<tr>
<td style="text-align: center"><code class="highlighter-rouge">simple</code></td>
<td style="text-align: center">No</td>
<td style="text-align: center">simple linear model algorithm</td>
</tr>
</tbody>
</table>
<p>First I will review the two IDL-based algorithms, then I will discuss the two
R-based algorithms.</p>
<h4 id="cloud-removal-using-idl-code">Cloud removal using IDL code</h4>
<p>If run with <code class="highlighter-rouge">algorithm="CLOUD_REMOVE"</code> (the default), <code class="highlighter-rouge">cloud_remove</code> runs an
IDL script provided by <a href="http://geography.osu.edu/grads/xzhu/">Xiaolin Zhu</a>. For
R to be able to run this script it must know the path to IDL on your machine.
For Windows users, this means the path to “idl.exe”. To specify this path you
will need to provide the <code class="highlighter-rouge">idl</code> parameter to the <code class="highlighter-rouge">cloud_remove</code> script. The
default value (<code class="highlighter-rouge">C:/Program Files/Exelis/IDL83/bin/bin.x86_64/idl.exe</code>) may or
may not work on your machine. I recommend you set the IDL path at the beginning
of your scripts:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">idl_path</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s2">"C:/Program Files/Exelis/IDL83/bin/bin.x86_64/idl.exe"</span></code></pre></figure>
<p>An optional <code class="highlighter-rouge">out_name</code> parameter can be supplied to <code class="highlighter-rouge">cloud_remove</code> to specify
the filename for the output file. If not supplied, R will save the filled image
as an R object pointing to a temporary file.</p>
<p>To run the cloud removal routine, call the <code class="highlighter-rouge">cloud_remove</code> function with the
appropriate parameters. Note that this computation may take some time (it takes
around 1.5 hours on a 2.9Ghz Core-i7 3520M laptop).</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># Takes 2-3 hours on a 2.9Ghz Core-i7 3520M laptop</span><span class="w">
</span><span class="n">start_time</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">Sys.time</span><span class="p">()</span><span class="w">
</span><span class="c1"># Ensure dataType is properly set prior to handing off to IDL</span><span class="w">
</span><span class="n">dataType</span><span class="p">(</span><span class="n">base_cloud_mask</span><span class="p">)</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s1">'INT2S'</span><span class="w">
</span><span class="n">filled_cr</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">cloud_remove</span><span class="p">(</span><span class="n">base_tc</span><span class="p">,</span><span class="w"> </span><span class="n">fill_tc</span><span class="p">,</span><span class="w"> </span><span class="n">base_cloud_mask</span><span class="p">,</span><span class="w">
</span><span class="n">algorithm</span><span class="o">=</span><span class="s2">"CLOUD_REMOVE"</span><span class="p">,</span><span class="w"> </span><span class="n">DN_min</span><span class="o">=</span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="n">DN_max</span><span class="o">=</span><span class="m">255</span><span class="p">,</span><span class="w">
</span><span class="n">idl</span><span class="o">=</span><span class="n">idl_path</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: ncdf</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">Sys.time</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">start_time</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Time difference of 1.637638 hours</code></pre></figure>
<p>Use <code class="highlighter-rouge">plotRGB</code> to check the output. Note that IDL does not properly code missing
values in the output - prior to plotting or working with the data be sure to
set any pixels with values less than <code class="highlighter-rouge">DN_min</code> (here <code class="highlighter-rouge">DN_min</code> is zero) to <code class="highlighter-rouge">NA</code>:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">filled_cr</span><span class="p">[</span><span class="n">filled_cr</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="m">0</span><span class="p">]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NA</span><span class="w">
</span><span class="n">plotRGB</span><span class="p">(</span><span class="n">filled_cr</span><span class="p">,</span><span class="w"> </span><span class="n">stretch</span><span class="o">=</span><span class="s2">"lin"</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/cloud_remove_cr_plot-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>The default cloud fill approach can take a considerable amount of time to run.<br />
There is an alternative approach that can take considerably less time to run,
with similar results. This option can be enabled by supplying the
<code class="highlighter-rouge">algorithm="CLOUD_REMOVE_FAST</code> parameter to <code class="highlighter-rouge">cloud_remove</code>.</p>
<p>The “fast” version of the algorithm makes some simplifications to improve
running time. Specifically, rather than follow the precise algorithm as
outlined by Zhu et al.<sup id="fnref:1:4"><a href="#fn:1" class="footnote">1</a></sup>, the “fast” routine uses k-means clustering to
divide the image into the number of classes specified by the <code class="highlighter-rouge">num_class</code>
parameter. The script then constructs a linear model of the temporal change in
reflectance for each class within the neighborhood of a given cloud. This
“temporal” adjustment is complemented by a “spatial” adjustment that considers
the change in reflectance in a small neighborhood around each clouded pixel.
For each clouded pixel, a weighted combination of the predicted fill values
from the spatial and temporal models determines the final predicted value for
that pixel. This version of the algorithm takes only 2.5 minutes to run on the
same machine as used above:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># Takes 2-3 minutes on a 2.9Ghz Core-i7 3520M laptop</span><span class="w">
</span><span class="n">start_time</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">Sys.time</span><span class="p">()</span><span class="w">
</span><span class="c1"># Ensure dataType is properly set prior to handing off to IDL</span><span class="w">
</span><span class="n">dataType</span><span class="p">(</span><span class="n">base_cloud_mask</span><span class="p">)</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s1">'INT2S'</span><span class="w">
</span><span class="n">filled_crf</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">cloud_remove</span><span class="p">(</span><span class="n">base_tc</span><span class="p">,</span><span class="w"> </span><span class="n">fill_tc</span><span class="p">,</span><span class="w"> </span><span class="n">base_cloud_mask</span><span class="p">,</span><span class="w">
</span><span class="n">algorithm</span><span class="o">=</span><span class="s2">"CLOUD_REMOVE_FAST"</span><span class="p">,</span><span class="w"> </span><span class="n">DN_min</span><span class="o">=</span><span class="m">0</span><span class="p">,</span><span class="w">
</span><span class="n">DN_max</span><span class="o">=</span><span class="m">255</span><span class="p">,</span><span class="w"> </span><span class="n">idl</span><span class="o">=</span><span class="n">idl_path</span><span class="p">)</span><span class="w">
</span><span class="n">Sys.time</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">start_time</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Time difference of 55.48255 secs</code></pre></figure>
<p>Use <code class="highlighter-rouge">plotRGB</code> to check the output:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">filled_crf</span><span class="p">[</span><span class="n">filled_crf</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="m">0</span><span class="p">]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NA</span><span class="w">
</span><span class="n">plotRGB</span><span class="p">(</span><span class="n">filled_crf</span><span class="p">,</span><span class="w"> </span><span class="n">stretch</span><span class="o">=</span><span class="s1">'lin'</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/cloud_remove_crf_plot-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<h4 id="cloud-removal-using-native-r-code">Cloud removal using native R code</h4>
<p>If you do not have IDL on your machine, there is a C++ implementation of the
NSPI cloud fill algorithm that will run directly in R, as well as a “simple”
cloud fill algorithm that uses linear models developed using the neighborhood
of each cloud to perform a naive fill. To run the R version of the NSPI
algorithm, call the <code class="highlighter-rouge">cloud_remove</code> function with the same parameters as above,
but specify <code class="highlighter-rouge">algorithm="teamlucc"</code>. This function also has a <code class="highlighter-rouge">verbose=TRUE</code>
option to tell <code class="highlighter-rouge">cloud_remove</code> to print progress statements as it is running
(this option is not available with the IDL scripts shown above). This version
is nearly identical to the IDL algorithm called with the
<code class="highlighter-rouge">algorithm="CLOUD_REMOVE"</code> option, but it takes much less time to run (only 3-4 minutes on my machine).</p>
<p>Note that when <code class="highlighter-rouge">cloud_remove</code> is run with <code class="highlighter-rouge">algorithm="teamlucc"</code> and
<code class="highlighter-rouge">verbose=TRUE</code>, there will be a large number of status messages printed to the
screen. For the purposes of this demo (so that the webpage is not unnecessarily
long), I have not used the <code class="highlighter-rouge">verbose=TRUE</code> argument, but I recommend using it if
you try this command yourself.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># Takes 4-5 minutes on a 2.9Ghz Core-i7 3520M laptop</span><span class="w">
</span><span class="n">start_time</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">Sys.time</span><span class="p">()</span><span class="w">
</span><span class="n">filled_tl</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">cloud_remove</span><span class="p">(</span><span class="n">base_tc</span><span class="p">,</span><span class="w"> </span><span class="n">fill_tc</span><span class="p">,</span><span class="w"> </span><span class="n">base_cloud_mask</span><span class="p">,</span><span class="w"> </span><span class="n">DN_min</span><span class="o">=</span><span class="m">0</span><span class="p">,</span><span class="w">
</span><span class="n">DN_max</span><span class="o">=</span><span class="m">255</span><span class="p">,</span><span class="w"> </span><span class="n">algorithm</span><span class="o">=</span><span class="s2">"teamlucc"</span><span class="p">)</span><span class="w">
</span><span class="n">Sys.time</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">start_time</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Time difference of 3.320532 mins</code></pre></figure>
<p>View the results with <code class="highlighter-rouge">plotRGB</code>:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plotRGB</span><span class="p">(</span><span class="n">filled_tl</span><span class="p">,</span><span class="w"> </span><span class="n">stretch</span><span class="o">=</span><span class="s1">'lin'</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/cloud_remove_tl_plot-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>The fastest cloud fill option is to run <code class="highlighter-rouge">cloud_remove</code> with
<code class="highlighter-rouge">algorithm="SIMPLE"</code>. This uses a simple cloud fill approach in which the value
of each clouded pixel is calculated using a linear model. The script develops a
separate linear model (with slope and intercept) for each band and each cloud.
For each cloud, and each image band, the script finds all pixels clear in both
the cloudy and fill images, and calculates a regression model in which pixel
values in the fill image are the independent variable, and pixel values in the
clouded image are the dependent variable. The script then uses this model to
predict pixel values for each band in each cloud in the clouded image. For
example:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># Takes 2-5 seconds on a 2.9Ghz Core-i7 3520M laptop</span><span class="w">
</span><span class="n">start_time</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">Sys.time</span><span class="p">()</span><span class="w">
</span><span class="n">filled_simple</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">cloud_remove</span><span class="p">(</span><span class="n">base_tc</span><span class="p">,</span><span class="w"> </span><span class="n">fill_tc</span><span class="p">,</span><span class="w"> </span><span class="n">base_cloud_mask</span><span class="p">,</span><span class="w"> </span><span class="n">DN_min</span><span class="o">=</span><span class="m">0</span><span class="p">,</span><span class="w">
</span><span class="n">DN_max</span><span class="o">=</span><span class="m">255</span><span class="p">,</span><span class="w"> </span><span class="n">algorithm</span><span class="o">=</span><span class="s2">"simple"</span><span class="p">)</span><span class="w">
</span><span class="n">Sys.time</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">start_time</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Time difference of 0.6300631 secs</code></pre></figure>
<p>View the results with <code class="highlighter-rouge">plotRGB</code>:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plotRGB</span><span class="p">(</span><span class="n">filled_simple</span><span class="p">,</span><span class="w"> </span><span class="n">stretch</span><span class="o">=</span><span class="s1">'lin'</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/cloud_remove_simple_plot-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<h3 id="compare-all-four-fill-algorithms">Compare all four fill algorithms:</h3>
<p>To plot the results of all four fill algorithms, make a layer stack of the
first band of all four images, then plot:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">filled_comp</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">stack</span><span class="p">(</span><span class="n">filled_cr</span><span class="p">[[</span><span class="m">1</span><span class="p">]],</span><span class="w"> </span><span class="n">filled_crf</span><span class="p">[[</span><span class="m">1</span><span class="p">]],</span><span class="w"> </span><span class="n">filled_tl</span><span class="p">[[</span><span class="m">1</span><span class="p">]],</span><span class="w">
</span><span class="n">filled_simple</span><span class="p">[[</span><span class="m">1</span><span class="p">]])</span><span class="w">
</span><span class="nf">names</span><span class="p">(</span><span class="n">filled_comp</span><span class="p">)</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="s1">'CLOUD_REMOVE'</span><span class="p">,</span><span class="w"> </span><span class="s1">'CLOUD_REMOVE_FAST'</span><span class="p">,</span><span class="w"> </span><span class="s1">'teamlucc'</span><span class="p">,</span><span class="w">
</span><span class="s1">'simple'</span><span class="p">)</span><span class="w">
</span><span class="n">filled_comp</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">linear_stretch</span><span class="p">(</span><span class="n">filled_comp</span><span class="p">,</span><span class="w"> </span><span class="n">pct</span><span class="o">=</span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="n">max_val</span><span class="o">=</span><span class="m">255</span><span class="p">)</span><span class="w">
</span><span class="n">plot</span><span class="p">(</span><span class="n">filled_comp</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-04-16-cloud-removal-with-teamlucc/cloud_remove_comparison_plot-1.png" title="center" alt="center" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<h2 id="automated-cloud-fill-from-an-image-time-series">Automated cloud fill from an image time series</h2>
<p>The <code class="highlighter-rouge">teamlucc</code> package also includes functions for automated cloud filling from
an image time series. Automatic cloud filling is performed using the
<code class="highlighter-rouge">auto_cloud_fill</code> function. This function automates the majority of the cloud
filling process. As multiple images are required to demonstrate this process,
the images required for this portion of the example are not available for
download from this site. I suggest you download the appropriate imagery for a
particular study site and preprocess the imagery using the <code class="highlighter-rouge">auto_setup_dem</code> and
<code class="highlighter-rouge">auto_preprocess_landsat</code> functions in the <code class="highlighter-rouge">teamlucc</code> package so that you can
follow along with this example. The <code class="highlighter-rouge">auto_preprocess_landsat</code> function will
also perform topographic correction, which is necessary prior to cloud filling
images in mountainous areas.</p>
<p>The <code class="highlighter-rouge">auto_cloud_fill</code> function allows an analyst to automatically construct a
cloud-filled image after specifying: <code class="highlighter-rouge">data_dir</code> (a folder of Landsat
images), <code class="highlighter-rouge">wrspath</code> and <code class="highlighter-rouge">wrsrow</code> (the WRS-2 path/row to use), and <code class="highlighter-rouge">start_date</code>
and <code class="highlighter-rouge">end_date</code> (a start and end date limiting the images to use in the
algorithm). The analyst can also optionally specify a <code class="highlighter-rouge">base_date</code>, and the
<code class="highlighter-rouge">auto_cloud_fill</code> function will automatically pick the image closest to that
date to use as the base image (otherwise <code class="highlighter-rouge">auto_cloud_fill</code> will automatically
pick the image with the least cloud cover as the base image).</p>
<p>As the <code class="highlighter-rouge">auto_cloud_fill</code> function automatically chooses images for inclusion in
the cloud fill process, it relies on having images stored on disk in a
particular way, and currently only supports cloud fill for Landsat CDR surface
reflectance images. To ensure that images are correctly stored on your hard
disk, use the <code class="highlighter-rouge">auto_preprocess_landsat</code> function to extract the original
Landsat CDR hdf files from the USGS archive. The <code class="highlighter-rouge">auto_preprocess_landsat</code>
function will ensure that images are extracted and renamed properly so that
they can be used with the <code class="highlighter-rouge">auto_cloud_fill</code> script.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># start_time <- Sys.time()</span><span class="w">
</span><span class="c1"># start_date <- as.Date('1986-01-01')</span><span class="w">
</span><span class="c1"># end_date <- as.Date('1987-01-01')</span><span class="w">
</span><span class="c1"># filled_image <- auto_cloud_fill("C:/Data/LEDAPS_imagery", wrspath=230, </span><span class="w">
</span><span class="c1"># wrsrow=62, start_date=start_date,</span><span class="w">
</span><span class="c1"># end_date=end_date)</span><span class="w">
</span><span class="c1"># Sys.time() - start_time</span></code></pre></figure>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Zhu, X., Gao, F., Liu, D., Chen, J., 2012. A modified neighborhood similar
pixel interpolator approach for removing thick clouds in Landsat images.
Geoscience and Remote Sensing Letters, IEEE 9, 521–525.
doi:10.1109/LGRS.2011.2173290 <a href="#fnref:1" class="reversefootnote">↩</a> <a href="#fnref:1:1" class="reversefootnote">↩<sup>2</sup></a> <a href="#fnref:1:2" class="reversefootnote">↩<sup>3</sup></a> <a href="#fnref:1:3" class="reversefootnote">↩<sup>4</sup></a> <a href="#fnref:1:4" class="reversefootnote">↩<sup>5</sup></a></p>
</li>
<li id="fn:2">
<p>Zhu, Z. and Woodcock, C. E., Object-based cloud and cloud shadow detection
in Landsat imagery, Remote Sensing of Environment (2012),
doi:10.1016/j.rse.2011.10.028 <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p><a href="http://www.jstatsoft.org/v43/i04/.">Sarah C. Goslee (2011). Analyzing Remote Sensing Data in R: The landsat
Package. Journal of Statistical Software, 43(4),
1-25.</a> <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="http://www.azvoleff.com/articles/cloud-removal-with-teamlucc/">Cloud removal with teamlucc</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on April 16, 2014.</p>http://www.azvoleff.com/articles/gfcanalysis-1-0-on-cran2014-04-01T00:00:00+00:002014-04-01T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>Version 1.0 of the <code class="highlighter-rouge">gfcanalysis</code> R package for working with the Hansen et al.<br />
2013<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> <a href="http://earthenginepartners.appspot.com/science-2013-global-forest">Global Forest Change
dataset</a> is
now on CRAN. See the <a href="http://cran.r-project.org/web/packages/gfcanalysis/index.html"><code class="highlighter-rouge">gfcanalysis</code> page on
CRAN</a> for more
information.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Hansen, M. C., P. V. Potapov, R. Moore, M. Hancher, S. A. Turubanova, A.
Tyukavina, D. Thau, S. V. Stehman, S. J. Goetz, T. R. Loveland, A. Kommareddy,
A. Egorov, L. Chini, C. O. Justice, and J. R. G. Townshend. 2013.
High-Resolution Global Maps of 21st-Century Forest Cover Change. Science 342,
(15 November): 850–853. Data available on-line from:
http://earthenginepartners.appspot.com/science-2013-global-forest. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="http://www.azvoleff.com/articles/gfcanalysis-1-0-on-cran/">gfcanalysis 1.0 released</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on April 01, 2014.</p>http://www.azvoleff.com/articles/analyzing-forest-change-with-gfcanalysis2014-03-25T00:00:00+00:002014-03-25T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<h2 id="overview">Overview</h2>
<p>This <code class="highlighter-rouge">gfcanalysis</code> R package facilitates simple analyses using the Hansen et
al. 2013<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> <a href="http://earthenginepartners.appspot.com/science-2013-global-forest">Global Forest Change
dataset</a>.
The package was written to analyze forest change in within the Zone of
Interaction surrounding each of the forest monitoring sites of the <a href="http://www.teamnetwork.org">Tropical
Ecology Assessment and Monitoring (TEAM) Network</a>.</p>
<p>If you need help with any of the functions in the package, see the help files
for more information. For example, type <code class="highlighter-rouge">?download_tiles</code> in R to see the help
file for the <code class="highlighter-rouge">download_tiles</code> function.</p>
<h2 id="getting-started">Getting started</h2>
<p>This post will outline an analysis using the <code class="highlighter-rouge">gfcanalysis</code> package. Note that
as the computations are intensive, some parts of this analysis may take some
time to run (about 30 minutes total to run all of the code outlined here). If
you do not already have the GFC product data downloaded on your computer,
downloading the dataset will also take some time (though this process is
automated by <code class="highlighter-rouge">gfcanalysis</code>).</p>
<p>To get started, first install the <code class="highlighter-rouge">gfcanalysis</code> package from CRAN. Also install
the <code class="highlighter-rouge">rgdal</code> package needed for reading/writing shapefiles if you do not already
have it.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">require</span><span class="p">(</span><span class="n">gfcanalysis</span><span class="p">))</span><span class="w"> </span><span class="n">install.packages</span><span class="p">(</span><span class="s1">'gfcanalysis'</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: gfcanalysis
## Loading required package: raster
## Loading required package: sp</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">require</span><span class="p">(</span><span class="n">rgdal</span><span class="p">))</span><span class="w"> </span><span class="n">install.packages</span><span class="p">(</span><span class="s1">'rgdal'</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: rgdal
## rgdal: version: 0.8-16, (SVN revision 498)
## Geospatial Data Abstraction Library extensions to R successfully loaded
## Loaded GDAL runtime: GDAL 1.10.1, released 2013/08/26
## Path to GDAL shared files: C:/Users/azvoleff/R/win-library/3.2/rgdal/gdal
## GDAL does not use iconv for recoding strings.
## Loaded PROJ.4 runtime: Rel. 4.8.0, 6 March 2012, [PJ_VERSION: 480]
## Path to PROJ.4 shared files: C:/Users/azvoleff/R/win-library/3.2/rgdal/proj</code></pre></figure>
<p>Indicate where we want to save GFC tiles downloaded from Google. For any given
AOI, the script will first check to see if these tiles are available locally
(in the below folder) before downloading them from the server - so I recommend
storing ALL of your GFC tiles in the same folder. For this example we will use
“.” - the current working directory of the R session.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">output_folder</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s2">"."</span></code></pre></figure>
<p>Set the threshold for forest/non-forest based on the treecover2000 layer in
the GFC product:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">forest_threshold</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="m">90</span></code></pre></figure>
<h2 id="downloading-data-from-google-server-for-a-given-aoi">Downloading data from Google server for a given AOI</h2>
<p>Load an area of interest. For this example we use a shapefile of the Zone of
Interaction (ZOI) of the <a href="http://www.teamnetwork.org">TEAM Network</a> site in
<a href="http://www.teamnetwork.org/network/sites/nam-kading-0">Nam Kading National Protected Area,
Laos</a>. Notice that first
we specify the folder the shapefile is in (here it is a “.” indicating the
current working directory), and then the name of the shapefile without the
“.shp”. To follow along with this example, <a href="/content/2014-03-25-analyzing-forest-change-with-gfcanalysis/ZOI_NAK_2012_EEsimple.zip">download this shapefile of the
ZOI</a>.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">aoi</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">readOGR</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span><span class="w"> </span><span class="s1">'ZOI_NAK_2012_EEsimple'</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## OGR data source with driver: ESRI Shapefile
## Source: ".", layer: "ZOI_NAK_2012_EEsimple"
## with 1 features and 3 fields
## Feature type: wkbPolygon with 2 dimensions</code></pre></figure>
<p>Calculate the tiles needed to cover the AOI:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">tiles</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">calc_gfc_tiles</span><span class="p">(</span><span class="n">aoi</span><span class="p">)</span><span class="w">
</span><span class="n">print</span><span class="p">(</span><span class="nf">length</span><span class="p">(</span><span class="n">tiles</span><span class="p">))</span><span class="w"> </span><span class="c1"># Number of tiles needed to cover AOI</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## [1] 1</code></pre></figure>
<p>To check the overlap between the tiles and the aoi, you can make a plot of the
needed tiles and the AOI using R’s plotting functions:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">tiles</span><span class="p">)</span><span class="w">
</span><span class="n">plot</span><span class="p">(</span><span class="n">aoi</span><span class="p">,</span><span class="w"> </span><span class="n">add</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">,</span><span class="w"> </span><span class="n">lty</span><span class="o">=</span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="n">col</span><span class="o">=</span><span class="s2">"#00ff0050"</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-03-25-analyzing-forest-change-with-gfcanalysis/tiles_versus_aoi.png" alt="center" /></p>
<p>Now, check to see if these tiles are already present locally, and download them
if they are not. By default the “first” and “last” composite surface
reflectance images are not downloaded. To also download these images specify
<code class="highlighter-rouge">first_and_last=TRUE</code>.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">download_tiles</span><span class="p">(</span><span class="n">tiles</span><span class="p">,</span><span class="w"> </span><span class="n">output_folder</span><span class="p">,</span><span class="w"> </span><span class="n">first_and_last</span><span class="o">=</span><span class="kc">FALSE</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## 1 tiles to download/check.
## 0 file(s) succeeded, 5 file(s) skipped, 0 file(s) failed.</code></pre></figure>
<h2 id="performing-thresholding-and-calculating-basic-statistics">Performing thresholding and calculating basic statistics</h2>
<p>Extract the GFC data for this AOI from the downloaded GFC tiles, mosaicing
multiple tiles as necessary (if needed to cover the AOI). Save this extract in
GeoTiff format in the current working directory (can also save as ENVI format,
Erdas format, etc.)</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">gfc_extract</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">extract_gfc</span><span class="p">(</span><span class="n">aoi</span><span class="p">,</span><span class="w"> </span><span class="n">output_folder</span><span class="p">,</span><span class="w"> </span><span class="n">filename</span><span class="o">=</span><span class="s2">"NAK_GFC_extract.tif"</span><span class="p">)</span></code></pre></figure>
<p>The extracted dataset has 5 layers (not yet thresholded):</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">gfc_extract</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## class : RasterBrick
## dimensions : 4358, 4761, 20748438, 5 (nrow, ncol, ncell, nlayers)
## resolution : 0.0002778, 0.0002778 (x, y)
## extent : 103.5, 104.8, 17.83, 19.04 (xmin, xmax, ymin, ymax)
## coord. ref. : +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0
## data source : C:\Users\azvoleff\Code\Misc\azvoleff.github.io\Rmd\2014-03-25-analyzing-forest-change-with-gfcanalysis\NAK_GFC_extract.tif
## names : treecover2000, loss, gain, lossyear, datamask
## min values : 0, 0, 0, 0, 1
## max values : 100, 1, 1, 12, 2</code></pre></figure>
<p>Threshold the GFC data based on a specified percent cover threshold (0-100),
and save the thresholded layers to a GeoTiff:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">gfc_thresholded</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">threshold_gfc</span><span class="p">(</span><span class="n">gfc_extract</span><span class="p">,</span><span class="w"> </span><span class="n">forest_threshold</span><span class="o">=</span><span class="n">forest_threshold</span><span class="p">,</span><span class="w">
</span><span class="n">filename</span><span class="o">=</span><span class="s2">"NAK_GFC_extract_thresholded.tif"</span><span class="p">)</span></code></pre></figure>
<h2 id="coding-of-the-thresholded-output">Coding of the thresholded output</h2>
<p>The thresholded dataset has 5 layers:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">gfc_thresholded</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## class : RasterBrick
## dimensions : 4358, 4761, 20748438, 5 (nrow, ncol, ncell, nlayers)
## resolution : 0.0002778, 0.0002778 (x, y)
## extent : 103.5, 104.8, 17.83, 19.04 (xmin, xmax, ymin, ymax)
## coord. ref. : +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0
## data source : C:\Users\azvoleff\Code\Misc\azvoleff.github.io\Rmd\2014-03-25-analyzing-forest-change-with-gfcanalysis\NAK_GFC_extract_thresholded.tif
## names : forest2000, lossyear, gain, lossgain, datamask
## min values : 0, 0, 0, 0, 1
## max values : 1, 12, 1, 1, 2</code></pre></figure>
<p>The output is coded using the following coding scheme:</p>
<h3 id="band-1-forest2000">Band 1 (forest2000)</h3>
<p>Based on the provided <code class="highlighter-rouge">forest_threshold</code>. Pixels wiwth percent canopy cover
greater than <code class="highlighter-rouge">forest_threshold</code> are coded as forest.</p>
<table>
<thead>
<tr>
<th style="text-align: left">Cover in 2000</th>
<th style="text-align: center">Code</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">Non-forest</td>
<td style="text-align: center">0</td>
</tr>
<tr>
<td style="text-align: left">Forest</td>
<td style="text-align: center">1</td>
</tr>
</tbody>
</table>
<h3 id="band-2-lossyear">Band 2 (lossyear)</h3>
<p>Note that lossyear is zero for pixels that were not forested in 2000.</p>
<table>
<thead>
<tr>
<th style="text-align: left">Year of loss</th>
<th style="text-align: center">Code</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">No loss</td>
<td style="text-align: center">0</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2001</td>
<td style="text-align: center">1</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2002</td>
<td style="text-align: center">2</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2003</td>
<td style="text-align: center">3</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2004</td>
<td style="text-align: center">4</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2005</td>
<td style="text-align: center">5</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2006</td>
<td style="text-align: center">6</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2007</td>
<td style="text-align: center">7</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2008</td>
<td style="text-align: center">8</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2009</td>
<td style="text-align: center">9</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2010</td>
<td style="text-align: center">10</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2011</td>
<td style="text-align: center">11</td>
</tr>
<tr>
<td style="text-align: left">Loss in 2012</td>
<td style="text-align: center">12</td>
</tr>
</tbody>
</table>
<h3 id="band-3-gain">Band 3 (gain)</h3>
<p>Note that gain is zero for pixels that were forested in 2000.</p>
<table>
<thead>
<tr>
<th style="text-align: left">Change</th>
<th style="text-align: center">Code</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">No gain</td>
<td style="text-align: center">0</td>
</tr>
<tr>
<td style="text-align: left">Gain</td>
<td style="text-align: center">1</td>
</tr>
</tbody>
</table>
<h3 id="band-4-lossgain">Band 4 (lossgain)</h3>
<p>Note that loss and gain is difficult to interpret from the thresholded
product, as the original GFC product does not provide information on the
sequence of loss and gain (loss then gain, or gain then loss). The product also
does not provide information on the levels of canopy cover reached prior to
loss (for gain then loss) or after loss (for loss then gain pixels). The layer
is calculated here as: <code class="highlighter-rouge">lossgain <- gain & (lossyear != 0)</code>, where <code class="highlighter-rouge">lossyear</code>
and <code class="highlighter-rouge">gain</code> are the original GFC gain and lossyear layers, respectively.</p>
<table>
<thead>
<tr>
<th style="text-align: left">Change</th>
<th style="text-align: center">Code</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">No loss and gain</td>
<td style="text-align: center">0</td>
</tr>
<tr>
<td style="text-align: left">Loss and gain</td>
<td style="text-align: center">1</td>
</tr>
</tbody>
</table>
<h3 id="band-5-datamask">Band 5 (datamask)</h3>
<table>
<thead>
<tr>
<th style="text-align: left">Class</th>
<th style="text-align: center">Code</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">No data</td>
<td style="text-align: center">0</td>
</tr>
<tr>
<td style="text-align: left">Land</td>
<td style="text-align: center">1</td>
</tr>
<tr>
<td style="text-align: left">Water</td>
<td style="text-align: center">2</td>
</tr>
</tbody>
</table>
<h2 id="calculating-statistics-on-forest-loss-and-forest-gain">Calculating statistics on forest loss and forest gain</h2>
<p>Calculate annual statistics on forest loss/gain:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">gfc_stats</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">gfc_stats</span><span class="p">(</span><span class="n">aoi</span><span class="p">,</span><span class="w"> </span><span class="n">gfc_thresholded</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Data appears to be in latitude/longitude. Calculating cell areas on a sphere.</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">gfc_stats</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## $loss_table
## year aoi cover loss
## 1 2000 AOI 1 433656 NA
## 2 2001 AOI 1 433125 530.6
## 3 2002 AOI 1 432108 1017.4
## 4 2003 AOI 1 430451 1656.4
## 5 2004 AOI 1 429590 861.6
## 6 2005 AOI 1 427738 1851.3
## 7 2006 AOI 1 425938 1800.2
## 8 2007 AOI 1 421196 4742.1
## 9 2008 AOI 1 420032 1164.0
## 10 2009 AOI 1 415982 4049.5
## 11 2010 AOI 1 412196 3786.5
## 12 2011 AOI 1 407462 4734.3
## 13 2012 AOI 1 403578 3884.1
##
## $gain_table
## period aoi gain lossgain
## 1 2000-2012 AOI 1 16194 12287</code></pre></figure>
<p>Save these statistics to CSV files for use in Excel, or other software:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">write.csv</span><span class="p">(</span><span class="n">gfc_stats</span><span class="o">$</span><span class="n">loss_table</span><span class="p">,</span><span class="w">
</span><span class="n">file</span><span class="o">=</span><span class="s1">'NAK_GFC_extract_thresholded_losstable.csv'</span><span class="p">,</span><span class="w"> </span><span class="n">row.names</span><span class="o">=</span><span class="kc">FALSE</span><span class="p">)</span><span class="w">
</span><span class="n">write.csv</span><span class="p">(</span><span class="n">gfc_stats</span><span class="o">$</span><span class="n">gain_table</span><span class="p">,</span><span class="w">
</span><span class="n">file</span><span class="o">=</span><span class="s1">'NAK_GFC_extract_thresholded_gaintable.csv'</span><span class="p">,</span><span class="w"> </span><span class="n">row.names</span><span class="o">=</span><span class="kc">FALSE</span><span class="p">)</span></code></pre></figure>
<p>To view the output format of the CSV files output by <code class="highlighter-rouge">gfcanalysis</code>, see the
<a href="/content/2014-03-25-analyzing-forest-change-with-gfcanalysis/NAK_GFC_extract_thresholded_gaintable.csv">loss
table</a>
and <a href="/content/2014-03-25-analyzing-forest-change-with-gfcanalysis/NAK_GFC_extract_thresholded_gaintable.csv">gain
table</a>
for Nam Kading.</p>
<h2 id="making-simple-visualizations">Making simple visualizations</h2>
<p>There is also a function in <code class="highlighter-rouge">gfcanalysis</code> to calculate and save a thresholded
annual layer stack from the GFC product (useful for simple visualizations,
etc.):</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">gfc_annual_stack</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">annual_stack</span><span class="p">(</span><span class="n">gfc_thresholded</span><span class="p">)</span><span class="w">
</span><span class="n">writeRaster</span><span class="p">(</span><span class="n">gfc_annual_stack</span><span class="p">,</span><span class="w"> </span><span class="n">filename</span><span class="o">=</span><span class="s2">"NAK_GFC_extract_thresholded_annual.tif"</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## class : RasterBrick
## dimensions : 4358, 4761, 20748438, 13 (nrow, ncol, ncell, nlayers)
## resolution : 0.0002778, 0.0002778 (x, y)
## extent : 103.5, 104.8, 17.83, 19.04 (xmin, xmax, ymin, ymax)
## coord. ref. : +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0
## data source : C:\Users\azvoleff\Code\Misc\azvoleff.github.io\Rmd\2014-03-25-analyzing-forest-change-with-gfcanalysis\NAK_GFC_extract_thresholded_annual.tif
## names : NAK_GFC_e//d_annual.1, NAK_GFC_e//d_annual.2, NAK_GFC_e//d_annual.3, NAK_GFC_e//d_annual.4, NAK_GFC_e//d_annual.5, NAK_GFC_e//d_annual.6, NAK_GFC_e//d_annual.7, NAK_GFC_e//d_annual.8, NAK_GFC_e//d_annual.9, NAK_GFC_e//_annual.10, NAK_GFC_e//_annual.11, NAK_GFC_e//_annual.12, NAK_GFC_e//_annual.13
## min values : 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
## max values : 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6</code></pre></figure>
<p>The annual stack output by <code class="highlighter-rouge">annual_stack</code> has one layer for each year:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">gfc_annual_stack</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## class : RasterBrick
## dimensions : 4358, 4761, 20748438, 13 (nrow, ncol, ncell, nlayers)
## resolution : 0.0002778, 0.0002778 (x, y)
## extent : 103.5, 104.8, 17.83, 19.04 (xmin, xmax, ymin, ymax)
## coord. ref. : +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0
## data source : C:\Users\azvoleff\AppData\Local\Temp\R_raster_azvoleff\raster_tmp_2014-04-04_141034_13628_76953.grd
## names : y2000, y2001, y2002, y2003, y2004, y2005, y2006, y2007, y2008, y2009, y2010, y2011, y2012
## min values : 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
## max values : 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6
## time : 2000-01-01, 2001-01-01, 2002-01-01, 2003-01-01, 2004-01-01, 2005-01-01, 2006-01-01, 2007-01-01, 2008-01-01, 2009-01-01, 2010-01-01, 2011-01-01, 2012-01-01</code></pre></figure>
<p>Forest change in each year is coded as:</p>
<table>
<thead>
<tr>
<th style="text-align: left">Cover/Change</th>
<th style="text-align: center">Code</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">No data</td>
<td style="text-align: center">0</td>
</tr>
<tr>
<td style="text-align: left">Forest</td>
<td style="text-align: center">1</td>
</tr>
<tr>
<td style="text-align: left">Non-forest</td>
<td style="text-align: center">2</td>
</tr>
<tr>
<td style="text-align: left">Forest loss</td>
<td style="text-align: center">3</td>
</tr>
<tr>
<td style="text-align: left">Forest gain</td>
<td style="text-align: center">4</td>
</tr>
<tr>
<td style="text-align: left">Forest loss and gain</td>
<td style="text-align: center">5</td>
</tr>
<tr>
<td style="text-align: left">Water</td>
<td style="text-align: center">6</td>
</tr>
</tbody>
</table>
<p>The <code class="highlighter-rouge">animate_annual</code> function can be used to save a simple visualization of the
thresholded annual layer stack.</p>
<p>Note: For this example, we are using the data in the WGS84 coordinate system.
For a real analysis or presentation, the data should be projected into UTM or
another projection system for this. The <code class="highlighter-rouge">utm_zone</code> function in the
<code class="highlighter-rouge">gfcanalysis</code> package and the <code class="highlighter-rouge">projectRaster</code> function in the <code class="highlighter-rouge">raster</code> package
could be used to automate this. Also see the <code class="highlighter-rouge">to_utm</code> option for the
<code class="highlighter-rouge">extract_gfc</code> function (type <code class="highlighter-rouge">?extract_gfc</code> in R).</p>
<p>To make an annual animation (in WGS84) type:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">aoi</span><span class="o">$</span><span class="n">label</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s2">"ZOI"</span><span class="w"> </span><span class="c1"># Label the polygon on the plot</span><span class="w">
</span><span class="n">animate_annual</span><span class="p">(</span><span class="n">aoi</span><span class="p">,</span><span class="w"> </span><span class="n">gfc_annual_stack</span><span class="p">,</span><span class="w"> </span><span class="n">out_dir</span><span class="o">=</span><span class="s1">'.'</span><span class="p">,</span><span class="w"> </span><span class="n">site_name</span><span class="o">=</span><span class="s1">'Nam Kading'</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## HTML file created at: C:\Users\azvoleff\Code\Misc\azvoleff.github.io\Rmd\2014-03-25-analyzing-forest-change-with-gfcanalysis/gfc_animation.html
## You may use ani.options(outdir = getwd()) or saveHTML(..., outdir = getwd()) to generate files under the current working directory.</code></pre></figure>
<p>The animation will be saved in the directory specified by <code class="highlighter-rouge">out_dir</code> (in this
example the current working directory). To view the animation, double-click the
new “.html” file in that directory. The animation will look <a href="/content/2014-03-25-analyzing-forest-change-with-gfcanalysis/gfc_animation.html">something like
this</a>.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Hansen, M. C., P. V. Potapov, R. Moore, M. Hancher, S. A. Turubanova, A.
Tyukavina, D. Thau, S. V. Stehman, S. J. Goetz, T. R. Loveland, A. Kommareddy,
A. Egorov, L. Chini, C. O. Justice, and J. R. G. Townshend. 2013.
High-Resolution Global Maps of 21st-Century Forest Cover Change. Science 342,
(15 November): 850–853. Data available on-line from:
http://earthenginepartners.appspot.com/science-2013-global-forest. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="http://www.azvoleff.com/articles/analyzing-forest-change-with-gfcanalysis/">Analyzing forest change with gfcanalysis</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on March 25, 2014.</p>http://www.azvoleff.com/articles/image-classification-with-teamlucc2014-03-19T00:00:00-00:002014-03-19T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<h2 id="getting-started">Getting started</h2>
<p>First load the <code class="highlighter-rouge">devtools</code> package, used for installing <code class="highlighter-rouge">teamlucc</code>. Install the
<code class="highlighter-rouge">devtools</code> package if it is not already installed:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">require</span><span class="p">(</span><span class="n">devtools</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">install.packages</span><span class="p">(</span><span class="s1">'devtools'</span><span class="p">)</span><span class="w">
</span><span class="n">library</span><span class="p">(</span><span class="n">devtools</span><span class="p">)</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>Now load the <code class="highlighter-rouge">teamlucc</code> package, using <code class="highlighter-rouge">devtools</code> to install it from github if
it is not yet installed:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">require</span><span class="p">(</span><span class="n">teamlucc</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">install_github</span><span class="p">(</span><span class="s1">'azvoleff/teamlucc'</span><span class="p">)</span><span class="w">
</span><span class="n">library</span><span class="p">(</span><span class="n">teamlucc</span><span class="p">)</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>Also load the <code class="highlighter-rouge">rgdal</code> package needed for reading/writing shapefiles:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">rgdal</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## rgdal: version: 0.9-1, (SVN revision 518)
## Geospatial Data Abstraction Library extensions to R successfully loaded
## Loaded GDAL runtime: GDAL 1.11.0, released 2014/04/16
## Path to GDAL shared files: C:/Users/azvoleff/R/win-library/3.1/rgdal/gdal
## GDAL does not use iconv for recoding strings.
## Loaded PROJ.4 runtime: Rel. 4.8.0, 6 March 2012, [PJ_VERSION: 480]
## Path to PROJ.4 shared files: C:/Users/azvoleff/R/win-library/3.1/rgdal/proj</code></pre></figure>
<h2 id="collect-training-data-for-supervised-classification">Collect training data for supervised classification</h2>
<p>The first step in the classification is putting together a training dataset.
<code class="highlighter-rouge">teamlucc</code> includes a function to output a shapefile that can be used for
collecting training data. Here we are collecting training data for the
<code class="highlighter-rouge">L5TSR_1986</code> raster (a portion of a 1986 Landsat 5 surface reflectance image)
that is included with the <code class="highlighter-rouge">teamlucc</code> package. Use the <code class="highlighter-rouge">get_extent_polys</code>
function to quickly construct a shapefile in the same coordinate system as the
image:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">train_polys</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">get_extent_polys</span><span class="p">(</span><span class="n">L</span><span class="m">5</span><span class="n">TSR_1986</span><span class="p">)</span></code></pre></figure>
<p>Add an empty field named “class_1986” to the object, and delete the extent polygon
(because we don’t need it, and just want an empty shapefile):</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">train_polys</span><span class="o">$</span><span class="n">class_1986</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s1">''</span><span class="w"> </span><span class="c1"># Add an empty column named "class_1986"</span><span class="w">
</span><span class="n">train_polys</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">train_polys</span><span class="p">[</span><span class="m">-1</span><span class="p">,</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="c1"># Delete extent polygon</span></code></pre></figure>
<p>Now save the <code class="highlighter-rouge">train_polys</code> object to a shapefile using <code class="highlighter-rouge">writeOGR</code> from the
<code class="highlighter-rouge">rgdal</code> package. The <code class="highlighter-rouge">"."</code> below just means “save the shapefile in the current
directory”.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">writeOGR</span><span class="p">(</span><span class="n">train_polys</span><span class="p">,</span><span class="w"> </span><span class="s2">"."</span><span class="p">,</span><span class="w"> </span><span class="s2">"training_data"</span><span class="p">,</span><span class="w"> </span><span class="s2">"ESRI Shapefile"</span><span class="p">)</span></code></pre></figure>
<p>Open the generated “training_data.shp” shapefile in a GIS program (I recommend
<a href="http://www.qgis.org">QGIS</a>) and digitize a number of polygons in each of the
land cover classes you want to map. For this example, we will simply classify
“Forest” and “Non-forest”. For each polygon you digitize, record the cover type
in the “class_1986” column. After digitizing a number of polygons within each
class, save the shapefile, and load it back into R using
<code class="highlighter-rouge">train_polys <- readOGR(".", "training_data")</code>.</p>
<p>Or: (for this example) you can use the thirty training polygons included in the
<code class="highlighter-rouge">teamlucc</code> package in the <code class="highlighter-rouge">L5TSR_1986_2001_training</code> dataset:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">train_polys</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">L</span><span class="m">5</span><span class="n">TSR_1986_2001_training</span></code></pre></figure>
<h2 id="classify-image">Classify image</h2>
<p>First we need to extract the training data from our training image,
for each pixel within the polygons in our <code class="highlighter-rouge">train_polys</code> dataset.
<code class="highlighter-rouge">get_pixels</code> will use the <code class="highlighter-rouge">training</code> parameter that we pass to
determine the fraction of the training data to use in training the classifier.
If set to 1, ALL of the training data will be used to train the classifier,
leaving no independent data for validation. If set to a fraction (for example
.6), then only 60% of the data (randomly selected) will be used in training,
and 40% will be preserved as an independent sample for use in testing.</p>
<p>Note: Validation data should generally be collected separately from training
data anyways, to ensure the image is randomly sampled (training data collection
is almost never random), so in most cases I don’t recommend making heavy use of
the <code class="highlighter-rouge">training</code> parameter. It can be useful though in testing.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">set.seed</span><span class="p">(</span><span class="m">0</span><span class="p">)</span><span class="w"> </span><span class="c1"># Set a random seed so results can be reproduced</span><span class="w">
</span><span class="n">train_data</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">get_pixels</span><span class="p">(</span><span class="n">L</span><span class="m">5</span><span class="n">TSR_1986</span><span class="p">,</span><span class="w"> </span><span class="n">train_polys</span><span class="p">,</span><span class="w"> </span><span class="n">class_col</span><span class="o">=</span><span class="s2">"class_1986"</span><span class="p">,</span><span class="w">
</span><span class="n">training</span><span class="o">=</span><span class="m">.6</span><span class="p">)</span></code></pre></figure>
<p>A summary method is provided by <code class="highlighter-rouge">teamlucc</code> for printing summary statistics on
training datasets:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">summary</span><span class="p">(</span><span class="n">train_data</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Object of class "pixel_data"
##
## Number of classes: 2
## Number of polygons: 30
## Number of pixels: 120
## Number of sources: 1
##
## Training data statistics:
## Source: local data frame [2 x 5]
##
## class n_polys n_train_pixels n_test_pixels train_frac
## 1 Forest 17 48 20 0.71
## 2 NonForest 13 24 28 0.46
##
## Number of training samples: 72
## Number of testing samples: 48
## Training fraction: 0.6</code></pre></figure>
<p>To perform the actual image classification, we will use the <code class="highlighter-rouge">classify</code>
function. Prior to using that function, we need to train a classifier. The
<code class="highlighter-rouge">train_classifier</code> function automates training a random forest or support
vector machine (SVM) classifier. There are many options that can be provided to
<code class="highlighter-rouge">train_classifier</code> - for this example we will just use the defaults. The
default is to use a random forest classifier.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">clfr</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">train_classifier</span><span class="p">(</span><span class="n">train_data</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: randomForest
## randomForest 4.6-10
## Type rfNews() to see new features/changes/bug fixes.
## Loading required package: lattice
## Loading required package: ggplot2</code></pre></figure>
<p>Now we can use the <code class="highlighter-rouge">classify</code> function to perform the image classification:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">cls</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">classify</span><span class="p">(</span><span class="n">L</span><span class="m">5</span><span class="n">TSR_1986</span><span class="p">,</span><span class="w"> </span><span class="n">clfr</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning in .local(x, ...): min value not known, use setMinMax</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning in .local(x, ...): min value not known, use setMinMax</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: parallel
##
## Attaching package: 'parallel'
##
## The following objects are masked from 'package:snow':
##
## clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
## clusterExport, clusterMap, clusterSplit, makeCluster,
## parApply, parCapply, parLapply, parRapply, parSapply,
## splitIndices, stopCluster
##
## Loading required package: iterators
## Loading required package: foreach
## foreach: simple, scalable parallel programming from Revolution Analytics
## Use Revolution R for scalability, fault tolerance and more.
## http://www.revolutionanalytics.com
##
## Attaching package: 'mmap'
##
## The following object is masked from 'package:Rcpp':
##
## sizeof</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning in int64(): unsupported int64, use int32 or real64</code></pre></figure>
<p>To see the predicted classes, use <code class="highlighter-rouge">spplot</code>:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">spplot</span><span class="p">(</span><span class="n">cls</span><span class="o">$</span><span class="n">classes</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-03-19-image-classification-with-teamlucc/predicted_classes-1.png" title="Predicted classes" alt="Predicted classes" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>We can also see the class probabilities (per pixel probabilities of membership of each class):</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">spplot</span><span class="p">(</span><span class="n">cls</span><span class="o">$</span><span class="n">probs</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-03-19-image-classification-with-teamlucc/class_probabilities-1.png" title="Predicted probabilities of each class" alt="Predicted probabilities of each class" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>The output from <code class="highlighter-rouge">classify</code> also includes a table indicating the coding for the
output:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">print</span><span class="p">(</span><span class="n">cls</span><span class="o">$</span><span class="n">codes</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## code class
## 1 0 Forest
## 2 1 NonForest</code></pre></figure>
<h3 id="parallel-processing">Parallel processing</h3>
<p>Training a classifier and predicting land cover classes is very CPU-intensive.<br />
If you have a machine that has multiple processors (or multiple cores), using
more than one processor can significantly increase the speed of some
calculations. <code class="highlighter-rouge">teamlucc</code> supports parallel computations (using the capabilities
of the <code class="highlighter-rouge">raster</code> package). To enable this functionality, first install the
<code class="highlighter-rouge">doParallel</code> package if it is not already installed, and load the package:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">require</span><span class="p">(</span><span class="n">doParallel</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">install.packages</span><span class="p">(</span><span class="s1">'doParallel'</span><span class="p">)</span><span class="w">
</span><span class="n">library</span><span class="p">(</span><span class="n">doParallel</span><span class="p">)</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: doParallel</code></pre></figure>
<p>Now, just call <code class="highlighter-rouge">registerDoParallel()</code>, and by default any calculations that are
coded to run in parallel will use half of the available CPUs on your machine.<br />
You can also specify a number of CPUs to use, by running, for example,
<code class="highlighter-rouge">registerDoParallel(2)</code> to use two CPUs. The <code class="highlighter-rouge">get_pixels</code>, <code class="highlighter-rouge">train_classifier</code><br />
and <code class="highlighter-rouge">classify</code> functions in <code class="highlighter-rouge">teamlucc</code> all support parallel computation, and
will run in parallel automatically if you have called <code class="highlighter-rouge">registerDoParallel</code>.<br />
Below is the code for the same classification problem we just ran, but this
time we run the classification in parallel:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">doParallel</span><span class="p">)</span><span class="w">
</span><span class="n">registerDoParallel</span><span class="p">(</span><span class="m">2</span><span class="p">)</span><span class="w">
</span><span class="n">set.seed</span><span class="p">(</span><span class="m">0</span><span class="p">)</span><span class="w"> </span><span class="c1"># Set a random seed so results match what we got earlier</span><span class="w">
</span><span class="n">train_data_par</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">get_pixels</span><span class="p">(</span><span class="n">L</span><span class="m">5</span><span class="n">TSR_1986</span><span class="p">,</span><span class="w"> </span><span class="n">train_polys</span><span class="p">,</span><span class="w"> </span><span class="n">class_col</span><span class="o">=</span><span class="s2">"class_1986"</span><span class="p">,</span><span class="w">
</span><span class="n">training</span><span class="o">=</span><span class="m">.6</span><span class="p">)</span><span class="w">
</span><span class="n">clfr_par</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">train_classifier</span><span class="p">(</span><span class="n">train_data</span><span class="p">)</span><span class="w">
</span><span class="n">cls_par</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">classify</span><span class="p">(</span><span class="n">L</span><span class="m">5</span><span class="n">TSR_1986</span><span class="p">,</span><span class="w"> </span><span class="n">clfr</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning in .local(x, ...): min value not known, use setMinMax</code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning in .local(x, ...): min value not known, use setMinMax</code></pre></figure>
<h2 id="accuracy-assessment">Accuracy assessment</h2>
<p>Conducting a thorough accuracy assessment is one of the most important
components of image classification. The <code class="highlighter-rouge">teamlucc</code> package includes an
<code class="highlighter-rouge">accuracy</code> function to assist with measuring the accuracy of image
classifications. In addition to the standard contingency tables often used for
describing accuracy, <code class="highlighter-rouge">accuracy</code> also calculates “quantity disagreement” and
“allocation disagreement” as introduced by Pontius and Millones 2011<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>.<br />
Unbiased contingency tables can be calculated with <code class="highlighter-rouge">accuracy</code> by supplying a
<code class="highlighter-rouge">pop</code> parameter to <code class="highlighter-rouge">accuracy</code>. <code class="highlighter-rouge">accuracy</code> provides 95% confidence intervals for
user’s, producer’s, and overall accuracies, calculated as in Olofsson et al.<br />
2013<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup>.</p>
<p>To calculate a basic contingency table, assuming that population frequencies of
the observed classes can be estimated from the classification output, and using
the 40% of pixels that were excluded from training the classifier as testing
data, run the <code class="highlighter-rouge">accuracy</code> function using the model calculated above:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">acc</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">accuracy</span><span class="p">(</span><span class="n">clfr</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Warning in calc_accuracy(predicted, observed, pop, reclass_mat): pop was
## not provided - assuming sample frequencies equal population frequencies</code></pre></figure>
<p>Note the warning from <code class="highlighter-rouge">accuracy</code>, which is reminding us that we did not provide
population frequencies for the classes.</p>
<p>A<code class="highlighter-rouge">summary</code> method for the <code class="highlighter-rouge">accuracy</code> object is provided by <code class="highlighter-rouge">teamlucc</code>, and
calculates user’s, producers, and overall accuracy, and quantity and allocation
disagreement:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">summary</span><span class="p">(</span><span class="n">acc</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Object of class "accuracy"
##
## Testing samples: 48
##
## Sample contingency table:
## observed
## predicted Forest NonForest Total Users
## Forest 16.0000 3.0000 19.0000 0.8421
## NonForest 4.0000 25.0000 29.0000 0.8621
## Total 20.0000 28.0000 48.0000
## Producers 0.8000 0.8929 0.8542
##
## Population contingency table:
## observed
## predicted Forest NonForest Total Users
## Forest 0.3333 0.0625 0.3958 0.8421
## NonForest 0.0833 0.5208 0.6042 0.8621
## Total 0.4167 0.5833 1.0000
## Producers 0.8000 0.8929 0.8542
##
## Overall accuracy: 0.8542
##
## Quantity disagreement: 0.0208
## Allocation disagreement: 0.125</code></pre></figure>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Pontius, R. G., and M. Millones. 2011. Death to Kappa: birth of quantity
disagreement and allocation disagreement for accuracy assessment.
International Journal of Remote Sensing 32:4407-4429. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Olofsson, P., G. M. Foody, S. V. Stehman, and C. E. Woodcock. 2013.
Making better use of accuracy data in land change studies: Estimating
accuracy and area and quantifying uncertainty using stratified estimation.
Remote Sensing of Environment 129:122-131. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="http://www.azvoleff.com/articles/image-classification-with-teamlucc/">Classifying an image with teamlucc</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on March 19, 2014.</p>http://www.azvoleff.com/articles/glcm-0-2-released2014-02-17T00:00:00+00:002014-02-17T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>I just released to CRAN<a href="http://cran.r-project.org/web/packages/glcm"> a
new version of the “glcm” R package</a> for calculating image texture measures
from grey-level co-occurrence matrices (GLCMs). Type</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">install.packages</span><span class="p">(</span><span class="s2">"glcm"</span><span class="p">)</span></code></pre></figure>
<p>at your R command prompt to download the latest CRAN release. To install the
latest development version from github using
<a href="http://cran.r-project.org/web/packages/devtools/index.html">devtools</a> type:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">install_github</span><span class="p">(</span><span class="s2">"glcm"</span><span class="p">,</span><span class="w"> </span><span class="n">user</span><span class="o">=</span><span class="s2">"azvoleff"</span><span class="p">)</span></code></pre></figure>
<p>For more information on the development version, see the <a href="http://github.com/azvoleff/glcm">github project
page</a> for glcm.</p>
<p><a href="http://www.azvoleff.com/articles/glcm-0-2-released/">glcm 0.2 released</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on February 17, 2014.</p>http://www.azvoleff.com/articles/calculating-image-textures-with-glcm2014-03-19T00:00:00-00:002014-02-17T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p><code class="highlighter-rouge">glcm</code> can calculate image textures from either a matrix or a <code class="highlighter-rouge">Raster*</code> object
from the <code class="highlighter-rouge">raster</code> package. First install the package if it is not yet
installed:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="p">(</span><span class="n">require</span><span class="p">(</span><span class="n">glcm</span><span class="p">)))</span><span class="w"> </span><span class="n">install.packages</span><span class="p">(</span><span class="s2">"glcm"</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: glcm</code></pre></figure>
<p>The below examples use an image included in the <code class="highlighter-rouge">glcm</code> package, a
red/green/blue cutout of a Landsat 5 image from 1986 from a Tropical Ecology
Assessment and Monitoring (TEAM) Network site in Volcan Barva, Costa Rica. The
image is included in the glcm package as <code class="highlighter-rouge">L5TSR_1986</code>:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">library</span><span class="p">(</span><span class="n">raster</span><span class="p">)</span><span class="w"> </span><span class="c1"># needed for plotRGB function</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## Loading required package: sp</code></pre></figure>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plotRGB</span><span class="p">(</span><span class="n">L</span><span class="m">5</span><span class="n">TSR_1986</span><span class="p">,</span><span class="w"> </span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="n">stretch</span><span class="o">=</span><span class="s1">'lin'</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-02-17-calculating-image-textures-with-glcm/L5TSR_1986_plot-1.png" title="1986 Landsat 5 image from Volcan Barva" alt="1986 Landsat 5 image from Volcan Barva" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p>To calculate GLCM textures from this image using the default settings, type:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">textures</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">glcm</span><span class="p">(</span><span class="n">raster</span><span class="p">(</span><span class="n">L</span><span class="m">5</span><span class="n">TSR_1986</span><span class="p">,</span><span class="w"> </span><span class="n">layer</span><span class="o">=</span><span class="m">3</span><span class="p">))</span></code></pre></figure>
<p>where <code class="highlighter-rouge">raster(L5TSR_1986, layer=3)</code> selects the third (red) layer. To see the
textures that have been calculated by default, type:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="nf">names</span><span class="p">(</span><span class="n">textures</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-text" data-lang="text">## [1] "glcm_mean" "glcm_variance" "glcm_homogeneity"
## [4] "glcm_contrast" "glcm_dissimilarity" "glcm_entropy"
## [7] "glcm_second_moment" "glcm_correlation"</code></pre></figure>
<p>This shows the eight GLCM texture statistics that have been calculated by
default. These can all be visualized in R:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">textures</span><span class="o">$</span><span class="n">glcm_mean</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-02-17-calculating-image-textures-with-glcm/mean-1.png" title="mean of GLCM texture" alt="mean of GLCM texture" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">textures</span><span class="o">$</span><span class="n">glcm_variance</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-02-17-calculating-image-textures-with-glcm/variance-1.png" title="variance of GLCM texture" alt="variance of GLCM texture" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">textures</span><span class="o">$</span><span class="n">glcm_homogeneity</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-02-17-calculating-image-textures-with-glcm/homogeneity-1.png" title="homogeneity of GLCM texture" alt="homogeneity of GLCM texture" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">textures</span><span class="o">$</span><span class="n">glcm_contrast</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-02-17-calculating-image-textures-with-glcm/contrast-1.png" title="contrast of GLCM texture" alt="contrast of GLCM texture" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">textures</span><span class="o">$</span><span class="n">glcm_dissimilarity</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-02-17-calculating-image-textures-with-glcm/dissimilarity-1.png" title="dissimilarity of GLCM texture" alt="dissimilarity of GLCM texture" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">textures</span><span class="o">$</span><span class="n">glcm_entropy</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-02-17-calculating-image-textures-with-glcm/entropy-1.png" title="entropy of GLCM texture" alt="entropy of GLCM texture" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">textures</span><span class="o">$</span><span class="n">glcm_second_moment</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-02-17-calculating-image-textures-with-glcm/second_moment-1.png" title="second moment of GLCM texture" alt="second moment of GLCM texture" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">plot</span><span class="p">(</span><span class="n">textures</span><span class="o">$</span><span class="n">glcm_correlation</span><span class="p">)</span></code></pre></figure>
<p><img src="/content/2014-02-17-calculating-image-textures-with-glcm/correlation-1.png" title="correlation of GLCM texture" alt="correlation of GLCM texture" style="display:block;margin-left:auto;margin-right:auto;" /></p>
<p><a href="http://www.azvoleff.com/articles/calculating-image-textures-with-glcm/">Calculating image textures with GLCM</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on February 17, 2014.</p>http://www.azvoleff.com/articles/wrspathrow-0-1-released2014-02-13T00:00:00+00:002014-02-13T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>A new R package for working with path and row numbers from the World Reference
System (WRS) grids (both WRS-1 and WRS-2) is <a href="http://cran.r-project.org/web/packages/wrspathrow">now available on
CRAN</a>.</p>
<p>For more information see the <a href="http://github.com/azvoleff/wrspathrow">github project
page</a> for the <code class="highlighter-rouge">wrspathrow</code> package, or
see <a href="https://stat.ethz.ch/pipermail/r-sig-geo/2014-February/020403.html">this
post</a>.</p>
<p><code class="highlighter-rouge">wrspathrow</code> includes functions for determining the path and row number(s) needed
to cover a given spatial object, or, conversely, for returning the polygon for
a given path and row. Note that installation of the <code class="highlighter-rouge">wrspathrow</code> package may take
a bit of time due to the need to download the
<a href="http://cran.r-project.org/web/packages/wrspathrowData">wrspathrowData</a> package
it depends on. The <code class="highlighter-rouge">wrspathrowData</code> package is approximately 26MB in
size, as it includes the full WRS-1 and WRS-2 vector grids in R format. My
thanks to the USGS for allowing the re-release of these reformatted datafiles
on CRAN. See the <a href="http://landsat.usgs.gov/tools_wrs-2_shapefile.php">USGS WRS-1 and WRS-2 shapefile download
page</a> for the original
source of these files.</p>
<p><a href="http://www.azvoleff.com/articles/wrspathrow-0-1-released/">wrspathrow 0.1 released</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on February 13, 2014.</p>http://www.azvoleff.com/articles/running-abms-in-the-cloud-with-amazon-ec22013-04-10T00:00:00+00:002013-04-10T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>PyABM can be installed on an Amazon Elastic Computing Core (EC2) instance to
allow you to run agent-based models (ABMs) in the cloud. If you are new to
Amazon EC2, see the <a href="http://aws.amazon.com/ec2/">EC2 overview</a>
before you get started. Amazon also has a special page on <a href="http://aws.amazon.com/hpc-applications/">high performance
computing (HPC) with EC2</a>. You will
also probably want to look at the available <a href="http://aws.amazon.com/ec2/instance-types/">Amazon EC2 instance
types</a>, and of course the
<a href="http://aws.amazon.com/ec2/pricing/">pricing</a>
information before you get started.</p>
<p>A basic cluster with a manager node and two worker nodes will run you about
$1.50 - $2.00 per hour, depending on the options you choose. You can vary the
number of CPU cores and the memory in your worker nodes depending on the needs
of your modeling. In general my models are CPU limited (not requiring large
amounts of memory) so I will create a small cluster of three Amazon EC2
instances, with one “Large Standard On-Demand” (m1.large) instance to manage
the cluster, and two “Extra Large High-CPU On-Demand” (c1.xlarge) instances as
worker nodes.This configuration gives me a total of 16 processor cores to work
with (8 per worker node), so I can run 16 model runs at the same time.</p>
<p>The cost to run this cluster is $1.58 per hour with current Amazon EC2 pricing.<br />
Note that pricing varies depending on the region you choose to place your
clusters in - the cheapest region is currently in northern Virginia in the
United States.</p>
<p>The easiest way I have found to get Amazon EC2 clusters up and running is using
<a href="http://star.mit.edu/cluster/">StarCluster</a> - a python program that makes
setting up, running, and managing Amazon EC2 clusters much easier.</p>
<p><a href="http://www.azvoleff.com/articles/running-abms-in-the-cloud-with-amazon-ec2/">Running ABMs in the Cloud with Amazon EC2</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on April 10, 2013.</p>http://www.azvoleff.com/articles/chitwan-abm-v1-5-released2013-02-23T00:00:00+00:002013-02-23T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>The latest release of Chitwan ABM (version 1.5) is now available at the
<a href="http://pypi.python.org/pypi/chitwanabm/1.5">Python Package Index</a>.</p>
<p><a href="http://www.azvoleff.com/articles/chitwan-abm-v1-5-released/">ChitwanABM 1.5 Released</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on February 23, 2013.</p>http://www.azvoleff.com/articles/pyabm-0-3-3-released2013-02-01T00:00:00+00:002013-02-01T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>The latest release of PyABM (version 0.3.3) is now available at the
<a href="http://pypi.python.org/pypi/pyabm/0.3.3">Python Package Index</a>.</p>
<p><a href="http://www.azvoleff.com/articles/pyabm-0-3-3-released/">PyABM 0.3.3 released</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on February 01, 2013.</p>http://www.azvoleff.com/articles/modifying-the-pyabm-source-code2014-03-19T00:00:00-00:002012-11-27T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<h2 id="modifying-the-latest-release-of-pyabm">Modifying the latest release of PyABM</h2>
<p>If you plan on making any changes to the PyABM source code, you can use a
<a href="http://www.pip-installer.org">pip</a> <a href="http://www.pip-installer.org/en/latest/reference/pip_install.html?highlight=editable%20install#editable-installs">“editable
install”</a> to
install PyABM in your local user folder so that you can edit the source code in
place without having to rebuild and reinstall the PyABM package every time you
make a change. To use this feature, first install pip, then open a command
window and type:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nb">sudo </span>pip install <span class="nt">-e</span> pyabm</code></pre></figure>
<p>in a command prompt in Linux, or</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"> pip install <span class="nt">-e</span> pyabm</code></pre></figure>
<p>on Windows. This will install the latest release of PyABM as an editable
install so that you can <code class="highlighter-rouge">import pyabm</code> from any python window or script, and
have the module imported from your own version of the source code (which will
be in a folder named something like <code class="highlighter-rouge">C:\users\azvoleff\src\pyabm</code> (on Windows) or
<code class="highlighter-rouge">/home/azvoleff/src/pyabm</code> (on Linux).</p>
<h2 id="modifying-the-development-version-of-pyabm">Modifying the development version of PyABM</h2>
<p>There are several ways to end up with an editable version of the current
development version of PyABM. If you have git installed on your system you can
use a pip editable install to download the latest version of PyABM from github
for you, and install it in an editable mode. If you do not have git you use
<a href="http://packages.python.org/distribute/">distribute</a>, which offers a
<a href="http://packages.python.org/distribute/setuptools.html#development-mode">“development
mode”</a>.
With both approaches, the end result is having the development version of PyABM
installed in such a way that any changes you make to the code take effect
immediately.</p>
<p>If you have git installed on your system, you can use pip to clone and install
the development version of PyABM from github by typing:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"> <span class="nb">sudo </span>pip install <span class="nt">-e</span> git+https://github.com/azvoleff/pyabm.git#egg<span class="o">=</span>pyabm</code></pre></figure>
<p>in a command prompt in Linux, or</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"> pip install <span class="nt">-e</span> git+https://github.com/azvoleff/pyabm.git#egg<span class="o">=</span>pyabm</code></pre></figure>
<p>on Windows. An advantage of cloning PyABM from github is that you can easily
update your copy of PyABM to include the latest changes by navigating to the
main PyABM folder and typing</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"> git pull</code></pre></figure>
<p>to pull the latest version of the PyABM source code from git and (depending on
your settings in git) merge any upstream changes with your local edits.</p>
<p>If you do not have git installed, download the latest development source of
PyABM as a <a href="https://github.com/azvoleff/pyabm/archive/master.zip">zip file</a>.
After downloading the PyABM source code, navigate to the main PyABM folder (the
one with setup.py) and type:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"> python setup.py develop</code></pre></figure>
<p>This will install a development version of PyABM and setup your system so that
you can <code class="highlighter-rouge">import pyabm</code> from your Python interpreter from your Python scripts.</p>
<p><a href="http://www.azvoleff.com/articles/modifying-the-pyabm-source-code/">Modifying the PyABM source code</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on November 27, 2012.</p>http://www.azvoleff.com/articles/pyabm-0-3-2-released2012-11-20T00:00:00+00:002012-11-20T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p>The latest release of PyABM (version 0.3.2) is now available at the
<a href="http://pypi.python.org/pypi/pyabm/0.3.2">Python Package Index</a>.</p>
<p><a href="http://www.azvoleff.com/articles/pyabm-0-3-2-released/">PyABM 0.3.2 released</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on November 20, 2012.</p>http://www.azvoleff.com/articles/pyabm-logging-setup2012-11-19T00:00:00+00:002012-11-19T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p><a href="/pyabm">PyABM</a> uses the python
<a href="http://docs.python.org/2/library/logging.html">logging</a> module, so that
warning and informational messages from the PyABM module can be written to the
console, and also saved to files along with model output. For flexibility, and
consistent with recommended usage of the logging module, configuration of the
logging output is left up to the user of PyABM. If you <code class="highlighter-rouge">import pyabm</code> directly
from a python session without setting up logging first, you will see a warning
message when PyABM tries to log a message:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">pyabm</span>
<span class="n">No</span> <span class="n">handlers</span> <span class="n">could</span> <span class="n">be</span> <span class="n">found</span> <span class="k">for</span> <span class="n">logger</span> <span class="s">"pyabm.rcsetup"</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span></code></pre></figure>
<p>For this example, I have forced PyABM to try to print an error message, by
specifying the wrong path to git in my pyabmrc. To see the error message, I
will need to <code class="highlighter-rouge">import logging</code> and configure a logging handler prior to
importing PyABM:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">()</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">pyabm</span>
<span class="n">WARNING</span><span class="p">:</span><span class="n">pyabm</span><span class="o">.</span><span class="n">rcsetup</span><span class="p">:</span><span class="n">Failure</span> <span class="k">while</span> <span class="n">reading</span> <span class="n">rc</span> <span class="n">parameter</span> <span class="n">path</span><span class="o">.</span><span class="n">git_binary</span> <span class="n">on</span>
<span class="n">line</span> <span class="mi">42</span> <span class="ow">in</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">azvoleff</span><span class="o">/</span><span class="n">pyabmrc</span><span class="p">:</span> <span class="o">/</span><span class="n">wrong</span><span class="o">/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">git</span> <span class="n">does</span> <span class="ow">not</span> <span class="n">exist</span><span class="o">.</span> <span class="n">Reverting</span>
<span class="n">to</span> <span class="n">default</span> <span class="n">parameter</span> <span class="n">value</span><span class="o">.</span>
<span class="n">WARNING</span><span class="p">:</span><span class="n">pyabm</span><span class="o">.</span><span class="n">rcsetup</span><span class="p">:</span><span class="n">git</span> <span class="n">version</span> <span class="n">control</span> <span class="n">features</span> <span class="n">disabled</span><span class="o">.</span> <span class="n">Specify</span> <span class="n">valid</span> <span class="n">git</span>
<span class="n">binary</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">your</span> <span class="n">pyabmrc</span> <span class="n">to</span> <span class="n">enable</span><span class="o">.</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span></code></pre></figure>
<p>I now see a warning telling me that <code class="highlighter-rouge">/wrong/path/to/git</code> does not exist, and
PyABM tries the default path to git specified in rcparams.default. This path
also does not exist (as I am running this example on Linux and PyABM is setup
for Windows by default) so I then see another warning ‘git version control
features disabled’ as PyABM is not able to find git. If I fix my pyabmrc file
to give the correct path to git (which, on my system, is <code class="highlighter-rouge">/usr/bin/git</code>) I can
<code class="highlighter-rouge">import pyabm</code> without seeing any error messages:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">logging</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">()</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">pyabm</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span></code></pre></figure>
<p><a href="http://www.azvoleff.com/articles/pyabm-logging-setup/">PyABM logging setup</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on November 19, 2012.</p>http://www.azvoleff.com/articles/pyabmrc-configuration2012-11-18T00:00:00+00:002012-11-18T00:00:00+00:00Alex Zvoleffhttp://www.azvoleff.comazvoleff@conservation.org<p><a href="/pyabm">PyABM</a> configuration is done using a pyabmrc text file. When loaded in
Python, using:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">pyabm</span></code></pre></figure>
<p>PyABM will search for a pyabmrc file. PyABM will search three locations, in
order:</p>
<ul>
<li>the current working directory</li>
<li>the current user’s home directory</li>
<li>the pyabm module directory</li>
</ul>
<p>PyABM will use the first pyabmrc file it finds, ignoring any others. Example
pyabmrc files are provided with PyABM versions < 0.3.1, in
<a href="https://raw.github.com/azvoleff/pyabm/master/pyabm/pyabmrc.windows">pyabmrc.windows</a>
and
<a href="https://raw.github.com/azvoleff/pyabm/master/pyabm/pyabmrc.linux">pyabmrc.linux</a>,
in the main module folder (under pyabm\pyabm in the development version). To
set custom values for any of the pyabmrc parameters, rename the proper file
from to ‘pyabmrc’ and move it to one of the three above locations. See the
pyabmrc.default file for details on each parameter and on possible parameter
values. Changes can also be made in the
<a href="https://raw.github.com/azvoleff/pyabm/master/pyabm/rcparams.default">rcparams.defaults</a>
file in the PyABM module directory, but this is not recommended as these values
will be overwritten when PyABM is upgraded.</p>
<p><a href="http://www.azvoleff.com/articles/pyabmrc-configuration/">PyABM configuration using 'pyabmrc' files</a> was originally published by Alex Zvoleff at <a href="http://www.azvoleff.com">Alex Zvoleff</a> on November 18, 2012.</p>