Shiny App to access NOAA data

Now that the US Government shutdown is over, it is time to download NOAA weather daily summaries in bulk and store them somewhere safe so that at the next shutdown we do not need to worry.

Below is the code to download data for a series of years:

NOAA_BulkDownload <- function(Year, Dir){
  URL <- paste0("ftp://ftp.ncdc.noaa.gov/pub/data/gsod/",Year,"/gsod_",Year,".tar")
  download.file(URL, destfile=paste0(Dir,"/gsod_",Year,".tar"),
                method="auto",mode="wb")
  
  if(dir.exists(paste0(Dir,"/NOAA Data"))==FALSE){dir.create(paste0(Dir,"/NOAA Data"))}
  
  untar(paste0(Dir,"/gsod_",Year,".tar"), 
        exdir=paste0(Dir,"/NOAA Data"))
}

An example on how to use this function is below:

Date <- 1980:2019
lapply(Date, NOAA_BulkDownload, Dir="C:/Users/fabio.veronesi/Desktop/New folder")

Theoretically, the process can be parallelized using parLappy, but I have not tested it.

Once we have all the file in one folder we can create the Shiny app to query these data.
The app will have a dashboard look with two tabs: one with a Leaflet map showing the location of the weather stations (markers are shown only at a certain zoom level to decrease loading time and RAM usage), below:

The other tab will allow the creation of the time-series (each file represents only 1 year, so we need to bind several files together to get the full period we are interested in) and it will also do some basic data cleaning, e.g. turn T from F to C, or snow depth from inches to mm. Finally, from this tab users can view the final product and download a cleaned csv.

The code for ui and server scripts is on my GitHub:
https://github.com/fveronesi/NOAA_ShinyApp

Street Crime UK - Shiny App

Introduction

This is a shiny app to visualize heat maps of Street Crimes across Britain from 2010-12 to 2018-01 and test their spatial pattern.

The code for both ui.R and server.R is available from my GitHub at: https://github.com/fveronesi/StreetCrimeUK_Shiny

Usage

Please be aware that this apps downloads data from my personal Dropbox once it starts and every time the user changes some of the settings. This was the only work-around I could think of to use external data in shinyapps.io for free. However, this also makes the app a bit slow, so please be patient.

Users can select a date with two sliders (I personally do not like the dateInput tool), then a crime type and click Draw Map to update the map with new data. I also included a option to plot the Ripley K-function (function Kest in package spatstat) and the p-value of the quadrat.test (again from spatstat). Both tools work using the data shown within the screen area, so their results change as users interact with the map. The Ripley K function shows a red dashed line with the expected nearest neighbour distribution of points that are randomly distributed in space (i.e. follow a Poisson distribution). The black line is the one computed from the points shown on screen. If the black line is above the red means the observations shown on the map are clustered, while if it is below the red line means the crimes are scattered regularly in space. A more complete overview of the Ripley K function is available at this link from ESRI.

The p-value from the quadrat test is testing a null hypothesis that the crimes are scattered randomly in space, against an alternative that they are clustered. If the p-value is below 0.05 (significance level of 5%) we can accept the alternative hypothesis that our data are clustered. Please be aware that this test does not account for regularly space crimes.

NOTE

Please not that the code here is not reproducible straight away. The app communicates with my Dropbox, though the package rdrop2, which requires a token to download data from Dropbox. More info github.com/karthik/rdrop2.

I am sharing the code to potentially use a taken downloaded from elsewhere, but the url that points to my Dropbox will clearly not be shared.

Preparing the dataset

Csv files with crime data can be downloaded directly from the data.police.uk website. Please check the dates carefully, since each of these files contains more that one years of monthly data. The main issue with these data is that they are divided by local police forces, so for example we will have a csv for each month from the Bedfordshire Police, which only covers that part of the country. Moreover, these csv contain a lot of data, not only coordinates; they also contain the type of crimes, plus other details, which we do not need and which makes the full collection a couple of Gb in size.

For these reasons I did some pre-processing, First of all I extracted all csv files into a folder named "CrimeUK" and then I ran the code below:

lista = list.files("E:/CrimesUK",pattern="street",recursive=T,include.dirs=T,full.names=T,ignore.case = T)

for(i in lista){
  DF = read.csv(i)

   write.table(data.frame(LAT=DF$Latitude, LON=DF$Longitude, TYPE=DF$Crime.type),
               file=paste0("E:/CrimesUK/CrimesUK",substr(paste(DF$Month[1]),1,4),"_",substr(paste(DF$Month[1]),6,7),".csv"),
               sep=",",row.names=F,col.names=F, append=T)
   print(i)
}

Here I first create a list of all csv files, with full link, searching inside all sub directory. Then I started a for loop to iterate through the files. The loop simply loads each file and than save part of its contents (namely coordinates and crime type) into new csv named after using year and month. This will help me identify which files to download from Dropbox, based on user inputs.

Once I had these files I simply uploded them to my Dropbox.

The link to test the app is:

fveronesi.shinyapps.io/CrimeUK/

A snapshot of the screen is below:

Learning R for Data Visualization [Video]

Last year Packt asked me to develop a video course to teach various techniques of data visualization in R. Since I love the idea of video courses and tutorials, and I also enjoy plotting data, I readily agreed.
The result is this course, published last March, which I will briefly present below.

The course is available here:
https://www.packtpub.com/big-data-and-business-intelligence/learning-r-data-visualization-video

I wanted to create a course that was easy to follow, and at the same time could provide a good basis even for the most advanced forms of data visualization available today in R.
Packt was interested in presenting ggplot2, which is definitely the most advanced way of creating static plots. Since I regularly use ggplot2 and I find it a tremendous tool, I was glad to be able to present its functionalities more in details. Three chapters are dedicated to this package. Here I present all the most important types of plots: histograms, box-plots, scatterplots, bar-charts and time-series. Moreover, a whole chapter is dedicated to embellish the default plots by adding elements, such as text labels and much more.

However, I am also very interested in interactive plotting, which I believe is now rapidly becoming commonplace for lots of applications. For this reason two chapters are completely dedicated to interactive plots. In the first I present the package rCharts, which is extremely powerful but also a bit tricky to use at times. In many cases there is little documentation to work with, and for developing the course I found myself often wondering through stackoverflow searching for answers. Luckily for all of us, Prof. Ramnath Vaidyanathan, the creator of rCharts, is always available to answer all the users' questions quickly and clearly. In chapter 5 the viewer will be able to start from zero and quickly create nice interactive versions of all the plots I covered with ggplot2. 

The last chapter is dedicated to Shiny and it is aimed at the creation of a full website for importing and plotting data. Here the reader will first learn the basics of Shiny and then will write the code to create the website and add lots of interesting functionalities.

I hope this video course will help R users become familiar with data visualization.
I would also like to take this opportunity to stress that I am open to support viewers throughout the learning process, meaning that if you have any question about the material in the course you should not hesitate one second in contacting me at info@fabioveronesi.net

Live Earthquake Map with Shiny and Google Map API

In the post Exchange data between R and the Google Maps API using Shiny I presented a very simple way to allow communication between R and javascript using Shiny.

This is an example of a practical approach for which the same system can be used. Here I created a tool to visualize seismic events, collected from USGS, in the Google Maps API using R to do some basic data preparation. The procedure to complete this experiment is pretty much identical to what I presented in the post mentioned above, so I will not repeated here.


The final map looks like this:

and it is accessible from this site, hosted on the Amazon Cloud: Earthquake

The colours of the markers depends on magnitude and they are set in R. For magnitudes below 2 the marker is green, between 2 and 4 is yellow, between 4 and 6 is orange and above 6 is red.
I also set R to export other information about the event to the json file that I then use to populate the infowindow of each marker.

The code for creating this map consists of two pieces, an index.html file (which needs to go in a folder names www) and the file server.r, available below:

Server.r

 # server.R  
 #Title: Earthquake Visualization in Shiny  
 #Copyright: Fabio Veronesi  
   
 library(sp)  
 library(rjson)  
 library(RJSONIO)  
   
    
 shinyServer(function(input, output) {  
   
 output$json <- reactive ({  
 if(length(input$Earth)>0){  
 if(input$Earth==1){  
 hour <- read.table("http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.csv", sep = ",", header = T)  
 if(nrow(hour)>0){  
 lis <- list()  
 for(i in 1:nrow(hour)){  
   
 if(hour$mag[i]<=2){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_green.png"}  
 else if(hour$mag[i]>2&hour$mag[i]<=4){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_yellow.png"}  
 else if(hour$mag[i]>4&hour$mag[i]<=6){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_orange.png"}  
 else {icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_red.png"}  
   
 Date.hour <- substring(hour$time[i],1,10)  
 Time.hour <- substring(hour$time[i],12,23)  
   
 lis[[i]] <- list(i,hour$longitude[i],hour$latitude[i],icon,hour$place[i],hour$depth[i],hour$mag[i],Date.hour,Time.hour)  
 }  
   
   
 #This code creates the variable test directly in javascript for export the grid in the Google Maps API  
 #I have taken this part from:http://stackoverflow.com/questions/26719334/passing-json-data-to-a-javascript-object-with-shiny  
    paste('<script>test=',         
       RJSONIO::toJSON(lis),        
       ';setAllMap();Cities_Markers();',        
       '</script>')  
 }  
 }  
   
 else if(input$Earth==4){  
 month <- read.table("http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv", sep = ",", header = T)  
 if(nrow(month)>0){  
 lis <- list()  
 for(i in 1:nrow(month)){  
   
 if(month$mag[i]<=2){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_green.png"}  
 else if(month$mag[i]>2&month$mag[i]<=4){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_yellow.png"}  
 else if(month$mag[i]>4&month$mag[i]<=6){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_orange.png"}  
 else {icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_red.png"}  
   
 Date.month <- substring(month$time[i],1,10)  
 Time.month <- substring(month$time[i],12,23)  
   
 lis[[i]] <- list(i,month$longitude[i],month$latitude[i],icon,month$place[i],month$depth[i],month$mag[i],Date.month,Time.month)  
 }  
   
   
 #This code creates the variable test directly in javascript for export the grid in the Google Maps API  
 #I have taken this part from:http://stackoverflow.com/questions/26719334/passing-json-data-to-a-javascript-object-with-shiny  
    paste('<script>test=',         
       RJSONIO::toJSON(lis),        
       ';setAllMap();Cities_Markers();',        
       '</script>')  
 }  
 }  
   
   
 else if(input$Earth==3){  
 week <- read.table("http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.csv", sep = ",", header = T)  
 if(nrow(week)>0){  
 lis <- list()  
 for(i in 1:nrow(week)){  
   
 if(week$mag[i]<=2){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_green.png"}  
 else if(week$mag[i]>2&week$mag[i]<=4){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_yellow.png"}  
 else if(week$mag[i]>4&week$mag[i]<=6){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_orange.png"}  
 else {icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_red.png"}  
   
 Date.week <- substring(week$time[i],1,10)  
 Time.week <- substring(week$time[i],12,23)  
   
 lis[[i]] <- list(i,week$longitude[i],week$latitude[i],icon,week$place[i],week$depth[i],week$mag[i],Date.week,Time.week)  
 }  
   
   
 #This code creates the variable test directly in javascript for export the grid in the Google Maps API  
 #I have taken this part from:http://stackoverflow.com/questions/26719334/passing-json-data-to-a-javascript-object-with-shiny  
    paste('<script>test=',         
       RJSONIO::toJSON(lis),        
       ';setAllMap();Cities_Markers();',        
       '</script>')  
 }  
 }  
   
   
   
   
 else {  
 day <- read.table("http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.csv", sep = ",", header = T)  
 if(nrow(day)>0){  
 lis <- list()  
 for(i in 1:nrow(day)){  
   
 if(day$mag[i]<=2){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_green.png"}  
 else if(day$mag[i]>2&day$mag[i]<=4){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_yellow.png"}  
 else if(day$mag[i]>4&day$mag[i]<=6){icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_orange.png"}  
 else {icon="http://maps.gstatic.com/mapfiles/ridefinder-images/mm_20_red.png"}  
   
 Date.day <- substring(day$time[i],1,10)  
 Time.day <- substring(day$time[i],12,23)  
   
 lis[[i]] <- list(i,day$longitude[i],day$latitude[i],icon,day$place[i],day$depth[i],day$mag[i],Date.day,Time.day)  
 }  
   
   
 #This code creates the variable test directly in javascript for export the grid in the Google Maps API  
 #I have taken this part from:http://stackoverflow.com/questions/26719334/passing-json-data-to-a-javascript-object-with-shiny  
    paste('<script>test=',         
       RJSONIO::toJSON(lis),        
       ';setAllMap();Cities_Markers();',        
       '</script>')  
 }  
 }  
 }  
 })  
 })  
   

Index.html

 <!DOCTYPE html>  
 <html>  
 <head>  
 <title>Earthquake Visualization in Shiny</title>  
   
 <!--METADATA-->    
 <meta name="author" content="Fabio Veronesi">  
 <meta name="copyright" content="©Fabio Veronesi">  
 <meta http-equiv="Content-Language" content="en-gb">  
 <meta charset="utf-8"/>  
   
 <style type="text/css">  
   
 html { height: 100% }  
 body { height: 100%; margin: 0; padding: 0 }  
 map-canvas { height: 100%; width:100% }  
   
 .btn {  
  background: #dde6d8;  
  background-image: -webkit-linear-gradient(top, #dde6d8, #859ead);  
  background-image: -moz-linear-gradient(top, #dde6d8, #859ead);  
  background-image: -ms-linear-gradient(top, #dde6d8, #859ead);  
  background-image: -o-linear-gradient(top, #dde6d8, #859ead);  
  background-image: linear-gradient(to bottom, #dde6d8, #859ead);  
  -webkit-border-radius: 7;  
  -moz-border-radius: 7;  
  border-radius: 7px;  
  font-family: Arial;  
  color: #000000;  
  font-size: 20px;  
  padding: 9px 20px 10px 20px;  
  text-decoration: none;  
 }  
   
 .btn:hover {  
  background: #f29f9f;  
  background-image: -webkit-linear-gradient(top, #f29f9f, #ab1111);  
  background-image: -moz-linear-gradient(top, #f29f9f, #ab1111);  
  background-image: -ms-linear-gradient(top, #f29f9f, #ab1111);  
  background-image: -o-linear-gradient(top, #f29f9f, #ab1111);  
  background-image: linear-gradient(to bottom, #f29f9f, #ab1111);  
  text-decoration: none;  
 }  
   
 </style>  
   
        
 <script type="text/javascript" src="http://google-maps-utility-library-v3.googlecode.com/svn/tags/markerclusterer/1.0/src/markerclusterer.js"></script>  
   
 <script src="https://maps.googleapis.com/maps/api/js?v=3.exp&signed_in=true&libraries=drawing"></script>  
   
 <script type="application/shiny-singletons"></script>  
 <script type="application/html-dependencies">json2[2014.02.04];jquery[1.11.0];shiny[0.11.1];ionrangeslider[2.0.2];bootstrap[3.3.1]</script>  
 <script src="shared/json2-min.js"></script>  
 <script src="shared/jquery.min.js"></script>  
 <link href="shared/shiny.css" rel="stylesheet" />  
 <script src="shared/shiny.min.js"></script>  
 <link href="shared/ionrangeslider/css/normalize.css" rel="stylesheet" />  
 <link href="shared/ionrangeslider/css/ion.rangeSlider.css" rel="stylesheet" />  
 <link href="shared/ionrangeslider/css/ion.rangeSlider.skinShiny.css" rel="stylesheet" />  
 <script src="shared/ionrangeslider/js/ion.rangeSlider.min.js"></script>  
 <link href="shared/bootstrap/css/bootstrap.min.css" rel="stylesheet" />  
 <script src="shared/bootstrap/js/bootstrap.min.js"></script>  
 <script src="shared/bootstrap/shim/html5shiv.min.js"></script>  
 <script src="shared/bootstrap/shim/respond.min.js"></script>  
   
        
      <script type="text/javascript">  
      var map = null;  
      var Gmarkers = [];  
        
   
      function Cities_Markers() {  
             
           var infowindow = new google.maps.InfoWindow({ maxWidth: 500,maxHeight:500 });  
   
           //Loop to add markers to the map based on the JSON exported from R, which is within the variable test  
           for (var i = 0; i < test.length; i++) {   
                var lat = test[i][2]  
                var lng = test[i][1]  
                var marker = new google.maps.Marker({  
                     position: new google.maps.LatLng(lat, lng),  
                     title: 'test',  
                     map: map,  
                     icon:test[i][3]  
                });  
             
           //This sets up the infowindow  
           google.maps.event.addListener(marker, 'click', (function(marker, i) {  
                return function() {  
                     infowindow.setContent('<div id="content"><p><b>Location</b> = '+  
                     test[i][4]+'<p>'+  
                     '<b>Depth</b> = '+test[i][5]+'Km <p>'+  
                     '<b>Magnitude</b> = '+test[i][6]+ '<p>'+  
                     '<b>Date</b> = '+test[i][7]+'<p>'+  
                     '<b>Time</b> = '+test[i][8]+'</div>');  
                     infowindow.open(map, marker);  
                }  
                })(marker, i));  
           Gmarkers.push(marker);  
           };  
             
   
      };  
   
      //Function to remove all the markers from the map  
      function setAllMap() {  
           for (var i = 0; i < Gmarkers.length; i++) {  
                Gmarkers[i].setMap(null);  
           }  
 }  
        
      //Initialize the map  
      function initialize() {  
           var mapOptions = {  
           center: new google.maps.LatLng(31.6, 0),  
           zoom: 3,  
           mapTypeId: google.maps.MapTypeId.TERRAIN  
      };  
   
      map = new google.maps.Map(document.getElementById('map-canvas'),mapOptions);  
        
    
 }  
   
   
 google.maps.event.addDomListener(window, 'load', initialize);  
   </script>  
        
   
 </head>  
   
   
 <body>  
   
  <div id="json" class="shiny-html-output"></div>  
   
    
  <button type="button" class="btn" id="hour" onClick="Shiny.onInputChange('Earth', 1)" style="position:absolute;top:1%;left:1%;width:100px;z-index:999">Last Hour</button>  
  <button type="button" class="btn" id="day" onClick="Shiny.onInputChange('Earth', 2)" style="position:absolute;top:1%;left:10%;width:100px;z-index:999">Last Day</button>  
  <button type="button" class="btn" id="week" onClick="Shiny.onInputChange('Earth', 3)" style="position:absolute;top:1%;left:20%;width:100px;z-index:999">Last Week</button>  
  <button type="button" class="btn" id="month" onClick="Shiny.onInputChange('Earth', 4)" style="position:absolute;top:1%;left:30%;width:100px;z-index:999">Last Month</button>  
    
  <div id="map-canvas" style="top:0%;right:0%;width:100%;height:100%;z-index:1"></div>  
    
    
    
 </body>  
 </html>  

Created with CodeFormatter

Exchange data between R and the Google Maps API using Shiny

A couple of years ago I wrote a post about using Shiny to exchange data between the Google Maps API and R: http://r-video-tutorial.blogspot.ch/2013/07/interfacing-r-and-google-maps.html

Back then as far as I remember Shiny did not allow a direct exchange of data between javascript, therefore I had to improvise and extract data indirectly using an external table. In other words, that work was not really good!!

The new versions of Shiny however features a function to send data directly from javascript to R:
Shiny.onInputChange

This function can be used to communicate any data from the Google Maps API to R. Starting from this I thought about creating an example where I use the Google Maps API to draw a rectangle on the map, send the coordinates of the rectangle to R, create a grid of random point inside it and then plot them as markers on the map. This was I can exchange data back and forth from the two platforms.

For this experiment we do not need  an ui.R file, but a custom html page. Thus we need to create a folder named "www" in the shiny-server folder and add an index.html file.
Let's look at the HTML and javascript code for this page:

 <!DOCTYPE html>  
 <html>  
 <head>  
 <title>TEST</title>  
   
 <!--METADATA-->    
 <meta name="author" content="Fabio Veronesi">  
 <meta name="copyright" content="©Fabio Veronesi">  
 <meta http-equiv="Content-Language" content="en-gb">  
 <meta charset="utf-8"/>  
   
 <style type="text/css">  
   
 html { height: 100% }  
 body { height: 100%; margin: 0; padding: 0 }  
 #map-canvas { height: 100%; width:100% }  
   
 </style>  
   
      
   
 <script type="text/javascript"  
    src="https://maps.googleapis.com/maps/api/js?&sensor=false&language=en">  
   </script>  
        
 <script type="text/javascript" src="http://google-maps-utility-library-v3.googlecode.com/svn/tags/markerclusterer/1.0/src/markerclusterer.js"></script>  
   
 <script src="https://maps.googleapis.com/maps/api/js?v=3.exp&signed_in=true&libraries=drawing"></script>  
   
   
   
        
 <script type="text/javascript">  
      //We need to create the variables map and cluster before the function  
      var cluster = null;  
      var map = null;  
        
      //This function takes the variable test, which is the json we will create with R and creates markers from it  
      function Cities_Markers() {  
           if (cluster) {  
                cluster.clearMarkers();  
                }  
           var Gmarkers = [];  
           var infowindow = new google.maps.InfoWindow({ maxWidth: 500,maxHeight:500 });  
   
           for (var i = 0; i < test.length; i++) {   
                var lat = test[i][2]  
                var lng = test[i][1]  
                var marker = new google.maps.Marker({  
                     position: new google.maps.LatLng(lat, lng),  
                     title: 'test',  
                     map: map  
                });  
         
           google.maps.event.addListener(marker, 'click', (function(marker, i) {  
                return function() {  
                     infowindow.setContent('test');  
                     infowindow.open(map, marker);  
                }  
                })(marker, i));  
           Gmarkers.push(marker);  
           };  
           cluster = new MarkerClusterer(map,Gmarkers);  
           $("div#field_name").text("Showing Cities");  
      };  
   
   
      //Initialize the map  
      function initialize() {  
           var mapOptions = {  
           center: new google.maps.LatLng(54.12, -2.20),  
           zoom: 5  
      };  
   
      map = new google.maps.Map(document.getElementById('map-canvas'),mapOptions);  
        
   
      //This is the Drawing manager of the Google Maps API. This is the standard code you can find here:https://developers.google.com/maps/documentation/javascript/drawinglayer  
  var drawingManager = new google.maps.drawing.DrawingManager({  
   drawingMode: google.maps.drawing.OverlayType.MARKER,  
   drawingControl: true,  
   drawingControlOptions: {  
    position: google.maps.ControlPosition.TOP_CENTER,  
    drawingModes: [  
     google.maps.drawing.OverlayType.RECTANGLE  
    ]  
   },  
   
   rectangleOptions: {   
    fillOpacity: 0,  
    strokeWeight: 1,  
    clickable: true,  
    editable: false,  
    zIndex: 1  
   }  
        
  });  
   
//This function listen to the drawing manager and after you draw the rectangle it extract the coordinates of the NE and SW corners
  google.maps.event.addListener(drawingManager, 'rectanglecomplete', function(rectangle) {  
   var ne = rectangle.getBounds().getNorthEast();  
      var sw = rectangle.getBounds().getSouthWest();  
   
      //The following code is used to import the coordinates of the NE and SW corners of the rectangle into R  
      Shiny.onInputChange("NE1", ne.lat());  
      Shiny.onInputChange("NE2", ne.lng());  
      Shiny.onInputChange("SW1", sw.lat());  
      Shiny.onInputChange("SW2", sw.lng());  
        
 });  
   
   
  drawingManager.setMap(map);  
    
 }  
   
   
 google.maps.event.addDomListener(window, 'load', initialize);  
</script>  
        
   
        
   
 <script type="application/shiny-singletons"></script>  
 <script type="application/html-dependencies">json2[2014.02.04];jquery[1.11.0];shiny[0.11.1];bootstrap[3.3.1]</script>  
 <script src="shared/json2-min.js"></script>  
 <script src="shared/jquery.min.js"></script>  
 <link href="shared/shiny.css" rel="stylesheet" />  
 <script src="shared/shiny.min.js"></script>  
 <meta name="viewport" content="width=device-width, initial-scale=1" />  
 <link href="shared/bootstrap/css/bootstrap.min.css" rel="stylesheet" />  
 <script src="shared/bootstrap/js/bootstrap.min.js"></script>  
 <script src="shared/bootstrap/shim/html5shiv.min.js"></script>  
 <script src="shared/bootstrap/shim/respond.min.js"></script>  
   
 </head>  
   
   
 <body>  
   
  <div id="json" class="shiny-html-output"></div>  
  <div id="map-canvas"></div>  
    
 </body>  
 </html>  

As you know an HTML page has two main elements: head and body.
In the head we put all the style of the page, the metadata and the javascript code. In the body we put the elements that would be visible to the user.

After some basic metadata (written in orange), such as Title, Author and Copyright, we find a style section (in yellow) with the style of the Google Maps API. This is standard code that you can find here, where they explain how to create a simple page with google maps: Getting Started

Below we have some script calls (in blue) where we import some elements we would need to run the rest of the code. We have here the scripts to run the Google Maps API itself, plus the script to run the drawing manager, which is used to draw a rectangle onto the map, and the js script to create the clusters from the markers, otherwise we would have too many overlapping icons.

Afterward we can write the core script of the Google Maps API; here I highlighted the start and the end of the script in red and all the comments in pink so that you can work out the subdivision I made.

First of all we need to declare two variables, map and cluster as null. This is because these two variables are used in the subsequent function and if we do not declare them the function will not work. Then we can define a function, which I call Cities_Marker() because I have taken the code directly from Audioramio. This function takes a json, stored into a variable called test, loops through it and creates a mark for each pair of coordinates in the json. Then it cluster the markers.

Afterward there is the code to initialize the map and the drawing manager. The code for the drawing manager can be found here: Drawing Manager

The crucial part of the whole section is the Listener function. This code, as soon as you draw a rectangle on the map, extracts the coordinates of the NE and SW corners and store them in two variables. Then we can use the function Shiny.onInputChange to transfer these variable from javascript to R.

The final step to allow the communication back to javascript from R is create a div element in the body of the page (in blue) of the class "shiny-html-output" with the ID "json". The ID is the part that allow Shiny to identify this element.

Now we can look at the server.R script:

 # server.R  
 library(sp)  
 library(rjson)  
    
 shinyServer(function(input, output, session) {  
    
 output$json <- reactive({  
 if(length(input$NE1)>0){  
   
 #From the Google Maps API we have 4 inputs with the coordinates of the NE and SW corners  
 #using these coordinates we can create a polygon  
 pol <- Polygon(coords=matrix(c(input$NE2,input$NE1,input$NE2,input$SW1,input$SW2,input$SW1,input$SW2,input$NE1),ncol=2,byrow=T))  
 polygon <- SpatialPolygons(list(Polygons(list(pol),ID=1)))  
   
 #Then we can use the polygon to create 100 points randomly  
 grid <- spsample(polygon,n=100,type="random")  
   
 #In order to use the function toJSON we first need to create a list  
 lis <- list()  
 for(i in 1:100){  
 lis[[i]] <- list(i,grid$x[i],grid$y[i])  
 }  
   
 #This code creates the variable test directly in javascript for export the grid in the Google Maps API  
 #I have taken this part from:http://stackoverflow.com/questions/26719334/passing-json-data-to-a-javascript-object-with-shiny  
    paste('<script>test=',   
       RJSONIO::toJSON(lis),  
       ';Cities_Markers();', # print 1 data line to console  
       '</script>')  
      }  
  })  
 })  

For this script we need two packages: sp and rjson.
The first is needed to create the polygon and the grid, the second to create the json that we need to export to the webpage.

Shiny communicates with the page using the IDs of the elements in the HTML body. In this case we created a div called "json", and in Shiny we use output$json to send code to this element.
Within the reactive function I first inserted an if sentence to avoid the script to start if no polygon has been drawn yet. As soon as the user draws a polygon onto the map, the four coordinates are transmitted to R and used to create a polygon (in blue). Then we can create a random grid within the polygon area with the function spSample (in orange).

Subsequently we need to create a list with the coordinates of the points, because the function toJSON takes a list as main argument.
The crucial part of the R script is the one written in red. Here we basically take the list of coordinates, we transform it into a json file and we embed it into the div element as HTML code.
This part was taken from this post: http://stackoverflow.com/questions/26719334/passing-json-data-to-a-javascript-object-with-shiny

This allow R to transmit its results to the Google Maps API as a variable named test, which contains a json file. As you can see from the code, right after the json file we run the function Cities_Markers(), which takes the variable test and creates markers on the map.


Conclusion
This way we have demonstrated how to exchanged data back and forth between R and the Google Maps API using Shiny.

Run Shiny app on a Ubuntu server on the Amazon Cloud

This guide is more for self reference than anything else. 
Since I struggled for two days trying to find all the correct setting to complete this task, gathering information from several websites, I decided to write a little guide on this blog so that if I want to do it again in the future and I do not remember anything (this happens a lot!!) at least I have something to resuscitate my memory.

I found most of the information and code I used from these websites:
http://tylerhunt.co/2014/03/amazon-web-services-rstudio/
http://www.howtogeek.com/howto/41560/how-to-get-ssh-command-line-access-to-windows-7-using-cygwin/
http://www.rstudio.com/products/shiny/download-server/


Preface
First I would like to point out that this guide assumes you (I am talking to myself of the future) remember how to open an instance in the Amazon Cloud. It is not that difficult, you go to this page:
http://aws.amazon.com/ec2/

you log in (if you remember the credentials) and you should see the "Amazon Web Services" page, here you can select EC2 and launch an instance. Remember to select the correct server from the menu on the top right corner since the last time you run all the instances from Oregon, and you live in freaking Switzerland!!


Guide
NOTE
Instead of installing cygwin and cover step 1 and 2 we can just first cover step 4 and connect to the ubuntu server using WinSCP, then start putty from WinSCP and it will be already connected.


1) Install Cygwin
This software is needed to communicate with the Ubuntu server.
It is important to follow the instructions on this page (http://www.howtogeek.com/howto/41560/how-to-get-ssh-command-line-access-to-windows-7-using-cygwin/) to install the software correctly.
In particular during the installation process a "select packages" windows appears where we need to select openssh and click on "skip", until there is a cross on the column bin.

When Cygwin is installed we need to click with the right button on the icon and select "run as administrator", then open it.
Now we can run the following line to install ssh:

ssh-host-config

During the process several questions will be asked, the following answers apply:
- Should privilege separation be used? YES
- New local account sshd? YES
- Run ssh as a service? YES
- Enter a value for daemon:  ntsec
- Do you want to use a different name? NO
- Create a new privilege account user? YES  -> then insert a password


After the installation we need to insert the following line to start the sshd service:

net start sshd

Then this line to configure the service:

ssh-user-config

Again it will ask a series of questions. There is a difference between the new version and what is written on the website.
Now it asks only about an SSH2 RSA identity file to be created, the answer is YES.
Then it asks other two questions regarding DSA files and another thing, the answers here are two NO.



2) Connect to the Amazon Server
Open Cygwin.
Go to the folder where the .pem file is saved, using the following line:

cd D:/<folder>/<folder>



NOTE:
Cygwin does not like folder names with spaces!


Now we need to be sure that the .pem key will not be publicly available using the following line

chmod 400 <NAME>.pem

and then we can connect to the ubuntu server using the following line:

ssh -i <NAME>.pem ubuntu@<PUBLIC IP>

These information are provided in Amazon if we click on "Connect" once the instance has properly been launched.
Once we are in we can installing R and Shiny.


3) Install R and Shiny
The first thing to do is set up the root user with the following line:

sudo passwd root

The system will ask to input a password.
Then we can log in using the following line:

su

Now we are logged in as root users.

Now we need to update everything with the following:

apt-get update

At this point we can install R with the following line:

apt-get install r-base


NOTE:
It may be that during the installation process an older version of R is installed and this may create problems with some packages.
To solve this problem we need to modify the file sources.list located in /etc/apt/sources.list
We can do this by using WinSCP, but first we need to be sure that we have access to the folder.
We should run the following two lines:

cd /etc/

chmod 777 apt

This gives us access to modify the files in the folder apt via WinSCP (see point 4).
This line of code gives indiscriminate access to the folder, so it is not super secure.

Now we can connect and add the following line at the end:

deb http://cran.stat.ucla.edu/bin/linux/ubuntu trusty/

Then we need to first remove the old version of R using:

apt-get remove r-base

or

apt-get remove r-base-dev

Then we need to run once again both the update and the installation calls.



We can check the amount of disk space left on the server using the following command:

df -h


Then we can start R just by typing R in the console.
At this point we need to install all the packages we would need to run shiny using standard R code:

install.packages("raster")


Now we can exit from R with q() and install shiny suing the following line in ubuntu:

sudo su - \
-c "R -e \"install.packages('shiny', repos='http://cran.rstudio.com/')\""

Now we need to install gdebi with the following lines (check here for any update:http://www.rstudio.com/products/shiny/download-server/):

apt-get install gdebi-core
wget http://download3.rstudio.org/ubuntu-12.04/x86_64/shiny-server-1.3.0.403-amd64.deb
gdebi shiny-server-1.3.0.403-amd64.deb



4) Transfer file between windows and Ubuntu
We can use WinSCP (http://winscp.net/eng/index.php) for this task.

First of all we need to import the .pem file that is need for the authentication.
From the "New Site" window we can go to "advanced", then click on "SSH -> Authentication".
From the "Private key file" we can browse and open the .pem file. We need to transform it into a ppk file but we can do that using the default settings.
We can just click on "Save private key" to save the ppk file and then import it again on the same field, and click OK.

Now the Host name is the name of the instance, for example:
ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com

The user name is ubuntu and the password is left blank. The protocol is SFTP and the port is 22.

The shiny server is located in the folder /srv/shiny-server

We need to give WinSCP access to this folder using again the command

cd srv
chmod 777 shiny-server





5) Transfer shiny app files on the server
In WinSCP open the folder /srv/shiny-server and create a new folder with the name of your shiny app.
Then transfer the files from your PC to the folder.
Remember to change the address of the files or working directory in the R script with the new links.


6) Allow the port 3838 to access the web
To do this we need to change the rule in the "security groups" menu.
This menu is visible in the main window where all your instances are shown. However, do not access the menu from the panel of the left, that area may be a bit confusing.
Instead select the instance in which you need to add the rule, look at the window at the bottom of the page (the one that shows the name of the instance) and click on the name in light blue near "security group".
This will open the security group menu specific for the instance you selected.
Here we need to add a rule by clicking on "add rule", select custom TCP, write the port number 3838 in the correct area and the select "anywhere" in the IP section.

Now if you go to the following page you should see the app:
<PUBLIC IP>:3838/<FOLDER>

It is sometimes necessary to open port 22 as well, using the same procedure to keep connecting to the server with cygwin or putty.


7) Stop, start and restart Shiny server

sudo start shiny-server

sudo stop shiny-server

sudo restart shiny-server


8) Installing rgdal
This package require some tweaks before its installations. On this site I found what I needed: http://askubuntu.com/questions/206593/how-to-install-rgdal-on-ubuntu-12-10

Basically from the Ubuntu console just run these three lines of code:
sudo apt-get install aptitude
sudo aptitude install libgdal-dev
sudo aptitude install libproj-dev

Then go back to R and install rgdal normally (with install.packages)



9) Installing rCharts
Before installing rCharts we need to run the line above to install rgdal (I also run the first two lines suggested in the comments, but I do not know if it helped). I also run the following lines from here: http://stackoverflow.com/questions/16363144/install-rcharts-package-on-r-2-15-2

sudo apt-get install libcurl4-openssl-dev
sudo apt-get install openjdk-6-jdk
export LD_LIBRARY_PATH=/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64/server
R CMD javareconf 

but I do not know if they helped or not. 

If we not do that "devtools" will not install and therefore we will not be able to install rCharts from github:

require(devtools)
install_github('rCharts', 'ramnathv')

To run rCharts from the ubuntu server we need to make sure that the links to the javascript library are not referring to


NOTE:
You may need to add packages in R after the installation. For doing that you always need to remember to access ubuntu as root user, so first thing to do is write the code:

su

and insert the password. Now you can start R and install the packages. Otherwise the lib folder where the packages are installed would not be accessible.




#UPDATE from Mike Rutter
To make the install easier, add the following PPAs:

sudo apt-add-repository ppa:marutter/rrutter
sudo apt-add-repository ppa:marutter/c2d4u

The first is the same as the CRAN repository, but you don't need to edit "sources.list". The second has over 2,500 R packages ready to install. For example:

sudo apt-get install r-cran-raster r-cran-rgdal r-cran-shiny

will install the R packages mentioned in the the post. No need to install the "dev" packages either, as that will be taken care of by apt. And the will be updated via the regular Ubuntu update process.

NOTE
Without updating the file in /etc/apt we can just run the first two lines suggested by Mike, then remove r-base and re-install it to have the updated version.

Interfacing R and Google maps

Introduction

I couple of weeks ago I had an idea for a website where people can collaborate to create the first real Audio Atlas, using the power of the Google Maps API. The problem was that I do some programming in R but I did know very few things about HTML and javascript. However, I knew that having a project was a good way to get serious about learning a bit of these two languages. So I started reading some books about how to use the Google API, I borrowed some code from other, more experienced, programmers (whom I thank very very much!!!) and in the end I created a website called Audioramio.com. Here you can come, record your voice and add it to the map. If everyone helps a tiny bit we can try and build the first Audio Atlas!!

While I was writing the code to build this site I realize that it would be very cool to be able to interface the Google maps API with R. I thought about it because in the past it happened that I created some map of soil properties where on the same location I had multiple data to show. The classic example is that you perform kriging and you end up with the actual estimation, plus the uncertainty. Normally, what you do is showing two maps or if you are familiar with webGIS you can produce an interactive website where the user can select which of the two maps to show. However, it would be good to have a way to perform some kind of analysis on these data “on the fly”. So I start thinking at a way to do exactly that, to create a website where on one end you have the map, while on the other it shows some plot or at least a summary of the data. I start looking into it and I found out that the team at RStudio had created a magnificent tool, called Shiny, which is able to create an interactive webpage in HTML and javascript where the user can change interactively some parameters and the page react accordingly, changing the plot or the summary. This is just one example: spark.rstudio.com/uafsnap/RV_distributions

To create a Shiny app you simply create two .R scripts, once for the user interface and one for the server side where you specify the analysis you want to perform and the variable the user can tweak. For more info you can read the tutorial or take a look at this blog, by Matt Leonawicz.

The interesting thing is that Shiny is also able to work with the server.r script, plus a user interface completely created in HTML5 and javascript. So potentially Shiny can be used for almost everything, and specifically it can be used to plot a Google map on the side of a Histogram, for example. The problem with this idea is that it is difficult to let the two application share information. Shiny has been built around an interactive console concept, meaning that the HTML pages are created for the user to interact with R inputs and generate dynamic outputs. However, with the Google Maps API, as far as I know, it is not possible to extract data from the markers on the map. You can add an infowindow with lots of information, but it is impossible to export these data to other applications. I solved this by working around it. I created a .json file with an array of all the coordinates of the points. Then in javascript I created a simple loop that generates, from the coordinates, a series of markers. Instead of attaching an infowindow to them, I created a function that for each elements in the loop, when the point is clicked it updates a certain text form in the Shiny interface. It seems complicated, but it is actually very basic stuff. I attached an image of the console, so that I can better explain.



As you can see, the Shiny console is very basic. There is a single text input and a single plot output. The text input is connected to a subset call, and the input number is the row of the data file to be subseted. So when I click on the point, its ID goes to update the ID field (in the Shiny interface), and the focus goes on that. Now, I need to click “enter” 2 times and the plot is updated.

Code Description

Now let’s take a closer look at the code, starting from the interface.
In Red you have the bits of code related to the Google maps API, while in Green you have the bits related to Shiny.

<!DOCTYPE html>
<html>

<head>
<title>Interfacing R and Google maps</title>
<meta charset=utf-8">
<script src="http://code.jquery.com/jquery-1.10.2.min.js" type="text/javascript"></script>
<script src="shared/shiny.js" type="text/javascript"></script>
<link rel="stylesheet" type="text/css" href="shared/slider/css/jquery.slider.min.css"/>
<script src="shared/slider/js/jquery.slider.min.js"></script>
<link rel="stylesheet" type="text/css" href="shared/shiny.css"/> 
<script type="text/javascript"
      src="https://maps.googleapis.com/maps/api/js?&sensor=false&language=en">
    </script>
 <script type="text/javascript" src="http://www.fabioveronesi.net/ShinyApp/Points2.json"></script>
  
<script type="text/javascript">
  var cluster = null;
  
  
  function SetValue(i) {
   document.getElementById("row").value = i;
   document.getElementById("row").focus();
   }
   
  
   
      function initialize() {
        var mapOptions = {
          center: new google.maps.LatLng(51.781436,-1.03363),
          zoom: 8,
          mapTypeId: google.maps.MapTypeId.ROADMAP
        };
        var map = new google.maps.Map(document.getElementById("map-canvas"),
            mapOptions);
   
  
  
  var Layer0 = new google.maps.KmlLayer("http://www.fabioveronesi.net/ShinyApp/layer0.kml");
  var Layer1 = new google.maps.KmlLayer("http://www.fabioveronesi.net/ShinyApp/layer1.kml");
  
  document.getElementById('lay0').onclick = function() {
  Layer0.setMap(map);
  };
  
  document.getElementById('lay1').onclick = function() {
  Layer1.setMap(map);
  };
  
  var Gmarkers = [];
  var infowindow = new google.maps.InfoWindow();

  for (var i = 0; i < Points.length; i++) {  
   var lat = Points[i][1]
   var lng = Points[i][0]
   var marker = new google.maps.Marker({
    position: new google.maps.LatLng(lat, lng),
    title: i.toString(),
    icon: 'http://www.fabioveronesi.net/ShinyApp/icon.png',
    map: map
   });
   
  google.maps.event.addListener(marker, 'click', 
   (function(i) {
   return function() {
    SetValue(i+1);
    
   }
   })(i));
   
  
   
  Gmarkers.push(marker);
  };
  

  
  document.getElementById('clear').onclick = function() {
  Layer1.setMap(null);
  Layer0.setMap(null);
  };
  
  };
  


 google.maps.event.addDomListener(window, 'load', initialize);
</script> 
  
  
</head>
 
<body>
  <h1>Interfacing R with Google maps</h1>

<label for="row">ID:</label>
<input name="row" id="row" type="number" value="1"/>  

<button type="button" id="lay1">Add Mean</button> 
<button type="button" id="lay0">Add SD</button> 
<button type="button" id="clear">Clear Map</button>  
 
 
 
<div id="plot" class="shiny-plot-output" 
       style="position:absolute;top:20%;right:2%;width: 40%; height: 40%"></div> 
  
<div id="map-canvas" style="position:absolute;top:20%;left:2%;width: 50% ; height: 50%"></div> 
</body>

</html>

As you can see I have two Shiny elements: the plot of the right side and and ID text input on the top left. These are the elements that interact directly with R. To these, I simply added a map frame plus three buttons to show my data and clear the frame, if needed. The communication between Shiny and the API is done purely by these lines:

google.maps.event.addListener(marker, 'click', 
 (function(i) {
  return function() {
   SetValue(i+1);
    
  }
 })(i));

What this says is that when I click on any marker the custom function SetValue will kick in. This function simply change the value in the ID text field to the one of i, which is the element of the loop and the ID of the marker.

Now, let's see what this ID text field controls. Here is the server.R code:

library(shiny)

data_file<-read.table("http://www.fabioveronesi.net/ShinyApp/point_grid.csv",sep=";",header=T) 


shinyServer(function(input,output){

    output$plot <- renderPlot({
  sub<-data_file[input$row,]
  data<-rnorm(10000,mean=sub$Wind_Media,sd=sub$StDev_smoo)
        hist(data)
    })
     

})

The R code is extremely simple. I have one input, ID, which is used to subset the data_file data.frame and one output, which is the histogram plot.

Problems with this approach

Now let’s talk about the problems with this approach.

The first is that the plot does not update automatically, the user needs to click “enter” 2 times before this to happen. However, this may not be a problem after all, simply because as soon as the R script becomes more complex, and its execution time longer, it is good to have a way to avoid updating the plot all the time I accidentally click a point on the map.

The second problem is related to the Google Maps API and the way it shares data. In general, when I plot markers on the map the only way to access their data is clicking on them and look at the infowindow. This is not true for KMZ map layers. When I plot a raster layer on the map it is treated exactly as an image (it is in fact a .png file georeferenced inWGS84), and it is therefore impossible to access its data. The only way to plot a map and give the user a way to access it is by plotting a marker layer on top of it, with invisible icons so that they do not disturb the visualization of the map. When the user clicks of the map, he is clicking on the invisible markers layer that triggers the Id field update. However, if I increase the zoom level the markers get smaller and therefore it becomes more difficult to click on them. So accessing the map is possible only at the zoom level at which it is presented. A way to solve this would be showing (by using a different icon, maybe a point) the markers, but when they are on a grid their visual impact is not very pretty at all.

Conclusions

I think this approach has the potential to be used for very cool mapping experiments. It relies directly on all the packages available in R and therefore it can virtually visualize every sort of statistics from the map. However, it is a bit difficult to set up, because you need to transform all of your data into KML, extract the coordinates of your cells into WGS84, and transform them into a .json array. For these steps I used ArcGIS and Notepad++. They are not difficult to complete, but they may take half of your working day or more, depending on your dataset.

A possible, quicker, alternative is http://www.jstat.org/. However, I never used it and so I do not know how to set it up for working with the Google Maps API. In addition, I do not think it has the same potential as R for performing mind blowing statistics.

Download

To run the App on your PC just open R, install the package shiny and run these two lines:

library(shiny)

runUrl("http://www.fabioveronesi.net/ShinyApp/InterfacingRGoogleMaps.zip")