AWS, R, RStudio, Parallel processing

In this post I will share my experiences with using a spot AMI instance for heavy parallel processing of R scripts.

Fist we start a Rstudio AMI:

Start a spot instance:

Request:

Search AMI:

Search for Rstudio:

select the needed compute power:

Its recommended to look at the pricing history:

In this case, we select a General Purpuse 16 CPU machine, which has a fair price per hour. Since we only use the machine for some heavy processing, we will decommission it in a few hours.

Leave the other settings as is.

select next, be sure to create a new key pair, in case you do not have the original key pair.

Its also good to add a new security group which has port 80 open:

open the ports:

The click 'Launch' instance, and then click op the capacity link;

Look up the public IP:

You should now be able to log on:

Log on credentials can be found here:
http://www.louisaslett.com/RStudio_AMI/

Now that the server is up and running we can start using R packages to allow parallel processing:

  
install.packages("doParallel",dependencies=TRUE)
install.packages("doMC",dependencies=TRUE)

We use the package

  
library(doParallel)
library(foreach)
library(doMC)

In order to initialize we;
- set the number of cores.
- initialize a cluster
-and we need to export our functions, variables and datasets to the clusters:

  
#prep parralel processing
# Calculate the number of cores
no_cores <- detectCores() - 1
# Initiate cluster
cl <- makeCluster(no_cores)
registerDoMC(no_cores) 
clusterExport(cl=cl, varlist=c("splitter", "create_ngram_table"))

There are a number of options for parralel processing roughly speaking you can choose from ParLapply and forEach. When you would like to split your dataset in multiple subsets and want to have a worker node perform an operation this can easily be done with a forEach, see example below. In both cases a list is returned. While we usualy would like to have a full dataset in return. In order to achieve this result I used the code do.call("rbind", result).

Many more examples can be found on: http://gforge.se/2015/02/how-to-go-parallel-in-r-basics-tips/

  
 result = foreach(j=seq(1,nl, by=(nl/parts))) %dopar% {
      }
    }
  }  
  do.call("rbind", result)
}

Build your custom Geo solution for QlikView in 5 minutes

In this post I will show how to build a Geo plot without the need for GeoAnalytics or Indevio. It will be way more limited functionality wise, you will be able to plot your data on a map. The result will look like this:

First you need a map of which you need to know the lat and long coordinates of the square corners. For this navigate to:
https://www.darrinward.com/lat-long/?id=2778191

Enter a set of coordinates for the corners:

Take a screen capture starting and ending on the specified corners:

In QlikView add a new scatter chart:

Add the lat and long dimensions:

Specify the axis:

Enter next untill this screen. Deselect 'forced 0' and enter the x and y axis ranges, by using the values you used on lat-long website:

As background image to the chart load the screen capture of the lat-long website:

Press finish and your result will look like this:

Obviously, you need to geo code your adress data. This can easily be done by most ETL tools.

Hope this is usefull.

BI and Analytics on your action cam video (GoPro, TomTom Bandit, etc)

In this post I will show some tips and tricks to add BI to your action cam video, specifically the TomTom Bandit, on Windows 10 with Dashware.

The result looks like this:

To start with the really tricky part, getting Dashware to work on Windows 10.

I also tried to use Racerender, but Dashware proved to be the tool of choice for me. Unfortunately, the latest version does not work on windows 10. I did get it to work with version 1.7. Which can be downloaded here: http://dashware.software.informer.com/1.7/

In order to get it working you first need to remove MFPLat.dll and then run the executable from the program folder:

Since I am using the TomTom Bandit, I will also extract the GPX information from the video. This can be done from TomTom Bandit Studio:

Right click the video to export the GPS data.

Then we start a new project in Dashware, remember to start the application by running the exe from the program folder. Select 'None'  as project template.

Click the plus to add a video:

We choose not to extract the GPS data:

Then we add the data file:

Select to synch the GPS file:

Now the fun part, we add our BI components:

And start customizing!

Enjoy!

Copy Paste Images from Clipboard in Blogger / Blogspot

I have always found it a bit annoying that it's necessary to upload images to Blogspot before being able to display them:

For this reason, I started looking for a windows application with a paste from clipboard option for images. Unfortunately, I was not able to find a free/open source solution. For this reason, I decided to build a workaround.

In this post I will explain how this works.

The basic idea is to use GreenShot for image capturing, an AWS server that hosts the images, a PuTTY pscp script to upload the file, and to copy a HTML text to the clipboard which you can paste into blogger.

This shows how the final solution works:

First start a screen capture with Greenshot:

And then select the option "upload". The script will then kick off and upload the screen capture to the AWS Server and return the HTML code that needs to be copied into Blogger:

This code is then copied into the HTML part of blogger:

Within Greenshot its possible to specify external commands. With this option a Bach file is started:

In here we configure the pointer to the batch file:

Also, in the settings, set the file format of the capture to hh mm to make sure the file names do not have spaces.

The batch file looks like this:

This script takes the path of the screen capture as a parameter and then uploads the file to the AWS Webserver. Unfortunately, the clip.exe command turned out the be buggy so I pasted the html code in a text document and then open this document with Notepad.

The webserver is a default (free of charge web-hosting AWS AMI):
First navigate to EC2 for elastic cloud options:

Launch an instance:

Search for the Wordpress AMI:

Select the free hosting option;

Then press next untill you get the option 'launch', press this as well. You will the see the screen to select a key pair, this is important:

Download the PEM file, we need this is the next step:
Use this file to create a pkk file, these steps have been described here:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html

Then, build the transfer script as described in the manual:

This script in part of the upload script:

Text prediction

In this post I would like to show off the text prediction app I created to obtain my Data Science specialization:

Converting webpage to epub file and transfer to Kobo E-reader easily

I like;
- Reading Wikipedia pages, especially regarding the history of nations.
- Reading from my e-reader.
- Reading at times when my laptop is shutdown, f.e. in bed.
- Being able to switch topic spontaneously.

I tried to find a solution for this.

First I tried the web browser on my Kobo e-reader (in beta). Unfortunately, this is very annoying, for a number of reasons. Later I found the website https://dotepub.com/ which allows you to convert a webpage to an epub file online. Unfortunately, this does not work on the e-reader web browser.

For this reason, I figured I will use my iPhone to convert a web page to an epub file. The next challenge would be to transfer the epub file from my Iphone to my e-reader. I tried using google drive, but this was messy in the e-reader web browser. So I need to use SD media. Since the Iphone has no SD slot, wireless transfer is needed. Therefore I bought the Verbatin Mediashare Wireless, which has an SD slot and Wifi capabilities, it also has an iPhone app. Unfortunately, it's impossible to transfer non photo or video media from the Iphone to the Verbatin Mediashare using the app! What a disappointment! I could have known since there a lot of negative reviews on the web with people complaining. But there is good news!

By coincidence, I learned that the Verbabin Mediashare hosts a website which has the option to upload files. When using this upload option you are able to use regular file apps.

In the screens below I show how to use this setup. Don't hesitate to give feedback in the comments section when you have a better solution.

First nagivate to the webpage, with safari(!), that you would like to convert to epub. Then open the bookmark with the dotepub.com scripts (see webpage for instructions).

select more and choose your file management solution ( I am using Documents 5). 

Store it in your folder: 

Connect to the Verbatin Mediashare using Wifi. 

In your browser navigate to 10.10.10.254

press enter. 

and add the file, select the second icon. 

as source choose your file management app, in my case 'Documents 5'.

select the file, in this case 'history of denmark'

select ok.

The file will be stored on the SD card. Shut down the Verbatin Mediashare. And insert the SD card in your e-reader. The e-reader will recognize the new file and you can start reading Wikipedia from your e-reader!