An auto-scaling Shiny Server on AWS
The code for this post is available in this GitHub repository
This post describes how to create your own Shiny Server on an EC2 instance on Amazon’s AWS. It consists of two parts:
- We’ll create just one server which auto-pulls your apps from a GitHub repository every few minutes
- Then, we create an auto scaling group so that your Shiny Server is elastic and spawns more servers when your apps suddenly get famous
The second part is optional. You can of course create just your single server.
Shiny apps are a way to build interactive data analyses and dashboards, which can be viewed from your web browser, in R. You can play around with parameters of your analysis, e.g. the timeframe on the x-axis, and instantly get an updated analysis (plots etc.) just for that part.
Some nice example apps can be found here. For a nice, simple example app see here.
You can run Shiny apps locally, when R is installed on your computer. To show your apps to the general public, you must host these apps on a Shiny Server. There is a service for that available at http://www.shinyapps.io/, but you can also run your own server for free.
1. Set up a single-instance Shiny Server
To set up a Shiny Server on a single instance, follow the steps in this part. If you want an auto-scaling, load balanced group of servers, you must do these steps too, because you have to create an AMI of your final configured server.
- Create a new VPC for your server(s). If you want auto-scaling and load
balancing later, make sure to add multiple public subnets, each in a different
availability zone. This ensures your server is still available if a single
availability zone’s data center ever fails. This is unlikely, but if you can
protect against it relatively easily, then why not?
- My VPC has the CIDR block 10.0.0.0/16, and the three subnets are 10.0..0/24
- Make sure to set ‘Auto-assign public IPv4 address’ to yes in each subnet.
- Also, make sure the Route Tables are routing traffic to 0.0.0.0/0 to the internet gateway, and that they are associated with each subnet.
- Create a big enough EC2 instance on AWS. As of now, a t2.micro is too small
and will fail while compiling the necessary R packages, but a t2.small works
- Use Ubuntu Server as the OS.
- Use a security group that allows SSH and port 3838 (the Shiny Server default port)
- Download my git repository with the setup scripts:
git clone https://github.com/AlexEngelhardt/auto-scaling-aws-shiny-server.git
- Make the script
bootstrap-EC2-instance.shexecutable and run it to set up the instance:
- This script also downloads my repository of shiny apps which will be served
- These two steps can be done with a bootstrap script from the AWS console, too.
- Now enter the public IP in your browser and you should see your Shiny server!
2. Add auto-scaling and load balancing
The steps in part 1 result in a running Shiny Server. But what if your apps suddenly get super famous? Your single EC2 instance will likely crash under the heavy load. AWS can create an auto scaling group, automatically creating extra “copies” of your webserver and then randomly distribute incoming traffic among these copies.
So, if you want to make your serverhighly available, that is, it should not break if your apps suddenly become highly accessed, you should enable auto scaling. Then, additional EC2 instances are spawned as soon as the existing one(s) are under too much load. A load balancer in front of the auto scaling group then distributes traffic among these instances.
Here’s how to do that:
- Create an image (an AMI) from your instance. Call it “Shiny Server golden image” or something.
- Create an Application Load Balancer.
- Create a new Target Group here. You don’t have to register actual instances there yet (because you don’t yet have them)
- You’ll get a DNS name for your ALB. Point your browser to this domain to let the ALB distribute the traffic to its EC2 instances
- Create a Launch Configuration in the Auto Scaling section of EC2, using your newly created AMI.
- Then create an Auto Scaling Group using this launch configuration.
- Under “advanced details”, make sure to tick “Receive traffic from one or more load balancers” and choose the Target Group you’ve created earlier.
- Use a scaling policy that keeps the average CPU utilization at 70 (percent). This way, new instances will be spawned as soon as the CPU gets too busy.
- Now you should see EC2 instances starting, because the Auto Scaling Group is
what actually spawns them.
- The spawned instances have their own IPv4 addresses, and you could browse to them directly. But the point of the load balanced ASG was to enter the load balancer’s domain name and he’ll decide which actual EC2 instance to direct your traffic to.
- You can also experiment and manually terminate an EC2 instance. The ASG should spin up a new one relatively quickly.
- The load balancer’s DNS names are ugly and long. Ideally, you’d create a subdomain of your personal homepage. I have chosen http://shiny.alpha-epsilon.de, for example.
Verify the load balancer works
To really verify my load balancer distributes the traffic, I’ve added a cronjob on my AMI that prints the public IP address in the Shiny Server’s index.html.
This only works for AWS EC2 instancers. To do so, create this script as
export IPV4=`curl http://169.254.169.254/latest/meta-data/public-ipv4` sudo sed -i "s@.*</body>@This instance's IP: $IPV4</body>@" /srv/shiny-server/index.html
Then add this line to your
crontab, which runs the script every minute:
* * * * * root /home/ubuntu/add-ip.sh