DIY Basic AWS EC2 Dashboard using Apache, Python, Flask and boto (PartII)

In part I of this tutorial we covered the basic stack setup as well as showing boto/Flask usage. In this part, I'll show how to handle posts, request instance details and rendering dynamic URLs.

So, arguably, you have your index page as your EC2 instance dashboard already working. Now, building on that let's say that you want to see the detail of any of the instances in display. So as per my explanation of the ideosyncracies of AWS' API, in order to see such details, you need two basic pieces of information: the region and the instance id. The way we are going pass this information from the dashboard page to Flask is through URLs. Flask has pretty neat ad-hoc URL routing with replaceable variables that you can then use in your code. So, in this instance, in the dashboard page we are going to dynamically generate a link that contains both the region end-point and the specific instance id. If you look closely, to the index.html template the link looks as follows:

<a href="/details/{{region['Name']}}/{{instance['Id']}}">See Details</button>

So, now we need to tell Flask to "listen" to that URL pattern. To do so, add the following line to your [WEB_APP_NAME].py file (please bear in mind that this is only for the sake of illustration only; so I'm putting code style aside):

@app.route("/details/<region_nm>/<instance_id>")

This tells Flask to match incoming requests to that URL and bind the incoming parameters to the variable names inside the angled brackets "<" ">".  Right below that line declare the method/function you are going to use explicitly declaring the parameters you expect.

def details(region_nm=None,instance_id=None)

That is it in so far as Flask is concerned. Now that we, presumably, have all the data we need, we leverage boto to do the "hard work" for us. So as seen in part I of this tutorial, whenever we need to issue calls to the AWS API, the first thing we need to do is to start a connection to whatever region you are interested in. So, we go ahead and do so:

rconn = boto.ec2.connect_to_region(region_nm,aws_access_key_id=akeyid,aws_secret_access_key=seckey)

With that active regional connection, we then query the API for the details of a particular instance (which in this case we are going to use the instance_id passed in):

instance = rconn.get_all_instances([instance_id])[0].instances[0]

the API call above takes an array of instance ids (in our case we are only interested in one) and it will return not the instance itself, but an array of the parent reservation that why we have that first [0] is for and then that reservation's instances collection with only one instance object in it, the one we requested. Once we have the instance object it should be clear and obvious that you can extract any information you want or need (type, tags, state, etc.).

Now what if we wanted to make changes to any of the instances or if we wanted to, say, start/stop any of them? It's actually not unlike what we've been doing thus far: we tell Flask route object what to look for, then use that machinery to get the data you want. So, for this step, as a matter of generally acceptable good practices, we are going to send the information via submitting a form through a POST request method. Our route should look something like this:

@app.route("/change", methods=['POST'])

We can now define out function to handle the request:

def change():

Within this method we can now leverage Flask's request object to get the form data, for instance:

        instance_id = request.form['instance_id']
        state_toggle = request.form['sstoggle']
        toggle_change = request.form['type_change']

And finally, to make changes to an instance, all you need is to modify the instance's attributes key-value pairs. Let's say we wanted to change the instance type, to do so we simply change the value via boto's modify_attribute method as shown below:

        instance.modify_attribute('instanceType', 'm1.large')

One thing to bear in mind is that regretfully AWS API does not provide a "valid types" list for each instance. So, if you are dealing with a mix of 32- and 64-bit machines, it is possible you can assign an instance a type it is not compatible with, so you must be mindful about that. Given that there is no list provided by the API also means you need to hard-code the instance types. A reference to the official instance type list can be found here (pay attention to the API name field).

To change the instance state, however, you do not use the attributes collection directly. Instead, use the API calls provided by boto to start/stop/terminate:

         instance.stop()
         instance.start() 
...

Hope this tutorial was of help. If you got questions or comments, feel free to ping me on Twitter @WallOfFire.

DIY Basic AWS EC2 Dashboard using Apache, Python, Flask and boto (Part I)

While Amazon Web Services offers a nice web-based UI to handle and manage EC2 instances, it might well be the case that you do wish to give access to some of this functionality to more people in your organization, but you do not with to provide them with full access to the AWS EC2 dashboard or wish to limit the type of API calls they make (for instance, you might want to allow users to start/stop instances, but you do now want them to be able to launch/terminate them). Whatever your use case might be, you can create your own "in-house" EC2 Dashboard with relative ease. Our software stack will consist of:

• Apache (for basic authentication, SSL and WSGI, virtual hosts) with mod_ssl and mod_wsgi.


• OpenSSL (for SSL and certificate generation).


• Python.


• Flask (py-based web services micro-framework).


• boto (py-based AWS API library).


• AWS KeyID/SecretKey credentials.


• Admin/sudo privileges

Please note, if you do not need SSL/htpasswd you can use Flask's bundled web server which is suitable for most in-house deployments; however in this example, I will be using Apache. Also to note: I'm not going to spend time in the installation process of the packages/SSL certicate generation above as it should be fairly straightforward for anyone with minimal dev/sysadmin experience as well as there being many well-written tutorials for the setup of  these tools floating around the 'net.

First we need to make sure your tools are working. Make sure Apache is working, make sure mod_ssl is working, etc. Open up a python prompt and try importing flask, boto and so forth. Once you are fairly confident your tools are good to go then let's get moving.

1. Create an Apache virtual host entry file specifying the SSL certificate location, port number, location of the wsgi file and other important parameters as show below:

<VirtualHost *:443>
        ServerAdmin webmaster@localhost

        DocumentRoot /var/www/[WEB_APP_NAME]
        SSLEngine On
        SSLCertificateFile /path/to/certs/server.crt
        SSLCertificateKeyFile /path/to/certs/server.key
        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>

        WSGIDaemonProcess [WEB_APP_NAME] user=[APACHE_USER] group=[APACHE_USER_GROUP] threads=5
        WSGIScriptAlias / /var/www/[WEB_APP_NAME]/[WEB_APP_NAME].wsgi

        <Directory /var/www/[WEB_APP_NAME]>
                WSGIProcessGroup [WEB_APP_NAME]
                WSGIApplicationGroup %{GLOBAL}
                WSGIScriptReloading On
                Options Indexes FollowSymLinks MultiViews
                AllowOverride All #important line for using htpasswd
                Order allow,deny
                allow from all
        </Directory>

		#other settings here

</VirtualHost>

2. We then need to write the wsgi file telling mod_wsgi which app and instance it should start when a request comes along. It's a very simple file. Just make sure its name matches the ones you provided in the vhost file above. Its contents should look something like this:

from [WEB_APP_NAME] import app as application

One thing to bear in mind is that this way of using wsgi requires that your web app be anywhere in the $PYTHONPATH environment variable. Save, close and restart Apache.

3. For this example, as said above, I'm working under the assumption that your authentication needs are very basic and those requirements can be fulfilled with Apache's htpasswd. Assuming you setup Apache correctly with the right mods, you can go to your application's root directory and tell htpasswd to create a password file (.htpasswd) with a username-password pair. You can do so by running the following command:

sudo htpasswd -c .htpasswd [USERNAME]

It will then prompt you for a password which it will then encrypt using basic symmetric encryption algorithms and the create the .htpasswd file.

4. Next step is to setup your .htaccess file so that Apache knows when to ask for the credentials. Your .htaccess should have (at least) the following rules:

AuthUserFile /path/to/[WEB_APP_NAME]/.htpasswd
AuthGroupFile /dev/null
AuthName "EnterPassword"
AuthType Basic
require valid-user

You can test it by pointing your web browser to whatever URL you set up for this site. You should now be prompted with a username/password dialog.

5. Now that we have the basics ready, let us get to the meat and substance. First I highly suggest you give a quick perusal to Flask's "Quickstart" tutorial which can be found here and trying out the first few trivial examples to make sure you have everything setup correctly.

6. Make a new file named [WEB_APP_NAME].py and copy-and-paste the text below:

from flask import Flask, flash, abort, redirect, url_for, request, render_template
from boto.ec2.connection import EC2Connection
import boto.ec2
app = Flask(__name__)
akeyid = '[AWS_KEY_ID]'
seckey = '[AWS_SECRET_KEY]'
conn = EC2Connection(akeyid,seckey)

7. One of the concepts to bear in mind with respect to AWS API connections is regions. There is no global end-point for your AWS API calls and calls made to that region's API only make sense for services within that Region. For instance, you can't "see" your instances in the west coast Region ("us-west-1") from any other region. So, whatever regions you wish to have access to, you need to specify those explicitly. By default, boto connects to the "us-east-1" region.

8. So, our index page will be the Dashboard itself, that is to say, a place where users will be able to see all the instances from all regions. You can choose to limit the regions you wish to show/scour fairly easily, but for the sake of this example I'm going to simply gather all the instance information from all of AWS' regions. I will the create an object data structure with all the data and eventually I'll pass that data to the template rendering engine that Flask comes with.  So, we're going to create an app route for the index page, use boto to create an EC2 connection, retrieve all available regions, get all the instance reservations in that region, get all the instances within each of those regions, then bundle the data in a data structure and pass it to the rendering engine.

a. Create the index route:

@app.route("/")

b.  declare your method:

def my_method_name():

c.  retrieve the list of all AWS available regions:

        allinfo  = []
        regions = conn.get_all_regions()
        for region in regions:

d. connect to each of those regions and retrieve all the instance reservations (something to note: I'm not sure if it's boto's or AWS' boo-boo, but retrieve_all_instances() method does not retrieve all instances per se, instead it retrieves all the instance reservations, which are instance "containers") :

                rconn = EC2Connection(akeyid,seckey,region=region)
                rsvs = rconn.get_all_instances() #read note above

e. loop over the reservations and gather all the instance information:

                   for rsv in rsvs:
                        insts = rsv.instances
                        for inst in insts:
                               #do stuff

f. now all together (including populating our instance info data structure and pass to the rendering engine):

@app.route("/")
def my_method_name():
        allinfo  = []
        regions = conn.get_all_regions()
        for region in regions:
                regioninfo = {}
                regioninfo['Name'] = region.name
                rconn = EC2Connection(akeyid,seckey,region=region)
                rsvs = rconn.get_all_instances()
                instances = []
                for rsv in rsvs:
                        insts = rsv.instances
                        for inst in insts:
                                instances.append({'Id': inst.id, 'Name':inst.tags['Name'],'State':inst.state, 'Type':inst.get_attribute('instanceType')['instanceType']})
                regioninfo['instances'] = instances
                allinfo.append(regioninfo)

        return render_template('index.html',all_info=allinfo)

9. Now that we have the route and the code, we are going to use Flask's nifty template rendering engine (Jinja). To do so we need to create a file that matches the name in the render_template call above.

a. create a file with nano or your favorite text editor named index.html (or whatever name you chose ). This file has to be (by Flask convention) inside a directory called 'templates' and this directory should be at the same level as your web app Py script.

b. copy-paste the following "boilerplate" html:

<!doctype html>
<html>
<head><title>EC2 Dashboard</title>
</head>
<body>
<div class="header">Welcome to EC2 Dashboard</div>
<div class="content">
<div class="region-text">Regions Available</div>
{% for region in all_info %}
	{% if region['instances'] %}
		<div class="region-info"><span style="font-weight:bold">Region: {{region['Name']}}</span>
		<span>Instances Avaliable</span></div>
		<div class="region-content">
			<table><tr><th>Instance Name</th><th>Instance State</th><th>Instance Type</th><th>Instance Id</th><th>Instance Actions</th></tr>
			{% for instance in region['instances'] %}
					<tr>
					<td><span>{{instance['Name']}}</span></td>
					<td><span>{{instance['State']}}</span></td>
					<td><span>{{instance['Type']}}</span></td>
					<td><span>{{instance['Id']}}</span></td>
					<td>
					<a href="/details/{{region['Name']}}/{{instance['Id']}}">See Details</button>
					</td>
					</tr>
			{% endfor %}
			</table>
		</div>
	{% else %}
	<div class="region-info">
		<span style="font-weight:bold">Region: {{region['Name']}}</span>
		<span>No Instances Avaliable in this region</span>
	</div>
	{% endif %}
{% endfor %}
</div>
</body>
</html>

It should be obvious that you can (and should!) use your own html and CSS styles. The above example was to illustrate the rendering engine usage and syntax, which is pretty self explanatory and simple. Flask's and Jinja's documentation is very good and when in doubt you should consult those as your primary source.

This concludes part I of this tutorial. In part II I will then show how to post information, how to use boto for modifying instance information and more on Flask's routing/url facilities.

Installing Citrix XenApp 6 Fundamentals on Amazon EC2 (from scratch)

Let me begin by saying that it has been a rather painful experience to learn/deal with Citrix XenApp 6. The documentation Citrix provides in their site regarding XenApp on Amazon EC2 is either outdated or slightly inaccurate (and therein a bigger problem). The problem is fundamental and one I'm afraid of Citrix has done on purpose: even a small mistake during installation can spoil your install forever and leaving you with very little options other than to start anew (literally! start from a clean OS image). The same is also true for upgrading/downgrading XenApp: it's just not possible to uninstall and upgrade. You must do fresh install from scratch. Citrix's XenApp forums seem to be packed with troubleshooting threads full of  "I have that problem too"  replies most of which with no official response/answer. They have two official blog entries specific to XenApp on EC2, but they are either no longer true or assume you have substantial knowledge of Citrix XenApp (and all its tricky parts). So, in this blog entry, I'm going to try to document my steps so that if should someone else find themselves on the same boat, they can at least go over these steps and see if they are of any help.

For the steps that follow, I'm going to assume that, just like me, the reader has little or no experience installing or administering Citrix XenApp and that the XenApp install is intended for external access to apps (sans VPN). I'm further going to assume the reader has an active EC2 account and that he or she is able to launch new instances and that he or she is well aware that this will incur in charges in accordance to AWS EC2 pricing.

1. Launch (i.e. new) a large instance with Microsoft Windows Server 2008 R2 with SQL Server Express and IIS (AMI Id: ami-42bd442b). SQL Server and IIS are requirements. Make sure you do so with a valid keypair so that you can later retrieve and decrypt the auto-generated Administrator password.

2. After waiting a few minutes retrieving the instance's auto-generated admin password, fire up your favorite Remote Desktop client and start a session to the instance you just launched.

3. Download (to the instance) and install whatever ISO-mounting software you prefer and install, I use PowerISO.

4. In your instance go to start menu and type "EC2Config" and wait until "EC2ConfigService Settings" shows up, select to run it.

4.a. Once the utility comes up, make sure to uncheck "Set Computer Name", "Initialize Drives", and "Set Password". Click Apply then OK.

5. Go to start menu, find "Computer", right click, click on Properties in the contextual menu.

5.a. In the Computer name, domain and workgroup settings click on Change settings, then click on the Change... button and give your computer a new name (prefearably something you easily remember and you'll use this name for installation and licensing steps as well).

5.b. Click OK couple times and you'll be prompted to reboot, click ok and then click on Close. Reboot.

6. Go to Citrix -> Product And Solutions -> XenApp -> Try (here) -> choose "Turnkey solution for small businesses up to 75 users" follow registration procedures until you're prompted to download and you're given a license number. If you already have a license number for XenApp 6 Fundamentals, you can try downloading directly here.

7. Enable .NET 3.5 (and make sure you don't have .NET 4+ installed).

8. Mount the ISO as a logical drive (with PowerISO or whatever ISO tool you use).

9. In your file explorer navigate to: {DRIVE LETTER}:\W2k8 and click on setup.exe (after agreeing to the license terms, ofter a dialog warning will show up telling you that other users might be logged on, ignore that and click OK). Please note that setup.exe must be "Run as Administrator", else it will fail.

9.a. In the setup workflow set Application Server as installation type.

9.b. Since this will be a test/trial environment, select "Disable Shadowing" then click next.

10. Set your admin username and password. I recommend using the same Domain\UserName and password you have in that machine.

Hopefully the installation will now complete successfully. If it failed along the way, it can be rather difficult to debug since the log file messages are rather devoid of any semantic meaning. As a last ditch effort, try uninstalling the very last module that failed to install/configure. and the try setup.exe again (of course, don't forget to "Run as Administrator").

 

Re/Sources:

PowerCram

Citrix XenApp on EC2 blog entry