For Schema Migrations Use App Engine Task Queue instead of Backends

Every application under iterative development, eventually is going to need a schema migration of some sort. In our case, we needed key/id field re-alignment as well as adding new common fields to all our models that needed to be populated. Given that even the most innocuous of migrations will probably take longer than the maximum time allowed by an HTTP request, our inclination was to resort to a GAE backend to do the hard work in the background independent of the HTTP requests deadlines. The problem with GAE backends is two-fold: the setup overhead and the dismal documentation and examples that the App Engine team provides -- although, to be fair, the former is exacerbated by the latter. 

Backends have their own .yaml config file with several options that are not entirely clear and what side effect (if any) they have. The documentation for the backends.yaml is here and as you can see chasing down the different choices for a given option can feel like going down the rabbit hole. To make matters worse, you have to "wire" your app's app.yaml with backend.yaml and figure out how they differ and what is their role with respect to backends. Yet another issue that is not entirely clear is how to kick off a backend instance, assuming of course that you have all the pieces in the right place. The docs, again, are dismal, in my opinion. One passage mentions /_ah/start as the way to kick start your backends; however, the docs also mention that a 404 HTTP error code is considered a success. Since there isn't a way to see whether the backend is doing anything or not, is outright confusing to consider a 404 a success. Eventually, after much searching, I came across a way to get the process going via a front-end request; unfortunately this ties the backend to the fron-end deadlines.

As I was about to /flipdesk, I walked through the dark caverns of App Engine's news group, I bumped into a post about best practices for schema migrations. The post mentioned this article posted on Dec, 2012 that lo-and-behold precisely addresses the issues at hand. While it seemed trivial to copy-pasta, I knew this would not apply to our app since we use App Engine's nbd datastore abstraction module. Fortunately, someone put the time to write a ext.db -> ext.ndb cheatsheet that made matters a lot easier. After moving things around a bit and switching the syntax to ndb and adding couple simple lines to app.yaml, I fired up my local dev server and tried it. It worked from the get go. I pushed to our staging server on App Engine's infrastructure, which contains ~40K records. Fired the process up and some ten minutes later, the migration was complete. The best part, I could see its progress and logs thereof through the admin console even how deep the Task Queue was and what task was next in line.

In all, setting up, changing code and testing schema migration thought Task Queue took roughly 1/5th the time I had thus far invested on getting this process running under a backend. So, my recommendation based on my experience is to avoid using backends unless you absolutely have to and instead leverage Task Queue for all other asynchronous/long-running processes.

PyCon 2013: an exhaustive, painfully comprehensive, encyclopedic, unabridged review

Just kidding. I'm going to make it short and sweet.

PyCon was the best tech conference I've been to. The sense of community that permeated the conference is simply bar none. The emphasis on helping out/charity/education like no other tech conference out there. The sponsor-organized after-parties were a blast and by in large, the talks and tutorials were of great value. As a relatively newcomer to Python, the conference was simply a resounding success, specially when you consider the shoestring budget it runs on and that it is organized and run by small contingent of community volunteers. PyLadies, a women's pythonistas outreach organization, was in large part to thank for a rather sizable female attendance to the conference (about twenty percent). Overall, while it's entirely too early to think about Montreal next year, I'm hoping I'll be able to not only attend but also help the community and organizers in whatever extent I can.

But, I don't want to bore you any further, so below are two links. One is to see the slideshows of the sessions, the other one is to watch the videos of the sessions. Enjoy.

Slides: https://speakerdeck.com/pyconslides/
Videos: http://pyvideo.org/category/33/pycon-us-2013

boto cheat sheet

I've been using python boto for nearly a year and have been greatly impressed with it from the get-go. However, I usually find myself forgetting key methods and parameter for the few AWS services I use the most, namely SQS, DynamoDB, S3 and EC2. So, in order to avoid going through the documentation every time, I made a cheat sheet for the most commonly used methods and functionality of the above said services. You can now download it here. If you see something amiss or inaccurate please let me know.

web2py: an MVC framework that will rock your socks

Living under the shadow of the venerable and much more famous Django (and outside the view of Rails' unwavering zealots), is an amazing MVC framework called web2py. Let me tell you why I find this framework to be absolutely great and why you should give it a try.

First: it's completely self-contained and self-dependent. All it requires to run is python (2.6+) installed in your computer.

Second: Documentation, documentation, documentation. Of all the MVC frameworks out there (including some that are much older and "mature"), I've never seen so much care and effort put into documentation as with web2py. The documentation is not just a collection of half-assed tutorials with a few code snippets here and there with some commentary and whatnot. No, web2py's documentation is actually a full-fledged and very well written book which you can read online or download and print via PDF. But don't be fooled thinking that just because it's a book that it will be a "slow" book or that it will be full of technical platitudes. On the contrary, its focus in the few chapters is to get you up and running fast without neglecting very useful technical side notes and leaving the thick of deep technical explanation for later chapters.  The documentation leaves the likes of django's and rail' in the dust.

Third: Meta-admin interface included. That's right, if you thought django's "freebie" admin interface was rad, wait till you see web2py's. It's not just an app-specific admin panel (like django's), it's also a panel to admin all of your available apps their contents and settings.  It's a really awesome feature, specially if you are developing/maintaining more than one web app.

Fourth: Built-in web-based IDE? Yes, web-based built-in IDE. When was the last time that, out of the box, you could have a rather complete web-based IDE to get your scaffolding, code generators, models and views and get you started with all the rest of your coding (Zen Conding included) with Rails --or django for that matter? Hmmm I'm guessing never. You could, theoretically, never have to run a shell command (other than to start the web2py driver app) to develop your app. I'd like to emphasize, it's not just a gimmick for "ooo's and ahhh's". It's a very powerful and time-saving tool.

Fifth: Adhering to standards and very well suited for RESTful apps. Classic and elegant /noun/verb (corresponding to controller/action) URLs makes app design straight forward.

Sixth: Built-ins. Web2py has an incredible collection of built-in widgets, forms, validators, types, and so forth. These are great for turn-key proof-of-concepts (or production-ready apps, if these do what you need).

Seventh: top-notch security. This is something the team who developed web2py worked hard to tackle. Web2py addresses OWASP's top ten security issues/vectors throughout their apps. That way you can focus on coding functionality and much less time worrying about the plethora of security risks web applications can have.

There's a lot more I could talk about, for instance its powerful database abstraction layer, it's intuitive templating engine , its form generators, its WSGI-compliance (which is almost a requirement to run in production environments), but I leave that as an exercise to the reader :)

DIY Basic AWS EC2 Dashboard using Apache, Python, Flask and boto (Part I)

While Amazon Web Services offers a nice web-based UI to handle and manage EC2 instances, it might well be the case that you do wish to give access to some of this functionality to more people in your organization, but you do not with to provide them with full access to the AWS EC2 dashboard or wish to limit the type of API calls they make (for instance, you might want to allow users to start/stop instances, but you do now want them to be able to launch/terminate them). Whatever your use case might be, you can create your own "in-house" EC2 Dashboard with relative ease. Our software stack will consist of:

• Apache (for basic authentication, SSL and WSGI, virtual hosts) with mod_ssl and mod_wsgi.


• OpenSSL (for SSL and certificate generation).


• Python.


• Flask (py-based web services micro-framework).


• boto (py-based AWS API library).


• AWS KeyID/SecretKey credentials.


• Admin/sudo privileges

Please note, if you do not need SSL/htpasswd you can use Flask's bundled web server which is suitable for most in-house deployments; however in this example, I will be using Apache. Also to note: I'm not going to spend time in the installation process of the packages/SSL certicate generation above as it should be fairly straightforward for anyone with minimal dev/sysadmin experience as well as there being many well-written tutorials for the setup of  these tools floating around the 'net.

First we need to make sure your tools are working. Make sure Apache is working, make sure mod_ssl is working, etc. Open up a python prompt and try importing flask, boto and so forth. Once you are fairly confident your tools are good to go then let's get moving.

1. Create an Apache virtual host entry file specifying the SSL certificate location, port number, location of the wsgi file and other important parameters as show below:

<VirtualHost *:443>
        ServerAdmin webmaster@localhost

        DocumentRoot /var/www/[WEB_APP_NAME]
        SSLEngine On
        SSLCertificateFile /path/to/certs/server.crt
        SSLCertificateKeyFile /path/to/certs/server.key
        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>

        WSGIDaemonProcess [WEB_APP_NAME] user=[APACHE_USER] group=[APACHE_USER_GROUP] threads=5
        WSGIScriptAlias / /var/www/[WEB_APP_NAME]/[WEB_APP_NAME].wsgi

        <Directory /var/www/[WEB_APP_NAME]>
                WSGIProcessGroup [WEB_APP_NAME]
                WSGIApplicationGroup %{GLOBAL}
                WSGIScriptReloading On
                Options Indexes FollowSymLinks MultiViews
                AllowOverride All #important line for using htpasswd
                Order allow,deny
                allow from all
        </Directory>

		#other settings here

</VirtualHost>

2. We then need to write the wsgi file telling mod_wsgi which app and instance it should start when a request comes along. It's a very simple file. Just make sure its name matches the ones you provided in the vhost file above. Its contents should look something like this:

from [WEB_APP_NAME] import app as application

One thing to bear in mind is that this way of using wsgi requires that your web app be anywhere in the $PYTHONPATH environment variable. Save, close and restart Apache.

3. For this example, as said above, I'm working under the assumption that your authentication needs are very basic and those requirements can be fulfilled with Apache's htpasswd. Assuming you setup Apache correctly with the right mods, you can go to your application's root directory and tell htpasswd to create a password file (.htpasswd) with a username-password pair. You can do so by running the following command:

sudo htpasswd -c .htpasswd [USERNAME]

It will then prompt you for a password which it will then encrypt using basic symmetric encryption algorithms and the create the .htpasswd file.

4. Next step is to setup your .htaccess file so that Apache knows when to ask for the credentials. Your .htaccess should have (at least) the following rules:

AuthUserFile /path/to/[WEB_APP_NAME]/.htpasswd
AuthGroupFile /dev/null
AuthName "EnterPassword"
AuthType Basic
require valid-user

You can test it by pointing your web browser to whatever URL you set up for this site. You should now be prompted with a username/password dialog.

5. Now that we have the basics ready, let us get to the meat and substance. First I highly suggest you give a quick perusal to Flask's "Quickstart" tutorial which can be found here and trying out the first few trivial examples to make sure you have everything setup correctly.

6. Make a new file named [WEB_APP_NAME].py and copy-and-paste the text below:

from flask import Flask, flash, abort, redirect, url_for, request, render_template
from boto.ec2.connection import EC2Connection
import boto.ec2
app = Flask(__name__)
akeyid = '[AWS_KEY_ID]'
seckey = '[AWS_SECRET_KEY]'
conn = EC2Connection(akeyid,seckey)

7. One of the concepts to bear in mind with respect to AWS API connections is regions. There is no global end-point for your AWS API calls and calls made to that region's API only make sense for services within that Region. For instance, you can't "see" your instances in the west coast Region ("us-west-1") from any other region. So, whatever regions you wish to have access to, you need to specify those explicitly. By default, boto connects to the "us-east-1" region.

8. So, our index page will be the Dashboard itself, that is to say, a place where users will be able to see all the instances from all regions. You can choose to limit the regions you wish to show/scour fairly easily, but for the sake of this example I'm going to simply gather all the instance information from all of AWS' regions. I will the create an object data structure with all the data and eventually I'll pass that data to the template rendering engine that Flask comes with.  So, we're going to create an app route for the index page, use boto to create an EC2 connection, retrieve all available regions, get all the instance reservations in that region, get all the instances within each of those regions, then bundle the data in a data structure and pass it to the rendering engine.

a. Create the index route:

@app.route("/")

b.  declare your method:

def my_method_name():

c.  retrieve the list of all AWS available regions:

        allinfo  = []
        regions = conn.get_all_regions()
        for region in regions:

d. connect to each of those regions and retrieve all the instance reservations (something to note: I'm not sure if it's boto's or AWS' boo-boo, but retrieve_all_instances() method does not retrieve all instances per se, instead it retrieves all the instance reservations, which are instance "containers") :

                rconn = EC2Connection(akeyid,seckey,region=region)
                rsvs = rconn.get_all_instances() #read note above

e. loop over the reservations and gather all the instance information:

                   for rsv in rsvs:
                        insts = rsv.instances
                        for inst in insts:
                               #do stuff

f. now all together (including populating our instance info data structure and pass to the rendering engine):

@app.route("/")
def my_method_name():
        allinfo  = []
        regions = conn.get_all_regions()
        for region in regions:
                regioninfo = {}
                regioninfo['Name'] = region.name
                rconn = EC2Connection(akeyid,seckey,region=region)
                rsvs = rconn.get_all_instances()
                instances = []
                for rsv in rsvs:
                        insts = rsv.instances
                        for inst in insts:
                                instances.append({'Id': inst.id, 'Name':inst.tags['Name'],'State':inst.state, 'Type':inst.get_attribute('instanceType')['instanceType']})
                regioninfo['instances'] = instances
                allinfo.append(regioninfo)

        return render_template('index.html',all_info=allinfo)

9. Now that we have the route and the code, we are going to use Flask's nifty template rendering engine (Jinja). To do so we need to create a file that matches the name in the render_template call above.

a. create a file with nano or your favorite text editor named index.html (or whatever name you chose ). This file has to be (by Flask convention) inside a directory called 'templates' and this directory should be at the same level as your web app Py script.

b. copy-paste the following "boilerplate" html:

<!doctype html>
<html>
<head><title>EC2 Dashboard</title>
</head>
<body>
<div class="header">Welcome to EC2 Dashboard</div>
<div class="content">
<div class="region-text">Regions Available</div>
{% for region in all_info %}
	{% if region['instances'] %}
		<div class="region-info"><span style="font-weight:bold">Region: {{region['Name']}}</span>
		<span>Instances Avaliable</span></div>
		<div class="region-content">
			<table><tr><th>Instance Name</th><th>Instance State</th><th>Instance Type</th><th>Instance Id</th><th>Instance Actions</th></tr>
			{% for instance in region['instances'] %}
					<tr>
					<td><span>{{instance['Name']}}</span></td>
					<td><span>{{instance['State']}}</span></td>
					<td><span>{{instance['Type']}}</span></td>
					<td><span>{{instance['Id']}}</span></td>
					<td>
					<a href="/details/{{region['Name']}}/{{instance['Id']}}">See Details</button>
					</td>
					</tr>
			{% endfor %}
			</table>
		</div>
	{% else %}
	<div class="region-info">
		<span style="font-weight:bold">Region: {{region['Name']}}</span>
		<span>No Instances Avaliable in this region</span>
	</div>
	{% endif %}
{% endfor %}
</div>
</body>
</html>

It should be obvious that you can (and should!) use your own html and CSS styles. The above example was to illustrate the rendering engine usage and syntax, which is pretty self explanatory and simple. Flask's and Jinja's documentation is very good and when in doubt you should consult those as your primary source.

This concludes part I of this tutorial. In part II I will then show how to post information, how to use boto for modifying instance information and more on Flask's routing/url facilities.