- Lessons Learned, Survival-Stories etc. von Firmen mit Amazon-Infrastruktur Before 5 AM Eastern on April 21st, 2011, an outage started at EC2's northern Virginia datacenter that brought down hundreds of web 2.0 and social media websites including Foursquare, Springpad, Reddit, Quora, BigDoor and Hootsuite. Specifically, attempts to use Amazon's elastic-disk and database services hung, failed, or were slow. Service was restored to some parts of the datacenter (three of four "availability zones" in Amazon's terms) by late afternoon Eastern time that day
Amazon Elastic Block Store (EBS) enables a single Amazon EC2 instance to attach to one or more highly available, highly reliable storage volumes of up to 1 TB of data each. Once attached, applications on a single Amazon EC2 instance can read or write from the Amazon EBS volume similar to a disk drive. With Amazon EBS, an Amazon EC2 instance can now be terminated without losing the data that resides on the Amazon EBS volume. One use case involves running a relational database within an Amazon EC2 instance, but maintaining the data within an Amazon EBS volume.
In the months prior to leaving Heavy, I led an exciting project to build a hosting platform for our online products on top of Amazon’s Elastic Compute Cloud (EC2). We eventually launched our newest product at Heavy using EC2 as the primary hosting platform. I’ve been following a lot of what other people have been doing with EC2 for data processing and handling big encoding or rendering jobs. We set out to build a fairly standard LAMP hosting infrastructure where we could easily and quickly add additional capacity. In fact, we can add new servers to our production pool in under 20 minutes, from the time we call the “run instance” API at EC2, to the time when public traffic begins hitting the new server. This includes machine startup time, adding custom server config files and cron jobs, rolling out application code, running smoke tests, and adding the machine to public DNS. What follows is a general outline of how we do this.
Disco is an oss implementation of the Map-Reduce framework for distributed computing. Disco supports parallel computations over large data sets on unreliable cluster of computers. The Disco core is written in Erlang. Users of Disco typically write jobs in Python, which makes it possible to express even complex algorithms or data processing tasks often only in tens of lines of code. This means that you can quickly write scripts to process massive amounts of data. Disco was started at Nokia Research Center as a lightweight framework for rapid scripting of distributed data processing tasks. This far Disco has been succesfully used, for instance, in parsing and reformatting data, data clustering, probabilistic modelling, data mining, full-text indexing, and log analysis with hundreds of gigabytes of real-world data. Linux is the only supported platform but you can run Disco in the Amazon's Elastic Computing Cloud.
Stax provides easy deployment of test and production environments, a local development model, and strong integration with existing development tools, frameworks, and processes. Get Started * Jump start your development with built-in application templates for popular Java technologies including Struts, GWT, Wicket, JRuby on Rails, Jython, Adobe Flex and ColdFusion * Create the MySQL databases required for your application. * Deploy to multiple environments such as test, staging, and production to support your application's lifecycle. Develop * Develop on your workstation using the Stax SDK, which provides a local reproduction of server production environment. * Use any editor or IDE (Eclipse projects are automatically generated for new applications). * Stax Ant plug-ins let you integrate Stax with your existing Java projects. Deploy * Publish applications from your local environment to the Stax cloud on Amazon EC2.
Cloud Tools is a set of tools for deploying, managing and testing Java EE applications on Amazon's Elastic Computing Cloud (EC2) and VMware environments. There are three main parts to Cloud Tools:
* EC2Deploy - the core framework. This framework manages virtual instances (e.g. EC2), configures MySQL, Tomcat, Terracotta and Apache and deploys the application. See this blog entry for an overview.
* Maven and Grails plugins that use EC2Deploy to deploy an application
* Amazon Machine Images (AMIs) that are configured to run Tomcat and work with EC2Deploy. See list of installed software.
Ubuntu Server Edition on Amazon - Beta programme now open Ubuntu Server Edition on Amazon gives you the power of Ubuntu combined with the flexibility of Amazon's cloud computing service. Ubuntu's modularity, virtualization capabilities, range of applications and optimised performance make it the perfect solution if you're deploying applications on Amazon's Elastic Computing (EC2) cloud. Ubuntu Server Edition on Amazon has recently been announced as a beta programme. This service allows you to create a fully running instance of Ubuntu Server Edition on EC2 in just a few clicks. All the applications you'll need such as a Web server, E-mail server and common development frameworks are available. The beta programme is open to anyone without charge from Canonical. Beta users will need to create an account with Amazon's Web Services and Amazon will charge you for your usage of EC2.
Apache's Hadoop project aims to solve these problems by providing a framework for running large data processing applications on clusters of commodity hardware. Combined with Amazon EC2 for running the application, and Amazon S3 for storing the data, we can run large jobs very economically. This paper describes how to use Amazon Web Services and Hadoop to run an ad hoc analysis on a large collection of web access logs that otherwise would have cost a prohibitive amount in either time or money.
Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.
Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. Amazon EC2 changes the economics of computing by allowing you to pay only for capacity that you actually use. Amazon EC2 provides developers the tools to build failure resilient applications and isolate themselves from common failure scenarios.
James Gardner The Pages About The Search search site archives The Associates Andrew Chapman Ben Bangert Ian Bicking Jason Kottke Signal vs Noise Simon Willison Vecosys The Storage September 2007 August 2007 July 2007 June 2007 May 2007 April 2007 March