Expedia Hosted by 2lemetry Through AWS2lementry (Amazon Web Services)
Logistics & Warehousing
Expedia provisions Hadoop clusters using Amazon Elastic Map Reduce (Amazon EMR) to analyze and process streams of data coming from Expedia’s global network of websites, primarily clickstream, user interaction, and supply data, which is stored on Amazon Simple Storage Service (Amazon S3). Expedia processes approximately 240 requests per second. “The advantage of AWS is that we can use Auto Scaling to match load demand instead of having to maintain capacity for peak load in traditional datacenters,” comments Gopalan. Expedia uses AWS CloudFormation with Chef to deploy its entire front and backend stack into its Amazon Virtual Private Cloud (Amazon VPC) environment. Expedia uses a multi-region, multi-availability zone architecture with a proprietary DNS service to add resiliency to the applications. Figure 2 demonstrates the architecture of the GDE service on AWS.
To simplify the management of GDE, Expedia developed an identity federation broker that uses AWS Identity and Access Management(AWS IAM) and the AWS Security Token Service (AWS STS). The federation broker allows systems administrators and developers to use their existing Windows Active Directory (AD) accounts to single sign-on (SSO) to the AWS Management Console. In doing so, Expedia eliminates the need to create IAM users and maintain multiple environments where user identities are stored. Federation broker users sign into their Windows machines with their existing Active Directory credentials, browse to the federation broker, and transparently log into the AWS Management Console. This allows Expedia to enforce password and permissions management within their existing directory and to enforce group policies and other governance rules. Additionally, if an employee ever leaves the company or takes a different role, Expedia simply make changes to Active Directory to revoke or changes AWS permissions for the user instead of inside of AWS.
The success of the ESS and GDE services sparked interest from other Expedia development teams, who began to use AWS for regional initiatives. By 2012, Expedia was hosting applications in the US East (Northern Virginia), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Tokyo), and US West (Northern California) Regions. Expedia Worldwide Engineering culled best practices from these initiatives to create a standardized deployment setup across all Regions. As Jun-Dai Bates-Kobashigawa, Principal Software Engineer explains, “We’re using Chef to automate the configuration of the Amazon Elastic Compute Cloud (Amazon EC2) servers. We can take any AWS image and use scripts stored in Chef to build a machine and spin up an instance customized for a team in just in a few minutes.”