Showing posts with label AWS. Show all posts
Showing posts with label AWS. Show all posts

Friday, 13 February 2015

Big Data In The Cloud

ETL and DWH are my fav and still that is my full time job, but I am just loving to work on some technical POC's of Hadoop and Big Data since last year.

Data is becoming more valuable, its really true. So the demand and value for the Big Data professionals in the market is high these days. That is the reason companies and products based on Hadoop are taking high positions.

As per my knowledge, today Cloudera is leading the technology quadrant.

Recently I was reading a paper published by Intel IT center, and found the information worth to share.

The paper describes how cloud and big data technologies are converging to offer a cost-effective delivery model for cloud-based big data analytics. It also includes: How cloud computing is an enabler for advanced analytics with big data. How IT can assume leadership for cloud-based big data analytics in the enterprise by becoming a broker of cloud services. Analytics-as-a-service (AaaS) models for cloud-based big data analytics. Practical next steps to get you started on your cloud-based big data analytics initiative.

The paper explains about data analytics is moving from batch to real time
 and real time supports predictive analytics.
you can read more on this here.

**************************************************************************** 

Apache Hadoop : is a set of algorithms (an open-source software framework written in Java) for distributed storage and distributed processing of very large data sets (Big Data) on computer clusters built from commodity hardware. All the modules in Hadoop are designed with a fundamental assumption that hardware failures (of individual machines, or racks of machines) are commonplace and thus should be automatically handled in software by the framework.

Cloudera Inc. : is an American-based software company that provides Apache Hadoop-based software, support and services, and training to business customers.
Cloudera's open-source Apache Hadoop distribution, CDH (Cloudera Distribution Including Apache Hadoop), targets enterprise-class deployments of that technology. 

***********************************************************

Monday, 29 July 2013

Amazon EC2...Amazing Cloud....

Amazon EC2’s simple web service interface allows enterprises to obtain and configure capacity with minimal friction. It provides enterprises with complete control of their computing resources that run on Amazon’s proven computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing enterprises to quickly scale capacity, both up and down, as the enterprise's computing requirements change. Amazon EC2 changes the economics of computing by allowing enterprises to pay only for capacity actually used. Amazon EC2 provides developers the tools to build failure resilient applications and isolate themselves from common failure scenarios.

On-Demand Instances Pricing



Amazon EC2 On-DEmand Instances Pricing


Reserved Instances Pricing



Amazon EC2 Reserved Instances Pricing

Spot Instances Pricing

Amazon EC2 Spot Instances Pricing
Spot Instances pricing fluctuates periodically depending on the supply of and demand for Spot Instance capacity. The illustration below takes a snapshot pricing for the EU Region at Wednesday January 13 10:28:06 UTC 2010.

Internet Data Transfer Pricing

The pricing below is based on data transferred "in" and "out" of Amazon EC2.

There is no Data Transfer charge between Amazon EC2 and other Amazon Web Services within the same region (i.e. between Amazon EC2 US West and Amazon S3 in US West). Data transferred between Amazon EC2 instances located in different Availability Zones in the same Region will be charged Regional Data Transfer. Data transferred between AWS services in different regions will be charged as Internet Data Transfer on both sides of the transfer.

Amazon Internet Data Transfer Pricing



Amazon Elastic Block Storage (EBS) Pricing


AWS Import/Export Service

AWS now offers physical data import/export service makes it easy to quickly transfer large amounts of data into and out of the AWS Cloud. It is an economical alternative to sending large volumes of data across the Internet. The AWS Import/Export service allow 2TB of data to be imported or exported globally from AWS S3. With that service, customers can send Amazon a blank storage device and Amazon will copy the contents of one or more Amazon S3 buckets to it before shipping it back. Alternatively, customers can send Amazon a storage device full of data that Amazon will copy it to the S3 buckets of the customer's choice. Customers can use AWS Import/Export for:

· Data Migration
· Offsite Backup
· Direct Data Interchange
· Disaster Recovery

Wednesday, 1 May 2013

Case Study - Cloud Computing Opportunities


Opportunities
The ICT team of GEC believes that cloud computing can help evaluate, plan, design and implement a dynamically scalable and virtualized software engineering environment. As such, cloud computing is perceived as an opportunity to build a more flexible and cost-effective development and test computing environment for the engineers working at GEC. It should allow the organization to accelerate the product life-cycle and reduce costs by centralizing and automating the deployment, configuration and tearing-down of this complex environment. Through this strategic decision, GEC expects to obtain the following benefits.
·        Reduce capital expenses while at the same timeoffering the elastic scalability to handle fluctuating business needs.
·  Reduce dependency by allowing engineers to allocate and manage computing resources themselves with minimal intervention from the ICT staff.
·  Reduce operating and labor costs of managing, deploying and supporting software test configurations while improving productivity by eliminating the time spent on the setting and tear-down of the lab environment.
·     Facilitate innovation and improve time-to-market by reducing development and testing setup times from weeks to minutes.
·   Improve quality by eliminating critical software issues prior to deployment through testing in environments that more closely match production environments.
·       Maximize efficiency from a centralized lab supporting multiple and/or remote teams.
Action Plan
The action plan for the migration of the development and test computing environment to the cloud is broken down as follows:
The analysis will be based on the Amazon Web Services (AWS) EC2 and EBS services, which cater for a world-class infrastructure-as-a-service cloud computing platform.
Sourcing of the development and test environment to the AWS cloud will rely on the Amazon Virtual Private Cloud (Amazon VPC) architecture. Amazon VPC enables enterprises to connect their existing infrastructure to a set of isolated AWS computing resources via a Virtual Private Network (VPN) connection, and to extend their existing management capabilities such as security services, firewalls and intrusion detection systems to include their AWS resources in a hybrid architecture that takes full advantage of the benefits of the AWS cloud in a secured and dedicated area as shown in illustration 2.
AWS Cloud
Illustration 2: Amazon Virtual Private Cloud Architecture (Graphic Courtesy of Amazon)
The ICT staff will create set Amazon Machine Image (AMI) templates—for the Solaris, Windows and Linux operating systems—containing the software necessary (middleware, libraries, data and associated configuration settings) for engineers to perform their customary tasks as if they were working locally.
One small AWS EC2 reserved instance will be permanently allocated per engineer. In addition, engineers will be able to allocate additional EC2 instances on demand through a modified version of the Virtual Instance Reservation Portal (VIRP) application. On average, engineers will be able to allocate two more small instances on demand when development and test tasks require running software on multiple nodes.
Development and test data will be hosted on Amazon Elastic Bloc Storage (EBS) volumes. Amazon EBS provides block-level storage volumes for use with Amazon EC2 instances. Amazon EBS volumes are off-instance storage that persist independently from the life of an instance. Amazon EBS provides highly available, highly reliable storage volumes that are particularly suited for applications that require a database, file system, or access to raw block-level storage. Amazon EBS also provides the ability to create point-in-time snapshots of volumes. These snapshots can be used as the starting point for new Amazon EBS volumes, and protect data for long-term durability. The same snapshot can be used to instantiate as many volumes as is needed. As such, engineers will be able to maintain multiple versions of the software components under development as well as multiple configurations and testing scenario setups without needing to hold a complete virtual machine for such purposes.
Developers will be allowed to create new EBS volumes of various sizes, ranging from as little as a few giga bytes to hundreds of giga bytes of storage, through the VIRP application and mount them onto their EC2 AMI instances dynamically. It is estimated that engineers will need on average no more than three EBS volumes ranging from 70 to 250 GB in size.
The number of managed servers—which for the most part are inefficient legacy machines whose use and purpose are not always clear to the IT team — and as such the virtual machines equivalent, will be drastically reduced. By transferring its computing assets to the cloud, GEC will also be able to “pull the plug”. on those unattended or quiescent systems and avoid the effect of server sprawling in the future.