Should you virtualize your pattern or not?

This post is replicated from the IBM Expert integrated system blog.  Please visit that article if you want to comment or share.


In this post, I compare and contrast the IBM PureSystems virtual application and virtual systems patterns.  I gently introduce you to these two options and help you decide how to select the correct pattern model for your workloads.  In two subsequent posts, I will explain how to create plug-ins to extend IBM PureSystems virtual application patterns.

Defining workloads on the cloud

In most IaaS cloud computing environments, the unit of deployment is the image that gets deployed (or runs) on a virtual machine.  Each image typically comprises operating system software, dependencies, libraries, and utilities, and also the application middleware and software.  So, deploying an image results in a complete software stack.  It is then the responsibility of the developer (or IT operations) to select the correct image and make the necessary customizations and additional setups.

The problem with this approach is that it results in an explosion of base images, each trying to cover one of the various combinations possible for various software stacks.  The other problem is that the customization and setup of images can be non-trivial, resulting in various scripts, patches, and custom tweaking that can be difficult to maintain and keep current.

Fundamentally, the issue is that workloads tend to be built on standard stacks but contain variations.  These variations can be customization information to various ways in which some parts of the stack can be configured.  The resulting software combination, including the VMs running these services constitute what is called a workload topology.

Typically, fixing a workload topology for a particular business process can be difficult because non-functional requirements — for example, security, privacy, scalability, resiliency, and so on — vary with the purpose of the workload.  For instance, deploying an expense reimbursement web-based application requires a very different topology if it is to be used by a local company with about one hundred employees than if it is to be used for a world-wide organization with thousands of employees that are geographically distributed.

Finally, to complicate matters even more, the reality of cloud computing is that failures will occur.  One cannot assume that a workload will work flawlessly once deployed.  Rather, it’s important to accept the reality that various types of failures will occur during the lifetime of the workload.  These failures range from hardware and VM failures, to IaaS management failures, to hypervisor and operating system failures, and to software stack failures.  The sad truth is that all of these mostly software systems contain bugs that will lurk at the inopportune time and cause a visible failure to your customer.  Therefore, a workload must be tunable, restartable, updatable, and malleable.

For these various reasons, the IBM PureApplication System platform provides two means to represent workloads as reusable patterns: virtual systems and virtual applications.

Why two types of patterns?

The two supported types of workload pattern definitions, virtual system pattern and virtual application pattern, both serve the same purpose: to help define an open representation of workload for the cloud.  However, they have also different strengths and capabilities.

Virtual systems allow the definition of workloads as components integrated into cloud images.  These patterns essentially help to easily create cloud images based on the software stack that it contains.  This approach is essentially analogous to providing an editor to typical cloud images.

The second approach for the definition of workloads is with virtual application patterns.  The simplest way to explain what a virtual application pattern is that it is an abstract representation of a workload.  The abstraction is over cloud components and the links between these workloads.  There is less focus on the entire stack or supporting resources, for example, virtual machines (VMs).  Instead, the workload is described by laying out its components, linking them together, and specifying optional parameters.

There are advantages and shortcomings to each approach; however, together they provide a level of flexibility that we hope covers all the bases when defining cloud workloads.

Virtual system patterns

As previously mentioned, virtual system patterns are a visual representation of an image’s content in a template format.  IBM PureSystems provides a graphical representation of the pattern that can then be visually modified.  Using the tooling, a PureSystems administrator can make changes to existing patterns to add new components and scripts for customizations.

The end result is a virtual system pattern that represents one image or a collection of images and be instantiated in a repeatable manner into an IBM PureSystems installation.  This process gives all the advantages of typical IaaS image deployment while providing metadata for the image’s content.  The resulting patterns can be shared and updated as needed.

Although virtual system patterns provide a means to quickly get a workload deployed into the cloud, it also has various drawbacks.  Principally, as we discussed before, approaches that use images for workloads assume that the topologies of these workloads are well defined and do not change much.  However, this assumption does not fit well with modern enterprises that want to be nimble and evolve with current market needs.

Virtual system patterns are thus excellent for repeatable deployment of workloads that are already well defined and do not need to change much in time.

Virtual application patterns

Contrary to virtual systems patterns, the approach of virtual application is to define workloads using higher-level concepts, such as cloud components and cloud services, and also the links between them.  There is no explicit notion of the required cloud resources required to realize the pattern.  Instead the system makes these decision on your behalf, allowing various optimizations that otherwise would be hard to achieve manually.

Although with virtual application patterns, a deployment realization is done by the system, a pattern user and developer can provide hints to the system.  For instance, a workload that contains components that can auto-scale can expose quality of service (QoS) properties that users can use to scale the component.  For instance, the WebSphere Application Server component provides auto-scaling properties to specify that the server should horizontally scale (be duplicated) when the CPU reaches some threshold or the number HTTP requests exceeds some value.

The components from a virtual application pattern are defined through an open plug-in mechanism.  This approach allows anyone to create components or services representing their software services and stack.  Because of the dynamic nature of the virtual application pattern approach, every deployment of a pattern using virtual application components can dynamically fetch the software and install and configure at deployment time.  In this manner, the patterns can evolve and be malleable.

Furthermore, because the system is managing the various components for the workloads, it can also monitor each deployment and recover from various failures.  Indeed, a deployment on an IBM Pure Application Systems that uses virtual application patterns automatically inherits monitoring features such as log viewers, and also VM and service-level monitoring.  The patterns can optionally provide various actions such as a “restart” service action.  In addition to all of this, when a component is scaling in a cluster, the components of the cluster can self-monitor and recover automatically from failures by creating new instances in the cluster when a failure is detected.

One shortcoming of this approach is that deployments can be a bit slower than with a virtual system approach; however, typically in practice, the difference is less than a few minutes of added time in the worst case.  The virtual application pattern approach provides the agility that enterprises require while also enabling the system to provide optimization that would be hard to do manually.

What next?

With the announcement of the IBM PureSystems family of platform-as-a-service and private clouds, IBM also announced hundreds of ISV partners who are committed to provide and support cloud components for the IBM PureSystems.  Most of these ISV cloud components are available today in the IBM PureSystems marketplace.  Using these ISV plug-ins, users of IBM Pure Application Systems can create complex patterns that not only include IBM middleware cloud components, but a wide variety of third-party software and open source software.

In a subsequent post, I will describe how you can create your own plug-ins to extend the IBM Pure Application Systems platform. I will provide instructions of how to create such a plug-in for a popular open source software component.  This plug-in can then be used as one more example to create your own plug-in.

White paper: Use the Apache Libcloud Python API to manage resources on IBM SmartCloud Enterprise

Abstract This article explains the basics of using the Apache Libcloud Python API with IBM SmartCloud Enterprise. The value of the Libcloud API is that it can be used across many different cloud providers. The article will demonstrate how to create and retrieve information about cloud resources and discusses managing virtual resources across clouds. This article is suitable for readers with basic knowledge of Python and a basic knowledge of the IBM SmartCloud Enterprise and who wish to combine the two.

Full article on IBM Techdocs White Paper site

Stormy forecast: Private clouds on the horizon are growing in importance

Wrote this for the IBM Thoughts on Cloud blog and reposting here in its entirety. Please use the ToC site for discussions.

Background

Cloud computing is primarily manifest by the availability of compute capacity (primarily servers, storage, and networks) as virtualize resources.

Using a self-serving portal, available on the web, one can provision a complete unit of computation and set it up with custom or predefined software. When it is no longer needed, the resources are discarded, and you are simply charged for the time you used the resources.

The ease of use and agility that one experiences with public clouds can be replicated in a private manner for enterprises. This is especially useful to enterprises that are worried about security or that want more control over where their data and software are setup and reside. So, private clouds can be seen as a microcosm of public clouds…

Although this view is true, it is also not complete. Private clouds have other, deeper purposes, to enterprises.

Case studies

Enterprises, no matter the industry, are about running a collection of business processes (procedures involving customers, employees, partners, and IT) to achieve some goal. At the end of the day, these collections of business processes need to execute (within the enterprise’s values) the enterprise’s mission, which results (hopefully) in maximizing profits for stakeholders.

Executing these diverse business processes requires various IT resources. Private clouds allow enterprises to be flexible and agile in how they support the execution of these business processes. However, because business processes are not simply about executing software on servers, a private cloud needs to provide more than virtualize resources… Consider the following two simplified use cases of applying private clouds to enterprise business processes.

Case study 1: Application development lifecycle

As an enterprise matures, so does its business processes. However, some of these processes need to also evolve to address new market shifts and new realities. One such example is how web and now mobile computing have affected how any enterprise communicates with its customers. The result is that new custom web or mobile development is a common undertaking inside modern businesses.

Enterprises that decide to keep such development in house, need to have application development lifecycle processes in place so that such development can happen in an orderly fashion while respecting the various legal and business procedures. Further, when released, an optimizing enterprise would also want to know the ROI of such efforts and how satisfied their customers are with the new offerings.

All of this requires setting up teams to develop, test, and push applications in production. After that, each application needs to be maintained and connected with existing customer satisfaction software and legal and business audit processes. Executing these steps in a repeatable manner is key. As an enterprise becomes larger, it loses some of its agility—that comes with the territory. However, large enterprises do not need to lose their innovative spirit.

The way to retain their leaderships, large enterprises need to have infrastructures that provides knobs for flexible resource planning and allocation, repeatable deployments of infrastructure, and software, consolidation of common services, centralized management, and rock solid fault-tolerant hardware substrate. A private cloud solution must provide all of these aspects to be useful to a large enterprise.

Case study 2: Data processing and analytics

With the explosion of data caused by the popularity of social media, enterprises are finding that an efficient and practical way to stay in touch with their customers is to have a social presence. The result of these interactions through Twitter, Facebook, and Google+ is an avalanche of unstructured data in the form of comments, tweets, likes, and so on.

Analyzing such social media data has huge potential for an enterprise to better understand how their customers are reacting to new products or service offerings introduction, and also advertising campaigns. Getting concrete feedback by a posting from a customer on your enterprise’s Facebook page is direct interaction that, not long ago, was reserved for special marketing surveys or studies. Instead of going to customers, they are coming to the enterprises.

The new challenge for an enterprise is how to collect this “real-time” data and make sense of it. This analysis can be outsourced, but large enterprises understand the potential goldmine hiding behind this social data and want to analyze it themselves.

Effective analysis of unstructured social media data requires analytics engines and algorithms that are resource hogs. Lots of storage must be allocated for the stream of data and the analysis must be performed constantly. The gems of insights are discovered by monitoring the results frequently and comparing them with background data and trends. All of these processes have only one thing in common, need for fault-tolerant high-grade storage and are compute intensive.

The utility of a private cloud is best measured if it is flexible enough to run such analytics workloads. Whether it is by quickly setting up large Hadoop clusters to support an analysis or by supporting the ETL (extract, transform, and load) of large data warehouses that support dashboards from customized OLAP queries.

Future of private clouds: IaaS and PaaS

The evolution of private clouds: Secure IaaS + extensible PaaS

So the point here is simply this: private clouds need to provide more than a microcosm of public clouds. In addition to the obvious potential in increased security, a private cloud must also offer an enterprise the following benefits:

1. Facilities to support repeatable execution of business processes


2. Centralized management of all resources (hardware and software)


3. Repeatable, manageable deployment of middleware with an open architecture that allows Open Source Software (OSS) components to be added to the mix


4. Facilities to manage to provenance and governance of software and middleware parts


5. Fault-tolerant, swappable, modular hardware design that can grow with the enterprise needs


6. Flexibility in defining how the hardware is divided to be able to support diverse workloads, for example, transactional, web, and analytics


7. Integrated support and service that can guarantee that your cloud is up and running 24×7 and that can help you minimize any downtime

IBM’s leadership and upcoming offerings

IBM’s leadership in private cloud with appliances such as WebSphere Cloudburst showed that our middleware was capable to support private clouds in an appliance fashion. What comes next is the complete soup to nuts, integrated solution that provides the seven characteristics that I listed and that determine a real private cloud.

Watch out, there is stormy weather ahead? Private clouds are more than you expect. Check out http://www.ibm.com/puresystems.

dr. max

Haiti Quake People Finder

Haiti Quake People Finder is a Facebook application created to socially collect, update, and share information on the survivors and victims of the devastating earthquake that hit Haiti on January 12th, 2010.

RSS feed of the survivors and victims. If you add the app to your Facebook profile then you can select survivors and victims who are friends and family and have a central place to see updates about them.





Consider donating to the relief effort via President Clinton’s Foundation Haiti Earthquake Relief.

OOPSLA 2009 - Videocast 4 - Microsoft Technology update with Dr. Erik Meijer of Microsoft

Dr. Erik Meijer is well know in language communities (and language lawyer circles) to have help create C# and LINQ (Language Integrated Query). He’s a frequent presenter at OOPSLA on these technologies as well as one of the early voices encouraging the merging of functional and object-oriented technologies and techniques.

In a brief telephone interview, Erik gives us a thorough update on what is new with C# and visual basic, the .NET platform, LINQ, and other up coming technologies at Microsoft (e.g., Project M and Azure).



Interestingly he mentions how some of of his early efforts on combining functional and object-oriented programming are surfacing to address multicore issues as well as how LINQ is finding various applications to solve parallel programming data issues and how LINQp and reactive LINQ are being used by Microsoft in their cloud platform environments: Azure and SQL Azure.

Finally, Erik finishes this video-podcast discussing what OOPSLA means to him, some of his colleagues at Microsoft, and also gives us a flavor of the strong presence of Microsoft technologies and presenters at OOPSLA this year.

OOPSLA 2009 - Videocast 3 - Cloud Analytics with Dr. Anant Jhingran of IBM

Anant Jhingran is well known for leading the technology direction of IBM’s information management division---which includes DB2, Informix, Cloudscape, and Cognos, to name a few. One of the important aspect of any information management portfolio is its analytics capabilities. Nowadays with companies collecting vast amount of data from their Web properties from end users of via social media sites, an obvious growth area for information management users and vendors is to facilitate fast and appropriate insights from this ever increasing big collection of data. Business Intelligence is not a new field, however, the opportunity to apply such technology has never been more pervasive and for so much and so varied domains and in many ways have never been more challenging...



In this videocast I caught up with Dr. Jhingran at his offices in IBM Silicon Valley Labs to discuss business analytics in a cloud computing world. That is, what is the impact of the cloud on big data? His explanation boils down to a simple equation of pain vs. gain. Dr. Jhingran also discusses how programming-oriented conferences such as OOPSLA have a lot to contribute to the challenges of analyzing big data when companies attempt to extract knowledge. The need for programming tricks and tools go beyond map reduce to scalable data analytics languages with well understood semantics and characteristics as SQL was for relational databases.

OOPSLA 2009 - Videocast 2 - Social Software with Patrick Chanezon of Google, Inc.

As we discussed in one of the previous OOPSLA 2009 blog post social software continues to impact modern lives in ways the inventors or users of the technologies are uncovering daily. By empowering everyone to be a “journalist” and a “broadcaster”, social media and associated tools are enabling everyone, everywhere, to have a voice and thus is encouraging democratic virtues to flow in all regions of the world. We are seeing only the beginning of this social media revolution.



In this videocast I caught up, at Google IO 2009 in San Francisco, with Patrick Chanezon: one of Google’s most outspoken evangelists on Social Media and Software. Patrick corroborates the importance and wide ranging impact of Social Media and Software. He also talks about Google’s effort to create an open social platform and the successes they have been able to achieve thus far. Importantly, Patrick explains why HTML, JavaScript, and Web APIs in conjunction with platforms like Youtube, Facebook, and Twitter have enabled a new style of Web programming which in turn has facilitated viral sharing of Web content to create this new social Web fabric where content flows in real time without national borders.

Patrick ends the videocast by discussing why OOPSLA matters to Google and how Googlers view deep computer science conferences such as OOPSLA.