Node.js meet IBM PureApplication System – Part 3 of 3

This is a re-post from the on on the Expert integrated systems blog.  Please go there for discussions and feedback.

In this final of three posts about our Node.js plug-in for IBM PureApplication System I will cover testing and debugging plug-ins and deployments.  Additionally, I briefly discuss some advanced features that could be useful moving forward and point to some references as well as listing some tips and tricks using IBM PureApplication System and the Node.js platform.

Recap: Part 1 and 2

In part 1 of this series I discussed the plug-in model for IBM PureApplication System along with details on how to create one.  Since the series is about demonstrating how open this platform is, I also picked and discussed the hot web application framework Node.js.  After a short background, I discussed how to design a plug-in for Node.js that could support simple patterns deploying Node.js applications from a Git repository such as Github.com.

The subsequent post (part 2) covered how to create the plug-in designed in part 1.  Some of the details covered in part 2 involved deep diving into how to create, build and install a new plug-in.  It also included discussions of the metadata.json and config.json files as well as the scripts to configure, install and start the Node.js server.  In that same post I also showed how to install the plug-in and use it in a simple pattern on an IBM PureApplication System environment.

In this post I will cover what you can consider as pitfall avoidance and tips and tricks, as well as pointers for advanced features.  In particular I also have a brief discussion of where to go from here.

Testing and debugging plug-ins

Testing any plug-in amounts to deploying it into a running PureApplication System setup, creating a pattern with the plug-in components and using that pattern.  However, such a test cycle can be time consuming and error prone.  It is therefore recommended that you individually test the different parts of the plug-in before doing the eventual complete tests.  I give three primary approaches to testing your plug-ins and debugging common issues that may arise:

1. Build / Deploy / Instantiate / Test

In this scenario, you build your plug-in according to the Plug-in Development Kit (PDK), and deploy it using the IBM PureApplication System dashboard (see part 2 for details).  Create a plug-in, and test it by deploying an instance. While this approach should always be included in your test plan, it is usually the most error prone, and it can be long.  Any issues with the plug-in will be known at a later phase and thus requires you to restart.

2. Build / Test / Deploy /Instantiate

In this approach we move the test phase earlier—before even deploying your plug-in.  This requires you to create tests for the various parts of your applications.  Using testing frameworks like PyUnit you create unit tests for the various Python files in your plug-in and make sure that while they can be built into a plug-in, they are also passing your tests.  You might need to isolate your code from the PDK files or stub or mock any dependencies.

3. Test / Build /Deploy / Instantiate

In this final approach we move the tests even earlier.  The idea is to create your Python scripts even before packaging them into a plug-in and testing (through PyUnit, for instance).  Once your configure.py, install.py, and start.py files work fine, you can then retrofit them to follow the PDK format.  In this case you are using a set of Python scripts that you are sure are able to configure and install your component, prior to even packaging them in a plug-in.

In any of these strategies, I want to highlight two common pitfalls that occur with plug-in developers.  First, once you build your plug-in, is verifying the IBM PureApplication System dashboard correctly shows and list your plug-in.  Most added plug-ins are shown under the “Other Components” in the Virtual Application Builder tool.  There you should see an icon matching the image you used in your metadata.json.

Second, if after successful installation of your plug-in you do not see your component then it’s likely that your IBM PureApplication System does not have the pattern type for your plug-in enabled.  Please refer to part 2 of this series on the steps to follow to enable your plug-in.

As one can easily notice, the main difference between the three approaches is when test is introduced.  Since I am of the school of thought that testing early and frequently is usually a great idea and a worthwhile investment, my primary recommendation is to move your tests as early as possible.  Finally, it’s worth noting that most pitfalls and debugging approaches discussed here are generally applicable to any approach used.

Node.js tips and tricks

In this section I give a quick primer with pointers on how to get started with Node.js.  This is not intended to be a complete overview of the subject but rather a set of reference links that I have found useful as I myself got started in Node.js and in creating this three-part blog post series.

1. Installing Node.js

The Installation wiki page for Node.js on Github.com contains what is the official set of instructions for getting Node.js running on your platform.  Most of the instructions assume that you have root access to the system you are using and that you use one of the following popular operating systems: Mac OS X, Windows 7, and various flavors of Linux.

Current Node.js releases depend on Google’s V8 engine.  So, one aspect of the installation for Node.js is getting the V8 engine installed onto your machine.  This might involve building it.  Generally, this step works fine for most operating systems.  However, you need to make sure you have correct Python interpreter as well as correct C/C++ compilers and libraries.

For the Mac OS X these come with installing the latest Xcode development package, for Windows the MS Visual Studio should contain the correct dependencies, and for Linux the latest GNU C/C++ compiler and libraries should suffice.

2.Adding packages via NPM

Like most modern software frameworks, Node.js’ architecture is modularized.  That is, while the basic Node.js installation comes with full features, it also lacks various components which are then added (as needed) after the fact.  This allows your Node.js installation to be easily customized and extended.  For instance, if you want to use the language CoffeeScript (a JavaScript-compatible language) then you simply add the coffee package.

To install and manage these modules, Node.js uses the Node Package Modules (NPM).  NPM is a standalone package manager that usually needs to be installed separately from Node.js itself.  However, once installed, you can use NPM to easily add new modules (or packages) to a Node.js installation as well as updating existing packages.  Even Node.js itself can be installed and updated using NPM!  Finally, the NPM web site (https://npmjs.org/) also constitutes a repository of various OSS modules you can readily access, as of this writing more than 18,744 modules were available.

3.Troubleshooting and debugging

I have found three areas that cause issues when getting started with Node.js.  First, the installation process, while usually flawless, can be painful for some users.  Primarily this has to do with not having the correct development environment when building Node from sources and setting up the V8 JavaScript engine.  Carefully following theInstallation wiki page for the operating system you are targeting is your best solution.

Second, while installing Node.js modules through NPM is as easy as issuing the command: $npm install coffee it has a couple of pitfalls.  First, NPM allows a user to have multiple module package directories (where the modules are installed) as well as a global one.  To install modules in the global directory, you must use the -g option when installing.  Also, since the global module directory usually defaults to: /usr/local/lib/node_modules in most UNIX compatible system, accessing this directory will require root privileges.  So all installation command must be done with that access:  $sudo npm -g install coffee.  It’s also recommended that the module directory be exported from the shell where the application will be executed, this is achieved with: $export NODE_PATH=/usr/local/lib/node:/usr/local/lib/node_modules

Finally, when running Node.js applications it’s usually important to run $sudo npm -g install in the application directory.  The application’s package.json file is then used to determine the dependencies and what packages need to be updated and/or installed.   The application can then be ran using the $node command.

Plug-in advanced features

Creating or re-using an existing cloud component plug-in is the first step to creating patterns for IBM PureApplication System.  While, as we demonstrated, you can use a cloud component to create a simple pattern, anything more complicated requires other cloud components as well as linking these components together and adding quality of service (QoS or policy) features to the current components.

While a thorough discussion of any of these features (linking, QoS) would warrant its own post, I want to highlight each here as well as monitoring and scaling.  The goal is to give hints at what is possible with IBM PureApplication System and in some cases point to where more information can be found.  Also, the paramount goal is to re-iterate the fact that the IBM PureApplication System plug-in model is flexible, open, and enables modeling and deployments of complex workloads.

Adding links

The most common advanced feature for creating comprehensive patterns is to create links between components.  For instance, one might want to create a pattern for Node.js applications and relational databases such as DB2,MySQL, or Postgres or a graph database like Neo4j.  For each of these cases, you would need to create a link plug-in that specifies the Node.js plug-in as its source and the datastore plug-in as its target (this assumes that you reuse or create a plug-in for the target datastore).

Link plug-ins are created like component plug-ins, they have a similar structure, however, they use “link” as their type and have additional parameters in their metadata.json such as “source” and “target”.  Like a component plug-in, a link plug-in can also include attributes.  For instance, connecting to a database you may want to include the database name as a link plug-in attribute as well as its JDBC JNDI URL or other types of URL references.  The link plug-in uses the attributes’ values (specified by the user) to configure the source and target components correctly.  Visually, this is represented by the screenshot above showing how the WebSphere component connects to the DB2 component in the J2EE pattern that ships with the IBM PureApplication System.

 QoS policies

Another important class of advanced features that could complement the Node.js                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 plug-in created in this series of posts is to modify it to support QoS policies.  An example of using QoS policies for a cloud component is illustrated in the figure below for the WebSphere Application Server (WAS).

The QoS policies for WAS are extensive.  In the screenshot above we show one aspect which allows the user to specify various options for the Java virtual machine that WAS will run on.  For instance, using the JVM policy one can specify the initial size of the heap as well as its maximum value.

For the Node.js plug-in an obvious set of QoS policy would be around how to fine tune Node.js and the V8

JavaScript engine.  For instance, specifying the max V8 stack size or passing options directly to the V8 JavaScript engine to fine-tune its performance.

Monitoring and scaling

The final set of features that would be needed for our Node.js cloud component to address most of the needs of modern production-level workloads is monitoring and scaling.  Most cloud environments suffer from failures.  These failures, while infrequent, are inevitable.  As such, any production-level pattern needs a means to recover from some failures and therefore some level of monitoring capability to know when failures occur or are about to occur.

The IBM PureApplication System comes with monitoring features than can be added to any plug-in.  The monitoring is done in the form of a plug-in that can be required in your own plug-in.   Without going into too much detail, some of its primary features are to help monitor processes on virtual machines (VMs) and trigger actions and aggregate the results into the IBM PureApplication System dashboard.  Additionally, the IBM PureApplication System comes built with some basic monitoring features readily available from the dashboard.  Some of these features are viewing current VM status as well remotely accessing logs for any VM and component.

A feature that works side-by-side to monitoring is scaling.  One of the advantages of using a cloud environment for your workload is the ability to quickly scale (up and down) the workload to address the immediate demand.  While scaling is not an easy feature and can be tricky and be workload-specific, some general scaling strategies such as replicating services and using a load balancer service to spread the load across a pool of services is a tried and true way to scale.

Adding scaling to a plug-in is another advance topic that can be achieved by adding a QoS policy to capture the scaling requirements of the user and then pass that information to the plug-in and use it to customize your startup scripts.  One example of using a scaling policy for a cloud component is how the WAS plug-in QoS scaling policy allows users to specify scaling rules to scale up and down based on the current HTTP request volume.

Wish list

While creating a new plug-in is relatively straightforward when you have some examples, as we discussed, there exists the possibility of various pitfalls during the development.  Simplifying and streamlining the plug-in development process is needed.  In particular the following list of potential improvements could help:

1. A plug-in generator.  This is a tool that would take some basic input like name, attributes list, image, and so on, and generate the scaffolding for a working plug-in that, of course, would have scripts that do not do anything.  The point is to get the user moving fast and have a working plug-in that they can modify and iterate multiple times after.

2. A plug-in simulator that could simulate the lifecycle of a plug-in during deployment.  The goal of this simulator is to reduce the time it takes to test the deployment of plug-ins during development.  Instead of using a real IBM PureApplication System and installing and testing a plug-in, developers would use the simulator to execute these steps in a simulated fashion in seconds versus minutes.

4. A plug-in mock testing environment. Along with the simulator, we also need a mock testing environment mimicking the real environment where plug-in lives.  Ideally, this could be extended to support full pattern development with mocked VM resources.

5. A developer plug-in catalog and browser so that all plug-ins created by a developer can be collected in one place.  The various versions of each plug-in can also be displayed along with notes for each plug-in.  This type of browser would facilitate long term development of plug-ins along with their maintenance.

Since we are always looking to improve the process of using developing and using our IBM PureApplication System environment, this list of wishful items has been communicated to the research and development teams.  Future versions of the PDK and the IBM PureApplication System platform might include all or some of these features.  If you have an opinion on these or have other development tools that you would like to see then please comment on this post.

What next?

In this three part series we did a deep dive investigation on how to support the Node.js web application stack in the IBM PureApplication Systems environment.  The investigation was thorough and took us from design to implementation and test of the plug-in.  Additionally, we discussed advanced features for expanding the current plug-in to support quality of service as well as features like scaling and monitoring.

• Preparing for IBM PureApplication System: a five part series
• Create and customize virtual application patterns
• Video demo: Install and configure the Virtual Pattern Kit for Developers
• Develop patterns for IBM PureApplication System
• Should you virtualize your pattern or not?
  
• PureApplication integration with existing monitoring
• IBM PureSystems Center
• IBM Virtual Pattern Kit for Developers

I hope you have enjoyed this series and that you now have enough information to start building your own set of plug-ins as well as consider contributing them to our IBM PureSystems Marketplace.

Open 24x7x365: The IBM open PaaS and private cloud platform

This is a re-post from the on on the Expert integrated systems blog.  Please go there for discussions and feedback.

In this post, I briefly highlight the features of the IBM PureSystems offering, and particularly focus on why I claim that it is an “open” platform.  I give a high-level overview of its plug-in mechanism and point to several example extensions that are available now for PureSystems customers.

What is IBM PureSystems?

IBM PureSystems is a turnkey private cloud solution that for the first time integrates a flexible, expandable hardware platform with layers of software to manage the system itself along with the workloads that it runs.  Essentially, the IBM PureSystems offering combines an infrastructure-as-a-service (IaaS) substrate with a platform-as-a-service (PaaS) layer that also provides means to codify workload components into patterns.  With these patterns, users can effectively deploy workloads and applications in a predictable and repeatable fashion.

Additionally, IBM PureSystems embodies all of the seven characteristics, which I mentioned in a previous post (“Stormy forecast”), to make this private cloud a modern incarnation of previous private clouds.  Features, such as extensibility, built-in support, autonomic self-healing, and customization, are all part of every IBM PureSystems.

However, one additional feature of IBM PureSystems places it above and beyond typical private IaaS and PaaS solutions — it’s based on an open architecture and design. Specifically it includes:

• Open extensible hardware platform
• Open plug-in mechanism to extend its software
• Open marketplace for patterns and software you can run on IBM PureSystems

Open ecosystem and APIs

First, it’s important to distinguish between open source and open ecosystem and open APIs.  As an analogy, in the mobile world there are two dominant platforms today: the Android and iOS mobile platforms.  Arguably, they are both open, however, while Android is fully open (source code, APIs, and ecosystem-open), the iOS platform is open in its ecosystem and APIs.  Anyone can join both platforms (for a small fee in the iOS case) and different rules apply to participants of each of the platforms.

Is it valid to say that having an open ecosystem and open APIs without open source is providing an open platform?  Well, it’s a matter of semantics and business model.  Because by open, we usually mean any participants (within some rules) can contribute and can create applications for the platform; using this definition, both models (Android and iOS) are open.

Providing an open platform that is based on open ecosystem and open APIs is not an iOS novel innovation.  Microsoft Windows has done exactly this for years and before that the IBM OS 360 had pioneered this model.  Although the participants don’t have the source code of the entire platform as in other models, in this participatory open platform, the APIs are fully published and the system provides key points of extensions and contributions.

IBM’s PaaS platform falls into the category of an open ecosystem and open APIs.  It does so by having open hardware architecture and an open plug-in mechanism to create extensions.

Open hardware

First, IBM PureSystems is based on standard industry server architecture.  Although the hardware architecture is sophisticated to include fault-tolerance attributes, nothing in there is proprietary.  The various units that make up an IBM PureSystems are designed with commercial off-the-shelf (COTS) parts that can be replaced.  For instance, each unit contains an Intel Sandy Bridge CPU that anyone can purchase from Intel.

However, beyond the hardware openness, what really sets the IBM PureSystems apart from other private cloud solutions is in its open plug-in mechanism, which includes open APIs and results in an open ecosystem.

Open plug-in mechanism

IBM PureSystem workloads are defined using patterns.  Think of a pattern as the definition of a template using cloud components and linking these components.  The components on a template represent the resources (typically software) and capture the necessary details to set up and configure that component.

For instance, a component representing the IBM WebSphere Application Server is part of the J2EE pattern and can be modified when users specify an EAR file for the application they want to add in their workload.  Patterns are built visually by using the IBM Pure Application System Designer and the library of components available on the left side of the interface.

A pattern can be saved and parametrized by leaving certain aspects unspecified, for example the EAR file name and location.  A completed pattern, which is one that is valid and has all of its missing parameters filled, can be deployed on an IBM Pure Application System instance.  Because patterns can be instantiated many times and be shared and customized, this process allows the predictable and repeatable means that enterprises need to manage their workloads.

The components and links library are defined in plug-ins that can be added dynamically to the system.  The plug-in architecture is open and allows a wide range of cloud components to be defined and connected together.  Each plug-in defines the metadata of the component or link including the attributes (required and optional) and also the help data, icons, and so on.  Additionally, each plug-in follows a strict structure that allows various parts (binaries) to be defined, and custom scripts to install and configure the processes that constitute the plug-in.

One key aspect of cloud components and links defined as plug-ins is that they do not dictate the exact cloud resources that they need; instead, they suggest this information by giving hints and limits to the IBM Pure Application System deployment process.  Using this information, the system can make appropriate placements and resource provisioning decisions resulting in an optimal configuration for your workload.

Ready for PureSystems: Plug-ins available now

IBM has been working with hundreds of independent software vendors (ISVs) to create various cloud components.  By the time you read this post, many of these plug-ins will either be included as part of every IBM Pure Application System or be available in the marketplace.

We also welcome your contribution to this open marketplace by downloading the Plug-in Development Kit and looking at the examples and tutorials available online.  In a future post, I will describe in simple steps how a common popular open source framework can be added to the IBM Pure Application System platform in the few hours that is required to create a plug-in for it.

Stormy forecast: Private clouds on the horizon are growing in importance

Wrote this for the IBM Thoughts on Cloud blog and reposting here in its entirety. Please use the ToC site for discussions.

Background

Cloud computing is primarily manifest by the availability of compute capacity (primarily servers, storage, and networks) as virtualize resources.

Using a self-serving portal, available on the web, one can provision a complete unit of computation and set it up with custom or predefined software. When it is no longer needed, the resources are discarded, and you are simply charged for the time you used the resources.

The ease of use and agility that one experiences with public clouds can be replicated in a private manner for enterprises. This is especially useful to enterprises that are worried about security or that want more control over where their data and software are setup and reside. So, private clouds can be seen as a microcosm of public clouds…

Although this view is true, it is also not complete. Private clouds have other, deeper purposes, to enterprises.

Case studies

Enterprises, no matter the industry, are about running a collection of business processes (procedures involving customers, employees, partners, and IT) to achieve some goal. At the end of the day, these collections of business processes need to execute (within the enterprise’s values) the enterprise’s mission, which results (hopefully) in maximizing profits for stakeholders.

Executing these diverse business processes requires various IT resources. Private clouds allow enterprises to be flexible and agile in how they support the execution of these business processes. However, because business processes are not simply about executing software on servers, a private cloud needs to provide more than virtualize resources… Consider the following two simplified use cases of applying private clouds to enterprise business processes.

Case study 1: Application development lifecycle

As an enterprise matures, so does its business processes. However, some of these processes need to also evolve to address new market shifts and new realities. One such example is how web and now mobile computing have affected how any enterprise communicates with its customers. The result is that new custom web or mobile development is a common undertaking inside modern businesses.

Enterprises that decide to keep such development in house, need to have application development lifecycle processes in place so that such development can happen in an orderly fashion while respecting the various legal and business procedures. Further, when released, an optimizing enterprise would also want to know the ROI of such efforts and how satisfied their customers are with the new offerings.

All of this requires setting up teams to develop, test, and push applications in production. After that, each application needs to be maintained and connected with existing customer satisfaction software and legal and business audit processes. Executing these steps in a repeatable manner is key. As an enterprise becomes larger, it loses some of its agility—that comes with the territory. However, large enterprises do not need to lose their innovative spirit.

The way to retain their leaderships, large enterprises need to have infrastructures that provides knobs for flexible resource planning and allocation, repeatable deployments of infrastructure, and software, consolidation of common services, centralized management, and rock solid fault-tolerant hardware substrate. A private cloud solution must provide all of these aspects to be useful to a large enterprise.

Case study 2: Data processing and analytics

With the explosion of data caused by the popularity of social media, enterprises are finding that an efficient and practical way to stay in touch with their customers is to have a social presence. The result of these interactions through Twitter, Facebook, and Google+ is an avalanche of unstructured data in the form of comments, tweets, likes, and so on.

Analyzing such social media data has huge potential for an enterprise to better understand how their customers are reacting to new products or service offerings introduction, and also advertising campaigns. Getting concrete feedback by a posting from a customer on your enterprise’s Facebook page is direct interaction that, not long ago, was reserved for special marketing surveys or studies. Instead of going to customers, they are coming to the enterprises.

The new challenge for an enterprise is how to collect this “real-time” data and make sense of it. This analysis can be outsourced, but large enterprises understand the potential goldmine hiding behind this social data and want to analyze it themselves.

Effective analysis of unstructured social media data requires analytics engines and algorithms that are resource hogs. Lots of storage must be allocated for the stream of data and the analysis must be performed constantly. The gems of insights are discovered by monitoring the results frequently and comparing them with background data and trends. All of these processes have only one thing in common, need for fault-tolerant high-grade storage and are compute intensive.

The utility of a private cloud is best measured if it is flexible enough to run such analytics workloads. Whether it is by quickly setting up large Hadoop clusters to support an analysis or by supporting the ETL (extract, transform, and load) of large data warehouses that support dashboards from customized OLAP queries.

Future of private clouds: IaaS and PaaS

The evolution of private clouds: Secure IaaS + extensible PaaS

So the point here is simply this: private clouds need to provide more than a microcosm of public clouds. In addition to the obvious potential in increased security, a private cloud must also offer an enterprise the following benefits:

1. Facilities to support repeatable execution of business processes


2. Centralized management of all resources (hardware and software)


3. Repeatable, manageable deployment of middleware with an open architecture that allows Open Source Software (OSS) components to be added to the mix


4. Facilities to manage to provenance and governance of software and middleware parts


5. Fault-tolerant, swappable, modular hardware design that can grow with the enterprise needs


6. Flexibility in defining how the hardware is divided to be able to support diverse workloads, for example, transactional, web, and analytics


7. Integrated support and service that can guarantee that your cloud is up and running 24×7 and that can help you minimize any downtime

IBM’s leadership and upcoming offerings

IBM’s leadership in private cloud with appliances such as WebSphere Cloudburst showed that our middleware was capable to support private clouds in an appliance fashion. What comes next is the complete soup to nuts, integrated solution that provides the seven characteristics that I listed and that determine a real private cloud.

Watch out, there is stormy weather ahead? Private clouds are more than you expect. Check out http://www.ibm.com/puresystems.

dr. max

OOPSLA 2009 - Videocast 3 - Cloud Analytics with Dr. Anant Jhingran of IBM

Anant Jhingran is well known for leading the technology direction of IBM’s information management division---which includes DB2, Informix, Cloudscape, and Cognos, to name a few. One of the important aspect of any information management portfolio is its analytics capabilities. Nowadays with companies collecting vast amount of data from their Web properties from end users of via social media sites, an obvious growth area for information management users and vendors is to facilitate fast and appropriate insights from this ever increasing big collection of data. Business Intelligence is not a new field, however, the opportunity to apply such technology has never been more pervasive and for so much and so varied domains and in many ways have never been more challenging...



In this videocast I caught up with Dr. Jhingran at his offices in IBM Silicon Valley Labs to discuss business analytics in a cloud computing world. That is, what is the impact of the cloud on big data? His explanation boils down to a simple equation of pain vs. gain. Dr. Jhingran also discusses how programming-oriented conferences such as OOPSLA have a lot to contribute to the challenges of analyzing big data when companies attempt to extract knowledge. The need for programming tricks and tools go beyond map reduce to scalable data analytics languages with well understood semantics and characteristics as SQL was for relational databases.

Cloud computing - a programming perspective

This is a repost of a blog entry created for the OOPSLA 2009 official blog.

Cloud computing is the “new hot” topic. Simply put, various business pressures, a multitude of pain points, and the maturity of a series of Web technologies (networking, APIs, and standards) have made it possible and cost-effective for businesses, small and large, to completely host data- and application-centers virtually... in the cloud, if you may.

Cloud computing providers, e.g., Amazon, reuse their expertise in efficiently managing and hosting their own Web systems and applications, and expose that core expertise as a set of Web APIs. Using the Amazon Web Services Elastic Compute Cloud (EC2), anyone with a credit card and some programming can provision a server instance and install a Web application on it and thus immediately have a presence on the Web. Using economies of scale for server hardware combined with virtual machine technologies, data- and application-centers automation expertise, as well as extensive instrumentations, Amazon is able to provide that service globally for pennies at the hour. There are no binding contractual agreements and Amazon will only bill you for the hours you have used.

In addition to compute instances, Amazon also provides various other compute resources on their cloud platform, e.g., storage (file and block), message queues, batch data processing, and others. Following Amazon’s lead, various companies, including Google, IBM, and Microsoft, are also exposing frameworks, services, platforms, and applications to a world-wide audience from within a Web browser and with simple Web APIs. Cloud computing is no less than a democratization of compute resources. With cloud computing, vast compute resources no longer require huge and long-term investments but instead can be had and consumed, as Amazon chairman Jeff Bezos, like to say, “by the drink”.

Whether cloud computing will fulfill the high-expectations that many are advocating is still to be determined. Various challenges remain and, in our opinion, we are reaching the peak of the typical hype curve that new technologies follow. However, regardless of whether cloud computing will be a bust or continue to be the hit that it has certainly been so far, there is one undeniable truth that some seem to ignore... The current success of cloud computing and, we believe, its future successes, are heavily tied to how easy the cloud and cloud applications are to program as well as to maintain and to scale. And this is precisely why OOPSLA matters to cloud computing advocates, users, and providers and vice-versa.

As we mentioned, with the cloud, computing resources are cheap and widely available. In a matter of minutes, one can provision 100s of server instances on the Amazon EC2 cloud along with terabytes of storage and more aggregate MIPS than what is available on most recent mini-computers. All of this for around $10 an hour. While most anyone could afford such computing capacity at these price points, what is hard for most is to take advantage of that cheap capacity. The problem is no longer one of provisioning the resources, but rather one of taking advantage of these resources and of efficiently doing so.

We are at the beginning of a new evolution of programming. One that is taking place with this move to cloud computing. For lack of a better moniker, we call it cloud programming. It is about being able to scale programs to take advantages of these on-demand cloud compute resources. Programming distributed nodes of computation has always been one of the classic ongoing problems of computer science. The cloud, it seems, has thrusted this problem and associated corollary issues to the forefront...

While cloud programming has some resemblance to old-style distributed programming or super computing or multicore programming, it is a different problem due to the changes in the core assumptions and constraints. On the cloud, most compute resources are essentially server instances with virtual compute capacities or virtualized services. The network is the Internet and assumptions about co-locations, latency, and errors cannot be made. The same concerns one has with real servers in your data centers also still persist. That is, securing, upgrading, automating, and managing these virtual instances are still very much part of the programming that one must do to reap the benefits of new cloud infrastructures. Scripting languages, e.g., Ruby, Python, and Groovy, are already taking center stage to solve some of these issues.

Additionally, now that storage can shrink and grow on demand and for very low costs, while keeping reasonably good qualities of service, the other issue is how to manipulate the vast amount of data that one can now store. Google had a similar concern years ago as it improved its search engine while managing expenses in growing its data centers to match the unprecedented growth of the Web. Google engineers and scientists cleverly figured out how to parallelize data computation over large clusters of cheap and replaceable compute nodes. The MapReduce programming model is specifically designed to help engineer algorithms that can scale and run on the resulting big data that one now accumulates...

Programming for massive scale is the key challenge. We firmly believe that new styles of programming, new programming frameworks, and new programming languages may be one of the key sources of innovations for the cloud. Imagine when cloud frameworks and cloud programming environments provide, in near real time: the cost, the energy impact, and the automation facilities that a cloud computing infrastructure enables. Plus now imagine being able to program these multiple cloud nodes either in batch or in real-time, while satisfying best practices of Web security and privacy. The combined results would be the nirvana of Web programming. Scaling automatically your compute resources in a cost-efficient and environmentally friendly fashion while managing the resulting deluge of data and potential influx of users...

Surely there are many PhD theses to be had to help address some of the fundamental scientific and engineering issues involved in achieving such an idealized state of Web computing. In some ways we maybe vastly simplifying the issues and that many of the challenges involved have been studied in various branches of computer science and software engineering for the past 30 years. However, the point here is not to claim that cloud computing is the assured next wave of computing, we don’t know; but rather, we would like this post to simply serve as a reminder that the various issues in system, data, and distributed computing that cloud computing brings to the forefront could be addressed from innovations in frameworks, programming styles, and programming languages... OOPSLA, it seems, from its long historical track record of ground breaking innovations in this space, may be a natural choice for the genesis of some of these new future eureka moments.


Updates
06/01/09 - fixed typos: accumulate => accumulates

Why OOPSLA matters?

This is a repost of a blog entry I created for OOPSLA 2009 official blog

These days, we all take for granted that software is best built incrementally, that testing while coding leads to better quality software, that virtual machine-based languages can be as fast as natively-compiled languages, that patterns are great way to bootstrap your thinking when designing, and that an object-oriented language with single inheritance is likely easier to deal with than one with multiple inheritance...

Many of these well-accepted tenets in the software industry and programming trade have their roots in one conference. A conference that started with a band of early programmers who were passionate about a powerful new style of programming: object-oriented programming. That style has evolved over the years to become a source of innovation for all things programming and software. Indeed, most of the assertions above can be traced back to their origins in papers, workshops, or ideas stemmed from that conference: OOPSLA. Such is the legacy of this conference.


Times and technologies change. That fact has implied that every year OOPSLA had to introspect and look for ways to rejuvenate and encourage exploring boundaries of software. The inventor of Self, Dave Ungar, likes to state it simply as always "question your assumptions."

What are OOPSLA’s basic assumptions? Well, over the years it has been a conference about software languages, software development, software development methodologies, and software systems. Should this still be our focus? Software is embedded everywhere and the success of devices like the iPhone and Blackberry are good indication that at least one immediate future of software is in mobile solutions that include a combination of hardware and software within an ecosystem (private or public).

The Web has also transformed our social lives and is increasingly a communication fabric unparalleled in scope, reach, and immediacy. Web services like Twitter and Facebook have transformed the Web into a real-time virtual social square. Information is flowing quickly and at ever-increasing volumes. This social software is not only near real-time and location-aware but it is also interconnected with complex executable logic. Mashups of Web APIs and data have led to a boom of innovations analogous to early days of commerce on the Web.

The current Web not only has resulted in the democratization of information and applications, but increasingly it is the gateway to reaching every business’s data centers and application centers. Using Web APIs, a startup can run its entire operation virtually on cloud computing infrastructure without concern for acquiring sufficient compute resources to scale should that startup become the next overnight success---that is, if they are TechCruched, Digged, or Slashdotted.

With so much happening around software and the Web, why should someone from academia or industry still attend an object-oriented conference?

This is an important question. It is one that cannot be completely answered in one blog post. However, I will give you a short answer now and elaborate each point over the next three months in various blog posts and podcasts. I hope to convince you that OOPLSA matters. It matters to both academic and industrial participants. It matters because of its tutorials, its workshops, its keynotes, all of its leading-edge content. Most of all, it matters because of the world-class people who regularly present their new ideas at OOPSLA. As you will see when we announce the program, all of the hot topics mentioned above will be represented in some fashion in this year’s program...


What makes any conference really worth while is the quality of the people who attend. OOPSLA has a tradition of attracting the best and most innovative students, professors, consultants, industry researchers, and practitioners. This year will be no different. Come to OOPSLA 2009 and you are sure to meet with members of the gang of four, the instigators of the Agile movement, the creators of the Web’s hottest languages and frameworks, as well as hear from researchers and practitioners at the leading universities and companies.

Yes, "the times they are a changing”. But just as Bob Dylan will forever have a certain “je ne sais quoi” that makes his music pertinent, classic, and always filled with relevant content and meaning. So too will the OOPSLA conference. As long as we keep welcoming a core group of innovators, keep including new topics in tutorials, workshops, keynotes, and keep attracting the quality content that you will hear when we announce the program, the conference’s future is very much assured and alive.

Check back frequently for other posts as we peel away at this year’s program and demonstrate why OOPSLA matters to you.

Updates
06/01/09 - added link to OOPSLA 2009 blog entry