PaaS Evaluation Framework for CIOs and Architects

An evaluation framework can help enterprise architects and solution architects evaluate and select PaaS offerings.  The evaluation framework can be used to shortlist PaaS providers, build questions posed in a Request for Proposal (RFP) document, or generate use case scenarios.  This blog post provides an overview of the PaaS Evaluation Framework published in the Selecting a Cloud Platform White paper.

At QCon International Software Development Conference 2011 and Gartner’s Application Architecture Development Integration Summit 2011, WSO2 will be distributing evaluation matrixes comparing Platform as a Service offerings.   We compiled scores of detailed requirements and attributes relevant to developers and project architects, and then we distilled the evaluation criteria into seven categories:

  • Cloud Characteristics
  • Cloud Dimensions
  • Production Ready
  • DevOps Activities and Phases
  • Cloud Architecture
  • Platform Services
  • Programming Model

Cloud Characteristics

Cloud characteristics define how the Cloud solution differs from traditional, terrestrial infrastructure.  As defined by NIST, Cloudy solutions exhibit the following characteristics:

  • On-demand self-service
  • Resource pooling
  • Rapid elasticity
  • Measured service or pay per use
PaaS offerings score high (10) when they exhibit high abstraction, fine granularity, and rapid response across the four categories.  Inflexible, static, and slow changing PaaS offerings exposing low-level infrastructure details will score low (1).

Cloud Dimensions

Before selecting PaaS infrastructure, understand how sharing, location, and responsibility impact your public/private Cloud and internal/external Cloud decision.  Public, private, or community attributes specify how widely the cloud service is shared; a sharing dimension.  Internal or external denote the consumer’s view of the Cloud’s service interface.  The view is associated with a consumer’s responsibility for service development, operations, and management; a responsibility dimension.  A third dimension, on-premise or outsourced, describes where the service assets are located; a location dimension.  A hybrid cloud strategy delivers, spans, and connects clouds across all dimension attributes.

PaaS offerings score high (10) when they can run within all dimension coordinates, and score low (1) when they only run within a subset of the dimensions.

Production Ready

The production ready criteria category measures whether

  • Production support warranties
  • Number of paying customers
  • Current service maturity
  • Service Level Agreement (SLA) quality
  • Warranted Quality of Service (QoS)

Service level management capabilities should include:

  • Resource monitoring
  • Resource management
  • Performance management
  • Traffic orchestration

PaaS offerings score high (10) when they are fully supported, mature, and fulfill comprehensive service level and quality of service requirements.  PaaS offerings score low (1) when they are pre-production releases and do not warrant service levels.

DevOps activities and Software Development Life Cycle phases

This criteria category measures support for DevOps activities across software development lifecycle phases (i.e. design, develop, test, build, deploy, manage).  Relevant sub-categories include:

  • Integration with on-premise software development life cycle tooling
  • Integration with on-premise automated service governance tooling and policy repositories
  • Supported DevOps activities (e.g. automated provisioning, self-service configuration, process automation, continuous integration, continuous deployment)
  • Automated governance including:
  1. Service catalogue and service tiers
  2. Demand and capacity management
  3. Lifecycle management
  4. Infrastructure Authority integration

PaaS offerings score high (10) when they are well integrated with preferred software development lifecycle tooling across all application life cycle phases.  PaaS offerings score low (1) when they deliver a disconnected and silo-ed design, development, deployment, and management experience.

Cloud Architecture

The Cloud Architecture criteria category measures conformance with architecture principles enabling:

Measured service or pay per use

  • Fine-grained metering
  • Billing
  • Reporting

Elastic and Scalable

  • Stateless services
  • Rapid provisioning
  • Flexible topology
  • Thirteen dwarf patterns (e.g. MapReduce)
  • Parallel processing

On-demand self-service

  • Flexible workload assignment
  • Standard service offerings
  • Quick startup and automation

Shared, virtual infrastructure and resource pooling

  • Multi-tenancy
  • Resource utilization
  • Resource pooling
  • Interoperability

PaaS offerings score high (10) when they deliver enhanced capabilities at high abstraction and fine granularity.  PaaS offerings score low (1) when they do not extend capabilities beyond levels offered by traditional application platform infrastructure.

Platform Services

Teams compose platform services to rapidly design, develop, deliver, deploy, and manage applications.  Platform services span the following broad categories:

  • Presentation services
  • Application and service container services
  • Business process and business rule services
  • Integration services
  • Complex event processing services
  • Data access and persistence services
  • Development governance
  • Application lifecycle management
  • Automated run-time governance services
  • Policy registry and repository services
  • Identity management
  • Security
  • Service level management
  • Compute, network, and storage infrastructure services

A complete, unified platform decreases integration cost, reduces solution complexity, and enables uniformly applying policy and process.  PaaS offerings score high (10) when they deliver a comprehensive platform service list and score low (1) when they offer a small subset of the platform required to effectively deliver complex applications and services.

Programming Model

A Cloud programming model facilitates building applications and services, which exhibit Cloud characteristics.  The programming model should be congruent with Cloud architecture principles and patterns.  Example detailed criteria include:

  • Actor model (i.e. message passing instead of function invocation
  • RESTful interactions
  • Consensus protocols
  • Asynchronous rather than synchronous interactions
  • Service orchestration
  • Functional programming
  • MapReduce

PaaS offerings score high (10) when the programming model explicitly supports parallel processing, distributed interactions, shared nothing architecture, workload decomposition, and hide infrastructure complexity.  PaaS offerings score low (1) when they expose data center infrastructure concepts (i.e. machines, storage parameters, network addresses) and do not elastically scale beyond traditional application server clusters.


How would you score your Platform as a Service offering?   What PaaS capabilities or attributes may I have missed?


One thought on “PaaS Evaluation Framework for CIOs and Architects

Leave a Reply