Oracle, AmberPoint, and the SOA Management ecosystem

Earlier this morning, Oracle announced it had entered into an agreement to acquire AmberPoint.  AmberPoint will be integrated and managed under Oracle’s Enterprise Manager division, and over time will be ported to the EM management framework.

For both existing customers and new prospects, this is positive news on many levels (i.e. broader/deeper capabilities after integration with EM, removing any doubts or concerns over vendor financial stability, product viability, solution’s strategic fit in the enterprise, etc).  AmberPoint will fill the gaps in Oracle’s SOA management capabilities today, and should play a key role for Oracle Cloud management in the future. With Oracle’s sales and distribution channels, AmberPoint will be able to reach new markets and extend market share.

In SOA management business, AmberPoint and SOA Software were the only key niche players left.  This acquisition puts Oracle in direct competition with SOA Software.  It will end the Oracle and SOA software partnership.  Looking forward, SOA Software is likely either going to get acquired – or - has to figure out a different product and partner strategy to compete. 

In terms of acquisition, the usual suspects for SOA Software include IBM, Microsoft, CA, HP, BMC or possibly even SAP.   Any of these vendors should be able to take their solutions and integrate them as part of a broader ESM/BSM solution.  SOA Software assets could help accelerate with that.  As far as product strategy, I think SOA Software will have to break away from just “SOA management” targeting enterprise customers into broader Cloud management also targeting Cloud service providers: self-service, asset/portfolio management, Cloud operational governance (quota, policies, SLA, billing, …) configuration, provisioning, automation…  They can do some of that on their own or go to market with new partners.

In summary, Oracle’s acquisition is definitely good for customers.  It also forces the other player to consider solutions and strategies.

The future of SOA is Cloudy…

A couple of months ago, I was at an Oracle event in Redwood Shores.  The event brought together some of Oracle’s marquee customers & Fusion middleware product management team to discuss challenges/issues with regards to SOA, BPM, infrastructure management... and provide an opportunity to learn more details about FMW roadmap and offer feedback…

I seized the opportunity to talk to several customers about their SOA implementation.  Most of the customers (small –> large) had passed the initial stage of SOA readiness assessment, transition planning, and initial service portfolio development.  They had already implemented and deployed multiple enterprise services into production.  The most common issue related to SOA infrastructure management, and making sure it offers the level of resiliency and availability their customers demanded.

SOA introduces additional layers in the already multi-tier distributed applications.  First, you have the SOA management layer that handles performance management and policy enforcement.  Next, you have the Enterprise Service Bus (ESB) that abstracts service endpoints, and offers integration logic intermediation between service consumers and providers.  Finally, there may be integration adapters used to facilitate semantic and protocol integration with backend applications (i.e. SAP).  During a service request, all of these components must be available and fully functional.  Otherwise, the request fails and either the infrastructure must handle automatic management of the exception and re-routing of the service request message to maintain SLAs or the client must re-try the request upon receiving the exception.

Another area of concern is related to capacity management.  It is common practice to use high volume / peak load metrics to calculate capacity.  The result is over-provisioning of resources  (AKA server proliferation) and low utilization of assets.  In terms of IT financial management, the impact is monumental from increased hardware and software licensing costs to additional FTEs to configure & maintain the assets, and finally data center floor space, power consumption, …

So, many customers are already in their next level of SOA maturity.  They are focused more sharply on SOA operational governance.  This is where “Cloud” and SOA converge.

Before we get to that, let’s do a quick review of “SOA business value”.

SOA promises lower IT costs, reduction of IT complexity, business agility, etc… However, rarely is SOA business value measured against metrics related to the above.  The universal measurement for SOA in many organizations continues to be “service reuse”.  The more reuse, the better…Often, it is not even clear at what level of the organization reuse occurs to map and measure the value more clearly, but that’s a whole different blog…

For SOA to deliver lower IT costs, reduction of IT complexity, and agility, it is required to change the infrastructure to be more adaptive and resilient.  Unfortunately, in my experience, this is often not properly considered in SOA programs.  The vendors give you an ESB, and you’re good to run :-[

The “cloudification of SOA” involves the following capabilities:

• Dynamic Resource Management – Rather than over-provisioning to meet peak demand, a private Cloud infrastructure can enable demand-based provisioning.  This enables the enterprise to utilize IT assets more effectively and realize reduction of IT costs in terms of HW/SW/datacenter.  For some workloads, policies can be established to spill over to public Clouds like Amazon (hybrid Cloud). 
• Automation – Automation is a key component of Cloud infrastructure.  Automation is used in a variety of scenarios from automatic scaling to align with resource demands as well as automatic error/exception recovery.  This is a key enabler for service level management.
• Performance Management – Performance visibility and management across different layers of the stack is fundamental to any Cloud infrastructure.  Without it, there is no DRM or automation.  Once the organization is able to capture and correlate performance metrics across different layers and map them to a service request, they can do a better job of capacity management.  This also helps SOA with service performance management and SLA.
• Self-Service capabilities – Finally, this is an area that may not be readily consumable in all organizations, but it is a vision.  The idea is simple.  The goal is to bring the same self-service capabilities available in public Clouds to the enterprise (i.e. self-service service registration, self-service provisioning, self-service resource configuration and policy specification, etc…)   Imagine an environment where you can go to a self-service portal and request a server from a list of pre-configured images, click a button to provision a server in a few minutes rather than a few weeks that typically takes for IT request review/approval, procurement, HW installation and configuration, and delivery… Many companies are looking at this approach or have already implemented initial self-service portals to enable self-service capabilities.  In this scenario, it is possible to communicate IT agility using concrete metrics (i.e. average time to provision a server, # of demands serviced / week, etc)

Here is a conceptual diagram to illustrate the above:

So, to summarize, SOA operational governance and management poses difficult challenges in the enterprise.  The common approach to SOA does not address infrastructure issues to enable realization of SOA business values.  A Cloud Computing approach can help organizations with that.

I would be very curious how many are looking at Cloud as a progression of their SOA programs, and if they are looking at self-service capabilities.  Look forward to comments or questions.

IT Portfolio Management & Cloud Computing

Earlier today, I presented at iCMG’s Architecture World 09 about the impact of Cloud Computing on IT Portfolio Management.   I started the discussion with a quick review, and an overview of Cloud Computing.  The first set of slides came from my Cloud Computing overview and case study.  Then, I spoke with IT Portfolio Management and its relationship to other IT management practices and IT Governance.

I reviewed IT Portfolio Management in the context of shared services and Cloud Computing.  Here are some of the key points:

• Cloud Computing lowers the barriers and risks for enterprises to experiment with new ideas (i.e. rapid prototyping and experimenting).  This is a huge benefit and capability for the enterprise.
• Cloud Computing makes it much easier for enterprises to perform ongoing financial analysis (i.e. TCO).  For example, Cloud Computing billing, by itself, facilitates understanding of the costs with more precision, because its delivery model eliminates a number of the variables associated with indirect costs (which are often the main contributors to different results)…
• With Cloud Computing, it is much faster to measure benefits due to readily available analytics…

You can find the slides here.

A quick overview of Cloud Computing

Here is a presentation that I recently shared with a group of IT managers:

Cloud Computing overview and case study

View more documents from babakh.

Is it a repository or dumpster?

One of the SOA infrastructure components is the repository. It is a critical tool that enables discovery and reuse as well as playing a key role in SOA portfolio management. Unfortunately, a lot of clients don’t implement or utilize the repository properly. Often, this is due to a combination of poor planning and challenges in technology & process integration. Many customers end up using it effectively as a dumpster or shared drive. It is a fundamental issue with SOA adoption, as it retards adoption and value return or in some cases completely derails it.

During service analysis and design, service architects and interface developers use the repository to search for artifacts such as design documents, WSDLs, schemas, policies, SLAs, service contracts, etc or catalog new ones into it. An artifact may be used by different services (i.e. a logging or security policies). Or at a higher level, a common service (i.e.CreditInquiry, TaxCalculation) may be part of a composite service (i.e. PremiumCalculation). A minor change to a common artifact could adversely impact dependents and consumers.

SOA program managers extract a lot of information from the repository for service reporting (reuse, demand pipeline, impact analysis, etc). Reports provide visibility and control and enable SOA design time governance.

As demand for new artifacts are identified, the program manager or the service architect submits a registration request. The requests are often reviewed by a committee, before approved. Once the asset is implemented, reviewed and approved, publication requests are sent to the librarian for promotion.

In a typical repository, artifacts are filed under a project. Projects are filed under business units/departments. So, a repository brings an integrated view of all services and related artifacts as well as information that indicates ownership and usage.

The above is a typical highlevel SOA design approach. In this scenario, multiple roles are involved participating in different processes. There are touchpoints with design & construction tooling (i.e. development tools, testing tools, SCM, requirements management DB…) So, there is a set of requirements around how a solution is designed and delivered in the organization (process and policies). There is also a set of requirements around what integration requirements are needed (technology). The repository, but itself, doesn’t provide the solution out of the box. There needs to be requirements analysis, planning and design around implementing the repository in a way that aligns with the organization’s practices.

Here are some basic questions to consider:

• Is there an SOA design & development process defined?
• Is there an asset publication process? Has it been modeled right (i.e. quality gates and roles)? Is the process understood and followed by all parties? Do you measure and monitor the process?
• Is there an asset specification template? What is the service naming convention and classification scheme?
• What are the policies for asset registration & publication?
• What are the policies for asset change management?
• How is compliance checked/reported?
• What are the requirments for supporting the organization (i.e. MNC, offshore, etc)? What would be the right topology and deployment model for the repository? Should you buy or build? What are the tradeoffs?
• …

What is your experience with SOA repository?

Service Portfolio Management (SPM) - An essential practice in SOA Governance

Last week, I attended a briefing that a vendor had put together to preview their solution strategy for SOA Governance. There wasn't anything new or different in their presentation. Like other vendors in this category, their approach was based on the same common division of design-time vs. run-time governance, with no provision around SOA portfolio planning & management.

This denotes a narrow view of SOA Governance shared by many vendors today. Unfortunately, it is a fundamental gap that contributes to an awful lot of SOA hiccups and failures (i.e. poor capabilities around business value measurement & alignment, increased costs in SOA adoption, service delivery delays, SLA violations, planned vs. actual reuse)

In terms of SPM, the following trends can be observed in the current SOA ecosystem:

• Some vendors place SPM under "design-time" governance. This doesn't make any sense, because design-time governance focuses on the IT aspects of service lifecycle management. SPM is a practice to maximize the value of SOA investments & assets to the organization (program/financial management).
  

• Most repository vendors have positioned their products as an SPM solution. I have a couple of issues with that:
  
  • Repositories are implemented by teams with narrow view of SOA assets and limited understanding of SPM requirements. They are not designed for SOA portfolio analysis & optimization.
    
  • They are metadata repositories, primarily designed to manage architectural assets and artifacts, and targeted towards architects & developers. Business executives & SOA program managers need to capture/aggregate different kind of data and require different kind of analysis & reporting in their role.
  
  
• Finally, a lot of consulting vendors take a cursory look at SPM at best. Most do a good job of delivering an initial plan and a service portfolio, but rarely do they prescribe any guidance around continuous & proactive service portfolio management, and linkage with other practices such as APM & PPM...
  

So, what's SPM?

There is more to SOA assets than WSDLs, XSDs, design documents, etc.

There are different categories of SOA assets such as capital resources, human resources, technologies, architecture, service, service provider... and within each category, there exist different asset types (i.e. design model, ESB, ESB administrator). Investment in the assets needs to be measured & balanced against business priorities.

SPM is a set of processes and techniques that continuously measures & scores SOA investments against business goals (i.e. cost reduction, IT simplification, process agility). The intent is to maximize the return of SOA assets & investments to the organization.

As an example, it involves the following activities:

• Service portfolio inventory & planning – maintaining an inventory of services including prioritization based on business value & strategic fit, service availability, service plans, provider information, subscription costs, billing, terms of service, policies, SLAs, etc
• Financial planning & management - capturing & aggregating the necessary data to do a better job at:
  
  • Planning for SOA investment (i.e. budget, resources, vendors…)
  • Financial analysis at different levels of granularity (i.e. development lifecycle, project, program)
    
  • Estimating average cost & efforts for initial delivery of services, service change requests, etc
    
  • Modeling subscription cost, service reuse incentives, and chargeback
    
  • Usage management, billing, and reporting
    
• Service sourcing – making informed decisions about sourcing (i.e. build vs. buy, on-premise vs. SaaS)
• Asset Management – tracking all SOA assets, perform utilization analysis, TCO…
  
• etc
  

In a typical organization, SPM sits between PPM & APM as follows:

As projects are submitted and loaded into PPM, a review process takes place to assess project's SOA relevancy, and identify project's service needs. Service demands are input to SPM, and an analysis is performed against the service portfolio, service pipeline, available resources, and other related information to estimate costs, delivery date, service reuse, ROI, ... and determine the optimal way of fulfilling the demand. [n.b. In some cases, enhancements to existing services may actually trigger new projects from SPM to PPM.]

On the right, APM maintains an inventory of existing applications, costs, future plans, etc. This information is used in service realization decisions. The objective is to fully leverage existing assets. so depending on the analysis, appropriate decisions and constraints will be documented and forwarded to downstream groups (SOA architecture team). Similarly, APM may trigger SOA projects in PPM.

From a solution tooling perspective, in addition to APM & PPM, an SPM tool would integrate with an SOA repository to help with service portfolio decisions.

So, why should we care about SPM?

I touched on this briefly in a couple of posts on LinkedIn.

Majority of SOA implementations are struggling to demonstrate and manage business alignment. I think the reason for this is partly approach, partly tooling, and partly the definition & prescription for SOA Governance. I have already touched upon these issues above. So, let's consider the following example.

In a typical organization, lots of roles, perspectives, and groups are involved in SOA delivery. Each role is focused on their specific activities, uses its own repository, and applies its own methods for tracking, management, and reporting:

This makes SOA program management extremely challenging, because it is often too difficult to have a consolidated snapshot of resources, demands, activities, status, costs, timelines, dependencies...So, SOA leadership team & program managers often end up making decisions without having all the necessary information. SPM enhances visibility into SOA assets.

Most organizations are struggling with SOA business justification and value reporting. Fundamental metrics such as avg cost of service, and business value indicators such as service contribution to cost reduction are partially captured at best. SPM provides a framework for continuous measurement & monitoring of key SOA metrics.

Summary

SPM is key to maintaining SOA alignment with business, maximizing reuse, making timely investments in SOA, and effectively managing change. Without SPM, there will be continued struggle with SOA change management, business costs, organizational impacts, visibility, control, etc.

"If you can't measure it, you can't manage it" – Andy Grove

----

Update: I am also trying a survey on this topic:
http://www.surveymonkey.com/s.aspx?sm=jgIMuSkJKmdcsmCWXVyWVA_3d_3d
You can also participate anonymously. Once I collect some responses, I will share the results in a separate post.
Thanks!

Socializing service reuse

SOA repository is the technology enabler for asset reuse. During design time, project teams search the repository for scheduled and published services to determine whether any asset can be integrated and assembled into their solution without building it.

Several scenarios may happen:

1 - The service is found in the repository and can be readily reused
2 - The service is found in the repository, but the consumer has to wait until it is available
3 - The service is found in the repository, but requires change (minor -> major)
4 - The service is available, registered, but couldn't be found in the repository
5 - The service is available, but not registered so it couldn't be found
...

There are different solutions to the problems above. At the end, it is the job of the SOA CoE to socialize reuse and promote services. The rate and level of reuse is a function of organizational complexities (i.e. culture, CoE model, process) as well as development approach (prescriptive and formal unified process vs. highly interactive agile methods).

Also, as teams collaborate on service production and consumption, they engage in frequent social interactions. Repository vendors should look beyond basic collaboration support (i.e. notification, workflow) and integrate social computing capabilities in their solutions to generate excitement and overall vitality, enhance productivity & knowledge which should facilitate and contribute to more effective reuse.

Delivering reusable services...Why does it fail?

IT asset reuse can be a force multiplier in reducing costs (TCO) and enhancing operational productivity (ROI).

Assets can range from knowledge delivered in an email to models, frameworks, applications... Regardless of asset type and whether they are tangible or specified formally, critical business operations may be dependent on them.

Service reuse is one of the most important objectives of SOA, but most projects struggle with it. Service reuse is typically achieved at low rates (or levels) or not at all. In terms of producing reusable services, the issues are related more on the approach rather than technology. It should be planned early on.

Current SOA approaches and methodologies focus on service reuse during service design & specification:

(1) Projects are reviewed by SOA CoE/CoC
(2) Candidate services are identified
(3) The CoE guides in designing reusable services and specification
(4) The project team rejects the design as too complicated/over-engineered that would delay delivery & impact schedule
(5) Lots of communication goes back and forth between the CoE and the project team over the merits of the design, enterprise benefits...
(6) The Issue gets escalated to senior leadership team
(7) After a few rounds of emails, the executive decision is to park it for now and revisit in the future
(8) The project team moves forward with building something that's specific to their project

The reality is that projects have no incentives for building reusable services. They are routinely measured by how consistently they meet the schedule and budget. By the time we get to service design, it is too late and difficult to balance the efforts to develop reusable services against project costs and deadlines. Incentives for building reusable services should be seeded at project inception time.

SOA Governance Maturity Model

Successful SOA rollout is dependent on SOA Governance. Despite the relative maturity of SOA, SOA Governance remains one of the most challenging barriers and key causes of SOA failure.

The issues commonly stem from approach & planning. In terms of approach, it should begin with a vision that sets some fundamental goals and objectives based on the culture and practices of the enterprise. Next, a roadmap should be developed to guide in the direction and incremental improvement of SOA Governance capabilities. SOA Governance Maturity Model (SGMM) is an effective tool to aid in the implementation of SOA Governance.

It would be appropriate to begin the discussion with a definition of SOA Governance. I am not going to do that, since there is plenty of good references on the Web. However, I am going to jump to the following diagram to review key SOA Governance domains and touchpoints:

Project Portfolio Management -In a typical enterprise, there are multiple concurrent projects that may be both producing or consuming services. After project funding and initiation, when solution implementation begins, a review of the project is typically done to determine what new services may be harvested or what existing services may be reused.

Service Modeling & Implementation - If the project produces new services (or requires change to existing services), there is a hand-off from PPM to Service Modeling & Implementation domain. Typically, multiple quality checks and reviews are planned to make sure services are implemented according to architectural guidelines and standards.

IT Service Management – Once the services are implemented, there is a hand-off from Service Modeling & Implementation back to PPM and on to IT Service Management domain for deployment and operation…

So, as noted at a high level, there are multiple hand-offs that take place between different groups. Sometimes it may even involve different organizations (i.e. outsourcing). All of this contributes to the complexities of SOA Governance implementation, as an enterprise tries to bring about the necessary changes to operationalize the model. It is impossible to bring all of this together in a single step. It is a “process” that requires management.

Capability Maturity Model (CMM) provides an appropriate model to develop SOA Governance capabilities. An SOA Governance Maturity Model (SGMM) helps realize a vision and continually make incremental improvements with minimal disruption to the business.

SGMM:

Level1 - Initial - At this stage, there is no or minimal SOA Governance. Services are developed ad hoc without appropriate quality control reviews or conformance to a service portfolio.

Level2 - Repeatable – At this stage, there are some processes defined. Typically, the practices are implemented at project level.

Level3 - Defined – At this stage, processes are fully implemented and practiced at the enterprise level.

Level4 - Managed – At this stage, processes are monitored and metrics are captured (i.e. # of projects, # of services implemented, % of service reuse…)

Level5 - Optimizing – At this stage, metrics are analyzed to implement process improvement.

SOA Governance, Registry, Repository ... 3 years later

A lot has happened since my post on SOA Registry & Repository 3 years ago.

In 2006, there were major acqusitions such as Mercury/Systinet and later Mercury itself by HP. BEA bought Flashline and that became AquaLogic Repositroy. Now, BEA is part of Oracle.

webMethods bought Infravio and in 2007, webMethods itself was acquired by Software AG.

Progress bought Actional. SOA Software bought Blue Titan.

IBM acquired Datapower, but decided to build its own Registry & Repository and bundled Rational Asset Manager for metadata management. Now, they are delivering a new product for Web Services security policy management.

Finally, AmberPoint's go-to-market strategey seems centered on the partner channel/OEM.

Vendors continue to enahnce products with new features and integration with other products (i.e. CCMDB for automated operations & management)...

In most cases, they fall short of delivering a solution. Even after a year, customers have difficulty to opertionalize the solution!

Fundamentally, most of the conversations with customers are product centric. Implementations are tool-driven with little or no understanding of the big picture at the client's side and requirements. There is little or no guidance/enablement around governance process design, transition/roadmap, and integration with customers' software development methodology, operational model, and existing tools...

Vendors should be very proactive on this issue to maintain that "trusted relationship" with customers.

A lot of value can be delivered and demonstrated through these solutions especially by capturing SOA governance metrics (i.e. ratio of service reuse) and operational metrics such as SLA... As customers try to figure out how to cut costs (i.e. consolidation), they can rely on these tools to provide some of the key metrics to help make the right decisions.