DTGov release workflow using 'Domain Deployment'

Introduction

DTGov ships with workflows which can help you with deploying releases to runtime environments. At the moment the deployment service supports 4 ways to deploy an application:

• copy, file copy to a directory
• rhq, using the RHQ REST API to deploy an application to a resource managed by RHQ.
• as_cli, using the JBoss AS Client to deploy to a JBoss Domain.
• mvn, deploy the application to a maven repository.

In this blog post we demonstrate how to deploy an application using the JBoss AS Client.

DTGov configuration

To use the JBoss AS Client we need to configure a target in the dtgov.properties file referencing the as_cli protocol. For example to deploy to a domain on server 192.168.1.31 we'd use

governance.targets=   qa|http://www.jboss.org/overlord/deployment-status.owl#InQa|as_cli|admin::admin123!::192.168.1.31::9999

Where admin/admin123! are the credentials to log into the remote domain.

JBoss Domain Setup

The DTGov deployment service can deploy to a JBoss Domain. Look here if you need more information on how set one up.

1. Open the domain/configuration/domain.xml file for editing, and make sure you have a 'qa' group, which is name of the target mentioned above. You should end up with something like
    <server-groups>
        <server-group name="qa" profile="full"> 
2. Open the domain/configuration/host.xml file for editing, and assign servers to this group. In this example we add 'server-one' to the 'qa' domain group. You want to add more then one server.
    <servers>
        <server name="server-one" group="qa">
 
3. Now you can start your domain using something like
   bin/domain.sh -b 192.168.1.31 -bmanagement=192.168.1.31

You can test your credentials (admin/admin123!) by logging into the management console, which in this case would run on http://192.168.1.31:9990, and/or using the client: bin/jboss-cli.sh. You may have to add an admin user using the bin/add-user.sh script. Also verify that you see the 'qa' domain group in the management console.

Deployment of your application

At this point the release workflow is ready to deploy an application to the 'qa' domain group. Once you've approved the 'dev' task, you should see the dtgov deployment REST API logging on the dtgov server:

Calling POST TO: http://localhost:8080/dtgov/rest/deploy/qa/0e7a209c-f298-433c-b9a6-3d40d4ac2f5c



This means that the dtgov is invoked the jboss client on the 'qa' domain to deploy the application, where '0e7a209c-f298-433c-b9a6-3d40d4ac2f5c' is the UUID of the application artifact that will get deployed. If something is wrong at this point you can use your favorite REST client tool and perform POST's to this url until things work. 

 Figure 1. Screenshot of the management console after the requirements jar was deployed (probably a bad choice to deploy the requirements!!)

 I hope this worked for you too, and that you picked a better artifact then I did as deploying the requirements from the dtgov-demos-project doesn't really make a whole lot of sense! Well other then demonstrating that it works :).

Cheers,

---Kurt


 

SOA Governance Projects and Organizational Roles

In large organizations building and maintaining services is not a one man job, instead it is a process that touches many people in the organization. Figure 1 is a copy of figure 5.6 on page 95 of the 'SOA Governance' book by Thomas Erl et al.

Figure 1. Common associations of organizational roles with different SOA project stages.

With this many people and roles involved, how to you manage a project like this? Especially since in a large organization you will have many of these project running simultaneously. We would have loved to implement a full project workflow based on the figure 1 process, but two things are currently standing in our way: time and real world input. Who is going to pick up the challenge?

To get you started we created a 'Simplied Project Lifecycle Workflow' demo that implements just 3 boxes; Requirements gathering, Service Design and Service Implementation, with one role responsible for each stage, which are a Business Analyst, SOA Architect and SOA Developer. Each role can be fulfilled by more then one person. In the demo we assigned all three roles to the 'admin' user so we don't have to log in and out as different users all time. The demo uses the workflow shown in Figure 2. Each column represents one of the phases. The first phase being Business Analysis. The hope is this demo provides you with the building blocks to create the real world implementation we talked about earlier.

Figure 2. Simplified Project Lifecycle Workflow.

Some benefits of using this workflow are:

• Helps your organization with adoption of SOA by following proven processes. It is clear who is responsible for approval.
• Provides insight in where your projects are.
• Helps your organization work together in different teams.
• Audit features allow full history tracking.
• Released artifacts are in the repository, the artifact is automatically in escrow this way, and documentation and sources are stored along side the binaries all in one place.

Some optional benefits:

• It is possible to send a BPMN event at the end of the workflow (or from anywhere else), which can kick off a release workflow to automate deployment. Though one could also write a governance query looking for a service implementation artifact with classification #ImplPass.
• Easy integration with other systems (think bug track systems, or time management systems)

If you want to follow along with the demo you should

1. Have DTGov running; see http://jboss-overlord.blogspot.com/2013/11/bleeding-edge-governance-getting-started.html.

2. Install the Eclipse BPMN2 Modeler into your Eclipse IDE, or you can try the early access JBoss Developer Studio.

Enjoy

Customize Managed Deployments Workflow

DTGov ships with a number of BPMN2 based processes that you may want to update so they fit your business processes. In this demo we will update the 'Manage Deployments' workflow. It is recommended you go over the Managed Deployments usinf DTGov demo if you haven't already done so.

If you want to follow along with the demo you should

1. Have DTGov running; see http://jboss-overlord.blogspot.com/2013/11/bleeding-edge-governance-getting-started.html.

2. Install the Eclipse BPMN2 Modeler into your Eclipse IDE, or you can try the early access JBoss Developer Studio.

Managed Deployments using DTGov

DTGov ships with a workflow to perform deployments of released artifacts to the dev, qa, stage and prod runtime. Some benefits of using this workflow are:

• Releases artifacts are in the repository before they go to a runtime, the artifact is automatically in escrow this way, and the sources are stored along side the binaries.
• Automated deployments (using file copy, RHQ RestAPI, or JBoss Domain RestAPI).
• Email notifications.
• Sign off using Human Tasks in the workflow. Users and roles can be tailored to fit your organization. The sign off enhances the visibility of the where an artifact is in the deployment lifecycle.
• Audit features allow full history tracking of the artifacts.
• One repository for everything; so things can be found easily. This answers the "Where is my stuff" question. The XPath like query language of S-RAMP can used for searching.

If you want to follow along with the demo you should

1. Have DTGov running; see http://jboss-overlord.blogspot.com/2013/11/bleeding-edge-governance-getting-started.html.

2. Install the Eclipse BPMN2 Modeler into your Eclipse IDE, or you can try the early access JBoss Developer Studio.

Enjoy

Bleeding Edge Governance - Getting Started

We just completed an ant script that will help you get started with the latest Governance code in five easy steps and a bit of building time (your mileage may vary depending on the speed of your machine).

First download the latest script overlord.zip script (or if you prefer to use git: https://github.com/KurtStam/overlord)

Project Overlord consists of a repository (S-RAMP), Design Time Governance (DTGov) and Run Time Governance (RTGov). This script will build S-RAMP and DTGov. Both of these projects have a GUI component. All components share login information using SAML bearer tokens. The SSO code lives in a shared project called overlord-commons. Because of these SSO complexities we currently only support deploying to JBossEAP.

1. Check out the code from Governance on Github

This clones overlord-commons, s-ramp and dtgov from github. If you want to check out your own clones, then you can update the urls in the build.properties file.

ant clone

going forward you can update using ant pull.

2. Dowload JBossEAP-6.1

Navigate to http://www.jboss.org/jbossas/downloads, login and download JBoss-6.1.0.Final.zip to this directory.

3. Build

While your download is running you can kick off the build

ant install

4. Deploy to JBossEAP

When both 2 and 3 are finished you can deploy to JBossEAP by running

ant deploy

When ready start of EAP in standalone mode (jboss-eap-6.1/bin/startalone.sh) using a different shell.

5. Seed

You can now add the following section to your .m2/settings.xml

<server>
   <id>local-sramp-repo-snapshots</id>
   <username>admin</username>
   <password>your-admin-password</password>
</server>
<server>
   <id>local-sramp-repo-snapshots</id>
   <username>admin</username>
   <password>your-admin-password<</password>
</server>

Insert the admin password you supplied during step 4.

Make sure JBossEAP is fully booted up before you start the seeding

ant seed

This will upload the ontologies as well as the workflows. You can verify success using the s-ramp-ui. For now you will need to reboot JBossEAP to activate the workflows.

Good luck!

Please use the overlord user forum if you need help or have any other questions. We hope you will become part of our community and start writing your own extensions. Looking forward to seeing some good Git pull requests.

Cheers,

The Overlord team.

Other useful getting started links:

• http://www.jboss.org/overlord/
  
• https://github.com/Governance/s-ramp/wiki/GuideGettingStarted
  
• https://github.com/Governance/dtgov/wiki/GuideGettingStarted
  

Overlord 1.0.0.Final has been released

The Overlord team is proud to announce the release of version 1.0.0.Final. There are three major components within this release:

1) S-RAMP - SOA Repository Artifact Model and Protocol (based on the OASIS standard)
Downloads Documentation

• Artifact repository
  • Built on top of ModeShape
  • Stores artifact content and meta-data including:
    • Core properties (name, description, version, etc)
    • Custom ad-hoc properties (name-value)
    • Relationships between artifacts
    • Hierarchical tagging system called Classifiers
  • Rich XPath-like query language to search the repository
  • Atom based REST API
  • Automatic decomposition of artifacts based on type
    • Built in support for common S-RAMP types (XSD, WSDL, etc)
    • Extensible to support additional types (Teiid VDB, switchyard.xml, etc)
• Command Line Interface
  • Supports interactive and batch modes
  • Built-in commands to manage and search for artifacts in the repository
  • Extensible to allow contributing custom commands
• Maven integration
  • Can "mvn deploy" a deployment artifact directly into S-RAMP
  • Able to use S-RAMP artifacts as dependencies in maven projects
• Web UI
  • Based on Errai 2.4
  • Allows granular management of content in the repository
  • Can be used to manipulate all artifact meta-data

2) DTGov - Design Time Governance
Downloads Documentation

• Built on top of the S-RAMP artifact repository
• Supports triggering Governance Workflows based on repository changes
  • Embeds jBPM 6 to support workflow execution
  • Maps an S-RAMP query to a workflow definition
• Includes an out-of-the-box Governance Workflow to handle Deployment Lifecycle Management
• Implements several REST services to support Governance Workflows
  • General purpose Notification (email) Service
  • Artifact meta-data Update Service (used to add meta-data to the S-RAMP artifact being governed)
  • Deployment Service used to deploy binary artifacts into runtime EAP systems
    • Supports RHQ
    • Supports JBoss AS CLI
    • Supports direct file copying
    • Can also deploy into a Maven repository
    • Supports multiple, configurable logical deployment targets (e.g. DEV, TEST, PROD)
• Web UI
  • Includes a Task Inbox for human tasks created by the Governance Workflows
  • Provides a UI specific to the Deployment Lifecycle Governance Workflow
    • Can drill into a deployment to see its contents
    • Shows the change history for a deployment
    • Provides a way to see which deployments are deployed to which environments

3) RTGov - Run Time Governance
Downloads Documentation

• Activity Collector
  • Activity events can be reported directly to the Activity Server via REST service, OR
  • Embedded collector to intercept activities, pre-process the information to extract correlation details and business relevant properties, before batching events for distribution to the server.
    • Events can be 'validated' using pluggable event processors, which can be used to block business transactions
• Activity Server
  • Provides a REST interface to enable events to be reported
  • Persists activity information
  • Distributes reported events to subscribed components.
• Event Processor Network
  • Subscribes to the Activity Server, to receive reported activity events
  • Distributes activity events through a series of registered networks, that consist of event processors to validate, transform and retransmit events. Processors can be specified using Java, Drools CEP rules and MVEL.
• Maintaining Derived Results
  • Information derived by the Event Processor Network can be presented as a set of 'active' collections
  • Components can directly access these collections, or issue queries through a REST API.
• Web UI
  • Runtime governance information is provided through a set of gadgets.
    • Service Response Time Gadget - displays response time information per service and optionally operation.
    • Situations Gadget - table for displaying situations detected by the event processing (e.g. SLA violation)
    • Call Trace Gadget - provides a tree based visual display of the call stack associated with a business transaction
    • Service Overview - graphical representation of the services and their dependencies, showing response time and invocation details.

If you have any questions about the Overlord governance capabilities, please post to the User Forum. If you find any issues or would like to raise a feature request, then please access the Jira issue tracking system for the particular sub-component. Links to these are found on the Overlord website.

Runtime Governance release 1.0.0.M5

The Overlord team are pleased to announce the availability of version 1.0.0.M5 of the Runtime Governance (RTGov) sub-project.

This release adds the following capabilities:

• Collection of activity events from SwitchYard BPEL and BPM components
• Enable security on REST services
• Ability to enable/disable activity event collection based on a property, and switchable at runtime through JMX
• XSLT transformer for processing activity event content
• Information processor support for header values
• Event processor for sending emails
• Event processor for persisting events (e.g. situations)
• Added a new context type called Link, for correlating two activity events within a specified timeframe, where the correlation value may not be unique outside of that timeframe
• H2 database configured on installation
• Various bug fixes and minor changes - see release notes for more details.

This release is compatible with SwitchYard 1.0.0.Alpha1, and should be installed in JBoss EAP.

For more information, please see the release notes at:
https://issues.jboss.org/secure/ReleaseNote.jspa?projectId=12313521&version=12322009

Runtime Governance 1.0.0.M4 released

The Overlord team are pleased to announce the availability of version 1.0.0.M4 of the Runtime Governance (RTGov) sub-project. This sub-project was until recently called 'Business Activity Monitoring', and was renamed to reflect the wider scope of its capabilities.

This release adds the following capabilities:

• Inline (synchronous) policy enforcement. The previous version enabled policies to evaluate activity events "after the fact" to determine if a situation had occurred, and then provide some enforcement capabilities for future business transactions (e.g. to block a customer's purchase requests if their account had been suspended). This new capability enables event processors to be configured to immediately evaluate activity events and if necessary block a business transaction.

• For business transactions that have associated 'principal' information, this information will be recorded with the activity events are made available in derived results (e.g. situations and call trace).

• Location information, recorded as part of the activity events, is propagated to derived results (e.g. situations and call trace) to enable users to understand where problems have occurred.

• Runtime governance now works in a clustered JBossAS environment.

• Activity events stored in H2 in-memory db. Other databases will be supported in the next version.

• Lightweight activity collector client configuration. It is now possible to install just the activity collector mechanism within an execution environment, and have it report the activity events to another fully configured runtime governance server via REST.

• Various bug fixes and minor changes - see release notes for more details.

This release is compatible with SwitchYard 0.8.Final.

For more information, please see the release notes at: https://issues.jboss.org/secure/ReleaseNote.jspa?projectId=12313521&version=12321356

S-RAMP v1.0 Specification submitted into the OASIS Review Process

[announcement]
Today the S-RAMP v1.0 specification was submitted for review by OASIS. You can download your own copy of the two spec documents from a public link on from OASIS. Randall Hauch, Eric Wittmann and yours truly worked on both the specification as well as on an implementation of the spec at the same time.

[plug for our project]
The open source S-RAMP implementation is under developement under the umbrella of the Overlord project and ships under the Apache v2 License. We hope that it will become the S-RAMP Reference Implementation (S-RAMP-RI). You can download the S-RAMP implementation from our downloads page. The source code is hosted on Github. Our implementation uses RESTEasy for the Atom based REST APIs and on the back-end it uses ModeShape (JCR) and Infinispan (persistence) for massive performance and scalability. Well that's what the ModeShape guys tell us anyway, someone should test it out! We are currently about 75% complete and -to speak in building a bridge terminology- we are 'out of the water  now': The foundations are finished and everything we are doing now is in full view. If you are interested in contributing please let us know, there is still lots to be done.

[your chance to get famous]
One area we are really looking for help is for someone who can help us develop some best practice governance scenarios: for example project workflows, with supporting document templates for approval etc.

Cheers,

--Kurt

Release S-RAMP-0.1.1 adds Governance Workflow

I'm proud to announce release 0.1.1 of the S-RAMP project. The S-RAMP project implements the OASIS SOA Repository Artifact Model and Protocol. Two of the developers (Eric Wittmann and yours truly) are members of the Technical Committee.  The spec is nearly complete and it will be submitted to OASIS for review in a matter of one to two weeks.

On the S-RAMP project we have implemented about 75% of the spec and we've layered governance features on top of that. The objective is to have a functional set of features before implementing features we do not need right away. Things are working now, but may not always be 100% compliant (think about things like error handling etc).

Some major features we have working now

• Working S-RAMP repo - sitting on top of JCR (ModeShape by default) and using RESTEasy for the S-RAMP Atom binding.
  
• Maven integration, so developers can release artifacts into the S-RAMP repository or they can reference artifacts in their builds as dependencies.
• S-RAMP ATOM Binding, anything that can consume a atom feed can interact with the repo.
• S-RAMP GWT based browser
• S-RAMP cmdline shell, so you interact by typing, or you can script interactions
• Governance Workflow - Integration with a BPMN2 engine (jBPM by default)
• custom REST-Based BPMN2 services to deploy artifacts from S-RAMP to runtime
• customer notification service (email + templates)
• custom S-RAMP interaction service to classify artifacts as they move through their life cycle.
•  Advanced XPATH-2 query capability into the S-RAMP repository.

Some things we are working on at the moment

• Security - we need SSO across the board. We are trying to use Picketlink IDM & OAuth, sticking point is that RESTEasy 3 supports it but SOA-P 6 wants to stick with 2.3.*. Maybe we can get Bill to back port.
  
• Governance console to manage workflow configuration and to embed Task Forms (right now we direct you to the jbpm-console)
• Add Documentation - We developed a kick ass process to push Asciidoc github wiki to docbook into the regular doc stream (https://github.com/Governance/s-ramp/wiki/GuideReadme). Now all we need is more content and demos.
  
• Implement all features an end2end for some interesting Switchyard use cases.
  
• Add more out of the box best practices governance workflow.
• RHQ and JBossAS7 domain model deployment integrations.
• Support for SLA and policies in general.
• Auditing of what goes in and out of the repo.

If you are interested check out the workflow demo, download the distro and come contribute so you can be crowned and become one of Overlords.

Cheers,

--Kurt

S-RAMP-0.1.1 Governance Workflow Demo

With the recent S-RAMP-0.1.1 release we added Governance Workflow capability to the S-RAMP repository. This release can be download from http://www.jboss.org/overlord/downloads/sramp. In the demo below we show how a mvn deploy of a Switchyard application to the S-RAMP repository kicks of a Simple Release Process as shown in Figure 1.

Fig 1. Simple Release Process, click to enlarge

This BPMN2 process runs on jBPM (default BPMN Engine) and calls deployment and notification services that ship with the S-RAMP Governance application. For more details on the SOA Governance configuration see the SOA Governance Wiki.

Fig 2. Click to start the demo.

Call Trace capability available as part of BAM M3

The Overlord team are pleased to announce the third milestone of the new Business Activity Monitoring (BAM) sub-project.

The new features include:

1) Call Trace capability, which is demonstrated in the following webcast: https://vimeo.com/54642223

2) Simplified the approach for defining and enforcing policies within SwitchYard: https://vimeo.com/54790975

3) Framework for deriving correlation identifiers and properties from activity content

This release works with SwitchYard 0.7.0.Final.

If you have an interest in business activity monitoring capabilities for your project, then we would love to hear from you! Our user forum is located here, or alternatively please raise feature requests in our jira system.