Articles

How to combine your own custom data with K2 Process information

chrisg — Tue, 10 Jun 2008 14:45:00 GMT

We have been working with a client recently and as their reporting requirements unfolded it became apparent that the OOB and custom reporting made available through blackpearl wasn’t going to cut the mustard in certain instances.

Why? Well these guys needed reporting that was going to provide them with visibility of information that resides in their own custom data store (SQL Server tables) and also process specific information that blackpearl holds. This information was to be presented via SQL Reporting Services reports and for each instance of a process (submission of a Investment Plan) they needed process information (Folio Name, Start Date, End Date, Duration, Number of times a particular activity has occurred – in this case how many times the Rework activity has occurred) combined with information from their own SQL tables (Customer name, custom Status info and other details). All of this information was available through SmartObjects. The blackpearl OOB reports use their own SmartObjects and we made use of the Dynamic SQL Service to provide the connectivity to the custom SQL database which in turn provides data to our SmartObjects.

Initially our thoughts were to provide the reporting utilizing SmartObjects. SmartObjects are indeed a fantastic component within the blackpearl platform but from a reporting perspective are relatively lightweight. They provide limited “TSQL” functionality, we were to discover that they do not allow for things like aggregation (Select * from SmartObjectA) or Sub Selects which in this case were critical in order for us to produce the information our client needed.

I have raised support tickets for these as I believe they should support this common querying capability.

So we turned our attentions to what was contained within the K2 Databases. Within the K2ServerLog database we looked at the following tables:

_Act

_ActInst

_ProcInstData

Time to take a look at the SQL: We use this piece of SQL to essentially count the number of times a particular activity has occurred in this case the Rework Activity.

SELECT

TEO.Name AS 'TEO Name'

, TEO.Edumis

, TEO.Email AS 'TEO Email'

, TEO.AdUserName AS 'TEO Username'

, TEO.TEOType AS 'TEO Type'

, TEO.XYZArea AS 'TEO Area'

, Activity.Name AS 'Activity Name'

, Instance.StartDate

, Instance.FinishDate

, DataFields2.value AS 'Number of reworks'

, Advisor.FirstName + ' ' + Advisor.LastName AS Advisor

FROM

K2ServerLog.dbo._ActInst AS Instance – K2 Activity Instance table

INNER JOIN K2ServerLog.dbo._Act AS Activity ON Instance.ActID = Activity.ID -- K2 Activity table

INNER JOIN K2ServerLog.dbo._ProcInstData AS DataFields1 ON (Instance.ProcInstID = DataFields1.ProcInstID AND (DataFields1.Name = 'TEOID'))-- Process Instance Data Fields (TEOID). So we have the

INNER JOIN K2ServerLog.dbo._ProcInstData AS DataFields2 ON (Instance.ProcInstID = DataFields2.ProcInstID) -- Process Instance Data Fields (ReworkCounter)

K2ServerLog.dbo._ProcInstData table is essentially a key value pair table per instance for its data fields, with data field name as the key value and data field value as the value data.

INNER JOIN XYZ_IIP_POC.dbo.tblTEO AS TEO ON (CAST(CAST(DataFields1.value AS nvarchar(50)) AS INT) = TEO.TEOID)

INNER JOIN XYZ_IIP_POC.dbo.tblAdvisor AS Advisor ON TEO.AdvisorID = Advisor.AdvisorID

WHERE

Activity.Name = 'Plan Rework'

The Output

This article written by Robin Doddridge and Jian Sun of www.intergen.co.nz

Actions, Outcomes and Lines Left Unruled

ken.wong — Thu, 22 May 2008 14:15:00 GMT

In an evolutionary fashion, new concepts have been added to K2 in addition to concepts being changed and refined. To the developer, it means adopting these new and changed concepts to model business processes better. With that in mind, this article addresses a point of confusion that has crept up to which I have fallen prey to. When should you use outcome rules over line rules and how do actions relate to this? The pitfall so easy to fall into occurs when modeling a multiple branching scenario from an activity. If the new K2 concepts have not been understood then where one would expect multiple branches to be followed only one is followed. This is the issue that this article addresses.

Collectively, the three concepts “actions”, “outcomes” and “lines” form the basis of abstracting lower level programming concepts and moving towards higher level concepts that an end user would understand. I am often asked to explain (to both developers and non-developers) the benefits of using a workflow modeling solution like K2 versus just coding it up directly. This article is not meant to be an exhaustive exposition on this matter but I would like to highlight one point to elucidate the “abstraction” concept. The point can be summarized as follows:

In any solution that models a business process, on one end of the spectrum there is the raw code that is written by developers. On the other end, we have business requirements written by business analysts. In the case that we are modeling business processes, the requirements are likely Visio diagrams.

Which is easier to understand (by an end user)? Which is easier to modify? It is of course the business requirements[1].

With these extremes mapped, I would say that K2 sits on the spectrum somewhere close to the business requirements (Visio diagrams) with the opportunity to drop down to the code level if needed; something to please the business analyst and something to please the developer. This is made possible because the concept of a “step” in a workflow has been brought up from the code level to a tangible concept understandable to the computer and something that can be modeled by a business analyst. This is made possible because of abstraction.

Those of you who have worked with K2.net 2003 know that lines and more specifically line rules are not new. They are the logic that represents many lines of code that directs the flow of execution from one step (activity) to another step. In the same tradition of abstraction, there are two new concepts that have been teased out of the code. They are Actions and Outcomes.

Actions

From the workflow, the steps that involves human interaction requires that the human do something or “action” that step. Actions are the options, typically surfaced in a form like “Approve” or “Reject”, that a human can act on a step he/she is assigned that will advance the workflow. . In K2 parlance, actions are defined on each activity that has a client event. This seemingly simple abstraction allows for the following:

· Actions are surfaced up to the K2 workspace which allows a user to action a step right from the workspace, if so desired.

· Actions can be secured. Each action can be assigned a specific access rights from the K2 workspace UI. This allows certain actions to be assigned to some people and restricted from others.

· Actions need not be hardcoded into the “actioning” form. This means that the list of possible actions can be retrieved from the system rather embedding it in form. The advantage of this approach is that the actions in the workflow can change without necessarily requiring the forms to be updated.

Outcomes

Outcomes are a natural extension of actions. Outcomes are formal definitions of a K2 activity that defines the end result of an activity once it has been “actioned”. In many cases, outcomes are one-to-one to actions but they need not be and it is when it is not that things can become confusing. The interesting point is that by design there can be only one outcome followed for an activity. In the one-to-one case, because the user selects one and only one action, it will correspond to one and only one outcome.

In the case that it is not one-to-one, consider the example of a K2 activity:

Actions defined:

· Complete

· Incomplete

Outcomes are not one-to-one and defined as:

1. Complete1: Action = Complete

2. Complete2: (Action = Complete) And (ProductType = Widget)

3. Complete3: (Action = Complete) And (ProductType = Doodad)

For each outcome above there is a line that branches the workflow; lines have no rules so that workflow can branch off in three different directions when the user actions “Complete”.

Because there can be only one outcome, only the outcome “Complete1” will activate. The other two will not be evaluated because an outcome has been found to be “true”.

The first line will be followed, the other two will not.

Outcome definitions must be defined such that only one is evaluated to true. They must be mutually exclusive. The extra clauses (ProductType = Widget, ProductType = Doodad) that I have above that make the outcomes not mutually exclusive should not be defined here. These rules are what control the flow of the process out from the activity. As such they should be defined in the lines coming out of the activity.

To fix the above scenario, there really are only two outcomes, Complete and Incomplete, because this is the set that is mutually exclusive. In this case, because there is no branching with the “Incomplete” outcome it can be removed. The extra clauses that control the branching need to be defined as line rules.

For the branch “Complete1”:

The “Outcome” green variable is a new activity level field that is accessed from the Context Browser:

For the branch “Complete2”:

And for the branch “Complete3”:

When the process is changed so that lines have the rules that control branching then the “expected” behaviour follows.

Summary

This article addresses the issue of when to use outcome rules and when to you use line rules. When this is understood, you will not fall prey to creating a multiple branching scenario that does not seem to follow the defined multiple branches, despite having defined the branching logic.

To summarize the rule of thumb, outcome definitions must be mutually exclusive as there can only be one outcome. Any rules that define the branching from the activity are meant to be defined on the lines and not part of the outcome definition.

[1] Some would argue that because code is exact and prose/diagrams generated by a business analyst is not necessarily clear then code is easier to understand. On the point of ambiguity, there is no questioning that code is exact; that “while (1) malloc (1);” is memory leak and that the business requirement “I want something that can solve my problem well” is ambiguous and open to interpretation. From an end user perspective, however, an end user can be brought up to speed quite fast on a well crafted Visio diagram and would not likely understand code.

Talkin SmartObjects. by Jason Apergis

chrisg — Thu, 24 Apr 2008 17:15:00 GMT

Background

Colin Murphy, Gabriel Malherbe and I held a workshop on SmartObjects at the K2 Insider Conference this April. It was an interesting discussion because the three of us had some conference calls beforehand where tried to gain a better understanding of how to position SmartObjects in the Enterprise Architecture. I wrote a blog that has not been published which all three of us iterated on before the conference. I made some statements in that article (to invoke debate) where I asserted that SmartObjects could be considered SOA. I got beat up a little by colleagues and now admit that SmartObjects are not SOA even though they adhere to many of the tenants of SOA like autonomy, manageability, discoverable, maintainable, exception handling, scalability, reliability, transactionality, security, logging, build ability, interoperability, testability, etc. Maybe it is low on the maturity scale of SOA (level 2 of 5) and I still debate that sometimes that is good enough for many organizations. It can be challenging to get a company to invest to get to a high maturity level of SOA.

So then what are SmartObjects? Here are some excerpts from the slides that we presented.

Before and Now

In the K2 2003 days we would have to go into the client event after the code was generated and start modifying it to get data from external places. Remember that there is no code generation in blackpearl; it is declarative.

Now all we do is drag and drop smart object references into the canvas to gain access to data; no code.

K2 [blackpearl] Solution

· Blackpearl introduces SmartObjects to solve many of these issues.

· SmartObjects:

o Are reusable business entities which can be deployed centrally and consumed by non-technical workflow authors

o Are a storage location for process data (SmartBox)

o Are a means to quickly access external LOB Data

o Allow developers to aggregate data from multiple backend systems into a single, composite object

o Can be created without writing any code

o Can be accessed outside of the workflow (ADO Provider) and can be reported on

K2 SmartObject in Enterprise Architecture

The following diagram is from a presentation I prepared in November of 2007 for the Texas User Group. At the time I was digging around SmartObjects trying to understand it. From what it looks like my diagram is still spot on.

K2 states that SmartObjects are used to “provision” data into your workflows. K2 uses it as the underpinning to the entire blackpearl platform. The two things you should know right off are there are SmartObject Services and SmartObjects.

SmartObject Services are:

· The providers that access line of business and expose the data through a common interface.

· They expose methods that can be executed; typically CRUD operations.

· They are registered with the K2 Server and instances of them are created through K2 Workspace.

SmartObjects are:

· Class definitions that have data elements that map into the methods that are provided by the SmartObject Service.

· SmartObjects can be graphically used within K2 to access data.

· SmartObjects have a data provider API which you can easily hook into other tiers of the architecture.

In the end SmartObjects is part of your Data Access layer and they allow you to basically write data providers which are access through SmartObjects.

K2 [blackpearl] Solution

The following are out of the box SmartObject Services that are provided by SharePoint.

· SmartBox (cannot be used for custom SQL Databases)

· SharePoint

· Active Directory – I cannot live without this service when building workflows.

· K2 [blackpearl] – Provides easy access to all data that is within K2.

· Salesforce.com

Blackmarket SmartObject Services

The Blackmarket is a K2’s version of CodePlex. It is an area where developers can share code with one another - http://k2underground.com/k2/ProjectLanding.aspx

· Dynamic SQL Stored Procedure Service – Build up Service based on stored procedures in SQL Server. I plan to start using this one because I prefer to be calling stored procedures versus accessing tables and columns directly.

· Dynamic SmartObject Services Service – Build up Service based on schema of a SQL Server instance.

· SharePoint Users in Groups ServiceObject - This ServiceObject when wrapped in a simple SmartObject allows you to leverage SharePoint Groups.

· Office Communication Server Service - The OCS Service Broker allows users to create a SmartObject to send notifications via Microsoft Office Communications Server 2007.

· Dynamic WebService Service– Build up Service based on a web service.

· SmartObject Service Provider for Oracle – Build up Service based on schema of an Oracle instance.

· Dynamic SmartObject Service – Dynamically builds up services that use existing SmartObjects as the data source. The author says this allows you to create SmartObject Service inheritance.

This catalog of services is going grow as more people start to contribute.

Conclusions

The following are my conclusions after the discussion.

· SmartObjects give you a significant edge in delivering timely solutions with K2. They have provided me the ability to immediately access Active Directory, SQL Databases, etc. We have personally seen that we can now deliver working processes in days instead of weeks because we no longer have to build up custom data access layers and go into the code of K2 and manually access the data. I can use the SmartObjects in a drag and drop fashion in line rules, succeeding rules, preceding rules, emails, destination users/rules, any configuration item in any event wizard, etc. It is ridiculously simple to configure my entire K2 process with data and I have zero custom code to get at the data. Plus I can write C# code in custom event handlers to use SmartObject. The API is really clean, simple and similar to using System.Data.

· Another great thing about SmartObjects is that if you carefully take the time to build them you will start being able to build catalogs of SmartObjects that business users can use when composing their K2 Processes. They will not be exposed to any of the details associated to knowing where the data comes from.

· Right now one concern is the ability to place business rules against the SmartObjects methods. For instance if you use the Dynamic SQL Service which build the interface of the service based on the schema of the database there is no way put business rules on the SmartObject to ensure that a value must be greater than X or cannot be null. So if a business user were to use SmartObject Event in their process they could possibly just start setting invalid data.

· The SmartBox is good for doing early iterations of processes but over the long term you may not want to us them. For instance you have limited control over the schema of the database (because it is generated). As well, you may not want to co-mingle data from different processes in the same database.

· Do not let you SmartObjects replace your Data Warehouse. With SmartObjects it is possible to perform CRUD operations across line of business data sources. It is very tempting to have a SmartObject read from many different places providing a single view of the data despite where the data is being read from. A simple example would have a SmartObject read from Active Directory and SQL Server. You must take performance into consideration when doing something similar to this. Performance issues associated to reading data will be vetted out in the SmartObject Service implementation however if you read in two large data sources and then join them together in the SmartObject layer you will not have the opportunity to index those joins. It would be better to have the SmartObject access data in the Data Warehouse itself.

· This comment will be a little at odds with the first but LINQ, Entity and .Net Data Services make it challenging to position SmartObjects. SmartObjects Services may use them to access data and surface it up through a SmartObject for K2. However many developers will be tempted just use LINQ directly inside your custom server events, middle and UI layers. I have not personally messed with LINQ and Entity as much as I should but from the hype it is pretty rich and provide a full Object Query Language and Object Relational Mapping (ORM). The one nice thing about SmartObjects is that they provide you a way to centralize and publish services.

Getting into BizTalk

chrisg — Thu, 13 Sep 2007 04:40:00 GMT

INTRO

This article will attempt to act as an overview of common message broking architectures, the BizTalk architecture and application, BizTalk components and BizTalk capabilities as a message orchestrator and potential application as a human to human and human to system workflow engine.

The message broker patterns commonly used for system integration are primarily concerned about the transport of data from one system to another.

Figure 1: Message Broker Pattern

The pattern has 4 main components.

Transport Layer: As the source or destination of the data could be anything from a file to a mobile text message, it is the job of the transport layer to pick the data up from any supported proprietary location. The transport layer is not concerned about the structure of the data. It only knows how to speak with the location and the format of the data.

Transformation layer: Once the data’s been picked up, the transport layer will need to understand the structure of the data. Most message brokers will use this layer to transform the data into a common format for internal use. Let’s call this our message

Mapping layer: Sometimes combined with the transport layer, the data, now in a common format, can be remapped and manipulated if needed.

Filtering/Routing Layer: This is the core of the message broker. The idea is that the broker can now route the message out based on the content of the data inside the message. The message could have enough information for the routing engine to determine where it should go and what it should do. Alternatively some work may need to be done to figure this out. This could be achieved by manipulating the message, or by applying complex routing rules and decisions and conditions… from here, the process is exactly the same, but in reverse.

Using this pattern, the BizTalk architecture overview looks like this:

Figure 2: BizTalk Architecture

Want to talk?

BizTalk is primarily a technology for integrating applications. In other words, BizTalk acts as a "broker" between applications, to enable them to work together to achieve a goal, solve a problem or provide extra value. By using a broker approach to application integration you provide a common connection point for these applications and avoid the complexity of custom integration, often significantly decreasing development time. The broker services provided by BizTalk can offer message transport and transformation and increased support for diverse communication mechanisms such as HTTP, MSMQ, FTP, etc. On the Windows platform, using BizTalk in this approach is often the best way to integrate with non-Windows applications, such as AS/400 and mainframes. Various third party adapters are available for BizTalk.

Show me the message!

It’s all about the message. As a message broker, BizTalk is primarily concerned about the transport of messages. It is the main entity in any BizTalk application. By using either the supplier adapters or utilizing third party adapters, BizTalk can pick up messages from a wide variety of sources including FTP, HTTP, MSMQ, etc. The receive pipeline (BizTalk’s implementation of the transformation layer) then converts the message into a known structure (XML as preference). Also here a wide variety of industry standards are supported including flat files, EDI documents, XML, etc. The configuration of ports and the mapping of data in the various pipelines and the optional orchestration a business processes can be done code-free. Current support for different transport and message formats include SAP, Oracle, Siebel and WSS.

Other features include a reliably delivery architecture that ensures that messages don’t get lost. Using a process of dehydration, processing times of messages in BizTalk can be seconds or years with no detrimental effect on the system.

So within this, what can it provide?

Some of the core capabilities are:

· XML messaging through different open Internet standards

· Message transformation/mapping

· Message validation

· Application/system connectivity through application adapters

· Visual business-process orchestration

· Business rules repository and engine

· Support for secure and reliable messaging

Support for long-running transactions

Moving beyond just passing message from one point to another, BizTalk gives a user the ability to graphically define the logic that gets applied via orchestrations for its application integration within the filtering and routing layer. This means that a business process can be dictated inside BizTalk by means of receiving messages, and effectively routing them (or related messages) based on a business process. In BizTalk world this is called orchestration. It can also provide state management and support for longer-running transactions. Building in transformation and or orchestrations is done graphically in BizTalk, and for those trained and knowledgeable, this is a great feature and makes BizTalk powerful yet easy to use.

Is BizTalk workflow?

Define: ‘Workflow’: "The automation of a business process, in whole or part, during which documents, information or tasks are passed from one participant to another for action, according to a set of procedural rules." – WMC (Workflow Management Coalition)

Indeed messages flowing from system to system through BizTalk, with rules dictating the routing logic is workflow. However, human-to-human and human-to-system workflow is not always message centric but normally state centric. During human workflow the routing is often based on variety of events and data that we collate into a case. The progression of the workflow normally represents a state change in the case as a whole. Together with that, human workflow requires facilities we as information workers need to effectively and easily interact with a business process.

So what makes human workflow different?

If the bulk of the process is system integration related (transforming, integrating, processing, chunking) and the level of human interaction (making decisions, escalating , displaying data, entering information, tasking, tracking etc) is minimal, a message brokering platform like BizTalk can be leveraged the best bet would be a message broking platform like BizTalk. However, if the primary focuses of your processes are based on human interaction, then a tool set focused on human-to-human or human-to-system processes and workflow would be more applicable. Not doing so could increase the amount of code, time and level of knowledge required to bring these solutions to production and the level of effort required to maintain them. Building processes and workflow with a human-focused platform often makes for easier development and deployment. Some items to keep in mind when considering human workflow the features and intricacies that often become important when building and deploying them:

Task management – How are people assigned tasks based on a business process and how are those tasks managed and escalated?

Task and/or work item notification – When someone is assigned a task, how are they notified? Is the notification mechanism flexible and does it conforms with common collaborative technologies like email or instant messaging?

Work item redirect – Can tasks be easily be re-assigned and routed?

Out of office – How are tasks and work items routed and escalated when the intended actor is not available? Is the actors availability based on some sort of presence notification technology?

Escalations – When work sits on one step, or with one individual, do you want the workflow system to take action, such as send reminder e-mails or escalate the work item to a manager in line with the business process?

Ad-hoc workflow – Informal human workflow and routing used today only have their routes decided at run time. How can the system support this?

Versioning – Human workflows tend to change. How will your system support these changes dynamically and allow for the different versions of the business process to exist at the same time?

Reporting – Human workflows often require the generation of reports and SLA’s based on the business process. This could be based on the meta data of the business case or the people involved in the business process.

Because a message broker platform like BizTalk does not natively address requirements placed on a human business process, accomplishing some of the above can be difficult and cumbersome.

What have we learned and how does that help me?

BizTalk is about applications, systems, messages, etc. Each of these can be called process or workflow, but don’t be confused: This is rarely about interaction with a person, or human, or role. The routing of messages is message centric while human workflow is case centric.

For BizTalk evaluations ask:

· Are my needs primarily message based?

· Do I need to do complex transformations on messages?

· Do I need application integration?

· Are humans going to be involved in this process and what human workflow activities will be required?

Once I have this mapped out, I can start to make decisions on what I need to write (what must I control completely) and what I am going to buy. BizTalk can help greatly in platform- and system- integration scenarios. It excels when you need to connect various systems or applications, especially when they use different communication methods or protocols. If you have simple integration needs, without the need for extensive transformations, and you do not anticipate many other needs of this type, then perhaps coding them manually is the way.

By considering the above, your options range from custom, point to point integration to implementing a robust, industry standard approach with added benefits of orchestration like BizTalk Server. For the automation of business processes that require human task driven execution, consider a human workflow platform like the K2. If a combination of the above is required make sure your human workflow engine and your system integration tool will happily speak to each other.

Where to now?

BizTalk is an information broker for disparate systems. It does not provide data, it does not service data, and it does not expose data. It moves data. So if you need a tool that can easily handle moving data from one format or system to another with built-in capabilities for atomic transactions, and planned roll backs, and the ability to orchestrate a message and system centric process around this, then BizTalk is a sound option.

War stories…

One customer I talked to had custom .NET business objects/entities (like a customer object or a price change object) that they had developed. They also had configured these to be serializable (so you could use them in a Web service where they're represented as XML). The benefit to this is that, in their BizTalk maps, the customer could map the data to the XML schema for the object in question and then just write a single line or two of .NET code to “object.deserialize(xml).” Then they have a fully populated .NET object without doing much at all. This makes creating your Web services simpler, because instead of having to pass in the individual parts, you could just pass in your business object whose parts had been populated for you. This is a good example of a customer that LOVES BizTalk.

This customer says, “For us, BizTalk is a one-stop shop for moving data from one system or format to another with full system-side, non-human-interactive process development flow.” For more human interactive workflow, the purpose of the tools is to better expose and use data interactively — be it decision criteria, expense report data — and have humans interact with it, use it, change it or reply to it.

I would like to take this opportunity to thank all those who helped contribute to this most specifically Gabriel Malherbe, Jeremy Ragan, Bob Maggio, and ChrisT