Part 1: This post summarises the announcements and vision for integration presented by Microsoft at the Integrate 2016 conference in London last week.
For starters, I'd like to thank Mexia for sending me to Integrate 2016 in London last week, the single largest Microsoft integration focused event of the year. With over 26 speakers from both the Microsoft product team and the MVP community, it is no wonder it attracted 380 attendees from 22 countries. While I very nearly claimed the title of "farthest traveller" with 10,579 miles and 26 consecutive hours, in the end I had to defer to our New Zealand colleague Wagner Silveira who endured one of the longest flight routes in the world with 12,230 miles at 31 hours!
Why did so many go the distance to get there? Because it was a unique opportunity not only to see and hear some enlightening and inspiring presentations, but also take advantage of actually meeting and conversing with the top minds in the industry. Within the tightly packed agenda (27 presentations in 3 days!), there were still enough breaks built in that allowed adhoc communication and networking - not only with our colleagues from around the world but also with the product team. I must say I was impressed with the degree that Microsoft made itself open and available; even though the 2nd & 3rd days were exclusively presentations from the MVPs, the Microsoft leaders stayed throughout the entire conference. I was thrilled to converse with leaders like Clemens Vasters, Jon Fancey, Kevin Lam, Jeff Hollan, Chris Anderson, Vladimir Vinogradsky, Dan Rosanova and others about real-world issues we have servicing our clients. Even better was seeing them take the time to listen and understand, and then hear directly from the horse's mouth about how they plan to address these points. Perhaps this was the greatest value of the conference, a justification for the enormous travel effort & expense compared to simply availing of the live stream offered to remote attendees.
This opening day was dominated by presentations from the Microsoft product team and (as expected) delivered all of the major announcements, some of the key ones being:
- Commitment to some exciting new features in the upcoming CTP2 release of BizTalk Server 2016 including a Logic App adapter, a REST API for the management database, and a repaint of the user interface (admin console)
- Addition of a BizTalk Connector for Logic Apps to enable seamless communication between on-prem and cloud-based workflows
- Details of the new Enterprise Integration Pack, a collection of API connectors to support advanced BizTalk functionality such as EDI/B2B, mapping, schema validation, etc
- Addition of a VETER pipeline template for Logic apps, similar to a Microsoft Azure BizTalk Services (MABS) bridge
- Establishment of Integration Accounts to store schemas, maps, certificates and eventually agreements, porting existing capabilities from MABS into Logic Apps
- Creation of a Logic Apps designer in Visual Studio which mimics the browser-based version currently available
While there was no firm date commitment, we were told that we should see these features become available for preview in the middle of the year.
Principal Group Program Manager Jim Harrer delivered the keynote address where he affirmed Microsoft's understanding that hybrid cloud solutions are the answer to enterprise organisations' integration problems (as opposed to suggesting a wholly cloud-based solution). He also emphasized Microsoft's commitment to establishing BizTalk Server and Logic Apps as the core for integration capability, whilst the "secret sauce" is our ability to leverage all of the vast array of other services presented in Azure. This is what differentiates Microsoft from its competitors.
(Photo courtesy of BizTalk360)
Jim also introduced the two major team leaders:
- Sanjiv Gupta as Development Manager of the BizTalk Server team
- Charles Lamanna as leader of the Logic Apps team
After all of the mixed messages we've been hearing from Microsoft throughout the previous years, it is comforting to finally see a firm commitment to a roadmap that places BizTalk Server and Logic Apps at the centre.
What's New in BizTalk Server 2016
Principal Program Manager Jon Fancey kicked off the first session by explaining in detail what new features have already been delivered in BizTalk Server 2016 CTP1 and what is planned for CTP2 (to be released in the middle of the year). Whilst we're already excited about the support for SQL Server "Always On" and the High Availability implications for IaaS BizTalk installations in Azure, the proposed introduction of a Logic Apps Adapter really peaks our interest. Logic Apps have always enabled an easy configuration-based way of connecting to various SaaS systems such as Salesforce and others. This new adapter will enable on-prem BizTalk Server applications to leverage Logic Apps and these connectors in a seamless fashion, underpinning the concept that "BizTalk and Logic Apps are better together".
Jon also demoed the ability for Logic Apps to call endpoints exposed in a BizTalk Server application using the new BizTalk Connectors. A REST API for the management database enhances the ability to enumerate BizTalk endpoints within a Logic App flow, providing the capability of new messaging patterns not easily achievable today (e.g. scheduling BizTalk processes to run via a Logic App). We also got a sneak preview of the new modernised "repaint" of the admin console, and assurance that a number of pain points would be addressed, as well as adding features like artefact name search, multiple-host settings configuration, and saving multiple suspended messages to the file system.
Powerful Integration & Workflow Automation
Microsoft Principal Program Manager Kevin Lam and Jeff Hollan presented the next session, bringing a slightly upscale version of "Logic Apps Live" to Integration 2016. Here they discussed the various features that enable integration in the cloud, including an impressive list of OOTB managed connectors to SaaS and LOB applications, custom APIs through API Apps, custom code via Azure Functions, and componentization through nested workflows. They also discussed the architecture for Logic Apps, various ways to connect to other services (HTTP + Swagger, Direct HTTP REST, or HTTP Webhook), and different ways to trigger a Logic App (recurring schedule, polling an API, HTTP POST request).
Some of the key new features we can look forward to are a Visual Studio designer, enhanced tracking capabilities (including support for both correlation tokens and custom tracked business properties), and support for reporting through Event Hubs and Operational Management Suite (OMS). I was also excited to hear that they are working on support for a fixed outbound IP address - as this is a key requirement for some of our clients!
Long term Microsoft MVP Howard Edidin also made a brief appearance to discuss his work on a real-world healthcare integration solution.
Enterprise Functionality Roadmap (EDI/B2B)
Next up were Principal Program Managers Jon Fancey and Kevin Lam again who went into further detail about the enterprise messaging features that will be added to Logic Apps by way of the new Integration Accounts and the VETER pipeline. Integration Accounts will provide a container for storing schemas, maps, trading partners, certificates and eventually agreements in support of B2B/EDI applications. They will be extensible, secure, scalable and easy to manage by way of Azure Resource Manager (ARM), a REST API, PowerShell, the Azure Portal, and Visual Studio. Working with the Integration Account will be a new VETER pipeline for performing Validation, Extraction, Transformation, Enrichment and Routing - replacing existing functionality in MABS. Jon used this pipeline to demo an app that:
- Received a message by HTTP
- Performed XML schema validation
- Transformed into a new message
- Performed conditional evaluation based on a node value (using XPath)
- Routed to an appropriate Azure Service Bus topic
The initial preview of the Enterprise Integration Pack (coming soon) will include support for EDI X12 and AS2. Subsequent releases will include support for EDIFACT, configurable tracking capability, managing trading partner agreements in the portal, and schema & party resolution. From the enterprise connectivity viewpoint, the initial offering will support connectors to SQL Server, Oracle and DB2, with planned support for MQ, Informix, SAP, Oracle EBS, File and CICS.
Integrations Made Simple
Principal Group Engineering Manager Charles Lamanna walked us through Microsoft Flow, explaining how it supports Microsoft's vision of "Citizen Integrators" by automating simple processes. With virtually no developer skills, business users can create workflows that synchronise files, collect data, automate approvals, and send notifications. Flow is built on Logic Apps, but provides a much simpler interface as well as numerous pre-defined templates for common integration scenarios. It is in fact so simple, that during Charles' presentation I created my own Flow for collecting #Integration2016 tweets and appending them to a CSV file in Dropbox. It took me about 3 minutes, mostly due to slow internet speeds from the saturated wifi at ExCel London.
So is it a good idea to provide non-devs the power to create integrations that could potentially spam email inboxes, flood Yammer with posts, or fill up file shares? That is a contentious point, but Charles provided some direction by pointing out that Flow is an appropriate tool for scenarios where it is okay to develop in production (e.g. when working with Excel, Access & SharePoint). More complicated integrations that would normally be developed in Visual Studio and Azure should probably use Logic Apps and BizTalk Server. Since Flow will always be a subset of Logic Apps, Flow integrations that grow more critical or complex over time can always be easily migrated to Logic Apps. Charles also pointed out that future capability will include the provision for organisations to disallow specific templates and connectors - thereby preventing careless users from flooding the Exchange server!
Charles also briefly mentioned how Power Apps can help create mobile-optimised applications by triggering workflows based on user actions. Using the same connectors as Flow and Logic Apps, Power Apps enable you to "Act" on events while Flow will help to "Automate" data flows.
Harness the Power of APIs
Principal PM Manager Vladimir Vinogradsky gave us an excellent overview of API Management, reminding us how it provides a facade to decouple, modernise and optimise exposure to existing APIs both within and outside of the organisation. In addition, it provides a means to secure access to functionality by way of policies, and monitor usage for both health tracking and monetization purposes.
Vladimir discussed policies in detail, explaining how they implement API Management capabilities such as mutating the context of the request message. Policies can be applied at multiple levels (global, API, product or operation scope) and in both inbound and outbound directions. Even better, expressions made up of C# snippets can be used to dynamically configure policies. A new announcement was the integration of Azure Functions into API Management, demonstrated by Vladimir routing a new operation on an existing API to an Azure Function.
Microsoft Integration Partner Program
The final session of the day was presented by Jim Harrer and Jeff Hollan, discussing the Microsoft Partner Program and reviewing the benefits to both customers and partners. Some questions were raised about the current certification process and its effectiveness, as well as the idea of having a partner for Logic Apps. To solicit feedback from the attendees, Jeff created an application which accepted SMS text message responses to a questionnaire, using Event Hubs, Stream Analytics and Power BI to present the results in real time to the audience. This interactive presentation was not only revealing but also a fun way to end the day,
The second day featured a mixture of presentations by the Microsoft product group and various MVPs. The Microsoft presentations are discussed here.
Leveraging Azure Functions & Azure App Service for Integration
Chris Anderson, PM for Azure App Service, started with a brief introduction of App Service, explaining how everything deployed to this fully managed enterprise grade platform is actually a Web App. Even Mobile Apps are simply Web Apps optimised for mobile and taking advantage of Notification Hubs for pushing information at a global scale. He then performed at brief demo of deploying a new application, before launching into his discussion of Azure Functions.
Introduced at Build 2016, Azure Functions provide the ability to run arbitrary code in response to events (hence the logo). Code can be written in C# or Node.js (support for other languages to follow) and you are able to configure triggers, inputs, and outputs by selecting available sources and targets (no messy connection strings!) There are several templates available to get you started, so you only need to write the code specific to your function. Perhaps best of all, code can run in "serverless" mode - so you only pay for when you use it.
I particularly liked Chris's discussion on three integration patterns and the specific Azure tools that provide the capability:
- Union (A + B): API Management (two operations implemented differently)
- Transform (A = A`): Azure Function
- Flow (A1 --> A2): Logic App
A Powerful, Scalable Service Bus in the Cloud
Dan Rosanova, Principal Program Manager, gave the first of two talks on Azure Service Bus. Dan's talk was more conceptual, accentuating both the good and bad aspects of BizTalk Server and highlighting some of the things we often get wrong with messaging (e.g. centralised schemas, mapping at the port level, and doing too much work in the middleware layer to compensate for bad LOB design). While he certainly reinforced his love of the BizTalk product (I doubt his wife will ever forgive him for superimposing the logo on his wedding photo!), he explained how much more he liked the versatility, scalability and affordability that Azure offers with its palette of 70+ services.
(Photo courtesy of Microsoft)
After showing some truly impressive numbers that reflect the massive scale on which Service Bus operates (>100 billion ingress events each day!), Dan cautioned against the expectation that any single product can do everything for you. He quoted the CAP Theorem which states the impossibility for a distributed system to simultaneously guarantee consistency, availability and partition tolerance all at once. This is where the true power of Azure comes into force, by offering the ability to leverage so much different services and combine them to build a super application.
Dan also announced availability of two items:
- Service Bus Premium Messaging - offers predictable performance by providing single tenant MUs which can process ~300 messages per second.
- Azure Event Hubs Dedicated Capacity - also a single tenant model for large scale customers
Service Bus - Roadmap, What's Next?
Whilst Dan's talk was predominately at the theoretical level, Azure Message Architect Clemens Vaster's presentation was quite deep and technical, discussing the particulars of the Advanced Message Queuing Protocol (AMQP) 1.0 and its architecture of containers, nodes and links. A powerful open standard, AMQP is becoming the protocol of choice for Service Bus and Event Hubs messaging, broadening the spectrum of interoperability and support through its efficiency, reliability, flexibility and broker model independence.
Clemens discussed in detail the differences between the multi-tenant "Standard" Service Bus offering and the single-tenant "Premium" offering, including the underlying storage engines (SQL Database vs."Jetstream" Azure Event Hubs). He also showed a few slides that presented the current offerings in the Microsoft Platform Broker space:
On Premises Infrastructure and IaaS
Service Bus for Windows Server 1.1
Windows Azure Pack
Public Cloud (Standard)
Azure Service Bus
Public Cloud (Premium)
Azure Service Bus Premium Messaging
There were many more presentations by a number of Microsoft MVPs, but these i will cover in a separate blog post.
In summary, it was an awesome event with lots of great messages coming from the product team about the future of integration at Microsoft!
(Photo by Tariq Sheikh, courtesy of BizTalk360)