Part 2: This post summarises the sessions presented by various Microsoft MVPs at the Integrate 2016 conference in London two weeks ago.
In my previous blog post, I summarised the messages that the Microsoft product team conveyed at the Integrate 2016 conference held in London two weeks ago. With a full day and a half of sessions, Microsoft clarified its vision and roadmap for integration and helped to generate a positive impression across the community about the future. Not the least of these key points is that BizTalk Server is here to stay, finding a solid home at the centre of Microsoft’s integration capability along with Logic Apps.
In this post I will briefly summarise the remaining half of the event’s sessions, delivered by the most widely respected MVPs in the worldwide integration community.
Since the first day was entirely Microsoft’s presentations, we start here with the second day.
BizTalk360 & Beyond – What’s Coming in 2016?
Since BizTalk360 was the major sponsor and organiser of the Integrate 2016 event, it is not surprising that CEO & founder Saravana Kumar would deliver one of the first non-Microsoft sessions. It is even less surprising when you consider how this comprehensive solution dominates the BizTalk Server monitoring space in the industry, especially with the latest 8.1 release which sports new features such as a fully customisable dashboard, a rules composer, Service Bus integration, and more. With an image in Azure Gallery, it is now even easier to bring this ever expanding functionality to the cloud.
But what was surprising was the revelation that BizTalk360 is now branching out by offering two new products in the near future:
- BizTalk360 Lite: This SaaS edition will be hosted in the cloud and offer a cut-down version of the full product. A small agent will need to be installed on-prem. The pricing model has yet to finalised, but Saravana promised that it would include a free version. We can look forward to a beta version to be released next month.
- ServiceBus360: All the great monitoring and alert features that we’ve come to know and love in the BizTalk360 product will soon be available for supporting Azure Service Bus queues & topics (a major component for hybrid integration solutions). While most of us rely on Service Bus Explorer to manage these resources, ServiceBus360 will take that a step further by providing alert notifications based on queue status, size, message count, deadletter count, and other key aspects. The beta version anticipated at the end of the month will initially support queues only, but this will be expanded to topics shortly after.
One of the great aspects of BizTalk360 has been their aggressive release cadence as the product continually expands and improves. It is great to see it moving into the cloud space now; I expect before long we will even have support for Logic Apps!
Open Source Messaging Options
Richard Seroter took us on a journey beyond the Microsoft integration space by introducing some alternative open source options, such as Kafka, RabbitMQ and NATS. By highlighting some of the key criteria of modern integration, Richard gave us some perspective on how to evaluate integration platforms:
- Support for messaging patterns
- High availability and resilience
- Significant throughput
- Programmable and automated
- Transient endpoints
- Low barrier to entry
He then went into an in-depth look and demonstration of each of the three options mentioned above, showing how they compare to the equivalent Microsoft platform.
Azure Service Fabric: Notes from the Field
This was the only place in the entire conference where Service Fabric was even mentioned. This piqued my interest, since the hyper-scale and auto-healing capabilities of this platform make it somewhat appealing for use in integration scenarios. Codit’s CTO Sam Vanhoutte discussed the architecture of Service Fabric and how it currently supports numerous Azure services such as SQL Database, Event Hubs. Cortana, etc. Essentially a hosting platform for microservices (remember that buzzword from Integrate 2014?), a Service Fabric instance consists of multiple VMs that are self-monitoring – bringing the best bits of both IaaS and PaaS together. The microservices can be either stateful or stateless, and are modelled as either a “Service” or an “Actor”. Sam gave us some insight on when to use what:
Sam then talked through four different scenarios where Codit has been using Service Fabric to solve real-world problems, including a “Connected Building”, a social network for sports fans, a city camp reservation system, and a high density integration cloud.
Service Fabric looks interesting for high volume scenarios, and also presents a solution to problem we have with the current PaaS offerings – setting an outbound static IP address. However, there is some question about whether the VMs are managed by Microsoft or by the customer (e.g. O/S patching, etc). If it’s the latter (as the documentation suggests), that would tend to make it more of an IaaS model in my opinion.
Integration in a World of Digital Transformation
Michael Stephenson is known for always giving thought-provoking presentations around integration architecture, and as usual this session did not disappoint. Instead of talking specifically about architecture or featuring cool Minecraft demos, this time Michael discussed tips on how to make integration succeed in a culture with high expectations of digital transformation. This Gartner Hype Cycle Model graph emphasizes the importance of integration in achieving digital transformation, and how both Azure and BizTalk Server remain at the top end of productivity:
Moving on to discuss the steps to success, Michael provided numerous recommendations around:
- managing expectations in a culture that expect a lots from integration
- promoting collaboration across teams instead of creating bottlenecks by centralising integration
- leveraging the best bits of technology across a vast array of choices in order to suit the requirements and the available skillset
- learning how to govern architecture when solutions expand beyond the original scope
- effectively performing build & deployment
- using insights to promote awareness and responsiveness
BizTalk Server: Behind Enemy Lines
Johan Hedberg gave an interesting presentation about implementing BizTalk projects in the real world, which often may be in an unfamiliar environment (e.g. Java Land). In these situations it is important to “blend in” by using the agile processes and technologies that fit within the environment and culture you are working in. Johan took us through a number of different tools including Jira (for work items), BitBucket (for source control), Bamboo and Jenkins (for build & deploy), and The Elastic Stack (for tracking, search & visualisations). A key takeaway was promoting “Integration, not Isolation” by ensuring visibility, transparency, openness, and adoption of common practices and vocabulary.
A New Set of BizTalk Server Tips & Tricks
Devscope’s Sandro Pereira is possibly best known for his massive blog which has earned him the nickname “Biztalkpedia”. He is also the author of the book “BizTalk Mapping Patterns and Best Practices”, originally offered as a free eBook but now also available in print form from BizTalk360. This session revealed some of his exhaustive knowledge by discussing a number of tips and pointers including:
- Overcoming common pitfalls when migrating a BizTalk solution to newer version, for example resolving ID conflicts with custom adapters and functoids
- Dealing with map transformation incompatibilities between the old “XslTransform” and the new “XslCompiledTransform” engine
- Estimating the cost of a migration by calculating the number of projects, features, maps, and adapters involved
- Extending the BizTalk Server engine using the open-source Grabcaster Adapter
- How to install custom adapters on a 64-bit machine
- Avoiding disk full errors by cleaning up archived messages using PowerShell or other tools
- Monitoring an environment through auto-healing PowerShell scripts
Sandro finished up by giving away a few free copies of his book and some BizTalk T-Rex stickers.
How to Perform Perfect BizTalk Server Solution Review
Finishing up Day 2 was the vibrant Nino Crudele from Italy who discussed some of the key aspects involved in performing a BizTalk environment review. Referencing his white paper which can be downloaded from the BizTalk360 site, Nino first talked about his engagement tactics, working to “identify the enemy” and associated big problems whilst retaining sensitivity to the politics and stress typically associated with organisation currently undergoing integration problems. His diagrams showing the CTO/CEO Mental Test were particularly poignant:
Nino also discussed some the tools which assist in performing an assessment (including a cost calculation spreadsheet and the BizTalk Maturity Questionnaire), as well as some tools for aiding development best practices like Nos, the Grabcaster Adapter, and Sandro’s blog. In addition, he mentioned the importance of minimum documentation requirements, such as naming conventions and Handshake Flow and Interface Matrices. Finally, he compared various options for monitoring, logging, deployments, management and governance. Nino’s passionate and humorous presentation style provided just the energy level needed at the end of a very long day!
Although this was the shortest day (ending at 1pm), it was nonetheless packed with no less than eight presentations, some which I felt could have done with more than the 30-45 minute time limit that the speaker was confined to.
Azure IaaS Essentials for the BizTalk Developer
Opening the day was a session by Stephen W. Thomas who started by recapping some of the great benefits of Azure IaaS like cost savings, flexibility of scale and making it easy for developers to provision temporary environments. On the flip side of the coin are the challenges such as the breaking changes that can occur from frequent updates and the speed of progression (especially with preview features). Stephen provided a very helpful guide for navigating through the plethora of different options we are confronted with, starting with a selection of 58 different VM sizes, basic vs. premium storage, and choosing classic vs. new Resource Manager VMs. His recommendation for BizTalk was to use a DS2v-2 VM with Premium Storage (about $110/month with an MSDN account). After reminding us to turn off VMs went not using it and to clean up our subscriptions, Stephen demoed the creation of a VM in the Azure portal, choosing a Windows Server 2012 Datacenter image with the Chef extension. He also noted that location matters, as an image created in East US costs more than the same image in Central US! Stephen finished up by discussing Azure Resource Manager (ARM) templates and the use of PowerShell, providing a link to his blog article and script that can create a full BizTalk domain using classic VMs in less than an hour!
Unified Tracking Across On Premise and the Cloud
In his presentation, M.R. Ashwin Prabhu stressed the importance of instrumentation and how it is too often overlooked or under-valued. The concept of an “informed application” becomes even more critical when we move into the hybrid solution space where applications span both on-premises and cloud. Ashwin demonstrated this clearly with progressive scenarios of exposing BizTalk Server BAM data in Power BI via Event Hubs and Stream Analytics, then doing the same for a Logic Apps solution, and finally combining the output of both to provide comprehensive telemetry across the hybrid solution. It was an impressive display of how the combination of these Azure services can elevate the visibility of what it typically a “black box” into reports that will enthuse business stakeholders and executives.
End-to-End IoT Solutions Leveraging Microsoft Azure Platform
Well-known TechNet Wiki Ninja and Seattle Seahawks fan Steef-Jan Wiggers talked about the ubiquitous Internet of Things (IoT) paradigm and how we see it today not only in cities and buildings but also in industries such as healthcare, energy and mobility. He described the “IoT Value Chain” consisting of event producers, data collection, data ingestors, transformation, storage and presentation – linking these items to the Azure services that support each of those capabilities. A key component for “learning about the past to predict the future” is Azure Machine Learning, a powerful platform for building and training models for predictive analytics.
Steef-Jan used his experience with both a wind energy and a greenhouse IoT solution in order to illustrate a scenario that leveraged Azure Event Hubs, Stream Analytics, Storage and Power BI to surface monitoring data. He also mentioned how IoT Suite can get you up & running in minutes via the pre-configured templates, and finally stressed that IoT is really more about data than technology.
Real World Industrial IoT
Kent Weare’s talk centred on Industrial IoT (IIoT) and that although it has its roots in Supervisory Control and Data Acquisition (SCADA), they are not one in the same. SCADA is heavily reliant in infrastructure, multi-level networks and firewalls, and proprietary protocols which make it an expensive proposition. It was this cost that prompted a client’s requirement for changing to a more modern methodology, leading to the solution that formed the basis of Kent’s example scenario. Kent’s team used a reference architecture of Event Hubs, Service Bus, Stream Analytics, App Service, Power BI, and BizTalk Server to deliver a hybrid solution for monitoring the state of devices and the Power Generation Output (MW). The solution went from Proof of Concept to Production deployment in less than six months! Kent provided three demos showing the telemetry architecture, real-time visualisations, and detection of the absence of events. He also discussed how BizTalk360 was used to monitor the solution, setting alarms based on a failure to receive an expected number of “heartbeat” messages.
Azure IoT Hub beyond Hello World
Capping off the IoT side of the conference was Mikael Hakansson’s talk which focussed on the mechanics of getting devices to talk to cloud services. After providing a brief walkthrough of the reference architecture, Mikael stepped bravely into an area few other presenters in the conference dared to: live coding. After successfully writing a Node.js program to process real-time data off of a Raspberry Pi 3, he explained the process for onboarding a new device:
Finally, he gave an interesting plug for his own company microServiceBus.com by demonstrating a facial recognition solution – one that even successfully identified a bubble-head statue of himself!
Implementing a real life scenario for the API Economy
Tomasso Groenendijk gave us a practical demonstration of how to implement your website as a Web App, and then use API Management to delegate security. For the first part, he took us through the process of creating the Web App using an MVC template in Visual Studio, storing the user data in an Azure SQL Database, configuring a custom domain and finally enabling SSL by creating DNS records and importing a certificate. One key step that often puzzles developers is how to generate the require .PFX certificate file; Tomasso showed how easy this is using OpenSSL.
In the second part, Tomasso then took us through the process of wiring up API Management to control delegation – which required setting up security in the Azure Management Portal first.
API Apps 101 for BizTalk Developers
Quicklearn’s Nick Hauenstein proved his exceptional skills as a trainer by delivering one of the most captivating and entertaining sessions of the conference. Using food as a metaphor, Nick first emphasized the priorities of designing an integration solution:
Then he walked through the process of designing a relatively simple integration solution for a business process which requires some familiar patterns: adapters, pipes & filters, pub/sub, process manager and content-based correlation. All of things we are used to doing with the rich toolset in BizTalk Server. However, Nick showed us a compelling solution using Logic Apps, Service Bus, and a custom API App – and his live demo proved that the concept of promoted properties and correlation works just as well in the cloud as it does in BizTalk!
This was a real eye opener, as many of us tend to think that the Azure integration tooling still falls behind the gargantuan capabilities of BizTalk. And in many respects, it does; but by widening our view to encompass the full range of Azure services and leveraging some clever programming, many of these capabilities can be replicated in the cloud without the huge upfront investment of the server product. Nick has published his entire presentation in blog format here for you to read.
How to Take a Customer from a Cloud Assessment to an Implementation
Wrapping up the conference was a session by one of the newest integration MVPs, Tom Canter. Tom exhibited his excellent story-telling talents by relating one of his experiences when he worked aboard a US nuclear submarine. I highly recommend watching the video (soon to be posted) to hear Tom’s version firsthand, but the punchline was that the CO was able to wake up from a dead sleep at 2am and within four minutes avert a near disaster by recalling a technical detail that escaped the five levels of highly trained officers below him. This underpins the Tom’s quote by Clemens Vasters: “"Software developers ought to have a solid understanding of at least one level of system abstraction below the one they build on."
Tom went on to highlight the five management pillars of an enterprise organisation, stressing that every successful business must do at least one well:
- Enterprise Resource Management
- Customer Relationship Management
- Supply Chain Management
- Financial Management
- Human Resource Management
Using the real-life examples of the National Academy of Sciences (NAS) and Louisiana State Employees Retirement System (LASERS), Tom showed how understanding what brings in revenue (for businesses) and what the most important function is (for non-profit organisations) drives the solution design.
Whilst Microsoft conveyed new and highly anticipated clarity about the integration roadmap in their sessions, the MVPs showcased their experience and knowledge with practical advice and examples of how to apply this broad range of technology in the real world. In this way, the product team and the MVPs complemented each other perfectly, contributing to an awesome event! Videos and presentation slides are slowly being made available for download here - I'd highly recommend checking them out!