In-Depth

SharePoint 2016 and the New 'Hybrid Search' Capability

Microsoft's new "Cloud Search Service Application" technology, timed for the release of SharePoint Server 2016, promises to bring a greatly improved SharePoint search experience for organizations.

• By Kurt Mackie
• 01/06/2016

One of the big features of SharePoint 2016 coming this quarter will be an improved "hybrid search" capability, which received a lot of speaker attention at 1105 Media's Live 360! event back in November.

One of the speakers who focused on that topic was Jeff Fried, chief technology officer at BA Insight. In his session, "Succeeding with Hybrid SharePoint and Search: Strategy and Implementation Including SharePoint 2016," Fried contended that Microsoft has gotten the message that organizations will continue to run SharePoint on premises, and that hybrid is going to stay. Fried is a search expert, having worked at Microsoft for three years as a product manager for search on the SharePoint team before taking his current position. He contributed to search technologies in SharePoint Server 2013 as well as the Office Graph solution that underlies Office 365 services.

SharePoint Server 2013 currently has a "disjointed" hybrid search experience, but a new technology called "Cloud Search Service Application" (Cloud SSA) will solve that problem, Fried contended. Cloud SSA, currently at the preview stage, is being built natively into SharePoint Server 2016, but it also will be available for SharePoint Server 2013, and the search crawling capability will extend across SharePoint Server 2010 and SharePoint Server 2007 workloads. Those prospects were outlined during the May Microsoft Ignite presentation that first introduced this new SharePoint hybrid search capability.

To use the new SharePoint hybrid search capability when it's commercially available sometime this year, Microsoft will require an Office 365 subscription, SharePoint Server on premises, the synchronization of user identities to the Azure Active Directory service (using Azure AD Sync), the use of file shares and BCS connectors, according to the Ignite presentation. The requirements are illustrated in this slide:

The setup process for the new SharePoint hybrid search involves selecting it to work with Office 365 services. Next, IT pros will have to run an "onboarding script" for SharePoint Server using PowerShell, according to the Ignite presentation. A reverse-proxy server will only be required in order for a document previewing function to work for customer premises-stored documents, per the Ignite session.

New vs. Old Search
Microsoft's new hybrid search capability will centralize the search index in the cloud. Instead of having separate search indices, with one index residing in Microsoft's datacenters and one index located on customer premises infrastructure, which is the current disjointed SharePoint Server 2013 hybrid search experience, the improved hybrid search capability will use Microsoft's cloud for the index. Other Office 365 services can then tap this same cloud-based index, such as Microsoft's Delve service, which surfaces information about people and activities within an organization. Anything put into Microsoft's cloud-based index will show up in the Office Graph, which is the search technology underlying Microsoft's Delve service, Fried noted. Microsoft's data loss prevention and e-discovery technologies also will work with the new hybrid search, according to the Ignite presentation.

The current hybrid search capability that's used with SharePoint Server 2013 and Office 365 services is described as a "federated hybrid search" approach. With this approach, a search query will return results from Microsoft's datacenters and also from the servers on the customer's premises, but there will be "no affinity between the two data sets," according to a description by Bill Baer, senior product marketing manager at Microsoft's SharePoint product group. 

Microsoft is promising that organizations will not have to maintain as many servers with the new hybrid search approach compared with the current federated hybrid search approach. The use of Cloud SSA possibly can reduce the server count by "65%," according to this Microsoft blog post.

Weaknesses
The need for reverse-proxy servers or a VPN to access premises-based server content remotely is the weakest part of the new SharePoint hybrid search experience, according to Fried. He explained that if an organization wants to provide remote access to documents and lists residing inside the organization's firewall (for so-called "inbound searches), then it will likely need a reverse-proxy capability, as well as identity solutions. Single sign-on identity capability can be used to ease the access experience for end users, for instance.

In short, organizations will need a synchronized directory such as Active Directory running with some sort of directory synchronization for the new hybrid search capability, Fried noted. He added that Azure Active Directory is associated with every Office 365 tenant, so many organizations likely already have access to this directory synchronization capability. The setup process has improved. Microsoft's DirSync directory synchronization was replaced by Azure AD Sync. It now takes an hour to set up directory synchronization, instead of days, according to Fried. He added that the Azure AD Sync capability is well documented by Microsoft but it's still hard to test it. He suggested provisioning another tenant to go through that process.

Key considerations for implementing the new hybrid search capability include security and regulatory concerns. Some countries have data export restrictions, which is the No. 1 reason not to put content in the cloud, Fried said.

While the search index resides in Microsoft's datacenters, Microsoft claims overall security for the new hybrid search capability. "The content metadata is encrypted when it's transferred to the search index in Office 365, so the on-premises content remains secure," Baer claimed in his blog post.

The new hybrid search capability will be fine for simpler use cases where organizations don't customize the search experience, according to Microsoft MVP Mikael Svenson. He described Microsoft's new hybrid crawl feature as a "cold hot potato" in a November blog post, citing certain restrictions. It can't be used where there are regulations against storing data outside the county (as in Europe), he contended. Organizations using dictionary mapping or a content enrichment Web service can't use the new hybrid crawl service, he added. He also suggested that organizations with client licenses for all of its SharePoint Online users can't tap this feature, based on its current implementation.

Hybrid Here To Stay
Microsoft's SharePoint customers currently are using hybrid infrastructures. Fried cited the example of the Kentucky Community and Technical College System's intranet, which couldn't bring 40 campuses together without using hybrid infrastructure. He explained that the college system uses the SharePoint Online service for most of its operations. Every classroom gets a Team Site and every student, faculty and employee gets their own Office 365 mailbox. It would be a nightmare to implement this intranet without some sort of hosted infrastructure, Fried contended. Nevertheless, the school system still has 30 custom solutions (such as PeopleSoft and payroll systems) that are built on SharePoint and those solutions aren't going to the cloud anytime soon, he said. The school system uses search as a bridge so that any campus can get a good SharePoint experience, he added.

There are some features in SharePoint Online that are missing, Fried said, which constitute reasons to keep some things on premises. Examples include the translation service and cross-site publishing, he noted. On the other hand, Microsoft likely won't bring some of its "cloud-first" capabilities to its premises-based server products. Those essentially are "cloud-only" capabilities, according to Fried. For instance, Microsoft's Yammer enterprise social networking service is only delivered via Microsoft's cloud. Other examples include Microsoft's Delve and Power BI solutions. Fried said that Power BI "could perfectly well go on prem, but Microsoft doesn't want to do that because it wants these things in the cloud." Organizations can respond by either migrating everything to the cloud at their own pace or they can coexist via a hybrid model, which are two different strategies, Fried said.

More insights from Fried about Microsoft's new SharePoint hybrid search capability can be found in this March interview, which was conducted by Christian Buckley, a Microsoft MVP, SharePoint expert and Redmond author. Microsoft is targeting the delivery of SharePoint Server 2016 sometime in Q1 of this year.