Introduction :

For Organizations that have very strict zero data egress, data residency, or customer controlled infrastructure requirements, selecting an enterprise AI solution means going beyond just “secure.”

While different vendors use different terminology for describing how their services handle enterprise data, such as service boundary, data security, tenant isolation, encryption, privacy, no training on customer data, and other terms. However, the terminologies used do not all answer the same question, which is:

Where is your enterprise data really being processed? Who manages the infrastructure processing your enterprise data via AI inferences?

For instance, take Microsoft Copilot.

While Microsoft Copilot is described to work inside the service boundary of Microsoft 365 and process organizational data based on existing permissions, the service boundary managed by Microsoft is not equivalent to exclusively customer-controlled infrastructure.

That means:

Microsoft 365 Service Boundary ≠ Customer-Controlled Infrastructure

That could be significant if the policy of the organization required AI processing to occur within infrastructure controlled by it.

However, if an organization does not have such a policy, cloud AI infrastructure provided by the provider can be fully consistent with its security and privacy policies.

Thus, before choosing whether the chosen AI system can meet the needs of your organization, it is reasonable to start with understanding what these words mean.

Let us consider the example of Microsoft Copilot and see the whole path of the data from entering a prompt to retrieving enterprise data, AI processing, and generating the response.

What Is an AI Service Boundary?

Before we can learn where data is processed by enterprises, we must first understand what AI service boundaries mean.

Service boundary is defined as the environment wherein the service operates along with the components that it delivers through the service provider. It may have relevance in issues relating to security, privacy, compliance, and data residency.

According to Microsoft, the AI service Copilot operates in the Microsoft 365 service boundary. Also, according to Microsoft, the AI service Copilot is a shared Microsoft 365 service running in the managed cloud architecture of Microsoft.

The critical difference here is that the term service boundary refers to the service environment; it does not imply automatically the exclusive control of the infrastructure underlying it by the client.

Thus, we have two different definitions:

• Service boundary: The specified space in which the service provider runs the AI service.
• Customer-controlled infrastructure: The infrastructure controlled solely by the organization using the AI service.

They are not necessarily identical. In order to understand why, we need to trace the process of processing enterprise data with AI.

What Happens When You Use AI With Enterprise Data?

When a user prompts an enterprise AI assistant with a query, the system may require some relevant organizational data to generate a meaningful answer.

In Microsoft Copilot, Microsoft outlines the way in which organizational data can be accessed via Microsoft Graph and only with the permission granted to the user.

In brief, the data flow can be represented this way:

User Prompt → Relevant Organizational Data → AI Processing → Answer

And the relevant data is then used to anchor the AI-generated answer.

For example, when the end-user raises a query to the enterprise AI assistant about any document, the enterprise AI assistant can seek data that is available to the user.

Where Does AI Process Enterprise Data?

Once an employee enters a prompt into the enterprise AI system, it starts searching for information associated with that prompt from all available sources such as emails, documents, files, or other applications. Using the collected data, the AI system gives its answer.

This process is known as AI inference. In general, inference is the process of getting some output from the system based on the prompt and enterprise data.

The process can be illustrated in the following simplified way:

Prompt → Data Retrieval & Grounding → AI Model Processing → Response

When it comes to Microsoft Copilot, according to Microsoft, inference happens inside Microsoft-managed cloud infrastructure. It means that, in addition to data available to the AI system, we should care about the location of the AI inference itself.

Security and Infrastructure Control Are Different

The differentiation does not imply that there are no security controls on provider-managed AI services. For enterprise AI services, there can be access controls, encryption, tenant isolation, compliance controls, and other types of security and privacy controls.

For instance, for Microsoft’s Copilot, the access controls depend on the permissions of the user to see information within the organization. Thus, when using Copilot, no one gets access to information that they would not have been able to see without it.

However, it should be noted that security controls and infrastructure controls relate to different components of AI architecture. Security controls establish the way of securing data and access to it. The infrastructure control establishes who is responsible for the environment of AI processing.

In that way, an organization can have high-level security and privacy controls, but still use provider-managed AI infrastructure.

This is important for those organizations that have strict infrastructure control requirements.

Understanding the Enterprise AI Data Journey

Using enterprise data with an AI platform usually consists of three main steps – retrieval, AI processing, and response creation.

First, the platform obtains the data to which the user is allowed to have access. Then the AI uses this information as context to provide a reply.

From the architectural point of view, the important considerations include where the processing happens and who controls the infrastructure where it is done.

Yet, since you have mentioned all of these aspects in your previous section “Where Does AI Process Enterprise Data?”, I would delete this section entirely.

Conclusion

The companies cannot characterize AI data processing in one word like secure or private. The architecture of the system dictates the process involved in retrieving and processing enterprise information to give an AI-based response.

The Microsoft Copilot case study shows the significance of understanding the difference between a service boundary and the infrastructure involved in making the AI inference.

The service boundary identifies where the organization consumes its services, whereas the infrastructure control is about who controls the infrastructure for AI inference.

The difference becomes more important for an enterprise especially if enterprise guidelines have certain stipulations on issues such as data residency, data mobility, or infrastructure control.

In summary, the most important thing is for organizations to analyze artificial intelligence not only from the perspective of what it does but also from the perspective of how it processes data for the company and what infrastructure it uses.

Payal Walunj

Payal transformed creativity into intelligence, moving from fashion trends to data trends. With a background in fashion design and expertise in business analytics & marketing, she now helps businesses harness AI-driven insights with VADY at Newfangled Vision. Passionate about making data simple, accessible, & actionable, she turns complexity into clarity, bridging the gap betwen intuition and inovation.