Knowledge Agent, released by Microsoft in public preview last week, opens up powerful content management features for SharePoint libraries that you can turn on across your entire Microsoft 365 tenant or limit to specific document libraries.
It’s an intelligent agent that accesses metadata and uses prompts to improve information retrieval from documents stored in SharePoint. Prompts allow you to query and ‘take action’ on documents. These new features use AI to enable more reliable AI for Copilot and agents.
Knowledge Agent differs from Copilot Studio agents, which don’t natively access metadata or any other external contextual elements (yet). Normally, we use third-party partner tools to add this feature. Copilot Studio agents can be developed and deployed for specific use cases. Knowledge Agent is your general SharePoint agent, and it can access that contextual data (metadata) that is key to agent reliability.
Is Knowledge Agent reliable?
We started testing Knowledge Agent immediately upon its release. Honest opinion: We were impressed with its out-of-the box performance but would like more robust taxonomy management.
For context, we have a set of control questions that we use to test native Copilot Studio agents (with no connected metadata) and enhanced Copilot Studio agents (metadata connected via partner technology).
Note: We already have a robust taxonomy for our test documents.
Once we set up and enabled new features (metadata extraction) in SharePoint, Knowledge Agent was able to natively answer most questions. It correctly responded to questions that our native Copilot Studio agents (no metadata) could not answer. On one 25-query test, Knowledge Agent passed on 24 responses, with one fail.
Note: We used Knowledge Agent with the autofill columns feature to add metadata in first tests. This means we did not review and adjust the AI-generated metadata that was automatically extracted from documents.
Let’s take a look behind the scenes at what makes Knowledge Agent work and how to improve reliability.
Microsoft 365 features that support Knowledge Agent reliability
Knowledge Agent relies on text that appears in the document and contextual metadata. The contextual metadata is what sets it apart from other agents.
To improve agent reliability, you can take advantage of new and existing features in SharePoint. We share a few of them here:
- Auto-tagging and autofill columns allows you to auto-populate columns using prompts to extract or generate what you need, including metadata, summaries, and triggers.
- Taxonomy tagging and managed metadata allows you to manage metadata using controlled language and auto-tag documents.
- Document classifier allows you to use classifier and extraction features to identify and classify documents by content type.
- Content assembly allows you to auto-generate net new documents based on templates.
Auto-tagging in SharePoint
SharePoint now supports auto-tagging. Well, to be honest, SharePoint already supported term extraction through Syntex, which was not reliable according to our tests. Now, term extraction and metadata auto-tagging is easier and more reliable. This is possible due to the new autofill columns feature in SharePoint. Each SharePoint document can have multiple columns. These columns can be used to extract metadata from documents, create summaries, or trigger actions.
Auto-tagging is a huge advantage in SharePoint, as it reduces the effort required for a time-consuming, manual process. The new autofill columns feature can recognise exact terms in documents and add them to columns you set up.
As you can imagine, using AI to extract metadata is hit and miss. Fortunately, this feature allows an admin to edit metadata. This is a great feature, as a subject matter expert or taxonomist can improve the extracted metadata for each document. That, of course, requires manual, human expertise and review.
Term extraction may have one downfall. If end users don’t use the exact terms found in the document, they may not get what they’ve asked for. There is no native thesaurus manager in Microsoft 365. This is the same reliability issue we have with native Copilot Studio agents. A third-party taxonomy manager and connector must be used to extend metadata to include synonyms, which is what we use when creating Copilot agents.
You still have the option (existing feature) to create a managed metadata column, which links to your term set and uses a controlled vocabulary – more on that later.
We’re investigating best practices for using term extraction, managed metadata, and term sets. We want end users to have the best experience when using natural language to query and instruct Knowledge Agent.
The game-changer here is that whether you use extracted metadata or managed metadata (or a combination of both), Knowledge Agent can read the metadata to gather additional context and improve reliability.
Why auto-tagging is important:
- Search and find documents easily
- Organise and manage unstructured documents
- Trigger workflows based on metadata
How autofill columns work
Autofill columns automate extraction, summarisation, or generation of net new content from files, allowing you to associate metadata and prompts with individual files. This adds context to the document, grounding the content and allowing the agent to respond or generate net new content more reliably.
Using prompts to autofill data
Now this is a cool feature. When setting up autofill columns, you can use a prompt to indicate what type of data you want auto-populated. For example, if you have contracts, you can enter a prompt to extract the client name. For invoices, you can set up two columns, with two prompts. In one column, prompt for the client name. In the next, prompt for the value (currency) of the invoice.
You can use prompts to make decisions, which may include ‘yes’ or ‘no’ as the metadata. You can prompt the model to autofill the status of the document, such as stage of review.
Keep in mind, that autofill columns only reads text in the document, so for the examples above, your documents must have this information available. (Other options are available to extend this capability to related text and related documents.)
You can use autofill columns alongside other document processing features in SharePoint.
Other document processing features in SharePoint
Autofill columns is just one of many document processing features in SharePoint. Other features allow you to auto-classify documents, manage metadata based on a term set (taxonomy), tag images, translate documents, use optical character recognition, assemble new content, and more.
Particularly interesting to us:
- Taxonomy tagging, managed metadata, and term store
- Document classifier and extractors
- Content assembly
Taxonomy tagging and managed metadata
This is not a new feature, but it certainly is underutilised by most companies with SharePoint. While the new autofill columns feature is impressive, extraction of terms from documents is only one step in creating and managing metadata that ensures reliability for agents.
Using the Microsoft term store, you can create a controlled vocabulary. The taxonomy tagging feature auto-populates a managed metadata column that tags documents based on your controlled terms.
Caveat: You should have a taxonomy for managed metadata, and taxonomy management in SharePoint is rudimentary.
Document classifier and extractors
SharePoint classifier can identify and classify documents by content type. For example, if you upload contracts to a SharePoint site, the classifier can identify all contracts and tag them as such. But this only works if you’ve identified the content type within the source document.
What this means: You need to develop an information architecture for your SharePoint libraries, creating a content model with all relevant content types. Your team must be trained to classify documents by content type.
Extractors work with classified documents to pull information from the documents. In the contract example, the extractor can identify client and start date and extract the specific information into columns. But you must define content models for the type of metadata you want extracted.
For both the classifier and extraction features, the model must be trained to identify and extract data. You do this by providing a ‘control’ set of documents.
Content assembly
This document processing feature auto-generates templated, structured documents from unstructured documents, such as contracts, proposals, supplier agreements, SOPs, and more.
Basically, you set up standard templates for documents that will be created repeatedly. Simply upload an unstructured document with relevant information, and ZING! Like magic, you have a new document ready to send out – after human review, of course.
We have yet to test Knowledge Agent extensively with content assembly, although we generate net new content reliably with other agents. We’re already imagining how the autofill columns feature could extend Knowledge Agent to generate structured, templated documents on the fly.
Final thoughts on SharePoint Knowledge Agent
Taxonomies, metadata, and content models make SharePoint a viable, even powerful, repository for unstructured documents. With the addition of Knowledge Agent and the autofill columns feature, the documents your teams store in SharePoint become a source of useful and valuable information that can be transformed into knowledge, insights, and highly efficient workflows.
By developing content models and applying metadata, SharePoint documents get context, and this is what improves the reliability of Knowledge Agent or a use-case-specific Copilot agent. While taxonomy management in SharePoint leaves much to be desired, you can still achieve agent reliability for many use cases using Microsoft 365 document processing features.
In short, Knowledge Agent and autofill columns take SharePoint to a whole new level. However, you still need professional, human intervention to ensure reliability.
How Altuent helps improve agent reliability
We’ve long been experts at developing information architecture for structured systems. Now, we develop information architecture for documents stored in SharePoint and other unstructured repositories. These advances in Microsoft 365 are music to our ears. With robust content models and custom business taxonomies, we see SharePoint as a viable knowledge repository.
When you use agents (going beyond Knowledge Agent with custom Copilot Studio agents), your SharePoint repository becomes a valuable source of information for everything from enabling faster search for internal teams to extracting deep insights for management teams.
Find out more about the Altuent AI Accelerator
Do you want more infromation about our AI Accelerator?
Do you have an AI related project you want support on?
