Report-only mode, and some more handy reporting functionality for Conditional Access and Azure AD

During its annual Microsoft Ignite 2019 conference this week, Microsoft announced a new feature for Conditional Access called Report-Only mode in preview.

So, what is Report-only mode?

Report-Only mode is a new option within a Conditional Access policy. Besides the option to turn the conditional access policy on or off, the option to Report-only has been added.

By enabling the Report-only mode the conditional access is evaluated on the client instead of enforced. Now you might ask yourself, wasn’t this functionality not already available by using the What if analysis tooling. The difference is that the What if tooling only moddeled the applicability of the Conditional Access policy based on the given scenario.

How does Report-only mode work and look?

The first thing to do is to create a Conditional Access policy and enable the policy for Report-only. In my case I have created a Conditional Access policy titled: “I24 – Accept User Terms” which grants access to the selected cloud apps, once the Insight24 terms of use have been accepted.

Grant Control for Conditional Access policy

After the conditional access policy is created, with Report-only mode selected, we have several options to determine the outcome of the evaluation.

Option 1: Azure Active Directory Sign-in logging

Go to the Azure AD administration portal | Monitoring | Sign-ins and select one of the listed sign-ins. Once selected, within the sign-in logging, a new tab has been added titled: “Report-only (Preview)”. Here you can see, in this example that the “I24 – Accept User Terms” conditional access policy reports the result:”Report-only: User action required”

Option 2: Use the power of Log Analytics and the new Azure AD workbooks

If you forward your sign-in logging to a Log Analytics workspace, you can use workbooks to report on the Report-only conditional access policies.

Once the SignInLogs are sent to the Log Analytics workspace you can open the Azure AD workbooks by selecting Monitoring | Workbooks.

Under Workbooks, several Workbooks are available:

Sign-ins
Sign-ins using Legacy Authentication
App Consent Audit
Conditional Access Insights (Preview)
Sign-ins by Conditional Access
Sign-ins by Grant Controls
Sign-ins Failure Analysis

For this example the “Conditonal Access Insights (Preview)” workbook is of special interest. By selecting the “I24 – Accept User Terms” under Conditional Access policy you get an overview of the applicability of the Conditional Access policy. As you can see in this example the Conditional Access policy is evaluated 5 times, it will not be applied 4 times and for one scenario, a User action is required.

Caveat so be aware: Policies in report-only mode that require compliant devices may prompt users on Mac, iOS, and Android to select a device certificate during policy evaluation

Conclusion

While at time of writing the Report-only mode is still in preview, it is a must have option if you want to implement your own custom conditional access policies. Before enabling your conditional access policy, first use the Report-only option to test it’s outcome for your users. This can really be a lifesaver, and gives more insights compared to the whatif tooling.

I will try to update the Conditional Access whitepaper and Conditional Access implementation flowchart as soon as possible to integrate this functionality. From now on, using Report-only will be used for sure.