*But Were Afraid to Ask

If you find the inner workings of Exchange data backups using Volume Shadown Copy (VSS) a bit mystifying take comfort in not being alone. Administrators may ask, “What’s all the “freezing” and “thawing” I’m seeing in my event logs? What is the Exchange VSS Writer really, and what is it doing to my databases? How does it create a snapshot of a 135GB database in less than 60 seconds?”

If you ever asked these questions but only felt more confused with the answers, here’s a guide to clear some of that up. To understand how a VSS backup of Exchange works it’s critical to understand the basics of VSS itself. There is some excellent documentation on TechNet and MSDN on this, as well as the Windows Server Core Team blog, “Ask the Core Team.” My esteemed colleague Randy Monteleone sums up the basics of VSS very nicely early in his post, while also providing links (repeated here) to some good TechNet primers on VSS:

How To: VSS Tracing – Randy Monteleone
http://blogs.technet.com/b/askcore/archive/2012/04/29/how-to-vss-tracing.aspx

How Volume Shadow Copy Service Works
http://technet.microsoft.com/en-us/library/cc785914(WS.10).aspx

Volume Shadow Copy Service
http://technet.microsoft.com/en-us/library/ee923636.aspx

If you’re already familiar with at least the basics of VSS, then look forward to Part 2 in this series, where we will break down the events that occur in a VSS Exchange backup, and how Exchange logs them in the application event log.

If you need a quick primer or refresher on VSS basics and the Exchange Writer I’ve condensed them into some visual points below to complement the references above.

Snapshots

Bear in mind that VSS solutions for Exchange, and for all applications, vary greatly between different hardware and software configurations. There are clone and COW snapshots, hardware and software solutions, just a very wide variety of technologies based on the core VSS subsystem. For the purposes of understanding Exchange backups we’re only going to illustrate one specific type of solution out of the multitude. Detailed below is what’s called “copy-on-write”, or “COW” snapshots.

In a COW snapshot-based VSS backup of Exchange we have the creation of snapshots of the disks where Exchange data is hosted. No matter what is getting backed up, even if it’s a single database file and a few logs, VSS creates a snapshot of the entire disk where any data is stored. If the data resides across multiple disks, such as when an Exchange database is on one disk, and the logs are on another, VSS will create snapshots of any and all of those disks.

So what is a “snapshot”? A volume snapshot is an area of space inside what’s called “shadow storage”, which is itself a typically small area of space on the disk located in its System Volume Information folder.

After a disk snapshot is created a change to any data block from that time forward cannot get written until a copy of that block’s data before the change (as it was when the snapshot was created) gets written to the differencing area in shadow storage. In this way the data on the disk at the time the snapshot was created is preserved, block by block, in the shadow storage area. The snapshot data is then available either from the original disk, if the data blocks requested haven’t changed, or from the differencing area if they have. The fundamentals of this are illustrated below:

Disk E: has a snapshot created at 1PM:

A minute later one of the blocks gets written to, but not before the data as it was at 1PM gets preserved in the differencing area:

As the actual disk changes the data as it was at 1PM gets written into shadow storage, preserving a record of the disk as it was in that moment:

The following step:

In the figure above a backup server requests data from the snapshot of blocks 2 and 53. Block 53 from 1PM is preserved in the snapshot, so it’s copied directly from shadow storage. Block 2 is unchanged since 1PM, so it is copied via the VSS driver VOLSNAP.SYS, which operates much like a filter driver underneath the NTFS.SYS file system driver. By working in the IRP stack (the part of kernel memory that manages disk I/O) underneath the file system it can read blocks of data without NTFS objecting that a file is in use. VOLSNAP.SYS is also responsible for ensuring blocks are copied over to shadow storage if a write is requested to them, hence the name “Copy On Write”. Here is more about VOLSNAP.SYS from Tim McMichael:

Exchange / VSS / and differential block size…
http://blogs.technet.com/b/timmcmic/archive/2011/07/12/exchange-vss-and-differential-block-size.aspx

Now that we’ve got the basics of a COW snapshot down let’s look at how it works with Exchange, along with some other major concepts:

Microsoft Exchange Writer

So we know that any disk that stores Exchange data gets a snapshot created of it by VSS. How exactly, though, does a backup application find out which disks those are? Oftentimes an administrator selects databases for backup without specifying anything about what disks their data files are stored in. So something is required to provide the information about where the data files are, and therefore what disks VSS needs to create snapshots of. This information also tells a backup application, also known as a VSS requestor, what specific data files should be copied out of the snapshots for preservation on backup media, as we don’t want to copy out anything from the disk we don't need.

The mechanism at work here is the Microsoft Exchange VSS Writer. Like any application’s VSS writer (there are many, just run VSSADMIN LIST WRITERS to see them) its first job is to tell the backup application about the data needed for backup, especially the EDB file, logs, and checkpoint file for each database requested. The information about these specific Exchange data files is known as writer metadata.

(click thumbnail for full size version)

In the figure above we see the initial steps of an Exchange backup. The Exchange Writer tells the backup server (the requestor) that there is a database located in a folder on volume E:, and that transaction logs for that database are in a folder on D:. Based on that information the backup application will request snapshots of the D: and E: volumes when the job progresses.

The Exchange VSS Writer serves another critical role besides providing metadata to VSS requestors. It also has the job of stopping writes to the databases and logs on disk, or “freezing” them, for the time it takes to create the necessary snapshots. A COW snapshot typically takes a small amount of time to create, as all it consists of initially is the designation of an area in shadow storage for blocks to be preserved in when they change on the actual disk. Despite this relatively quick operation it can still take up to a minute, which is plenty of time for blocks of data to change on a disk between the start and the end of its snapshot creation process. If blocks of data change but don't have the originals preserved from the exact time the snapshot creation begins those blocks may become inconsistent with other snapshot data, especially between logs, database, and checkpoint files. Hence, the Exchange Writer forces the Information Store Service, or the MS Exchange Replication Service, from writing what’s in RAM to the frozen database files. In the case of the Information Store Service, the current transaction log file (Exx.log) gets rolled and closed out before the Exchange Writer allows VSS to take the snapshot. This ensures nothing changes in the file data between the beginning of the snapshot and the completion, at which point the databases are “thawed”. When databases are thawed write I/O held in RAM is allowed to go to disk again.

Here's more information on how an application's VSS writer interacts with VSS with regards to freeze, thaws, and the time needed to get a snapshot completed:

CVssWriter::OnFreeze method
http://msdn.microsoft.com/en-us/library/windows/desktop/aa381563(v=vs.85).aspx

The last major responsibility of the Exchange Writer is to tell the Information Store Service (MS Exchange Replication Service in the case of a passive copy backup) that the backup was completed and, if applicable, carry out post-backup tasks like log truncation, marking the database as no longer with a backup in progress, etc.

In the part two and part three of this series we’ll look at a play-by-play breakdown of how the elements described above all come together in an Exchange backup, the application log events that get generated, and compare the process for a mounted database to that for a passive database copy.

Thanks go out for the collaboration on the content in these posts to Michael Blanton, Tim McMichael, Randy Monteleone, Dave Vespa, and Tom Kern.

Jesse Tedoff

TechEd North America 2012 is coming and folks on the Exchange team will be there to talk to you about Exchange hybrid deployments and Exchange Online migrations! We've gotten a lot of questions on hybrid deployments, especially after releasing Exchange 2010 SP2 with the Hybrid Configuration Wizard and updating the Exchange Deployment Assistant with more streamlined instructions to deploy hybrid - so here are some answers.

Ben Appleby and I recently did a three-part TechNet Radio set of video podcasts to shed additional light on hybrid topics as part of the "Road to TechEd 2012" series.  These podcasts should help provide some insights into our hybrid TechEd sessions if you're attending TechEd but also provide solid guidance and information if you're not going this year.

1. We dive into discussing migration options for you on your journey to the cloud in Office 365 Migration Options and Choosing a Hybrid Exchange Deployment. Everyone's business and IT needs are different -  whether that means moving to the cloud over a weekend or migrating over a few months. Hybrid is special because it can serve a couple of distinct goals:
   • To migrate all of your mailboxes to Exchange Online while maintaining a great experience for your end users such as availability of free/busy calendar data and preservation of headers so MailTips and OOF messages look right between on-premises and online users. Some of you need to take a more measured approach to the cloud  but keep users productive as you work with business timeframes and constraints - such as change moratoriums during holiday periods for retail organizations or as you test and update LOB applications to work with Office 365 services.
   • Keep mailboxes on-premises and online  for a longer period with the ability to move mailboxes back and forth between the two environments seamlessly. The City of Rome expects to maintain some servers and mailboxes on-premises for a longer period of time in order to leverage the rest of their IT investment so hybrid is an ideal deployment state.

2. The second podcast is How the Hybrid Configuration Engine Works in Exchange Server so you can get an idea of what the Hybrid Configuration Wizard actually does once you click 'update' and trigger the Update-HybridConfiguration cmdlet to start the hybrid configuration engine.  We'll cover this a little more in depth during TechEd 2012 as well as provide a demo of the updated wizard.

3. The final podcast is FAQ's and Common Gotchas with a Hybrid Exchange Deployment which covers important topics you should be aware of. In this episode, Ben mentions the Exchange Online Migration Performance Guide Whitepaper which is a great reference point. Or if you sometimes neglect to read documentation really carefully (like me, whoops!), I highly recommend checking out Understanding Certificate Requirements for Hybrid Deployments on TechNet.

Hopefully those podcasts give you a taste of some of the amazing content that's coming your way at TechEd 2012 this month as well as reference points  that dive into some key hybrid topics.

Here are the migration and hybrid TechEd sessions you should check out:

TechEd North America

• EXL302: Exchange Online Migration - Ram Poornalingam
• EXL303: Configuring Hybrid Exchange the Easy Way - Neil Axelrod

TechEd Europe

• EXL302: Exchange Online Migration - Ben Appleby
• EXL303: Configuring Hybrid Exchange the Easy Way -  Ben Appleby

We'll also be at MEC and we'll be excited to talk to you later in the year on these topics as well. So if you haven't registered already and want to chat with us about hybrid, migration and Exchange 15, go do it!

Ann Vu

TechEd North America 2012 happens next week in Orlando, Florida. This year, there are several Exchange break-out sessions and hands-on labs for IT pros and developers, including sessions on Exchange 2010 high availability, virtualization, retention and archiving, load balancing, connecting with SharePoint, and more!

You can use the Schedule Builder on the TechEd web site to select the sessions you want to attend and sync session info with your Outlook calendar (and have the info handy on your mobile device). For more info, head over to the TechEd North America 2012 web site.

If you’re attending, swing by the Exchange/Unified Communications booths to meet Exchange team folks. We’d love to hear from you and answer your Exchange-related questions. See you in Orlando!

Fifth Annual UC Roundtable

The TechEd UC Roundtable event is an independent, UC-related event

This is a great opportunity to meet and greet the Exchange MCMs, MVPs and the Microsoft Exchange Product Group at Microsoft TechEd North America 2012 in Orlando. Appetizers and drinks will be served, compliments of our friends at F5. Bring your Exchange and Lync questions for a fun evening of discussions around both products. MCM Program Manager David Bjurman-Birr will be there to answer questions about the MCM:Exchange program, as well.

To attend the 5th Annual UC Roundtable, June 13th, 6:00PM – 8:00PM EDT, please register directly using the link below. Prior to registering for the event using the Event Direct Link provided below, you will be prompted to sign in or create a new user account (user profile of your name, address, phone, email, etc.) if you do not already have one.

Event Direct Link - Registration required

Instructions to create a User Account and Register for the Event:

1. After clicking Event Direct Link provided above, click “External Candidate” or “Microsoft Employee”
2. Complete all requested information, and select “Submit”
3. Click “Register” button to complete your registration for the event.

For assistance with registration, please contact advcert@microsoft.com.

We look forward to seeing you Wednesday, June 13th in Orlando!

Exchange Team

Back in December 2011, the Exchange team first started making the Exchange Online December 2011 Service Update available to customers.  This rollout was staggered globally, with full deployment expected to be completed within six months.  For more information on the new features and enhancements included as part of this update, please refer to the original blog post.

While we are pleased to announce that most Office 365 tenants have already received the Exchange Online December 2011 Service Update, there are still some regions that are awaiting this update.  Full deployment to these remaining tenants is expected to be completed over the next six months.  Approximately one week before the service update is deployed to these regions we will post more information in the Planned Maintenance section in the Service Health Dashboard panel.  Administrators should check this resource for awareness of when their update is scheduled to begin.

Thanks very much for your continued patience and support as we finish deploying this important update to the service.

Regards,

Exchange Online Team

* But Were Afraid to Ask

Part 2 of this series (Part 1 is here) breaks down the events that take place during the backup of a mounted and active replicated database in an Exchange 2010 Database Availability Group called, simply enough, “DAG”. In this example the backup server is asked to create a full backup of database DB1 on server ADA-MBX1, using non-persistent COW snapshots:

(please click thumbnails for full size version of graphics in this post)

Event 9606 indicates that the VSS requestor has engaged the Exchange writer, and reports the instance GUID for the backup job that is starting. In this case the instance is 830705de-32d9-4059-94ea-b9e9aad38615. This instance GUID persists throughout each job, and changes with each subsequent one. You can therefore use it to track the sequence of events for each individual job. At this time the Exchange Writer provides metadata about the databases and logs present to the backup application.

Events 2005 and 9811 indicate an instance number assignment for ESE. So along with the writer instance GUID from event 9606 we can also track a job’s progress using these ESE instance numbers which increment by one with each job. At this stage the database is marked with “backup in progress” in the Information Store Service's memory space.

Just after the backup application has determined which disks need snapshots created, based on the data locations provided by the Exchange Writer metadata, it goes ahead and requests those snapshots. As the snapshot requests arrive event 9608 gets generated, indicating the Exchange writer’s acknowledgment of what’s about to happen. It then must halt disk writes to the database(s) and logs, also known as a “freeze” for the duration of the snapshot generation process.

When event 2001 is generated the current transaction log is closed, and the freeze begins. Writes from STORE.exe to the disks are held in memory.

Once these events appear we know the snapshot(s) have been created, and writes are allowed to database data blocks again.

Once the snapshots are created the backup application can copy blocks of data from the VSS subsystem, getting blocks of data from shadow storage if they’ve been preserved due to a change, or from the actual disk volume if they haven’t. The Exchange Writer waits for the signal that the transfer of data is complete. This flow of data is represented by the purple arrows, which in this case indicates data getting copied out of the snapshots in storage, through I/O of the Exchange server, and on to the backup server.

Once the backup application finishes copying data it will signal VSS that it’s done. VSS in turn signals the Exchange writer, which then initiates post-backup steps, signified by the above events. Event 225 appears to state that log truncation won’t occur, but that event is misleading. For a standalone database, upon backup completion, ESE would go ahead and clear logs accordingly. However, when a DAG replicated database is involved a check of other database copies must be performed in coordination with the Exchange Replication Service to ensure log truncation can continue. Once that check is complete the logs eligible for truncation are deleted. The database header is marked with information about the backup and the backup in progress bit is switched off in memory. In this case the snapshots used for the job are destroyed as part of the completion. In other types of backups, such as incremental, the persistence of the snapshot varies, but in this case they are removed.

In the next post in this series we'll break down the backup of a passive DAG replicated database copy.

Jesse Tedoff

We've made several enhancements and bug fixes based on community feedback.

Please go to our Mailbox Server Role Storage Requirements Calculator updates tracking page to see what is in this new version!

A blog post explaining the calculator is here and or you can download the calculator directly.

Comments welcome!

Ross Smith IV
Principal Program Manager
Exchange Customer Experience

6/22/12 Update - This article and the accompanying download have been updated.

The latest release of the Exchange Client Network Bandwidth Calculator includes several updates but arguably the most significant is time zone support. I have been looking at how to deal with the time zone problem for the last 12 months or so and it’s taken a while to come up with a practical solution. However, I am getting ahead of myself, so firstly let’s look at what the time zone problem actually is.

What is the time zone problem?

I am going to assume that everyone knows what time zones are and why we have them, however for those that want to know more I recommend reading the Wikipedia article below;

http://en.wikipedia.org/wiki/Time_zone

The actual problem with time zones from a network bandwidth prediction perspective is that we may be trying to model workloads from users in different parts of the world who are sharing either the same network connection or the same end service. This causes us a problem since the peak usage times for most users is relative to their local time zone;

For example, if we look at a normal working day for an average 1000 user organization we see two typical peaks, one in the morning around 10am that lasts for 2 hours and a later peak in the afternoon at around 2pm that lasts for 4 hours. When we plot this it looks as follows;

Now, let’s imagine that we are modelling the requirements for 5 different locations around the world, each supporting 1000 users accessing a shared resource in New York. At this stage let’s assume that the shared resource is a load balancer fronting Exchange 2010 on-premises (I thought I would choose an on-prem example for a change J)

• London (GMT) = 1000 users
• Warsaw (GMT+1) = 1000 users
• Jakarta (GMT+7) = 1000 users
• Redmond (GMT-8) = 1000 users
• New York (GMT-5) = 1000 users

If we model this using our legacy technique of predicting the peak for each set of users and adding them together we get the following;

What this chart is showing is that each site of 1000 users required roughly 1.56Mbits/sec of bandwidth at their peak each day and so when we add them all together to account for all of the users sharing the load balancer in New York, we get a peak requirement of 7.81Mbits/sec. This is how we have historically dealt with bandwidth planning for distributed users, predict their peak requirements then stick them all in a table and add them together.

The issue here is that the users in Europe will be going home when the users in Redmond are just waking up and the users in Jakarta will be tucked up in bed!

If we take the time zone of these sites into account then the chart changes significantly;

This chart shows how the workloads would actually combine to form a very different usage profile than we would have historically planned for. What is really interesting here is that our peak workload is now much lower at only 3.78Mbits/sec (our original prediction was 7.81Mbits/sec). The usage profile is also very different to our original prediction.

How can we deal with this problem?

Well, as you may have guessed from the charts above we have extended the network calculator to allow you to include time zone details!

What we actually did to achieve this was to abandon the idea of predicting only the peak workload and instead we now predict bandwidth usage per hour of the day based on the usage patterns provided such as when is the morning peak time and afternoon peak time etc. This allows the calculator to not only know when your peak usage will be, but also what the usage will be like throughout the rest of the day. Once we know what this curve looks like it becomes possible to add the data together accounting for time zones.

Does anyone really do this single consolidation?

Well, the simple answer to this is yes – many organisations are consolidating workloads as much as possible. This is requiring design teams to plan for service workloads from very distributed users; often with different profiles and in different time zones. This is especially common where workload is moved to the cloud since Office 365 provides only single regional tenant locations and so a global organisation using Office 365 will have to plan for a large proportion of its users accessing the service in a totally different region and time zone, often over shared infrastructure.

Many customers I work with are also consolidating many smaller datacentres into fewer, larger datacentres – these consolidated sites must be able to handle the workload from the previously distributed users, quite often these users will be in different time zones and so when we try to accommodate their workload we need a way to figure out how they will combine with other distributed workloads.

Obviously, if all of your users exist in the same time zone then you do not need to concern yourself with all of this and just use the calculator as normal.

Using the new time zone feature

Ok, so you have a scenario that requires time zone support, how on earth do you make use of the new feature?

First things first, we need to configure the TimeZone configuration table in on the input sheet. The parameters entered here control the shape of the usage curve used to combine the workloads. The values here need to reflect the average usage patterns within your organisation, I typically will take a look at performance data from running Exchange servers to create this, combined with asking the customer how they think that their users work and when their peak times are.

Once the input sheet is completed, we move to the Client Mix sheet – on here we have two new areas to configure time zone data.

First is the Model Timezone, this represents the time zone of the resource we are planning for, i.e. the network link or load balancer. In the example earlier you can see that I have set the model time zone to be GMT-5 for New York which is where our load balancer was.

Second we have a new column called TimeZone – this represents the time zone of each individual site relative to GMT (note I am British and went with GMT here, but I may move this to UTC in the future if enough people complain).

The resulting prediction is shown in a chart underneath the client mix table as shown earlier. The values in this table are Mbits/sec and represent the predicted network usage at each hour of the day.

An Added Bonus

One of the other nice features is that the calculator will provide a table that can be copied into the Mailbox Role Calculator to help with DAG network replication prediction.

If you look to the right of the prediction chart (Client Mix Sheet) in the Network Calculator you will see a table that contains % change per hour of the day… if you copy this onto your clipboard…

Then scroll to the bottom of the Input sheet in the Mailbox Server Role Calculator, you will find a table for Log Replication Configuration. Paste the figures from the Network Calculator into this table.

And there you have it, the Mailbox Server Role Calculator is now able to predict the bandwidth requirements for DAG replication taking both your organisation data and timezone configuration into account!

Conclusion

Hopefully this new feature will help many of you more accurately predict your network bandwidth requirements; it is not necessary for all deployments but for those large enterprise architects out there struggling with this problem I hope that the time zone feature helps you.

Please continue to provide your valuable feedback - both positive and negative, to the netcalc AT microsoft.com address. We love to read your comments!

Neil Johnson
Senior Consultant, MCS UK

One of the first actions most Exchange Administrators generally take when troubleshooting suspected problems with Exchange Content Indexing will be to rebuild the impacted Mailbox Database's content index files (either manually or by using the ResetSearchIndex.ps1 script found in the \Exchange Server\Scripts directory). I've also worked with many Exchange administrators over the years who choose to proactively rebuild search indexes at various points during the calendar year or as various milestones within a project are met (say in a migration project; as a single example).

Irrespective of the rationale for resetting the indexes, most administrators, if and when asked, will be unable to provide realistic estimates for how long this process will take from start to finish. The undesired consequences of not estimating these times accurately will differ from organization to organization. Some IT departments may discourage not having server side indexes available to users during the business day citing losses in end-user productivity and upticks in escalated issues hitting the Help Desk. From an operational perspective, not having knowledge of the anticipated rebuild times might also prevent Exchange administrators from being alerted to potential problems within the rebuild process itself. Whatever your rationale, having a sound understanding of how long the process might take is valuable.

Admittedly there's very little information about the amount of time it takes (or better yet, the time it should take) to rebuild an Exchange content index available today. Ostensibly, this is because the actual rebuild times are always variable. There are many factors that influence the rebuild rates and time to complete. Most notably:

• Variability in the total number of end-user mailboxes “homed” on an Exchange Mailbox Database
• Variability in the size of mailboxes contained on an Exchange Mailbox Database
• Variability in item counts between user mailboxes homed on an Exchange Mailbox Database
• Variability in item counts between Exchange Mailbox Databases (when performing concurrent rebuilds)
• Variability in the size of items residing within an Exchange Mailbox Database
• Variability in number and size of mail attachments residing within an Exchange Mailbox Database
• The types and number of enabled IFilters on an Exchange Mailbox Server (allows for the indexing of various file formats)
• The overall system resource utilization of a Mailbox Server performing crawl (think throttling)
• Many more…

At Microsoft (both in our corporate implementation as well as in various Office-365 offerings), we utilize a Search Rebuild Framework that was developed by my colleague Anatoly Girko and I. This framework was originally designed to provide our internal operations staff with a set of comprehensive validation steps and progress indicators they could leverage when performing rebuilds of content indexes. These techniques are utilized at various key milestones within the overall rebuild process to ensure successful completion.

As the framework evolved, we decided to add additional functionality that would allow us to track and store historic throughput metrics for any and all rebuild operations. As these data collections grew, and as the subsequent trending data came to light, we found we were able to make significantly more informed and more accurate estimates for how long a given rebuild operation might take. This, in turn, allowed us as an operational team to make better decisions on when to schedule rebuilds so that we could minimize disruption to our end user customer base. Since inception, this framework has been utilized to oversee rebuild operations for several thousand content indexes within the various environments we support.

In a series of articles, we will discuss our “Rebuild Framework” so that interested parties might apply a similar methodology to their own environments should such a need arise. Each stage of the framework will be detailed including discussions on the various toolsets we have authored to assist with this process. This series will conclude with a series of graphs and tables that detail our observed content indexing rebuild statistics and conclusions to date. For customers who have not tracked statistics in this area previously, we hope this will serve as a valuable reference point. Presumably it should provide the ability for you to make more informed estimates concerning Exchange content index rebuild times in your own environments. That said, it's worth noting that because all Exchange environments are unique your rebuild metrics may differ dramatically from the rates we have observed and are presenting here.

Before diving in “head first” it’s also worth mentioning that this series is not intended as a troubleshooting guide. Our expectation is that your own troubleshooting has led you to a decision to perform rebuild(s) either in response to a problem or as a proactive measure. All examples presented within this series will focus on Exchange 2007. I made the decision to concentrate on 2007 for this post because the likelihood of rebuilding 2007 indexes are significantly higher when contrasted with 2010 (unlike 2007, 2010 has the ability to reseed Content Indexes from Healthy redundant sources thus making the need to perform complete index rebuilds much more rare in multi-copy architectures). In the coming weeks Anatoly and I will release a supplemental post that provides the script reference for the Exchange 2010 version of the Content Index Rebuild Analyzer script as we accumulate corresponding examples for its usage.

Index Rebuild Analyzer Script

Internally at Microsoft the primary toolset we leverage when rebuilding Content Indexes is the IndexRebuildAnalyzer script. This script was authored by Anatoly and I specifically for establishing Content Index rebuild baselines. As previously noted, there are two versions of this script; an Exchange 2007 version; and an Exchange 2010 version. To calculate your statistics properly always use the script that corresponds to the Exchange Mailbox Database version whose index will be rebuilt. The IndexRebuildAnalyzer script generates two types of metrics depending on the mode of operation passed by the operator. Internally we refer to these two modes as “pre-rebuild metrics” and “post-rebuild metrics” (all properties documented within the Script Reference section below).

Although this script is primarily leveraged to track Content Index rebuild operations, Exchange Administrators could certainly utilize the script in “pre-mode” to obtain Point-In-Time (PIT) statistics for various mailbox centric purposes (e.g. “Number of Mailboxes”, “Number of Items in a Database”, “Average Message Size” for your entire organization, etc.). This might, for example, provide you with additional optics and capabilities for user trending should the tool be leveraged regularly depending on your own business requirements or needs.

The E2K7_IndexRebuildAnalyzer.ps1 script parameters as well as examples for usage can be obtained by passing the -Help parameter in the PowerShell session prior to script execution.

The following table outlines each parameter:

Database Metrics Headers

Pre-Rebuild

As defined above the “mode of operation” of the script is determined by the presence or absence of the -PostRebuild switch. To obtain pre-rebuild metrics the -PostRebuild switch would not be utilized. When the script is instantiated in pre-mode the following Headers will be presented with corresponding metrics:

Post-Rebuild (Utilizing -PostRebuild Parameter)

When the -PostRebuild switch is utilized IndexRebuildAnalyzer will attempt to calculate throughput metrics for Content Index rebuild operation(s). It does this by parsing the Application Event Log to obtain both the start (denoted by Event ID 109) and completion time of the rebuild (denoted by Event ID 110) for each Mailbox Database on the Mailbox Server. To calculate post-rebuild metrics successfully the complete Event ID pair must be present in Event Viewer for each Mailbox Database whose corresponding Content Index was reset. In situations where the Event ID pair is not available the script will be unable to calculate the statistics. In these situations the string “NoEventsFound” will be returned. The most common reasons why this string would be returned are:

• The Content Index rebuild operation for a given or set of databases has not completed.
• The Application Event Log wrapped or was cleared (best practice is to set the Maximum Log Size value to the highest sustainable value.
• The Mailbox Databases reporting “NoEventsFound” did not recently have their Content Indexes reset (hence the absence of the Event ID pair from the Event Log). By leveraging the -CSVFile option and Excel these strings can be easily filtered out of the result set. I will address and provide examples for filtering in Step-5 of the framework.

All “pre-rebuild” headers and metrics are also calculated whenever the -PostRebuild switch is passed for script execution. Use of the -PostRebuild switch will include the addition of the following headers and metrics:

Conclusion

You can download the Exchange 2007 Index Rebuild Analyzer Script here.

In part 2 of this series, I will discuss the search rebuild framework and in part 3 of this series, I will discuss what we have seen to date within Microsoft.

Eric Norberg
Service Engineer
Office 365-Dedicated

As the original blog post about OCAT release was quite popular, we wanted to let you know that we have just released the version 2.0 of the tool, with many features that you have been asking for!

If you have OCAT v1 installed, the installation of OCAT v2 will automatically remove v1 for you.

Here are the key features added in OCAT version 2:

• Automatic downloading of new detection rules

As we create new rules to detect issues or to collect additional information about your Outlook profile, we will post a new updated rule file to the Internet. With the default OCAT version 2 configuration, OCAT will automatically check for a new rule file and prompt you to update OCAT if a new file is found.

• Automatic downloading of OCAT installation files

As we update and fix the core OCAT application, we will post a new Windows Installer package file (.Msi) to the Internet. With the default OCAT version 2 configuration, OCAT will automatically check for a new installer file and prompt you to update OCAT if a new file is found.

• Addition of the CalCheck tool

The Calendar Checking Tool (CalCheck) for Outlook is a command-line program that checks Outlook Calendars for problems. This tool is now included in OCAT to scan for and report on any known problems with items on your primary Calendar.

• Addition of new detection rules

To greatly enhance the list of known issues detected by OCAT, approximately 75 new rules were added to OCAT version 2.

• Improved support for Outlook 2003

OCAT v1 supported Outlook 2003 using offline scans, however, most people did not realize it because of the error shown when you tried to run an Online scan.

• Command-line version of OCATcmd.exe

OCAT version 2 includes a command-line version of OCAT (OCATcmd.exe) that administrators can use to scan computers in their organization. Please see the OCAT v2 Supplemental Information Download.docx file for details on how to use the command-line version of OCAT.

The last bullet item is also what allowed us to include OCAT v2 in the Outlook Baseline diagnostic.

If you want to submit feedback or improvement suggestions for OCAT, click the feedback link in the See also section in the left panel of OCAT. The link opens a new email message addressed to OCATsupp.

Greg Mansius

In Part 1 of this series, I explained the E2K7_IndexRebuildAnalyzer.ps1 script.  In this article, I will discuss the Search Rebuild Framework that Anatoly Girko and I developed. This framework was originally designed to provide our internal operations staff with a set of comprehensive validation steps and progress indicators they could leverage when performing rebuilds of content indexes.

Stage 1: Pre-Rebuild Statistics Gathering

In our environment whenever a decision is made to rebuild an Exchange Mailbox Database’s Content Index files the Operator begins by first calculating pre-rebuild statistics for the impacted store. These statistics are always written to CSV for documentation purposes and are eventually inserted into our historical data collection set. However as discussed, the use of the -CSVFile parameter) is optional. In situations where the -CSVFile parameter is not passed the corresponding output will be written to the shell console window. For best readability you should adjust both the Screen Buffer Size Width and Window Size Width of the console to properly accommodate for all of the various headers and metrics that will be outputted. This will allow you to read the values more easily in the console session. From there I typically “pipe” the output to Format-Table (ft) with the -AutoSize (-a) parameter.

Examples

Console Example:

.\E2K7_IndexRebuildAnalyzer.ps1 -CMS NA1-ERICNOR-1 | ft -a

Output:

CSV Example:

.\E2K7_IndexRebuildAnalyzer.ps1 -CMS NA1-ERICNOR-1 -CSVFile c:\ericnor\NA-1ERICNOR-1_PreRebuild.csv

Output:

The pre-metrics are then contrasted against the historical data collection and an average time to complete rebuild estimate is derived. We take into account the regional location of the mailbox database and corresponding end user mailboxes. Based upon geography of the store and the estimated time to complete we schedule the rebuild work for a date and time where user activity on that store is minimalized.

Stage 2: Reset Content Index for Impacted Mailbox Database(s)

The operators in our environment then proceed to reset the Content Index for the Mailbox Database utilizing either of the techniques documented in How To Rebuild the Full-Text Index Catalog.

In the examples that follow I will be resetting the Content Index files for two Mailbox Databases within my environment. These two stores also happen to have the largest Mailbox Counts, EDB File Sizes and collective Database Item Counts on the NA1-ERICNOR-1 Clustered Mailbox Server:

Reset of Content Index Files:

Stage 3: Validating Search Indexer Has Detected Full Crawl Required

After the existing Content Indexing files have been removed from the file system and the Microsoft Exchange Search Indexer service has been subsequently restarted, it is the responsibility of the MonitorAndUpdateMDBList thread to determine the current state of the Content Indexes for all Mailbox Databases on the Mailbox Server that are enabled for Content Indexing. Once the MonitorAndUpdateMDBList thread has determined the Content Index Health State for each Mailbox Database it places each Mailbox Database’s Health State value into memory. If the value of the Content Index Health State is equal to “New”, the Microsoft Search Indexer Service has determined that a complete Full Crawl is required to bring the Content Index files into a Healthy state (“Healthy” being the point where the index is being kept up to date via Store Notification). It’s at this time that the Exchange Search Indexer service initiates a Full Crawl operation against the impacted Mailbox Database.

The time that the Full Crawl operation actually begins is denoted in the Application Event log by Event ID 109.

To ensure that all Full Crawl operations have begun on a system the operator performing the work will validate the presence of Event ID 109 for each Mailbox Database whose Content Index was previously reset in Step-2.

Example

As illustrated in the previous example the Content Index files for NA1-ERICNOR-1-DB1 and NA1-ERICNOR-1-DB18 were reset utilizing the ResetSearchIndex script. To validate that the Microsoft Exchange Search Indexer service has begun Full Crawl operations, a unique Event ID 109 should be present for each Mailbox Database on the Mailbox Server. This appears to be the case:

Event Type: Information
Event Source: MSExchange Search Indexer
Event Category: General
Event ID: 109
Date: 5/10/2012
Time: 2:22:19 PM
Computer: NA1-ERICNOR-1-A
Description: Exchange Search Indexer has created a new search index and will perform a full crawl for the Mailbox Database NA1-ERICNOR-1\NA1-ERICNOR-1-SG1\NA1-ERICNOR-1-DB1 (GUID = 5a1122be-b9bb-4d5b-853a-e689b1ea1129).

Event Type: Information
Event Source: MSExchange Search Indexer
Event Category: General
Event ID: 109
Date: 5/10/2012
Time: 2:22:20 PM
Computer: NA1-ERICNOR-1-A
Description: Exchange Search Indexer has created a new search index and will perform a full crawl for the Mailbox Database NA1-ERICNOR-1\NA1-ERICNOR-1-SG18\NA1-ERICNOR-1-DB18 (GUID = 2faba54d-1699-441e-8ac8-1a136d0b7b16).

Note: As opposed to inspecting the Event Log visually another viable technique would be to simply leverage the Get-EventLog cmdlet and write the Start events to CSV. Example:

Get-EventLog "Application" | Where-Object {$_.EventID -eq 109} | Select-Object EventID,TimeGenerated,Message | Export-CSV -NoTypeInformation -Path c:\Search_StartEvents.csv

Stage 4: Validating Search Indexer Progress and Completion

Having validated that Full Crawl operations have begun (via presence of Event ID 109) the Operator monitors the overall rebuild progress via System Monitor. Specifically the following Objects and Counters would be monitored:

• MSExchange Search Indexer\Number of Databases Being Crawled
• MSExchange Search Indexer\Number of Indexed Databases Being Kept Up-to-Date by Notifications
• MSExchange Search Indices\<DB Instance Undergoing Crawl>\Full Crawl Mode Status
• MSExchange Search Indices\<DB Instance Undergoing Crawl>\Document Indexing Rate
• MSExchange Search Indices\<DB Instance Undergoing Crawl>\Number of Mailboxes Left to Crawl
• MSExchange Search Indices\<DB Instance Undergoing Crawl>\Number of Recently Moved Mailboxes Being Crawled
• MSExchange Search Indices\<DB Instance Undergoing Crawl>\Number of Outstanding Batches
• MSExchange Search Indices\<DB Instance Undergoing Crawl>\Throttling Delay Value

By reviewing the MSExchangeSearchIndexer object an Operator can easily ascertain how many Content Indexes on the Server are up to date and how many are actively being crawled. When a Mailbox Server is completely up to date the value for the “Number of Databases Being Crawled” counter will always be equal to 0, and the “Number of Indexed Databases Being Kept Up-to-Date by Notifications” counter value will be equal to the number of Mailbox Databases that are enabled for Content Indexing on the server.

In my example, I have eighteen Mailbox Databases total on my mail server and two of those databases are currently undergoing Full Crawl. Thus, the value for “Number of Databases Being Crawled” should be equal to 2, and the value of “Number of Indexed Databases Being Kept Up-to-Date by Notifications” should be equal to 16 which they are:

As individual Mailbox Databases complete Full Crawl you will notice specific changes to the various object counters in System Monitor.

At the MSExchange Search Indexer level:

• The “Number of Databases Being Crawled” counter is decremented by 1
• The “Number of Indexed Databases Being Kept Up-to-Date by Notification” is incremented by 1.

At the Mailbox Database Index/indices level:

• The value for “Full Crawl Mode Status” on the specific database has been decremented to 0 (which reflects the new value for Content Index Health state as determined by the MonitorAndUpdateMDBList monitor thread).
• The “Number of Mailboxes Left to Crawl” should reflect a value of 0.
• Although not specifically related to Full Crawl rebuild operation per se, the value for the “Number of Recently Moved Mailboxes Being Crawled” counter should also be 0 for each index. As Exchange Mailboxes are successfully moved between Exchange Mailbox Databases (provided the target is enabled for Content Indexing), the Search Notifications on the target database will be temporarily suspended. The Exchange Search Indexer Service does this so that it can perform 1-off crawls of the recently moved mailboxes to bring the Master Index completely up to date. Once 1-Off crawling has been completed, Store Notifications are resumed. Point here is that although Full Crawl has technically completed, it is still possible that 1-off crawling might be occurring if mailbox moves were occurring at the same time as Content Index rebuild. Provided this counter is equal to 0, all crawling activity occurring on the Mailbox Database will be definitively complete. Thus, it should be validated as such.

On an Exchange Server where all Content Indexes are completely up to date you would expect to see the following:

• “MSExchange Search Indexer\Number of Databases Being Crawled” is equal to 0.
• “MSExchange Search Indexer\Number of Indexed Databases Being Kept Up-to-Date by Notifications” is equal to the number of Mailbox Databases enabled for indexing on the Mailbox Server.
• “Full Crawl Mode Status” for each Exchange Mailbox Database instance on the Mailbox Server is equal to 0.
• “Number of Mailboxes Left to Crawl” for each Exchange Mailbox Database instance on the Mailbox Server is equal to 0.
• “Number of Recently Moved Mailboxes Being Crawled” for each Exchange Mailbox Database instance on the Mailbox Server is equal to 0.

In my example, all of these values are True thus I can reasonably assume that the indexes have been rebuilt successfully and that from a Content Index perspective that the server is completely Healthy:

Once all Content Indexes appear to be up to date via System Monitor the Operator performing the work would proceed to review the Application Event Log and validate the presence of Event ID 110 which is the Microsoft Exchange Search Indexer completion event for Full Crawl. Similar to Event ID 109 there will be a unique 110 event entry for each Mailbox Database that has completed Full Crawl:

Event Type: Information
Event Source: MSExchange Search Indexer
Event Category: General
Event ID: 110
Date: 5/10/2012
Time: 5:39:47 PM
Computer: NA1-ERICNOR-1-A
Description: Exchange Search Indexer completed a full crawl (indexing) of Mailbox Database NA1-ERICNOR-1\NA1-ERICNOR-1-SG1\NA1-ERICNOR-1-DB1 (GUID = 5a1122be-b9bb-4d5b-853a-e689b1ea1129).

Event Type: Information
Event Source: MSExchange Search Indexer
Event Category: General
Event ID: 110
Date: 5/10/2012
Time: 5:11:47 PM
Computer: NA1-ERICNOR-1-A
Description: Exchange Search Indexer completed a full crawl (indexing) of Mailbox Database NA1-ERICNOR-1\NA1-ERICNOR-1-SG18\NA1-ERICNOR-1-DB18 (GUID = 2faba54d-1699-441e-8ac8-1a136d0b7b16).

Note: As opposed to inspecting the Event Log visually another viable technique would be to simply leverage the Get-EventLog cmdlet and write the Completion events to CSV. Example:

Get-EventLog "Application" | Where-Object {$_.EventID -eq 110} | Select-Object EventID,TimeGenerated,Message | Export-CSV -NoTypeInformation -Path c:\Search_CompletionEvents.csv

Stage 5: Collect Post-Rebuild Metrics

At Stage-5, it is assumed that the operator has validated the completion of Full Crawl for each Mailbox Database. It’s at this point that the operator would collect Post-Rebuild metrics to determine throughput characteristics for each Mailbox Database that underwent crawl.

To collect these metrics the E2K7_IndexRebuildAnalyzer script is utilized leveraging the -PostRebuild switch.

As previously mentioned -PostRebuild parses the Application Event logs for the presence of both Event ID 109 and Event ID 110. In the case of Continuous Cluster Replication the Application Event Log for each node in the CCR Cluster is evaluated. If the script can locate these Event IDs it will calculate the total time (in various time increments) required to complete Full Crawl against each Mailbox Database. It will then return to the operator the throughput characteristics for each unique rebuild operation.

It is once again worth noting that all Mailbox Databases on the Mailbox Server will be evaluated irrespective of whether or not all Mailbox Databases had their Search Indexes rebuilt. Additionally if instantiated without the -CSVFile option the result set will be outputted to the console window. When calculating Post Rebuild statistics I would strongly encourage leveraging -CSVFile as it makes reporting, sorting, filtering, and pivoting very easy when Excel is leveraged.

Example

.\E2K7_IndexRebuildAnalyzer.ps1 -CMS NA1-ERICNOR-1 -PostRebuild -CSVFile c:\ericnor\NA1-ERICNOR-1_PostRebuild.csv

After E2K7_IndexRebuildAnalyzer has completed execution the CSV file can then be inspected. Review of the CSV shows that all Exchange Mailbox Databases on the Mailbox Server were processed. The string “NoEventsFound” is being returned for many of the Exchange Mailbox Databases because the Search Indexer Event ID pair combination cannot be located. In the example case, there are 16 Mailbox Databases reporting “NoEventsFound” and two Mailbox Databases that have valid post rebuild metrics:

This result set can be further refined by applying a simple text based filter within Excel. By applying filtering to any of the post rebuild column headers and unticking the “NoEventFound” string, only databases with valid post rebuild metrics will be displayed:

The result of applying this filter on the example result set now shows only NA1-ERICNOR-1-DB1 and NA1-ERICNOR-1-DB18 (e.g. the Mailbox Databases that had their Content Indexes reset). It is also clear that post rebuild metrics were successfully ascertained as there are valid values for the various post rebuild headers:

Post String Filter Application:

Stage 6: Test-ExchangeSearch Validation

Stage 6 in the Rebuild Framework validates that every mailbox homed on Mailbox Databases that had their Content Indexes reset can now return valid search responses. This goal can be accomplished by leveraging the core functionality contained within the Test-ExchangeSearch cmdlet. Final validation is complete only when all mailboxes return a True response to the cmdlet.

Example:

Get-Mailbox -Database NA1-ERICNOR-1\ NA1-ERICNOR-1-SG1\ NA1-ERICNOR-1-DB1 | Test-ExchangeSearch | ft -a

Get-Mailbox -Database NA1-ERICNOR-1\ NA1-ERICNOR-1-SG18\ NA1-ERICNOR-1-DB18 | Test-ExchangeSearch | ft -a

This process will take a significant amount of time to complete depending on the number of mailboxes contained within the database. It’s also worth noting that when performing bulk operations with Test-ExchangeSearch that the ResultFound and SearchTime properties will only be visible in the console window after all mailboxes have been processed (regardless of True or False results). In some cases it may also make sense to store all results in CSV for documentation purposes.

Any end user mailbox reporting False to the Test-ExchangeSearch test are treated as 1-off issues and remediated accordingly.

Stage 7: Analysis of Post Rebuild Metrics

For readability purposes I’m going to display the various metrics in table format as opposed to displaying the metrics natively in Excel columns. As previously noted there were two Exchange Mailbox Databases on the NA1-ERICNOR-1 Clustered Mailbox Server that had their Content Indexes rebuilt: NA1-ERICNOR-1-DB1 and NA1-ERICNOR-1-DB18

Mailbox Metrics Post Rebuilds:

Mailbox Database Rebuild Metrics:

The various rebuild metrics and pre-rebuild metrics are then added to the historical data collection set so that they can contribute to the historical averages for future rebuild exercises.

Conclusion

In the second post of the series, I covered the stages of the Search Rebuild Framework.  My next and final post will cover the observed averages we have seen within our deployments at Microsoft.

Eric Norberg
Service Engineer
Office 365-Dedicated

Update 07/03/2012: Expanded the Workaround section to include more details.

We wanted to bring to your attention an issue that we have been seeing recently in CSS where many mailboxes on the same mailbox database become quarantined seemingly for no reason. Upon inspecting the Application log, you will see many 10018 events with text containing to the following information.

Log Name: Application
Source: MSExchangeIS
Event ID: 10018
Task Category: General
Level: Error
Description:
The mailbox for user <guid>: /o=Contoso /ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=UserMailbox has been quarantined. Access to this mailbox will be restricted to administrative logons for the next 6 hours.

Additionally, the following event ID 5410 will appear in the Microsoft-Exchange-Troubleshooters/Operational event log for each mailbox quarantined by the database space troubleshooter:

Log Name: Microsoft-Exchange-Troubleshooters/Operational
Source: Database Space
Event ID: 5410
Level: Warning
Keywords: Classic
Description:
The database space troubleshooter quarantined mailbox <guid> in database <DBName>.

The key here is the last sentence. “The database space troubleshooter”. In environments where System Center Operations Manager has been deployed, a Monitor is in place by default to check free disk space for database and log volumes. This monitor uses the PowerShell script Troubleshoot-DatabaseSpace.ps1, which ships with Exchange 2010 by default. To read more about the Troubleshoot-DatabaseSpace.ps1 script, refer to the following TechNet article:

Manage Database Log Growth by Using the Troubleshoot-DatabaseSpace.ps1 Script in the Shell
http://technet.microsoft.com/en-us/library/ff477617.aspx

The System Center team recently released Service Pack 2 for the Exchange Management Pack. Among other things, this adjusted a timeout value for running the script. Prior to the Service Pack, the timeout value was 300 seconds, and the monitor was set to fire every 5 minutes. This meant that in many large organizations, the script was virtually guaranteed to timeout, causing it to not complete. After installing the upgrade, the new timeout value is 1200 seconds. In addition, the monitor now runs once per hour instead of every 5 minutes. The effects of this change are that the troubleshooter is now reliably executing code that it previously did not execute prior to the SP2 update. This has caused the troubleshooter to find a bug in the Information Store perfmon counter used by the troubleshooter to determine rate of log byte generation for the database (the perfmon rate can exceed 1.0E+19 Bytes/Hr). When the estimated hourly rate of log generation exceeds the capacity of remaining free disk space on database or log volume, the troubleshooter begins to quarantine mailboxes to stop log generation from consuming all free space and causing the database to dismount. The troubleshooter appears to hit this condition when the perfmon counter has a bogus value (the perfmon counter is updated once per minute and should not have a bogus value for more than 1 min)… so it doesn’t occur frequently but does occur with moderate probability.

What does this mean for you?

Well, as documented in the TechNet article, the main function of the script is to keep track of the log generation rate, and to check free disk space for the database and log volumes. When the script runs, it uses a store perfmon counter to determine the rate of log generation, and calculates whether the current rate will cause the disk to run out of space within the threshold of hours (default is 12 hours), and if so, it will optionally start quarantining mailboxes (up to 150 at a time). The script will then also optionally quarantine mailboxes when the percentage of free disk space goes below a specified value. The default free disk space percentage value used is 25%. In order to quarantine mailboxes when either of these conditions are met, the –Quarantine parameter must be passed to the script.

The monitor utilized by SCOM 2007 and higher uses the Quarantine parameter by default, and there is no option supported option to change this because the Management Pack is sealed. This means that if the free disk space for either the database or log volume for a database goes below 25%, AND if the log generation rate is calculated to consume remaining disk space within the next 12 hours, the heaviest users will be quarantined. When there are many mailboxes on a database, this can mean many mailboxes can be quarantined in a short period of time.

What can you do?

Obviously, having many mailboxes quarantined is not a desired behavior, as it causes outages for those users, though it is still better than the alternative of having the entire database dismount due to lack of disk space, which will cause an outage for all users on that database. Although quarantined mailboxes can be manually released by removing that mailbox GUID from the registry, this is not an optimal solution, as the next time the monitor runs, the same mailboxes may end up being placed into quarantine again.

Quarantined mailboxes are identified in the following location:

Hkey_Local_Machine\SYSTEM\CurrentControlSet\Services\MSexchangeIS\Servername\Private-<D Bguid>\Quarantined Mailboxes\ {Mailbox GUID}

We want you to know that we are aware of this situation and are working to implement a fix. While we work to identify the best way to solve this going forward, we want to let you know of a workaround that can be implemented to prevent mailboxes from being quarantined. In the meantime, to help prevent more customers running into this problem, we have temporarily removed the Exchange 2010 SP2 Management Pack from the Download Center.

Workaround

Create an override to disable the monitors that check the free disk space. This will prevent the Troubleshoot-DatabaseSpace.ps1 file from being run at all.

When creating overrides, it is best practice to place them into a new management pack specifically for customizations within System Center Operations Manager. This allows you to quickly and easily manage your overrides or other custom settings. If you do not already have a management pack for this, you can create one by following the steps below:

1. In the Operations Console, click the Administration button. In the Administration pane, right-click Management Packs and then click Create Management Pack. The Create a Management Pack wizard displays.
2. In the General Properties page, type a name for the management pack in Name, the correct version number in Version, and a short description in Description. Click Next and then Create.

Once you have designated a management pack for the overrides, you will want to create them for the 4 monitors listed here:

In System Center Operations Manager, go to the Authoring module, and then Monitors under Management Pack Objects. The monitors can be disabled by setting an Override and choosing to disable the Monitor for all objects of class Database Copy DB Logical Disk Space (as shown below):

Check the box next to Enabled, and then modify the override value by selecting False:

On the same Override dialog Properties box, be sure to select the management pack designated for customizations.

At this time, we feel that if you are being impacted by mailboxes being quarantined, the option to disable the Monitors is the only available workaround. We recognize that this will potentially prevent administrators from receiving alerts when disk space is low, but feel this is the best option to prevent mailboxes from being quarantined until a fix is released.

Note that when Database disk space is low, an alert will still be received when the transaction log drive coexists with database since this is generated by a different rule that is purely perfmon based.

Ben Winzenz, Kevin Carker

In Part 1 of this series, I explained the E2K7_IndexRebuildAnalyzer.ps1 script and in Part 2, I discussed the Search Rebuild Framework that Anatoly Girko and I developed. Before concluding this series I did want to provide a series of graphs as well as an “observed averages” table that illustrate the rebuild characteristics we have observed since the inception of the framework. Hopefully this allows for better conceptualization as well as empowering you to make better estimates when calculating your own rebuild rates..

Observed Averages to Date within Microsoft

Anatoly and I went back and forth on how to best to present this. As one might imagine there are an infinite number of possibilities for presentation. We decided to scope the graphs and table to the message size that most Exchange Storage Architects design to: 150 KB per mail item. We then performed a secondary filter on Mailbox Count and only took into account Mailbox Databases within our data collections that had 100 or more active mailboxes to build the averages below. Once done, we removed the 10% best performing and 10% worst performing rebuild operations from within our collection to derive the averages used to build the graphs and tables.

Note: In the various graphs and table to follow Averages Mailbox Sizes in several range increments are noticeably missing. This data was not overlooked or purposefully omitted. Absence of statistical data for these ranges is due to the fact that no valid data exists in our historical collections. Put another way we have never performed Content Index rebuild operations and/or collected Post-Rebuild metrics for databases where the Average Mailbox Sizes for end user mailboxes were in the following ranges:

• 1700-1799 MB
• 1800-1899 MB
• 2000-2099 MB
• 2100-2199 MB

Graphs

We will be presenting four Excel Pivot Charts that reflect the throughput characteristics we have observed to date based upon the filtered collection as described above. These Pivot Charts are meant to illustrate the relationship that exists between various properties as they occur within and around the Mailbox Store (e.g. mailbox count, item count and EDB file sizes) and contrast them with the historical throughput times required to complete Full Crawl against Mailbox Stores with similar characteristics.

Graph 1

The view contained within Graph-1 specifically depicts the relationship between Mailbox Counts per Database, the relative size of the Mailbox Databases in gigabytes and how this ultimately impacts the total time to complete Content Index rebuild of the Mailbox Stores in minutes.

This graph makes a clear argument that as the total number of active mailboxes on an Exchange Mailbox Database increases there will also tend to be a parallel relationship for increases to the EDB file size on the storage subsystem. This relationship subsequently has an impact on the overall time to complete a Full Crawl of a Content Index. This is really just a fancy way of saying that: with more active mailboxes typically comes more mail items; with more mail items comes a larger EDB file size on disk; the larger an EDB file size is on disk the longer it will “usually” take to rebuild a Content Index. The only situation where this hypothesis will never hold true is in the case of a Mailbox Database that had a large amount of whitespace present in the file. In such a case the overall time to complete a Content Index rebuild will be noticeably faster than anticipated. This anomaly situation has been seen within the environments we support but the subsequent statistics were removed from our collection by leveraging the filtering technique discussed above.

Graph 2

Graph-2 depicts the relationship that exists between the Average Mailbox Size (for mailboxes that exist on databases contained within the same filtered sample set) and how it impacts the Content Index rebuild throughput at the Mailbox Database level in Seconds per/mailbox.

This graph essentially restates the argument presented in Graph-1 albeit at the active mailbox level. Specifically as the active mailbox size averages increase so do the average number of mail items within those mailboxes. On average the more mail items within a mailbox the longer it will take the Search Indexer to complete crawl against a given mailbox which in turns impacts how long it will take to complete Full Crawl for all mailboxes within the database.

Graph 3

Graph-3 depicts the relationship that exists between the Average Mailbox Size (for mailboxes that exist on databases contained within the same filtered sample set) and how it impacts the Content Index rebuild throughput in Megabytes per/second.

Graph-3 builds on the initial hypothesis alluded to in Graph-2. Specifically it shows that as the Average Mailbox Size and Average Item Counts within a Mailbox Database increase there is a negative relationship with respect to Search Indexer throughput. Graph-3 shows this relationship in megabytes per second.

Graph 4

Graph-4 depicts the relationship that exists between the Average Mailbox Size (for mailboxes that exist on databases contained within the same filtered sample set) and how it impacts the Content Index rebuild throughput in Items per Second (based upon Average Message Size of 150KB):

As was the case with Graph-3, Graph-4 shows the negative performance impact with respect to throughput in Items per/second.

Observed Averages Table

To present the table we utilized the same filtered set (described above and presented in the graphs) but elected to create focused averages based upon Average Mailbox Size. These rows are subsequently delineated as independent rows in increments of 99 megabytes. The throughput characteristic for each row represents the aggregate averages for all similarly sized databases that have completed rebuild operations in our collection. Specifically where the Average Message Size was 150KB and the Average Mailbox Size for all active mailboxes on those databases were within the ranges defined by Column-A.

The historical averages presented in this table (at least to me) produce three potential ways of estimating Content Index rebuild times:

• A “Historical Average” could be implemented based upon Average Mailbox Size where the Average Message Size for items residing within those mailboxes is 150KB. Because we have large amounts of historical rebuild data in our collection we choose to leverage this average. We derive our estimate by determining the Average Mailbox Size value via “pre-rebuild” metrics and compare this to the historical average. We then take the composite average for Rebuild: Seconds per/Mailbox and then multiply that figure by the number of mailboxes on the database which require crawling to determine the total time necessary to complete.
• An “Organizational Average” could also be established based upon Average Message Size irrespective of the number of items and average size of the mailboxes across the organization (that Organizational Average is provided in the Averages Row above).
• A composite average between historical average and organizational average.

For example, if I have a Content Index that needs to be rebuilt for a database whose users have an aggregate Average Mailbox Size in the 500-599 MB range and assuming the Average Message Size is 150KB, if that database has 200 users on it, I could derive the estimate in one of three potential ways:

The Historical Averages Table:

200 mailboxes * 63 seconds = 12,600 seconds total. This equates to 210 minutes or roughly 3.5 hours to complete Full Crawl.

The “Organizational Average”:

200 mailboxes * 108 seconds = 21,600 seconds total. This equates to 360 minutes or roughly 6.0 Hours to complete Full Crawl.

Composite Average (Average of “Historic” + “Organizational”):

3.5 + 6.0 = 9.5 Hours

9.5 / 2 = 4.75 Hours

Conclusion

The total time it takes to rebuild a Content Index will always be variable because mail populations and items therein are always variable too. When rebuilding Content Indexes the most accurate and robust estimates will always come from leveraging historical averages. I also wanted to mention that when I/we make decisions to rebuild Content Index internally at MSFT we do our best to schedule them for “least user impact” time intervals. However, our implementations are global so it is more or less impossible to completely eliminate end-user impact. The best you can hope for is to minimize the impact on the surface area. Furthermore, within our data collections we don’t factor in Search Indexer Throttling Delays. Any and all Search Indexer Rebuild Throttle Delays are handled and understood within the moment and are representative within the individual tickets as they are presented to operations. By leveraging the filtering techniques used throughout this post you insulate your figures from those negative averages (same is true for “overly high throughput” rebuild operations) making your overall estimates considerably more accurate.

If you are the type of personality that that has a propensity for betting with averages I fully stand by our table. If a more exact science is required, I would suggest implementing a framework such as the one we describe within this post series.

We hope you find this post series useful, and better yet, have learned something new along the way!

Happy Trails!

Eric Norberg
Service Engineer
Office 365-Dedicated

We are happy to announce improvements to the Remote Connectivity Analyzer CAPTCHA experience.  Several of you told us how frustrating the previous experience was to use.  And we agree… it was very annoying to have both password fields get blanked out if you got the challenge incorrect.  While we can't remove this "feature" completely, we hope the enhancements will make a big difference.

The latest version (V1.4) of the Remote Connectivity Analyzer has the following improvements:

• We are using a new CAPTCHA service provided by an internal team!
  • The challenge is NOT case sensitive, so it doesn't matter if you type upper or lower case letters.  We also note this on the web page.
  • The CAPTCHA challenges will not include hard to distinguish letters/numbers.  For example 2 and Z or O and 0.
  • If you get the challenge wrong, the password entries will not be removed.
  • Once you enter a correct response to the challenge, you will be verified for a set amount of time.  This means you will not see additional CAPTCHA challenges until the timeout period expires.

(We can hear the cheers already!!)

A few more:

• The inbound SMTP test now inserts the IP address of the user performing the test into the test email message. The IP is also inserted into an SMTP Header (X-Originating-IP).
• Fixed an issue in the Sender-ID test where certain DNS responses while evaluating the "exists" mechanism were incorrectly being treated as a TempError
• The outbound SMTP Sender-ID tests now conform to the RFC specified limit of ten DNS-based mechanisms that can be used during the evaluation of the SPF record.
• And more! Read about it here.

Here's a fun 1 minute video which demonstrates some of these improvements.

Enjoy!

Brad, Nicole, and Shawn

We eat our own Dogfood in Exchange and talk about it often so we wanted to share some very special insights with you on the process.  What is Dogfood? It's a slang term we use for using our own software, including alpha and beta varieties internally at Microsoft prior to shipping our products, in order to validate scenarios at scale, in a real world environment with real users who need to be productive. This is an important part of our ship cycle since it not only allows us to work out production-level bugs or issues but also validate deployment guidance, documentation, support paths and align scenarios with other partner products such as Windows, Office, SharePoint and Lync. 

In the Exchange world, this means that we deploy extremely early version of Exchange in production starting with a few mailboxes, hundreds of mailboxes, then working with MSIT to move thousands of mailboxes - and eventually deploying to the entire company. 

This also means we actually dogfood Exchange Online as well and therefore we maintain an ongoing hybrid Exchange deployment at Microsoft. We’ve recently posted some TechNet Radio podcasts on hybrid on the road to TechEd as well as publishing some Exchange Deployment Assistant updates and many of these tips, tricks and processes were validated through the magic of dogfood. Check out this IT Showcase Whitepaper Microsoft IT Plans for Hybrid Deployment of Microsoft Office 365 Messaging which covers how we deployed hybrid early on, what important lessons we learned, and how this helps Microsoft IT beyond our usual dogfooding validation goals.

Exchange folks are crazy about Dogfood! But don't just take my word on it… Check out this ESE Access to Exchange video to get a taste of our Dogfooding sense of adventure.

Comment back and please let us know what you think.

Ann Vu

* But Were Afraid to Ask

In Part 1 and Part 2 of this series we looked at the fundamentals of Exchange backups using VSS, and the flow of an active DAG database backup.

In Part 3 we break down how a passive DAG database copy undergoes a full backup. The Exchange Writer responsible for passive copy backups doesn't run in the Information Store Service, but rather as part of the MS Exchange Replication Service. Among other functions, this service coordinates the backup process between the passive copy node and the active copy server. Similar to the backup of an active database described in Part 2, this post describes the backup of a passive database copy of DB1, hosted on server ADA-MBX1. The active mounted database copy is on ADA-MBX2, and again, a non-persistent copy-on-writer (COW) snapshot is utilized by the backup solution:

(please click thumbnails for full size version of graphics in this post)

The first steps to back up a passive database copy are about the same as for an active one. The backup application gets the metadata for DB1 from the Exchange Writer, but again, the writer is running in the MS Exchange Replication Service. A new writer instance GUID is generated which will persist throughout the job, as with an active database backup.

Event 2021 indicates that the backup application, or VSS requestor, has engaged the Exchange Writer. It will appear numerous times throughout the backup as different components are read from metadata, such as log and database file locations.

Events 2110 and 2023 indicate that the backup application has requested a particular set of components to back up, and the backup type.

The replication service for the passive copy’s server signals the active copy server that a backup is in progress. Events 910 and 210 on the active copy server, as well as 960 on the passive copy server, signify two things: First, they establish which server backing up a passive copy of the database; Second, the STORE service on the active copy server has marked the database with “backup in progress” in memory and acknowledges that the surrogate backup will proceed. Once this occurs it is not possible to backup the database again until either the current surrogate backup completes, or the “backup in progress” status is otherwise cleared.

Events 2025 and 2027 are generated when the replication writer prevents the replication service from writing logs copied from the active copy server to the local disk. Replay of logs also stops, thereby keeping the contents of the database files unchanged. At this point writes of data for the database getting backed up are “frozen”. VSS can now create the snapshots in shadow storage for each disk specified in the metadata.

VSS creates snapshots of disks D: and E:. Once these complete it signals the Exchange Writer, which in turns allows the replication service to resume log copy and replay. Events 2029 and 2035 are generated when the “thaw” is completed and normal disk writes are allowed to continue.

Once the snapshots are created the backup application can copy blocks of data through VSS, which transfers blocks of data from shadow storage if they’ve been preserved due to a change, or from the actual disk volume if they haven’t. The replication service writer waits for the signal that the transfer of data is complete. This flow of data is represented by the purple arrows, which in this case indicates data getting copied out of the snapshots in storage, through I/O of the Exchange server, and on to the backup server.

When the files necessary for backing up DB1 are safely copied to backup media, the backup application signals VSS that the job is complete. VSS in turn signals the replication writer, and Exchange generates events 963 and 2046 on the passive copy server. The replication service then signals the Information Store service on the active copy server that the job is done, and that log truncation can proceed if all necessary conditions are met. The active copy node generates events 913 and 213 signaling that the surrogate backup is done, and that the database header will be updated with the date and time of the backup.

Events 2033 and 2037 signal the end of the backup. The active copy node flushes and rolls the current transaction log containing database header updates. That log is then shipped and made eligible for replay according to schedule so that the passive database copy is marked with the new header information at the earliest available time. Log truncation also proceeds if possible. In this case the snapshots are destroyed, and normal operations continue.

For more on the subject of this series here are some more great references:

Volume Shadow Copy Service
http://technet.microsoft.com/en-us/library/ee923636(WS.10).aspx

Exchange VSS Writers
http://msdn.microsoft.com/en-us/library/bb204080.aspx

Overview of Processing a Backup Under VSS
http://msdn.microsoft.com/en-us/library/aa384589(VS.85).aspx

Backup Sequence Diagrams
http://msdn.microsoft.com/en-us/library/aa579076(v=exchg.140)

Troubleshooting the Volume Shadow Copy Service
http://technet.microsoft.com/en-us/library/ff597980(EXCHG.80).aspx

Jesse Tedoff

In Exchange 2010, we introduced the ability for discovery managers to search mailboxes across an Exchange 2010 organization using Multi-Mailbox Search. Along with Personal Archives, which allow you to eliminate the risk from PST files and store historical messaging data in Exchange, Legal Hold, which allows you to preserve data for eDiscovery, and Retention Policies, which allow you to manage email lifecycle, it has been one of the more popular features of Exchange 2010.

Multi-Mailbox Search required an Enterprise Client Access License (CAL) for each mailbox searched. We’ve heard your feedback on how you use this feature and the licensing requirements. Today we’re making a change to Exchange 2010 licensing so you’ll no longer require an Enterprise CAL for Multi-Mailbox Search. The Exchange Server Licensing page will be updated to reflect this.

Update 7/16/2012: We've updated the Exchange Server Licensing page.

Bharat Suneja

Last Monday, at an event in San Francisco, Steve Ballmer introduced the new, modern Office to the world. An exciting set of capabilities were showcased, including Windows 8 integration, Office as a subscription, and an enhanced social story.

Exchange is one of the cornerstones of communication and collaboration in Office. Over the past few years, we have seen significant changes in the way people communicate – a multitude of devices, an explosion of information, complex compliance requirements, social networks, and a multi-generational workforce. This world of communication challenges has been accompanied by a major shift towards cloud services.

The Exchange team has been hard at work in building a product and service that helps to address these challenges and better prepare our customers for the future of communications and productivity. We are excited to announce an important milestone on this journey – the preview of the next version of Exchange is now available!

With Exchange 2010, we redesigned the product with low-cost large mailboxes and cloud services in mind. We then extended this vision through Office 365 where tens of thousands of organizations with millions of users have accompanied us on this journey to the cloud. Now, customers can look forward to the new release of Exchange which offers a wide variety of exciting benefits:

• Remain in control, online and on-premises, by tailoring your solution based on your unique needs and ensuring your communications are always available on your terms.
• Keep the organization safe by protecting business communications and sensitive information in order to meet internal and regulatory compliance requirements.
• Increase productivity by helping users manage increasing volumes of communications across multiple devices.

As of last week the new version of Office, including Exchange and Office 365, has been made available to customers. I would encourage everyone to download the preview version of Exchange Server 2013 and try out the service preview of Office 365 Enterprise. As with pre-release versions, please use them to preview but not for production use.

Here are some of the great benefits you get with the next release of Exchange:

1. Reduced costs by optimizing for next generation of hardware

   Exchange is now optimized for 8TB disks, by reducing database IOPS by +50% and optimizing for multiple databases per volume to increase aggregate disk utilization while maintaining reasonable database sizes. Ever growing memory capacity is used to improve search query performance and reduce IOPS. All this allows you and your end users to have larger mailboxes at lower costs.

2. Significantly reduced operational overhead for high availability

   DAG management is simplified via automatic DAG network configuration, enhancements to DAG management cmdlets, support for multiple databases per disk, and enhancements to lagged copies. Auto-recovery capabilities – inherently built into DAGs – are now extended to the rest of Exchange and all protocols. Client-initiated, automatic recovery allows you to reduce recovery time for site failures from hours to under a minute.

3. Decrease the amount of time spent managing your system while maintaining control

   Exchange now provides a single, easy-to-use, Web-based administration interface – the Exchange Administration Center (EAC). Role based access control (RBAC) empowers your helpdesk and specialist users to perform specific tasks which are surfaced appropriately in the EAC – without requiring full administrative permissions. This streamlined and intuitive experience helps you manage Exchange efficiently, delegate tasks, and focus on driving your business forward.

   
   Figure 1: The Exchange Administration Center (EAC) in the next release of Exchange

4. Automatically protect Exchange availability from surges in traffic

   Exchange now offers easy to administer controls to protect against unexpected surges in traffic. System work that is not interactive is automatically deferred to non-peak hours in order to preserve the end user experience and higher priority tasks. This improved overall system through-put leads to reduced costs by saving you from planning capacity for those infrequent, unexpected peaks.

5. Cloud on your terms

   Exchange provides you tools to move to the cloud on your terms – whether that’s onboarding to the cloud overnight or easily managing a hybrid deployment with mailboxes on-premises and online to meet your business needs. Provide your end users with a seamless experience including sharing calendars and scheduling meetings between on-premises and online users and have minimal user disruption when user mailboxes are smoothly moved across environments. Remain in control in the cloud by testing out upcoming enhancements via previews.

6. Automatically protect your email from malware

   Exchange now offers built in basic anti-malware protection. Administrators can configure and manage their protection settings right from within the Exchange Administration Center. Integrated reporting provides visibility into emerging trends. This capability can be turned off, replaced, or paired with premium services such as Exchange Online Protection for layered protection.

7. Protect your sensitive data and inform users of internal compliance policies with Data Loss Prevention (DLP) capabilities

   Keep your organization safe from users accidentally sharing sensitive information with unauthorized people. The new Exchange DLP features identify, monitor, and protect sensitive data through deep content analysis. Exchange offers built-in DLP policies based on regulatory standards such as PII and PCI, and is extensible to support other policies important to your business. New Policy Tips in the new release of Outlook inform users about policy violations as content is being created and about how information should be handled according to organizational standards.

   
   Figure 2: Protect your sensitive data with Data Loss Prevention (DLP) capabilities

8. Allow compliance officers to run In-Place eDiscovery across Exchange, SharePoint, and Lync – from a single interface

   The ability to immutably preserve and discover data across your entire organization is essential to ensuring internal and regulatory compliance. Allow your compliance officers to autonomously use the new eDiscovery Center to identify, hold, and analyze your organization’s data from Exchange, SharePoint, and Lync. And, the data always remains in-place, so you never have to manage a separate store. With the eDiscovery Center, you can reduce the cost of managing complex compliance needs, while ensuring you are prepared for the unexpected.

   
   Figure 3: Run In-Place eDiscovery across Exchange, SharePoint and Lync from a single interface

9. Allow users to naturally work together – while compliance is applied behind the scenes

   Site Mailboxes bring Exchange emails and SharePoint documents together. Like a filing cabinet, they provide a place to file project emails and documents and can only be seen by project members. Document storage, co-authoring, and versioning is provided by SharePoint while messaging is handled by Exchange – with a complete user experience within the new release of Outlook. Compliance policies are applied at the site mailbox level and are transparent to the users – thus preserving their productivity.

   
   Figure 4: Site Mailboxes bring Exchange emails and SharePoint documents together

10. Modern public folders provide a great way of managing and storing shared conversations and discussions

    Public folders are now available in Exchange Online. Both on-premise as well as online, public folders provide the same capabilities customers are already familiar with. And more – they now share the same storage, indexing, and HA capabilities of regular mailboxes and public folder content can now be found via end-user search.

11. Give your users an intuitive, gorgeous, touch-optimized experience on all screens

    Your end users will get more done from anywhere with a clean and uncluttered experience. Users can now take advantage of the fresh, easy, and intuitive Windows 8 style experience across Outlook and OWA. OWA user experience scales beautifully for any form factor and size – PC or slate or phone – and has a modern user experience voice with great support for touch and motion. OWA now offers three different UI layouts optimized for desktop, slate, and phone browsers.

    
    Figure 5: An intuitive, gorgeous, touch-optimized experience on all screens

12. Offline support in OWA allows your users to be productive when offline or on intermittently connected networks

    You can now launch OWA in the browser and start working even if there is no network connectivity. Your emails and actions are automatically synchronized the next time connectivity is restored. This allows your users to be productive and have a great OWA experience even from remote locations with slow or intermittently connected networks or no network connection at all.

13. Bring all of your contacts together and automatically keep them up-to-date

    People’s professional networks span many different places. In Office 365, your users can import contact information from LinkedIn (and other networks in the future) so that they have all of their information in one place. Exchange will even find the same person across your personal contacts, GAL, and other networks and consolidate their information into one contact card, avoiding duplication and multiple contact cards with different information.

    
    Figure 6: Bring all your contacts together from Exchange's GAL, your personal contacts and other networks

14. Modern people search experience lets you quickly find the right person

    People search experience is consistent everywhere – from people hub to nickname cache when composing an email. Search spans across all of your people – personal contacts, GAL, networks. Search results are relevance based and contain rich results – photos, phone number, location, etc.

    
    Figure 7: Quickly find the right person across your personal contacts, GAL and networks

15. Updated canvas makes calendar more useful for everyone

    Like Outlook, OWA now supports simple entry of reminders and to-do’s by typing right on the calendar. Users get quick, glance-able day and item “peeks”. New views for day, week, and month – like the “month + agenda” (or “Mogenda”) view – makes it really easy to manage your time.

    
    Figure 8: Manage your time easily with the new views for day, week, and month

    
    Figure 9: Calendar item "peek" shows useful information

16. Customize Outlook and OWA easily by integrating apps from the Office marketplace

    Help your users be more productive via 3rd party apps for Outlook adding contextual information and functionality to emails and calendar. Apps for Outlook are easy to develop using the new cloud-based extensibility model. The same apps work across the new release of Outlook and OWA – including on OWA’s slate and phone optimized layouts. Users and Exchange administrators can easily discover and install apps via the Office marketplace. You can control which apps different end users can use.

    
    Figure 10: Customize Outlook and OWA with 3rd party apps from the Office marketplace

This is the first of a series of blog posts which will cover the next release of Exchange. In future posts, we will cover the full set of capabilities, including all of the features mentioned above, in more detail.

To get fully up-to-speed on the next release of Exchange:

• Download the preview version of Exchange Server 2013.
• Try the new Exchange Online in the Office 365 Enterprise Preview.
• Follow the Exchange Team Blog.
• Attend the Microsoft Exchange Conference (MEC) in September. Go deep on the next release of Exchange with industry experts and the Exchange engineering team. It’s been a decade since the last MEC, and we’ve got some big surprises in store for our community!

As always, we welcome your comments and feedback. We've also gone live with the Exchange Server 2013 Forum and will monitor it regularly to collect your feedback.

Thanks so much for your interest in Exchange, and we hope you find the next version of the product as exciting and innovative as we do. The entire team looks forward to your feedback!

Rajesh Jha
Corporate Vice President
Exchange

Yesterday Microsoft Security Research Center issued Microsoft Security Advisory (2737111) - Microsoft is investigating new public reports of vulnerabilities in third-party code, Oracle Outside In libraries, that affect Web-ready document viewing in Microsoft Exchange Server 2007 and Microsoft Exchange Server 2010. We recommend that customers apply the workarounds described in this advisory so you are not exposed to the vulnerabilities described in Oracle Critical Patch Update Advisory - July 2012.

The reported vulnerability that’s being investigated impacts web-ready document viewing in Exchange 2010/Exchange 2007. Web-ready document viewing is a feature that allows Outlook Web App users to view supported attachments in an email without having to download them to a computer and using locally-installed applications to view them.

For more information, see Microsoft Security Advisory (2737111) and More information on Security Advisory 2737111 on the Microsoft Security Research & Defense blog.

Bharat Suneja

Earlier this month we announced a change on this blog related to licensing for Multi-Mailbox Search. Since then we’ve received questions about when it’ll go into effect. The change will go into effect starting October 1, which is when it’ll be reflected in the product use rights (PUR). The PUR contain the terms and conditions for how you can use software license purchased through a Microsoft Volume Licensing program and is updated quarterly. You’ll see the change represented in the October Product List. We’ve updated the original post to include this information.

While we’ll have more to share about licensing for the new Exchange in the coming months, I can tell you this change will also carry over into licensing for Exchange Server 2013.

Stay tuned to this blog and Office Next for more details about the new Exchange and the new Office.

Bharat Suneja

Last week, we shared information about The New Exchange. Today we’re delighted to share some of the exciting new developments in Outlook Web App (OWA).

The primary purpose of OWA is to give Exchange users access to their mailboxes from anywhere through the same Outlook UI and functionality they already know and love. In this release, OWA has evolved to reach not only laptop or desktop computers, but also tablets and phones. As we did with desktop browsers, we’ll continue investing in wider and richer support for tablet and phone platforms in the future. You can try our latest release by signing up for the Office 365 Customer Preview Small Business Premium or Enterprise plans.

Figure 1: The new OWA on desktop, tablet and phone screens

Desktops, tablets and phones have different UI requirements. To ensure people get a consistent experience across all their devices and to make it possible for us to continue evolving the UI for all device types in parallel, we built one OWA application which knows how to scale the UI to different screen sizes and across touch, mouse and keyboard input:

• For screens found on laptops or larger devices, OWA projects a UI optimized for mouse and keyboard input. We refer to this mode as Desktop.
• For tablet size screens, OWA projects a UI optimized for touch input. We refer to this as touch-wide
• For phone size screens, OWA projects a narrower UI for touch. This is touch-narrow.

The application logic running underneath the UI, representing >90% of the application code, is shared between the three UI modes. Because the modes are very similar in function and look-and-feel, users don’t need to relearn how to manage email or calendaring as they switch between different devices. Because the application logic is the same for the three modes, administrators can rest assured that the policies they put in place for this application applies on any device where it is used.

Windows 8 style UI

OWA has a new design language that gives it a fresh, modern look that aligns to Windows 8, Windows Phone and the rest of the Office 2013 products. It’s a clean, light, and open approach that enables people to focus more on the communication content, with less distraction. The simplified visual language and interaction models elevate people and content in the experience, and aims to strike a balance between form and function.

Figure 2: Photos of people and attachment thumbnails

To help users avoid context switching, we also added email inline compose in the main window. Users can pop out certain compose forms, or change a setting to have all compose forms pop out automatically. Inline compose without losing view of the email list view you’re working your way through is a good time saver for most users.

Although we have spent a lot of time ensuring the default colors of OWA will feel great, different people have differing tastes, so the OWA capability of selecting a different theme has been enhanced with more themes that affect more aspects of the UI for a more delightful experience.

Figure 3: Some of the themes users can choose from in OWA

Offline support

In addition to the new User Interface, we made an additional major architectural change in OWA: offline support. You can now launch OWA in the browser and start working even if there is no network connectivity. The work you do while disconnected is automatically synchronized the next time connectivity is restored. This allows your users to be productive using OWA even when they have no network connectivity. It also makes OWA faster and a better experience from remote locations with slow or intermittently connected networks.

Our goal for OWA offline is to support the most common user actions, as far as HTML5 offline capabilities allows. This does not take us to the complete offline support of Outlook on the Windows or Mac desktop, but it matches or exceeds the offline capabilities offered by most Exchange ActiveSync phone clients. Most email triage and calendar management actions work in OWA while offline. You can send email and calendar invites, delete email, get reminders, decline meetings, flag and categorize messages, etc. If you access a folder often, its contents will automatically be made available offline. An example of something not yet possible with OWA offline, or HTM5 offline in general, is full-text indexed search. Over time as the offline capabilities of browsers mature, so will OWA’s offline capabilities. Users must enable OWA offline functionality on each computer where they want to be able to use OWA without network connectivity. Administrators have the ability to control which users can use OWA offline.

You can enjoy OWA offline in IE10+ (using HTML5’s IndexedDB for storage), Chrome 17+ (with WebSQL) or Safari 5 (also using WebSQL).

Outlook Apps

Past extensibility models for OWA have all relied on installing 3rd party code on Exchange servers. That model doesn’t work well for customers who rely on Microsoft to run Exchange servers for them through Office 365 and even on premises this model often meets with resistance from Exchange administrators who don’t want to risk the stability and security of their servers. As part of the new Office cloud app model, the new Exchange includes a platform for what we call “Outlook Apps”. These are web apps which a 3rd party can host somewhere on the Internet, and which Exchange end users and administrators can easily and safely integrate into the Outlook experience. The apps can show up in email and calendaring forms and are either always present, or more commonly, activate for a given email or calendar item when there is something special about that email or calendar item. For example, the Bing Maps Outlook App activates when there are one or more addresses present in the email or calendar item body text. Another example is the Groupon by PowerInbox Outlook App which shows up only for emails sent from Groupon. LinkedIn’s Outlook App activates on all email, showing information from the LinkedIn social network about the people on the email thread.

You can go see what Outlook Apps are currently available for use with the Office Customer Preview in the Office Store. Administrators have a lot of control over what Outlook Apps they allow their users to use, or promote to their user populations. Outlook Apps work in Outlook 2013 for the Windows desktop and in the new Outlook Web App on desktop, tablets and phones. The exact same Outlook App code runs in both Outlook and OWA, so developers can quickly and easily target users of the two clients with the same extension.

Figure 4: The LinkedIn app for Outlook activated in OWA, showing information about all the people included in an email conversation

In addition to enabling developers to extend Outlook without putting code on Exchange servers, this new extensibility model also overcomes the challenge of Office plug-ins that worked with only one version of Office. We overcome that challenge by offering very simple version and platform-agnostic JavaScript APIs for the applications to use for all integration with the Outlook/OWA UI, avoiding scenarios where developers, on purpose or by accident, might hook into any version-specific aspects of the Outlook experience.

New in Email

Managing and sending email in OWA is more efficient than it has ever been. Inline compose, mentioned above, allows users to quickly compose and send messages without popping out a window. Inline compose also comes with an improvement in draft management. It’s now easier to see in the inbox which conversations have unsent drafts, helping you remember to finish and send that email you started working on yesterday before leaving work. Once you’re composing an email, the Forgotten Attachment Detector notices if you just composed an email which looks like it probably intended to include an attachment. When you press send without having added an attachment, it warns you that you might have forgotten to add one.

Figure 5: Improved draft indicators

Figure 6: Forgotten Attachment Detector in action

Drag-and-drop of files from the desktop to the compose form helps users of IE10, Safari, Firefox and Chrome easily attach files to emails they’re composing. For email triage, OWA, just like Outlook 2013, includes a quick delete button directly on items in the email list view. The ability to filter email according to whether they’re unread, flagged, or have your name in the to-line has been promoted to the top level of the UI for easier discoverability.

Figure 7: Quick delete buttons inline on items and all/unread/toMe/flagged filters above email list view

More efficient calendaring

Although some OWA users have dense calendars which are best viewed in day or week view, most of our users only have one or a few meetings or appointments entered for each day, making their calendar most suitable for viewing through a monthly or agenda view. To give a great default calendar experience to this majority of users, we have added the new “MOGENDA” calendar view to OWA, combining MOnthly and aGENDA view.

Figure 8: Mogenda calendar view

Meeting scheduling has gotten a significant makeover, making it easier to find times when all attendees can attend. Like Outlook, OWA now supports simple entry of calendar items and to-do’s by typing right on the calendar. Users get quick, glance-able day and item “peeks”.

Figure 9: Calendar item “peek”

All information about people in one place

The new OWA helps you get a clear view of all the information you have about people by linking all data about a given person to get combined views from a corporation’s Global Address List (GAL) and the contact folders in the mailbox. In Office 365, you also get LinkedIn integration. This makes the linking extend to LinkedIn contacts, enhancing the combined views of information about people even more. We aim to extend this linking to other networks in the future.

Figure 10: All information about a person from contacts, Global Address List and LinkedIn in one place

All this information compiled about people is also put to good use as you try to find the right people to communicate with. Search for people spans across all of your people – personal contacts, GAL and connected social networks. Search results are relevance based and contain photos, phone number, location, etc.

Accessibility

As we announced a little while ago, we have been excited about the Accessible Rich Internet Application (ARIA) standard for a while. In the new OWA, we use ARIA to make the OWA Premium desktop UI mode accessible through screenreaders, for high contrast needs, keyboard-only navigation needs, etc. We have focused on ensuring the experience with IE10 and JAWS or Win8 Narrator will be great for people with accessibility needs. Just as for people without accessibility needs, OWA Light is still available for interoperability with accessibility technologies we have not yet verified work great with OWA Premium.

A common challenge for complex web applications and accessibility, even when using ARIA, is how to provide convenient navigation within the application given that the browsers use the top level F6 key for navigation, making it impossible for the web application to also use the convenient F6 top level navigation mechanism. As a solution to this problem OWA and the other Office Web Access Companions are introducing the Ctrl+F6 shortcut for top level navigation within the web application. We hope others in the industry will follow along and make this a de-facto standard as more complex web applications add ARIA support.

Server Architecture

Managing OWA is very similar in Exchange Server 2013 to what it was like in Exchange Server 2010 and Exchange Server 2007, with OWA settings being set on the Internet Information Server (IIS) virtual directories on the Client Access Server (CAS) and in OWA mailbox policies. Internally there has been a big change in architecture for OWA though. OWA rendering logic runs on Mailbox (MBX) servers, whereas in the past this rendering logic ran on the CAS. The Exchange Server 2013 CAS only does user authentication, redirection and proxying. This architecture change results in a significant simplification in how load balancing needs to be deployed for OWA and all the other Exchange protocols. You can find more information on the changes for the CAS role in Exchange 2013 Server Roles.

Monitoring to ensure OWA service continues undisrupted has also changed significantly in Exchange Server 2013. We have made the monitoring and auto-recovery mechanisms we use in Office 365 datacenters also run on premises, giving all Exchange customers the most robust and feature rich monitoring and self-healing system we have ever had for Exchange.

Browser and Operating System support

Just like in the last release of OWA, the new OWA comes in two flavors:

• OWA Premium is what serves up the new desktop, tablet and phone optimized UI. You can get to this UI from desktops using Internet Explorer (IE) 8+, and newer versions of Safari, Chrome and Firefox. To get to the tablet and phone OWA UI, we are starting out with support for Windows 8 tablets, and iOS6 on iPad2 and iPhone 4 or newer in our initial release. In the preview releases of OWA, you can use iOS5 since iOS6 isn’t released yet, but this will change since iOS6 includes several bug fixes necessary for the OWA experience to be great.
• OWA Light, which largely remains the same as with the last release of OWA, is what you get on any browser not supported with OWA Premium or OWA Mini. It uses a very simple HTML4 based UI which works in pretty much any browser in existence. The most significant addition to the list of browsers which get OWA Light with OWA 2013 is IE7.

So what do you think we should do next?

We’re very excited to finally share the new OWA, its updated Windows 8 style UI and added tablet and phone support. We can’t wait to hear your feedback on where to go from here!

Kristian Andaker
on behalf of the OWA team