On your Exchange 2016 databases you may encounter a situation in which the content index fails. A failed content index will impact Outlook on the web (OWA) users trying to search their mailbox contents, and can also cause database switchover and failover problems for you.

Failed content indexes can be seen in the output of Get-MailboxDatabaseCopyStatus.

[PS] C:\scripts\>Get-MailboxDatabaseCopyStatus * | sort name | Select name,status,contentindexstate

Name               Status  ContentIndexState
----               ------  -----------------
DB05\EX2016SRV1   Mounted            Healthy
DB05\EX2016SRV2   Healthy FailedAndSuspended
DB06\EX2016SRV1   Healthy            Healthy
DB06\EX2016SRV2   Mounted            Healthy
DB07\EX2016SRV1   Healthy            Healthy
DB07\EX2016SRV2   Mounted            Healthy
DB08\EX2016SRV1   Healthy            Healthy
DB08\EX2016SRV2   Mounted            Healthy

For an Exchange 2016 server that is not a member of a database availability group the failed content index can be rebuilt using the following procedure. Note that rebuilding the content index may cause high CPU utilization on the server. If you have any concerns about performance impact you should perform this work at an off-peak time for your customer.

First, stop the search services on the server.

[PS] C:\>Stop-Service MSExchangeFastSearch

[PS] C:\>Stop-Service HostControllerService

Next, open Windows Explorer and go to the folder that contains the files for that database. If you aren’t sure where that is located you can query the database attributes.

[PS] C:\>Get-MailboxDatabase DB05 | select EdbFilePath

EdbFilePath
-----------
E:DB05DB05.edb

In that folder (E:DB05 in this example) there will be a sub-folder with a GUID for the name, which contains the content index files. Delete that GUID folder.

exchange-2016-content-index-01

Start the search services on the server again.

[PS] C:\>Start-Service MSExchangeFastSearch

[PS] C:\>Start-Service HostControllerService

The content index will be crawling and rebuilding, which will take some time depending on the size of the database.

For an Exchange 2016 server that is a member of a database availability group the content index can be fixed by reseeding it from a healthy copy.

[PS] C:\>Update-MailboxDatabaseCopy DB05\EX2016SRV2 -CatalogOnly -BeginSeed

When the content index has finished rebuilding or reseeding you should see all healthy results again.

[PS] C:\>Get-MailboxDatabaseCopyStatus * | sort name | Select name,status,contentindexstate

Name               Status  ContentIndexState
----               ------  -----------------
DB05\EX2016SRV1   Mounted            Healthy
DB05\EX2016SRV2   Healthy            Healthy
DB06\EX2016SRV1   Healthy            Healthy
DB06\EX2016SRV2   Mounted            Healthy
DB07\EX2016SRV1   Healthy            Healthy
DB07\EX2016SRV2   Mounted            Healthy
DB08\EX2016SRV1   Healthy            Healthy
DB08\EX2016SRV2   Mounted            Healthy

Cybersecurity Risk Management for Active Directory

Discover how to prevent and recover from AD attacks through these Cybersecurity Risk Management Solutions.

About the Author

Paul Cunningham

Paul is a former Microsoft MVP for Office Apps and Services. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server. Paul no longer writes for Practical365.com.

Comments

  1. Bobby

    Very grateful for you detailing this – working a charm reindexing now, after Exchange 2016 CU19 must have screwed this up on the quiet. Much thanks 🙂

  2. John F

    Awesome you da’ man. This happened after application of CU19 to Exchange 2016 for Zero day exploit.
    Micros*** blah.

    1. Detrich

      Yes. This happened to us too.

  3. Waseem

    I got same issue of ContentIndex “failed” after upgrading to Exchange 2016 CU19.
    Checked the below services, one of them was disabled, after enabling/starting everything is fine.

     Microsoft Search (Exchange)
     Host Controller service for Exchange services

    1. T Nguyen

      Thanks Waseem,

      We applied the Exchange 2016 CU19 on March 8 2021 and you were right!

      Our Host Controller service was disabled and after enabling and starting the service the search functionality started working.

      1. Kenneth Chukwu

        Such was the case with me. I found the service disabled as well after upgrading from CU17 to CU19. Re-enabling the service resolved the issue with ContextIndexing.

    2. K. Tu

      We ran into this same problem last night after upgrading to Exchange 2016 CU19.

      Thanks!!!

    3. Syed Tahir Hussain shah

      Bro I have Also Exchange server 2016 with CU19 but this issue mostly when replication between database and database copies on a different servers. when I fix my replication issues between servers in my environment after that no copy fails on any servers.

  4. Joao Leandro

    Muito obrigado, fiz o processo como mencionado, e para minha alegria, após reiniciar os serviços, a pesquisa estava funcionando. TOP

  5. Senthil

    Hello,

    After migrating from EX2010 to EX2016, we have similar search issues
    I could see all ‘ContentIndexState’ of 2016 servers is Unknown.
    could you update us how to proceed to fix the issue.

    Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex
    Length Length State
    —- —— ——— ———– ——————– ————
    Mailbox Database 2047897560\XXXX Mounted 0 0 Unknown
    Mailbox Database Primary\XXXX Mounted 0 0 Unknown

    Cheers.

  6. Scotty

    Paul,

    Great article just what i was looking for this morning.

  7. Luis Garcia

    I have now foxed the issue.
    Found in the event viewer that there was an error with space, eventhough I still have heaps of space, it said it has passed the 70% threshold. So all I did was to relocate some files out of the drive to have extra space in the exchange drive, then I followed the above steps again, and after several hours, this time it worked.

    Thanks a lot.

  8. Aakif Mohamed Mohamed Anees

    Hi Paul,
    Thanks for the note really helpful

  9. Luis

    Hello,

    I have this problem on Exchange 2013.
    I have followed this steps every day for a week now.
    It goes to Healthy. (search still won’t work in outlook or owa)
    Then the next few hours is back to FailedAndSuspended.

    Any idea what could be causing it?
    I have only one Database no DAG
    Cheers.

  10. John

    Running a test-exchangesearch from EMS reports time outs/errors and Outlook searches are inconsistent. Exchange shows the content index as healthy for all DAG servers containing the database copy.

    It’s been advised to rebuild the content index from scratch. In a case where Exchange lists the content index as healthy and you have a DAG, what should be the correct steps to re-build the content index from scratch? I don’t want it attempting to copy over an index from the other DAG servers since they all have same ‘healthy’ but unsearchable indexes. I haven’t found anything yet but my google-fu is a bit lacking since I’ve working other issues related to this restore.

    In my mind, I think you would just stop the MSExchangeFastSearch/HostControllerService(s) on all 3 members, disable the services on the two non-active members, remove the index on the active server and then let it re-build on service restart. When complete, repeat the index delete on the remaining two DAG server, enable/restart services and let them complete.

    My reasoning for disabling services on the other two DAG server is to not have them attempt to auto-start/build while trying to get a clean index made on the active server.

    Has anyone had to do this? Have I missed an obvious article for such a process?

  11. thierry

    Anyone familiar with this error?

    ContentIndexErrorMessage : Could not find registry value of Index Status for database {2134923498 9083 8-02384-0 8-0823940}

    (standalone on premise)

    thanx in advance

  12. Rennex Callaghan

    Hey Paul,

    Just want to say thanks! Great resource.

  13. Ray P

    Hi Paul,
    We were updating our Exchange 2016 CU9 servers today in our dags one at a time. After a few were done we noticed a lag in our environment. When checking we see our Passive Databases reseeding. Would be nice to know why.

  14. Jim L

    If you’re having regular issues with the index failing, check to see if there is enough hard drive space on the drive that the database/index is on for a copy/expansion.

    We encountered a problem where we had to rebuild the index daily. Everyday we rebuilt the index based on your recommendations above and the next morning it failed.

    Upon further investigation we found in the event log an event 1006 followed by an immediate 1009. This appeared to happen every night around the same time.

    The daily recurrence was a backup of the database. We assume the software that we used must’ve stopped the database, made a temporary copy and restarted the database. Finally we came to the understanding that there was not enough space to complete this process and the end result was the failure of the index. There was no indication on the backup software or in the event log of the failure.

    We added drive space and the issue went away.

    1. bubu

      @Jim L says

      What backup solution you’re using?

    2. Luis

      Hey Jim,

      After an extensive search online, your description of the problem is the closest to mine.
      Mine also fails every day and I do have a daily backup solution. (shadowprotect)

      Did you just add drive space to the backup unit? or to the local drive?
      I do have plenty of space available in both, not sure if there is a threshold there.
      Cheers.

  15. TimA

    Hi Paul,

    I want to express that you are greatly appreciated!
    in my scenario (3 databases), I ran a series of commands to try to get things to work (below after command that fixed my situation).
    This is what fixed it for me:
    Update-MailboxDatabaseCopy -identity DB01\MBX01 -CatalogOnly

    note: when I used this command there was no change:
    Update-MailboxDatabaseCopy -identity DB01\MBX01 -SourceServer MBX02 -CatalogOnly
    But the command without -SourceServer gave a warning:
    The mailbox database copy ‘DB01\MBX01’ has failed to update from server . Do you want to clean up that update request now? Seeding cannot be
    requested for the same database copy until the failed request has been cleaned up by the server, which should automatically happen within 15 minutes.

    After clicking through the warning, 2 of the 3 databases were Healthy immediately, and the DB01 had to complete the seed.

    Here are the commands I tried before my solution:
    Set-ServerComponentState MBX01 -component “HighAvailability” -requester Maintenance -State Active

    Set-ServerComponentState MBX01 –Component ServerWideOffline –State Active –Requester Maintenance

    Set-MailboxServer MBX01 –DatabaseCopyAutoActivationPolicy Unrestricted

    Set-MailboxServer MBX01 –DatabaseCopyActivationDisabledAndMoveNow $false

    Resume-ClusterNode –Name MBX01

    Set-ServerComponentState MBX01 –Component HubTransport –State Active –Requester Maintenance

  16. Melissa

    Hi Paul,

    I’m showing Database01 and Database02 in the same folder and two GUID-named folders there as well. Only Database01’s Content index state is showing as FailedAndSuspended. So how do I know which folder to delete?

    Thank you in advance 🙂

    1. Melissa

      NM. May be the long way around, but I was able to dig around in my AD Microsoft Exchange System Objects until I found the right property under Attribute Editor. Booya!

  17. Sebastain Gomes

    Hello Paul,

    I have two servers in a DAG but it is giving error while creating a mailbox database copy and the error is related with content Index. But ran a command with -Skip ClientExperienceChecks and it worked. Now, I am not able to either switchover the database nor move the mailbox to this database.
    Tried with this below command but still the that content indexing is in failed and suspended state.
    Update-MailboxDatabaseCopy DB05EX2016SRV2 -CatalogOnly -BeginSeed

  18. Enrique Lopez

    Some of my database copies are with ContentIndexState: Disable, although are configures as True.

    Can the IndexSearch “fail” due a low ram memory configuration?

    My DAG hosts (2) are in a virtual environment and I want to know if increment Ram will be usefull.

  19. jast

    Hi Paul,

    When you run Update-MailboxDatabaseCopy, this command mus be run with the MSExchangeFastSearch and HostControllerService stop?

    Regards

  20. Joe

    My indexes are showing healthy
    I have a server with 3 databases DB 1 and DB3 work fine DB2 when I run a test-exchangesarch -idenetiy name gives me a time out of thread the other 2 work fine. I repaired the index on those servers but still has same results. any suggestions?

  21. Lyle Epsteiin

    Thanks Paul. This worked great on my Exchange 2016 server and fixed the searching issue!

  22. bpoelen

    Hi Paul,

    again, I had to repair this because New-MailboxImportRequest could not import anymore.

    greetz bpoelen,
    the netherlands

  23. bpoelen

    Oh Boy, Paul,

    you saved my Day, I was in a endless for days loop to resolve this problem.
    I was still a bit afraid to delete a folder in the exchange folder.
    But is is working now, amazing

    thanks,thanks, thanks

  24. Patrick

    Thank you for your efforts on this Exchange blog.
    My Content index state = failed.
    My services list does not have those two present.
    Why might they be missing (hence my real problem)?
    I am pursuing other steps to identify this.

    I always learn something from you and the commentators.

    1. semper

      try run “Stop-Service …” from powershell.
      not exchange mgmt shell

  25. Darren

    Thank You! Great instructions and worked perfectly!

    1. TeakWoodKite

      I agree. 🙂 Thank you Paul. Very Helpful.

  26. Kyle

    Paul

    Thanks as always for the great Exchange related content. A lot of us out here rely on your insight.

    I have a situation that maybe you can offer some guidance on. I have a test environment setup testing Exchange 2016. I have three mailbox servers that are in a DAG. The servers currently have 16GBs of ram and a 40GB hard drive where the databases are stored (test mailboxes only, but no real content). The DAG has its own replication 10G network.

    It seems like every time I add a new database copy, the content index comes up as FailedAndSuspended and never reaches a healthy state. I have followed several different posts that say to use the Update-MailboxDatabaseCopy, but nothing seems to be helping. I also added an AD group called ContentSubmitters based on an article I read.

    Any thoughts on why the content indexes never get healthy or take a long time to reach a healthy state? (example: I added a test database around 6:40PM along with two copies, it has been 2.5 hours and the content indexes for the copies are still in a state of FailedAndSuspended.)

    Thanks
    Kyle

    1. Avatar photo
      Paul Cunningham

      Make sure you’ve configured antivirus exclusions.

      Also try simplifying your networking by removing the replication network.

  27. SAM

    After reseeding the Lag copy database we are seeing the content indexing state as Autosuspend and replay queue length is also too high (in thousands). Replay Lag time and truncation is set to 7 Days. Is this normal behaviour? Please advice as we reseeded one database again but still it is showing Autosuspended and the replay queue length is high. We are using Exchange 2016. Safety net hold time is set to 1 day.

    1. krishna

      its expected on lagged copies..I confirmed it

  28. Klaus

    Are you running RU3? If so, the recommendation is: Reinstall your cluster with RU2.
    Unfortunately not joking – we’ve been there.

  29. frank

    im getting failed for content now on ALL . since i renamed the old guid.

    Name Status ContentIndexState
    —- —— —————–
    PublicFolderDatabaseSRV-EXCHANGE Mounted Failed
    QCDBEXCH01SRV-EXCHANGE Mounted Failed
    QCDBEXCH02SRV-EXCHANGE Mounted Failed
    QCDBEXCH03SRV-EXCHANGE Mounted Failed
    QCDBEXCH04SRV-EXCHANGE Mounted Failed

    where before it was

    Name Status ContentIndexState
    —- —— —————–
    PublicFolderDatabaseSRV-EXCHANGE Mounted FailedAndSuspended
    QCDBEXCH01SRV-EXCHANGE Mounted Healthy
    QCDBEXCH02SRV-EXCHANGE Mounted Healthy
    QCDBEXCH03SRV-EXCHANGE Mounted Healthy
    QCDBEXCH04SRV-EXCHANGE Mounted Healthy

    any help ?

  30. Sameer B

    Hi Paul,

    Thanks for the great post !

    BTW after migrating from EX2010 to EX2016, we have similar search issues for some huge mailboxes. Numeric searches are not properly working (M/B Size 25 GB)

    I could see all ‘ContentIndexState’ of 2016 servers as Healthy.

    Please advise.

  31. Eric

    Hello Paul,

    Thank you for the excellent posts on Exchange! I followed it and did not run into any obvious errors, but still have a problem. I hope you can help me out.

    Last week our Exchange 2016 database got corrupted when power was cut to the server. I have restored the database but forgot the index folder.

    Users are complaining that the search option in is not working in Outlook and OWA.

    With the “Get-MailboxDatabase | select EdbFilePath” I located the index folder and deleted it. When I started the search services the folder is not recreated and I am still not able to search for e-mails.

    When I run the “Get-MailboxDatabaseCopyStatus” command the output states that the ContentIndexState is healthy.

    I hope that you can point me in the right direction.

    1. Jo-Jo

      I have the same issue,
      But i can see that the utilization of the command line takes a lot of time based on how big the database is. So i keep this running overnight to see if this resolves it.

      Other tip , create new DB and move the databases to that new database , as the index will recreate itself on the new DB

  32. Oskar

    Thank you for your great resources!

    I have seen, that after following your guide and restarting the services Get-MailboxDatabaseCopyStatus will still show “FailedAndSuspended” for a while (you will see the noderunner.exe task(s) working at full speed).
    After a while it will change to “Crawling” and after that hopefully to “Healthy”, but I’m not there, yet.

  33. Andrzej

    In our environment after installing CU1 for Exchange 2016 crashed serach service.
    We couldn’t find any way to resolve this issue (we try index repair, etc.)

    Yesterday we installed CU2 for Exchange 2016 and do repairing content index as described above. Now it’s working fine.

  34. Lister

    is the command Update-MailboxDatabaseCopy DB05EX2016SRV2 -CatalogOnly -BeginSeed only for Exchange 2016 or it also works with 2013?

    1. Avatar photo
      Paul Cunningham

      If you’re ever wondering whether a parameter is available for a cmdlet, or you want to find out more about a parameter, just check the TechNet page for that cmdlet. That’s what I do.

      1. Bob Nine

        No. No. No. No.

Leave a Reply