While exploring the Office 365 Import Service in my test lab I encountered repeated timeout errors trying to upload PST files to Azure using AzCopy.exe.
[2015/07/26 23:26:36.206+10:00] >>>>>>>>>>>>>>>> [2015/07/26 23:26:36.221+10:00][VERBOSE] 3.1.0 : AzCopy /source:\mgmtpst /dest:https://uploadurl.blob.core.windows.net/ingestiondata/ /DestKey:****** /S /V:C:AdminPSTupload.log [2015/07/26 23:26:39.680+10:00][VERBOSE] Start transfer: Alan Reid.pst [2015/07/26 23:26:40.352+10:00][VERBOSE] Start transfer: Dawn Evans.pst [2015/07/26 23:26:40.352+10:00][VERBOSE] Start transfer: Elaine West.pst [2015/07/26 23:26:40.352+10:00][VERBOSE] Start transfer: Vik Kirby.pst [2015/07/26 23:41:41.552+10:00][VERBOSE] Transfer FAILED: Alan Reid.pst. [2015/07/26 23:41:41.568+10:00][ERROR] Alan Reid.pst: The client could not finish the operation within specified timeout. The client could not finish the operation within specified timeout. [2015/07/26 23:52:41.450+10:00][VERBOSE] Transfer FAILED: Elaine West.pst. [2015/07/26 23:52:41.450+10:00][ERROR] Elaine West.pst: The client could not finish the operation within specified timeout. [2015/07/26 23:52:41.450+10:00][VERBOSE] Transfer FAILED: Vik Kirby.pst. [2015/07/26 23:52:41.450+10:00][ERROR] Vik Kirby.pst: The client could not finish the operation within specified timeout. [2015/07/26 23:53:23.429+10:00][VERBOSE] Transfer FAILED: Dawn Evans.pst. [2015/07/26 23:53:23.429+10:00][ERROR] Dawn Evans.pst: The client could not finish the operation within specified timeout. The client could not finish the operation within specified timeout. [2015/07/26 23:53:23.445+10:00] Transfer summary: ----------------- Total files transferred: 4 Transfer successfully: 0 Transfer skipped: 0 Transfer failed: 4 Elapsed time: 00.00:26:47
I was trying to upload using my home internet connection, which doesn’t have high upload speed. The upload process was also showing a transfer speed of 0 KB/s. Clearly it was not making any progress.
Reviewing the AzCopy command line options I saw that there is a /NC parameter that can be used to control the number of concurrent operations.
AzCopy by default starts a certain number of concurrent operations to increase the data transfer throughput. Note that large number of concurrent operations in a low-bandwidth environment may overwhelm the network connection and prevent the operations from fully completing. Throttle concurrent operations based on actual available network bandwidth.
Lowering the thread count to 2 (by specifying /NC:2) seems to have resolved the issue for me, and the upload was able to finish on the next attempt.
It is easy now, there are many tools that can do this job easily and quickly like Shatregate, Gs Richcopy 360, and Syncback
Thanks a lot for the switch /NC.
It worked for me too.
This has 100% just saved my bacon, after a couple of days trying to upload a PST – the M$ documentation for this is pretty poor.
Thank you Chumrade!
Still a problem in September of 2021. I was only trying to upload one 8GB PST file. Thanks for this solution!
Well, I thought this can’t help my case as I only have one file to upload. But it did
Thanks – Great find. /NC:2 worked a treat.
This fix was extremely helpful. I was having failures exporting the initial large files and almost got frustrated. I have successfully transfer over 650GB of data thanks to the /NC:2 switch.
Thank you Paul. May your toughest challenge have the easiest solution.
what was the final command that you ran to do this ?
yes I would love to see a full example — the /NC:2 option does not seem to be supported anywhere in my command.
with the current version of azcopy we need to set the environmental variable:
set AZCOPY_CONCURRENCY_VALUE=2
before launching azcopy
https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-optimize#increase-the-number-of-concurrent-requests
i am very frustruting knowing that in 2020 MS hasn’t a proper tool for PST migration to O365 especially when AzCopy is a pain in the ass and never works!!!
thanks a lot for your post!
A HUGE thanks to Paul. It is 2019 and STILL an issue. The /NC:2 switch saved the day. Well, Paul did 😉
This seems to work! I couldn’t even get started before but with this switch it started uploading…albeit very slow at 525 KB/s which translates to around 4.2 Mb/s. Meh, as long as I can keep uploading steadily for the next 36 hours I’ll definitely make progress on the project. Thanks!
Worked a treat 🙂 MSFT should add this to the instructions for Azcopy.
To resume the transfer, just run the same .\azCopy.exe command or use a script like:
[int]$Failed = $Result[5].Split(“:”)[1].Trim() $Journal = “$env:LocalAppData\Microsoft\Azure\AzCopy” $i=1 while ($Failed -gt 0) { $i++ [int]$Failed = $Result[5].Split(“:”)[1].Trim() $Result = .\AzCopy.exe /Z:$Journal $Result $i }
This script will continue to run the same .azCopy.exe command until “Transfer Failed” output is down to zero. azcopy tool stores two journal files under %\AppData%\Local\Microsoft\Azure\AzCopy like C:\Users\samb\AppData\Local\Microsoft\Azure\AzCopy folder. It looks them up upon starting to detect if this is a transfer that did not complete and resumes it.
Ref : https://social.technet.microsoft.com/wiki/contents/articles/26528.resuming-timed-out-uploadsdownloads-tofrom-azure-blob-storage-with-azcopy.aspx
Beautiful post – fixed my issue; thank you.
I have a 40mbp/s upload speed, but AzCopy just stuck at 0kb the whole time. When it eventually bombed out with the error above, it had somehow managed to copy one 4MB file, but left a 4GB and a 50MB PST. I think there are serious issues with AzCopy. Having to search Google to fix the most basic of problems is a pain in the ass. Perhaps Microsoft might even manage to make a GUI to do this job one day, and just maybe with friendly error messages as well, and hell, even a “retry” option??? Just a thought…
Maybe someone knows what is maximal size of pst file?
There is no official info about that. I’ve read about 10GB, but nothing confirmed. I’m preparing to migration via pst and have users with even 70GG files.
Thanks Paul,
its so frustrating searching relevant information for azcopy.
Dokumentation and advice for importing pst to office 365 varies in every post. I first stuck at the commandline, since i never read sth that you dont need a key as parameter anymore – just use the dest url.
When it finally started i had the files on an attached usb3 drive and started without your nc:2 switch – broken connection every 15 minutes, average speed 50kb/s.
Moving files on hd and using your switch works fine and leads to 700kb/s – and no loss of connections. Thanks again
After about 10 calls to Microsoft Technical Support and hours of frustration, I finally found this post and it has fixed my problem…..Thank you!! Thank you!! Thank you!!
Thanks Paul! 2 years later and your post is still helping. It’s going to take about 24 hours to complete but it has already gone far beyond where it was failing before I added the /NC switch.
I have a 100Mbps connection, and was getting transfer speeds of 1k. This resolved the problem for me, the transfer speed shot up. (Although still not exactly fast, it was 200k-1MBps)
Would be nice if there was an RETRY option in the command.
Each time you issue a command to AzCopy, it checks whether a journal file exists in the default folder, or whether it exists in a folder that you specified via this option. If the journal file does not exist in either place, AzCopy treats the operation as new and generates a new journal file.
For a Lower connections, NC:2 It’s worked to me. Many thnks!
Thanks – we had the same problem when trying to upload from any on-prem server in any physical location – adding this parameter worked. Any idea what the ‘certain number of concurrent operations’ that AZCopy starts by default is? It would be nice if they would tell us so we have something to gauge it by. FYI – we uploaded our test PST file (1.3 GB in size) to an azure VM and ran the tool and it worked perfectly. The upload was very fast – 2 minutes for a 1.3 GB file.
I think it may be a default of 8, but I couldn’t find a definite answer when I was looking into it.
I actually got a reply from a MS guy – you are kind of correct – the default is 8 per core. I quote from the MS reply “The default thread count is 8*core count, such on a machine with 8 CPU cores, the thread count would be 8*8=64.”
Thank you for the guidance, this worked perfectly!