How to Calculate Copy Process Time

[Sachin]

Hi

What is the efficient way of calculatring the the required to copy X MB of
data ?
in my program i have information of total amount of data to be copied in MB
, and i want to nofity user the tentative time it will take.

The copy progresses in separate thread , and after each file copy it
notofies other thread currently copied file size , then the other threads
calculates the total % of data copied and moves progress bar ...
now in this UI threads i also have to show how long the operation will take
..


Thanks
sachin

[Alex Blekhman]

Post by Sachin
What is the efficient way of calculatring the the required to
copy X MB of data ?

There is none. Accurate estimation of remaining time can be very
hard task, since speed fluctuations are unpredictable most of the
time. It is especially true when you copy over the network.
Usually people do linear interpolation of current transfer speed
with some averaging in order to smooth out the unusual
peaks/drops.

HTH
Alex

[Nathan Mates]

Post by Sachin
What is the efficient way of calculatring the the required to copy X
MB of data ? in my program i have information of total amount of
data to be copied in MB, and i want to nofity user the tentative
time it will take.

Some sort of linear interpolation is usually used. Basically,
periodically note down that the first N MB copied has taken Y
seconds. So, 2*N MB would take 2*Y seconds. For example, show
"copying..." for the first 2-3 seconds, then measure progress at that
point, and do the math to figure out the remaining time. Every 2-3
seconds, redo your calculations.

Whatever you do, it won't be much worse than Windows Explorer's
guesstimating on file copy times, which can show values from "a few
days" to "10 seconds remaining" very quickly.

Nathan Mates
--
<*> Nathan Mates - personal webpage http://www.visi.com/~nathan/
# Programmer at Pandemic Studios -- http://www.pandemicstudios.com/
# NOT speaking for Pandemic Studios. "Care not what the neighbors
# think. What are the facts, and to how many decimal places?" -R.A. Heinlein

[Tamas Demjen]

Post by Nathan Mates
Whatever you do, it won't be much worse than Windows Explorer's
guesstimating on file copy times, which can show values from "a few
days" to "10 seconds remaining" very quickly.

That's a good implementation. I have a couple of suggestions to the OP:

1. Don't display high precision values, such as 732.134278 seconds. It's
not a good idea to display more digits than what you know accurately.
Since you know it's a guess, you may want to start with a ballpark
figure (like "a few minutes", or "about half an hour"), go down to
minute resolution as you get more data, and when there's less than a
minute, go down to 10-second resolution, or say "less than a minute". If
the estimate keeps fluctuating too much, reduce the accuracy until it
gets stable again.

2. Don't update the estimate very frequently. It's annoying when it
keeps jumping up and down a few times a second. If the estimate changes
sharply, wait a few seconds before you display it -- maybe it's just a
brief anomaly.

It depends, however, whether you develop airplane/atomic reactor
instrumentation, or a consumer product. And on how much time you have
for this feature, because what I outlined above is not necessarily easy
to implement and test.

Tom

[Sachin]

Thanks all suggestions were very informative . .

I just wonder to what extent these calculations affect my overall throughput.

I will have to this calculation periodically after X Seconds in a different
thread ( UI THREAD ) and the copy is going in Worker thread , so this
involve periodic context switching and doing extra calculation ..

if not periodically can we , given X MB determine how long it will take to
copy it on external USB Drive ? Approximation will do if we cant predict
accurate ..

Post by Tamas Demjen

Post by Nathan Mates
Whatever you do, it won't be much worse than Windows Explorer's
guesstimating on file copy times, which can show values from "a few
days" to "10 seconds remaining" very quickly.

1. Don't display high precision values, such as 732.134278 seconds. It's
not a good idea to display more digits than what you know accurately.
Since you know it's a guess, you may want to start with a ballpark
figure (like "a few minutes", or "about half an hour"), go down to
minute resolution as you get more data, and when there's less than a
minute, go down to 10-second resolution, or say "less than a minute". If
the estimate keeps fluctuating too much, reduce the accuracy until it
gets stable again.
2. Don't update the estimate very frequently. It's annoying when it
keeps jumping up and down a few times a second. If the estimate changes
sharply, wait a few seconds before you display it -- maybe it's just a
brief anomaly.
It depends, however, whether you develop airplane/atomic reactor
instrumentation, or a consumer product. And on how much time you have
for this feature, because what I outlined above is not necessarily easy
to implement and test.
Tom

[Scot T Brennecke]

Such predictions are almost always inaccurate anyway. Note that the hardware involved is almost assuredly going to be the
bottleneck, so any CPU time consumed in calculating will have negligible, if any, impact on throughput.

Post by Sachin
Thanks all suggestions were very informative . .
I just wonder to what extent these calculations affect my overall throughput.
I will have to this calculation periodically after X Seconds in a different
thread ( UI THREAD ) and the copy is going in Worker thread , so this
involve periodic context switching and doing extra calculation ..
if not periodically can we , given X MB determine how long it will take to
copy it on external USB Drive ? Approximation will do if we cant predict
accurate ..

Post by Tamas Demjen

Post by Nathan Mates
Whatever you do, it won't be much worse than Windows Explorer's
guesstimating on file copy times, which can show values from "a few
days" to "10 seconds remaining" very quickly.

1. Don't display high precision values, such as 732.134278 seconds. It's
not a good idea to display more digits than what you know accurately.
Since you know it's a guess, you may want to start with a ballpark
figure (like "a few minutes", or "about half an hour"), go down to
minute resolution as you get more data, and when there's less than a
minute, go down to 10-second resolution, or say "less than a minute". If
the estimate keeps fluctuating too much, reduce the accuracy until it
gets stable again.
2. Don't update the estimate very frequently. It's annoying when it
keeps jumping up and down a few times a second. If the estimate changes
sharply, wait a few seconds before you display it -- maybe it's just a
brief anomaly.
It depends, however, whether you develop airplane/atomic reactor
instrumentation, or a consumer product. And on how much time you have
for this feature, because what I outlined above is not necessarily easy
to implement and test.
Tom

[Ben Voigt [C++ MVP]]

Post by Sachin
Thanks all suggestions were very informative . .
I just wonder to what extent these calculations affect my overall throughput.
I will have to this calculation periodically after X Seconds in a different
thread ( UI THREAD ) and the copy is going in Worker thread , so this
involve periodic context switching and doing extra calculation ..
if not periodically can we , given X MB determine how long it will take to
copy it on external USB Drive ? Approximation will do if we cant predict
accurate ..

For any rotating media, the primary thing that controls the time is the
number of other tasks accessing the same disk. Because the heads then spend
more time seeking than actually copying. I really wish Windows Explorer
would implement an intelligent queue (and by this I mean use some new Win32
API that knows the necessary internal information) for operations involving
the same physical rotating disk and execute them sequentially instead of in
parallel.

I really hate that when copying 300MB files to a USB flash drive, each copy
takes 45 seconds if I wait for the prior one to finish. But if I try to
copy files from three different directories (so I can't multi-select), then
copying 3*300MB takes about 10 minutes, from my internal hard disk seeking
between the three files continuously.

Post by Sachin

Post by Tamas Demjen

Post by Nathan Mates
Whatever you do, it won't be much worse than Windows Explorer's
guesstimating on file copy times, which can show values from "a few
days" to "10 seconds remaining" very quickly.

1. Don't display high precision values, such as 732.134278 seconds. It's
not a good idea to display more digits than what you know accurately.
Since you know it's a guess, you may want to start with a ballpark
figure (like "a few minutes", or "about half an hour"), go down to
minute resolution as you get more data, and when there's less than a
minute, go down to 10-second resolution, or say "less than a minute". If
the estimate keeps fluctuating too much, reduce the accuracy until it
gets stable again.
2. Don't update the estimate very frequently. It's annoying when it
keeps jumping up and down a few times a second. If the estimate changes
sharply, wait a few seconds before you display it -- maybe it's just a
brief anomaly.
It depends, however, whether you develop airplane/atomic reactor
instrumentation, or a consumer product. And on how much time you have
for this feature, because what I outlined above is not necessarily easy
to implement and test.
Tom