what is "Not enough quota is available to process this command" ?

[Michael Mol]

Recently my MFC application is giving a strange error
System error code 1816
"Not enough quota is available to process this command."
one of the functionality  of the program is hash ( MD5 and SHA1 )  the
incoming file , copy the file ,
hash   ( MD5 and SHA1 ) it again , see if both hashes matches .
when i turn this feature off the error doesn't reproduce . i am hashing the
content .
Any idea why such issues comes ?
The memory foootprint of the program was 27 MB at the time of error ..

Very likely, you're exceeding the disk quota of the user with your
file copy. It sounds like this is a data archival/backup feature.
Essentially, your target volume has run out of disk space the computer
(or network server) is willing to allot to your user. Suggested
solutions: Compress your data prior to storing it on the archival
target (You can use streaming decompression to check the integrity of
the data, or you can pass a copy of the data through your hash
function as you feed the compression algorithm for the archived
file.), periodically purge old archived data, reduce the amount of
data you're archiving, or get a bigger disk quota. Or all three.

[Nathan Mates]

Recently my MFC application is giving a strange error
System error code 1816
"Not enough quota is available to process this command."

Usual cause: out of memory, or out of memory to service the
allocation requested. I'd suspect you're getting this message when
trying to allocate memory to read in a large file all at once. Don't
do that. Read in small (1-4MB chunks), and process the chunks in
order.

On a standard 32-bit Windows environment, you can get about 75-80%
of the 2GB address space. But, the largest contiguous memory chunk you
can allocate is probably only in the 800MB range. If you really need
more, and streaming in chunks isn't doable, then go to a 64-bit OS and
application.

Nathan Mates
--
<*> Nathan Mates - personal webpage http://www.visi.com/~nathan/
# Programmer at Pandemic Studios -- http://www.pandemicstudios.com/
# NOT speaking for Pandemic Studios. "Care not what the neighbors
# think. What are the facts, and to how many decimal places?" -R.A. Heinlein

[jjoohhnn]

Nathan,

Read in small (1-4MB chunks), and process the chunks in order.

I would like to know about this chunk size while processing huge
data (from files). To get better(high like cpu, memory) performance on win
32 applications, which is the chunk size?

How can we identify the each chunk with unique value? But
performance should be very high.

what is the difference between fat & ntfs file systems? Is there any
other file systems are there?

If this is off topic, please guide me to correct forum.

Thanks in advance.

Regards,
John.

Recently my MFC application is giving a strange error
System error code 1816
"Not enough quota is available to process this command."

Usual cause: out of memory, or out of memory to service the
allocation requested. I'd suspect you're getting this message when
trying to allocate memory to read in a large file all at once. Don't
do that. Read in small (1-4MB chunks), and process the chunks in
order.
On a standard 32-bit Windows environment, you can get about 75-80%
of the 2GB address space. But, the largest contiguous memory chunk you
can allocate is probably only in the 800MB range. If you really need
more, and streaming in chunks isn't doable, then go to a 64-bit OS and
application.
Nathan Mates
--
<*> Nathan Mates - personal webpage http://www.visi.com/~nathan/
# Programmer at Pandemic Studios -- http://www.pandemicstudios.com/
# NOT speaking for Pandemic Studios. "Care not what the neighbors
# think. What are the facts, and to how many decimal places?" -R.A. Heinlein

[Nathan Mates]

Post by jjoohhnn

Read in small (1-4MB chunks), and process the chunks in order.

I would like to know about this chunk size while processing huge
data (from files). To get better(high like cpu, memory) performance on win
32 applications, which is the chunk size?

The ideal/optimum chunk size (which you seem to be asking for)
probably depends on your HD, what other applications are running at
the same time, memory, etc. Write your code with a tuning value that
adjusts the chunk size. Then, experiment.

Post by jjoohhnn
How can we identify the each chunk with unique value? But
performance should be very high.

I don't know what you mean by "unique value" here. You'll have to
be clearer. Much clearer. If you want performance, the best thing to
do is probably some form of overlapped I/O. Look at CreateFile --
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx ,
especially the FILE_FLAG_NO_BUFFERING & FILE_FLAG_SEQUENTIAL_SCAN
flags. (And, pay attention to all the comments as to the special work
you must do when using the first!) Then call ReadFile --
http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx with an
overlapped handle. Going double-buffered where you request reads into
a second buffer when you're processing the first might help as well.

Post by jjoohhnn
what is the difference between fat & ntfs file systems? Is there any
other file systems are there?

Get it running, first. Then make it fast. You've not managed to get
your code running for huge files. Caring about FAT vs NTFS is
irrelevant at this point. You're committing the error of premature
optimization, or premature "what if" nonsense. Working code that's not
stupidly slow is far better than thoughts on paper that care about
every detail under the sun.

Nathan Mates
--
<*> Nathan Mates - personal webpage http://www.visi.com/~nathan/
# Programmer at Pandemic Studios -- http://www.pandemicstudios.com/
# NOT speaking for Pandemic Studios. "Care not what the neighbors
# think. What are the facts, and to how many decimal places?" -R.A. Heinlein

[jjoohhnn]

Nathan,

Post by Nathan Mates
The ideal/optimum chunk size (which you seem to be asking for)
probably depends on your HD, what other applications are running at
the same time, memory, etc. Write your code with a tuning value that
adjusts the chunk size. Then, experiment.

Chunk size should be constant for our applicaiton. So, I am sorry. I
cannot go with this. We had given constant value for that. It is running
fine without cpu & memory issues.

Post by Nathan Mates
I don't know what you mean by "unique value" here. You'll have to be
clearer. Much clearer.

Checksum value for each chunk like MD5

Post by Nathan Mates
You've not managed to get your code running for huge files.

I can manage for huge files. I just want to know VC-Experts
views. Just for that.

I would like to know the difference between FAT & NTFS. Why Volume
Shadow Copies will not support FAT, Is it true?

Regards,
John

Post by Nathan Mates

Post by jjoohhnn

Read in small (1-4MB chunks), and process the chunks in order.

I would like to know about this chunk size while processing huge
data (from files). To get better(high like cpu, memory) performance on win
32 applications, which is the chunk size?

The ideal/optimum chunk size (which you seem to be asking for)
probably depends on your HD, what other applications are running at
the same time, memory, etc. Write your code with a tuning value that
adjusts the chunk size. Then, experiment.

Post by jjoohhnn
How can we identify the each chunk with unique value? But
performance should be very high.

I don't know what you mean by "unique value" here. You'll have to
be clearer. Much clearer. If you want performance, the best thing to
do is probably some form of overlapped I/O. Look at CreateFile --
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx ,
especially the FILE_FLAG_NO_BUFFERING & FILE_FLAG_SEQUENTIAL_SCAN
flags. (And, pay attention to all the comments as to the special work
you must do when using the first!) Then call ReadFile --
http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx with an
overlapped handle. Going double-buffered where you request reads into
a second buffer when you're processing the first might help as well.

Post by jjoohhnn
what is the difference between fat & ntfs file systems? Is there any
other file systems are there?

Get it running, first. Then make it fast. You've not managed to get
your code running for huge files. Caring about FAT vs NTFS is
irrelevant at this point. You're committing the error of premature
optimization, or premature "what if" nonsense. Working code that's not
stupidly slow is far better than thoughts on paper that care about
every detail under the sun.
Nathan Mates
--
<*> Nathan Mates - personal webpage http://www.visi.com/~nathan/
# Programmer at Pandemic Studios -- http://www.pandemicstudios.com/
# NOT speaking for Pandemic Studios. "Care not what the neighbors
# think. What are the facts, and to how many decimal places?" -R.A. Heinlein

[Sachin]

hi all,

the typical workflow of my application is .

1. Iteratively go inside given directory
2. Pick up one file
3. read various attributes of the file store them
4. Do MD5 & SHA1 hash of the file
5. Copy the file to give destination folder
6. Apply original timestamp and file attributes
7. Do MD5 & SHA1 hash again.
8. Check if hashes match .( this is core requirement of the tool to make
sure that source and destination files are exactly same.
9. Do this for all files.

i am using SDK's cryptograhy libraries for hashing "CryptoAPI functions"

I have two version of this tool
1.Console mode
2. UI Mode ( MFC )

both offer same set of features
but i find the issue in UI mode ( MFC program ) and not in Console versio n
of it ..

another point is when i set the MD5 hashing OFF then there is no error

Post by Nathan Mates

Post by jjoohhnn

Read in small (1-4MB chunks), and process the chunks in order.

I would like to know about this chunk size while processing huge
data (from files). To get better(high like cpu, memory) performance on win
32 applications, which is the chunk size?

The ideal/optimum chunk size (which you seem to be asking for)
probably depends on your HD, what other applications are running at
the same time, memory, etc. Write your code with a tuning value that
adjusts the chunk size. Then, experiment.

Post by jjoohhnn
How can we identify the each chunk with unique value? But
performance should be very high.

I don't know what you mean by "unique value" here. You'll have to
be clearer. Much clearer. If you want performance, the best thing to
do is probably some form of overlapped I/O. Look at CreateFile --
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx ,
especially the FILE_FLAG_NO_BUFFERING & FILE_FLAG_SEQUENTIAL_SCAN
flags. (And, pay attention to all the comments as to the special work
you must do when using the first!) Then call ReadFile --
http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx with an
overlapped handle. Going double-buffered where you request reads into
a second buffer when you're processing the first might help as well.

Post by jjoohhnn
what is the difference between fat & ntfs file systems? Is there any
other file systems are there?

Get it running, first. Then make it fast. You've not managed to get
your code running for huge files. Caring about FAT vs NTFS is
irrelevant at this point. You're committing the error of premature
optimization, or premature "what if" nonsense. Working code that's not
stupidly slow is far better than thoughts on paper that care about
every detail under the sun.
Nathan Mates
--
<*> Nathan Mates - personal webpage http://www.visi.com/~nathan/
# Programmer at Pandemic Studios -- http://www.pandemicstudios.com/
# NOT speaking for Pandemic Studios. "Care not what the neighbors
# think. What are the facts, and to how many decimal places?" -R.A. Heinlein