File Copy - Problems

[venkat.siddareddy]

Dear friends,
I am facing a major problem with File Copy operation in AMS. I have no problems when copying files of sizes less than 5GB. When the file sizes are above 5GB the File.Copy operation takes 10 times more time than normal file copy operation. This happens only when I show the File Copy status using in built status dialog or Progress Bar. With out these the file copying is fast enough. But I would like a way to show the status and also copy files faster.

I tried a different method to overcome this problem. What I did was to perform File Copy with out call back function and also no status dialog. While the file is being copied I wrote a small piece of code in on timer event to check the file size of the file being copied and calculate the percentage of file copied and display the status using Progress Bar.

But the problem here is When I do File.GetSize operation of the file being copied AMS returns 0. So to over come this problem I tried doing a file refresh using SHChangeNotify DLL call but the file is not getting refreshed. I used the below code.

DLL.CallFunction(_SystemFolder.."\\SHELL32.DLL", "SHChangeNotify", "134217728, SHCNE_ALLEVENTS, NULL, NULL", DLL_RETURN_TYPE_LONG, DLL_CALL_STDCALL)

But what I realised is that while file is being copied if I open the explorer and go to the folder in to which the file is being copied and refresh manually, the progress bar starts updating. But I need to keep the explorer window open all the time.

Is there any way to solve this problem?

Please help me with your valuable suggestions.

Thanking you in advance.

Venkat

[Ulrich]

Don't forget that the actions provided in AutoPlay Media Studio are not the only possible way to perform the tasks. They are included in the product to assist you with common tasks, but may not always be the best solution. Sometimes you can get better results if you think about the problem and tackle it with standard Lua code.

Consider this code:

local source = "G:\\dummy.iso";
local destination = "Q:\\dummy.iso";

local size = File.GetSize(source);
Label.SetText("Label1", "File size: " .. size .. " bytes");

-- test 01: built-in File.Copy() action
StatusDlg.Show();
local start = os.time();
File.Copy(source, destination, false, true, false, true);
local stop = os.time();
StatusDlg.Hide();
Label.SetText("Label2", "File.Copy() took " .. (stop-start) .. " seconds");

-- test 02: OS file copy
local cmd = _SystemFolder.."\\cmd.exe";
start = os.time();
File.Run(cmd, "/c COPY /B "..source.." /B "..destination.." /Y", "", SW_SHOWNORMAL, true);
stop = os.time();
Label.SetText("Label3", "CMD COPY took " .. (stop-start) .. " seconds");

-- test 03: Lua I/O file copy
local BUFSIZE = 2^14;
local fInHandle = io.open(source, "rb");
local fOutHandle = io.open(destination, "wb");
local blocks = Math.Floor(size / BUFSIZE);
local remainder = size - (blocks * BUFSIZE);
StatusDlg.Show();
StatusDlg.SetMeterRange(0, 1000);
StatusDlg.SetMeterPos(0);
StatusDlg.SetMessage(source);
start = os.time();
for i = 1, blocks do
   data = fInHandle:read(BUFSIZE);
   fOutHandle:write(data);
   pos = 1000 * i * BUFSIZE / size;
   StatusDlg.SetMeterPos(pos);
   x = string.format("%0.1f%% copied", pos/10);
   StatusDlg.SetStatusText(x);
end
if (remainder > 0) then
   data = fInHandle:read(remainder);
   fOutHandle:write(data);
end
fInHandle:close();
fOutHandle:close();
stop = os.time();
Label.SetText("Label4", "Lua I/O took " .. (stop-start) .. " seconds");
StatusDlg.Hide();

-- tests finished
Button.SetEnabled("Button1", true);

This example shows three different ways to copy a file. I have tested this with a file over 7 GB in size, with the following results:



Indeed, the built-in File.Copy() action is slower than a direct copy in the operation system, where you don't have any visual feedback about the progress. However, this does not mean that you can't get reasonable results and visual feedback with Lua directly, as shown in the third test:



Finally, if you are really experiencing delays ten times as long in AutoPlay Media Studio, then there is something else happening / interfering, because - as you can see - the time difference is not that big as you say.

Ulrich

[venkat.siddareddy]

thanks Ulrich.

will try the third method and post the results.

[naxo]

Ulrich I cant post anything here its says working when click submit or save. I have one question for you

[BioHazard]

First, apologies for bumping such an ancient thread, folks - but I thought this was interesting and worth the bump.

In light of the recent thread request on File.Copy (here), this should be relevant to some. After stumbling upon Ulrich's test code above, I found it quite interesting and ran some tests of my own. Figured some of you might want to try likewise.

So, to make testing a little quicker / more easily accessible, I've embedded Ulrich's code for Test 1 (Built-In File.Copy) and Test 3 (Lua I/O) into an APZ which permits relative file selection. (See attached)

Here are my comparative results for a test on a 902mb file (using WinXP SP3) and copying across HDDs:

Test 1: (Built-In File.Copy)


Test 2: (Lua I/O)


Obviously, the results will vary enormously depending on what system resources you have available at the time, but these results are based on identical system resources being available at execution. Interesting that Lua I/O seems marginally faster.

Nb.
Didn't bother with the Command Line test which will no doubt be fastest of all. Was only interested in methods which graphically display copy progress.

Thanks for the very interesting code, Ulrich. Good stuff!

[naxo]

Hi biohazard from your apz posted here
I got in the test 2 an error:

Error. Attempt to index local "fOutHandle" a nil value while testing an 3.5 gb iso file from the hhd to another external hdd
Line 32

[BioHazard]

Hi biohazard from your apz posted here
I got in the test 2 an error:

Error. Attempt to index local "fOutHandle" a nil value while testing an 3.5 gb iso file...
Line 32

Hmm, okay. Thanks for the feedback. I ran a few more tests, trying to duplicate the error. Used Riz's old Bloat File Maker tool, to create some dummy-files (each with .iso extension) of 100mb, 1gb & 4gb in size:



And then ran tests (copying across HDDs). Each time, executed without problems. Thought at first that Lua I/O might have some issues with large file-sizes. But the 4gb file worked just fine, too:



... and showed the following test result:



The apz seems to work (no hitches) on my system (WinXP SP-3). And have been unable to duplicate the reported error. Do you get the same error with different size / type files? Nb. Have attached a copy of Riz's Bloat File Maker tool below. Use it to make some dummy-files and run some more tests, if you like.

At this stage, can't see any errors in the code. Is basically untouched from Ulrich's original script. Have only replaced original source/destination variables with path variables returned by Dialog.FileBrowse / Dialog.FolderBrowse actions.

But, consider also the following. Error is reported as occurring on Line-32, right?:

-- Line 32
fOutHandle:write(data);

... whereby fOutHandle is the local variable, referenced from Line-20:

-- Line 20
local fOutHandle = io.open(destination, "wb");

... which uses native Lua io.open command to write file to destination. Error reports fOutHandle is coming back as a 'nil' value while iterating over table, indicating 'nothing' to write to destination.

Paying particular attention to the "wb" flag in aforementioned command-parameter, we establish that "wb" means "write binary" (as opposed to plain "w" flag used to write non-binary filetypes). So we know Lua write-command should be good for .iso files. No problemo.

Logically next, we consider possibly the error might originate at Line-19:

-- Line 19
local fInHandle = io.open(source, "rb");

... which uses native Lua io.open command to read file from source. Again, paying attention to flag in command-parameter, "rb" which means "read binary" (as opposed to plain "r" flag used to read non-binary filetypes). So we know Lua read-command also should be good for .iso files. Still, no problemo.

From where then does error originate? Not a clue! And at this stage, has me stumped. Unless error can be duplicated by others (and across different binary filetypes), I can't see any deeper into the problem. Likely, this'll need a resident coding-god to solve?

Sorry, my man - wish I could be of more help.

Maybe Ulrich knows?

[BioHazard]

@naxo

Update:

Okay, found some errors in the code. Still could not duplicate the exact error described. But did find that Line-32 was producing a 'nil return on string' error if attempting to copy to same location as the source file.

Have now coded in a redundancy-check to prevent this. And made some minor mods to remove superfluous code / make code more uniform across both tests. As a result (and for your reference) there's 3 extra lines of code in Test 2. So Line-32 is now Line-35, etc...

Anyway, updated apz is now attached below. Let us know if it's still giving errors.
Cheers.

[naxo]

hI BIO
I found it out that if you copy both methods from an hdd with O.S (usually c:\ directly no subfolder only root dir) installed I could get error, test 1 directly it does give a zero bytes copying and test 2 gives that error I told you earlier. Maybe its a sort of permission write or maybe my antivirus could prevent somewhat their access? Its strange because you can copy whatever hdd disk without o.s installed on it in both methods...

[BioHazard]

Update:

Okay, have now thoroughly refined all the code for maximum efficiency. Much improved - does not appear to be throwing ANY more errors. And code is now clearly 'commented' throughout for easier reference (see attached). Is actually looking 'very pretty', LOL -now that it's had my full attention!

Nb..
One thing I couldn't figure out, though. Added a Cancel button to the StatusDlg, viz:

StatusDlg.ShowCancelButton(true, "Cancel");

... in the File.Copy() test (which works just dandy). But when adding it to the StatusDlg in the Lua I/O test, it doesn't function (at all). Does anyone know why that is?

............
@naxo,
Well, I guess we'll just wait to see if other users turn up any bugs in the code. My system doesn't 'appear' to be throwing any more error-messages. But that of course doesn't make it definitive. Just saying, can't find any more problems, myself.

[naxo]

hey biohazard I tested your last code snippets and Im still having that same kind of nil error for both codes I will test in another laptop and I will tell the results ...

[BioHazard]

hey biohazard I tested your last code snippets and Im still having that same kind of nil error for both codes ...

If you're getting the same error on BOTH tests, it kinda sounds more like a 'write-permissions' issue. Am assuming you're using Win7 of Win10, correct? Am using WinXP here (I know, I'm a dinosaur right?) so don't have these issues. Would be helpful if someone else with Win7/Win10 could check to see if they're getting the same error.

Have you tried running the apz with a 'Test for error' code inserted? Try sticking this code-block into both tests:

-- Test for error
error = Application.GetLastError();
if (error ~= 0) then
    Dialog.Message("Error", _tblErrorMessages[error], MB_OK, MB_ICONEXCLAMATION);
end

In Test-1, place it at: Line-43 after the File.Copy command
In Test-2, place it at Line-42 after local fOutHandle = io.open(destination, "wb") and at Line-55 after fOutHandle:write(data)

... And then let us know what happens, yeah?

Nb.
The above LineNumber references apply to the latest apz, File.Copy Tests (refined & improved).apz

[colc]

Hi Bio ,
Tested on Win10 x64 No Probs opened copied files with UltraISO OK


Cheers

[naxo]

Bio I can confirm in another lapton win same o.s and I got no issues at all, strange you are right its a write permission problem :0

[BioHazard]

@colc,
@naxo,
Beer & skittles for everyone, then? 👍