Tags: 512mb, anexperiment, calls, code, dell, error, experiencing, ifft, matlab, out-of-memory, production, programming, running, strange, win, workstation

ifft Out-Of-Memory error after 640 calls

On Programmer » Matlab

13,895 words with 5 Comments; publish: Tue, 06 May 2008 17:06:00 GMT; (20078.13, « »)

Hello,

We're experiencing a strange problem in a production code running an

experiment.

Matlab 7.0.1 on a Dell P4 workstation Win XP 512MB RAM.

Experiment does many scans, sequence is like this

Step - Acquire - Save - Analyze - Display - Repeat

Each "Save" saves 10 datafiles. Each "Analyze" performs 2 calls to

fft() and 4 calls to ifft(). Data length at present is 147456 =

16348*9 = 2^14 * 3^2

After 160 steps, we've written 1600 datafiles, performed 320 fft()

calls and 640 ifft() calls. And then we get an out-of-memory error.

Occasionally 161 or 162 steps, but we always get an error.

OK, so what have we tried so far ?

1. Tried pack. No difference.

2. Looked at feature('dumpmem'), always reports 800MB+ free contiguous

space. No evidence of memory leak. Now this number is greater than

physical memory installed, I'm guessing there's a good explanation for

this in terms of virtual memory.

3. Wondered about Windows issues, maybe putting too many files in a

directory. Rewrote the file save portion so we put approx 100 files

max in a single directory. No change in behavior.

I'm looking for (a) suggestions as to what might be going on (b)

possible workarounds to try. I read some old threads re: fftw and

plans and such, definitely we'll try modifying the data length, see if

that has any impact. I wonder if there may be alternate strategies and

functions for in R7 for repetitive fixed-length fft/ifft calls, hoping

someone experienced with this would be able to drop a hint !

Thanks for *any* suggestions,

-rajeev-

All Comments

Leave a comment...

  • 5 Comments
    • I have personally done hundreds of thousands of ifft calls in one

      matlab session without problems, so i'm suspecting that the problem

      is somewhere else in the code. My suggestion is to keep commenting

      things out until the bug dissapears. The acquire step seems like the

      most likely place where the problem might be. Can you try writing

      some code that fakes data output from the acquire step?

      Some other things that you might want to try to see if they help:

      clear as many variables as you can after each step... make sure all

      files are closed... You can use fclose('all').

      Aslak Grinsted

      <http://www.glaciology.net/ag>

      #1; Tue, 06 May 2008 17:07:00 GMT
    • Aslak Grinsted wrote:

      > I have personally done hundreds of thousands of ifft calls in one

      > matlab session without problems, so i'm suspecting that the problem

      > is somewhere else in the code. My suggestion is to keep commenting

      > things out until the bug dissapears. The acquire step seems like the

      > most likely place where the problem might be. Can you try writing

      > some code that fakes data output from the acquire step?

      > Some other things that you might want to try to see if they help:

      > clear as many variables as you can after each step... make sure all

      > files are closed... You can use fclose('all').

      > Aslak Grinsted

      > <http://www.glaciology.net/ag>

      Aslak,

      Thanks for your input and these suggestions. We should definitely

      check that files are being closed. The thing with clear is that the

      Analysis occurs in a function call, once the function is terminated

      presumably there's nothing left to clear ! (We did have the idea to

      try 'clear functions' ... we may not have tried it yet.) Besides

      Analysis, there is a GUI with pretty modest memory footprint, and other

      things are C programs executed by dos('abc.exe') type calls. I guess

      we should check if we're doing something in plotting the data that's

      causing the memory requirement of the GUI figure to grow... Running

      the code without acquiring the data, that also seems like a really good

      thing to try.

      Thanks !

      Regards,

      -rajeev-

      #2; Tue, 06 May 2008 17:08:00 GMT
    • What exactly is running out of memory? Is it java or A MATLAB function?

      Can you post the out of memory error message?

      MATLAB 7.1 can use a different memory manager try setting the environment

      variable MATLAB_MEM_MGR to fast before starting matlab see

      [url]http://www.mathworks.com/support/solutions/data/1-1CAJH.html?solution=1-1CAJH[/url

      ]

      or matlab_install_dir\bin\matlab.bat for more information.

      "Rajeev" <rrr.matlab.todaysummary.com.ieee.org> wrote in message

      news:1123121369.409069.36490.matlab.todaysummary.com.g47g2000cwa.googlegroups.com...

      > Hello,

      > We're experiencing a strange problem in a production code running an

      > experiment.

      > Matlab 7.0.1 on a Dell P4 workstation Win XP 512MB RAM.

      > Experiment does many scans, sequence is like this

      > Step - Acquire - Save - Analyze - Display - Repeat

      > Each "Save" saves 10 datafiles. Each "Analyze" performs 2 calls to

      > fft() and 4 calls to ifft(). Data length at present is 147456 =

      > 16348*9 = 2^14 * 3^2

      > After 160 steps, we've written 1600 datafiles, performed 320 fft()

      > calls and 640 ifft() calls. And then we get an out-of-memory error.

      > Occasionally 161 or 162 steps, but we always get an error.

      > --

      > OK, so what have we tried so far ?

      > 1. Tried pack. No difference.

      > 2. Looked at feature('dumpmem'), always reports 800MB+ free contiguous

      > space. No evidence of memory leak. Now this number is greater than

      > physical memory installed, I'm guessing there's a good explanation for

      > this in terms of virtual memory.

      > 3. Wondered about Windows issues, maybe putting too many files in a

      > directory. Rewrote the file save portion so we put approx 100 files

      > max in a single directory. No change in behavior.

      > --

      > I'm looking for (a) suggestions as to what might be going on (b)

      > possible workarounds to try. I read some old threads re: fftw and

      > plans and such, definitely we'll try modifying the data length, see if

      > that has any impact. I wonder if there may be alternate strategies and

      > functions for in R7 for repetitive fixed-length fft/ifft calls, hoping

      > someone experienced with this would be able to drop a hint !

      > Thanks for *any* suggestions,

      > -rajeev-

      >

      #3; Tue, 06 May 2008 17:09:00 GMT
    • ---

      "Rajeev" <rrr.matlab.todaysummary.com.ieee.org> writes:

      > Hello,

      > We're experiencing a strange problem in a production code running an

      > experiment.

      > Matlab 7.0.1 on a Dell P4 workstation Win XP 512MB RAM.

      > Experiment does many scans, sequence is like this

      > Step - Acquire - Save - Analyze - Display - Repeat

      > Each "Save" saves 10 datafiles. Each "Analyze" performs 2 calls to

      > fft() and 4 calls to ifft(). Data length at present is 147456 =

      > 16348*9 = 2^14 * 3^2

      > After 160 steps, we've written 1600 datafiles, performed 320 fft()

      > calls and 640 ifft() calls. And then we get an out-of-memory error.

      > Occasionally 161 or 162 steps, but we always get an error.

      I'm not aware of any memory usage problems related to MATLAB's FFT functions

      . I

      suggest that you try this:

      and run your code. When the out-of-memory error occurs, you'll be put into

      the

      debugger at that point, so you can poke around to see if there's anything fu

      nny

      about the variables in scope at that point.

      Steve Eddins

      Development Manager, Image Processing Group

      The MathWorks, Inc.

      #4; Tue, 06 May 2008 17:10:00 GMT
    • Steve Eddins wrote:

      > ---

      > "Rajeev" <rrr.matlab.todaysummary.com.ieee.org> writes:

      > I'm not aware of any memory usage problems related to MATLAB's FFT functio

      ns. I

      > suggest that you try this:

      >

      > and run your code. When the out-of-memory error occurs, you'll be put int

      o the

      > debugger at that point, so you can poke around to see if there's anything

      funny

      > about the variables in scope at that point.

      > --

      > Steve Eddins

      > Development Manager, Image Processing Group

      > The MathWorks, Inc.

      Gentlemen,

      Thank you both for your interest and suggestions. I regret how long

      it's taken me to get back to this thread.

      We have done and tried a number of things following upon your

      suggestions and Aslak's earlier. we are continuing to experience this

      problem, however we are now satisfied that ifft() is not implicated.

      ====

      #1 (per Aslak) running the scan without the acquisition : no problem

      observed.

      ====

      #2 (per Philip) I have no idea what is running out of memory. The

      error message is copied below. Note that function('dump_mem') reports

      about 800 MB free space both before and after the error. (per Steve) I

      did poke around in the debugger, everything looks normal up the call

      trace, in particular the argument to ifft is real and has the right

      size 147456 x 1. I am at a loss to understand how it may be possible

      for an out-of-memory message to be generated when so much memory is

      available.

      ---

      The Out-Of-Memory Error Message

      ---

      <...snip...>

      AnalyzeOneTrace 0.390s Done

      ReadHeader0010: Read header from

      [C:\MATLAB701\work\scan\tmpdata\D1721.dat]

      AnalyzeOneTrace 0.389s Done

      ReadHeader0010: Read header from

      [C:\MATLAB701\work\scan\tmpdata\D1731.dat]

      AnalyzeOneTrace 0.375s Done

      ReadHeader0010: Read header from

      [C:\MATLAB701\work\scan\tmpdata\D1741.dat]

      ? Error using ==> ifft

      Out of memory. Type HELP MEMORY for your options.

      Error in ==> <a

      href=" error:C:\MATLAB701\work\scan\AnalyzeOneT

      race.m,199,1">AnalyzeOneTrace

      at 199</a>

      vDataBPh = 2*imag(ifft(wData.*uBandSS)); % Use

      single sided mask to generate imaginary term

      Error in ==> <a

      href="error:C:\MATLAB701\work\scan\pointzmap.m,14,1">pointzmap at

      14</a>

      [EnvelopeA sDAQ] = AnalyzeOneTrace(filename);

      Error in ==> <a

      href=" error:C:\MATLAB701\work\scan\DataCollect

      .m,934,1">DataCollect>autoscan

      _Callback

      at 934</a>

      x2 = pointzmap(filename);

      Error in ==> <a

      href=" error:C:\MATLAB701\toolbox\matlab\uitool

      s\gui_mainfcn.m,75,1">gui_main

      fcn

      at 75</a>

      feval(varargin{:});

      Error in ==> <a

      href=" error:C:\MATLAB701\work\scan\DataCollect

      .m,44,1">DataCollect at

      44</a>

      gui_mainfcn(gui_State, varargin{:});

      ? Error while evaluating uicontrol Callback.

      ====

      #3 (per Philip) we have not tried -fast memory manager yet. The

      reference to version 7.1 us, seeing now that the latest

      version listed on TMW website is 7.0.4 we're assuming that's a typo. I

      have 7.0.3 but haven't installed it on this machine.

      #4 Other things we've explored

      We were concerned about repetitively writing data to the GUI figure, so

      we turned that off, made no difference. Tried increasing the amount of

      virtual memory, error moved from iteration number 1600 to 1740. Oddly

      enough it is still at 1740 even though we decreased virtual memory back

      to 1.5GB and rebooted. Reduced amount of data acquired and analyzed,

      the error occurred at a later iteration count in inverse proportion.

      Tried clear functions (per Aslak), made no difference.

      ***MORE INFO***

      I need to tell more about our acquisition setup. The acquisition is a

      standalone C program running acquisition from a National Instruments M

      Series card using the daqmx library. Communication with Matlab is via

      external engine calls (COM interface); the Matlab session is invoked

      with the /automation command. Recently we have noticed that the COM

      interface between programs appears to work even from a 'normal' Matlab

      session, ie no /automation flag. We've also been noticing that often

      as not it's the acquisition program that terminates abnormally. (Our

      earlier indications had been that the acquisition program failure was

      relatively rare whereas the ifft memory error happened "always") So

      now our suspicions are centered on the COM interface.

      Also, the first item in our repetitive sequence is called "Step" ie

      move to the next point in the scan, and involves sending an RS232

      command to an external device. This RS232 is implemented using a Matlab

      serial port object. It also shows occasional timeout errors which it

      should not.

      And here's what ver reports...

      ----

      --

      MATLAB Version 7.0.1.24704 (R14) Service Pack 1

      MATLAB License Number: XXXXXX

      Operating System: Microsoft Windows XP Version 5.1 (Build 2600: Service

      Pack 2)

      Java VM Version: Java 1.4.2_04 with Sun Microsystems Inc. Java

      HotSpot(TM) Client VM

      ----

      --

      So here are some other things we're planning to try soon:

      5. run acquisition from a 'normal' session

      6. run acquisition program by itself, no Matlab or COM interface

      7. close and reopen the COM engine connection periodically

      The COM interface is kind of a black box mystery at this point, if any

      of you have pointers on where to start diagnosing its problems, that

      would sure be helpful.

      I'd also like to have pointers to understand what if any differences

      there are between a Matlab instance running with the /automation flag

      vs a normal session.

      Thanks !

      -rajeev-

      #5; Tue, 06 May 2008 17:11:00 GMT