[molpro-user] possible bug: moldpro not releasing deleted scratch files

[Jörg Saßmannshausen]

Dear Andy,

thanks for the feedback and the explanation. 

What puzzled me was that a normal ls -l command did not show these files and 
even the du -sh command did not 'see' them. It is only when I done a df 
command (and the lsof) that I found them. Hence I believed they were deleted 
by the program but the file handle has not been released. 

Shame for my calculation. I would have thought that 1 TB of scratch is enough 
here. So the conclusion is I cannot to it the way I want it. Just my luck 
again.

Thanks for your help!

All the best from a grey and cold London

Jörg

On Thursday 21 February 2013 10:26:24 Andy May wrote:
> Jörg,
> 
> When Molpro opens a scratch file it immediately calls unlink() for that
> file. This means the file is scheduled for deletion, but postponed until
> either all processes close the file (i.e. when Molpro's finished with
> them), or when the processes linked to them terminate. So the reason
> these files are not actually deleted until you kill the process is
> likely because they are still being used. My guess is that your
> calculation really does need this amount of disk space.
> 
> The reason we use unlink() instead if putting in a call to delete the
> file is to cover the case of the program crashing, or the user killing
> the job. In both of these cases without the call to unlink() the files
> would remain as permanent files on the disk since the file delete code
> would have never been reached. Of course that's no problem on clusters
> when the batch scheduler has been set up to clean scratch, but in most
> other cases the scratch files would otherwise have to be removed manually.
> 
> Best wishes,
> 
> Andy
> 
> On 20/02/13 22:18, Jörg Saßmannshausen wrote:
> > Dear all,
> > 
> > I have come across something highly anoying in Molpro: When I am doing a
> > rather large LCCSD(T)/cc-pVTZ calculation, the program does not release
> > the deleted files and thus is filling up the scratch space:
> > 
> > df /scr/
> > Dateisystem           Size  Used Avail Use% Eingehängt auf
> > /dev/md0              932G  932G   88K 100% /scr
> > 
> > but:
> > du -sh /scr/molpro/
> > 35G     /scr/molpro/
> > 
> > The program crashes as the scratch is full but one process is still
> > running. When I am then doing a lsof (output attached), I see a number
> > of these files:
> > 
> > molpro.ex  2452       sassy   19u      REG    9,0          73  134219798
> > /scr/molpro/init_job5000002452 (deleted)
> > 
> > Once I kill the program in the queue, it is releasing the deleted files
> > and I get my space back.
> > 
> > $ df /scr/
> > Dateisystem           Size  Used Avail Use% Eingehängt auf
> > /dev/md0              932G   39G  893G   5% /scr
> > 
> > This is not what I expect.  I would have thought once the file is deleted
> > it is released to the file system can remove it. As that is not
> > happening, the scratch of 1 TB is filling up and my calculation crashes
> > :-(
> > 
> > I am using that version of Molpro.
> > 
> >   SHA1      : 2c68d29c09da70e1723824271fadde4bcd5f07a0
> >   ARCHNAME  : Linux/x86_64
> >   FC        : /opt/intel/compilerpro-12.0.2.137/bin/intel64/ifort
> >   FCVERSION : 12.0.2
> > 
> > Is that a bug or is there something I am doing wrong here. I am using xfs
> > for scratch.
> > 
> > All the best from London!
> > 
> > Jörg
> > 
> > 
> > 
> > _______________________________________________
> > Molpro-user mailing list
> > Molpro-user at molpro.net
> > http://www.molpro.net/mailman/listinfo/molpro-user

-- 
*************************************************************
Jörg Saßmannshausen
University College London
Department of Chemistry
Gordon Street
London
WC1H 0AJ 

email: j.sassmannshausen at ucl.ac.uk
web: http://sassy.formativ.net

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html