aklog: Couldn't determine realm of user:aklog: No credentials cache found  while getting realm
/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/pyfits/__init__.py:22: PyFITSDeprecationWarning: PyFITS is deprecated, please use astropy.io.fits
  PyFITSDeprecationWarning)  # noqa

Likelihood Analysis Tool
Support Contact: Daniel Kocevski (daniel.kocevski@nasa.gov)


Waiting 215 seconds before starting...
Proceeding.

Importing modules...
This is make3FGLxml version 01r0.
The default diffuse model files and names are for pass 8 and assume you have v10r00p05 of the Fermi Science Tools or higher.
This is make4FGLDR3xml version 01r09.
The default diffuse model files and names are for P8R3, 4FGL DRs, and the default fermitools directory locations.

WARNING: version mismatch between CFITSIO header (v3.43) and linked library (v3.41).

/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/matplotlib/cbook/deprecation.py:106: MatplotlibDeprecationWarning: The mpl_toolkits.axes_grid module was deprecated in version 2.1. Use mpl_toolkits.axes_grid1 and mpl_toolkits.axisartist provies the same functionality instead.
  warnings.warn(message, mplDeprecation, stacklevel=1)

WARNING: version mismatch between CFITSIO header (v3.43) and linked library (v3.41).

Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700699833_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700711032_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700722425_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700728122_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700733819_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700737219_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700743677_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700755710_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700761660_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700767597_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700773526_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700779452_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700785263_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700790477_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700802178_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700807874_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700819268_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700822902_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700829190_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700841199_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700847147_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700853081_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700859010_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700864936_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700887626_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700893323_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700899020_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700904716_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700908493_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700920718_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700932632_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700938565_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700944493_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700961681_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700711032_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700716728_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700722425_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700728122_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700733819_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700737219_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700749729_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700755710_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700767597_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700790477_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700796481_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700807874_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700819268_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700822902_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700829190_v000.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700853081_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700859010_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700864936_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700870653_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700881930_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700887626_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700893323_v002.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700914697_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700938565_v001.fit; input/output error')
Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700944493_v001.fit; input/output error')
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
rmdir: failed to remove `/scratch/kocevski/699168': No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Done.


Performing analysis on: 4FGL_J1923.2+1408e_daily_bin1780
RA = 290.818, Dec = 14.145
tmin = 700704001, tmax = 700963201
emin = 100.0, emax = 100000.0
irfs = P8R3_SOURCE_V2
ROI = 12
zmax = 90

Fitting strategies:
Fix sources to their catalog values = True
Leave variable source normalizations free = True
Perform pre-fit where ALL sources are frozen to catalog values = False
Perform secondary fit with a free spectral index = True
Remove all catalog sources = False
Remove weak (TS < 2) catalog sources = False
Freeze Galactic component = False

Likelihood fit parameters:
Statistic: UNBINNED
Optimizer: MINUIT

Catalog: 4FGL

Repository job = True

LAT data source:
Data catalog

Updating database...

UPDATE lightcurve_data_v2 SET job_id='699168', job_status='RUN', time_executed='2023-03-23 14:44:57.081289', time_epoch_executed='1679607897.08' WHERE source_name='4FGL J1923.2+1408e' and cadence='daily' and bin_id='1780'

Database successfully updated.
Done.

Creating custom output directory:
/scratch/kocevski/699168

Creating custom pfile directory:
/scratch/kocevski/699168/pfiles/

Retrieving the FT1 file...

Issuing command:
/afs/slac.stanford.edu/g/glast/ground/bin/datacat find --group FT1 --filter 'nMetStop>=700704001.0 && nMetStart<700963201.0' /Data/Flight/Reprocess/P305

Issuing command:
/afs/slac.stanford.edu/g/glast/ground/bin/datacat find --group FT1 --filter 'nMetStart>564945418 && nMetStop>=700704001.0 && nMetStart<700963201.0' /Data/Flight/Level1/LPA

Retrieving 47 files...
Done.


Merging files...

Empty or corrupt FITS file
Traceback (most recent call last):
  File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py", line 2836, in sourceAnalysis
    dataAvailibility_ft1 = dataCatalogQuery(FT1_options, verbose=verbose)
  File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py", line 1938, in dataCatalogQuery
    FTmerge.ft1merge(dataCatalogFiles_local, mergedFile, verbose=verbose)
  File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/FTmerge.py", line 59, in ft1merge
    tstart, tstop = _getTimeKeywords(infiles)
  File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/FTmerge.py", line 36, in _getTimeKeywords
    header = pyfits.open(infiles[0])[extnum].header
  File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 148, in fitsopen
    lazy_load_hdus, **kwargs)
  File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 402, in fromfile
    lazy_load_hdus=lazy_load_hdus, **kwargs)
  File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1040, in _readfrom
    raise IOError('Empty or corrupt FITS file')
IOError: Empty or corrupt FITS file


Retrieving the FT1 file...

Issuing command:
/afs/slac.stanford.edu/g/glast/ground/bin/datacat find --group FT2 --filter 'nMetStop>=700704001.0 && nMetStart<700963201.0' /Data/Flight/Level1/LPA

Retrieving 47 files...
Done.


Merging files...
Empty or corrupt FITS file
Traceback (most recent call last):
  File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py", line 2862, in sourceAnalysis
    dataAvailibility_ft2 = dataCatalogQuery(FT2_options, verbose=verbose)
  File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py", line 1943, in dataCatalogQuery
    FTmerge.ft2merge(dataCatalogFiles_local, mergedFile, verbose=verbose)
  File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/FTmerge.py", line 133, in ft2merge
    tstart, tstop = _getTimeKeywords(infiles_arg)
  File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/FTmerge.py", line 40, in _getTimeKeywords
    header = pyfits.open(item)[extnum].header
  File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 148, in fitsopen
    lazy_load_hdus, **kwargs)
  File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 402, in fromfile
    lazy_load_hdus=lazy_load_hdus, **kwargs)
  File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1040, in _readfrom
    raise IOError('Empty or corrupt FITS file')
IOError: Empty or corrupt FITS file


UPDATE lightcurve_data_v2 SET job_status='FAIL', comment='No available data', time_epoch_completed='1679607909.22', time_completed='2023-03-23 14:45:09.215093', job_id='699168' WHERE bin_id='1780' AND source_name='4FGL J1923.2+1408e' AND cadence='daily';

Database successfully updated.

Error: could not find cmap file: /scratch/kocevski/699168/cmap_HighResolution_700704001_700963201_4FGL_J1923.2+1408e_daily_bin1780.fits
Continuing... 

Cleaning up...

** Nuclear option initiated! **
rmdir /scratch/kocevski/699168


Analysis Complete.


Thu Mar 23 14:45:09 PDT 2023


------------------------------------------------------------
Sender: LSF System <lsf@bullet0063>
Subject: Job 699168: <4FGL_J1923.2+1408e_daily_bin1780> in cluster <slac> Done

Job <4FGL_J1923.2+1408e_daily_bin1780> was submitted from host <rhel6-64p> by user <kocevski> in cluster <slac> at Thu Mar 23 14:40:55 2023
Job was executed on host(s) <bullet0063>, in queue <medium>, as user <kocevski> in cluster <slac> at Thu Mar 23 14:40:56 2023
</u/ki/kocevski> was used as the home directory.
</u/ki/kocevski> was used as the working directory.
Started at Thu Mar 23 14:40:56 2023
Terminated at Thu Mar 23 14:45:09 2023
Results reported at Thu Mar 23 14:45:09 2023

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py 4FGL_J1923.2+1408e_daily_bin1780 290.818 14.145 700704001 700963201 irfs=P8R3_SOURCE_V2 zmax=90 ROI=12 association=4FGL_J1923.2+1408e fixedModel=True performFreeIndexFit=True freeVariableSources=True nuke=True repositoryJob=True verbose=False
------------------------------------------------------------

Successfully completed.

Resource usage summary:

    CPU time :                                   11.93 sec.
    Max Memory :                                 37 MB
    Average Memory :                             18.10 MB
    Total Requested Memory :                     -
    Delta Memory :                               -
    Max Swap :                                   924 MB
    Max Processes :                              3
    Max Threads :                                19
    Run time :                                   253 sec.
    Turnaround time :                            254 sec.

The output (if any) is above this job summary.

