/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/matplotlib/font_manager.py:281: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment. 'Matplotlib is building the font cache using fc-list. ' /nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/pyfits/__init__.py:22: PyFITSDeprecationWarning: PyFITS is deprecated, please use astropy.io.fits PyFITSDeprecationWarning) # noqa Likelihood Analysis Tool Support Contact: Daniel Kocevski (daniel.kocevski@nasa.gov) Waiting 211 seconds before starting... Proceeding. Importing modules... This is make3FGLxml version 01r0. The default diffuse model files and names are for pass 8 and assume you have v10r00p05 of the Fermi Science Tools or higher. This is make4FGLDR3xml version 01r09. The default diffuse model files and names are for P8R3, 4FGL DRs, and the default fermitools directory locations. WARNING: version mismatch between CFITSIO header (v3.43) and linked library (v3.41). /nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/matplotlib/cbook/deprecation.py:106: MatplotlibDeprecationWarning: The mpl_toolkits.axes_grid module was deprecated in version 2.1. Use mpl_toolkits.axes_grid1 and mpl_toolkits.axisartist provies the same functionality instead. warnings.warn(message, mplDeprecation, stacklevel=1) WARNING: version mismatch between CFITSIO header (v3.43) and linked library (v3.41). Last ser WARNING: version mismatch between CFITSIO header (v3.43) and linked library (v3.41). Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700961681_v002.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700967378_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700973074_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700978771_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0700984467_v002.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701000201_v000.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701006210_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701018116_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701024048_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701029976_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701035892_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701041413_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701047129_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701052825_v000.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701058522_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701064218_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701069915_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701075611_v000.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701079589_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701097659_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701103599_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft1/gll_ph_p305_r0701115458_v000.fit; input/output error') /nfs/farm/lsb_spool2/kocevski/1679604754.682109: line 8: 36965 Quit (core dumped) /nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py 4FGL_J0325.7+2225_daily_bin1781 51.4421 22.4326 700963201 701222401 irfs=P8R3_SOURCE_V2 zmax=90 ROI=12 association=4FGL_J0325.7+2225 fixedModel=True performFreeIndexFit=True freeVariableSources=True nuke=True repositoryJob=True verbose=False t server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700984467_v002.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0700990164_v000.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0701000201_v000.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0701012173_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0701018116_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0701075611_v000.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0701079589_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0701103599_v001.fit; input/output error') Last server error 3007 ('Unable to read /glast/Data/Flight/Level1/LPA/prod/5.9/ft2/gll_pt_p310_r0701115458_v000.fit; input/output error') shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory rmdir: failed to remove `/scratch/kocevski/682112': No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory Done. Performing analysis on: 4FGL_J0325.7+2225_daily_bin1781 RA = 51.4421, Dec = 22.4326 tmin = 700963201, tmax = 701222401 emin = 100.0, emax = 100000.0 irfs = P8R3_SOURCE_V2 ROI = 12 zmax = 90 Fitting strategies: Fix sources to their catalog values = True Leave variable source normalizations free = True Perform pre-fit where ALL sources are frozen to catalog values = False Perform secondary fit with a free spectral index = True Remove all catalog sources = False Remove weak (TS < 2) catalog sources = False Freeze Galactic component = False Likelihood fit parameters: Statistic: UNBINNED Optimizer: MINUIT Catalog: 4FGL Repository job = True LAT data source: Data catalog Updating database... UPDATE lightcurve_data_v2 SET job_id='682112', job_status='RUN', time_executed='2023-03-23 14:01:23.473049', time_epoch_executed='1679605283.47' WHERE source_name='4FGL J0325.7+2225' and cadence='daily' and bin_id='1781' Database successfully updated. Done. Creating custom output directory: /scratch/kocevski/682112 Creating custom pfile directory: /scratch/kocevski/682112/pfiles/ Retrieving the FT1 file... Issuing command: /afs/slac.stanford.edu/g/glast/ground/bin/datacat find --group FT1 --filter 'nMetStop>=700963201.0 && nMetStart<701222401.0' /Data/Flight/Reprocess/P305 Issuing command: /afs/slac.stanford.edu/g/glast/ground/bin/datacat find --group FT1 --filter 'nMetStart>564945418 && nMetStop>=700963201.0 && nMetStart<701222401.0' /Data/Flight/Level1/LPA Retrieving 28 files... Done. Merging files... Empty or corrupt FITS file Traceback (most recent call last): File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py", line 2836, in sourceAnalysis dataAvailibility_ft1 = dataCatalogQuery(FT1_options, verbose=verbose) File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py", line 1938, in dataCatalogQuery FTmerge.ft1merge(dataCatalogFiles_local, mergedFile, verbose=verbose) File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/FTmerge.py", line 59, in ft1merge tstart, tstop = _getTimeKeywords(infiles) File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/FTmerge.py", line 36, in _getTimeKeywords header = pyfits.open(infiles[0])[extnum].header File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 148, in fitsopen lazy_load_hdus, **kwargs) File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 402, in fromfile lazy_load_hdus=lazy_load_hdus, **kwargs) File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1040, in _readfrom raise IOError('Empty or corrupt FITS file') IOError: Empty or corrupt FITS file Retrieving the FT1 file... Issuing command: /afs/slac.stanford.edu/g/glast/ground/bin/datacat find --group FT2 --filter 'nMetStop>=700963201.0 && nMetStart<701222401.0' /Data/Flight/Level1/LPA Retrieving 28 files... Done. Merging files... Empty or corrupt FITS file Traceback (most recent call last): File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py", line 2862, in sourceAnalysis dataAvailibility_ft2 = dataCatalogQuery(FT2_options, verbose=verbose) File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py", line 1943, in dataCatalogQuery FTmerge.ft2merge(dataCatalogFiles_local, mergedFile, verbose=verbose) File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/FTmerge.py", line 133, in ft2merge tstart, tstop = _getTimeKeywords(infiles_arg) File "/nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/FTmerge.py", line 40, in _getTimeKeywords header = pyfits.open(item)[extnum].header File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 148, in fitsopen lazy_load_hdus, **kwargs) File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 402, in fromfile lazy_load_hdus=lazy_load_hdus, **kwargs) File "/nfs/farm/g/glast/g/LCRepository/bin/miniconda2/envs/fermitools_v1.0.5/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1040, in _readfrom raise IOError('Empty or corrupt FITS file') IOError: Empty or corrupt FITS file UPDATE lightcurve_data_v2 SET job_status='FAIL', comment='No available data', time_epoch_completed='1679605315.62', time_completed='2023-03-23 14:01:55.617155', job_id='682112' WHERE bin_id='1781' AND source_name='4FGL J0325.7+2225' AND cadence='daily'; Database successfully updated. Error: could not find cmap file: /scratch/kocevski/682112/cmap_HighResolution_700963201_701222401_4FGL_J0325.7+2225_daily_bin1781.fits Continuing... Cleaning up... ** Nuclear option initiated! ** rmdir /scratch/kocevski/682112 Analysis Complete. Thu Mar 23 14:01:55 PDT 2023 ------------------------------------------------------------ Sender: LSF System Subject: Job 682093: <4FGL_J0325.7+2225_daily_bin1781> in cluster Done Job <4FGL_J0325.7+2225_daily_bin1781> was submitted from host by user in cluster at Thu Mar 23 13:52:34 2023 Job was executed on host(s) , in queue , as user in cluster at Thu Mar 23 13:52:36 2023 was used as the home directory. was used as the working directory. Started at Thu Mar 23 13:52:36 2023 Terminated at Thu Mar 23 13:56:22 2023 Results reported at Thu Mar 23 13:56:22 2023 Your job looked like: ------------------------------------------------------------ # LSBATCH: User input /nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py 4FGL_J0325.7+2225_daily_bin1781 51.4421 22.4326 700963201 701222401 irfs=P8R3_SOURCE_V2 zmax=90 ROI=12 association=4FGL_J0325.7+2225 fixedModel=True performFreeIndexFit=True freeVariableSources=True nuke=True repositoryJob=True verbose=False ------------------------------------------------------------ Successfully completed. Resource usage summary: CPU time : 11.00 sec. Max Memory : 141 MB Average Memory : 55.44 MB Total Requested Memory : - Delta Memory : - Max Swap : - Max Processes : 5 Max Threads : 19 Run time : 225 sec. Turnaround time : 228 sec. The output (if any) is above this job summary. ------------------------------------------------------------ Sender: LSF System Subject: Job 682097: <4FGL_J0325.7+2225_daily_bin1781> in cluster Done Job <4FGL_J0325.7+2225_daily_bin1781> was submitted from host by user in cluster at Thu Mar 23 13:52:34 2023 Job was executed on host(s) , in queue , as user in cluster at Thu Mar 23 13:52:37 2023 was used as the home directory. was used as the working directory. Started at Thu Mar 23 13:52:37 2023 Terminated at Thu Mar 23 13:57:05 2023 Results reported at Thu Mar 23 13:57:05 2023 Your job looked like: ------------------------------------------------------------ # LSBATCH: User input /nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py 4FGL_J0325.7+2225_daily_bin1781 51.4421 22.4326 700963201 701222401 irfs=P8R3_SOURCE_V2 zmax=90 ROI=12 association=4FGL_J0325.7+2225 fixedModel=True performFreeIndexFit=True freeVariableSources=True nuke=True repositoryJob=True verbose=False ------------------------------------------------------------ Successfully completed. Resource usage summary: CPU time : 11.02 sec. Max Memory : 131 MB Average Memory : 49.64 MB Total Requested Memory : - Delta Memory : - Max Swap : - Max Processes : 5 Max Threads : 19 Run time : 268 sec. Turnaround time : 271 sec. The output (if any) is above this job summary. ------------------------------------------------------------ Sender: LSF System Subject: Job 681915: <4FGL_J0325.7+2225_daily_bin1781> in cluster Done Job <4FGL_J0325.7+2225_daily_bin1781> was submitted from host by user in cluster at Thu Mar 23 13:52:28 2023 Job was executed on host(s) , in queue , as user in cluster at Thu Mar 23 13:52:29 2023 was used as the home directory. was used as the working directory. Started at Thu Mar 23 13:52:29 2023 Terminated at Thu Mar 23 14:00:21 2023 Results reported at Thu Mar 23 14:00:21 2023 Your job looked like: ------------------------------------------------------------ # LSBATCH: User input /nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py 4FGL_J0325.7+2225_daily_bin1781 51.4421 22.4326 700963201 701222401 irfs=P8R3_SOURCE_V2 zmax=90 ROI=12 association=4FGL_J0325.7+2225 fixedModel=True performFreeIndexFit=True freeVariableSources=True nuke=True repositoryJob=True verbose=False ------------------------------------------------------------ Successfully completed. Resource usage summary: CPU time : 11.67 sec. Max Memory : 102 MB Average Memory : 60.76 MB Total Requested Memory : - Delta Memory : - Max Swap : - Max Processes : 4 Max Threads : 19 Run time : 472 sec. Turnaround time : 473 sec. The output (if any) is above this job summary. ------------------------------------------------------------ Sender: LSF System Subject: Job 682088: <4FGL_J0325.7+2225_daily_bin1781> in cluster Done Job <4FGL_J0325.7+2225_daily_bin1781> was submitted from host by user in cluster at Thu Mar 23 13:52:33 2023 Job was executed on host(s) , in queue , as user in cluster at Thu Mar 23 13:52:36 2023 was used as the home directory. was used as the working directory. Started at Thu Mar 23 13:52:36 2023 Terminated at Thu Mar 23 14:01:27 2023 Results reported at Thu Mar 23 14:01:27 2023 Your job looked like: ------------------------------------------------------------ # LSBATCH: User input /nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py 4FGL_J0325.7+2225_daily_bin1781 51.4421 22.4326 700963201 701222401 irfs=P8R3_SOURCE_V2 zmax=90 ROI=12 association=4FGL_J0325.7+2225 fixedModel=True performFreeIndexFit=True freeVariableSources=True nuke=True repositoryJob=True verbose=False ------------------------------------------------------------ Successfully completed. Resource usage summary: CPU time : 12.00 sec. Max Memory : 102 MB Average Memory : 20.37 MB Total Requested Memory : - Delta Memory : - Max Swap : - Max Processes : 4 Max Threads : 19 Run time : 531 sec. Turnaround time : 534 sec. The output (if any) is above this job summary. ------------------------------------------------------------ Sender: LSF System Subject: Job 682112: <4FGL_J0325.7+2225_daily_bin1781> in cluster Done Job <4FGL_J0325.7+2225_daily_bin1781> was submitted from host by user in cluster at Thu Mar 23 13:52:34 2023 Job was executed on host(s) , in queue , as user in cluster at Thu Mar 23 13:52:39 2023 was used as the home directory. was used as the working directory. Started at Thu Mar 23 13:52:39 2023 Terminated at Thu Mar 23 14:01:56 2023 Results reported at Thu Mar 23 14:01:56 2023 Your job looked like: ------------------------------------------------------------ # LSBATCH: User input /nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py 4FGL_J0325.7+2225_daily_bin1781 51.4421 22.4326 700963201 701222401 irfs=P8R3_SOURCE_V2 zmax=90 ROI=12 association=4FGL_J0325.7+2225 fixedModel=True performFreeIndexFit=True freeVariableSources=True nuke=True repositoryJob=True verbose=False ------------------------------------------------------------ Successfully completed. Resource usage summary: CPU time : 11.64 sec. Max Memory : 141 MB Average Memory : 44.79 MB Total Requested Memory : - Delta Memory : - Max Swap : - Max Processes : 7 Max Threads : 19 Run time : 556 sec. Turnaround time : 562 sec. The output (if any) is above this job summary. ------------------------------------------------------------ Sender: LSF System Subject: Job 682109: <4FGL_J0325.7+2225_daily_bin1781> in cluster Exited Job <4FGL_J0325.7+2225_daily_bin1781> was submitted from host by user in cluster at Thu Mar 23 13:52:34 2023 Job was executed on host(s) , in queue , as user in cluster at Thu Mar 23 13:52:38 2023 was used as the home directory. was used as the working directory. Started at Thu Mar 23 13:52:38 2023 Terminated at Thu Mar 23 14:02:06 2023 Results reported at Thu Mar 23 14:02:06 2023 Your job looked like: ------------------------------------------------------------ # LSBATCH: User input /nfs/farm/g/glast/g/LCRepository/Likelihood/Scripts/LikelihoodAnalysis.py 4FGL_J0325.7+2225_daily_bin1781 51.4421 22.4326 700963201 701222401 irfs=P8R3_SOURCE_V2 zmax=90 ROI=12 association=4FGL_J0325.7+2225 fixedModel=True performFreeIndexFit=True freeVariableSources=True nuke=True repositoryJob=True verbose=False ------------------------------------------------------------ TERM_SWAP: job killed after reaching LSF swap usage limit. Exited with exit code 131. Resource usage summary: CPU time : 9.36 sec. Max Memory : 146 MB Average Memory : 57.20 MB Total Requested Memory : - Delta Memory : - Max Swap : 18698 MB Max Processes : 5 Max Threads : 19 Run time : 568 sec. Turnaround time : 572 sec. The output (if any) is above this job summary.