Raw data from the Whipple 10m gamma-ray telescope, Mt Hopkins, AZ

Description This dataset provides the raw data from observations made with the Whipple 10-meter imaging atmospheric Cherenkov telescope at the Fred Lawrence Whipple Observatory (FLWO) on Mt. Hopkins, Arizona between September 1994 and July 2011. The data are stored in a proprietary format, based on the CERN FZ framework, for which we have developed a pure Python reader/decoder with which you can extract the images and metadata for display or analysis. The observation logsheets which describe the data taking are also available for the majority of nights. Unfortunately a substantial fraction of the observations has been lost to the passage of time, so the dataset here is not complete. It is difficult to estimate the total quantity of data lost. The observations were generally made in 28-minute long runs with the telescope tracking a source in the sky during dark, moonless nights. These observations are organized into darkruns between the full-moon periods when no data-taking could be made, and into seasons around the Summer monsoon period in Arizona when the camera was disconnected to protect it from damage from lightning storms. The imaging camera used on the Whipple 10m telescope evolved during the time from 1994 to 2011, with this evolution being described in detail by Kildea et al., Astroparticle Physics, 28, 2, 182-195, (2007). Raw data files: raw10/dYYYYMMDD/gtNNNNNN.fz.xz Data recorded during each observation run are stored in a single data file in the Granite data format (GDF). These files contain multiple records of different types, as described in some detail in the readme page for the Python reader/decoder. The following record types are available and are supported by the reader: Run header: Nominal start time of the run, comments entered by the observer, and other miscellaneous information. 10m event: The data recorded for each event, including time, digitized image, trigger type, etc. 10m frame: Data recorded for special calibration events in early Whipple data. Phased out in GDF version 80 (starting on 1997-10-29), after which the calibration data appears in the 10m event records, flagged by a specific trigger bit. Tracking status: Target name and coordinates, and position of telescope (azimuth and elevation angles) read out periodically from the tracking computer, if available. HV status: Measured voltages and currents on each PMT, read out periodically from the high-voltage system, if available. Here the data files are grouped into tar archives for each season, e.g. raw10_1998_1999.tar, which will yield data files with the following hierarchy when unpacked: raw10/dYYYYMMDD/gtNNNNNN.fz.xz, where YYYYMMDD is the UTC date of the observations and NNNNNN is a zero-padded run number of the observation. The data files are compressed using the standard LZMA compression format, which can be read directly by the Python reader (there is no need to decompress the files manually). Users can download the full dataset (tar files) from this repository, unpack them and operate on them locally. Alternatively, the Python reader can download individual runs from the repository on-the-fly, which allows the data to be explored without downloading in advance. The raw data repository comprises: 22,805 observation runs, taken on 1,827 nights of observation, where 620,331,605 events were recorded during 24,633,108 seconds of observation (6,842.5 hours). Log sheet files: log10/dYYYYMMDD.log_10 The observations for each night are described by a plain-text (ASCII) log sheet, written by the observers that operated the telescope each night. These log sheets are also available in the repository, grouped into a tar archive, log10.tar, which unpacks into a set of files with the following hierarchy: log10/dYYYYMMDD.log_10. The logs have been anonymized to remove references to the names of the observers and support staff. The logs describe the observations made, problems encountered, weather conditions, modifications to the hardware, and give the results of a "quicklook" analysis of the data files (significance of any detection and rate of gamma rays detected). The logsheet archive comprises 2,582 logsheets describing the observations made on individual nights. Run database: raw10_run_summary.csv.xz Two summary databases are also available in CSV form. The file raw10_run_summary.csv.xz contains one entry for each data file in the archive, obtained by scanning the data file and processing the various records. Each record contains the following entries: filename: filename in the archive in the form raw10/dYYYYMMDD/gtNNNNNN.fz.xz (string) run_number: run number, extracted from the filename (integer) gdf_version: version number of the Granite data format (integer, between 21 and 83) nheader: number of GDF header records in the file (integer) nevent: number of GDF event records in the file (integer) nframe: number of GDF frame records in the file (integer) ntracking: number of GDF tracking records in the file (integer) nhv: number of GDF high-voltage records in the file (integer) nccd: number of GDF CCD records in the file (integer) nadc: number of ADC channels recorded in the each event. The LeCroy ADCs each have 12 channels, all of which were read out, so this is larger than the number of PMT pixels in the camera (integer): nadc=120: the 109-pixel camera (before 1996-12-05); nadc=156: the 151-pixel camera (from 1996-12-05 until 1997-02-29) nadc=336: the 331-pixel camera (from 1997-02-29 until 1999-09-11) nadc=492: the 490-pixel camera (from 1999-09-11 until 2009-12-16) nadc=384: the 379-pixel camera (from 2009-12-16) nadc=0: if no event or frame records are found in the run gps_not_good: number of event or frame records for which the Python reader has returned gps_not_good=True (integer) mjd: UTC date of the observation in modified Julian date (MJD), taken from the GPS clock (if available). For robust determination of the date of the observation the mjd_min and mjd_max elements should be preferred, as these use the system time on the DACQ computer, which is available even when the GPS is bad. (integer) ut_max: UTC time of the final event in the run, in seconds since midnight, taken from the GPS clock (integer) ut_min: UTC time of the first event in the run, in seconds since midnight, taken from the GPS clock (integer) mjd_max: nominal (anticipated) MJD at the end of the run, as determined by the DACQ system (based on the computer time) when the run starts. May be different from the true end of the run if run terminates early for some reason (float) mjd_min: nominal MJD at the start of the run, as determined by the DACQ system (based on the computer time) when the run starts (float) nsky: number of sky events triggered by the multiplicity or pattern trigger when a sufficiently bright flash of light is detected in the camera (integer) npedestal: number of pedestal calibration events, initiated by a periodic trigger (integer) el_max: maximum elevation reported by the tracking system during the run in degrees (float), or empty string if no tracking records are available el_min: minimum elevation reported by the tracking system during the run in degrees (float), or empty string if no tracking records are available source_name: target (source) name reported by the tracking system (string), or empty string if no tracking records are available tracking_mode: tracking mode reported by the tracking system, one of: on, off, standby, slewing, check, or empty string if no tracking records available (string) ra: right ascension of the target in the format HHhMMmSS.Ss, e.g. 12h34m56.7s, or empty string if no tracking records are available(string) dec: declination of the target in the format +/-DDdMMmSSs, e.g. +12h34m56.7s, or empty string if no tracking records are available (string) ped: median pedestal value across the camera in digital counts, calculated from pedestal calibration events (float) ped_rms: median pedestal RMS across the camera in digital counts, calculated from pedestal calibration events (float) bytes_read: number of bytes read, which should be the same as the number of bytes in the file unless a decoder or read error occurs (integer) error_message: error message returned by the reader, or empty string if full data file is successfully decoded (string) Logsheet database: logsheet_database.csv.xz The second summary database, logsheet_database.csv.xz, contains one entry for each run mentioned in the logsheets, obtained by parsing the plain-text logsheets and extracting data for each run. Each record contains the following entries: Code: target object ID code, a 2 or 3 character identifier for the source, e.g. cr for the Crab (string) Run Num: run number (integer) Source: target object name (string) UTC Date: date of observation run in the form YYYY-MM-DD (string) UTC Time: time of start of run, if noted by observer HH:MM or HH:MM:SS (string) Duration: duration of run in minutes, if noted by observer (integer) Mode: observation mode, if noted by observer, one of track, on, off, drift, slew, home, wobble (string) Sky Q: sky quality, if noted by observer, one of a+ (best), a, a-, b+, b, b-, c+, c, c-, ng (string). Very subjective, based entirely on observer's feeling of the sky. Elevation: elevation at start of observation in degrees (integer) N2 run: run number for flat-fielding calibration run to use with this run (integer) Off run: run number for off-source run to use with this run in on/off or pair mode (integer) Comment: comment from observer (string). The logsheet database comprises 32,102 entries. Index files: index.json, raw10_index.csv.xz and log10_index.csv.xv Used by the Python reader to download the individual data files and logsheets from the repository. Data completeness The data and logsheet archives in this repository are not complete. In its entirety the Whipple dataset was too large to be stored affordably on the disk of the early era of observation. The data were instead stored, and distributed to the participating institutions, on compact disk. Data handling and archiving proceedures improved as time went on. As an example, in earlier days FTP was used to manually transfer data from the VMS-based DACQ computer to the Unix-based analysis and archiving machines - data could be lost if the transfer of the binary data files was inadvertently done in ASCII mode which led to the VMS<->UNIX linefeed translations being incorrectly applied. In the early 2000's the transfer of data and the production of CDs was automated by scripts, and by the late 2000's the data was stored in the VERITAS data archive at UCLA and an effort was made to integrate all remaining data in the archive. Before 2008 the plain-text logsheets were distributed by the observers to the collaboration by email. Starting in 2008 an electronic log book was used to archive and distribute the logsheets. From mid-1998 to 2008 the logsheets were ingested into a database system which allowed collaborators to search for runs. The system also archived injested logsheets. The logsheet archive is therefore rather complete from about mid-1998 until the end of the observations in 2011. This repository is built from the surviving data stored in the VERITAS archive and the logsheets from the electronic log book, the database system, and from old email mailboxes. The Whipple DACQ system assigned run numbers consecutively when runs were started. We can therefore estimate the completeness of the data and logsheet archive by examining what fraction of the consecutive run numbers are present in the archive. This can be seen in a plot of the data and logsheet completeness as a function of run number binned in groups of 1,000 run numbers. The plot shows that data files are present from approximately 40%-50% of the run numbers from 1994 to mid-1998, increasing to approximately 60% until about 2007, and then to approximately 90% until the end of observations in 2011. The logsheets are mostly complete from mid-1998 to 2011, and somewhat patchy before this time. Accounting for some known gaps in the run number sequence we expect that 37,547 run numbers were assigned, whereas 22,805 data files are present in the archive (60.7% completeness), and 32,102 run log entries have been entered into the logsheet database (85.5% completeness). This estimate is a lower limit on the completness, as it does not account for engineering runs and crashes where a run number is assigned but no useful data is produced. In some cases these runs may have been taken during the daytime and not reported in the logs. In other cases they were logged but no data was produced. Engineering runs and crashes are more common in the early period as the camera was upgraded from the 109 pixel camera, to the 151, 331 and finally to 490 pixel cameras. Additionally the Granite DACQ system and data format were in development during this period. As discussed above, the logsheet database is relatively complete for the period from mid-1998 onwards. We alternatively estimate the data completeness in this period by counting the number of nights for which at least one observation run is recorded in the logsheet database versus the number of nights for which data is present in the archive. A total of 1,884 nights of observation are logged in this period, whereas data are present in the archive for 1,369 nights, giving an overall completeness of 72.6% in this period. Downloading and analyzing data An example of how to use this repository is provided in a Python notebook that implements a Whipple-style analysis of a large flare from the AGN Markarian 421 observed by the Whipple telescope on 07 May 1997. Caveat This dataset is provided as-is by the Whipple Collaboration with the hope that it will be useful to students and researchers. We do not advise that these data are used for astronomical research, as we cannot provide instrument response functions, a Monte Carlo model of the instrument, or the details of the numerous modifications to the camera settings during the period. It may none-the-less be useful for student projects; to illustrate air-shower images; as a test for image processing software; as a dataset to develop algorithms to work on hexagonally-sampled images; and probably many more.