ASKAP Science Data Processor software - ASKAPsoft Version 0.20.0

ASKAPsoft, the ASKAP Science Data Processor, provides data processing functionality, including:* Calibration* Spectral line imaging* Continuum imaging* Source detection and generation of source catalogs* Transient detectionASKAPsoft is developed as a part of the CSIRO Australian Square Kilometre Array Pathfinder (ASKAP) Science Data Processor component. ASKAPsoft is a key component in the ASKAP system. It is the primary software for storing and processing raw data, and initiating the archiving of resulting science data products into the data archive (CASDA).The processing pipelines within ASKAPsoft are largely written in C++ built on top of casacore and other third party libraries. The software is designed to be parallelised, where possible, for performance.ASKAPsoft is designed to be built and executed in a standard Unix/Linux environment and core dependencies must be fulfilled by the platform. These include, but are not limited to, a C/C++/Fortran compiler, Make, Python 2.7, Java 7 and MPI. More specific dependencies are downloaded by the ASKAPsoft build system and are installed within the ASKAPsoft development tree. Specific to the Debian platform, after a standard installation of Debian Wheezy (7.x) the following packages will need to be installed with apt-get:* g++* gfortran* openjdk-7-jdk* python-dev* flex* bison* openmpi-bin* libopenmpi-dev* libfreetype6-dev* libpng12-devMore information regarding the building, installation and running of the software can be found in the README file in the root of the file structure that forms this collection.Source code can be accessed via the links in Related Materials section.-----Pipelines:* Corrected the use of the $ACES environment variable when running the continuum validation script, so that pecularities of the local environment are appropriately dealt with.* Some corrections in pipeline scripts regarding FITS mode processing: - Ensures the continuum linmos image is copied at the field-level mosaicking job. - Ensures the spectral-line selavy job uses the correct file extensions. - Ensures the imcontsub job converts the contsub cube to fits at the end if we are working in FITS mode. - Updates the naming of the contsub cube to ensure consistency (removing .fits from the middle of it). * Improve copying of spectral weights images when running linmos to avoid ambiguities and prevent unnecessary files.* Added a parameter, DO_SOURCE_FINDING_FIELD_MOSAICS, to turn off source finding for individual fields and rely on the source finding for the final mosaic instead. This prevents unnecessary source finding jobs being launched.* Selavy source finding jobs now have scheduling block ID (SBID) passed in parsets.* The casdaupload utility can now handle cubelets (as well as spectra & moment-maps). These are included by the casda script in the pipeline. * TIME selection options in flagging are now exposed in pipeline scripts via TIME_FLAG_SCIENCE, TIME_FLAG_SCIENCE_AV and TIME_FLAG_1934. It is up to the user to provide suitable values. * Pipelines allow processing of scheduling blocks (SB) where the number of measurement sets (MS) is different to the number of beams. This addresses an issue where the SB have recorded 36 MSs but only a subset of them are valid.* The use of dcp for copying MSs from the archive is turned off by default to minimise the load on the hpc-data nodes (the method for doing this is not ideal).Processing Software:* Modifications to Selavy to include additional information in the headers of the spectra & related images (Object name, date-obs and duration, Project ID and SBID, history comments).* Fixed a problem where mslist output was corrupted by long field names.* Shortened objectID strings are now used in catalogues. No longer uses image name, but instead SBID + catalogue/data product type + sequence ID.ASKAPsoft即ASKAP科学数据处理器（ASKAP Science Data Processor），提供以下数据处理功能： - 校准（Calibration） - 谱线成像（Spectral line imaging） - 连续谱成像（Continuum imaging） - 源探测与源目录生成（Source detection and generation of source catalogs） - 暂现源探测（Transient detection） ASKAPsoft是澳大利亚联邦科学与工业研究组织（CSIRO）旗下澳大利亚平方千米阵列路径探测器（Australian Square Kilometre Array Pathfinder, ASKAP）科学数据处理器组件的开发成果之一，也是ASKAP系统的核心组成部分。它是存储与处理原始数据、并将生成的科学数据产品归档至数据存档系统（CASDA）的核心软件。 ASKAPsoft内的处理流水线大多基于casacore与其他第三方库，采用C++语言编写。该软件设计为尽可能实现并行化以提升运行性能。 ASKAPsoft可在标准Unix/Linux环境下编译与运行，平台需满足核心依赖要求，包括但不限于C/C++/Fortran编译器、Make、Python 2.7、Java 7以及消息传递接口（MPI）。更多特定依赖项将由ASKAPsoft的构建系统自动下载，并安装至ASKAPsoft开发目录中。针对Debian平台，在标准安装Debian Wheezy（7.x）后，需通过apt-get工具安装以下软件包： - g++ - gfortran - openjdk-7-jdk - python-dev - flex - bison - openmpi-bin - libopenmpi-dev - libfreetype6-dev - libpng12-dev 有关该软件的构建、安装与运行的更多详细信息，可在本数据集根目录的README文件中查阅。源代码可通过"相关材料"部分的链接获取。 --- 流水线更新 1. 修复了运行连续谱验证脚本时对`$ACES`环境变量的使用逻辑，以适配本地环境的特殊配置。 2. 对FITS模式处理相关的流水线脚本进行了若干修正： - 确保在视场级拼接（linmos）作业中正确复制连续谱拼接图像。 - 确保谱线Selavy（Selavy）源探测作业使用正确的文件扩展名。 - 确保在FITS模式下，`imcontsub`作业在处理结束时将连续谱扣除立方体转换为FITS格式。 - 优化连续谱扣除立方体的命名规则，确保命名一致性（移除文件名中间的`.fits`后缀）。 3. 改进了运行拼接（linmos）时谱权重图像的复制逻辑，以避免命名歧义并减少冗余文件的生成。 4. 新增参数`DO_SOURCE_FINDING_FIELD_MOSAICS`，用于关闭单个视场的源探测，转而依赖最终拼接结果的源探测，避免启动不必要的源探测作业。 5. 现在Selavy（Selavy）源探测作业可通过参数集（parsets）传入调度块ID（SBID）。 6. `casdaupload`工具现已支持处理立方体切片（cubelets）（同时支持频谱与矩量图），流水线中的`casda`脚本会自动包含此类文件。 7. 标记（flagging）功能中的时间选择选项现已通过`TIME_FLAG_SCIENCE`、`TIME_FLAG_SCIENCE_AV`与`TIME_FLAG_1934`三个参数在流水线脚本中开放，用户需自行提供合适的参数值。 8. 流水线现已支持处理测量集（Measurement Set, MS）数量与波束数量不一致的调度块（Scheduling Block, SB），解决了部分调度块记录了36个测量集但仅部分有效的场景下的问题。 9. 默认关闭使用`dcp`从存档复制测量集的功能，以降低对`hpc-data`节点的负载（该复制方式并非最优方案）。 --- 处理软件更新 1. 对Selavy（Selavy）源探测工具进行了改进，在频谱及相关图像的文件头中新增了更多元数据信息，包括天体目标名称、观测日期与时长、项目ID以及调度块ID（SBID）、历史注释。 2. 修复了长视场名称导致`mslist`输出损坏的问题。 3. 目录中现已采用缩短版的对象ID字符串，不再使用图像文件名，转而采用调度块ID（SBID）+目录/数据产品类型+序列ID的命名规则。