Ultra-low-light computer vision using trained photon correlations

This data repository contains the information necessary to reproduce the main results of the paper “Ultra-low-light computer vision using trained photon correlations”.This repository contains the data and the code for generating the figures in the manuscript "Ultra-low-light computer vision using trained photon correlations", including figures in the main text and in supplementary materials. The repository also contains the code for controling the experiment setup and running the experiments conducted in the paper: Folder 'DATA_Figure2' contains the raw camera data of different SPDC illumination patters generated with different pump spectra and the correspoding images of the pump spectra taken on a Bassler camera. It also contains a notebook describing how to fit for the phasematching function, a notebook that predicts different SPDC illumination patterns from the pump spectrum, and an example data collection notebook. Data_Collection.ipynb - example data collection notebook Fitting_parameters_for_phase_matching_function.ipynb - notebook used to fit the phasematching function parameters SPDC_dataset.npz - a dataset constructed from different SPDC illumination patterns to be used in Fitting_parameters_for_phase_matching_function.ipynb Experiment_digital_twin_predictions_Fancy.ipynb - example notebook to generate SPDC illumination patterns from pump angular spectra Folder 'MPEG7_DATA_UNTRAINED_ILLUMINATION_Figure3/Experimental_DATA' contains the raw camera data of the MPEG7 objects under untrained SPDC (correlated) and coherent (uncorrelated) illumination patterns, the dataset made from these raw images, the estimated illumination photon-flux lists, and the accuracies from a trained set Transformer. Illumination_list_computing_example_SPDC.ipynb - Example notebook to extract illumination power levels Dataset_Untrained_Coherent_Illumination_MPEG7.zip - Untrained coherent (uncorrelated) illumination dataset file Dataset_Untrained_Illumination_MPEG7.zip - Untrained SPDC (correlated) illumination dataset file Untrained_illumination_SPDC_vs_Coherent_mpeg7_Set_Transformer_accuracies.npz - file containing accuracies for the untrained illumination points Folder 'MPEG7_DATA_UNTRAINED_ILLUMINATION_Figure3' contains example scripts for data collection and set Transformer training. postprocess_Set_Transformer_training.py - script for training the set Transformer utils.py - helper functions for postprocess_Set_Transformer_training.py Data_Collection_example_SPDC_untrained_illumination.ipynb - example notebook for data collection Folder 'E2E_EXPERIMENTAL_TRAINING_EXAMPLE_Figure3' contains an example script for running end-to-end optimization on the experimental setup. Experimental CAT E2E Training.ipynb - example notebook to run the end-to-end optimization protocol on the experimental setup Folder 'E2E_EXPERIMENTAL_TRAINING_EXAMPLE_Figure3/Accuracies_on_MPEG7' contain the illumination pattern details and the accuracies for the 3 different power levels of trained SPDC illumination show in main text Figure 3 test_errs_Pow2.npz - File containing the accuracies for trained correlated illumination test_errs_Pow3.npz - File containing the accuracies for trained correlated illumination test_errs_Pow4.npz - File containing the accuracies for trained correlated illumination Folder 'PLOTTING_CODE_Figure3' contains an example script to plot the experimental accuracies Plotting_MPEG7_results_Zenodo.ipynb - example code used to generated Figure 3 plots Folder 'E2E_TRAINING_Figure4' contains all the simualtion scripts, the datasets, and EMCCD noise distribution used to generate the results in Figure 4. It also contains 4 example notebooks on how to train the illumination for each illumination case considered. E2E_Training_arbitrary_SPDC_Example.ipynb - Example notebook to train the engineered phase matched SPDC source on the cell organelle task E2E_Training_baseline_Example.ipynb - Example notebook to train the conventional computer vision approach on the cell organelle task E2E_Training_Coherent_Example.ipynb - Example notebook to train the coherent (uncorrelated) source on the cell organelle task E2E_Training_simulation_of_experiment_SPDC_Example.ipynb - Example notebook to train the digital model of our SPDC source on the cell organelle task modules_original.py - helper function py file utils_training.py - helper function py file data - folder containing the cell organelle dataset and the noise distribution derived from the EMCCD camera train_e2e_coherent_cells.py - script used to generated the uncorrelated illumination data in Figure 4 train_e2e_spdc_cells.py - script used to generate the correlated illumination data in Figure 4 All files need to be unzipped and all paths to data files and folders in the scripts/notebooks will need to be checked before any code is run. All code was written in Python 3.XX. Python environment requirements: _libgcc_mutex 0.1 _openmp_mutex 4.5 antlr-python-runtime 4.9.2 appdirs 1.4.4 asttokens 2.2.1 attrs 23.2.0 autograd 1.7.0 autoray 0.6.12 backcall 0.2.0 blas 1.0 bokeh 2.4.3 bottleneck 1.3.5 brotli 1.1.0 brotli-bin 1.1.0 brotli-python 1.0.9 ca-certificates 2025.8.3 catalogue 2.0.7 certifi 2023.5.7 cffi 1.15.1 charset-normalizer 2.0.4 click 8.1.3 cloudpickle 2.2.1 cmake 3.26.3 cmasher 1.6.3 colorama 0.4.6 colorspacious 1.1.2 comm 0.1.3 confection 0.0.4 contourpy 1.0.5 cryptography 41.0.3 cycler 0.12.1 cymem 2.0.6 cython-blis 0.7.10 cytoolz 0.12.0 dask 2023.5.0 dask-core 2023.4.1 dbus 1.13.18 debugpy 1.6.7 distributed 2023.4.1 e13tools 0.9.6 executing 1.2.0 expat 2.5.0 filelock 3.12.0 fire 0.5.0 freetype 2.12.1 fsspec 2023.5.0 giflib 5.2.1 glib 2.69.1 gst-plugins-base 1.14.1 gstreamer 1.14.1 heapdict 1.0.1 icu 58.2 idna 3.4 importlib-metadata 6.6.0 importlib_resources 6.1.1 intel-openmp 2023.1.0 ipykernel 6.23.1 ipython 8.12.2 jedi 0.18.2 jinja2 3.1.2 joblib 1.4.2 jpeg 9e jsonschema 4.17.3 jupyter-client 8.2.0 jupyter-core 5.3.0 jupyter_core 5.8.1 kaleido-core 0.2.1 keyutils 1.6.1 kiwisolver 1.4.5 krb5 1.20.1 langcodes 3.3.0 lark-parser 0.12.0 lcms2 2.12 ld_impl_linux-64 2.38 lerc 3.0 libbrotlicommon 1.1.0 libbrotlidec 1.1.0 libbrotlienc 1.1.0 libclang 10.0.1 libdeflate 1.17 libedit 3.1.20191231 libevent 2.1.12 libexpat 2.5.0 libffi 3.4.4 libgcc-ng 13.2.0 libgfortran-ng 11.2.0 libgfortran5 11.2.0 libllvm10 10.0.1 libllvm14 14.0.6 libpng 1.6.39 libpq 12.15 libstdcxx-ng 13.2.0 libtiff 4.5.1 libwebp 1.3.2 libwebp-base 1.3.2 libxcb 1.16 libxkbcommon 1.0.1 libxml2 2.9.14 lit 16.0.5 llvm-openmp 14.0.6 llvmlite 0.40.0 locket 1.0.0 lz4-c 1.9.4 markupsafe 2.1.2 mathjax 2.7.7 matplotlib 3.7.2 matplotlib-base 3.7.2 matplotlib-inline 0.1.6 mkl 2023.1.0 mkl-service 2.4.0 mkl_fft 1.3.8 mkl_random 1.2.4 mpmath 1.3.0 msgpack-python 1.0.3 munkres 1.1.4 murmurhash 1.0.7 nbformat 5.10.4 ncurses 6.4 nest-asyncio 1.5.6 networkx 3.1 ninja 1.11.1.4 nspr 4.35 nss 3.89.1 numba 0.57.0 numexpr 2.8.4 numpy 1.24.3 numpy-base 1.24.3 nvidia-cublas-cu11 11.10.3.66 nvidia-cuda-cupti-cu11 11.7.101 nvidia-cuda-nvrtc-cu11 11.7.99 nvidia-cuda-runtime-cu11 11.7.99 nvidia-cudnn-cu11 8.5.0.96 nvidia-cufft-cu11 10.9.0.58 nvidia-curand-cu11 10.2.10.91 nvidia-cusolver-cu11 11.4.0.1 nvidia-cusparse-cu11 11.7.4.91 nvidia-nccl-cu11 2.14.3 nvidia-nvtx-cu11 11.7.91 openjpeg 2.4.0 openssl 3.0.17 packaging 23.1 pandas 2.0.3 parso 0.8.3 partd 1.4.0 pathy 0.10.1 patsy 0.5.6 pcre 8.45 pennylane 0.27.0 pennylane-lightning 0.28.0 pennylane-sf 0.20.1 pexpect 4.8.0 pickleshare 0.7.5 pillow 10.0.1 pip 23.3 piq 0.8.0 pkgutil-resolve-name 1.3.10 platformdirs 3.5.1 plotly 5.24.1 ply 3.11 pooch 1.7.0 preshed 3.0.6 prompt-toolkit 3.0.38 psutil 5.9.5 pthread-stubs 0.4 ptyprocess 0.7.0 pure-eval 0.2.2 pycparser 2.21 pydantic 1.10.12 pygments 2.15.1 pynndescent 0.5.13 pyopenssl 23.2.0 pyparsing 3.0.9 pypng 0.20220715.0 pyqt 5.15.10 pyqt5-sip 12.13.0 pyrsistent 0.20.0 pysocks 1.7.1 python 3.8.18 python-dateutil 2.8.2 python-dotenv 0.21.0 python-fastjsonschema 2.15.1 python-kaleido 0.2.1 python-lmdb 1.4.1 python-tzdata 2023.3 python_abi 3.8 pytz 2023.3.post1 pyyaml 6.0 pyzbar 0.1.9 pyzmq 25.0.2 qrcode 7.4.2 qt-main 5.15.2 quantum-blackbird 0.5.0 quantum-xir 0.2.2 qutip 4.7.3 readline 8.2 requests 2.31.0 retworkx 0.15.1 rustworkx 0.15.1 scikit-learn 1.3.2 scipy 1.10.1 seaborn 0.13.2 seaborn-base 0.13.2 semantic-version 2.10.0 setuptools 68.0.0 shellingham 1.5.0 sip 6.7.12 six 1.16.0 smart_open 5.2.1 sortedcontainers 2.4.0 spacy 3.5.3 spacy-legacy 3.0.12 spacy-loggers 1.0.4 sqlite 3.41.2 srsly 2.4.8 stack-data 0.6.2 statsmodels 0.13.5 strawberryfields 0.23.0 sympy 1.12 tbb 2021.8.0 tblib 1.7.0 tenacity 8.5.0 termcolor 2.1.0 thewalrus 0.20.0 thinc 8.1.10 threadpoolctl 3.5.0 tk 8.6.12 toml 0.10.2 tomli 2.0.1 toolz 0.12.0 torch 2.0.1 torchaudio 2.0.2 torchvision 0.15.2 tornado 6.3.2 tqdm 4.65.0 traitlets 5.9.0 triton 2.0.0 typer 0.4.1 typing-extensions 4.8.0 typing_extensions 4.7.1 umap-learn 0.5.7 unicodedata2 15.1.0 urllib3 1.26.18 wasabi 0.9.1 wcwidth 0.2.6 wheel 0.41.2 xanadu-cloud-client 0.3.1 xorg-libxau 1.0.11 xorg-libxdmcp 1.1.3 xz 5.4.2 yaml 0.2.5 zict 3.0.0 zipp 3.15.0 zlib 1.2.13 zstd 1.5.5