HDF5

Hierarchical Data Format (HDF) is a set of file formats (HDF4, HDF5) designed to store and organize large amounts of data.
Document for HDF5 Python API: https://docs.h5py.org/en/stable/build.html

Installation

Installation with conda:

conda install h5py

Installation with pre-built wheels

pip install h5py

Usage

import h5py

h5_file_name = "my_data.h5"
h5_writer = h5py.File(h5_file_name, 'a')  # indicate the file to store the data

for index in len(**YOUR_DATA_LIST**):
    set_intensity.create_dataset(f"{index:06}", data=**YOUR_DATA**) # save data

    if (index + 1) % 1500 == 0: # force to save once every 1500 records.
        print('Finish processing one section\n')
        h5_writer.close()
        time.sleep(1)
        h5_writer = h5py.File(h5_file_name, 'a')
        continue

# when finished!
h5_writer.close()

References

[1] https://en.wikipedia.org/wiki/Hierarchical_Data_Format
[2] https://docs.h5py.org/en/stable/build.html

Acknowledgement

Codes are from Qiuliang Ye