HDF5
Hierarchical Data Format (HDF) is a set of file formats (HDF4, HDF5) designed to store and organize large amounts of data.
Document for HDF5 Python API: https://docs.h5py.org/en/stable/build.html
Installation
Installation with conda:
conda install h5py
Installation with pre-built wheels
pip install h5py
Usage
import h5py h5_file_name = "my_data.h5" h5_writer = h5py.File(h5_file_name, 'a') # indicate the file to store the data for index in len(**YOUR_DATA_LIST**): set_intensity.create_dataset(f"{index:06}", data=**YOUR_DATA**) # save data if (index + 1) % 1500 == 0: # force to save once every 1500 records. print('Finish processing one section\n') h5_writer.close() time.sleep(1) h5_writer = h5py.File(h5_file_name, 'a') continue # when finished! h5_writer.close()
References
[1] https://en.wikipedia.org/wiki/Hierarchical_Data_Format
[2] https://docs.h5py.org/en/stable/build.html
Acknowledgement
Codes are from Qiuliang Ye