Easy NetCDF Toolbox

1. Introduction

Highlighted features

Features

Known issues

File structure

Frequently used functions

How to add this package to the MATLAB path

GUI (MATLAB Desktop)

Command line

addpath('/path/to/Easy_NetCDF');
savepath

2. Read data from NetCDF file(s)

data = FUN_nc_load_all_variables( fn );

data = FUN_nc_load_all_variables( fn, 'time_var_name', var_time );

2.1 Read a variable from one NetCDF file

2.1.1 Read data in its original type and values

INPUT

OUTPUT

Example 1

data = FUN_nc_varget( 'Demo_SST_2001.nc', 'sst');

2.1.2 Apply scales, add_offsets and _FillValues to the raw data

INPUT

OUTPUT

Example 2

data = FUN_nc_varget_enhanced( 'Demo_SST_2001.nc', 'sst'); 

2.2 Read a subset of a variable from one NetCDF file

2.2.1 Specify the subset boundary by [start, count stride];

INPUT

example 3

% parameters
fn = 'Demo_SST_2001.nc';
lonlimit = [-110 -20];
latlimit = [15 70];
% read lon/lat
lon = FUN_nc_varget_enhanced( fn, 'lon' );
lat = FUN_nc_varget_enhanced( fn, 'lat' );
% calculate [start, count] from range for lon/lat.
[x_start, x_count, xloc] = FUN_nc_varget_sub_genStartCount( lon, lonlimit );
[y_start, y_count, yloc] = FUN_nc_varget_sub_genStartCount( lat, latlimit );
nc_start = [ x_start, y_start, 0 ] % the third value is for time. 
nc_count = [ x_count, y_count, 1 ] 
nc_stride= [1, 1, 1];
% load data
data = FUN_nc_varget_enhanced_region( fn, 'sst', nc_start, nc_count, nc_stride);

2.2.2 Specify the subset boundary by longitude, latitude, ...

INPUT

OUTPUT

example 4: Read May 2001 SST between 110W-20W, 15N-70N

fn = 'Demo_SST_2001.nc';
dim_name = {'lon','y', 'time'}; 
dim_limit = { [-110 -20], [15 70], [datenum(2001,5,1) datenum(2001,5,31)] }; 
time_var_name = 'time';
dim_varname = {'lon','lat','time'};
varname = 'sst';
[ out_dim, data ] = FUN_nc_varget_enhanced_region_2( fn, 'sst', dim_name, dim_limit, time_var_name, dim_varname );
% Plot
%FUN_MAP_pcolor_lonlat_quick( lon, lat, data(:,:,1)');
pcolor( out_dim.lon, out_dim.lat, data' );
cbar = colorbar;
shading interp
title(datestr(out_dim.time))

example 5: Read SST between 180W-180E, 0N-50N in the second month of 2001

fn = 'Demo_SST_2001.nc';
dim_name = { 'y' 'time'}; 
dim_limit = { [0 50], [2,2] }; 
time_var_name = [];
dim_varname = {'lat', nan}; 
varname = 'sst';
[ out_dim, data ] = FUN_nc_varget_enhanced_region_2( fn, 'sst', dim_name, dim_limit, time_var_name, dim_varname );
% Plot
pcolor( out_dim.lon, out_dim.lat, data' );
cbar = colorbar;
shading interp
axis equal
%title(datestr(out_dim.time))

2.3 Read a variable from multiple NetCDF files

INPUT:

 filelist [struct array]: name and folder of the NetCDF file
 filelist must include two attributes: `name` and `folder`. For 
 each element of filelist (e.g. the ith one), the full path
 will be generated by fullfile( filelist(ith).folder, filelist(ith).name)
 It can also be a cell array contain paths of files,
 or a char matrix, each row of which contains one path.
 varname [char]: name of the variable
 dim_limit_str [cell]: name of dimensions, like {'lon','lat'}. Dimensions with custom limits must be listed here. Other dimensions are optional.
 dim_limit_limit [cell]: limits of dimensions, like {[-85 -55], [30 45]}.
 + It can be empty, indicating no limit for all dimensions.
 + For any specific dimension, the limit contains more than 2 values, 
 it will be treated as a list of discrete indexes to be read from the file.
 see Example 8 & 9 in readme.md .
 merge_dim_name [string]: name of the dimension in which the variables 
 from different files will be concatenated. If merge_dim_name is
 empty, the variable will be concatenated after its last
 dimension.
 + Example 6: if you want to read gridded daily
 temperature given in [lon, lat, depth, time] from a set of
 files, and each file contains temperature in one day,
 the merge_dim_name should be 'time'. 
 + Example 7: if you want to read gridded daily temperature given in
 [lon, lat, depth], in which time is not given
 explicitly in each file, you can leave merge_dim_name
 empty.
 time_var_name [char, optional]: name of the time axis
 + variable defined by this will be loaded into time in "MATLAB units" (days since 0000-01-00)
 + This is helpful for setting timelimit in a easy way, avoiding
 calculating the timelimit from units in NetCDF files.
 For example, to read data between 02/15/2000 00:00 and
 02/16/2000 00:00 from a NetCDF file, which includes a time variable "ob_time" 
 in units of "days since 2000-00-00 00:00", you need to set 
 timelimit as [46 47] when time_var_name is empty. However, you
 should set timelimit as [datenum(2000,2,15),
 datenum(2000,2,16)] if the tiem_var_name is set to "ob_time".
dim_varname [cell, optional]: name of the variable defining the axis at each dimension.
 + by default, each axis is defined by a variable sharing the same name as the dimension. 
 + "`dim_varname{1} = nan` indicates that the axis is not defined by any variable in file. It will be defined 
 as 1, 2, 3, ... Nx, where Nx is the length of the dimension.

OUTPUT:

 out_dim : dimension info (e.g., longitude, latitude, if applicable)
 data : data extracted from the given NetCDF file. 

Example 6: Read SST from Dec 2001 to May 2003

filelist = dir('Demo_*.nc');
varname = 'sst';
dim_name = { 'time' };
dim_limit = { [datenum(2001,12,1) datenum(2003,5,31)] };
merge_dim_name = 'time';
time_var_name = 'time';
dim_varname = [];
[ out_dim, data ] = FUN_nc_varget_enhanced_region_2_multifile( filelist, varname, dim_name, dim_limit, merge_dim_name, time_var_name, dim_varname );

filelist = { 'Demo_SST_2001.nc'
 'Demo_SST_2002.nc'
 'Demo_SST_2003.nc'
 'Demo_SST_2004.nc'
 'Demo_SST_2005.nc'
 'Demo_SST_2006.nc'
 'Demo_SST_2007.nc'
 'Demo_SST_2008.nc'
 'Demo_SST_2009.nc'
 'Demo_SST_2010.nc'};

filelist = ['Demo_SST_2001.nc'
 'Demo_SST_2002.nc'
 'Demo_SST_2003.nc'
 'Demo_SST_2004.nc'
 'Demo_SST_2005.nc'
 'Demo_SST_2006.nc'
 'Demo_SST_2007.nc'
 'Demo_SST_2008.nc'
 'Demo_SST_2009.nc'
 'Demo_SST_2010.nc'];

Example 7: Read SST from Dec 2001 to Nov 2003 in Northwest Atlantic

filelist = dir('Demo_*.nc');
varname = 'sst';
dim_name = { 'lon', 'y', 'time' }; % In the demo files, the meridional dimension is named as "y".
dim_limit = { [-110 -20], [15 70], [datenum(2001,12,1) datenum(2003,11,30)] };
merge_dim_name = 'time'; % merge data in "time" dimension.
time_var_name = 'time'; % convert values in "time" to MATLAB units (days since 0000-01-00 00:00).
dim_varname = {'lon','lat','time'}; % This is to force the function to read values for the meridional dimension from the variable "lat".
[ out_dim, data ] = FUN_nc_varget_enhanced_region_2_multifile( filelist, varname, dim_name, dim_limit, merge_dim_name, time_var_name, dim_varname );

Example 8: read data by incontinuous indexes

filelist = dir('./Documents_and_demo/Demo_*.nc');
varname = 'sst';
dim_name = { 'lon', 'y', 'time' }; % In the demo files, the meridional dimension is named as "y".
dim_limit = { [1, 2, 3, 9, 10, 16, 19], [4, 5, 6, 8, 9], [datenum(2001,12,1) datenum(2002,6,30)] };
merge_dim_name = 'time'; % merge data in "time" dimension.
time_var_name = 'time'; % convert values in "time" to matlab units (days since 0000-01-00 00:00).
dim_varname = []; % {'lon','lat','time'}; % This is to force the function to read values for the meridional dimension from the variable "lat".
[ out_dim, data ] = FUN_nc_varget_enhanced_region_2_multifile( filelist, varname, dim_name, dim_limit, merge_dim_name, time_var_name, dim_varname );

out_dim = 
 struct with fields:
 lon: [7x1 double]
 y: [4 5 6 8 9]
 time: [731186 731217 731248 731276 731307 731337 731368]
 Name Size Bytes Class Attributes
 data 7x5x7 1960 double 

Example 9: read salinity in Northwest Atlantic from a CMIP6 experiment (unconstructed grid)

% list files
filelist = dir('W:\Data_climate\CMIP6\ssp245_AWI-CM-MR_mon\so_Omon_AWI-CM-1-1-MR_ssp245_r1i1p1f1_gn_*.nc');
% read time for years from 2068 to 2075
varname = 'time';
dim_name = { 'time' }; % 
dim_limit = { [datenum(2068,1,1) datenum(2075,12,31)] };
merge_dim_name = 'time'; % merge data in "time" dimension.
time_var_name = 'time'; % convert values in "time" to matlab units (days since 0000-01-00 00:00).
dim_varname = []; % the variables defines each dimension in dim_name. It is assigned automatically if left empty.
[ ~, time ] = FUN_nc_varget_enhanced_region_2_multifile( filelist, 'time', dim_name, dim_limit, merge_dim_name, time_var_name, dim_varname );
% read longitude/latitude
[ ~, lon ] = FUN_nc_varget_enhanced_region_2_multifile( filelist, 'lon', dim_name, dim_limit, merge_dim_name, time_var_name, dim_varname );
[ ~, lat ] = FUN_nc_varget_enhanced_region_2_multifile( filelist, 'lat', dim_name, dim_limit, merge_dim_name, time_var_name, dim_varname );
% get the index for the focus longitude/latitude
ind = find( lon >= -90 & lon <= -50 & lat >= 15 & lat <= 45);
% read surface data (salinity here) within the region.
varname = 'so';
dim_name = { 'ncells', 'depth', 'time' }; 
dim_limit = { ind, [1 1], [datenum(2068,1,1) datenum(2075,12,31)] }; 
merge_dim_name = 'time'; % merge data in "time" dimension.
time_var_name = 'time'; % convert values in "time" to matlab units (days since 0000-01-00 00:00).
dim_varname = {[], nan,'time'}; % the variables defines each dimension in dim_name. It is assigned automatically if left empty.
 % nan indicate that the limit ([1 1]) in dim_lim is a limit of index instead of a limit of actual depth
 % [] will be handled automatically. Here, it is equal to nan.
[ out_dim, so ] = FUN_nc_varget_enhanced_region_2_multifile( filelist, varname, dim_name, dim_limit, merge_dim_name, time_var_name, dim_varname );
% read full-depth data (salinity here) within the region.
varname = 'so';
dim_name = { 'ncells', 'time' }; % `depth` is removed since the full depth should be loaded without any spcific limits applied.
dim_limit = { ind, [datenum(2068,1,1) datenum(2075,12,31)] }; 
merge_dim_name = 'time'; % merge data in "time" dimension.
time_var_name = 'time'; % convert values in "time" to matlab units (days since 0000-01-00 00:00).
dim_varname = {[], 'time'}; % the variables defines each dimension in dim_name. It is assigned automatically if left empty.
 % nan indicate that the limit ([1 1]) in dim_lim is a limit of index instead of a limit of actual depth
 % [] will be handled automatically. Here, it is equal to nan.
[ out_dim, so ] = FUN_nc_varget_enhanced_region_2_multifile( filelist, varname, dim_name, dim_limit, merge_dim_name, time_var_name, dim_varname );

2.4 Read a subset from hundreds of files quickly

%% generate info -----------------------------------------------------------
 filelist = dir('Demo_*.nc');
 merge_dim_name = 'time'; % merge data in "time" dimension.
 dim_name = { 'lon', 'y', 'time' }; % In the demo files, the meridional dimension is named as "y".
 dim_varname = {'lon','lat','time'}; % This is to force the function to read values for the meridional dimension from the variable "lat". 
 time_var_name = 'time'; % convert values in "time" to MATLAB units (days since 0000-01-00 00:00). This is optional
 output_file_path = 'Presaved_info_demo.mat';
 % Please note that **absolute** file paths are saved in the generated file. If you moved the data, you need to run this again
 pregen_info = FUN_nc_gen_presaved_netcdf_info_v2( filelist, merge_dim_name, dim_name, dim_varname, time_var_name, output_file_path );
%% read data --------------------------------------------------------------
 varname = 'sst';
 dim_name = { 'time' };
 dim_limit = { [datenum(2001,12,1) datenum(2003,5,31)] };
 merge_dim_name = 'time';
 presaved_info = load(output_file_path);
 presaved_info = presaved_info.pregen_info;
 [ out_dim, data_out ] = FUN_nc_varget_enhanced_region_2_multifile( presaved_info, varname, dim_name, dim_limit);

3. Download files and merge files

3.1 Download NetCDF files via OpenDAP (FUN_nc_OpenDAP_with_limit)

INPUT:

 filename0 : source of the NetCDF file (OpenDAP URL here) 
 filename1 : Name of output NetCDF file 
 dim_limit_var : which axes you want to set the limit 
 dim_limit_val : the limit of each axis 
 var_download : the variable you'd like to download. [var_download = [] will download all variables.] 
 var_divided : the variables that need to be downloaded piece by piece in a specific dimension. In many cases, OpenDAP may not respond if you try to download too much data at once. A solution for this is to download data piece by piece 
divided_dim_str: which dimension you'd like to download piece by piece (e.g., 'time', or 'depth'). divided_dim_str = [] means all variables will be downloaded completely at once. 
 Max_Count_per_group: Max number of points in the divided dimension. 

Optional parameters:

Output

N/A

Example 10: download a subset of HYCOM data from its OpenDAP server

% HYCOM dataset at an OpenDAP server
filename0 = 'http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1/2012';
% output filename
filename1 = 'HYCOM_test2.nc';
% calculate time limits
 timelimit = [datenum(2012,1,1) datenum(2012,1,3)];
 time = FUN_nc_varget(filename0,'time');
 time_unit = FUN_nc_attget(filename0,'time','units');
 [time0, unit_str, unit_to_day] = FUN_nc_get_time0_from_str( time_unit );
 timelimit = (timelimit - time0)/unit_to_day ;
% set limits
 lonlimit = [-76 -70 ];
 latlimit = [32 39];
 depthlimit = [0 100];
 dim_limit_var = {'lon','lat','depth','time'};
 dim_limit_val = {lonlimit, latlimit depthlimit timelimit};
% variable to be downloaded
 var_download = {'water_temp','lon','lat','depth','time'}; % empty indicates downloading all variables
% Variables that should be downloaded block by block
 var_divided = {'water_temp'};
% which dim you'd like to download block by block (e.g., 'time', or 'depth')
divided_dim_str = 'depth'
% max size of each "piece"
Max_Count_per_group = 5;
 FUN_nc_OpenDAP_with_limit( filename0, filename1, dim_limit_var, dim_limit_val, var_download, var_divided, divided_dim_str, Max_Count_per_group )

Example 11: A new template

% HYCOM dataset at an OpenDAP server
opendapURL = 'http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1/2012';
% output filename
filename_out = 'HYCOM_test1.nc';
% calculate time limits
 timelimit = [datenum(2012,1,1) datenum(2012,1,3)];
 time_varname = 'time'; %Tell the code which variable contains time
% set limits
 lonlimit = [-76 -70 ];
 latlimit = [32 39];
 depthlimit = [0 100];
 dim_limit_var = {'lon', 'lat', 'depth', 'time' };
 dim_limit_val = {lonlimit, latlimit, depthlimit, timelimit};
% variable to be downloaded
 var_download = {'water_temp','lon','lat','depth','time'}; % empty indicates downloading all variables
% Variables that should be downloaded block by block
 var_divided = var_download;
 
% which dim you'd like to download block by block (e.g., 'time', or 'depth')
divided_dim_str = 'depth';
% max size of each "piece"
N_divided_rec_per_group = 2;
 FUN_nc_OpenDAP_with_limit( opendapURL, filename_out, dim_limit_var, dim_limit_val, var_download, var_divided, divided_dim_str, N_divided_rec_per_group, 'time_var_name', time_varname);

3.2 Merge multiple NetCDF files in time (FUN_nc_merge)

INPUT:

 input_dir: The folder in which all input NetCDF files given by `filelist` are located.
 filelist : the list of files which will be merged. This should be generated by the MATLAB built-in command `dir`. 
 This function will merge the NetCDF files following the order given in this variable. Please make sure this variable has been sorted properly.
 output_fn : Name of output NetCDF file
 merge_dim_name : name of the dimension in which all variables will be merged.
 compatibility_mode: 
 compatibility_mode = 1: write netCDF in 'CLOBBER'; Compression would be disabled.
 compatibility_mode = 0: write netCDF in 'NETCDF4'.

Output:

Example 12

% input_dir: path for the folder containing the files
 input_dir = '.';
% filelist
 filelist = dir(fullfile(input_dir,'Merge_Demo*.nc'));
% output filename
 output_fn = 'Merged_Output.nc';
% name of the demension to be merged.
merge_dim_name = 'time';
% compatibility_mode:
% compatibility_mode = 1: write netCDF in 'CLOBBER'; Compression would be disabled.
% compatibility_mode = 0: write netCDF in 'NETCDF4'.
compatibility_mode = 0;
strvcat( filelist(:).name )
FUN_nc_merge( input_dir, filelist, output_fn, merge_dim_name, compatibility_mode )

3.3 FUN_nc_merge_save_mean

% input_dir: path for the folder containing the files
 input_dir = '.';
% filelist
 filelist = dir(fullfile(input_dir,'Merge_Demo*.nc'));
% Output filename
 output_fn = 'Merged_Output_mean.nc';
% Name of the demension to be merged.
 merge_dim_name = 'time';
% compatibility_mode:
% compatibility_mode = 1: write netCDF in 'CLOBBER';
% compatibility_mode = 0: write netCDF in 'NETCDF4';
 compatibility_mode = 0;
% Variable should not be included in the output file.
 list_var_excluded = [];
% Execute
FUN_nc_merge_save_mean( input_dir, filelist, output_fn, merge_dim_name, compatibility_mode, list_var_excluded );

4. Write NetCDF Files

4.1 FUN_nc_easywrite_enhanced

INPUT

 filename [char]: name of the output NetCDF file (e.g., 'test.nc')
 dim_name [cell]: names of dimensions (e.g., {'lon','lat'}
 dim_length [array]: length of each dimension (e.g., [ 360, 180 ] )
 varname [cell]: names of variables (e.g., {'lon','lat','sst','ssh'}
 dimNum_of_var [cell]: dimensional ID for each variable (e.g., `{ 1, 2, [1,2],[1,2]}`). A value of 1 indicates the first dimension in `dim_name`, 2 indicates the second, and so on.
 data [cell]: values for each variable (e.g., {lon,lat,sst,ssh})
 global_str_att: global attribute

Optional paramaters

OUTPUT

 N/A

Example 13

% ---- generate random data ----
lon = [-75:-55] ;
lat = [26 : 55] ;
depth= [0:5:100] ;
temp = rand( length(lon), length(lat), length(depth) );
%% ---- write nctCDF ----
filename = 'Test_random_values.nc';
dim_name = {'lon','lat','depth'};
dim_length = [length(lon), length(lat), length(depth)];
varname = {'temp', 'lon','lat','depth'};
dimNum_of_var = {[1,2,3], 1, 2, 3 };
data = { temp, lon, lat, depth };
global_att = 'This is a test.';
FUN_nc_easywrite_enhanced( filename, dim_name, dim_length, varname, dimNum_of_var, data, global_att )
% FUN_nc_easywrite_enhanced('temp.nc',...
% {'Node','Cell','time'},[1000 2000 500],...
% {'node_lon','node_lat','lon_cell','lat_cell','sst'},{1,1,2,2,[1 3]},...
% {lon_node,lat_node,lon_cell,lat_cell,sst},'This is an example');

4.2 Other functions for writting a NetCDF file