Supported Instrument Data Sets
Currently, support is included for files from the following sources:
SuperMAG (
ocbpy.instruments.supermag)SuperDARN Vorticity (
ocbpy.instruments.vort)
These routines may be used as a guide to write routines for other data sets.
A ocbpy.instruments.general loading sub-module is also provided for
ASCII files. All the non-boundary data routines are part of the
ocbpy.instruments module. Support for time-handling that may be useful
for specific data sets are provided in ocbpy.ocb_time.
General Instrument Module
General loading routines for data files.
- ocbpy.instruments.general.load_ascii_data(filename, hlines, gft_kwargs={}, hsplit=None, datetime_cols=None, datetime_fmt=None, int_cols=None, str_cols=None, max_str_length=50, header=None)[source]
Load an ASCII data file into a dict of numpy arrays.
- Parameters:
filename (str) – data file name
hlines (int) – number of lines in header. If zero, must include header.
gft_kwargs (dict) – Dictionary holding optional keyword arguments for the numpy genfromtxt routine (default=dict())
hsplit (str, NoneType) – character seperating data labels in header. None splits on all whitespace characters. (default=None)
datetime_cols (list, NoneType) – If there are date strings or values that should be converted to a datetime object, list them in order here. Not processed as floats. NoneType produces an empty list. (default=None)
datetime_fmt (str, NoneType) – Format needed to convert the datetime_cols entries into a datetime object. Special formats permitted are: ‘YEAR SOY’, ‘SOD’. ‘YEAR SOY’ must be used together; ‘SOD’ indicates seconds of day, and may be used with any date format (default=None)
int_cols (list, NoneType) – Data that should be processed as integers, not floats. NoneType produces an empty list. (default=None)
str_cols (list, NoneType) – Data that should be processed as strings, not floats. NoneType produces an empty list. (default=None)
max_str_length (int) – Maximum allowed string length. (default=50)
header (list, NoneType) – Header string(s) where the last line contains whitespace separated data names. NoneType produces an empty list. (default=None)
- Returns:
header (list of strings) – Contains all specified header lines
out (dict of numpy.arrays) – The dict keys are specified by the header data line, the data for each key are stored in the numpy array
Notes
Data is assumed to be float unless otherwise stated.
SuperMAG Instrument Module
Perform OCB gridding for SuperMAG data.
Notes
SuperMAG data available at: http://supermag.jhuapl.edu/
- ocbpy.instruments.supermag.load_supermag_ascii_data(filename)[source]
Load a SuperMAG ASCII data file.
- Parameters:
filename (str) – SuperMAG ASCI data file name
- Returns:
header (list) – List of strings containing the header
out (dict) – The dict keys are specified by the header data line, the data for each key are stored in the numpy array
- ocbpy.instruments.supermag.supermag2ascii_ocb(smagfile, outfile, hemisphere=0, ocb=None, ocbfile='default', instrument='', max_sdiff=600, min_merit=None, max_merit=None, scale_func=<function normal_curl_evar>, **kwargs)[source]
Covert and scales the SuperMAG data into OCB coordinates.
- Parameters:
smagfile (str) – File containing the required SuperMAG file sorted by time
outfile (str) – Filename for the output data
hemisphere (int) – Hemisphere to process (can only do one at a time). 1=Northern, -1=Southern, 0=Determine from data (default=0)
ocb (ocbpy.OCBoundary, ocbpy.DualBoundary, or NoneType) – OCBoundary or DualBoundary object with data loaded already. If None, looks to ocbfile and creates an OCBoundary object. (default=None)
ocbfile (str) – File containing the required OC Boundary data sorted by time, or ‘default’ to load default file for time and hemisphere. Only used if no OCBoundary object is supplied (default=’default’)
instrument (str) – Instrument providing the OCBoundaries. Requires ‘image’ or ‘ampere’ if a file is provided. If using filename=’default’, also accepts ‘amp’, ‘si12’, ‘si13’, ‘wic’, and ‘’. (default=’’)
max_sdiff (int) – Maximum seconds between OCB and data record in sec (default=60)
min_merit (float or NoneType) – Minimum value for the default figure of merit or None to not apply a custom minimum (default=None)
max_merit (float or NoneTye) – Maximum value for the default figure of merit or None to not apply a custom maximum (default=None)
scale_func (function or NoneType) – Scale the magnetic field observations unless None (default=ocbpy.ocb_scale.normal_curl_evar)
kwargs (dict) – Dict with optional selection criteria. The key should correspond to a data attribute and the value must be a tuple with the first value specifying ‘max’, ‘min’, ‘maxeq’, ‘mineq’, or ‘equal’ and the second value specifying the value to use in the comparison.
- Raises:
IOError – If unable to open the input or output file
Notes
May only process one hemisphere at a time.
See also
ocbpy.ocb_scale.normal_curl_evar
SuperDARN Vorticity Instrument Module
Perform OCB gridding for SuperDARN vorticity data.
Notes
Specialised SuperDARN data product, available from the British Antarctic Survey.
References
Chisham, G. (2023). Ionospheric vorticity across the northern hemisphere ionosphere determined from particular SuperDARN radar pairs - 2000 to 2005 inclusive (Version 1.0) [Data set]. NERC EDS UK Polar Data Centre. https://doi.org/10.5285/8EEDC594-730B-4AAD-B9CE-827912320C3A
- ocbpy.instruments.vort.load_vorticity_ascii_data(vortfile, save_all=False)[source]
Load SuperDARN vorticity data files.
- Parameters:
- Returns:
vdata – Dictionary of numpy arrays
- Return type:
- Raises:
IOError – If an unexpected data format is encountered
- ocbpy.instruments.vort.vort2ascii_ocb(vortfile, outfile, hemisphere=0, ocb=None, ocbfile='default', instrument='', max_sdiff=600, save_all=False, min_merit=None, max_merit=None, scale_func=<function normal_curl_evar>, **kwargs)[source]
Covert the location of vorticity data from AACGM to OCB coordinates.
- Parameters:
vortfile (str) – file containing the required vorticity file sorted by time
outfile (str) – filename for the output data
hemisphere (int) – Hemisphere to process (can only do one at a time). 1=Northern, -1=Southern, 0=Determine from data (default=0)
ocb (ocbpy.ocboundary.OCBoundary, ocbpy.DualBoundary, or NoneType) – OCBoundary or DualBoundary object with data loaded already. If None, looks to ocbfile and creates an OCBoundary object. (default=None)
ocbfile (str) – File containing the required OC boundary data sorted by time, or ‘default’ to load default file for time and hemisphere. Only used if no OCBoundary object is supplied (default=’default’)
instrument (str) – Instrument providing the OCBoundaries. Requires ‘image’ or ‘ampere’ if a file is provided. If using filename=’default’, also accepts ‘amp’, ‘si12’, ‘si13’, ‘wic’, and ‘’. (default=’’)
max_sdiff (int) – maximum seconds between OCB and data record in sec (default=600)
save_all (bool) – Save all data (True), or only that needed to calcuate OCB and vorticity (False). (default=False)
min_merit (float or NoneType) – Minimum value for the default figure of merit or None to not apply a custom minimum (default=None)
max_merit (float or NoneTye) – Maximum value for the default figure of merit or None to not apply a custom maximum (default=None)
scale_func (function or NoneType) – Scaling function for Vorticity data or None to not scale (default=ocbpy.ocb_scale.normal_curl_evar)
kwargs (dict) – Dict with optional selection criteria for quality boundaries. The key should correspond to a data attribute and the value must be a tuple with the first value specifying ‘max’, ‘min’, ‘maxeq’, ‘mineq’, or ‘equal’ and the second value specifying the value to use in the comparison.
- Raises:
IOError – If unable to open input or output file
ValueError – If unable to retrieve all necessary data from the input file
Notes
Input header or col_names must include the names in the default string.
pysat Instrument Module
Perform OCB gridding for appropriate instrument data loaded in pysat.
Notes
pysat is available at: http://github.com/pysat/pysat or pypi
- ocbpy.instruments.pysat_instruments.add_ocb_to_data(pysat_inst, mlat_name='', mlt_name='', height_name='', evar_names=None, curl_evar_names=None, vector_names=None, height=350.0, hemisphere=0, ocb=None, ocbfile='ocb', instrument='', max_sdiff=60, min_merit=None, max_merit=None, loc_coord='magnetic', vect_coord='magnetic', **kwargs)[source]
Covert the location of pysat data into OCB, EAB, or Dual coordinates.
- Parameters:
pysat_inst (pysat.Instrument) – pysat.Instrument class object containing magnetic coordinates
mlat_name (str) – Instrument data key or column for latitudes (default=’’)
mlt_name (str) – Instrument data key or column for local times (default=’’)
height_name (str) – Instrument data key or column for altitude (default=’’)
evar_names (list or NoneType) – List of Instrument data keys or columns pointing to measurements that are proportional to the electric field (E); e.g. ion drift (default=None)
curl_evar_names (list or NoneType) – List of Instrument data keys or columns pointing to measurements that are proportional to the curl of E; e.g. ion vorticity (default=None)
vector_names (dict or NoneType) – Dict of Instrument data keys or columns pointing to measurements that are vectors that are proportional to either E or the curl of E. The key should correspond to one of the values in the evar_names or curl_evar_names list. If this is not done, a scaling function must be provided. The value corresponding to each key must be a dict that indicates the names holding data needed to initialise the ocbpy.ocb_scaling.VectorData object (default=None)
height (float or array-like) – Altitude value(s) to use if no height variable is provided by height_name (default=350.0)
hemisphere (int) – Hemisphere to process (can only do one at a time). 1=Northern, -1=Southern, 0=Determine from data (default=0)
ocb (ocbpy.OCBoundary, ocbpy.DualBoundary, or NoneType) – OCBoundary or DualBoundary object with data loaded already. If None, looks to ocbfile and creates an OCBoundary object. (default=None)
ocbfile (str) – file containing the required OC boundary data sorted by time, ignorned if OCBoundary object supplied. To use the default for a boundary type, supply the desired boundary type; ‘eab’, ‘ocb’, or ‘dual’. (default=’ocb’)
instrument (str) – Instrument providing the OCBoundaries. Requires ‘image’ or ‘ampere’ if a file is provided. If using filename=’default’, also accepts ‘amp’, ‘si12’, ‘si13’, ‘wic’, and ‘’ (default=’’)
max_sdiff (int) – maximum seconds between OCB and data record in sec (default=60)
min_merit (float or NoneType) – Minimum value for the default figure of merit or None to not apply a custom minimum (default=None)
max_merit (float or NoneTye) – Maximum value for the default figure of merit or None to not apply a custom maximum (default=None)
loc_coord (str) – Name of the coordinate system for mlat_name and mlt_name; one of ‘magnetic’, ‘geocentric’, or ‘geodetic’. If not ‘magnetic’, height_name or height will be used to convert the data to magnetic coordinates. (default=’magnetic’)
vect_coord (str) – Name of the coordinate system for vect_n and vect_e; one of ‘magnetic’, ‘geocentric’, or ‘geodetic’. If not ‘magnetic’, height_name or height will be used to convert the data to magnetic coordinates. (default=’magnetic’)
kwargs (dict) – Dict with optional selection criteria or criteria for initializing a DualBoundary class object (in combination with ocb=None and ocbfile=’dual’). For the optional selection criteria, the key should correspond to a data attribute and the value must be a tuple with the first value specifying ‘max’, ‘min’, ‘maxeq’, ‘mineq’, or ‘equal’ and the second value specifying the value to use in the comparison.
- Raises:
ValueError – If the pysat Instrument doesn’t have the necessary data values or if the input provided is not a pysat Instrument.
Notes
This may be run on a pysat instrument or as a custom function when loading pysat data.
Examples
# Example vector name input looks like: vector_names={'vel': {'vect_n': 'vel_n', 'vect_e': 'vel_e', 'dat_name': 'velocity', 'dat_units': 'm/s'}, 'dat': {'vect_n': 'dat_n', 'vect_e': 'dat_e', 'scale_func': local_scale_func}}
- ocbpy.instruments.pysat_instruments.add_ocb_to_metadata(pysat_inst, ocb_name, pysat_name, overwrite=False, notes='', isvector=False)[source]
Update pysat metadata for OCB data.
- Parameters:
pysat_inst (pysat.Instrument) – pysat.Instrument class object containing magnetic coordinates
ocb_name (str) – Data column name for boundary data
pysat_name (str) – Data column name for non-adaptive boundary version of this data
overwrite (bool) – Overwrite existing metadata, if present (default=False)
notes (str)) – Notes about this boundary data (default=’’)
isvector (bool) – Is this vector data or not (default=False)
- Raises:
ValueError – If input pysat Instrument object is the wrong class
- ocbpy.instruments.pysat_instruments.reshape_pad_mask_flatten(data, mask)[source]
Reshape, pad, mask, and flatten data.
- Parameters:
data (xr.DataArray) – Data to be reshaped, padded, masked, and flattened for processing.
mask (xr.DataArray) – Mask with the desired dimensions
- Returns:
flat – Flattened array of good data, as specified by the mask
- Return type:
np.array
Time Handling Module
Routines to convert from different file timekeeping methods to datetime.
- ocbpy.ocb_time.convert_time(year=None, soy=None, yyddd=None, sod=None, date=None, tod=None, datetime_fmt='%Y-%m-%d %H:%M:%S')[source]
Convert to datetime from multiple time formats.
- Parameters:
year (int or NoneType) – Year or None if not in year-soy format (default=None)
soy (int or NoneType) – Seconds of year or None if not in year-soy format (default=None)
yyddd (str or NoneType) – String containing years since 1900 and 3-digit day of year (default=None)
sod (int, float, or NoneType) – Seconds of day or None if the time of day is not in this format (default=None)
date (str or NoneType) – String containing date information or None if not in date-time format (default=None)
tod (str or NoneType) – String containing time of day information or None if not in date-time format (default=None)
datetime_fmt (str) – String with the date-time or date format (default=’%Y-%m-%d %H:%M:%S’)
- Returns:
dtime – Datetime object
- Return type:
dt.datetime
- ocbpy.ocb_time.datetime2hr(dtime)[source]
Calculate hours of day from datetime.
- Parameters:
dtime (dt.datetime) – Universal time as a timestamp
- Returns:
uth – Hours of day, includes fractional hours
- Return type:
- ocbpy.ocb_time.fix_range(values, min_val, max_val, val_range=None)[source]
Ensure cyclic values lie below the maximum and at or above the mininum.
- Parameters:
- Returns:
fixed_vals – Values adjusted to lie min_val <= fixed_vals < max_val
- Return type:
- ocbpy.ocb_time.get_datetime_fmt_len(datetime_fmt)[source]
Get the lenght of a string line needed for a specific datetime format.
- Parameters:
datetime_fmt (str) – Formatting string used to convert between datetime and string object
- Returns:
str_len – Minimum length of a string needed to hold the specified data
- Return type:
Notes
See the datetime documentation for meanings of the datetime directives
- ocbpy.ocb_time.glon2slt(glon, dtime)[source]
Convert from geographic longitude to solar local time.
- ocbpy.ocb_time.hr2deg(lt, min_deg=-180.0, max_deg=360.0)[source]
Convert from degrees to hours.
- Parameters:
- Returns:
lon – Longitude-like value in degrees
- Return type:
float or array-like
Notes
If min_deg and max_deg specify a range less than 360 degrees, max_deg will be used to identify the desired range. Ranges greater than 360 degrees (e.g., -180 to 360) are allowed.