Supported Instrument Data Sets

Currently, support is included for files from the following sources:

  1. SuperMAG (ocbpy.instruments.supermag)
  2. SuperDARN Vorticity (ocbpy.instruments.vort)
  3. pysat (ocbpy.instruments.pysat_instruments)

These routines may be used as a guide to write routines for other data sets. A ocbpy.instruments.general loading sub-module is also provided for ASCII files. All the non-boundary data routines are part of the ocbpy.instruments module. Support for time-handling that may be useful for specific data sets are provided in ocbpy.ocb_time.

General Instrument Module

General loading routines for data files.

ocbpy.instruments.general.load_ascii_data(filename, hlines, gft_kwargs={}, hsplit=None, datetime_cols=None, datetime_fmt=None, int_cols=None, str_cols=None, max_str_length=50, header=None)[source]

Load an ASCII data file into a dict of numpy arrays.

Parameters:
  • filename (str) – data file name
  • hlines (int) – number of lines in header. If zero, must include header.
  • gft_kwargs (dict) – Dictionary holding optional keyword arguments for the numpy genfromtxt routine (default=dict())
  • hsplit (str, NoneType) – character seperating data labels in header. None splits on all whitespace characters. (default=None)
  • datetime_cols (list, NoneType) – If there are date strings or values that should be converted to a datetime object, list them in order here. Not processed as floats. NoneType produces an empty list. (default=None)
  • datetime_fmt (str, NoneType) – Format needed to convert the datetime_cols entries into a datetime object. Special formats permitted are: ‘YEAR SOY’, ‘SOD’. ‘YEAR SOY’ must be used together; ‘SOD’ indicates seconds of day, and may be used with any date format (default=None)
  • int_cols (list, NoneType) – Data that should be processed as integers, not floats. NoneType produces an empty list. (default=None)
  • str_cols (list, NoneType) – Data that should be processed as strings, not floats. NoneType produces an empty list. (default=None)
  • max_str_length (int) – Maximum allowed string length. (default=50)
  • header (list, NoneType) – Header string(s) where the last line contains whitespace separated data names. NoneType produces an empty list. (default=None)
Returns:

  • header (list of strings) – Contains all specified header lines
  • out (dict of numpy.arrays) – The dict keys are specified by the header data line, the data for each key are stored in the numpy array

Notes

Data is assumed to be float unless otherwise stated.

ocbpy.instruments.general.test_file(filename)[source]

Test to ensure the file is small enough to load.

Parameters:filename (str) – Filename to test
Returns:good_flag – True if good, bad if false
Return type:bool

Notes

Python can only allocate 2GB of data without crashing

SuperMAG Instrument Module

Perform OCB gridding for SuperMAG data.

Notes

SuperMAG data available at: http://supermag.jhuapl.edu/

ocbpy.instruments.supermag.load_supermag_ascii_data(filename)[source]

Load a SuperMAG ASCII data file.

Parameters:filename (str) – SuperMAG ASCI data file name
Returns:out – The dict keys are specified by the header data line, the data for each key are stored in the numpy array
Return type:dict
ocbpy.instruments.supermag.supermag2ascii_ocb(smagfile, outfile, hemisphere=0, ocb=None, ocbfile='default', instrument='', max_sdiff=600, min_merit=None, max_merit=None, scale_func=<function normal_curl_evar>, **kwargs)[source]

Covert and scales the SuperMAG data into OCB coordinates.

Parameters:
  • smagfile (str) – File containing the required SuperMAG file sorted by time
  • outfile (str) – Filename for the output data
  • hemisphere (int) – Hemisphere to process (can only do one at a time). 1=Northern, -1=Southern, 0=Determine from data (default=0)
  • ocb (ocbpy.OCBoundary, ocbpy.DualBoundary, or NoneType) – OCBoundary or DualBoundary object with data loaded already. If None, looks to ocbfile and creates an OCBoundary object. (default=None)
  • ocbfile (str) – File containing the required OC Boundary data sorted by time, or ‘default’ to load default file for time and hemisphere. Only used if no OCBoundary object is supplied (default=’default’)
  • instrument (str) – Instrument providing the OCBoundaries. Requires ‘image’ or ‘ampere’ if a file is provided. If using filename=’default’, also accepts ‘amp’, ‘si12’, ‘si13’, ‘wic’, and ‘’. (default=’’)
  • max_sdiff (int) – Maximum seconds between OCB and data record in sec (default=60)
  • min_merit (float or NoneType) – Minimum value for the default figure of merit or None to not apply a custom minimum (default=None)
  • max_merit (float or NoneTye) – Maximum value for the default figure of merit or None to not apply a custom maximum (default=None)
  • kwargs (dict) – Dict with optional selection criteria. The key should correspond to a data attribute and the value must be a tuple with the first value specifying ‘max’, ‘min’, ‘maxeq’, ‘mineq’, or ‘equal’ and the second value specifying the value to use in the comparison.
  • min_sectors (int) – Minimum number of MLT sectors required for good OCB. Deprecated, will be removed in version 0.3.1+ (default=7).
  • rcent_dev (float) – Maximum number of degrees between the new centre and the AACGM pole. Deprecated, will be removed in version 0.3.1+ (default=8.0)
  • max_r (float) – Maximum radius for open-closed field line boundary in degrees/ Deprecated, will be removed in version 0.3.1+ (default=23.0)
  • min_r (float) – Minimum radius for open-closed field line boundary in degrees. Deprecated, will be removed in version 0.3.1+ (default=10.0)
  • scale_func (function or NoneType) – Scale the magnetic field observations unless None (default=ocbpy.ocb_scale.normal_curl_evar)
Raises:

IOError – If unable to open the input or output file

Notes

May only process one hemisphere at a time.

See also

ocbpy.ocb_scale.normal_curl_evar()

SuperDARN Vorticity Instrument Module

Perform OCB gridding for SuperDARN vorticity data.

Notes

Specialised SuperDARN data product, available from: gchi@bas.ac.uk

ocbpy.instruments.vort.load_vorticity_ascii_data(vortfile, save_all=False)[source]

Load SuperDARN vorticity data files.

Parameters:
  • vortfile (str) – SuperDARN vorticity file in block format
  • save_all (bool) – Save all data from the file (True), or only data needed to calculate the OCB coordinates and normalised vorticity (False). (default=False)
Returns:

vdata – Dictionary of numpy arrays

Return type:

dict

ocbpy.instruments.vort.vort2ascii_ocb(vortfile, outfile, hemisphere=0, ocb=None, ocbfile='default', instrument='', max_sdiff=600, save_all=False, min_merit=None, max_merit=None, scale_func=<function normal_curl_evar>, **kwargs)[source]

Covert the location of vorticity data from AACGM to OCB coordinates.

Parameters:
  • vortfile (str) – file containing the required vorticity file sorted by time
  • outfile (str) – filename for the output data
  • hemisphere (int) – Hemisphere to process (can only do one at a time). 1=Northern, -1=Southern, 0=Determine from data (default=0)
  • ocb (ocbpy.ocboundary.OCBoundary, ocbpy.DualBoundary, or NoneType) – OCBoundary or DualBoundary object with data loaded already. If None, looks to ocbfile and creates an OCBoundary object. (default=None)
  • ocbfile (str) – File containing the required OC boundary data sorted by time, or ‘default’ to load default file for time and hemisphere. Only used if no OCBoundary object is supplied (default=’default’)
  • instrument (str) – Instrument providing the OCBoundaries. Requires ‘image’ or ‘ampere’ if a file is provided. If using filename=’default’, also accepts ‘amp’, ‘si12’, ‘si13’, ‘wic’, and ‘’. (default=’’)
  • max_sdiff (int) – maximum seconds between OCB and data record in sec (default=600)
  • save_all (bool) – Save all data (True), or only that needed to calcuate OCB and vorticity (False). (default=False)
  • min_merit (float or NoneType) – Minimum value for the default figure of merit or None to not apply a custom minimum (default=None)
  • max_merit (float or NoneTye) – Maximum value for the default figure of merit or None to not apply a custom maximum (default=None)
  • kwargs (dict) – Dict with optional selection criteria. The key should correspond to a data attribute and the value must be a tuple with the first value specifying ‘max’, ‘min’, ‘maxeq’, ‘mineq’, or ‘equal’ and the second value specifying the value to use in the comparison.
  • min_sectors (int) – Minimum number of MLT sectors required for good OCB. Deprecated, will be removed in version 0.3.1+ (default=7)
  • rcent_dev (float) – Maximum number of degrees between the new centre and the AACGM pole. Deprecated, will be removed in version 0.3.1+ (default=8.0)
  • max_r (float) – Maximum radius for open-closed field line boundary in degrees. Deprecated, will be removed in version 0.3.1+ (default=23.0)
  • min_r (float) – Minimum radius for open-closed field line boundary in degrees. Deprecated, will be removed in version 0.3.1+ (default=10.0)
  • scale_func (function or NoneType) – Scaling function for Vorticity data or None to not scale (default=ocbpy.ocb_scale.normal_curl_evar)
Raises:
  • IOError – If unable to open input or output file
  • ValueError – If unable to retrieve all necessary data from the input file

Notes

Input header or col_names must include the names in the default string.

pysat Instrument Module

Perform OCB gridding for appropriate instrument data loaded in pysat.

Notes

pysat is available at: http://github.com/pysat/pysat or pypi

ocbpy.instruments.pysat_instruments.add_ocb_to_data(pysat_inst, mlat_name='', mlt_name='', evar_names=None, curl_evar_names=None, vector_names=None, hemisphere=0, ocb=None, ocbfile='default', instrument='', max_sdiff=60, min_merit=None, max_merit=None, **kwargs)[source]

Covert the location of pysat data into OCB, EAB, or Dual coordinates.

Parameters:
  • pysat_inst (pysat.Instrument) – pysat.Instrument class object containing magnetic coordinates
  • mlat_name (str) – Instrument data key or column for magnetic latitudes (default=’’)
  • mlt_name (str) – Instrument data key or column for magnetic local times (default=’’)
  • evar_names (list or NoneType) – List of Instrument data keys or columns pointing to measurements that are proportional to the electric field (E); e.g. ion drift (default=None)
  • curl_evar_names (list or NoneType) – List of Instrument data keys or columns pointing to measurements that are proportional to the curl of E; e.g. ion vorticity (default=None)
  • vector_names (dict or NoneType) – Dict of Instrument data keys or columns pointing to measurements that are vectors that are proportional to either E or the curl of E. The key should correspond to one of the values in the evar_names or curl_evar_names list. If this is not done, a scaling function must be provided. The value corresponding to each key must be a dict that indicates the names holding data needed to initialise the ocbpy.ocb_scaling.VectorData object (default=None)
  • hemisphere (int) – Hemisphere to process (can only do one at a time). 1=Northern, -1=Southern, 0=Determine from data (default=0)
  • ocb (ocbpy.OCBoundary, ocbpy.DualBoundary, or NoneType) – OCBoundary or DualBoundary object with data loaded already. If None, looks to ocbfile and creates an OCBoundary object. (default=None)
  • ocbfile (str) – file containing the required OC boundary data sorted by time, ignorned if OCBoundary object supplied (default=’default’)
  • instrument (str) – Instrument providing the OCBoundaries. Requires ‘image’ or ‘ampere’ if a file is provided. If using filename=’default’, also accepts ‘amp’, ‘si12’, ‘si13’, ‘wic’, and ‘’ (default=’’)
  • max_sdiff (int) – maximum seconds between OCB and data record in sec (default=60)
  • min_merit (float or NoneType) – Minimum value for the default figure of merit or None to not apply a custom minimum (default=None)
  • max_merit (float or NoneTye) – Maximum value for the default figure of merit or None to not apply a custom maximum (default=None)
  • kwargs (dict) – Dict with optional selection criteria. The key should correspond to a data attribute and the value must be a tuple with the first value specifying ‘max’, ‘min’, ‘maxeq’, ‘mineq’, or ‘equal’ and the second value specifying the value to use in the comparison.
  • min_sectors (int) – Minimum number of MLT sectors required for good OCB. Deprecated, will be removed in version 0.3.1+ (default=7)
  • rcent_dev (float) – Maximum number of degrees between the new centre and the AACGM pole. Deprecated, will be removed in version 0.3.1+ (default=8.0)
  • max_r (float) – Maximum radius for open-closed field line boundary in degrees. Deprecated, will be removed in version 0.3.1+ (default=23.0)
  • min_r (float) – Minimum radius for open-closed field line boundary in degrees. Deprecated, will be removed in version 0.3.1+ (default=10.0)
Raises:

ValueError – If the pysat Instrument doesn’t have the necessary data values or if the input provided is not a pysat Instrument.

Notes

This may be run on a pysat instrument or as a custom function when loading pysat data.

Examples

# Example vector name input looks like:
vector_names={'vel': {'aacgm_n': 'vel_n', 'aacgm_e': 'vel_e',
                      'dat_name': 'velocity', 'dat_units': 'm/s'},
               'dat': {'aacgm_n': 'dat_n', 'aacgm_e': 'dat_e',
                       'scale_func': local_scale_func}}
ocbpy.instruments.pysat_instruments.add_ocb_to_metadata(pysat_inst, ocb_name, pysat_name, overwrite=False, notes='', isvector=False)[source]

Update pysat metadata for OCB data.

Parameters:
  • pysat_inst (pysat.Instrument) – pysat.Instrument class object containing magnetic coordinates
  • ocb_name (str) – Data column name for OCB data
  • pysat_name (str) – Data column name for non-OCB version of this data
  • overwrite (bool) – Overwrite existing metadata, if present (default=False)
  • notes (str)) – Notes about this OCB data (default=’’)
  • isvector (bool) – Is this vector data or not (default=False)
Raises:

ValueError – If input pysat Instrument object is the wrong class

Time Handling Module

Routines to convert from different file timekeeping methods to datetime.

ocbpy.ocb_time.convert_time(year=None, soy=None, yyddd=None, sod=None, date=None, tod=None, datetime_fmt='%Y-%m-%d %H:%M:%S')[source]

Convert to datetime from multiple time formats.

Parameters:
  • year (int or NoneType) – Year or None if not in year-soy format (default=None)
  • soy (int or NoneType) – Seconds of year or None if not in year-soy format (default=None)
  • yyddd (str or NoneType) – String containing years since 1900 and 3-digit day of year (default=None)
  • sod (int, float, or NoneType) – Seconds of day or None if the time of day is not in this format (default=None)
  • date (str or NoneType) – String containing date information or None if not in date-time format (default=None)
  • tod (str or NoneType) – String containing time of day information or None if not in date-time format (default=None)
  • datetime_fmt (str) – String with the date-time or date format (default=’%Y-%m-%d %H:%M:%S’)
Returns:

dtime – Datetime object

Return type:

dt.datetime

ocbpy.ocb_time.datetime2hr(dtime)[source]

Calculate hours of day from datetime.

Parameters:dtime (dt.datetime) – Universal time as a timestamp
Returns:uth – Hours of day, includes fractional hours
Return type:float
ocbpy.ocb_time.deg2hr(lon)[source]

Convert from degrees to hours.

Parameters:lon (float or array-like) – Longitude-like value in degrees
Returns:lt – Local time-like value in hours
Return type:float or array-like
ocbpy.ocb_time.fix_range(values, min_val, max_val, val_range=None)[source]

Ensure cyclic values lie below the maximum and at or above the mininum.

Parameters:
  • values (int, float, or array-like) – Values to adjust
  • min_val (int or float) – Maximum that values may not meet or exceed
  • max_val (int or float) – Minimum that values may not lie below
  • val_range (int, float, or NoneType) – Value range or None to calculate from min and max (default=None)
Returns:

fixed_vals – Values adjusted to lie min_val <= fixed_vals < max_val

Return type:

int, float, or array-like

ocbpy.ocb_time.get_datetime_fmt_len(datetime_fmt)[source]

Get the lenght of a string line needed for a specific datetime format.

Parameters:datetime_fmt (str) – Formatting string used to convert between datetime and string object
Returns:str_len – Minimum length of a string needed to hold the specified data
Return type:int

Notes

See the datetime documentation for meanings of the datetime directives

ocbpy.ocb_time.glon2slt(glon, dtime)[source]

Convert from geographic longitude to solar local time.

Parameters:
  • glon (float or array-like) – Geographic longitude in degrees
  • dtime (dt.datetime) – Universal time as a timestamp
Returns:

slt – Solar local time in hours

Return type:

float or array-like

ocbpy.ocb_time.hr2deg(lt)[source]

Convert from degrees to hours.

Parameters:lt (float or array-like) – Local time-like value in hours
Returns:lon – Longitude-like value in degrees
Return type:float or array-like
ocbpy.ocb_time.hr2rad(lt)[source]

Convert from hours to radians.

Parameters:lt (float or array-like) – Local time-like value in hours
Returns:lon – Longitude-like value in radians
Return type:float or array-like
ocbpy.ocb_time.rad2hr(lon)[source]

Convert from radians to hours.

Parameters:lon (float or array-like) – Longitude-like value in radians
Returns:lt – Local time-like value in hours
Return type:float or array-like
ocbpy.ocb_time.slt2glon(slt, dtime)[source]

Convert from solar local time to geographic longitude.

Parameters:
  • slt (float or array-like) – Solar local time in hours
  • dtime (dt.datetime) – Universal time as a timestamp
Returns:

glon – Geographic longitude in degrees

Return type:

float or array-like

ocbpy.ocb_time.year_soy_to_datetime(yyyy, soy)[source]

Converts year and soy to datetime.

Parameters:
  • yyyy (int) – 4 digit year
  • soy (float) – seconds of year
Returns:

dtime – datetime object

Return type:

dt.datetime

ocbpy.ocb_time.yyddd_to_date(yyddd)[source]

Convert from years since 1900 and day of year to datetime.

Parameters:yyddd (str) – String containing years since 1900 and day of year (e.g. 100126 = 2000-05-5).
Returns:dtime – Datetime object containing date information
Return type:dt.datetime