Supported Data Sets

Currently, support is included for files from the following sources:

These routines may be used as a guide to write routines for other data sets. A general data loading routine is also provided for ASCII files. These routines are part of the ocbpy.instruments module. Supporting time-handling routines that may be useful for specific data sets are provided in ocbpy.ocb_time.

General

General loading routines for data files

Functions

test_file(filename)
Test to see whether file exists and is small enough to load
load_ascii_data(filename, hlines, kwargs)
Load time-sorted ascii data file
ocbpy.instruments.general.load_ascii_data(filename, hlines, gft_kwargs={}, hsplit=None, datetime_cols=[], datetime_fmt=None, int_cols=[], str_cols=[], max_str_length=50, header=[])[source]

Load an ascii data file into a dict of numpy array

Parameters:
filename : (str)

data file name

hlines : (int)

number of lines in header. If zero, must include header.

gft_kwargs : (dict)

Dictionary holding optional keyword arguments for the numpy genfromtxt routine (default=dict())

hsplit : (str, NoneType)

character seperating data labels in header. None splits on all whitespace characters. (default=None)

datetime_cols : (list of ints)

If there are date strings or values that should be converted to a datetime object, list them in order here. Not processed as floats. (default=[])

datetime_fmt : (str or NoneType)

Format needed to convert the datetime_cols entries into a datetime object. Special formats permitted are: ‘YEAR SOY’, ‘SOD’. ‘YEAR SOY’ must be used together; ‘SOD’ indicates seconds of day, and may be used with any date format (default=None)

int_cols : (list of ints)

Data that should be processed as integers, not floats. (default=[])

str_cols : (list of ints)

Data that should be processed as strings, not floats. (default=[])

max_str_length : (int)

Maximum allowed string length. (default=50)

header : (list of str)

Header string(s) where the last line contains whitespace separated data names (default=list())

Returns:
header : (list of strings)

Contains all specified header lines

out : (dict of numpy.arrays)

The dict keys are specified by the header data line, the data for each key are stored in the numpy array

Notes

Data is assumed to be float unless otherwise stated.

ocbpy.instruments.general.test_file(filename)[source]

Test to ensure the file is small enough to read in. Python can only allocate 2GB of data without crashing

Parameters:
filename : (str)

Filename to test

Returns:
good_flag : (bool)

True if good, bad if false

SuperMAG

Perform OCB gridding for SuperMAG data

Functions

supermag2ascii_ocb(smagfile, outfile, kwargs)
Write and ASCII file with SuperMAG data and the OCB coordinates for each data point
load_supermag_ascii_data(filename)
Load SuperMAG ASCII data files

Data

SuperMAG data available at: http://supermag.jhuapl.edu/

ocbpy.instruments.supermag.load_supermag_ascii_data(filename)[source]

Load a SuperMAG ASCII data file

Parameters:
filename : (str)

SuperMAG ASCI data file name

Returns:
out : (dict of numpy.arrays)

The dict keys are specified by the header data line, the data for each key are stored in the numpy array

ocbpy.instruments.supermag.supermag2ascii_ocb(smagfile, outfile, hemisphere=0, ocb=None, ocbfile='default', instrument='', max_sdiff=600, min_sectors=7, rcent_dev=8.0, max_r=23.0, min_r=10.0)[source]

Coverts and scales the SuperMAG data into OCB coordinates

Parameters:
smagfile : (str)

file containing the required SuperMAG file sorted by time

outfile : (str)

filename for the output data

hemisphere : (int)

Hemisphere to process (can only do one at a time). 1=Northern, -1=Southern, 0=Determine from data (default=0)

ocb : (OCBoundary or NoneType)

OCBoundary object with data loaded from an OC boundary data file. If None, looks to ocbfile

ocbfile : (str)

file containing the required OC boundary data sorted by time, or ‘default’ to load default file for time and hemisphere. Only used if no OCBoundary object is supplied (default=’default’)

instrument : (str)

Instrument providing the OCBoundaries. Requires ‘image’ or ‘ampere’ if a file is provided. If using filename=’default’, also accepts ‘amp’, ‘si12’, ‘si13’, ‘wic’, and ‘’. (default=’’)

max_sdiff : (int)

maximum seconds between OCB and data record in sec (default=600)

min_sectors : (int)

Minimum number of MLT sectors required for good OCB (default=7).

rcent_dev : (float)

Maximum number of degrees between the new centre and the AACGM pole (default=8.0)

max_r : (float)

Maximum radius for open-closed field line boundary in degrees default=23.0)

min_r : (float)

Minimum radius for open-closed field line boundary in degrees (default=10.0)

Notes

May only process one hemisphere at a time. Scales the magnetic field observations using ocbpy.ocb_scale.normal_curl_evar.

SuperDARN Vorticity

Perform OCB gridding for SuperDARN vorticity data

Functions

vort2ascii_ocb(vortfile, outfile, kwargs)
Write and ASCII file with SuperDARN data and the OCB coordinates for each data point
load_vorticity_ascii_data(filename, save_all=False)
Load vorticity block ASCII data files

Data

Specialised SuperDARN data product, available from: gchi@bas.ac.uk

ocbpy.instruments.vort.load_vorticity_ascii_data(vortfile, save_all=False)[source]

Load SuperDARN vorticity data files.

Parameters:
vortfile : (str)

SuperDARN vorticity file in block format

save_all : (bool)

Save all data from the file (True), or only data needed to calculate the OCB coordinates and normalised vorticity (False). (default=False)

Returns:
vdata : (dict)

Dictionary of numpy arrays

ocbpy.instruments.vort.vort2ascii_ocb(vortfile, outfile, hemisphere=0, ocb=None, ocbfile='default', instrument='', max_sdiff=600, save_all=False, min_sectors=7, rcent_dev=8.0, max_r=23.0, min_r=10.0)[source]

Coverts the location of vorticity data from AACGM to OCB coordinates

Parameters:
vortfile : (str)

file containing the required vorticity file sorted by time

outfile : (str)

filename for the output data

hemisphere : (int)

Hemisphere to process (can only do one at a time). 1=Northern, -1=Southern, 0=Determine from data (default=0)

ocb : (ocbpy.ocboundary.OCBoundary or NoneType)

Object containing open closed boundary data or None to load from file

ocbfile : (str)

file containing the required OC boundary data sorted by time, ignorned if OCBoundary object supplied. (default=’default’)

instrument : (str)

Instrument providing the OCBoundaries. Requires ‘image’ or ‘ampere’ if a file is provided. If using filename=’default’, also accepts ‘amp’, ‘si12’, ‘si13’, ‘wic’, and ‘’. (default=’’)

max_sdiff : (int)

maximum seconds between OCB and data record in sec (default=600)

save_all : (bool)

Save all data (True), or only that needed to calcuate OCB and vorticity (False). (default=False)

min_sectors : (int)

Minimum number of MLT sectors required for good OCB. (default=7)

rcent_dev : (float)

Maximum number of degrees between the new centre and the AACGM pole (default=8.0).

max_r : (float)

Maximum radius for open-closed field line boundary in degrees. (default=23.0)

min_r : (float)

Minimum radius for open-closed field line boundary in degrees (default=10.0)

Notes

Input header or col_names must include the names in the default string.

pysat

Time Handling

Routines to convert from different file timekeeping methods to datetime

Functions

get_datetime_fmt_len(datetime_fmt)
Gets the length of a string line needed to hold a specified datetime format
year_soy_to_datetime(yyyy, soy)
Converts from seconds of year to datetime
yyddd_to_date(yyddd)
Converts from years since 1900 and day of year to datetime
convert_time(kwargs)
Convert to datetime from multiple time formats
deg2hr(lon)
Convert from degrees to hours
hr2deg(lt)
Convert from hours to degrees
rad2hr(lon)
Convert from radians to hours
hr2rad(lt)
Convert from hours to radians
datetime2hr(dtime)
Calculate fractional hours of day from timestamp
slt2glon(slt, dtime)
Convert from solar local time to geographic longitude
glon2slt(glon, dtime)
Convert from geographic longitude to solar local time
fix_range(values, max_val, min_val)
Ensure cyclic values lie within a specified range

Moduleauthor

Angeline G. Burrell (AGB), 15 April 2017, University of Texas, Dallas

ocbpy.ocb_time.convert_time(year=None, soy=None, yyddd=None, sod=None, date=None, tod=None, datetime_fmt='%Y-%m-%d %H:%M:%S')[source]

Convert to datetime from multiple time formats

Parameters:
year : (int or NoneType)

Year or None if not in year-soy format (default=None)

soy : (int or NoneType)

Seconds of year or None if not in year-soy format (default=None)

yyddd : (str or NoneType)

String containing years since 1900 and 3-digit day of year (default=None)

sod : (int,float or NoneType)

Seconds of day or None if the time of day is not in this format (default=None)

date : (str or NoneType)

String containing date information or None if not in date-time format (default=None)

tod : (str or NoneType)

String containing time of day information or None if not in date-time format (default=None)

datetime_fmt : (str)

String with the date-time or date format (default=’%Y-%m-%d %H:%M:%S’)

Returns:
dtime : (datetime)

Datetime object

ocbpy.ocb_time.datetime2hr(dtime)[source]

Calculate hours of day from datetime

Parameters:
dtime : (dt.datetime)

Universal time as a timestamp

Returns:
uth : (float)

Hours of day, includes fractional hours

ocbpy.ocb_time.deg2hr(lon)[source]

Convert from degrees to hours

Parameters:
lon : (float or array-like)

Longitude-like value in degrees

Returns:
lt : (float or array-like)

Local time-like value in hours

ocbpy.ocb_time.fix_range(values, min_val, max_val, val_range=None)[source]

Ensure cyclic values lie below the maximum and at or above the mininum

Parameters:
values : (int, float, or array-like)

Values to adjust

min_val : (int or float)

Maximum that values may not meet or exceed

max_val : (int or float)

Minimum that values may not lie below

val_range : (int, float, or NoneType)

Value range or None to calculate from min and max (default=None)

Returns:
fixed_vals : (int, float, or array-like)

Values adjusted to lie min_val <= fixed_vals < max_val

ocbpy.ocb_time.get_datetime_fmt_len(datetime_fmt)[source]

Get the lenght of a string line needed for a specific datetime format

Parameters:
datetime_fmt : (str)

Formatting string used to convert between datetime and string object

Returns:
str_len : (int)

Minimum length of a string needed to hold the specified data

Notes

See the datetime documentation for meanings of the datetime directives

ocbpy.ocb_time.glon2slt(glon, dtime)[source]

Convert from geographic longitude to solar local time

Parameters:
glon : (float or array-like)

Geographic longitude in degrees

dtime : (dt.datetime)

Universal time as a timestamp

Returns:
slt : (float or array-like)

Solar local time in hours

ocbpy.ocb_time.hr2deg(lt)[source]

Convert from degrees to hours

Parameters:
lt : (float or array-like)

Local time-like value in hours

Returns:
lon : (float or array-like)

Longitude-like value in degrees

ocbpy.ocb_time.hr2rad(lt)[source]

Convert from hours to radians

Parameters:
lt : (float or array-like)

Local time-like value in hours

Returns:
lon : (float or array-like)

Longitude-like value in radians

ocbpy.ocb_time.rad2hr(lon)[source]

Convert from radians to hours

Parameters:
lon : (float or array-like)

Longitude-like value in radians

Returns:
lt : (float or array-like)

Local time-like value in hours

ocbpy.ocb_time.slt2glon(slt, dtime)[source]

Convert from solar local time to geographic longitude

Parameters:
slt : (float or array-like)

Solar local time in hours

dtime : (dt.datetime)

Universal time as a timestamp

Returns:
glon : (float or array-like)

Geographic longitude in degrees

ocbpy.ocb_time.year_soy_to_datetime(yyyy, soy)[source]

Converts year and soy to datetime

Parameters:
yyyy : (int)

4 digit year

soy : (float)

seconds of year

Returns:
dtime : (dt.datetime)

datetime object

ocbpy.ocb_time.yyddd_to_date(yyddd)[source]

Convert from years since 1900 and day of year to datetime

Parameters:
yyddd : (str)

String containing years since 1900 and day of year (e.g. 100126 = 2000-05-5).

Returns:
dtime : (dt.datetime)

Datetime object containing date information