The LoRS API for Sets

Function Documentation

int lorsSetEnum (  LorsSet *    set, LorsEnum *    list )

Form an enumeration of all mappings within the Set. In order to enumerate through all of the mapping within a Set, you may use lorsSetEnum(). It returns a LorsEnum which can then be passed to lorsEnumNext() to get each successive mapping. Parameters: set  A LorsSet list  A pointer to a LorsEnum

int lorsEnumNext (  LorsEnum    list, LorsEnum *    iterator, LorsMapping **    ret )

Iterate through a LorsEnum list of mappings. Iterate through a list of LorsMappings. Parameters: list  A LorsEnum list returned from lorsSetEnum(). iterator  A pointer to an iterator. Must be initially NULL. ret  A pointer into which a LorsMapping pointer is returned. Returns: On list exhaustion, LORS_END is returned.

int lorsEnumFree (  LorsEnum    list )

Free a LorsEnum. Frees resources associated with a LorsEnum. Parameters: list  Returns: There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetAddMapping (  LorsSet *    set, LorsMapping *    map )

Append a mapping to a given set. If the mapping is already present, and error will be returned. A mapping should exist only once per set. Parameters: set  map  Returns: There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetRemoveMapping (  LorsSet *    set, LorsMapping *    map )

Remove a mapping from a given set. Parameters: set  map  Returns: There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetInit (  LorsSet **    set, ulong_t    data_blocksize, int    copies, int    opts )

Initialize LorsSet internal data structures. lorsSetInit() initializes a LorsSet structure with the specified data_blocksize and number of copies. Before calling either lorsSetStore() or lorsSetCopy() you should call lorsSetInit() to create a target LorsSet into which new mappings will be added. Parameters: set  A pointer to a pointer of a LorsSet into which the new structure will be returned. data_blocksize  The blocksize of allocated fragments. copies  The number of replicas to create. opts  This is always zero for now. In the future there may be more options. Returns: There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetStore (  LorsSet *    set, LorsDepotPool *    dp, char *    buffer, longlong    offset, longlong    length, LorsConditionStruct *    lc, int    nthreads, int    timeout, int    opts )

Allocate storage and Send data to the IBP allocations. lorsSetStore() sends data to IBP allocations. The number of threads used to execute the operation will depend heavily on the initial blocksize given to lorsSetCreate() and the size of buffer. Only one thread can be active per Mapping (i.e. per replica). In the event of depot failure during Store or Allocate, a new buffer will be allocated from the LorsDepotPool. Parameters: set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation. dp  A LorsDepotPool from which new allocations will be drawn. buffer  The data a user wishes to upload. offset  The absolute offset relative to the exnode to store the buffer data. This allows multiple calls to SetStore simultaneously, if necessary. length  The length of buffer. lc  An array of LorsConditionStructs which specify the order to apply end2end operations. This may be NULL if no conditioning is desired. nthreads  The maximum number of threads to use. By default, only (length/blocksize + 1 * copies) can be used, timeout  Return after timeout seconds, even if operation is incomplete. opts  LORS_RETURN_ON_ANY_ERROR LORS_RETRY_UNTIL_TIMEOUT Returns: There are various error conditions. See list of all possible errors The LoRS Error List.

long lorsSetLoad (  LorsSet *    set, char *    buffer, longlong    offset, long    length, ulong_t    block_size, LorsConditionStruct *    lc, int    nthreads, int    timeout, int    opts )

Retrieve data from IBP allocations. lorsSetLoad() retrieves data from IBP and stores it into buffer. buffer must contain at least length bytes, or risk overwriting memory. The number of threads used depends strictly on the amount passed to nthreads and the opts Policy. lorsSetLoad() operates on 'length/nthreads' Blocksizes. However, in the case where 'Blocksize' is larger than the length to next boundary point (i.e. when mappings size are not an even multiple of 'Blocksize') the less of the two will be chosen to assign the next thread. The return value is the number of bytes read. Parameters: set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation. buffer  A pointer to data area into which lorsSetLoad() stores. offset  The offset relative to the Set from which this operation is to begin. length  The desired size of set to retrieve, and length of buffer block_size  The maximum datasize for one IBP_load call. lc  An array of all possible LorsConditionStructs used to decode any conditioning applied to the mappings in set. nthreads  The maximum number of threads to use. Because IBP allocations are memory mapped, any number of simultaneous threads may be used. timeout  Return after timeout seconds, even if operation is incomplete. opts  LORS_RETURN_ON_ANY_ERROR LORS_RETRY_UNTIL_TIMEOUT Returns: On error, a negative number is returned. Otherwise the number of bytes successfully written to the buffer is returned.

longlong lorsSetRealTimeLoad (  LorsSet *    set, char *    buffer, int    fd, longlong    offset, longlong    length, ulong_t    block_size, int    pre_buf, int    cache, LorsConditionStruct *    lc, int    nthreads, int    timeout, int    max_thds_per_depot, int    thds_per_job, int    progress_n, int    opts )

Provide real-time streaming of data over a file descriptor. To satisfy the requirements of real time streaming applications, lorsSetRealTimeLoad() writes to a file descriptor rather than a buffer. Most of the semantics remain the same as for lorsSetLoad(). Parameters: set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation. buffer  A pointer to data area into which lorsSetLoad() stores. fd  A valid file descriptor to which lorsSetRealTimeLoad() will write in order the successfully downloaded length requested. offset  The offset relative to the Set from which this operation is to begin. length  The desired size of set to retrieve, and length of buffer block_size  The maximum datasize for one IBP_load call. pre_buf  This is an integer count of the number of block_size sections to fully download before writing to the file descriptor. cache  The maximum amount of buffering the call will provide in addition to the memory allocated per-thread. lc  An array of all possible LorsConditionStructs used to decode any conditioning applied to the mappings in set. nthreads  The maximum number of threads to use. Because IBP allocations are memory mapped, any number of simultaneous threads may be used. timeout  Return after timeout seconds, even if operation is incomplete. max_thds_per_depot  In order to minimize the impact of our algorithm on depots, this parameter will provide a cap for individual depot connections. thds_per_job  This is a measure of redundancy. progress_n  When progress_n is greater than 0, and thds_per_job is greater than 1 opts  LORS_RETURN_ON_ANY_ERROR LORS_RETRY_UNTIL_TIMEOUT Returns: On error, a negative number is returned. Otherwise the number of bytes successfully written to the buffer is returned.

int lorsSetCopy (  LorsSet *    srcSet, LorsSet *    dstSet, LorsDepotPool *    dp, LboneResolution *    lr, longlong    offset, longlong    length, int    nthreads, int    timeout, int    opts )

Create replicas of data in srcSet allocation to another. lorsSetCopy() copies data from the @offset and length of srcSet to the seek_offset of dstSet. If the length of of srcSet is too large to fit within dstSet, it may or may not generate an error condition depending on the options passed in opts. Parameters: srcSet  A set from which a copy begins. dstSet  A pointer to a set to which the copy is placed. dp  A LorsDepotPool from which new allocations will be drawn. lr  An LboneResolution matrix as returned by lorsCreateResolutionFromFile() or others. From this information, decision regarding routing are made. offset  The exnode offset of the mappings in srcSet length  Length relative to srcSet nthreads  By default, this will depend on the number of mappings in set , i.e. one thread per fragment. timeout  Return after timeout seconds, even if operation is incomplete. opts  LORS_RETURN_ON_ANY_ERROR LORS_RETRY_UNTIL_TIMEOUT LORS_MCOPY LORS_COPY Returns: There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetMerge (  LorsSet *    src, LorsSet *    dest )

Combine the mappings from two Sets into one Set. lorsSetMerge() combines two Sets into a single dest, It does not perform a deep copy on internal datastructures, and mappings. The two are not independent after lorsSetMerge(). Parameters: src  The pointer to the first and source Set. dest  The pointer to second and destination Set. Returns: There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetTrim (  LorsSet *    set, longlong    offset, longlong    length, int    nthreads, int    timeout, int    opts )

Remove particular extents or Mappings from Set. After an allocation has expired it may be of no use to leave it in the Set. lorsSetTrim() provides the mechanism by which expired or unneeded mappings may be removed. Alternately, the operation is more subtle, allowing for readdressing of existing mappings, such that, pieces of a mapping may be ignored, while still present, in IBP. The number of threads needed to perform this operation are determined automatically. Parameters: set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation. offset  The offset relative to the Set. length  The desired size of set to trim. nthreads  By default, this will depend on the number of mappings in set , i.e. one thread per fragment. timeout  Return after timeout seconds, even if operation is incomplete. opts  LORS_TRIM_ALL - LORS_TRIM_DEAD - LORS_TRIM_KILL - LORS_TRIM_NOKILL - LORS_RETURN_ON_ANY_ERROR LORS_RETRY_UNTIL_TIMEOUT Returns: There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetRefresh (  LorsSet *    set, longlong    offset, longlong    length, time_t    duration, int    nthreads, int    timeout, int    opts )

Refresh the expiration time on allocated storage. Because the underlying storage of the LoRS tools is IBP, the allocations will eventually expire. However, before they expire, it may be possible to renew the lease on these allocations. The number of threads are determined automatically if 'nthreads' is -1. Parameters: set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation. offset  The offset relative to the Set. length  The desired size of set to refresh. duration  The new time in seconds. nthreads  By default, this will depend on the number of mappings in set , i.e. one thread per fragment. timeout  Return after timeout seconds, even if operation is incomplete. opts  LORS_REFRESH_MAX - Attempt to refresh to the largest allowable expiration time. LORS_REFRESH_EXTEND_BY - Attempt to add duration seconds to the time out. LORS_REFRESH_EXTEND_TO - Attempt to normalize all duration seconds from now. LORS_REFRESH_ABSOLUTE - Attempt to set expiration to duration seconds. LORS_RETURN_ON_ANY_ERROR LORS_RETRY_UNTIL_TIMEOUT Returns: There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetStat (  LorsSet *    set, int    nthreads, int    timeout, int    opts )

Perform a status operation on the Mappings of a Set. In order to have some view of the Mapping level, lorsSetStat() is provided to probe the data of individaul allocations. This allows for more complex operations such as 'Balance' which could guarantee a minimum number of replicas per-offset, or for determining other reliability statistics. Parameters: set  A LorsSet from a previous call to lorsSetCreate() or a deserialize operation. nthreads  The maximum number of threads to use for this operation. a value of -1 will use 1 thread per mapping. timeout  The maximum amount of time to allow for the entire operation, regardless of success or failure. opts  LORS_STAT_LIVE LORS_STAT_CACHE LORS_RETURN_ON_ANY_ERROR LORS_RETRY_UNTIL_TIMEOUT Returns: There are various error conditions. See list of all possible errors The LoRS Error List.

int lorsSetFree (  LorsSet *    set, int    opts )

Perform 'free' on internal data structures. Release memory back to local system. Parameters: set  A LorsSet opts  LORS_FREE_MAPPINGS Returns: 0

int lorsSetTrimCaps (  LorsSet *    set, int    opts )

Trim IBP caps from a set of mappings. Trim IBP caps from a set of mappings. Parameters: set  A LorsSet opts  LORS_READ_CAP LORS_WRITE_CAP LORS_MANAGE_CAP Returns: There are various error conditions. See list of all possible errors The LoRS Error List.