Trees       Indices       Help   
DebTorrent
Package DebTorrent :: Package BT1 :: Module StorageWrapper :: Class StorageWrapper
[hide private]
[frames] | no frames]

Class StorageWrapper

source code

Manage the storage of data at the piece level.

Instance Methods [hide private]
    Initialization
 
__init__(self, storage, request_size, hashes, piece_sizes, datalength, finished, failed, statusfunc=<function dummy_status at 0x2b55193a3b18>, flag=<DebTorrent.BT1.StorageWrapper.fakeflag instance at 0x2b55193a..., check_hashes=True, data_flunked=<function <lambda> at 0x2b55198ce230>, backfunc=None, config={}, unpauseflag=<DebTorrent.BT1.StorageWrapper.fakeflag instance at 0x2b55193a...)
Initialize the instance and begin the background initialization tasks.		 source code
 
_bgsync(self)
Periodically flush the files to disk.		 source code
boolean
old_style_init(self)
Initialize the storage in the foreground.		 source code
 
initialize(self, donefunc)
Schedule the background initialization of the storage.		 source code
 
_initialize(self)
Perform the background initialization of the storage.		 source code
boolean
init_hashcheck(self)
Initialize the hash check initialization task.		 source code
 
_markgot(self, piece, pos)
Mark a piece as complete during the initial hash check.		 source code
float
hashcheckfunc(self)
Hash check the current data as part of the initialization tasks.		 source code
boolean
init_movedata(self)
Initialize the move data initialization task (only for sparse allocation).		 source code
float
movedatafunc(self)
Move pieces around as part of the initialization tasks.		 source code
boolean
init_alloc(self)
Initialize the allocation initialization task.		 source code
int
_allocfunc(self)
Find the next location that needs allocating		 source code
float
allocfunc(self)
Allocate space for pieces.		 source code
boolean
bgalloc(self)
Enable the backround allocation of all pieces.		 source code
 
_bgalloc(self)
Allocate the file storage in the background.		 source code
    Information
boolean
_waspre(self, piece)
Determine if the piece has been pre-allocated.		 source code
int
_piecelen(self, piece)
Get the size of a piece.		 source code
long
get_amount_left(self)
Determine the amount of the download still remaining.		 source code
boolean
do_I_have_anything(self)
Determine if anything has been downloaded.		 source code
boolean
am_I_complete(self)
Determine if the download is currently complete.		 source code
string
get_have_list(self)
Get this peer's bitfield as a string.		 source code
(string, list of int)
get_have_list_cloaked(self)
Get this peer's incomplete bitfield as a string, and a have list.		 source code
boolean
do_I_have(self, index)
Determine if the piece has been downloaded.		 source code
string
get_hash(self, index)
Get the piece's hash.		 source code
long, long
get_stats(self)
Get some statistics about the download.		 source code
    Requests
 
_make_inactive(self, index)
Create the list of inactive requests for a piece.		 source code
boolean
is_endgame(self)
Determine if the download is in end-game mode.		 source code
 
reset_endgame(self, requestlist)
Reset all lost requests to inactive.		 source code
boolean
do_I_have_requests(self, index)
Determine if the piece has requests that could be sent out.		 source code
boolean
is_unstarted(self, index)
Determine if the piece has been started downloading.		 source code
int, int
new_request(self, index)
Get a new request for the piece.		 source code
 
request_lost(self, index, begin, length)
Add a lost request back to the list of inactive requests.		 source code
    Read/Write
boolean
write_raw(self, index, begin, data)
Write some data for the piece to disk.		 source code
boolean
_write_to_buffer(self, piece, start, data)
Write some data to the piece buffer.		 source code
boolean
_flush_buffer(self, piece, popped=True)
Flush the piece in the buffer to disk.		 source code
 
sync(self)
Flush all the buffered pieces to disk in order by their current place in the download.		 source code
int
_move_piece(self, index, newpos)
Move a piece from it's current location to a new one.		 source code
boolean
_clear_space(self, index)
Clear a space for a new piece to be written to the storage.		 source code
boolean
piece_came_in(self, index, begin, piece, source=None)
Process some received data for a piece.		 source code
DebTorrent.piecebuffer.SingleBuffer or string
get_piece(self, index, begin, length)
Get a chunk of a piece.		 source code
DebTorrent.piecebuffer.SingleBuffer
read_raw(self, piece, begin, length, flush_first=True)
Read some piece data from the disk.		 source code
 
set_file_readonly(self, n)
Sync a file to disk and set it read only.		 source code
    Resuming
boolean
has_data(self, index)
Determine whether a piece location has data in it.		 source code
boolean
doublecheck_data(self, pieces_to_check)
Double check all the listed pieces hashes.		 source code
 
reblock(self, new_blocked)
Recreate the various blocked lists based on new blocked pieces.		 source code
dictionary
pickle(self)
Pickle the current state for later resuming.		 source code
list of int
unpickle(self, data, valid_places)
Unpickle the state for resuming.		 source code
Instance Variables [hide private]
string alloc_buf
dummy data to write to allocated but not downloaded piece locations
string alloc_type
the type of allocation to do, always 'pre-allocate'
long amount_desired
the amount of the total download that is desired
length amount_inactive
the amount of data that is still in inactive requests
long amount_left
the amount of the total download that is still left to download
long amount_obtained
the amount of the desired download that has been received
method backfunc
method to call to schedule future invocation of requester functions
boolean bgalloc_active
whether background allocation is in process
boolean bgalloc_enabled
whether the allocation of all pieces has been enabled
list of boolean blocked
whether the piece location is blocked
list of int blocked_holes
the holes that have been blocked
Olist blocked_movein
the piece locations that are currently blocking a piece from moving into its proper place
Olist blocked_moveout
the pieces that are currently blocked from moving into their proper place
boolean check_hashes
whether to re-check hashes of data already downloaded, during startup
list of int check_list
the pieces needing hash checking during initialization
float check_numchecked
not used
dictionary check_targets
keys are piece hashes, values are a list of potential pieces
int check_total
the number of pieces needing hash checking during initialization
dictionary config
the configuration parameters
method data_flunked
method to call when a hash check fails for a piece
long datalength
the total length of the download
dictionary dirty
keys are the pieces that have received partial downloads, values are a list of the offset within the piece and length of the partial downloads received
boolean double_check
whether to double-check data being written to the disk for errors
dictionary download_history
keys are downloaded pieces, values are dictionaries with keys of the request offset within the piece and values of the source of the download
method failed
method to call when a critical failure occurs
dictionary failed_pieces
keys are downloaded pieces, values are dictionaries with keys of the source of the download
method finished
method to call when the download is complete
threading.Event flag
the flag that indicates when the program is to be shutdown
list of string hashes
the hashes of the pieces
DebTorrent.bitfield.Bitfield have
the bitfield that this peer has
(string, list of int) have_cloaked_data
the cached incomplete bitfiled as a binary string, and the haves needed to complete it, or None if it has not yet been generated
list of int holes
the piece locations that are currently empty and unallocated
list of int or list of (int, int) inactive_requests
for each piece, the inactive requests remaining for the piece.
method initialize_done
method to call when the background initialization is complete
method initialize_next
current method to call in the initialization tasks
list of [string, int, method, method] initialize_tasks
the initialization tasks to perform, the status message, initial fraction done, method to call to initialize the task, and the task
list of int movelist
for sparse allocation, the pieces that need to be moved during initialization
list of int numactive
the number of active requests pending for a piece
float numchecked
the number of pieces that have been hash checked during initialization
int numholes
the number of holes that need to be filled by allocation during initialization
int out_of_place
for sparse allocation, the number of pieces that are out of place during initialization
list of long piece_begins
the beginning offset of the pieces in the total download
list of int piece_lengths
the lengths of the pieces
list of int piece_sizes
the lengths of the pieces
dictionary places
keys are the pieces, values are their current piece locations
int request_size
the number of bytes to query for per request
dictionary stat_active
keys are the pieces that currently have active requests
dictionary stat_new
keys are the pieces that have a newly sent request but have not downloaded anything
int stat_numdownloaded
total number of pieces that have successfully downloaded
int stat_numflunked
the number of downloaded pieces that have failed a hash check
int stat_numfound
number of pieces that were found from a previous download
method statusfunc
the method to use to diplay status updates
BT1.Storage.Storage storage
the file storage instance
float tomove
for sparse allocation, the numebr of pieces that still need to be moved during initialization
long total_length
the total length of the download
boolean triple_check
whether to thoroughly check data being written to the disk
threading.Event unpauseflag
the flag to unset to pause the download
list of boolean waschecked
whether the piece has been hash checked
dictionary write_buf
keys are pieces in the piece buffer, values are lists of the offset within the piece and the piece data: [(start, data), ...]
list of int write_buf_list
the pieces currently in the piece buffer, ordered so that the most recently received piece is at the end of the list
long write_buf_max
the maximum number of bytes to store in the piece buffer
long write_buf_size
the current number of bytes stored in the piece buffer
Method Details [hide private]

__init__(self, storage, request_size, hashes, piece_sizes, datalength, finished, failed, statusfunc=<function dummy_status at 0x2b55193a3b18>, flag=<DebTorrent.BT1.StorageWrapper.fakeflag instance at 0x2b55193a..., check_hashes=True, data_flunked=<function <lambda> at 0x2b55198ce230>, backfunc=None, config={}, unpauseflag=<DebTorrent.BT1.StorageWrapper.fakeflag instance at 0x2b55193a...)
(Constructor)

source code 
Initialize the instance and begin the background initialization tasks.
Parameters:
  • storage (BT1.Storage.Storage) - the file storage instance
  • request_size (int) - the number of bytes to query for per request
  • hashes (list of string) - the hashes of the pieces
  • piece_sizes (list of int) - the lengths of the pieces
  • datalength (long) - the total length of the download
  • finished (method) - method to call when the download is complete
  • failed (method) - method to call when a critical failure occurs
  • statusfunc (method) - the method to use to diplay status updates (optional, defaults to not displaying status updates)
  • flag (threading.Event) - the flag that indicates when the program is to be shutdown (optional, defaults to using a dummy flag)
  • check_hashes (boolean) - whether to re-check hashes of data already downloaded, during startup (optional, defaults to True)
  • data_flunked (method) - method to call when a hash check fails for a piece (optional, defaults to doing nothing)
  • backfunc (method) - method to call to schedule future invocation of requester functions
  • config (dictionary) - the configuration parameters
  • unpauseflag (threading.Event) - the flag to unset to pause the download (optional, defaults to using a dummy flag)

old_style_init(self)

source code 
Initialize the storage in the foreground.
Returns: boolean
whether the init completed

initialize(self, donefunc)

source code 
Schedule the background initialization of the storage.
Parameters:
  • donefunc (method) - method to call when the initialization is complete

init_hashcheck(self)

source code 
Initialize the hash check initialization task.
Returns: boolean
whether the task should proceed

_markgot(self, piece, pos)

source code 
Mark a piece as complete during the initial hash check.
Parameters:
  • piece (int) - the piece to mark as complete
  • pos (int) - the place the piece currently occupies

hashcheckfunc(self)

source code 
Hash check the current data as part of the initialization tasks.
Returns: float
the fraction of the task that is complete, or None if the task is complete

init_movedata(self)

source code 
Initialize the move data initialization task (only for sparse allocation).
Returns: boolean
whether the task should proceed

movedatafunc(self)

source code 
Move pieces around as part of the initialization tasks.
Returns: float
the fraction of the task that is complete, or None if the task is complete

init_alloc(self)

source code 

Initialize the allocation initialization task.

Only runs if the allocation type is pre-allocate, or background and there are pieces waiting to move into their proper place.
Returns: boolean
whether the task should proceed

_allocfunc(self)

source code 
Find the next location that needs allocating
Returns: int
the next piece hole to allocate, or None if there are none

allocfunc(self)

source code 

Allocate space for pieces.

Allocation is done both as part of the initial startup (see init_alloc), and also periodically (see _bgalloc).
Returns: float
the fraction of the task that is complete, or None if the task is complete

bgalloc(self)

source code 
Enable the backround allocation of all pieces.
Returns: boolean
False

_waspre(self, piece)

source code 
Determine if the piece has been pre-allocated.
Parameters:
  • piece (int) - the piece to check
Returns: boolean
whether a storage location for the piece has been allocated

_piecelen(self, piece)

source code 
Get the size of a piece.
Parameters:
  • piece (int) - the piece to get the size of
Returns: int
the size of the piece

get_amount_left(self)

source code 
Determine the amount of the download still remaining.
Returns: long
the amount still remaining to download

do_I_have_anything(self)

source code 
Determine if anything has been downloaded.
Returns: boolean
whether anything has been downloaded

_make_inactive(self, index)

source code 
Create the list of inactive requests for a piece.
Parameters:
  • index (int) - the piece to make the inactive list for

is_endgame(self)

source code 

Determine if the download is in end-game mode.

End-game mode occurs when there are pending requests for all the chunks of pieces in the download.
Returns: boolean
whether the download is in end-game mode

am_I_complete(self)

source code 

Determine if the download is currently complete.

Complete here means that all the pieces that are desired have been downloaded, not that all possible pieces ahve been downloaded.
Returns: boolean
whether the download is complete

reset_endgame(self, requestlist)

source code 

Reset all lost requests to inactive.

Only occurs as the result of a loss of all peers, or a pause during end-game mode.
Parameters:
  • requestlist (list of (int, int, int)) - the requests that were lost, the piece index, offset within the piece, and length of the request

get_have_list(self)

source code 
Get this peer's bitfield as a string.
Returns: string
the have list as a binary string

get_have_list_cloaked(self)

source code 

Get this peer's incomplete bitfield as a string, and a have list.

Random bits are removed from the bitfield and added to the list of haves.
Returns: (string, list of int)
the incomplete bitfield as a binary string, and the haves to complete it

do_I_have(self, index)

source code 
Determine if the piece has been downloaded.
Parameters:
  • index (int) - the piece to check
Returns: boolean
whether the piece has been downloaded

do_I_have_requests(self, index)

source code 
Determine if the piece has requests that could be sent out.
Parameters:
  • index (int) - the piece to check
Returns: boolean
whether the piece has unsent requests to send

is_unstarted(self, index)

source code 
Determine if the piece has been started downloading.
Parameters:
  • index (int) - the piece to check
Returns: boolean
whether the piece has been started

get_hash(self, index)

source code 
Get the piece's hash.
Parameters:
  • index (int) - the piece to get the hash of
Returns: string
the SHA1 hash of the piece

get_stats(self)

source code 
Get some statistics about the download.
Returns: long, long
the amount of the download that has been completed, and the amount that is still desired to be downloaded

new_request(self, index)

source code 
Get a new request for the piece.
Parameters:
  • index (int) - the piece to get a request for
Returns: int, int
the offset within the piece and length of the request

write_raw(self, index, begin, data)

source code 
Write some data for the piece to disk.
Parameters:
  • index (int) - the piece to write to
  • begin (int) - the offset within the piece to start writing at
  • data (string) - the data to write
Returns: boolean
whether the write was successful

_write_to_buffer(self, piece, start, data)

source code 

Write some data to the piece buffer.

If the piece buffer has exceeded the maximum, the oldest pieces in it are first flushed to disk to make space. If the maximum write buffer size is set to 0, the data is immediately written to disk.
Parameters:
  • piece (int) - the piece buffer to write to
  • start (int) - the offset within the piece to start writing at
  • data (string) - the data to write
Returns: boolean
whether the write was successful

_flush_buffer(self, piece, popped=True)

source code 
Flush the piece in the buffer to disk.
Parameters:
  • piece (int) - the piece buffer to write to disk
  • popped (boolean) - whether the piece was already removed from the write_buf_list (optional, defaults to False)
Returns: boolean
whether the flush was successful

_move_piece(self, index, newpos)

source code 

Move a piece from it's current location to a new one.

The piece hash may be double checked if this is it's final position, or triple checked regardless.
Parameters:
  • index (int) - the piece to move
  • newpos (int) - new position to move the piece to
Returns: int
the old position od the piece, or -1 if an error occurred

_clear_space(self, index)

source code 

Clear a space for a new piece to be written to the storage.

If the proper place is available it will use it, otherwise a new place is created. The new place is alos first checked to make sure a piece is not waiting to be moved in there.
Parameters:
  • index (int) - the piece that needs a space
Returns: boolean
whether the function needs to be called again to complete the operation

piece_came_in(self, index, begin, piece, source=None)

source code 

Process some received data for a piece.

If this is the first data for a piece, it will first clear a space for the piece. If the piece is complete then the data structures are cleaned up, and the data is flushed to disk and hash checked.
Parameters:
  • index (int) - the piece that data was received for
  • begin (int) - the offset within the piece that the data starts at
  • piece (string) - the data that was received
Returns: boolean
False if the piece is complete but failed the hash check, otherwise returns True

request_lost(self, index, begin, length)

source code 
Add a lost request back to the list of inactive requests.
Parameters:
  • index (int) - the piece requested
  • begin (int) - the offset within the piece of the request
  • length (int) - the length of the request

get_piece(self, index, begin, length)

source code 

Get a chunk of a piece.

If the piece has never been hash checked, the whole piece will be read and checked first. Otherwise only the requested data will be read.
Parameters:
  • index (int) - the piece requested
  • begin (int) - the offset within the piece of the request
  • length (int) - the length of the request
Returns: DebTorrent.piecebuffer.SingleBuffer or string
the requested data (in a piecebuffer if the request was for the entire piece), or None if there was a problem

read_raw(self, piece, begin, length, flush_first=True)

source code 
Read some piece data from the disk.
Parameters:
  • piece (int) - the piece requested
  • begin (int) - the offset within the piece of the request
  • length (int) - the length of the request
  • flush_first (boolean) - whether to flush the files before reading the data (optional, default is not to flush)
Returns: DebTorrent.piecebuffer.SingleBuffer
the requested data, or None if there was a problem

set_file_readonly(self, n)

source code 
Sync a file to disk and set it read only.
Parameters:
  • n (int) - the file index to set read only

has_data(self, index)

source code 
Determine whether a piece location has data in it.
Parameters:
  • index (int) - the piece location to check
Returns: boolean
whether the location has data in it

doublecheck_data(self, pieces_to_check)

source code 
Double check all the listed pieces hashes.
Parameters:
  • pieces_to_check (dictionary) - keys are the piece indexes to hash check
Returns: boolean
whether all the pieces passed the hash check

reblock(self, new_blocked)

source code 

Recreate the various blocked lists based on new blocked pieces.

Assume downloads have already been canceled and chunks made inactive.
Parameters:
  • new_blocked (list of boolean) - whether the pieces are now blocked

pickle(self)

source code 

Pickle the current state for later resuming.

Pickled data format: 
   d['pieces'] = a string containing a bitfield of complete pieces
   d['partials'] = [ piece, [ offset, length... ]... ]
               a list of partial data that had been previously
               downloaded, plus the given offsets.  Adjacent partials
               are merged so as to save space, and so that if the
               request size changes then new requests can be
               calculated more efficiently.
   d['places'] = [ piece, place, {,piece, place ...} ]
               the piece index, and the place it's stored.
               If d['pieces'] specifies a complete piece or d['partials']
               specifies a set of partials for a piece which has no
               entry in d['places'], it can be assumed that
               place[index] = index.  A place specified with no
               corresponding data in d['pieces'] or d['partials']
               indicates allocated space with no valid data, and is
               reserved so it doesn't need to be hash-checked.
Returns: dictionary
the pickled state

unpickle(self, data, valid_places)

source code 
Unpickle the state for resuming.
Parameters:
  • data (dictionary) - the pickled state (see pickle)
  • valid_places (list of int) - the pieces that are still considered valid based on their modification time and size
Returns: list of int
the pieces that are partially downloaded

Instance Variable Details [hide private]

inactive_requests

for each piece, the inactive requests remaining for the piece. Will be 1 if the requests have not yet been generated, None if thee piece has already been downloaded, or a list of offsets within the piece and request lengths.
Type:
list of int or list of (int, int)

   Trees       Indices       Help   
DebTorrent
Generated by Epydoc 3.0beta1 on Sun Aug 19 16:26:54 2007 http://epydoc.sourceforge.net