Package org.apache.storm.hdfs.blobstore
Class HdfsBlobStoreImpl
- java.lang.Object
-
- org.apache.storm.hdfs.blobstore.HdfsBlobStoreImpl
-
public class HdfsBlobStoreImpl extends Object
HDFS blob store impl.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description classHdfsBlobStoreImpl.KeyInHashDirIterator
-
Field Summary
Fields Modifier and Type Field Description static org.apache.hadoop.fs.permission.FsPermissionBLOBSTORE_DIR_PERMISSION
-
Constructor Summary
Constructors Constructor Description HdfsBlobStoreImpl(org.apache.hadoop.fs.Path path, Map<String,Object> conf)HdfsBlobStoreImpl(org.apache.hadoop.fs.Path path, Map<String,Object> conf, org.apache.hadoop.conf.Configuration hconf)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected voiddelete(org.apache.hadoop.fs.Path path)voiddeleteKey(String key)Delete a key from the blob store.booleanexists(String key)Check if the key exists in the blob store.voidfullCleanup(long age)protected intgetBlobReplication(String key)protected org.apache.hadoop.fs.PathgetKeyDir(String key)longgetLastBlobUpdateTime()Get the last update time of any blob.protected Iterator<BlobStoreFile>listBlobStoreFiles(org.apache.hadoop.fs.Path path)Iterator<String>listKeys()List relevant keys.protected Iterator<String>listKeys(org.apache.hadoop.fs.Path path)BlobStoreFileread(String key)Get an input stream for reading a part.voidshutdown()protected intupdateBlobReplication(String key, int replication)voidupdateLastBlobUpdateTime()Updates the last updated time of existing blobstores to the current time.voidvalidateBlobUpdateTime()Validates that the last updated blob time of the blobstore is up to date with the current existing blobs.BlobStoreFilewrite(String key, boolean create)Get an object tied to writing the data.
-
-
-
Constructor Detail
-
HdfsBlobStoreImpl
public HdfsBlobStoreImpl(org.apache.hadoop.fs.Path path, Map<String,Object> conf) throws IOException- Throws:
IOException
-
HdfsBlobStoreImpl
public HdfsBlobStoreImpl(org.apache.hadoop.fs.Path path, Map<String,Object> conf, org.apache.hadoop.conf.Configuration hconf) throws IOException- Throws:
IOException
-
-
Method Detail
-
listKeys
public Iterator<String> listKeys() throws IOException
List relevant keys.- Returns:
- all keys that are available for reading
- Throws:
IOException- on any error
-
listKeys
protected Iterator<String> listKeys(org.apache.hadoop.fs.Path path) throws IOException
- Throws:
IOException
-
read
public BlobStoreFile read(String key) throws IOException
Get an input stream for reading a part.- Parameters:
key- the key of the part to read- Returns:
- the where to read the data from
- Throws:
IOException- on any error
-
write
public BlobStoreFile write(String key, boolean create) throws IOException
Get an object tied to writing the data.- Parameters:
key- the key of the part to write to.create- whether the file needs to be new or not.- Returns:
- an object that can be used to both write to, but also commit/cancel the operation.
- Throws:
IOException- on any error
-
exists
public boolean exists(String key)
Check if the key exists in the blob store.- Parameters:
key- the key to check for- Returns:
- true if it exists else false.
-
deleteKey
public void deleteKey(String key) throws IOException
Delete a key from the blob store.- Parameters:
key- the key to delete- Throws:
IOException- on any error
-
getKeyDir
protected org.apache.hadoop.fs.Path getKeyDir(String key)
-
fullCleanup
public void fullCleanup(long age) throws IOException- Throws:
IOException
-
listBlobStoreFiles
protected Iterator<BlobStoreFile> listBlobStoreFiles(org.apache.hadoop.fs.Path path) throws IOException
- Throws:
IOException
-
getBlobReplication
protected int getBlobReplication(String key) throws IOException
- Throws:
IOException
-
updateBlobReplication
protected int updateBlobReplication(String key, int replication) throws IOException
- Throws:
IOException
-
delete
protected void delete(org.apache.hadoop.fs.Path path) throws IOException- Throws:
IOException
-
shutdown
public void shutdown()
-
getLastBlobUpdateTime
public long getLastBlobUpdateTime() throws IOExceptionGet the last update time of any blob.- Returns:
- the last updated time of blobs within the blobstore.
- Throws:
IOException- on any error
-
updateLastBlobUpdateTime
public void updateLastBlobUpdateTime() throws IOExceptionUpdates the last updated time of existing blobstores to the current time.- Throws:
IOException- on any error
-
validateBlobUpdateTime
public void validateBlobUpdateTime() throws IOExceptionValidates that the last updated blob time of the blobstore is up to date with the current existing blobs.- Throws:
IOException- on any error
-
-