Skip to content

Raw Storage

Raw Storage Key Features

  • Upload Files: Easily upload files to the Raw Storage for persistent storage and retrieval.

  • Download Files: Retrieve files from the Raw Storage for local use or processing.

  • Delete Files: Manage file storage by deleting unwanted or obsolete files.

  • Search Capabilities: Leverage search functionalities to find specific files based on criteria such as filename, metadata, or tags.

Examples

Functions

List dataset's files

.list() example

List all files in a dataset.

Arguments

  • resource_dto (DatasetDto): Dataset resource to list files from.
  • metadata_filter (Optional[dict]): Metadata filter.

Yields

  • File metadata for each file in the dataset.

Example

from odp.client import OdpClient
from odp.dto import Metadata
from odp.dto.catalog import DatasetDto, DatasetSpec
from odp.dto.common.contact_info import ContactInfo    
client = OdpClient()

# Getting Raw dataset
my_dataset = client.catalog.get("3d797de8-f4ec-48a5-b211-cae1bcfa432c")
# Optional OQS filter
my_filter = {...}

for file_metadata in client.raw.list(my_dataset, my_filter):
    print(file_metadata)

List page of dataset's files

.list_paginated() example

List page of files in a dataset.

Arguments

  • resource_dto (DatasetDto): Dataset resource to list files from.
  • metadata_filter (Optional[dict]): Metadata filter.
  • cursor (Optional[str]): Cursor for pagination
  • page_size (Optional[int]): Optional limit for each page

Returns

  • Paginated list of File metadata for files in the dataset.

Example

from odp.client import OdpClient
from odp.dto import Metadata
from odp.dto.catalog import DatasetDto, DatasetSpec
from odp.dto.common.contact_info import ContactInfo    
client = OdpClient()

# Getting Raw dataset whose data we will want to see
my_dataset = client.catalog.get("3d797de8-f4ec-48a5-b211-cae1bcfa432c")
my_filter = {...}
cursor = None
page_size = 10

paginated_result = client.raw.list_paginated(my_dataset, my_filter, cursor, page_size)
print(paginated_result)

Create and upload dataset's file

.create_file() example

Create and upload a file to a dataset.

Arguments

  • resource_dto (DatasetDto): Dataset resource to associate the file with.
  • file_metadata_dto (FileMetadataDto): Metadata for the file. Parameters can be viewed here.
  • contents (bytes): Contents of the file.

Returns

  • The metadata of the created file.

Example

from odp.client import OdpClient
from odp.dto import Metadata
from odp.dto.catalog import DatasetDto, DatasetSpec
from odp.dto.common.contact_info import ContactInfo
from odp.client.dto.file_dto import FileMetadataDto

client = OdpClient()

# Getting Raw dataset in which we will create a file
my_dataset = client.catalog.get("3d797de8-f4ec-48a5-b211-cae1bcfa432c")

file_metadata_dto = FileMetadataDto(...)

# File contents
contents = b"Hello, World!"

file_metadata = client.raw.create_file(my_dataset, file_metadata_dto, contents)
print("File created successfully:", file_metadata)

Upload dataset's file contents

.upload_file() example

Upload a file to a dataset.

Arguments

  • resource_dto (DatasetDto): Dataset resource to associate the file with.
  • file_metadata_dto (FileMetadataDto): Metadata for the file. Parameters can be viewed here.
  • contents (bytes): Contents of the file.
  • overwrite (Optional[bool], default = False): Should existing file be overwritten

Returns

  • The metadata of the created file.

Example

from odp.client import OdpClient
from odp.dto import Metadata
from odp.dto.catalog import DatasetDto, DatasetSpec
from odp.dto.common.contact_info import ContactInfo
from odp.client.dto.file_dto import FileMetadataDto

client = OdpClient()

# Getting Raw dataset in which file is already created
my_dataset = client.catalog.get("3d797de8-f4ec-48a5-b211-cae1bcfa432c")

file_metadata_dto = FileMetadataDto(...)

# File contents
contents = b"Hello, World!"

file_metadata = client.raw.upload_file(my_dataset, file_metadata_dto, contents)
print("File uploaded successfully:", file_metadata)

Download dataset's file

.download_file() example

Download a file from a dataset.

Arguments

  • resource_dto (DatasetDto): Dataset resource containing the file.
  • file_metadata_dto (FileMetadataDto): Metadata of the file to download. Parameters can be viewed here.
  • destination_path (str): Local path to save the downloaded file.

Example

from odp.client import OdpClient
from odp.dto import Metadata
from odp.dto.catalog import DatasetDto, DatasetSpec
from odp.dto.common.contact_info import ContactInfo
from odp.client.dto.file_dto import FileMetadataDto

client = OdpClient()

# Getting Raw dataset with file
my_dataset = client.catalog.get("3d797de8-f4ec-48a5-b211-cae1bcfa432c")

file_metadata_dto = FileMetadataDto(...)

# Destination_path
destination_path = "path/to/save/file.txt"

client.raw.download_file(my_dataset, file_metadata_dto, destination_path)
print("File downloaded successfully.")

Delete dataset's file

.delete_file() example

Delete a file from a dataset.

Arguments

  • resource_dto (DatasetDto): Dataset resource containing the file.
  • file_metadata_dto (FileMetadataDto): Metadata of the file to delete. Parameters can be viewed here.

Raises

  • OdpResourceNotFoundError: If the file or dataset is not found.

Example

from odp.client import OdpClient
from odp.dto import Metadata
from odp.dto.catalog import DatasetDto, DatasetSpec
from odp.dto.common.contact_info import ContactInfo
from odp.client.dto.file_dto import FileMetadataDto

client = OdpClient()

# Getting Raw dataset whose file we will want to delete
my_dataset = client.catalog.get("3d797de8-f4ec-48a5-b211-cae1bcfa432c")

file_metadata_dto = FileMetadataDto(...)

client.raw.delete_file(my_dataset, file_metadata_dto)
print("File deleted successfully.")