Projects API

The godata projects API is the primary tool you will use to interact with your project. It provides tools for creating and loading projects, as well as adding, getting and removing project data. If you’re using godata in a script, Jupyter notebook or even a larger application, you probably won’t ever have to go beyond the tools documented here.

Projects & Collections

Godata projects can be organized into collections, which can be used to group reltated projects together. Collections are automatically create when you create a project inside a collection, and are automatically deleted when the last project in a collection is deleted. Creating a project inside a collection is as simple as:

from godata import create_project

# Create 'my_project' inside 'my_collection'
project = create_project('my_project', collection='my_collection')

If you do not specify a collection, the project will be created in the ‘default’ collection.

Loading projects that you have previously created is just as easy:

from godata import load_project

# Load 'my_project' from 'my_collection'
project = load_project('my_project', collection='my_collection')

Projects can only be created once, but can be loaded any number of times from any number of python sessions including those executing concurrently.

If you forget the name of a project or collection, you can list them using the list_projects and list_collections functions:

from godata import list_projects, list_collections

# List all collections
print(list_collections())

# List all projects in 'my_collection'
print(list_projects("my_collection"))

Output:

['default', 'my_collection']
['my_project']

Hidden Collections and Projects

Hidden collections and projects are hidden from the list_collections and list_projects functions by default. You can list hidden collections and projects by passing the hidden=True argument to the list functions. In general, hidden collections and projects are designed to be used by libraries built on top of godata, which store data that is not intended to be accessed directly by the user. Note that projects and collections are hidden seperately. You can place a hidden project inside a visible collection, or a visible project inside a hidden collection.

Creating a hidden project is as simple as prepending the project name with a period:

from godata import create_project, list_projects

# Create a hidden project
project = create_project('.my_hidden_project', collection='my_collection')

# List all projects in 'my_collection'
print(list_projects("my_collection"))

# List all projects in 'my_collection', including hidden projects
print(list_projects("my_collection", hidden=True))

Output:

['my_project']
['.my_hidden_project', 'my_project']

Working with Projects

Once you have created or loaded a project, working with it is simple. The godata.project.GodataProject class provides a number of methods for adding, getting, listing, and removing data from the project. In general, you should never create a godata.project.GodataProject object directly. You should always use the create_project and load_project functions.

Project Paths and Folders

Within a project, all data is stored and accessed using a project path. A project path looks a lot like a file path, but it is always relative to the project root. For example, if you have a project with the following structure:

my_project/
    data/
        file1.txt
        file2.txt
    results/
        file3.txt
        file4.txt
        more_results/
            file5.txt

The project path for file1.txt would be data/file1.txt, the project path for file3.txt would be results/file3.txt, and the project path for file5.txt would be results/more_results/file_five.txt. File extensions are not required, and you could just as easily use data/file1, results/file3, and results/more_results/file5.

Everything prior to the last slash in a project path is considered a folder. Folders are automatically created when you add data to a project, and are automatically deleted when the last file in a folder is removed. Your folders can contain subfolders and files, just like a regular file system.

Linking vs. Storing Data

When you add data to a project, you can choose to link to the data or store it in the project. When it comes to using the data, there is no difference between linked and stored data.

GodataProject.link is used to link pre-existing data to a project without actually copying the data into godata. This is useful when you have a large amount of pre-existing data that you want to work with in godata. If the data is in a format that godata can read, it will be returned as a python object when you request it.

One key difference between linked and stored data is that linked data does not have to be a known format. If you link to a file that godata does not recognize, it will always be returned as a path when you request it.

A second key difference is the way that linked and stored data will be handled when you delete them from a project (or delete the project itself). Linked data will never be deleted by godata under any circumstances.

GodataProject.store is used to store data in godata. This is most useful when you are creating and storing objects in python that you want to able be access later. Godata will handle the serialization and deserialization of the data for you, and track it on disk.

Unlike linked data, stored data will be deleted by godata when you delete it from a project or when the project itself is deleted.

You can also store data that already exists on disk by passing a path to the store method. In this case, godata will create a copy of the data, while leaving the original data untouched. If you delete the stored data, the copy will be deleted, but the original data will be left on disk in its original location.

API Reference

The godata.create_project() and godata.load_project() return a godata.project.GodataProject object, which is used to interact with the project. You should never create a godata.project.GodataProject object directly.

godata.create_project(name: str, collection: str | None = None, storage_location: str | None = None) → GodataProject

Create a new project in the given collection. If no collection is given, this will create a project in the default collection. If the collection does not exist, it will be created. You can also specifiy a custom storage location for the project. If no storage location is given, the project will be stored in the default location.

Godata supports hidden projects and collections, which are not listed by default. This can be useful if your building a tool on top of godata that uses godata to store information you don’t intend a user to see. To reference a hidden hidden project or collection, just prepend the name with a period.

Parameters:

name (str) – The name of the project to create
collection (str, optional) – The collection to create the project in. If no collection is given, the project will be created in the default collection.
storage_location (str, optional) – A custom storage location for the project. If no storage location is given, the project will be stored in the default location.

Returns:

The newly created project.

Return type:

GodataProject

Raises:

GodataProjectError – If the project already exists in the given collection.
FileNotFoundError – If the given storage location does not exist.
NotADirectoryError – If the given storage location is not a directory.

godata.load_project(name: str, collection: str = 'default') → GodataProject

Load an existing project in the given collection. If no collection is given, this will load a project in the default collection. If the project does not exist, this will throw an error.

Parameters:

name (str) – The name of the project to load
collection (str, optional) – The collection to load the project from. If no collection is given, the project will be loaded from the default collection.

Returns:

The loaded project.

Return type:

GodataProject

Raises:

GodataProjectError – If the project does not exist in the given collection.

godata.delete_project(name: str, collection: str = 'default', force=False) → bool

Remove a project and all data stored in godata’s internal storage. This will delete any data that was stored in the project using godata.project.GodataProject.store, but will not delete any data that was linked using godata.project.GodataProject.link.

Parameters:

name (str) – The name of the project to delete
collection (str, optional) – The collection to delete the project from. If no collection is given, the project will be deleted from the default collection.
force (bool, optional) – Required to be set to True to delete the project. This is a safety measure to prevent accidental deletion of projects.

Returns:

True if the project was deleted successfully.

Return type:

bool

Raises:

GodataProjectError – If the project does not exist in the given collection.

godata.has_project(name: str, collection: str = 'default') → bool

Check if a project exists in the given collection. If no collection is given, this will check if the project exists in the default collection.

Parameters:

name (str) – The name of the project to check for
collection (str, optional) – The collection to check for the project in. If no collection is given, the project will be checked for in the default collection.

Returns:

True if the project exists.

Return type:

bool

godata.has_collection(name: str) → bool

Check if a collection exists. This will return True if the collection exists, and False if it does not.

Parameters:: name (str) – The name of the collection to check for.
Returns:: True if the collection exists.
Return type:: bool

godata.list_projects(collection: str = 'default', show_hidden: bool = False, display: bool = False) → list[str]

Return a list of projects in the given collection. If no collection is given, this will return a list of projects in the default collection.

By default, hidden projects are not listed. This function outputs a list of strings of the project names. If display is set to True, this function will print the list of projects to the console. This is used for the godata CLI, or for working in a Jupyter notebook.

Parameters:

collection (str, optional) – The collection to list the projects from. If no collection is given, the projects will be listed from the default collection.
show_hidden (bool, optional) – If set to True, hidden projects will be listed.
display (bool, optional) – If set to True, the list of projects will be printed to the console.

Returns:

A list of project names in the given collection.

Return type:

list[str]

godata.list_collections(show_hidden=False, display=False) → list[str]

Return a list of collections. By default, hidden collections are not listed. This function outputs a list of strings of the collection names. If display is set to True, this function will print the list of collections to the console. This is used for the godata CLI, or for working in a Jupyter notebook.

Parameters:

show_hidden (bool, optional) – If set to True, hidden collections will be listed.
display (bool, optional) – If set to True, the list of collections will be printed to the console.

Returns:

A list of collection names.

Return type:

list[str]

class godata.project.GodataProject(collection: str, name: str)

A GodataProject object is the main interface for interacting with projects. It contains tools for adding, removing, and listing files in the project.

The GodataProject is responsible for interfacing with the godata server via the client module. It is also responsible for delegating file reading and writing to the io module.

You should not create a GodataProject object directly. Instead, use the create_projecct or load_project functions to create a new project or load an existing one, respectively.

link(file_path: str | Path, project_path: str, metadata: dict = {}, recursive: bool = False, overwrite=False, verbose=True, _force=False) → bool

Link a pre-existing file or folder to the project. This will not actually move any data around. External data that is linked to the project will not be deleted from disk under any circumstances, including if the project itself is deleted. If you prefer to move the data into the project, use the godata.project.GodataProject.store method instead.

You can link any file you want to the project, regardless of whether it is a known file type. If the file is of a known type, subsequent gets will return the file as a python object by default.

Parameters:

file_path (str | pathlib.Path) – The path to the file or folder to link to the project.
project_path (str) – The path in the project to link the file to.
metadata (dict, optional) – A dictionary of metadata to attach to the file. This can be used to store additional information about the file.
recursive (bool, optional) – If the file_path is a folder, the link will recursively add data in all subfolders to the project as well. Otherwise only files within the folder will be included.
overwrite (bool, optional) – If a file or folder already exists at the given project path, this will overwrite it. Note that if an overwritten file was added to the project using the store method, the file will be deleted from disk.
verbose (bool, optional) – If set to True, this will print a message to the console indicating the result of the operation.

store(object: Any, project_path: str, overwrite=False, verbose=True, format: str | None = None, writer_kwargs: dict = {}) → bool

Stores a given python object or file in godata’s internal storage at the given project path. If the object is a path, the file will be copied into the project’s storage.

Store is different from link in that it will actually move data into godata’s internal storage, rather than just creating a reference to the file. This also means that subsequent removals of the file from the project will actually delete the data from disk. This can be done either via an overwrite (with link or store), explicit removal with the GodataProject.remove method, or by deleting the project itself.

The “format” and “writer_kwargs” arguments can be used to customize the writing process. For example, a pandas dataframe by default will be written as a csv file. If you want to write it as a parquet file instead, you can pass format=”.parquet”. The writer_kwargs argument will be passed directly to the underlying method, such as pd.DataFrame.to_csv.

Parameters:

object (Any) – The object to store in the project. This can be any python object, or a path to a file.
project_path (str) – The path in the project to store the file.
overwrite (bool, optional) – If a file or folder already exists at the given project path, this will overwrite it. Note that if an overwritten file was added to the project using the store method, the file will be deleted from disk.
verbose (bool, optional) – If set to True, this will print a message to the console indicating the result of the operation.
format (str, optional) – The format to write the file in. If no format is given, godata will attempt to infer the format from the object. If the format is not known, this will throw an error.
writer_kwargs (dict, optional) – A dictionary of keyword arguments to pass to the writer function. This can be used to customize the writing process.

Returns:

True if the object was stored successfully.

Return type:

bool

Raises:

GodataProjectError – If the object is not a known type and no format is given.
godataIoException – If the object is a known type, but no writer is found.

get(project_path: str, as_path: bool = False, load_type: type | None = None, reader_kwargs: dict = {}) → Any

Get an object at a given project path. This method will return a python object whenever possible. If godata doesn’t know how to read in a file of this type, it will return a path. The path can also be returned explicitly by passing as_path = True.

The “load_type” and “reader_kwargs” arguments can be used to customize the reading process. For example, a csv file is by default read into a pandas dataframe, but can also be read as a Polars dataframe by passing load_type = polars.DataFrame. Note this is the class itself, not a string. The reader_kwargs argument will be passed directly to the underlying read method.

Parameters:

project_path (str) – The path in the project to get the file from.
as_path (bool, optional) – If set to True, this will return the path to the file, rather than the object itself.
load_type (type, optional) – The type to load the file as. This can be any python type, such as a pandas DataFrame, or a polars DataFrame.
reader_kwargs (dict, optional) – A dictionary of keyword arguments to pass to the reader function. This can be used to customize the reading process.

Returns:

The object at the given project path, or the path to the file: if as_path is set to True or the object cannot be read.

Return type:

Any

Raises:

godataIoException – If the object is a known type, but no reader is found.
GodataProjectError – If the file does not exist in the project.

move(src_project_path: str, dest_project_path: str, overwrite: bool = False, verbose: bool = True) → bool

Move a file or folder from one location in the project to another. This will throw an error if the destination already exists. If you want to overwrite the destination, set overwrite to True. Note that overwriting a file that was added to the project using the store method will delete the file from disk.

If the data being moved is stored in godata’s internal storage, this will not necessarily move the data on disk.

Parameters:

src_project_path (str) – The path in the project to move the file from.
dest_project_path (str) – The path in the project to move the file to.
overwrite (bool, optional) – If a file or folder already exists at the given project path, this will overwrite it. Note that if an overwritten file was added to the project using the store method, the file will be deleted from disk.
verbose (bool, optional) – If set to True, this will print a message to the console indicating the result of the operation.

Returns:

True if the file was moved successfully.

Return type:

bool

Raises:

GodataProjectError – If the destination already exists and overwrite is not set to True.

remove(project_path: str) → bool

Remove a file or folder from the project. If this file exists outside of the project’s storage and was added using the link method, this will not delete the file from disk. If the file was added to the project using the store method, this will delete the file from disk.

Parameters:: project_path (str) – The path in the project to remove the file/folder from.
Returns:: True if the file or folder was removed successfully.
Return type:: bool
Raises:: GodataProjectError – If the file does not exist in the project.

list(project_path: str | None = None) → dict[str, str]

List the contents of a given project path. This will return a dictionary containing the files and folders at the given path. If no path is given, this will list the contents of the project root.

The return will always be of the form

{"files": ["file1", "file2", ...], "folders": ["folder1", "folder2", ...]}

Parameters:: project_path (str, optional) – The path in the project to list. If no path is given, this will list the contents of the project root.
Returns:: A dictionary containing the names of the files and folders at the given path.
Return type:: dict[str, list[str]]
Raises:: GodataProjectError – If the given path does not exist in the project or is not a folder.

ls(project_path: str | None = None) → None

Utility function for listing the contents of a given directory in a human-readable format. This is used for the godata CLI, or for working in a Jupyter notebook. Example output:

>> project = load_project("my_project")
>> project.ls("folder1")

    my_project/folder1:
    -------------------
    subfolder1/
    subfolder2/
    file1
    file2

Parameters:: project_path (str, optional) – The path in the project to list the contents of. If no path is given, this will list the contents of the project root.
Raises:: GodataProjectError – If the given path does not exist in the project or is not a folder.

has_path(project_path: str) → bool

Check if a given path exists in the project.

Parameters:: project_path (str) – The path in the project to check for.
Returns:: Whether the given path exists in the project.
Return type:: bool