Object Storage for AI Workloads: Where Should Datasets, Artifacts and Logs Live?
April Wong
Store shared input datasets, published artifacts and retained log batches in object storage when applications can read them through an object API. Keep frequently changing job records in a database, and working files on storage suited to the execution environment. The important boundary is not simply hot versus cold data. It is which copy is authoritative, who may change it, and how a consumer knows that a dataset or result is complete.
TL;DR
- Separate source data, published artifacts, temporary working files and operational records. They have different access and retention requirements.
- Publish a manifest containing exact object keys, checksums and provenance. A folder named
latestis not a reproducible dataset. - Treat the object upload and the database update as separate operations. Make retries safe and expose results only after verification.
- Verify access scope, retention, restore and client compatibility on the actual service. S3 compatibility does not establish support for every S3 feature.
Which AI data belongs in object storage?
Object storage is useful when several services need named data without attaching the same disk to every consumer. It can serve active datasets and application content, not only archives. A worker can retrieve an input, produce a result and publish that result for other services to use.
Start with the data's role:
| Data | Useful starting location | What to record elsewhere |
|---|---|---|
| Original documents, images and dataset shards | Object storage with stable, version-specific keys | Owner, permitted uses, schema and dataset release |
| Model packages, evaluation reports and generated outputs | Object storage, published as identifiable artifacts | Run ID, input release, code revision and approval status |
| Temporary unpacked files and intermediate working state | Worker filesystem or persistent workspace, according to restart needs | Which outputs must survive loss of the worker |
| Job status, permissions and current artifact pointers | Transactional application database | Object identifiers and lifecycle state |
| Recent logs needed for interactive investigation | A suitable log or analytical service | Retention and access policy |
| Finalized logs retained for replay or longer-term analysis | Partitioned object batches | Schema version, time range and collection completeness |
MLflow's architecture separates large run artifacts from parameters, metrics and tags held in its backend store. It supports S3-compatible artifact locations, but that does not certify every provider or configuration.
For the underlying interface choices, see our block storage versus object storage guide. Here, the next question is how to make the data usable throughout its lifecycle.
How do you make datasets and artifacts reproducible?
Give each published dataset a stable release identity. Avoid overwriting the same object keys every time the source changes. A model evaluation that records only datasets/current/ can become impossible to reproduce after the next ingestion job.
An illustrative layout could contain:
datasets/support-docs/release-004/part-0001.parquet
datasets/support-docs/release-004/manifest.json
artifacts/run-028/report.json
artifacts/run-028/manifest.json
logs/service-a/date=2026-11-18/batch-001.jsonl
These are example keys, not a Nirvana naming requirement or a benchmark dataset. A dataset manifest should identify its objects, byte sizes, checksum algorithm and values, schema version, source references and permitted-use classification. An artifact manifest should additionally identify the input release, code revision, relevant configuration and producing run.
Keep sensitive inputs and credentials out of broadly accessible manifests.
Calculate and verify an explicit checksum. Do not treat every ETag as a whole-file MD5: Amazon S3 documents that multipart uploads and other circumstances change that interpretation. Verify the checksum operations supported by your endpoint and client.
Where supported, storage versioning can help recover overwritten or deleted objects. It does not replace the application manifest that explains which versions belong to one release. Amazon S3 Versioning, for example, is explicitly enabled and retains whole object versions. Do not assume those settings or semantics are enabled on another service.
How should workers publish results without exposing partial work?
Design publication as an application protocol. An object upload and a row written to your job database are not automatically one transaction.
A practical sequence is:
- Assign the attempt a unique run ID and output keys. Record the job as running.
- Upload the output objects. Handle interrupted transfers and retries using the supported client operations.
- Verify the expected objects, sizes and checksums. Write a manifest listing the complete output set.
- Commit a database record that references the verified manifest and marks the result available.
- Have consumers discover available records, rather than treating any key found in a bucket listing as a completed result.
This is a recommended design pattern, not a claim that the storage service provides cross-system transactions. Use a database uniqueness constraint or equivalent concurrency control to prevent two retries from publishing conflicting final records for the same logical job.
Test the awkward cases deliberately: the worker stops after uploading but before updating the database; the database commits but the response is lost; a consumer requests an object that has been removed. Decide how reconciliation finds orphaned objects and missing references. Do not immediately delete every unreferenced key, because an upload may still be in progress.
For logs, use bounded batches and record their time range and schema. A bucket is not itself an incident console. Define the live investigation layer separately from retained data for replay or analytical queries.
What should access, retention and recovery tests cover?
Separate environments and trust boundaries before choosing convenient bucket names. A production/ prefix is an organizational convention unless your access system actually enforces it.
Nirvana's access-key documentation states that a key is scoped to a project and region and works across that project's buckets in that region. To limit a workload to a subset of buckets, the documented approach is to place those buckets in a separate project. Design around that boundary instead of assuming per-prefix credentials.
The public-object documentation says objects are private by default and describes presigned URLs for time-limited sharing. Treat such URLs as bearer access: avoid embedding them in public logs, permanent dataset manifests or support screenshots.
Retention needs an owner and a recovery objective. Distinguish temporary outputs, retained artifacts, production releases and logs. Check whether deletion must also remove derived datasets or application records. Test lifecycle rules on disposable data first: Amazon S3's documentation notes that new rules can apply to existing objects too. This AWS example does not establish Nirvana lifecycle support.
Before production, test retrieving a known release from a replacement worker, detecting a corrupted download, denying an unrelated project's credentials and recovering after accidental deletion. Distinguish service guarantees from your independent copies and procedures.
How can you evaluate this design on Nirvana?
Nirvana launched Object Storage on 17 November 2026. The launch adds shared, S3-compatible buckets alongside compute and high performance block storage.
Use the Object Storage quickstart to create a bucket through the Nirvana API, CLI or Terraform, then configure an S3 client. Read the returned endpoint and signing region rather than copying assumptions from another deployment. Keep credentials outside article examples and source control.
Start with one representative dataset and one producer-consumer flow. Measure upload and download time, request failures, retry behavior and first useful application result at realistic concurrency. Test the actual SDK version, multipart behavior and permissions you intend to deploy. This article reports no new performance measurements.
Pair object storage with high performance block storage when a supporting database or application needs a disk. The value is a clear division of responsibilities: shared data and artifacts in buckets, active application state in the appropriate service, and an explicit publication and recovery protocol connecting them.
FAQ
Can object storage hold active AI datasets?
Yes. The API access pattern can suit shared inputs and published artifacts used continuously. Test transfer patterns and concurrency; object storage is not synonymous with infrequently accessed data.
Is a bucket a model registry?
No. A bucket stores objects. A registry or application catalog adds identities, approvals, relationships and discoverability. Keep an explicit link between its records and the exact artifact objects.
Does S3-compatible mean every Amazon S3 feature works?
No. Verify the operations, security controls, checksums, versioning and retention behavior your application requires against the provider's current documentation and a controlled test.
Is an artifact upload a complete agent checkpoint?
Only if the application deliberately saved everything needed to restart. A stored file does not automatically include process memory, in-flight network calls or external side effects.
Reviewed 18 November 2026. Examples describe an application design and evaluation method, not completed tests or universal storage guarantees.
About Nirvana Labs
Nirvana Labs is a high performance storage cloud built for blockchain, AI and databases: demanding, real-time, stateful workloads. Its high performance block storage, compute, private networking, Nirvana Kubernetes Service and Agent Sandboxes help teams build and operate data-intensive applications.
Learn more at Nirvana Labs.
Nirvana Cloud | Pricing | Blog | Docs | Changelog | LinkedIn | Twitter | Telegram | YouTube
Related Posts

ClickHouse® Storage Architecture: Hot Data, Object Storage and Merge Pressure
Choose a ClickHouse storage layout by separating authoritative data, the fast serving path and recovery. Test cache behavior, merges and failures together.

How to Choose a Cloud Region for Stateful Applications
Choose a cloud region by application latency, service availability, failure domains and recovery requirements. A practical evaluation for stateful workloads.

What Does a Kubernetes Database Operator Actually Manage?
Understand what Kubernetes database operators automate, what your team still owns, and how to evaluate backups, failover and upgrades before production.