0:0018:44

Storing your data in the cloud: doing it right

Orit Wasserman 18:44 2021

Treating cloud object storage like a file system wastes its best features

Unfinished multipart uploads keep sitting in a bucket, using capacity someone pays for, and ten saved versions of a 5 MB document take up 50 MB. Orit Wasserman, who worked on Ceph object storage at Red Hat and supported large customers, has seen this kind of suboptimal usage often. Which built-in features stop the waste automatically, and which settings keep cloud data from leaking?

Learn when to use multipart upload, versioning, lifecycle rules and bucket security settings

Watch on YouTube

In short

Orit Wasserman, OpenShift Container Storage Architect at Red Hat, explains what cloud object storage is and why its flat, immutable design suits the scale of the cloud. Drawing on her work developing Ceph object storage, she walks through multipart upload, versioning, lifecycle rules and prefixes, and closes with practical security and privacy dos and don'ts.

“You cannot update a part of the object. You overwrite it completely.”
“And the change, what changed and made object storage so popular is the cloud.”

What you will learn

  • Explain how object storage differs from block and file storage: flat namespace, immutable objects and rich metadata.
  • Decide when to use multipart upload, and check SDK thresholds so small objects avoid it and objects over 5 GB use it.
  • Configure lifecycle rules to expire old versions, clean up incomplete uploads and move unread objects to an archive tier.
  • Organize objects with prefixes to simulate folders and filter bucket listings.
  • Secure buckets with HTTPS, client-side encryption, no public buckets and least-privilege bucket and user policies.

Key takeaways from the speaker

  • What is Cloud Object Storage?
  • When and how to use Cloud Object Storage?
  • Cloud Object Storage special features

Details

Level
Intermediate
Format
Talk
Technical depth
Practitioner
For
Early career, Mid-career, Senior individual contributor
Length
18:44
Published
June 2021

Questions this talk answers

What makes object storage different from block and file storage?

Object storage uses a flat namespace of buckets holding objects referenced by name, and objects are written as a whole rather than partially updated. This simplifies management, lowers cost and allows rich user metadata.

When should multipart upload be used, and when not?

Use it for large objects, and always above the 5 GB single-upload limit, because it allows resuming, pausing and parallel uploads. Avoid it for small objects, where the extra requests can make throughput up to three times slower.

How do you stop versioning and failed uploads from wasting storage?

Set lifecycle rules, for example to delete noncurrent versions older than 30 days and incomplete multipart uploads older than seven days. Tiering rules can also move objects not read for a year to an archive.

Can object storage have folders?

Not natively, but prefixes added to object keys simulate a hierarchy. Listing a bucket with a prefix returns only the objects under that prefix.

How do you keep data in cloud object storage secure and private?

Use HTTPS, use client-side encryption with your own keys or a key management system, never use public buckets, and grant the most limited access possible with bucket and user policies.

About this talk

We all use cloud object storage extensively, like Amazon S3, Azure Blob Storage, Google Cloud Storage, and others. However, not many utilize its unique capabilities and advanced features, such as versioning, security, life cycle...

Throughout my work developing Ceph object storage and supporting large customers, I’ve seen a lot of incorrect and suboptimal usage. This session will briefly discuss object storage design, highlight valuable features and review practical dos and don’ts.

Topics

Related talks

Browse the library