Treating cloud object storage like a file system wastes its best features
Unfinished multipart uploads keep sitting in a bucket, using capacity someone pays for, and ten saved versions of a 5 MB document take up 50 MB. Orit Wasserman, who worked on Ceph object storage at Red Hat and supported large customers, has seen this kind of suboptimal usage often. Which built-in features stop the waste automatically, and which settings keep cloud data from leaking?
Learn when to use multipart upload, versioning, lifecycle rules and bucket security settings
Orit Wasserman, OpenShift Container Storage Architect at Red Hat, explains what cloud object storage is and why its flat, immutable design suits the scale of the cloud. Drawing on her work developing Ceph object storage, she walks through multipart upload, versioning, lifecycle rules and prefixes, and closes with practical security and privacy dos and don'ts.
“You cannot update a part of the object. You overwrite it completely.”
“And the change, what changed and made object storage so popular is the cloud.”
What you will learn
Explain how object storage differs from block and file storage: flat namespace, immutable objects and rich metadata.
Decide when to use multipart upload, and check SDK thresholds so small objects avoid it and objects over 5 GB use it.
Configure lifecycle rules to expire old versions, clean up incomplete uploads and move unread objects to an archive tier.
Organize objects with prefixes to simulate folders and filter bucket listings.
Secure buckets with HTTPS, client-side encryption, no public buckets and least-privilege bucket and user policies.
Key takeaways from the speaker
What is Cloud Object Storage?
When and how to use Cloud Object Storage?
Cloud Object Storage special features
Details
Level
Intermediate
Format
Talk
Technical depth
Practitioner
For
Early career, Mid-career, Senior individual contributor
Length
18:44
Published
June 2021
Questions this talk answers
What makes object storage different from block and file storage?
Object storage uses a flat namespace of buckets holding objects referenced by name, and objects are written as a whole rather than partially updated. This simplifies management, lowers cost and allows rich user metadata.
When should multipart upload be used, and when not?
Use it for large objects, and always above the 5 GB single-upload limit, because it allows resuming, pausing and parallel uploads. Avoid it for small objects, where the extra requests can make throughput up to three times slower.
How do you stop versioning and failed uploads from wasting storage?
Set lifecycle rules, for example to delete noncurrent versions older than 30 days and incomplete multipart uploads older than seven days. Tiering rules can also move objects not read for a year to an archive.
Can object storage have folders?
Not natively, but prefixes added to object keys simulate a hierarchy. Listing a bucket with a prefix returns only the objects under that prefix.
How do you keep data in cloud object storage secure and private?
Use HTTPS, use client-side encryption with your own keys or a key management system, never use public buckets, and grant the most limited access possible with bucket and user policies.
About this talk
We all use cloud object storage extensively, like Amazon S3, Azure Blob Storage, Google Cloud Storage, and others. However, not many utilize its unique capabilities and advanced features, such as versioning, security, life cycle...
Throughout my work developing Ceph object storage and supporting large customers, I’ve seen a lot of incorrect and suboptimal usage. This session will briefly discuss object storage design, highlight valuable features and review practical dos and don’ts.
Speaker 1: Hello, everyone. Welcome to storing your data in the cloud, doing it right. Today, I'm going to introduce you to Optics Storage, the right way to store data in the cloud. I'm Orit Wasserman. I'm a senior principal software engineer at Red Hat. Speaker 1: I have been developing software for many, many years already, and I'm almost half of my career in open source. I love low level programming and especially storage and being working on different storage systems distributed for me in quite a while.
Speaker 1: In the last few years, I'm focusing on Ceph, which is a social defined storage, completely open source. And of course, it provides also object storage, also block and file. Speaker 1: And today, I'm an OpenShift Data Foundation architect at Red Hat. OpenShift Data Foundation provides persistent data services for containers application running in OpenShift. Speaker 1: It is Red Hat platform as a service infrastructure for the hybrid cloud based on Kubernetes. And I'm from Israel if anyone interested working from home even before COVID.
Speaker 1: So today, I want to introduce you to object storage. I want to tell you what's special about it and why it's such a good fit to the cloud. Speaker 1: Then go very cool, unique feature it has that make it even perfect, more perfect for the cloud. Speaker 1: And as we are talking about cloud, we also have to have a section about security and privacy.
Speaker 1: So storage is. Speaker 1: Is a very important infrastructure been around for quite a while. This is actually the evolution and. Speaker 1: And now in today's cloud era, it's even is transformed even more. Speaker 1: And usually when we talked about storage, we would divide it to block storage or file system storage. Speaker 1: And but now today we also have a new kind of storage called object storage. So why do we need this first type?
Speaker 1: So basically what happened is that we create so much data set a scale data actually it's not. Speaker 1: As you see, it's like 1,300 extra bite which is actually it's at the bite and it scores exponentially, which means that we need different kind of storage system to store the data. Speaker 1: And it can be very, very scalable and also be cost effective because it's also quite expensive to store storage data and block storage and file system.
Speaker 1: I've inherited limitation that make it harder. So that's why there is the carrying the object storage. Speaker 1: So what is object storage? It's first of all, it's a flat namespace. Speaker 1: We have buckets buckets contain objects and we reference object by name. Speaker 1: So so it's bucket. So if you still bucket, you want to, if it's an object, you tell this bucket and that object name, but a bucket cannot contain another bucket.
Speaker 1: Unlike with file system, the directories can have other directories inside. We are talking about flat namespace. Speaker 1: This limitation helps with reducing the cost of the storage and make the management much, much more easier for the storage. Speaker 1: The second difference from files is that objects are immutable. Speaker 1: What do you mean immutable? I don't mean that objects cannot change and be updated, but the way we have them.
Speaker 1: Then them is as one entity. You cannot update a part of the object. You overwrite it completely. Speaker 1: You can of course read whatever part of the object, read read, whatever you want, but when you write an object, it's always the full object. Speaker 1: And because it's much easier to manage the storage with object storage, we can add very rich metadata to the object and better access control, but also user metadata.
Speaker 1: So how I'm as a user of object storage can store my metadata on the object. Speaker 1: And we can see example where it can help us. It can helps with analytics, with searching, search of object and so on. Speaker 1: Object storage has been around for a while. People don't know, but from the middle of the nineties, it's quite a while, but it didn't break out.
Speaker 1: It was like a niche, EMC at Sentera, which provided, they call it content addressable storage, but that was it. Speaker 1: And the change, what changed and made object storage so popular is the cloud. In 2006, AWS announced their cloud service, EC2, Elastic Compute Cloud, and a storage service that comes with it called S3. Speaker 1: Simple storage service, and I really like that. And S3 is object storage. Object storage with one addition that is very important.
Speaker 1: All the access to the storage, all the management is done with Westfall API. Speaker 1: It means that API is always by using an HTTP request, which is perfect for the cloud because in cloud, we always have lots of service all around. Speaker 1: Usually all our access are remote and HTTP makes it easier. It's quite easy to use HTTP to access a remote server or remote storage.
Speaker 1: You don't need to do it. So the firewalls are open. Everything is standard, standard networks and much, much easier than the storage protocols. Speaker 1: And today, object storage is dominating the cloud. Of course, AWS S3, which is huge. Every big cloud vendor has its own object storage, Google Cloud Storage, for example, as low storage and so on.
Speaker 1: And on premise, we have staff that also provide S3-compatible object storage. Speaker 1: And I noticed I want to use DigitalOcean because DigitalOcean are using safe to provide cloud object storage as well. Speaker 1: So that's a little exam of uses of object storage. So the classical, as we say, is photos. So if you think about photos are nature immutable.
Speaker 1: Because of the format of photos, even if we just edit the photo, let's say rotate, we have to form to write the photo as one unit. Speaker 1: You never write parts of the photo. So objectoid is classical for photos. This is Flickr. Speaker 1: I don't know what's going on Flickr. It used to be our, but I don't know. I always use Flickr because they use F as well.
Speaker 1: So here in Flickr, we have albums. Album is a bucket. And of course, it's photo is a different object. Speaker 1: So our first problem is uploading the photos to the cloud. Photos are growing actually bigger. We have better cameras on the phone. Speaker 1: So when we upload the photo in the cloud, it's very annoying if we have some network or some other problem and we need to upload them again.
Speaker 1: And also, it's not always photos. Sometimes we want a video. Those are even bigger. So we need a different way to upload the photos, the object to the cloud. And object storage has what we call multi-part upload to solve the problem. Speaker 1: So multi-part upload with the name. We take the big object and we cut it into parts and we upload each part separately.
Speaker 1: It's a transaction based protocol. First of all, you have an init command. You go to the object storage, tell him, I want to update this object. Give it the name and you can tell him how many parts and parts sides, but you don't have to. And the object storage will return and upload ID. Speaker 1: You need to use it all keep it. And then we upload each part separately and without correct upload ID. And when we are done, we send the complete command and only then the object will be created on the object storage and visible them.
Speaker 1: So now when I upload it for a big object, first of all, in case I have a network problem, Speaker 1: All the parts that were already successfully uploaded will be there. Speaker 1: Then I also can parallelize the upload, getting much better network throughput, much shorter time to upload. I can pause and resume object. So for example, for my phone, I only upload photos when I'm Wi-Fi area. So if I go out of the Wi-Fi area, that would Speaker 2: I will pause and when I go back, it will continue from the last part that I still didn't upload.
Speaker 1: You can upload an object. You don't know the actual, the final size, for example, when you stream video and you can use it as a transaction instead of what you use, which file system rename. Speaker 1: But there are some pitfalls from using a multi-power upload. First of all, Speaker 1: You need that to make, to understand the performance instead of one request, we have free. So if the object is small, you want to make sure that you actually don't use multiple upload Speaker 1: Because it can get up to three times slower throughput. And also if you use regular upload, you should know that the object size is limited to five giga. So always for object larger than five giga, you should use multiple upload, which is limited by five.
Speaker 1: Luckily, when we use S3, we have lots of frameworks and SDKs that can handle all those for us. And you just need to make sure that the defaults are set correctly and the threshold to using multiple upload is correct. Speaker 1: Another problem we have is, is what happens if I never complete the upload, I'm storing the parts in the storage. So I'm using capacity and actually paying for it, but it's a waste.
Speaker 1: We call those our friend uploads and those needs to be sealed up. I can do it manually. And later in the presentation, I'm going to present you how you can do it automatically. Speaker 1: Let's look at another example of documents. So first, it seems that documents are not a good fit for object storage because documents are actually very immutable. You've changed them a lot in all, in every part.
Speaker 1: But as they are relatively not large and the updates are infrequent, you're in frequency and slow compared to computers, they actually are a good fit. Speaker 1: And when object storage adds to documents, which is a very important feature, as you can see here, I can keep version from the document. So if I did a mistake and I will get back, I can go to the search and get an older version.
Speaker 1: So what happens if I don't specify which version I want to read, I'll always read with the current version, the latest, but I can specify the version ID and get an old version. If I want to, I can revert to it. Speaker 1: So basically, versioning is a bucket feature. You enable it per bucket. And then you'll get every time you overwrite an object, it won't be the old data.
Speaker 1: The old data won't be deleted, but the new version is added. As you can see, one problem here is that it uses lots of space. So if I have a document, it's five mega usually. If I have 10 version of it, I'm using 50 mega. That's a lot. Speaker 1: So what we need to do to remember when we use versioning is to delete the old version that I don't, no longer need. So we free up space.
Speaker 1: Sadly, I don't remember to do anything, but here the object storage help us. Speaker 1: And you can see this is an example from drive. The older version are only kept for 30 days. The object storage will delete them for us. This is, I think, with the best feature object storage has. And it's called life cycle.
Speaker 1: Life cycle and life cycle. You can configure a world on the object that the object storage and when the condition met the object storage will execute the rules. You have two types of transitions. Speaker 1: One is expiration deleting the data and one is to moving the object. So for example, with expiration, I can tell the storage, I want to delete all the versions that are not current version, but are older than 30 days. This is one kind of rule. The other kind of rule, I can say, I don't want to, I want to delete all the multi-par uploads that haven't completed for a week, for seven days. This will clean all of Speaker 1: Failed multiple upload ahead and clean the space automatically without any need of my involvement.
Speaker 1: And usually Turing is used for archiving. I can say any object that wasn't read for the last year should be moved to the archive as it's not used. It's a very cool feature. I think it's very useful. Speaker 1: And an example I said we are flat namespace, but you can actually have a much complex hierarchy. For example, here you are for the one containing for the true that content for the free. You can do it in object storage.
Speaker 1: And this is with what it's called in a free prefix prefix is basically a key, a metadata key that you can add to an object. Speaker 1: So for example, an object that time for the one will have prefix prefix slash for the one object for the two will ever prefix slash for the one slash for the two and so on.
Speaker 1: And when we do a listing of a bucket with the prefix, you'll get only the object that specified this prefix. Speaker 1: So if I want to see only the object in for the two, I will give that prefix to the listing and get them. Speaker 1: As we are in the cloud, it's very important to make sure your data is secure and to keep it in private.
Speaker 1: I've sadly seen many MS configuration that made the data public. Speaker 1: So first of all, use HTTP unless it's your own private network or you have VPN. Speaker 1: Sadly, it's not default. Speaker 1: I wish they would do that, but make sure HTTPS transport is encrypted. Speaker 1: Next, next, the storage itself always encrypt the data when it gets the data.
Speaker 1: This is service side encryption, but that's not good for your privacy. Speaker 1: Object storage has client side encryption, meaning the data is already encrypted in the SDK basically. Speaker 1: And even before it gets to the network actually. Speaker 1: And when the storage gets it, it's already encrypted this way. Speaker 1: The storage provider can erase your data.
Speaker 1: You have two types of keys. Speaker 1: You can provide your keys or use the key management system. Speaker 1: Another thing, we have access control for bucket. Speaker 1: And one of the default, but it's not default, one of the options public bucket. Speaker 1: Please don't use public buckets. Speaker 1: There are lots of leaks every few months of public buckets.
Speaker 1: Today, AWS actually can warn you if you have such buckets. Speaker 1: So we have lots of granularity of access control to buckets and object. Speaker 1: Sadly, I don't have time to go over them. Speaker 1: Please use them and try to use the most limited access you can. Speaker 1: And we have new, we have bucket and user policy to allow you much better granularity.
Speaker 1: For example, you can share bucket between user and you can give read only for anonymous user. Speaker 1: You can restrict for specific IP or MAC address. Speaker 1: So presenting you today object storage. Speaker 1: It's designed for the scale of the cloud and its unique features make it perfect to the cloud and it can help you to develop much better application for the cloud.
Speaker 1: But to use the feature, please use the storage, the APIs and not use it as a file system as that will, won't allow you to use features like versioning or lifecycle and so on. Speaker 1: We are in the cloud. Speaker 1: Make sure your data is safe and it's private. Speaker 1: And if you use a private cloud, please use safe Windows Gateway for your object storage.
Speaker 1: Feel free to pin for question in the, I assume will be the conference chat that is set. Speaker 1: And always like thank you. Speaker 1: Thank you.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why cloud object storage works best when you stop treating it like a file system
Red Hat storage architect Orit Wasserman explains the object storage features most teams leave unused, and the security mistakes that expose data.
In short
Cloud object storage such as Amazon S3 was designed for the scale of the cloud, and its real value lies in features like multipart upload, versioning and lifecycle rules. Orit Wasserman argues that teams get these benefits only when they use the object storage APIs directly rather than a file system layer, and only when they lock down access and encryption. Drawing on her work developing Ceph object storage, she shares the practical dos and don'ts she has seen with large customers.
Key takeaways
What cloud object storage is
When and how to use cloud object storage
The special features of cloud object storage
What object storage is and why the cloud made it popular
Wasserman opens by explaining that storage has traditionally been divided into block and file storage, but the sheer volume of data being created today demands something more scalable and cost effective. Object storage answers that with a deliberately simple design: a flat namespace in which buckets contain objects, objects are referenced by name, and a bucket cannot contain another bucket.
"This limitation helps with reducing the cost of the storage and make the management much, much more easier for the storage." (3:36)
The second difference is immutability. Objects can still be replaced, but never partially changed: "You cannot update a part of the object. You overwrite it completely." (4:05) Because management is simpler, object storage can also carry rich metadata, including user metadata that helps with analytics and search.
Wasserman notes that object storage has existed since the mid-1990s, citing EMC Centera, but remained a niche until AWS launched EC2 and S3 in 2006. "And the change, what changed and made object storage so popular is the cloud." (5:03) The key addition was an HTTP-based RESTful API, which works over standard networks and open firewalls, ideal when every access is remote. Today every major cloud vendor offers object storage, and Ceph provides S3-compatible object storage on premises.
Uploading large objects with multipart upload
Using photo albums as an example, where an album is a bucket and each photo an object, Wasserman turns to a common pain: large uploads that fail halfway. Multipart upload splits a big object into parts uploaded separately inside a transaction. An init command returns an upload ID, each part is uploaded with that ID, and a complete command makes the object visible.
The benefits are real: parts already uploaded survive network failures, uploads can run in parallel for better throughput, and they can pause and resume, for example only uploading on Wi-Fi. It also works when the final size is unknown, such as streaming video.
There are pitfalls too. For small objects, the extra requests can make throughput up to three times slower, as the speaker cited, while regular uploads are limited to 5 GB, so larger objects must use multipart upload. "Luckily, when we use S3, we have lots of frameworks and SDKs that can handle all those for us." (10:39) Her advice is to check that the defaults and the multipart threshold are set correctly. Uploads that never complete leave orphaned parts that consume capacity you pay for.
Versioning and lifecycle rules
Documents seem a poor fit for immutable objects, but because they are relatively small and updated infrequently, they work well, especially with versioning. Enabled per bucket, versioning keeps old data whenever an object is overwritten, so a mistake can be undone by reading or reverting to an earlier version ID. The cost is space: ten versions of a 5 MB document use 50 MB.
"Sadly, I don't remember to do anything, but here the object storage help us." (13:03) Lifecycle rules, which Wasserman calls one of the best features object storage has, let the storage act automatically when conditions are met. Expiration rules can delete noncurrent versions older than 30 days or remove multipart uploads that have not completed within seven days. Transition rules handle tiering: "I can say any object that wasn't read for the last year should be moved to the archive as it's not used." (14:19)
She also shows how prefixes recreate a folder-like hierarchy inside the flat namespace. Each object carries a prefix such as a folder path, and listing a bucket with that prefix returns only the matching objects.
Keeping data secure and private
Wasserman has seen many misconfigurations that made data public. Her recommendations are direct. Use HTTPS unless you are on your own private network or VPN, since it is not the default. Server-side encryption protects stored data, but for privacy she points to client-side encryption, where the SDK encrypts data before it reaches the network, using your own keys or a key management system.
On access control, she is emphatic: "Please don't use public buckets." (16:37) She notes that leaks from public buckets happen every few months and that AWS can now warn about them. Fine-grained access controls, bucket policies and user policies allow sharing between users, read-only anonymous access or restrictions by IP address. "Please use them and try to use the most limited access you can." (17:00)
Use the APIs, not a file system layer
Her closing message ties the talk together. Object storage is designed for cloud scale, and its unique features help teams build better cloud applications, but only if they use them. "But to use the feature, please use the storage, the APIs and not use it as a file system as that will, won't allow you to use features like versioning or lifecycle and so on." (17:57) For private clouds, she recommends Ceph RADOS Gateway for object storage.
Watch the talk
"Storing your data in the cloud: doing it right" by Orit Wasserman, 19 minutes: https://www.womentech.net/video/storing-your-data-in-cloud-doing-it-right
About the speaker
Orit Wasserman is OpenShift Container Storage Architect at Red Hat, focusing on storage for containers, hybrid cloud, multi cloud and edge. An experienced software engineer passionate about open source and infrastructure, she previously worked at Red Hat on Ceph object storage (Ceph Rados Gateway) and on live migration for KVM/QEMU, and developed nested virtualization for KVM at IBM Research Labs.
Questions this talk answers
What makes object storage different from block and file storage?
When should multipart upload be used, and when not?
How do you stop versioning and failed uploads from wasting storage?
Can object storage have folders?
How do you keep data in cloud object storage secure and private?
Drafted with AI from the talk transcript. Editor review pending.
Resources mentioned
Ceph RADOS Gateway (recommended for private cloud object storage)
Cite this talk
Orit Wasserman. "Storing your data in the cloud: doing it right." WomenTech Network, 2021. https://www.womentech.net/video/storing-your-data-in-cloud-doing-it-right