Podcast: Quad Core
Faster, Hotter, Scalier
Jill
David
Jason
Joyent Weblog
Strongspace meet Thumpers, Thumpers meet Strongspace
A year and a half ago (July of 2005) we launched Strongspace as a means to securely upload, store and share files in a closed system that is only accessible via encrypted channels.
It quickly became very popular as secure backup storage for a number of individuals, businesses, non-profit organizations and schools.
Then a year ago, we migrated it from FreeBSD+UFS systems to Solaris Nevada + ZFS and it become the first and largest ZFS file system publicly put into production outside of Sun Microsystems.
That was cool and quite a bit of fun. At the very least we got to know the ZFS team at Sun well, and Jeff Bonwick (architect of ZFS and now the CTO of storage of Sun) was even posting about things on the forum.
At the time the hardware was EMC Clariions that were fiber attached to Dell 2850s. This served us fine but it was a solution that was still keeping our per GB per month pricing higher than we would like. It was also a bit more complicated then I now like things to be: we needed to operate and maintain a separate fiber network, and each server needed fiber HBAs.
So over the last couple of weeks we’ve been transferring all the data over to containers that live on Sun x4500 nodes, and from our San Diego, CA facility to the one in Emeryville, CA.
Strongspace, like Bingo, is a perfect application for these servers because the processor and RAM needs are much less than the need for a lot of storage, and what we were able to do is get rid of any networking between “storage” and “servers” (they’re now one and the same). It’s also greatly simplifies things when you have the same operating system throughout and don’t have to involve other proprietary storage management tools.
One thing you’ll notice is that each Strongspace node will be significantly faster (especially when backing up from servers and services on our own network), and the web part of the application itself runs faster as a result of there being so many disc spindles in the mix as well.
I’m simply excited to have the pricing for Strongspace storage come down to Bingo levels per GiB and for us to be able to start bundling it along with hosting and other applications in Joyent Core
Commenting is closed for this article.
Wow, the new Strongspace is way faster. I haven’t been backing up since the ssh keys changed, and I was pleasantly surprised how fast my rsync backup was running through.
— Ralph von der Heyden 564 days ago #Thanks!
“I’m simply excited to have the pricing for Strongspace storage come down to Bingo levels per GiB and for us to be able to start bundling it along with hosting and other applications in Joyent Core”
Does this mean we can get 100 of GiB of strongspace for 199 dollar a year?
— Reinier 564 days ago #Could you explain what the difference is between Bingo and Strongspace? I feel like they’re the same service except one is SFTP and one is WebDAV. Why not just combine them and offer both protocols?
— Garrett Murray 564 days ago #I have a dumb question about Thumpers and reliability. Not to question it but to ask out of interest if my thinking is valid and, if so, how it’s dealt with.
I assume if 1 drive goes belly up, the Thumper quickly recovers in some way. I don’t know if it brings one of the spares into the pool or if that’s a manual process. But either way, I imagine there’s a period of time while it populates the new drive where you really wouldn’t want a second drive to go right now.
Assuming drives are manufactured to close tolerances, I would think that the time to failure of the 48 drives would be quite tightly clustered around the mean. So, if one drive goes, maybe that’s an outlier. But I’m imagining this like making a batch of popcorn: one, then one, then two, then suddenly half the remainder in no time at all. And clearly you’d want to take some action well before that point. Or am I wrong and even a tight clustering of failures still be days or weeks or months apart?
— andrew 564 days ago #@ Garrett, yes one is SFTP and one is webdav.
They happen to be two largely incompatible ways of accessing files.
But besides that, Strongspace is about encrypted closed storage (space is the commodity, it doesn’t care about bandwidth) and there is no possible way of leaking a file.
Bingo is about ease of use and open delivery, space is just a simple price (like buying a hard drive) and bandwidth is the consumable.
I do realize that they’re seemingly close but they in fact hit two different, largely incompatible areas. If we combined the two, then it would be overkill to encrypt someone’s publicly available podcast while in transit and on disk (strongspace for example will use encrypted ZFS filesystems when it becomes available), and we would (and have gotten) get comments about how space could be “strong” when it’s possible to accidently put a file into a “public” folder and allow the outside world to download it.
— Jason Hoffman 564 days ago #@andrew if you take a look at this post then you’ll see
raidz2 c{5,4,7,6,1,0}t4d0 c{4,7,6,1,0}t0d0 \ {can lose 2; notice how it cuts across controllers}
raidz2 c{5,4,7,6,1,0}t5d0 c{4,7,6,1,0}t1d0 \ {can lose 2; notice how it cuts across controllers}
raidz2 c{5,4,7,6,1,0}t6d0 c{4,7,6,1,0}t2d0 \ {can lose 2; notice how it cuts across controllers}
raidz2 c{5,4,7,6,1,0}t7d0 c{4,7,6,1,0}t3d0 \ {can lose 2; notice how it cuts across controllers}
spare c5t{1,2,3}d0 {three hot spares}
We can lose 8 drives (across 4 double parity groups), but then also have 3 hot spares (and hot does mean that they’ll just take over in case of a failure and resilver).
So of the 46 drives (substract 2 from the 48 that are kept out for boot and boot recovery), we can lose 11 drives before it would be a good idea to head in and replace some drives.
Also there’s no controller cabling or the like to fail because of the design.
Then in the case of containers, add mirroring across thumpers to that … and we’re pretty comfortable with the setup.
— Jason Hoffman 564 days ago #@ Reinier Yep that’s the idea.
— Jason Hoffman 564 days ago #So, when will the Strongspace pricing reflect this wonderful new era? Currently it still lists 100GiB as being $100/monh.
— Matthew Fitzsimmons 564 days ago #That’s a reassuring level of redundancy. Seriously.
I guess tho’ what I was asking was by the time 11 drives have gone, are you right in the busy part of the bell curve? Would you preemptively replace a pack of drives after, say, half a dozen had gone? Or swap a few around between Thumpers to mix it up a little? Or has Sun thought of all this and done some mixing of its own? Or is it all really a non issue because the lease will have expired before even 2 or 3 drives have gone?
Like I said, just asking out of interest.
— andrew 564 days ago #As far as merging the two products go, I don’t really care about encryption as I’d use an encrypted file / image for anything truly sensitive but I’d really like to be able to access my storage using rsync as well as WebDAV. Now I’m wondering whether to go with Bingo now or wait for the Strongspace pricing to adjust.
— Chris Adams 564 days ago #@Chris: Bingo supports rsync. Mount your Bingo disk via WebDAV, then use rsync to do the copies.
— David Young 563 days ago #I tried that rsync method, but it kept re-uploading the same stuff.
— Matthew Fitzsimmons 563 days ago #Do the Joyeurs have a sample rsync command that should work with Bingo?
I probably did something that doesn’t work like preserving ownership or something.
— Matthew Fitzsimmons 563 days ago #Ah…I added -u to the mix to not overwrite newer files. That seems to have done the trick. It seems that it doesn’t retain dates.
— Matthew Fitzsimmons 563 days ago #David – doesn’t that cause it to download a large file again to do a comparison? I prefer running rsync on the remote end to avoid all of the metadata & comparison traffic for syncing a large tree with relatively few changes.
— Chris Adams 562 days ago #