With over 500 million registered users across more than 180 countries, file-hosting and collaboration service Dropbox is an industry leader in cloud storage, file synchronisation, personal cloud and client software.
A core component of the San Francisco-based company’s success lies in its in-house storage system, dubbed ‘Magic Pocket’.
Housing several exabytes of data (essentially trillions of megabytes), its genesis stemmed from the company’s desire to reduce its dependence on other external parties for storage and ultimately provide a better user experience.
The person charged with overseeing the software project through to completion was Dropbox’s Principal Engineer, James Cowling.
“I got really lucky and was able to play a key role in some once-in-a-lifetime projects,” says James, who graduated with a Bachelor of Computer Science and Technology (Advanced) from the School of Information Technologies in 2005.
“I’ve worked on a variety of large projects spanning storage systems, databases and filesystems but my largest project has been leading the design and implementation of ‘Magic Pocket’.
“It was a very demanding project, but a tremendous opportunity and a really gratifying experience to work closely with people I deeply respect.”
Launching publicly in 2016, the project required James and his team to build a system geared to Dropbox’s specific needs that could be upscaled as required, and to architect the data migration.
Crucial to the design and build phase included the design of a new storage software system and storage hardware based on shingled magnetic recording (SMR) technology – a dense data recording technique in which one track on a hard drive is partly overwritten by the next, similar to the position of shingles overlapping on a roof.
Dubbed ‘Diskotech’, the creation of a compact unit of storage capable of holding a petabyte (PB) of data (one quadrillion bytes) was also instrumental to the project and measured slightly longer than a standard rackmount server tray yet only 4U (18 centimetres) in a rack.
Dedicated physical storage centres were also built to house a mass-scale version of what James and his team had created, followed-up by a rigorous testing of all components.
The crowning achievement of the two-and-a-half-year project though was the successful migration of data across private lines and the internet – all 500PB of it.
“We not only delivered on time but were also able to achieve a significant technical undertaking without any major service disruptions or any loss of data,” says James.
The University of Sydney alumnus had been at Dropbox for less than a year when initially tasked lead that that ‘once-in-a-lifetime’ project.
Prior to joining the technology firm, James was undertaking his PhD in Computer Science and Engineering at Massachusetts Institute of Technology (MIT), specialising in distributed systems and consensus protocols.
His doctorate was to be the final piece of the puzzle before he set out on his originally intended career path – one he ultimately did not pursue.
“My original aspiration was to become a professor and move back to Australia where I would hopefully have my own research group one day,” says James.
“This had been my intention right up towards the end of my PhD program and my seven years at MIT. However, as I got closer to my graduation, I couldn’t shake the desire to get out and build things.
“Dropbox had a lot going for it as a company and felt like the smallest place I could find that had very large engineering challenges.”
Despite this change of heart, James notes that his success abroad in part stems back to his experiences while studying within the Faculty of Engineering and Information Technologies.
“My computer science degree served as really great preparation for my further studies abroad,” says James.
“I received an exchange scholarship while at the University of Sydney which encouraged me to leave my comfort zone and spend a semester aboard studying in the United States. I would never be where I am now if it hadn’t been forced to leave the nest and discover opportunities I hadn’t previously entertained.
“My Honours year at the University of Sydney also gave me a taste of research and encouraged me to pursue a PhD after graduation. I remember working in the computer labs until sunrise on countless occasions – but I had a lot of fun in the process.
“I also had some really supportive professors who served as personal mentors and inspired me to pursue further education in the field.”