Data Liberation

June 02, 2020



“Freedom!” – William Wallace, Braveheart

Data liberation: It’s the epic struggle of our industry today. Here at Oliasoft, we like to believe that if William “Braveheart” Wallace were alive in 2020, he’d be part of our mission to liberate data for the oil & gas industry. But what is data liberation? And where is there a struggle?

Data liberation can be defined as the process of freeing data from the systems that were used to create the data. Historically, computer data was stored in binary formats inside data files that could only be read and written by the specialized software that created the data. The data itself was mostly considered “black magic”. If you didn’t have that piece of software, the data mostly looked like random garbage. This was nice for the software vendors, but typically less nice for the customers, who needed a valid license at all times to access their own data.

Skjermbilde 2020-07-09 kl. 11.09.59

As more and more software programs came into use, the need to exchange more data more easily between different systems became paramount. New software vendors typically wanted to help their customers support their old data files. The customers themselves wanted the freedom to switch vendors for various reasons (vendors going out of business, adopting better products, and otherwise avoiding the pitfalls of lock-in). Over time, software converters were written to let one application read another application’s data files. This was still somewhat inefficient, so sometimes rival providers of a particular software application would agree on common interchange formats.

Spreadsheet software offers a useful example that illustrates the power of data liberation. Spreadsheet software providers standardized on a file format known as CSV (comma separated values) which made it possible to move data from one system to another.

Let’s assume there were N providers of spreadsheet software. Before CSV, if a vendor wanted to support the proprietary file formats of the other vendors, the vendor had to write N-1 “converters”. But each vendor had to do that for its software. That meant the industry paid for N x N - 1 integrations in total.

If there were 6 vendors, that meant 6 x (6 – 1) or 30 integrations total. If there were 10 vendors, that meant 10 x (10 – 1) or 90 integrations total! And that math assumes that all the binary formats could actually be decoded enough to make it possible to read the data at all. And as for writing data – well, that was even more difficult, and most vendors were content if their converter was good enough to read something from an old system into a new system.

This is why the standardized CSV file format was so revolutionary. Once all the vendors supported reading from and writing to CSV, data could be moved from and to any system without needing converters. Instead of writing N-1 integrations, each vendor only had to write one (CSV). CSV had another very important feature: Humans could ALSO read and write CSV. Even if every vendor of spreadsheet software disappeared all at once, the data would not be lost. Somebody would write new software from scratch which could read and write those CSV files just as before.

Officer workers today take for granted that their spreadsheet software can read and write the same data as everyone else’s spreadsheet software. It’s only the nerdy people like me who remember what data liberation did for productivity. But there are very important lessons to be learned from it! Let’s consider those lessons from the perspective of a user of well design/planning/construction software.

A lot of the software used in the oil & gas industry today was developed in the same era as the first spreadsheet applications. And the programs worked the same way: Users would punch data into the software, and the software would hide that data inside proprietary binary files which were only readable by the same software. But the market forces were less competitive, so the outcome was worse. It was a market with very few players. A small number of companies were able to acquire most, if not all, of the software vendors servicing the industry. The acquiring companies were not even software companies – they were oil service companies. Software wasn’t their focus, and data liberation wasn’t even a consideration.

The oil service companies simply saw no need to open or standardize their data formats. Since they owned more or less all the software, they saw no need to interface with anybody else. Often they didn’t even interface with themselves. An oil service company might acquire three different software packages X, Y, and Z from three different vendors and then expect the users to manually re-enter the data into each of the different packages. This sort of inefficiency lasted for decades, and years passed with little progress towards data liberation.

Finally, with the advent of LANs (local area networks) and WANs (Internet etc.), customers began demanding the ability to share data between software packages. In response, the oil service companies did implement some features to make their customer’s lives easier. To facilitate data sharing between software packages, they converted from file-based data to data hosted and shared on database servers accessible via LANs or WANs. They wrote their own custom adapters, where some data could be shared between the proprietary software packages X, Y and that they controlled. But these were very small steps. Most of the data was STILL just unreadable, only now the unreadable data was stored inside a database instead of a file.

These small steps served the oil service “software” companies well for a while. But with a tiny number of enormously powerful companies still controlling most of the critical software being used by the industry, customers were still suffering from vendor and data lock-in that was stronger than ever.By the early 2000s the internet revolution was in full swing. Software systems were now being moved from closed LANs and WANs onto the open internet, accessible to anyone, available from anywhere. Customers decided they no longer wanted to have their own IT departments running their internal systems – they just wanted to use the systems. The IT industry had an answer: Citrix.

Citrix is a solution (one of many) that essentially “extends” the screen and keyboard of a computer across the internet. Citrix allows a company to run old proprietary software on local computers (as it did before), but with the software remotely controllable from a keyboard and screen on the far side of the internet. Based on this primitive solution, virtually every software vendors declared itself cloud- and internet-ready. While Citrix worked… kind of… IT industry professionals knew better. The whole internet revolution that started in the early 1990s on university computer networks was built on client/server protocols. These protocols powered the internet, starting with email and then later web browsing. To truly be cloud- and internet-ready, software had to be developed natively on these same protocols.

Developing software for the web turned out to be a lot easier than developing separate software for Windows, Macs and UNIX computers respectively. The tools and languages quickly became standardized, and software written for the web could run anywhere, accessible to all, both local and remote users. With a few exceptions where locally running software remained more suited to running on local computers, most software soon moved into the cloud, and became SAAS (Software as a Service). The trend has only accelerated as computers became more powerful and the internet became more ubiquitous.

Now, back to the oil industry software giants. What’s the status quo? Despite the progress in SAAS that has benefited most industries, Citrix is still – by far – the most commonly used solution in oil & gas. Worse, the actual software that is run with Citrix is still from the 1990s. It’s proprietary Windows-native software, storing a mix of relational data and binary files on local database servers. Customers are demanding new cloud-native solutions. But the existing oil service companies face a classical innovator’s dilemma. When and if the oil industry software giants start opening up, they risk losing business to new entrants in the market. Up until recently, nobody has been able to challenge the giants. And for the same reasons, the giants haven’t seen any reason to open up.

But customers know how much other industries have benefited from data liberation, and they are unceasing in their demands for free access to their own data

In addition to open read- and writable- data, they require the ability to store the data themselves, mostly in their own enterprise cloud solutions. They expect to be able to run (or even write) software that can take any of their own data as input and deliver the output as open accessible data as well. They want APIs which can fetch data, trigger calculations, and retrieve data without human interference. Only then can they drive their own business processes forward, improving efficiency with digitalization and automation, even all the way to fully autonomous operations.

The oil service industry giants are promising solutions, even showcasing new solutions, claiming they will address all these customer demands. But the giants are not actually DELIVERING solutions. Partly that’s because they don’t want to. It’s not in the existing giants’ interests to deliver what the customers want and need, not until they absolutely have to. And that’s the classical innovator's dilemma; by then it is probably too late.

It’s not even clear that the oil software giants could deliver solutions even if they wanted to. They have acquired software vendors, but they have never been experts in software development. Modern software solutions and architectures are very different from the products they bought up in the past.

From working both with the oil software giants directly and with common customers, we know that the giants are making small steps in the right direction. But most of their “new solutions” are nothing more than simple extensions of existing product suites – typically products that can deliver some output from their legacy systems into new fancy dashboards. Instead of redesigning their core systems to modern standards, they are making add-on systems that simplify access to some output data. But they are not committing to data liberation, where both their own input data and output data is totally open. And they are not committing to a strategy that will allow human labor to be replaced with software and automated processes for engineering.

Inspired by our industry partners, we have borrowed the motto "Data liberation" - because this is a common battle for anyone who wants to digitize the oil industry. Read more about how Cognite Data Fusion (CDF) releases data from siloed source systems here.

At Oliasoft, we have done things differently. Oliasoft was built on the ever-growing expectancy gap between suppliers and vendors in this space. Oliasoft is a software company, a native child of the open-data open-protocol modern world of cloud/SAAS computing. Data liberation is in our genes. We have a complete platform for well planning/construction/engineering, with solutions for trajectory design with error modelling, casing- and tubing design, T&D, hydraulics, blowout and kill, and more. All our offerings are available either through either machine interfaces (APIs), or rich cloud- and browser based SAAS, offering interactive 2D and 3D graphics for humans. And most importantly, all of our customer’s data is available to, and owned by, the customers themselves.

This effectively means Oliasoft’s solutions are future-proofed. We support completely manual workflows with humans (like those in use today). But we also support 100% automated workflows through our APIs, or any combination in between. We leave that decision (and process) to our customers. Our job is to liberate them to choose how they want to do business. And that’s the real outcome of data liberation.

Curious about what “in the cloud” actually means when it comes to well planning?

Our CTO wants to make sure that you know what “the cloud” is, and explains why Oliasoft´s cloud is able to shape the future of well design. Read the article here.