The process needed to transform Google Earth Enterprise, or GEE, from a closed-source commercial offering to an open-source, publicly available suite of applications is pretty complex and still ongoing. GEE was in development first at Keyhole, then at Google for 12 years. During that time a lot changed with the code, but at the same time, much of it stayed the same. The saying goes “if it ain’t broke, don’t fix it.” Unfortunately, many things that worked seamlessly in a proprietary, closed-source environment won’t work for an open-source project.
In this series we’ll explore what had to change and what had to be left behind in order to get this code into the hands of the open source and geospatial communities. In this post, part 1, we’ll focus on compiler differences and challenges involved with building on multiple Linux distributions.
During the time that GEE was developed at Google, the engineering team used a custom cross-compiler, based on GCC 4.4. The software and all the required libraries for many Linux distributions were built on a single, controlled Linux environment. Versions of compilers, libraries, and even paths, are not guaranteed to be the same across the many distributions of Linux used in an open source development environment. The code will need to compile on popular Linux distributions with modern compilers. We decided that Ubuntu 14.04 and Red Hat Enterprise Linux (RHEL) 7 would be our initial targets, and we would expand the supported list as needed. These distributions use GCC 4.8 as their default compiler.
As anyone who has upgraded their compiler while working on C++ source code likely knows, as compilers are upgraded, they tend to become more strict on syntax standards. This is a great trend for improving the quality of your source code. However, older source code repositories may contain many instances of source code not conforming to the latest standards. This is compounded by the fact that we have our compiler configuration set so that warnings are treated as errors and will halt the build process. This is also a good practice to improve source code quality. When syntax errors are encountered during the upgrade, the solution is typically straightforward. The code must be updated to conform to the specification. If a new warning is encountered, on the other hand, an engineer would have to determine the level of effort required to either change the source code, which may require altering dozens of source files, which may introduce new bugs to the software, or simply configuring the build system to ignore the warning. Due to time constraints, sometimes the former is chosen, and sometimes the latter. Solving these issues is typically a manual process of compiling and testing on each target distribution and solving issues as they appear.
Occasionally, new bugs involving run-time errors can result from changes in the C++ standard, and corresponding compiler behavior, such as the order of calling class constructors to initialize variables. In-depth explanation of these problems and corresponding examples would warrant a blog post of their own.
Additionally, libraries may be located in different locations between different distributions. For example, gtest, our unit testing framework, was especially interesting. In Debian-based distributions the development package for gtest provides the source code. This is the same as what you’d get if you download the source for the library from its home page. This matched the configuration expected in the previous, closed-source build environment. However, Red Hat distributions include the library as a pre-compiled, shared library. This difference required changes to our build system, SCons. We had to detect which style was available.
Additionally, package names and organization of packages are different between the Debian and Red Hat environments. To handle this, we have different configuration requirements between the two systems. The details for this, of course, are included in our Build Instructions page on the Earth Enterprise GitHub Wiki.
Still today, work is being performed to expand supported distributions, including CentOS 7, RHEL 6, and CentOS 6. Upon initial investigation, the differences between these environments will provide different challenges in the way their build systems work.
In our next topic in the series, we’ll take a deeper look at the various libraries that had to be upgraded, removed, or replaced in order to bring the code to the open source world.