Originally published by Bernard Marr on LinkedIn: Big Data From Mars - Real Time Analytics 150 Miles From Earth

The team in control of NASA’s Mars Rover spacecraft now have Big Data driven analytical engines at their fingertips.

The same open source Elastic Search technology used by the likes of Netflix and Goldman Sachs is now being put to use planning the actions of the Curiosity rover, which landed on the red planet in 2012. And the next mission, due to launch in 2020 with the primary mission of finding evidence of ancient life on the planet, is being built around the technology from the ground up.

NASA’s Jet Propulsion Labs, which runs the day-to-day mission planning, has rebuilt its analytics systems around ElasticSearch which now processes all of the data transmitted from the Rover during its four daily scheduled uploads.

These uploads cover tens of thousands of data points, including every reading taken by Curiosity’s onboard sensors such as temperature on the Martian surface, atmospheric composition, and precise data on the rover’s equipment, tools and actions.

All of Curiosity’s operations are planned a day in advance based on data received the previous day. This move to a real time analytics vastly speeds up the time in which decisions can be taken by mission control.

ElasticSearch – which recently achieved its 50 millionth download – means that patterns and anomalies in the datasets can be spotted far more quickly. Correlations which could provide mission-critical insights are more likely to become apparent, leading to a greater rate of scientific discovery and less danger of malfunction or failure.

NASA JPL data scientist Dan Isla told me “This has been very transformational from an operational point of view. We can get very quick insights over multiple days’ worth of data, or even the entire mission, in just a few seconds, without delays. We can actually ask questions and get answers quicker than we can think up new questions.”

One application is anomaly resolutions. When it appears that there is a problem with the spacecraft, precise details of its operations can immediately be analyzed to find out when was the last time this situation occurred, and what other elements were in play at the time.

The system is also used for the vital function of managing power generated by the rover’s radioisotope thermoelectric and solar power generators. The trickle charge system only makes around 100 watts of power available to the craft and its instruments at any one time consistently. This means that power management must be carefully planned in advance. As Isla explains, “using ElasticSearch we can budget this power much more effectively, we can look at the data and see ‘ok, this is what we used yesterday, and this is the state of the batteries, and this is our plan coming up…’ we can build a power model very quickly.