We Have Created a Data Lake for Storing Operational Data of Masaryk University

The team headed by Jindřich Zechmeister managed to create a functional Data Lake, which is used to store operational records (logs) for IT service administrators at Masaryk University. Thanks to a successful project, they do not have to deal with their own storage and significantly save time and resource capacity.

The Data Lake was created in cooperation with the CESNET Development Fund, which also financed the project for the most part. Thanks to this, a complete solution was created for the collection of logs on stations and servers, their storage and further analysis. It uses S3 CESNET storage and components from Elasticsearch B.V. The Data Lake will thus make it possible to process logs faster and react to events, but also save administrators the trouble of collecting logs and their long-term retention.

And that's not all. “We've created an analytics layer on top of the data in the Data Lake to make it easier for service managers to work with the data. It will help them monitor data in real time and evaluate it immediately, but in the event of incidents they can analyze it retrospectively and look for answers to their questions," project manager Jindřich Zechmeister explains other advantages of the chosen solution. The newly created Data Lake will also be included in the IT MUNI service catalog, where other potential applicants for centralized data storage can find it.

The results of the project in the form of created instructions can be viewed in the git repository at this address.

• The final report of the project located on the website of the CESNET Development Fund​​