Rocode >> Contracting >> StockPrice

Using a Ubuntu Linux Virtual Machine on NeCTAR.


Linux Horse Race

A NeCTAR virtual machine running Ubuntu Linux 12.04 LTS was used for a horse race.

Ubuntu VM on NeCTAR

Ubuntu VM on NeCTAR

The virtual machine had an operating system disc, a scratch disc and an ephemeral disc. Finally it had 920GB of ram.

Ubuntu Linux Desktop with Disc and Memory details

Ubuntu Linux Desktop with Disc and Memory details


A bash script was written to drive MySQL through a test of loading data, cleaning the data and then running a sequence of queries on the scratch disk, the ephemeral disc and in memory.

Notice the --local_infile=1 as needed with MySQL on Ubuntu!

Details of this command option may be found on stackoverflow:
http://stackoverflow.com/questions/10762239/mysql-enable-load-data-local-infile

Bash Script

Bash Script

A logger was written, using a bash script, to capture each second the memory usage of mysqld, the disc usage on the scratch disc and the disc usage on the ephemeral disc.

It was adapted from the logger detailed on stackoverflow:
http://stackoverflow.com/questions/7998302/graphing-a-processs-memory-usage

Logging Script

Logging Script

The logger output its results to a file in the /tmp folder. These results were downloaded and put in to Excel.

Logger Output

Logger Output


Using the following timestamplog table, the progress of the sql scripts are logged and maybe reviewed afterwards.

It was adapted from the time stamp log table detailed on stackoverflow:
http://stackoverflow.com/questions/3488536/mysql-create-time-and-update-time-timestamp

Timestamp log table

Timestamp log table

This is a screen shot of what the sql looks like that is used in the testing of the scratch and ephemeral discs.

Example SQL to add records to the timestamp log

Example SQL to add records to the timestamp log

This is a screen shot of the records generated by the above sql.

Example timestamp log entires for the above SQL

Example timestamp log entires for the above SQL


The final step was to run the race between the scratch disc, the ephemeral disc and memory.

This is the disc usage on the scratch disc after running the horse race.

Scratch Disc Results

Scratch Disc Results

The logger output was cut using the timestamplog table details and loaded in to Excel. 3 worksheets were created being scratch, persistent and ram. A duration key column was added that contained the offset from the test start for each entry.

Scratch Log Details in Excel

Scratch Log Details in Excel

A summary of results was generated combining all three logger outputs. The spreadsheets formulas used are shown here. These resulting columns were graphed as a chart.

Horserace Results Worksheet

Horserace Results Worksheet

The horserace results chart shows the scratch disc usage, the persistent disc usage and ram usage on a second by second basis.

Horserace Results Chart

Horserace Results Chart

The lines are synchronised at the starting second of each individual test script. The chart shows that scratch and persistent usage were almost identical.

As the logger was synchronised to the nearest second, there is a slight phase shift with all three lines.