PK's Online Adventure || Technical

Introduction

The dynamic nature of the IT industry requires a commitment to continuous learning and adaptation.

I made the decision to get back into the IT game late 2023. As I am returning to the IT industry after a lengthy hiatus, my priority has been to bridge the gap between my foundational knowledge and the latest technical services and offerings. I therefore embarked on a self-directed journey to explore the latest in Cloud computing and Artificial Intelligence services.

My first activity was to complete the ISC2 CC certification and complete the ISC2 CCSP exam. Both provided a good overview of the state of play, at least from a risk, regulatory and governance perspective. I then attained the AWS Practitioner certification, and recently completed the Finops Practitioner certification from the FinOPs Foundation.

To complement the certifications I've gone for a more hands-on approach to upskilling, completing a series of projects involving as many Cloud, AI and data analysis tools as I can lay my hands on for free. The projects listed below illustrate this journey, showcasing my hands-on approach to acquiring new skills, and applying them in practical scenarios to solve realistic problems.

Cloud Resume Challenge

Project Summary

This project represents the beginning of my re-education and re-aquaintance with Cloud services after a pretty long absence. The project brief is defined in Forrest Brazeal's Cloud Resume Challenge. In essence the challenge requires us to build a simple website using as many cloud-related services as possible. I do believe that getting hands-on with tech is the best way to learn and come up to speed with what’s possible.

Re-educate myself on the latest Cloud concepts and practices (PaaS, serverless, CI/CD,)
Build a thing using as many current Cloud services from a prominent provider as possible
Get exposed to the prevalent coding languages used today (HTML/CSS, Javascript, Python)

Write the HTML/CSS code for the website (based on a project from John Smilga's "HTML&CSS Tutorial and Projects Course")
Domain name registerd with Cloudflare (DNS services only - no proxy) DNSSEC enabled
Azure storage account created and configured for delivery of static web content
Visitor Counter app written in Python, deployed as an Azure Function.
CosmosDB database to store Visitor Counter app data.
Azure Key Vault to store CosmosDB credentials and endpoint data.
Github repository for source code control and code CI/CD.
Visual Studio Code with Azure/Github extensions.

The deployment model for this solution looks something like this: Cloud Resume Challenge Solution Components

The Visitor Counter process flow looks roughly like this: visitorCounter process

Cloud has certainly democratized IT. Its pretty impressive that you can now have a website, app runtime environment, storage, database and secrets management services without having even one server OS to worry about, and all for the princely sum of $0.17/mth from Microsoft!
Resilience, patience and some humility were required to swallow the number of new concepts and technologies that went into this solution. Chipping away at it every day helped to make sure I felt that progress was being made, no matter how small.
Visual Studio Code plugins for Azure, Python and Github made life really easy. I could create and test the Azure Function locally with the assistance of the Azurite emulator, and the Azure plugin translated any local json settings to Azure Application Settings when deploying to Azure itself. Github Actions mean I can synch code and rebuild my entire website with a single command.
Regarding CosmosDB, I want to acknowledge a fellow challenger techbrett for the SQL query solution, as its a much more databasey way of doing things than the Python code I was struggling with.

Weather Forecast Project

Project Summary

To get a 5 day weather forecast these days one can simply enter a prompt in ChatGPT or Copilot....... but where's the fun in that??

Gain further experience with Azure Functions and other Azure services (e.g. Key Vault, storage)
Gain experience in working with APIs, specifically OpenAi
Expose myself to more complex Python and Javascript concepts, specifically passing stuff between front and backend, data handling and security

My static website, hosted in Azure blob storage, providing javascript capabilities
Azure Functions, using Python 3.11
Azure Key Vault
OpenWeatherMaps free API
OpenAI API, using the 4o-mini model
Extract from SimpleMaps WorldCities data, basic v1.76
My OpenAI GPT Plus subscription

The deployment model for this app looks something like this: weatherForecast_components

The end to end process flow looks roughly like this: weatherForecast process

So.... pick a city, pick a voice, be patient and see what comes back!

5 day forecast for in the style of :

My GPT Plus subscription came into its own on this project. It took a lot of wrangling to get the raw data from OpenWeatherMaps into the right format for OpenAI to make use of, and then again to get the OpenAI response presented in something readable, which meant iterating several times back and forward with the Python code. It also gave me options as to how to capture user-supplied variables, passing data between front and backend.
I learnt the value of logging, given the number of iterations I went through with data, variables, API key retrieval etc etc. This Udemy course by Steve Avon provided a great logging template.
A word on using GPT Plus - I do not wish to represent myself as something I'm not at this point in the journey. I haven't suddenly gained a decade of coding experience. But I can explain every line of code committed to Github. At the end of the day, the onus is on the individual - on me - to make sure I am always learning from the GPT, not just leaning on it. I do this by ensuring GPTPlus comments all code, and explains every line to me. I also include various phrases in the prompt such as 'best practice', 'most secure', and I also ask for alternative approaches. It does get it wrong sometimes - its still pushing completion methods that OpenAI themselves have actually deprecated! But - as long as I remember this, and don't do a Michael Cohen or a Channel 9, at a minimum it will fast track my education.

Data Anaysis Project - Reddit, COVID and Gamestop

Project Summary

My initial goal for this project was to determine whether there was any value in trawling through stock-related social media posts, in order to identify or even predict large moves up or down of the mentioned stocks. While this is still the end goal, another story emerged as I started to look at the high-level data, that I thought was worth documenting.

The approach was to plot the number of comments or posts on a social media platform against actual market prices, to see whether there was any correlation between retail investor sentiment expressed through social media, and actual stock or market prices. I would start at the highest level (total posts from a popular social media platform plotted against the S&P500), and then drill down into specific stocks. To continue the theme of exposing myself to the latest Cloud services, I would present this data through Tableau Public.This project is my first move away from infrastructure and towards data.

Expose myself to Cloud-based BI service Tableau Public
Gain experience in extracting data from social media platforms
Determine whether there are any predictive benefits in correlating social media data with stock market price data

Archive of all stock related subreddits from Academic Torrents.
Python scripts supplied by Watchful1
S&P500 historical data sourced from CME Group
Gamestop historical data source from Yahoo Finance
Tableau Public account
Microsoft Excel

The final Tableau Viz (also available online at Tableau Public): Reddit Viz

Data can be tricky to find - sometimes too expensive (Elon's $100/mth charge for access to the X API) or with restricted access (the usage policy on Reddit). Sometimes a search can yield amazing results - who knew someone was keeping an archive of ALL (yes ALL - even the dodgy ones!) subreddits!
Sometimes attempting to answer one question actually leads to a better story. In this case, I wasn't expecting the data to show the link between COVID and the formation of a mob so clearly. Fear, layoffs, furloughs, capital destruction, lockdowns and time on our hands proved the perfect honeypot for people wanting to protect their wealth, find alternative sources of income, and probably most importantly, feel connected to something. Add to that a burgeoning Robin Hood investor mentality, a desire to "stick it to the man", and we now have a very large number of emotive speculators in one spot needing an opportunity to express themselves. And didn't they do just that!

Data Anaysis Project - ABS National Study of Mental Health and Wellbeing 2020-2022

Project Summary

Following on from the Gamestop exercise, this project focused on exploring the presentation capabilities of Tableau.

The data set provided by ABS contains a large array of data sliced and diced several ways. However, it is also delivered in what you might call a final state - tables that are designed to be cut and pasted into a presentation, rather than raw data with well defined fields that can be uploaded and manipulated in a platform like Tableau.

So, with much wailing and gnashing of teeth I reverse engineered a more appropriately formatted data file with properly defined fields that could then be translated by Tableau into the graphs you see here.

The Tableau Viz is too large to show here, but can be viewed online at Tableau Public

The joy of data prep :-) If I were to do this again I would register myself with the ABS under the guise of whatever commercial operation I was representing, and gain direct access to the source data. My data file bore little resemblance to the ones made aailable by ABS. The numbers are unchanged, but I can't programmatically prove data provenance back to the source.
Although Tableau Public allows you to create feature rich presentations, all the cool data management stuff is unfortunately not available.
Whatever your views are on the current societal conflicts around gender and sexuality, the fact remains that there are people hurting. Specifically, and sadly, its the people at the centre of these battles (young women, LGBTQI+). I hope this conversation can be detached from the partisan grip of social media and be approached by humans with greater empathy and care towards each other regardless of the label.

The data was sourced from the ABS National Study of Mental Health and Wellbeing 2020-2022.

Technical Stuff

On This Page:

Introduction

The dynamic nature of the IT industry requires a commitment to continuous learning and adaptation.

Cloud Resume Challenge

Project Summary

Weather Forecast Project

Project Summary

Data Anaysis Project - Reddit, COVID and Gamestop

Project Summary

Data Anaysis Project - ABS National Study of Mental Health and Wellbeing 2020-2022

Project Summary