March 20

Starting to Piece Azure Data Factory Together

After a couple of weeks of playing around in Azure, largely focusing on Azure Data Factory (ADF), Data Lakes, Synapse and other similar offerings from Microsoft, I feel that it is all starting to fall into place and make sense. Admittedly, the vague level of information provided did not help on areas to focus on but in the end, general curiousity has started to prevail.

As part of my future work building various pipelines within ADF, I have started to make notes on similarities and differences between existing tools that is used at work and Microsoft’s offering as I am aware that in the future I will more then likely be writing internal documentation (oh the joy!). Whilst doing this deep dive, I have come to love how linear and clean the different activities are laid out within the UI, but behind the clean UI and fairly simple design, is quite a powerhouse of ELTL tooling. This power is increased greatly once you realise how easy it is to integrate different services like Function and Logic apps.

Over the next few months, I am going to start to piece together and post here about solutions to problems that I come across whilst building Pipelines, as well as any useful tools and apps that I build along the way as it will be a long but fruitful journey.

March 8

Starting Down the Azure Road

Microsoft Azure Logo

Over the past few months at work, we have been hearing about the possibility that we will be moving our ETL package away from Alteryx to a Microsoft based tool. For the team that I work in, this would be a significant shift as a good 85% of our workload is made easier through the use of Alteryx with a library of Macros and flows that have been built up over the past four or five years. Due to the lack of clarity provided by the working group that was setup to assess the migration, I decided to take a bit of a look at Microsoft’s ETL offerings as I knew we would either be left on Alteryx (potentially with a heavily limited number of licenses) or thanks to some research, we would either be left with SQL Server Integration Services or Azure based tools like Azure Data Factory.

As time went on, it started to become more and more likely that Alteryx was going to be replaced due to the cost of licensing. Fortunately I had been working in SSIS by replicating several of our key data ETL flows to make sure that all the functionality we needed was available. I was hoping for SSIS partially as it meant I could work with Visual Studio at work (I am not a developer professionally, just hobby) but alas it was not meant to be, last week it was confirmed that Azure tools would be the solution that we are moving to in the near future. Unfortunately, nothing was clear around how much access we will have to Azure, what functionality we can use without having to raise support tickets, this makes the difference in how usable it will be for our team, especially in the initial replication process.

Until questions are answered and we get given access to actually work in Azure, I thought that it would be a good time to redeem the free 30 day Azure trial and start to build some workflows with dummy data that could loosely represent the core functionality of what will be needed for work. It is a bit of a win-win situation, I get to learn how to work with Azure Data Factory, Logic Apps, Databricks etc, there is no cost to me (for 30 days) and it gives me a bit of a head start at work when it comes to the understanding of how everything works. On another note, I thought that if I document my learning throughout my time with Azure, it would provide a good reason to reboot the blog and provide reference points for anyone else who wishes to start their time with Azure.

May 24

Find a bug and fix multiple bugs

Earlier in the year, whilst working on Omniworlds RPG, I came across an issue on the Discord Bot where it was constantly throwing an error of invalid item < at position 0 when trying to access a list of NPCs. On top of this, when calling the same endpoint via Insomniac, I also found that the API was limiting the response to 10 entries for some unknown reason.

After an initial search and attempt at rectifying the error on NodeJS, I decided to give up on that side and move on to the response issue and after a decent search, it was mentioned by a viewer that the potential NodeJS error was to do with json_encode() failing silently in PHP. I applied some error handling on that function and amazingly it was throwing the following error: Malformed UTF-8 characters, possibly incorrectly encoded!

I went back further through the code to trace back the errors cause and added ;charset=utf8” to the PDO connection string for the database calls and amazingly both issues were solved! The API was no longer limiting responses to 10 entries and the NodeJS invalid JSON error was suddenly gone.¬† It goes to show that sometimes, there can be a silent bug in your code that you are unaware of that has a knock on effect with other parts of your code!

 

December 2

Triggering Organisation

Since the hassle that was working with the follower event handler, I decided to move onto the raid and host event triggers despite leaving issues in the follower event. After the challenges I had just faced, I thought I had enough in the bank to be able to easily handle the upcoming code challenge. after about 15 minutes of reading and typing, I ended up with the following which compiled fine, the issue was it did not want to trigger.

private void OnRaidNotification(object sender, OnRaidNotificationArgs e)
{
form.WriteChat(e.RaidNotificaiton.DisplayName + ” raided!!!!!!!!”);
}

private void OnBeingHosted(object sender, OnBeingHostedArgs e)
{
form.WriteChat(e.BeingHostedNotification.Channel + ” hosted with ” + e.BeingHostedNotification.Viewers + ” viewers”);
client.SendMessage(Resources.channel_name,e.BeingHostedNotification.HostedByChannel);
form.AddEvent(e.BeingHostedNotification.Channel + ” hosted (” + e.BeingHostedNotification.Viewers + “)”);
form.AddEvent(e.BeingHostedNotification.HostedByChannel + ” hosted (” + e.BeingHostedNotification.Viewers + “)”);
}

After testing a few times to see if something is wrong (as you can see with the e.BeingHostedNotification.Channel AND e.BeingHostedNotification.HostedByChannel) to see which would work, it was time to throw in breakpoints. At least this way I would be able to tell if the issue lay with my code of with TwitchLib/Twitch… Recompiled the bot, dropped the host and… The breakpoint never triggered, I even waited a couple of minutes in case it was delayed, it was not. So there is a very likely chance that the issue lies with TwitchLib as neither trigger was fired, I decided to pop a message into the TwitchLib discord but after waiting two days, I still have no response other then “it should work”.

Last night was the first advent stream so I thought I would build the bot and give it a live test and all in all it performed as well as I had expected it to. Certain methods weren’t working, others did and it highlighted a few niggles and bugs that need to be worked on which after the stream caused me to get organised. I ended up creating a feature and bug/niggle tracker, yes I could have used Trello and/or GitHub but I felt like running it this way for now and you can see the results here:

 

Tomorrow I am on my first day back to work after my holiday so development and posts will slow down but it has been a good bit of progress so far!

November 25

Working with a new Language and TwitchLib API

This week I decided it was finally time to delve into the world of C# properly, I have previously dabbled in the language but never worked on it solidly for a prolonged period of time so that I would be able to learn the system properly.

One project that I have worked on before which would help ease the transition is working on a Twitch Bot. My most recent PHP project on stream was a bot so the general logic is going to be a breeze. After all, Twitch is just an IRC protocol which is easy enough to work with and manipulate. Despite the ease, I had found a library called TwitchLib which has been made by a decent team of C# developers.

Whilst I won’t be able to stream the development of the bot due to the system limitations, I am going to use this blog to show the progress of the development as well as any challenges that I come across in the development, especially as I haven’t found any real tutorials or resources that could help.