March 20

Starting to Piece Azure Data Factory Together

After a couple of weeks of playing around in Azure, largely focusing on Azure Data Factory (ADF), Data Lakes, Synapse and other similar offerings from Microsoft, I feel that it is all starting to fall into place and make sense. Admittedly, the vague level of information provided did not help on areas to focus on but in the end, general curiousity has started to prevail.

As part of my future work building various pipelines within ADF, I have started to make notes on similarities and differences between existing tools that is used at work and Microsoft’s offering as I am aware that in the future I will more then likely be writing internal documentation (oh the joy!). Whilst doing this deep dive, I have come to love how linear and clean the different activities are laid out within the UI, but behind the clean UI and fairly simple design, is quite a powerhouse of ELTL tooling. This power is increased greatly once you realise how easy it is to integrate different services like Function and Logic apps.

Over the next few months, I am going to start to piece together and post here about solutions to problems that I come across whilst building Pipelines, as well as any useful tools and apps that I build along the way as it will be a long but fruitful journey.

Category: Azure, Development | LEAVE A COMMENT
March 8

Starting Down the Azure Road

Microsoft Azure Logo

Over the past few months at work, we have been hearing about the possibility that we will be moving our ETL package away from Alteryx to a Microsoft based tool. For the team that I work in, this would be a significant shift as a good 85% of our workload is made easier through the use of Alteryx with a library of Macros and flows that have been built up over the past four or five years. Due to the lack of clarity provided by the working group that was setup to assess the migration, I decided to take a bit of a look at Microsoft’s ETL offerings as I knew we would either be left on Alteryx (potentially with a heavily limited number of licenses) or thanks to some research, we would either be left with SQL Server Integration Services or Azure based tools like Azure Data Factory.

As time went on, it started to become more and more likely that Alteryx was going to be replaced due to the cost of licensing. Fortunately I had been working in SSIS by replicating several of our key data ETL flows to make sure that all the functionality we needed was available. I was hoping for SSIS partially as it meant I could work with Visual Studio at work (I am not a developer professionally, just hobby) but alas it was not meant to be, last week it was confirmed that Azure tools would be the solution that we are moving to in the near future. Unfortunately, nothing was clear around how much access we will have to Azure, what functionality we can use without having to raise support tickets, this makes the difference in how usable it will be for our team, especially in the initial replication process.

Until questions are answered and we get given access to actually work in Azure, I thought that it would be a good time to redeem the free 30 day Azure trial and start to build some workflows with dummy data that could loosely represent the core functionality of what will be needed for work. It is a bit of a win-win situation, I get to learn how to work with Azure Data Factory, Logic Apps, Databricks etc, there is no cost to me (for 30 days) and it gives me a bit of a head start at work when it comes to the understanding of how everything works. On another note, I thought that if I document my learning throughout my time with Azure, it would provide a good reason to reboot the blog and provide reference points for anyone else who wishes to start their time with Azure.

Category: Azure, Development | LEAVE A COMMENT
May 24

Find a bug and fix multiple bugs

Earlier in the year, whilst working on Omniworlds RPG, I came across an issue on the Discord Bot where it was constantly throwing an error of invalid item < at position 0 when trying to access a list of NPCs. On top of this, when calling the same endpoint via Insomniac, I also found that the API was limiting the response to 10 entries for some unknown reason.

After an initial search and attempt at rectifying the error on NodeJS, I decided to give up on that side and move on to the response issue and after a decent search, it was mentioned by a viewer that the potential NodeJS error was to do with json_encode() failing silently in PHP. I applied some error handling on that function and amazingly it was throwing the following error: Malformed UTF-8 characters, possibly incorrectly encoded!

I went back further through the code to trace back the errors cause and added ;charset=utf8” to the PDO connection string for the database calls and amazingly both issues were solved! The API was no longer limiting responses to 10 entries and the NodeJS invalid JSON error was suddenly gone.  It goes to show that sometimes, there can be a silent bug in your code that you are unaware of that has a knock on effect with other parts of your code!

 

Category: Development | LEAVE A COMMENT
December 24

The End of the Year and End of the Decade

Once again, the year is wrapping up and it is time to review both the highs and the lows of the year. This year however is slightly different as it is not just the year that is ending, it is also the end of the decade. There were plenty of lows this year but yet again, I have had far more positives over the year that has far outweighed any low that I have experienced as well as plenty of opportunities to start off 2020 in a positive fashion. As always, I will split this up into a few sections with a quick review of last year’s goals and then what the plan is for 2020!

 

Streaming

This year has been one of my worst for consistency unfortunately. It started off well, keeping to my 12 streams a month average but after 4 or 5 months, it took a turn for the worst with a few personal issues that was going on. On the positive side, I managed to exceed the 5 viewer average goal I set myself for the year, this was largly thanks to the high average viewer streams between June and August. Unfortunately I didn’t hit the 500 follower goal I had set either. On the positive side, I did get accepted to The Live Coders which will bring some good challenges and opportunities over the next year.

To prepare for 2020, myself and Clare have been spending some time planning out how the first quarter of the year is going to look and if everything goes to plan then it is going to be a busy start to the year. In the first month or two, the streams will only be on the weekend whilst I get re-adjusted. Below are the goals for 2020:

1) Average a minimum 12 times a month over the year (144 times overall).

2) At a minimum, maintain 2019’s average view count of 13. Stretch goal is to knock it up to 15 average over the year.

3) Look for a minimum 100 new followers over 2020, 200 as a stretch goal.

 

Professionally

Now this one has been both the cause of my lowest moments of 2019 but also my highest moments of the year too. Those that know me on a personal level, understand fully know hellish my current boss is. Well, working for her finally took me to breaking point which resulted in quite a bad deterioration in my mental health which with a little push from Clare, I went to see my GP who decided it was best that I take a leave of abcense from work for a two month period.

During this period, I felt so much about my personality change in a positive manner. It was almost like a weight had been lifted from my my mind, my shoulders and everywhere else. In the first week of being off, I also applied at a couple of companies. One as a PHP developer and one as a Supply Chain Analyst. Suprisingly enough, both of them wanted to hire me which was a massive tick in the box for finding a new role and I felt a million times better as well!

I start the new job in January and whilst it is a shame that I am leaving o2 after nine and a half years, it is a fresh start for a fresh decade… Same as when I started at o2 so it feels like it bodes well! My goal for 2020 is to really get into this role and develop myself massively!

 

I am not going to put out a personal section this year but I am really looking forward to what 2020 is going to bring in all aspects of my life. I am really greatful to all the friends, both physical and virtual that has helped me and been there throughout the year.

 

Category: General | LEAVE A COMMENT
August 18

It Has Been A While

Every month, I come back to here and think “damn I have been neglecting this blog so much, I need to write something” and then close down the tab (bad, I know). I keep telling myself that I cannot think of something to put together of adequate quality to present to a wider audience but sub-consciously I know that is just an excuse and one I need to get past!

To make the comeback easier, I thought it would be best to put together a summary of what I have been up to so far this year and then see where I go over the next few months in terms of blog posts… So lets get it started!

Professional Life

Despite the want to leave retail life, unfortunately opportunities have not been as forthcoming as I would have liked, I managed to get one interview to just be pipped at the post which is unfortunate but as always, just need to pick myself up, dust myself off and start looking again which is the plan!

Working in a more professional environment is the end goal but mainly I have been looking at technical roles (Software Configuration, Programming etc) and will keep pushing my skills to be more attractive to potential employers!

Streaming and Community

Well what can I say on this one? I have a small but amazing community that continually surprise me with the different levels of support from just turning up to financial. One amazing example is that in January/February I started a community goal to put money towards building a modern PC and retire the ancient machine I have been using for a decade! I expected this goal to still be going at Christmas this year but we managed to smash the goal by April which was crazy.

 

Other Tidbits

I have been doing a lot more programming in my spare time which has been excellent. Part of this is in the form of a website that I have released for my streamer personality https://itslit.uk which is a great showcase of leveraging the Twitch API as almost all the content is provided by data from the API services. The site also showcases a couple of side projects that I have been working on and released to the community as a whole and will continually add to it as I release new projects

Category: General | LEAVE A COMMENT
December 26

A New Year and New Goals

The Christmas lights are still twinkling, a beer is sat next to me and I am looking back through the achievements of 2018, professionally, personally and as a streamer.

Fortunately, despite some lows, there are plenty of positives to look back on over the year which is setting the tone for 2019 which is what we are going to focus on for this post and I will split it over a few sections to keep it organised.

Streaming

Despite taking a couple of large breaks throughout 2018 due to not being quite into it mentally and a few PC related issues, there were plenty of highs to outweigh the lows which was excellent! The two biggest highs that instantly comes to mind is the most recent event, Naughty or Nice which brought in record numbers over December and helped me to be more consistent in general. The second high that always sticks to my mind is the two occassions that MopGarden, a partnered Twitch streamer raided me which opened up a few new opportunities and new friends.

In that clip, it wasn’t until Mop said hi that I even knew how many people she had brought along! #coolAsACucumber
For 2019 my goals are going to be quite challenging but it will require me to step my game up and really work hard and here they are:
1) Stream a minimum of 12 days on average in a month (so around 3 streams a week). Stretch goal is 16 days on average in a month.
2) Push for an increase of viewers month on month / maintain a minimum of a 5.0 average as set for December. Stretch goal is to reach 8.0 average month.
3) Aim for 500 followers by the end of 2019, with a stretch goal of 750.

I decided to keep it to three goals as this will be a lot of work, especially mentally, so I feel that they will be challenging enough to keep me going and to show improvement year on year!

Professionally

From the tail end of 2017, through to June 2018, I found myself in a secondment at work which allowed me to work in my company’s supply chain team instead of my usual retail work, to develop a new system from scratch that was going to benefit the company massively. Whilst it was only fixed term, it was still disappointing that I was unable to continue the role after the 9 months of a better work/life balance and so many benefits that made my job feel blissful whilst still being challenging.
Due to this, I have been bitten by the “proper” job bug and I am aiming to look at new opportunities outside of retail work that will make my life better, reduce my stress (partially from customers, partially from useless Line Managers) and increase my happiness overall.
My goal on this side is to look for new opportunities and apply a JFDI mentality and put myself out there. Worst case scenario, I get rejected and I start again, best case scenario, I end up in a better role, with better hours which will make life so much better overall!

Personally

A lot has gone on in my personal world this past year, much of which I wish to leave behind me. Brief, to the point and as vague as I can write it eh 😉

My aim in 2019 is to balance everything better so that I do not let Clare feel neglected but at the same time, I get to learn and enjoy as much of my spare time as I can!

I have so many side projects in my head that I want to work on, some of which I will stream like the Bot development tasks for G4G, a Destiny 2 clan that me and Clare are both a part of. Also there is the API for Twitch Streamers and communities that I have been working on over the past 12 months.

As you can see, 2019 should be a busy but hopefully positive year for me and I hope that is the same for you! There are several people who have made this year as great as it is, all of you will know who you are but some of the real stand out people who helped a lot in some way this year, in no particular order are:

– Synxiec
– Clare
– Mopgarden
– Diddy

Category: General | LEAVE A COMMENT
December 2

Triggering Organisation

Since the hassle that was working with the follower event handler, I decided to move onto the raid and host event triggers despite leaving issues in the follower event. After the challenges I had just faced, I thought I had enough in the bank to be able to easily handle the upcoming code challenge. after about 15 minutes of reading and typing, I ended up with the following which compiled fine, the issue was it did not want to trigger.

private void OnRaidNotification(object sender, OnRaidNotificationArgs e)
{
form.WriteChat(e.RaidNotificaiton.DisplayName + ” raided!!!!!!!!”);
}

private void OnBeingHosted(object sender, OnBeingHostedArgs e)
{
form.WriteChat(e.BeingHostedNotification.Channel + ” hosted with ” + e.BeingHostedNotification.Viewers + ” viewers”);
client.SendMessage(Resources.channel_name,e.BeingHostedNotification.HostedByChannel);
form.AddEvent(e.BeingHostedNotification.Channel + ” hosted (” + e.BeingHostedNotification.Viewers + “)”);
form.AddEvent(e.BeingHostedNotification.HostedByChannel + ” hosted (” + e.BeingHostedNotification.Viewers + “)”);
}

After testing a few times to see if something is wrong (as you can see with the e.BeingHostedNotification.Channel AND e.BeingHostedNotification.HostedByChannel) to see which would work, it was time to throw in breakpoints. At least this way I would be able to tell if the issue lay with my code of with TwitchLib/Twitch… Recompiled the bot, dropped the host and… The breakpoint never triggered, I even waited a couple of minutes in case it was delayed, it was not. So there is a very likely chance that the issue lies with TwitchLib as neither trigger was fired, I decided to pop a message into the TwitchLib discord but after waiting two days, I still have no response other then “it should work”.

Last night was the first advent stream so I thought I would build the bot and give it a live test and all in all it performed as well as I had expected it to. Certain methods weren’t working, others did and it highlighted a few niggles and bugs that need to be worked on which after the stream caused me to get organised. I ended up creating a feature and bug/niggle tracker, yes I could have used Trello and/or GitHub but I felt like running it this way for now and you can see the results here:

 

Tomorrow I am on my first day back to work after my holiday so development and posts will slow down but it has been a good bit of progress so far!

Category: Development, Updates | LEAVE A COMMENT