Month: June 2024

Last week, several Docker containers repeatedly crashed despite my attempts to restart them. I consulted Mr. Peter and discovered that the disk had reached its maximum capacity of 20GB. To resolve this, I needed to increase the disk size by 50GB. I sought Mr. Peter’s guidance on how to proceed, but since I was not very familiar with Docker commands, I researched Docker containers further. Before making any changes to the disk, I created backup files to ensure data integrity in case of any issues. Despite several attempts, I was unsuccessful, and Mr. Peter eventually helped me expand the disk size while preserving the data.

Afterward, I resumed optimizing a query by implementing pagination instead of fetching the entire list in one API call. While working on a query to read a list of entities from the backend, I encountered an issue where I could not directly query a specific column which was not present in the table. Previously, filtering was done on the front end using BackgroundDataGridCollection, and I was assigning the data names by using a foreach loop into each of the wrapper. I sought Mr. Peter’s advice on the best approach. The next day, he suggested creating three additional tables to store unique names, establishing relationships with the existing table to facilitate querying without manually assigning names.

Following this, I initialized the models for the three new tables, configured their relationships, and added integration tests for each table.

By the end of the week, there were still many parts of the code needing optimization for a cleaner approach, which I plan to continue next week.

Monday, June 10, 2024 – Last week, I initiated the generation of 200,000 dummy data points. Although the data generation process worked, it was quite slow. Over the weekend, the process stopped midway and responded with a 401 unauthorized error. I restarted the program and the generation, and reported the status to Mr. Peter. It turned out that the reason for the interruption was a user API access limit of finite hours. Mr. Peter instructed me to extend the access duration on the identity server.

I then continued with my current task: debugging and investigating why the API calls for a list query were taking so long. While debugging a specific handler that Mr. Peter had previously improved by implementing a parallel foreach loop, I noted that I had initially used a standard foreach loop to assign additional data to each list item. The parallel foreach has the advantage of leveraging multiple threads to process items in the collection concurrently. This can significantly reduce the overall execution time, especially for large datasets or operations involving intensive computation. By distributing the workload across multiple threads, it maximizes CPU resource utilization, leading to faster task completion compared to a standard foreach loop, which processes each item sequentially.

However, despite testing different degrees of parallelism to optimize performance, where I used a stopwatch to measure the execution time of each configuration, having more than 4 degrees of parallelism did not show any improvements in execution times, so I settled on using 4 as the maximum degree of parallelism. Even after implementing the parallel foreach, the query performance was still unsatisfactory, taking over a minute to retrieve the list on the front end. This prompted me to search for additional optimization strategies.

Upon finding a suggestion on Stack Overflow, I realized that using FirstOrDefault might be contributing to the slow performance. An article confirmed that FirstOrDefault could be less efficient compared to other methods. I decided to replace it with ToDictionary, which was reported to have faster performance. After implementing this change, I observed a significant improvement in the retrieval time, reducing it to 19 seconds for the two lists called asynchronously, which was approximately 3 times faster.

Despite this improvement, the retrieval time of over 19 seconds was still far from ideal. Consequently, Mr. Peter advised me to implement pagination for the two lists to further enhance performance.

In my previous blog, I discussed the challenges I faced while creating a function to generate a large amount of data. This week, I continued my journey in debugging to make the generation function work effectively.

Several issues were causing the failure. First, I was using the wrong function to execute the data generation. To generate data with this specific method, I needed to execute three other functions consecutively to prepare the event before making the main function work.

While debugging the next function, I noticed I was calling the wrong function. I quickly corrected this based on guidance provided by Mr. Peter. Following this, I had two more functions to test. However, I got stuck on one function whose role was to confirm if data was being picked by staff. When I ran this function, the data status was not updated, and no specific error was thrown in the log.

Then, I decided to process the data manually to see what exception will be thrown from there. I realized that the “picked” button was not working because there were errors in the quantity and price of some data entries. I manually fixed the data one by one.

Next, I worked on the function to change the status to as being packed. I created new data, and the new data ran successfully while the old data still caused errors. I tried to delete these erroneous data manually but not implemented yet. I mistakenly deleted an entire table of data when I intended to delete only a specific line, forgetting there was a “cancelled” button I should have used instead.

Later, I asked Mr. Peter for advice. Following his guides, environment restored. Towards the end of the week, all the generation functions finally worked successfully. Mr. Peter suggested combining them into one button to avoid repeatedly clicking multiple buttons. However, when I combined the functions, the process kept failing, only completing the first out of eight steps.

The next day, Mr. Peter helped debug the issue and found that an asynchronous process in the progress dialog bar was causing the failure. Once it was successfully resolved, I input 200,000 to be generated and left the PC running over the weekend until it completed.

Next week, I will focus on the purpose of creating this generation button, which is to simulate the scenario of slow performance caused by handling a large amount of data.