Author: nurlailiyuzi

Last week, several Docker containers repeatedly crashed despite my attempts to restart them. I consulted Mr. Peter and discovered that the disk had reached its maximum capacity of 20GB. To resolve this, I needed to increase the disk size by 50GB. I sought Mr. Peter’s guidance on how to proceed, but since I was not very familiar with Docker commands, I researched Docker containers further. Before making any changes to the disk, I created backup files to ensure data integrity in case of any issues. Despite several attempts, I was unsuccessful, and Mr. Peter eventually helped me expand the disk size while preserving the data.

Afterward, I resumed optimizing a query by implementing pagination instead of fetching the entire list in one API call. While working on a query to read a list of entities from the backend, I encountered an issue where I could not directly query a specific column which was not present in the table. Previously, filtering was done on the front end using BackgroundDataGridCollection, and I was assigning the data names by using a foreach loop into each of the wrapper. I sought Mr. Peter’s advice on the best approach. The next day, he suggested creating three additional tables to store unique names, establishing relationships with the existing table to facilitate querying without manually assigning names.

Following this, I initialized the models for the three new tables, configured their relationships, and added integration tests for each table.

By the end of the week, there were still many parts of the code needing optimization for a cleaner approach, which I plan to continue next week.

Monday, June 10, 2024 – Last week, I initiated the generation of 200,000 dummy data points. Although the data generation process worked, it was quite slow. Over the weekend, the process stopped midway and responded with a 401 unauthorized error. I restarted the program and the generation, and reported the status to Mr. Peter. It turned out that the reason for the interruption was a user API access limit of finite hours. Mr. Peter instructed me to extend the access duration on the identity server.

I then continued with my current task: debugging and investigating why the API calls for a list query were taking so long. While debugging a specific handler that Mr. Peter had previously improved by implementing a parallel foreach loop, I noted that I had initially used a standard foreach loop to assign additional data to each list item. The parallel foreach has the advantage of leveraging multiple threads to process items in the collection concurrently. This can significantly reduce the overall execution time, especially for large datasets or operations involving intensive computation. By distributing the workload across multiple threads, it maximizes CPU resource utilization, leading to faster task completion compared to a standard foreach loop, which processes each item sequentially.

However, despite testing different degrees of parallelism to optimize performance, where I used a stopwatch to measure the execution time of each configuration, having more than 4 degrees of parallelism did not show any improvements in execution times, so I settled on using 4 as the maximum degree of parallelism. Even after implementing the parallel foreach, the query performance was still unsatisfactory, taking over a minute to retrieve the list on the front end. This prompted me to search for additional optimization strategies.

Upon finding a suggestion on Stack Overflow, I realized that using FirstOrDefault might be contributing to the slow performance. An article confirmed that FirstOrDefault could be less efficient compared to other methods. I decided to replace it with ToDictionary, which was reported to have faster performance. After implementing this change, I observed a significant improvement in the retrieval time, reducing it to 19 seconds for the two lists called asynchronously, which was approximately 3 times faster.

Despite this improvement, the retrieval time of over 19 seconds was still far from ideal. Consequently, Mr. Peter advised me to implement pagination for the two lists to further enhance performance.

In my previous blog, I discussed the challenges I faced while creating a function to generate a large amount of data. This week, I continued my journey in debugging to make the generation function work effectively.

Several issues were causing the failure. First, I was using the wrong function to execute the data generation. To generate data with this specific method, I needed to execute three other functions consecutively to prepare the event before making the main function work.

While debugging the next function, I noticed I was calling the wrong function. I quickly corrected this based on guidance provided by Mr. Peter. Following this, I had two more functions to test. However, I got stuck on one function whose role was to confirm if data was being picked by staff. When I ran this function, the data status was not updated, and no specific error was thrown in the log.

Then, I decided to process the data manually to see what exception will be thrown from there. I realized that the “picked” button was not working because there were errors in the quantity and price of some data entries. I manually fixed the data one by one.

Next, I worked on the function to change the status to as being packed. I created new data, and the new data ran successfully while the old data still caused errors. I tried to delete these erroneous data manually but not implemented yet. I mistakenly deleted an entire table of data when I intended to delete only a specific line, forgetting there was a “cancelled” button I should have used instead.

Later, I asked Mr. Peter for advice. Following his guides, environment restored. Towards the end of the week, all the generation functions finally worked successfully. Mr. Peter suggested combining them into one button to avoid repeatedly clicking multiple buttons. However, when I combined the functions, the process kept failing, only completing the first out of eight steps.

The next day, Mr. Peter helped debug the issue and found that an asynchronous process in the progress dialog bar was causing the failure. Once it was successfully resolved, I input 200,000 to be generated and left the PC running over the weekend until it completed.

Next week, I will focus on the purpose of creating this generation button, which is to simulate the scenario of slow performance caused by handling a large amount of data.

Monday, May 27, 2024 – Last week, after pushing all my changes to cloud, Mr. Peter provided feedback on the latest feature he tested. Unfortunately, there were many issues that needed to be addressed. The major problem was how the data was being loaded in the UI when retrieved from the API. Since the real data was significantly larger than my dummy data, I hadn’t anticipated the issues that would arise. This became the most critical problem to fix.

To replicate the slow performance issue, I needed to generate a large amount of data to mimic the real scenario. This led to my next task for the week: creating functions to generate a specific amount of data as specified by the user. A few months ago, I developed a similar feature, so I reused the same module but expanded it with more functions to meet the current requirements.

By the end of the week, I had created all the necessary functions. However, only half of them ran successfully, while the rest required further debugging. I plan to continue this debugging in the upcoming week. Once resolved, I will generate an additional 100,000 to 200,000 data points to better simulate the heavy use case scenario and help me debug the problem more efficiently.

Monday, May 19, 2024 –Continuing from my previous progress on publishing the project, I encountered an issue where the launch crashed. It turned out this was because I hadn’t republished the other API project with the latest changes. After publishing everything and rebuilding the Docker containers, the app launched successfully and worked just as it did during debugging.

Immediately, I focused on cleaning up one particular code, which I had been postponing to prioritize another task, which was to merge an old integration test with a new one. Once the merging completed, I jumped into testing out the application to make sure everything in the UI run as expected, then I noticed there was an error coming from logs. This error only appeared when executing a specific command. Interestingly, during debugging, no error was shown; instead, it just reported a transaction error in the log, even though the function executed perfectly.

I informed Mr. Peter about this issue, and he advised that it was an opportune moment to implement error-catching in my code. This led me to a debugging session to trace the error’s source. Finally, I discovered the issue: it stemmed from my attempt to delete a table with a one-to-one relationship with another table. Deleting Table A would already delete Table B, but I mistakenly deleted Table B first, which caused the transaction error in the logs. After correcting this mistake, no errors appeared in the logs.

The next bug I tackled involved deleting an entity in the report. After deletion, the table’s counter ID didn’t reset to 1, so I modified my query to address this.

After ensuring everything worked as expected, I pushed all my changes for Mr. Peter to review and supervise my latest update.

Last week, I began by integrating a previously completed test with an existing one. After that, I focused on cleaning up the code, particularly the sections related to a recently developed report feature. I then tested the app to identify any overlooked bugs. During this process, I discovered that the date filter was not functioning as expected. The issue was that I had made the date a query parameter for the API while other filters were applied to the background data grid collection. To address this, I removed the date query from the API and handled all filtering and search logic within the view model, aiming to reduce the number of API calls and improve loading times.

Next, I tackled a bug where the app sometimes became unresponsive when a progress bar was used. The logs did not show any errors, and there was no consistent event to pinpoint the cause. After discussing this with Mr. Peter, he suggested publishing the app to see if the issue persisted, as it might be related to Visual Studio. This was my first time publishing independently, so I asked Mr. Peter numerous questions to ensure I did it correctly. My initial attempt at publishing failed, and I was unsure where to look for the issue. After doing a quick research on Stack Overflow, I learned that I should check the output rather than the error list when publishing. Following this advice, I successfully published the app, but clicking on the System.exe file yielded no response.

I consulted Mr. Peter again, and he guided me on what to check next. It turned out the newest API was not built correctly. Upon reviewing the schema in the docker-compose.yml file, I discovered I had inserted an incorrect port name. After fixing this and rebuilding the API, I reopened the application. This time, the app launched, but it crashed when attempting to read or retrieve data from the API. As the week ended, I planned to continue addressing these issues in the following week.

Last week, I developed a UI table to facilitate the ongoing use of the latest entity for storing calculated margins. During this process, I recognized the necessity for an additional column—a boolean or flag from the tracker—to indicate the calculated margin as true. Consequently, I proceeded to implement the functionality for undoing margins, enabling the deletion of all data in the table. This deletion serves the purpose of resetting the table to regenerate and track based on the latest updates.

However, since the table that displays margins are interconnected, I ensured to address the logic for deleting all entities in the table and resetting several conditions in their respective tables. Subsequently, I created controller tests to verify the functionality. Although the tests were successful, I sought confirmation from Mr. Peter to validate my approach. Additionally, I added a button to the UI for easier access.

This week, I made adjustments to the UI displaying the data log. Upon completion, I noticed discrepancies in the creation of the tracker, as it did not produce the expected results as previously simulated. After debugging, it was revealed that there were errors in the logic governing the creation of the tracker data table.

After ensuring that all data was stored and displayed correctly, after I’ve showed to Mr. Peter, he pointed out a flaw in the design Where the system could only track the output of log data without being able to track the input. Consequently, I reverted to the original approach as initially discussed. Furthermore, I fixed several filters on the page that were not functioning properly. As the week draws to a close, there are still some adjustments needed, but I will continue addressing them in the upcoming week.

Monday, April 29, 2024 – Last week, I began by creating controller tests for the API responsible for recording inventory data. These tests were invaluable in helping me fully grasp the necessary logic to achieve the desired results. As I delved deeper into refining this logic, I found the need to add boolean-type columns to the entity to indicate processed and calculated inventory, preventing duplicate data retrieval.

Although it took some trial and error, as well as simulated data testing, once the tests were successful, I swiftly proceeded to develop the UI for displaying the recorded inventory.

Moving forward, I shifted my focus to another entity’s section that will store margin calculation . As I worked on creating the entity, I sought advice and opinions from Mr. Peter to ensure the best outcome. During the process of developing the entity’s service, which execute the logic for entity’s creation, I realized a previous error in the relationship of the current entity model with the preceding entity. Upon recognizing this, I promptly corrected the related services and logic, and tested them thoroughly post-changes.

Throughout the creation of this latest entity, whenever I encountered challenges with the logic, I endeavored not to dwell too long and alternately worked on creating or improving the UI. This approach allowed me time to contemplate better solutions.

As the week drew to a close, there were a few things needed to complete the task and some clean-up tasks to complete but I shall continue next week.

Monday, April 22, 2024 – Last week, following the end of the Hari Raya holiday, I resumed my work on developing the report section. Before the holiday began, I was assigned a new task to work on once the break was over – the margin section. Margins typically refer to profit margins, indicating the percentage of revenue that remains as profit after accounting for all associated costs. The margin section will allow users to view the overall margin calculated based on the latest input and output, previously developed and equipped with the latest data for comprehensive consideration.

To show the margin accurately, I needed to track the the goods in and out of the inventory. Therefore, Mr. Peter introduced me to a new entity designed to record and track the necessary data. Creating this entity would enable me to achieve the required next part of margin calculation. Initially, understanding how data would be recorded in the entity’s table was a bit confusing, but Mr. Peter simplified it visually, making it easier to grasp the basic flow.

I promptly worked on creating the model for the new entity and tested its type in the integration tests. Then, I proceeded to develop its services and outlined the necessary logic to achieve the desired results. Though some aspects were confusing, Mr. Peter patiently explained them to me and was open to any necessary changes or improvements.

Towards the end of the week, I developed an API responsible for calling the entity’s service to record inventory data. Additionally, I performed migration of the newly created entity model, updated the database, and generated its SQL file before adding them to the cloud. However, there is still more needed to be done to complete this task and shall continue working on it next week.

Last week, I had three primary objectives to accomplish before the week’s end. Firstly, I aimed to develop the view and view model for showcasing the latest table, alongside creating the necessary API for both command and query functionalities to be utilized by the report module. Following Mr. Peter’s recommendation, I also integrated SignalR into the API command to facilitate a progress bar, ensuring users are informed about ongoing processes. Given that this command involved calling upon 10 services, Mr. Peter advised sequencing them meticulously to avoid conflicts during implementation.

Once the foundation for the view and view model was laid, I delved into creating the API command, incorporating SignalR. Initially, I tested the functionality with just one service to ensure smooth execution before scaling it up to encompass all 10 services. Integrating SignalR necessitated adjustments to the service’s return states. With the necessary SignalR setup in place, I proceeded to conduct the controller test. Although the initial test was unsuccessful, I was able to identify the missing configurations causing the failures.

With the API command completed, I promptly integrated it into the UI. However, I soon realized the need for two separate API queries to retrieve data from the latest tables. Without wasting time, I dived into the creation of the two api query along with their tests. After confirming the success of the tests I devised, I refocused on optimizing the UI for seamless functionality. Approaching the end of the week, a significant challenge remained: devising a method to consolidate progress updates from all 10 services into a single progress indicator. This will be the first thing that I will work on this upcoming week.