Author: nurlailiyuzi

Monday, August 26, 2024 – Last week, I continued my exploration and implementation of RabbitMQ message queues. Following Mr. Peter’s advice, I focused on encapsulating methods to be used by multiple components, a best practice that helps reduce duplication and simplifies error detection. It took several attempts to perfect the encapsulation for both the publisher and consumer before I could move on to actually using them to publish messages.

When setting up message publication, I faced a few challenges. I had to consider the type of messaging pattern: whether the message queue would be from a single publisher to a single consumer, or from a publisher to multiple consumers. For scenarios involving multiple consumers, the publisher needs to implement an ‘exchange’. An exchange in RabbitMQ is a routing mechanism that decides how messages should be distributed to queues based on routing rules. It essentially determines where messages should go. There are different types of exchanges, such as direct, fanout, topic, and headers, each with its own routing logic.

Next, I turned my attention to implementing the consumer side. One tricky part was figuring out how to call the Consume method effectively and ensure it returned the published message for further processing. To handle message consumption, I used a background service provided by Microsoft. After setting up both the consume and publish methods, I checked the RabbitMQ management UI and confirmed that the published messages appeared in the queue. However, the messages were not being consumed as expected. After revising the consumer setup, I consulted Mr. Peter and discovered that the issue was with the publisher failing to send the message properly. I’ll be tackling this issue in the coming week.

Monday, August 19, 2024 – Last week, I started by thoroughly testing the new feature for importing Excel spreadsheets, ensuring that it could handle various errors to provide a smooth user experience. Once I felt confident that most scenarios were covered, I deployed the latest changes to the cloud server.

Next, I shifted my focus to researching message queues, specifically RabbitMQ. I began by creating console projects following the provided documentation, but I encountered a roadblock when trying to run the Docker command to build the RabbitMQ container. I forwarded this issue to Mr. Peter, who guided me the next day on running the correct command, and thankfully, the Docker container was successfully built.

Simultaneously, I worked on developing a small feature that involved implementing a single button. Given the simplicity of the task, I prioritized completing it before returning to RabbitMQ testing. The following day, I successfully ran the feature but noticed that it lacked detailed error handling, and additional filters were needed. Moreover, the query associated with this feature required optimization, as it initially took about 7 seconds to execute. By the end of the week, I managed to optimize the query, reducing the execution time to just 1.5 seconds. However, I discovered a discrepancy in the calculated data from this feature compared to the existing data from another module.

Next week, my priority will be identifying and resolving this bug before I can proceed with the message queue task.

Monday, August 12, 2024 – Towards the end of last week, I was tasked with creating a new functionality that allows data to be imported and read from an Excel file. This was a straightforward task requiring only two buttons for functionality. I started by working on the view and view model to get a comprehensive understanding of how the UI should operate and what data needed to be sent to the API.

Since this was my first time working with Excel import functionality, I spent some time familiarizing myself with the existing code to understand how previous imports were handled. After drafting the view and view model, I moved on to developing the API. My plan was to extract the data list from the imported Excel file, send it to the API, and then process it accordingly. After some work, I successfully completed the task, and the new feature worked as expected. However, after a review by Mr. Peter, I realized I had overlooked validation error handling for cases where the imported Excel file might contain faults or unexpected data.

In addition to this task, I also worked on fixing a few minor bugs related to previously developed features. These included adjusting the UI to fit the standard screen resolution of 120% used by most user, adding progress percentage logs to keep track of the progress via log files, and optimizing an API query that was taking too long to execute. By the end of the week, the Excel import feature required some minor checks for additional conditions before I could push the latest changes to the cloud server.

Monday, August 5, 2024 – After last week’s failed migration attempt, I identified the root cause as a mismatch between the data structure of the migration and the current table. After several attempts to resolve the issue, I decided to remove the problematic table entirely and recreate it using the existing SQL migration files.

Moving on to my next task, I needed to debug an issue where data generation was failing, causing the progress bar to close automatically without any visible error. This behavior led to errors going unnoticed. To address this, I added more robust error handling and logging to ensure that any errors would be flagged to the user. However, even after implementing these measures, the source of the error remained unclear. At this point, Mr. Peter suggested using stack traces for complex functions and demonstrated how to apply them effectively. With stack trace error handling, we could pinpoint the exact line where the error was thrown, making troubleshooting much easier.

Later in the week, Mr. Peter assigned me to research message queues and identify the best platform for our upcoming tasks. After exploring various options, I became particularly interested in RabbitMQ and delved into its documentation to learn more about this service.

Monday, July 29, 2024 – Last week, after I pushed all my changes to cloud server, Mr. Peter found two API tests failing. I quickly investigated the tests and found that one was failing because I had added a new table but hadn’t updated the data for that specific test. However, the second test revealed a loophole in my logic execution.

The issue stemmed from two newly created tables meant to store information about user type A and user type B. These tables were designed to serve as references to reduce the amount of data in a single table. During the unit test for this specific API execution, it was discovered that user type A actually required a reference from the user type B table, which only had an established relationship with a different kind of the table. This error necessitated rearranging the tables and combining user type A and user type B into one table so that both could be referenced without any obstruction.

As the week was ending, I made the necessary changes but encountered a migration failure. I plan to resolve this issue on Monday as quickly as possible before moving on to the next task.

Monday, July 22, 2024 – In the previous week, I implemented a raw SQL query to set a column in a table to a condition, resolving a bug where the loading bar never ended. While the query executed correctly, Mr. Peter advised that, since we are using the EF Core approach, we should minimize the use of raw SQL queries unless absolutely necessary.

During this time, I also discovered another bug where a page displayed an “object set to reference” error. Upon careful debugging, I found that the error was due to the recent addition of a new table. The query from this table set a reference but didn’t correctly group it, resulting in an inability to map the correct data.

After solving the bug, I quickly worked on removing the raw SQL query. Initially, I tried using a foreach loop and experimented with multiple foreach iterations. However, waiting for all loops to finish before sending the results back to the UI caused a waiting time of about one minute, which was impractical. I then experimented with executing the task using async/await to see if there was any improvement in performance, but the total execution time remained similar.

After some consideration, to avoid long waiting times for an unknown response, I decided to implement the foreach loop along with a progress bar for the delete process.

Monday, July 15, 2024 –Last week, I encountered a few bugs related to the report section. The first bug was with the overall margin calculations, which were incorrect even though the formula was right. After generating the margins and filtering a group of data with the same reference, some groups had a similar bug pattern. Specifically, one margin in the list was calculated as zero when it shouldn’t have been. I realized the issue was due to committing the generated margins in batches. I had forgotten to include an important function in the last batch, which caused the bug.

The second bug occurred in another section with multiple combo boxes for selecting dates and years. When a user selected a year, the combo box that should have displayed all years was null. Initially, I thought the problem was in the view model. However, after thorough debugging, I discovered the issue was with the API, which lacked a query to retrieve all existing years for the specific data. Once I identified the source of the bug, I was able to fix it.

The third bug of the week involved a button intended to remove a table of data. The query was supposed to be straightforward, but the loading bar never seemed to end. The problem was that the query took too long for the loading bar to handle. I solved this by implementing a raw SQL query, which made the function run in under ten seconds.

I pushed all the changes and will continue with my next task next week.

Monday, July 1, 2024 – Last week, while waiting for the data generation to be completed, in order to test the stock generation function, I retested the controller tests to ensure everything was running successfully. However, I discovered that I had overlooked a test, particularly the generate stock test. The cause of the failure was due to my implementation of a task when creating the stock as an alternative to a foreach loop. This task caused the failure because, as Mr. Peter explained, we cannot run tasks when calling or using services from the repository due to the multiple DB contexts involved.

To resolve this, I modified the code to use a foreach loop instead. However, my next concern was how to handle the same data being called again every time the table is generated. Although the table configurations include settings for unique data, looping through already stored data is time-consuming and unnecessary. After consulting with Mr. Peter, he suggested using the Except function to exclude existing data from the list, ensuring that the list only contains new data without the need for a foreach loop to check each item individually.

Next, while examining and testing the logic for the specific data generation, which was previously developed but had performance issues, I found that the commit transaction was causing a slowdown at the end of the progress bar. Committing a large volume of data at once proved to be inefficient. I experimented with moving the transaction into every loop, but this still resulted in a very slow process, with the progress bar updating every second, meaning thousands of loops would take thousands of seconds to complete. 

Mr. Peter suggested committing transactions in batches of 500, which improved the speed. I then tested increasing the batch size to 1,000, 10,000, and 20,000. I found that committing every 10,000 records was the most efficient for now, with a commit time of about 5-7 seconds per batch. This significantly sped up the generation process compared to committing every 500 records in one batch, which took about 2 seconds per batch.

Monday, July 8, 2024 – This week, I was tasked with adding pagination to another segment of the stock management UI. This required changes to both the API query and the UI view model. First, I focused on the API query by incorporating all the necessary additions into the request query, based on the existing filters from the front-end UI filter. Previously, all the data was retrieved in a single API call and stored in the background data grid collection. The filtering logic was then handled in the view model. The initial purpose of this approach was to ensure a faster response based on the user’s search.

However, after testing the UI with an expanded dataset reaching up to a hundred thousand records, retrieving all data at once degraded the page’s performance, which could be taking up to 8-10 seconds to load a single page. This led to the decision to implement pagination, which would significantly improve performance by fetching only 50-100 records per page.

While working on the pagination logic for the query, one challenge was calculating the average total margin directly within the API query. Initially, I fetched all stock data into a list, which slowed down performance and defeated the purpose of pagination by fetching a large amount of data at once. To address this, I adopted a different approach.

I was reminded of LINQ’s concept of Deferred Execution, where data is not immediately retrieved from the database. Instead, an expression tree representing the query is constructed, and the query is executed only when iterated over or when a method forcing execution (such as ToList() or ToListAsync()) is called. This insight led me to try the more efficient approach.

Thus, rather than immediately fetching the entire query result into a list, I applied filters to retrieve only the necessary data. I then utilized the Select() method to specify and retrieve only the required fields from the database, enabling efficient data transformation into the desired format. This approach offered greater flexibility for handling data, allowing for transformations, calculations, or filters directly within the Select() projection. By implementing these transformations efficiently, I optimized performance by fetching the transformed data into a data format that I want and only turning them into a list only when necessary, resulting in improved speed and efficiency.

Once the API query was complete, I fixed the controller tests and began working on the view and view model changes, ensuring the controller tests passed successfully. By the end of the week, I had completed most of the assigned tasks. There are a few remaining UI issues that I will continue to check next week to ensure a flawless user experience.

Monday, June 24, 2024 – In my previous blog, I mentioned that I created three additional tables to store unique names, establishing relationships with the existing table to facilitate querying without manually assigning names.

After adding these three new models, I needed to update many parts of the code logic. The first step was to ensure all three new tables had the latest data before generating stock data. To achieve this, I created a repository and the necessary services to retrieve data from source names and items from different contexts and store them in the new tables before creating the stock history data.

I then tested the new logic and encountered a few errors. The first error occurred because I didn’t rebuild the Cassandra container correctly. The next error was an “unknown database” issue, which stumped me for a while. Mr. Peter identified that the debugger was running too quickly, causing it to end prematurely. He advised me to use a while loop to keep the debugger running while the status was still pending. This adjustment allowed the code to execute correctly without ending too fast.

After integrating the new tables, I realized that nearly all of the controller tests needed updates. This involved reassigning the item and source names for each test to reflect the new data structure. I meticulously went through each test, ensuring they were correctly updated and consistent with the new models. In the process, I took the opportunity to refactor and clean up the tests, improving their readability and efficiency. This comprehensive update not only fixed the immediate issues but also enhanced the overall robustness and maintainability of the test suite.