Author: nurlailiyuzi

Monday, November 4, 2024 – Last week,  which was a short week for me, I focused on building entities for our new API project. My first step was to review our entity-relationship diagram (ERD), a clear diagram we rely on for a clear visual structure of the database. This diagram, equipped with attributes and data types, serves as a reference point for any updates or improvements to the database. After double-checking data types and discussing potential enhancements with my colleague, I proceeded with entity creation within the project.

When defining entity models, I intentionally saved relationship configurations for last to avoid complications, allowing me to focus on careful assignments. After adding a migration, my initial attempt failed, so I reviewed and corrected some errors. Once the migration was successful, I moved on to developing CRUD operations, starting with a “Read by ID” query since it’s the most straightforward. Building the repository required extra attention as, unlike previous projects using EF Core, this one uses Dapper for database mapping, requiring SQL queries to be written manually. Since the entity I queried has a one-to-many relationship with another entity, I researched the practices for crafting SQL queries in such cases.

Before advancing further in the CRUD process, I created a unit test for the query. Running tests allowed me to debug and identify issues in my code and query. Mr Peter also provided valuable advice on tracing errors within the query based on error messages. By the end of the week, I successfully passed the test and plan to expand the test scenarios next week before proceeding to the next CRUD operation.

Monday, October 28, 2024 – Last week, Mr. Peter assigned me the task of refining and to push the seed data generator that was previously created to automate sample data generation. Initially, the code was tailored for use only in my local environment, hence it required significant cleaning before it could be shared. Since I had been testing it extensively in a trial-and-error setup, there was quite a bit of redundant code to sift through.

To start, I revamped the UI page, removing unused buttons and giving it a more streamlined appearance. Then, I focused on optimizing the view model for better functionality.

The seed data generator was designed to handle multiple entities with a single button click, which led to long stretches of code that required careful verification to ensure smooth data generation. Although most of the functions leveraged existing code, a few necessitated extra queries. This meant creating new functions in a separate project, which hadn’t been pushed previously.

In my initial implementation, I used a parallel foreach loop to expedite data generation by processing multiple records simultaneously. However, this approach occasionally caused errors when handling more than two processes concurrently. Switching back to a standard foreach loop resolved these errors but slowed down the data generation, making the process less efficient.

Apart from needing Mr. Peter’s approval for certain functions, as I was limited to using DLL-based functions. Towards the end of the week, the generator worked well in my environment and everything was successfully operational, although further testing might be necessary.

Monday, October 21, 2024 –  Last week, I was given access to an API sandbox project. In this sandbox, Mr. Peter had already outlined the necessary components for building the API, serving as a reference for the new project. To get started, I needed to set up a Docker container for one of the databases we would be using, PostgreSQL. After configuring everything, I ran the application and tested the Swagger page, successfully verifying that the API was working.

However, I later noticed that the intended database was not appearing in PostgreSQL. After consulting with Mr. Peter, it was discovered that one of the configuration settings had not been correctly applied. Once this was resolved, the database was successfully created. I then quickly dove into creating a table for an entity and took the opportunity to learn more about how Dapper and PostgreSQL were integrated, ensuring I fully understood the technology stack I’d be working with.

Since we used EF Core to create the tables, much of the code structure remained familiar, even with PostgreSQL as the database. Without wasting time, I began working on an API query for one entity. Toward the end of the week, Mr. Peter advised that when working on the real project, I should start by working on smaller, less complex tables first and tackle the more complex ones later. Initially, I had assumed the opposite approach was better, but his guidance gave me a clearer perspective on how to approach the project.

Monday, October 14, 2024 – At the start of the previous week, Mr. Peter introduced me to our upcoming project, which will use Next.js for the website and ASP.NET for the backend API. However, instead of using EF Core, we’ll be adopting Dapper for mapping database queries. To help me get up to speed with Next.js, Mr. Peter provided access to a course and other resources to familiarize myself with the technologies involved in this project.

In regard to last week’s issues with message queue consumers in RabbitMQ, Mr. Peter addressed all of my concerns and provided clear explanations. I was very grateful for his guidance in correcting my misconceptions, especially on the Dead Letter Exchange (DLX). With the RabbitMQ issues resolved, I shifted my focus back to learning Next.js.

After completing the Next.js course, I immediately started applying what I had learned. Since we’re using TypeScript in our project, I initially struggled a bit with ensuring that the code I wrote adhered to clean code principles, especially avoiding duplication. Next.js is a powerful React framework that simplifies both server-side rendering and client-side rendering. To create a user interface, I applied Tailwind CSS, which made styling much easier and more efficient.

For data display, I created a dummy JSON dataset and used an online free API that provides images to enhance the product list. This allowed me to develop a simple page where a list of products is shown, and each product can be clicked to view its details on a separate page.

Towards the end of the week, I successfully built a basic dummy page that simulates a product listing with clickable details. Mr. Peter also explained the overall project flow and the different entities we will be working on in the coming week. This project will incorporate multiple databases, including PostgreSQL, MongoDB, and AWS, each chosen based on their strengths for specific tasks. I’m looking forward to diving into this exciting project of multi-database architecture as we move forward.

Monday, October 7, 2024 – Last week, I integrated a new logic for updating bulk modified entities from one context into another within an existing function. Initially, everything worked fine. However, during further testing, I discovered a loophole in the implementation. If only the name was changed, the update worked as expected. But when the user also changed the associated data code (which was used as the search key), the system created a new data entry instead of updating the existing one. After discussing this with Mr. Peter, we decided not to continue with this implementation, as it became unnecessary once the RabbitMQ message handling stabilized.

I then refocused on refining the message queue service. I encountered an issue when running the queue three times in a row: the system threw an error stating, “A second operation started on this context instance before a previous operation completed.” This was due to different threads trying to use the same instance of DbContext concurrently. To resolve this, I implemented a scope within the update data function, ensuring that each operation had its own context instance. Once completed, I tested it and it worked. However, a new issue emerged, the message execution was not happening in the correct order.

After some research, I found that implementing a semaphore could help enforce proper message execution order. Still unsure if this was the best solution, I consulted Mr. Peter. He pointed out that using a semaphore wasn’t necessary, as the issue likely stemmed from unrefined code elsewhere. The next day, Mr. Peter made minor adjustments to the code and explained that the unnecessary use of the using() function was limiting how the message queue was consumed.

With this issue resolved, I then revisited the message flow to identify potential errors or loopholes. I encountered another problem on the consumer side. If an error occurred during message processing, the consumer would issue a basic nack (negative acknowledgment), and the message would be discarded. To avoid data loss, I researched how to handle this more effectively. RabbitMQ’s documentation suggested using Dead Letter Exchange (DLX), a method that redirects failed messages to a dedicated queue for further inspection or retries.

By the end of the week, I was able to implement the DLX mechanism, so if a message received a basic nack instead of an ack, it would be sent to the DLX queue. However, a concern arose as instead of just one message being sent to the DLX queue, four messages were sent. This remains an issue to address.

Monday, September 30, 2024 – Last week, I continued the cleanup of old and unused features, focusing on the entity models that were no longer relevant. This process involved verifying that all removed features were not in use elsewhere in the code before I executed the migration and updated the database.

Previously, I had tried to create a separate connection service for RabbitMQ to avoid hardcoding the connection details. However, when testing this service, I found it failed to read the connection parameters. After consulting with Mr. Peter, I learned that the RabbitMQ connection service was unnecessary and could be simplified by replacing specific hostnames with a general hostname defined in the config file.

Next, while enhancing the message queue, I encountered an issue with publishing messages. I had initially implemented a “Publisher Confirm” mechanism, but it had drawbacks. Mr. Peter demonstrated an alternative approach that involved using transactions, along with commit and rollback procedures. This method helps ensure that messages are processed reliably, mitigating the risk of data loss.

As the message queue service is designed to update data from one context to another, it effectively processes updates one at a time after a message is successfully published. However, this raised a question: what happens to previously created data that has already undergone changes? One solution would be to delete and regenerate this data, but that would be highly inefficient, especially considering the processing time involved.

To address this, I initially considered creating a button to pull any modified changes from the original context. However, I later realized this was not the best approach. Instead, I decided to use the existing regenerate button to call out or update changes effectively. By the end of the week, I had successfully integrated this new logic into the existing function, although further testing is necessary to ensure that no errors or bugs will be introduced in the process.

Monday, September 23, 2024 – Last week, I focused on refining the controller tests, particularly in contexts that typically throw errors when executed all at once, but function correctly when run individually. The root cause was the unavoidable sharing of data within a single test file that contained both query and command API tests. Following Mr. Peter’s advice, I decided to separate the tests by query and command to eliminate this issue.

I also continued enhancing the message queue I was working on. I encountered a significant problem: when the RabbitMQ server is down from the start, starting the application caused a collapse. Initially, I thought this behavior was acceptable, but Mr. Peter pointed out that since RabbitMQ is implemented on the backend, it shouldn’t impact the user interface unless an API call triggers it. Upon further investigation, I found that several frontend services also utilized the message queue service, which contributed to the issue.

To address this, I extracted the used services into a new service that integrates the message queue functionality. By the end of the week, I presented my progress to Mr. Peter, and he noted that several aspects of the message queue service still needed enhancements, which I plan to tackle next week.

Monday, September 16, 2024 – Last week, I continued my work on completing the message queue. Although I had successfully consumed and received the messages, I was initially unable to execute the intended purpose of these messages. After presenting my attempts to Mr. Peter, I learned that the issue stemmed from an unnecessary transaction I had included. It turned out that the system only required a simple update function.

Once I ensured that this part of the message queue was functioning correctly, I proceeded to implement the message queue in various components that required it. This message queue facilitates communication between microservices, ensuring that relevant data is consistently updated across the system.

Mr. Peter tasked me with cleaning up several features and services that would no longer be needed in the future. My next challenge with the RabbitMQ implementation was determining the best way to handle errors. To test this, I stopped the RabbitMQ server and attempted to start the application, which caused it to shut down immediately. To mitigate this, I added a try-catch block inside the message service constructor to ensure that errors would be thrown rather than causing the application to crash. While this approach worked, I recognized that it was not a clean or best practice solution.

Determined to refine my approach, I created specific files for managing the RabbitMQ connection, aiming to avoid hardcoding and to handle errors more effectively. However, I encountered issues during testing; it appeared that I might have missed some initializations. I plan to revisit this in the coming week.

Monday, September 9, 2024 – Last week, while continuing to test the new button functionality, I encountered an error during the data cleanup process. The issue arose because the cleanup only read data from a single table, while the UI page displayed a combination of data from Table A and Table B. After identifying the problem, I created a new UI page to ensure the cleanup query also retrieves data from Table B, preventing discrepancies.

In addition to resolving the UI data issue, I needed to ensure the data cleanup logic was accurate and effective. Initially, the cleanup process was set up to automatically convert any null or zero values to zero. However, this approach did not fully address all potential data scenarios. After discussing the requirements with Mr. Peter, he provided insights into the different scenarios and the best approach for handling data cleanup, which refined my solution.

Next, I focused on the message queue implementation. I was tasked with creating a controller test to verify that the publish message functionality was correctly utilized. Although it took some time to understand the testing process for the message queue, Mr. Peter’s guidance helped me successfully create and execute the test. I then shifted my attention back to the consumer side, working on handling the execution after a message is received. I sought Mr. Peter’s advice to ensure the implementation was correct, and while there are still some adjustments and testing needed, I plan to continue with this next week.

Monday, September 2, 2024 – Last week, I discovered that the issue with not receiving messages was due to a failure in publishing them correctly. I identified and corrected the problem, particularly focusing on the message format in the publish function. After several adjustments, the message was successfully queued in RabbitMQ. I verified this by checking the RabbitMQ management UI, where I confirmed that the message had been published successfully. With the publishing issue resolved, I turned my attention back to the consumer side and made necessary fixes to ensure that messages are read correctly.

In addition, I was assigned a new task to clean up existing data, which had some inaccuracies in previous calculations. The cleanup process involves resetting the data and updating it with the correct values. I started by creating an API query in context A to retrieve the latest invoice and its corresponding data price. This latest data price is then used in a query from context B to update the data in context B.

Once both queries were completed, I moved on to the UI updates. I utilized an existing cleanup page to incorporate a new button feature, given its similar functionality to the current cleanup process. By the end of the week, most of the new button functionality was implemented, although I haven’t fully tested it yet. Testing will be a priority for next week.