Tun Jing Ang's Team Project Journal

In week 1, we form our group, our group member is Illya, Wenyu, Voon Dic, Zeyan and me.

Progress:

We had communicated about our different skills that we learn because we never work before but only me and Illya team together last year.

After the discussion I get the role ScrumMaster, backend lead developer will be more focused on database and server stuff.

Getting the ScrumMaster role is a quite big challenge for me since by having last year experience, I found that scrum master is actually an important task and also quite hard to act as a scrum master. That is because scrum master needs to keep track of the whole project and also like confirming the project is going in the right direction. Besides that, estimating the backlog item is also quite hard and need to use experience to enhance that technic. 

After we form our team, each of us pitch few idea about our team project.

At week 2, we had discussed more the idea we had pitch last week, and after discussing with the lecturer, we reduce the number of ideas to consider from 9 to 4.

And also we start to discuss more detail about the 4 ideas. 

Topic:

• Discuss Project
• Discuss Language/Framework/Tech will be Used
• Creating Sprint Backlog Item

Progress:

On this Friday, Gerry assigned us to scrumwise group, because I am the ScrumMaster in this team, so I started to plan the backlog item and set up the sprint 0 tasks on Saturday noon. At the evening I had a meeting with all the teammate to discuss the point and hour should assign to each task, and any backlog Item we need to do.

 And also at this week, we all work on further investigate about the idea we pitched by ourself and thinking about the requirement and hardware-software needed.

After reading the Mahara page about Aquality V1.0, I start to think about what feature we can enchant or add into it and become Aquality V2.0, I don't have much direction but I thinking about maybe we can automate the project instead of the participant need to go to the river bank and taking the kick sample. Because after I reading the Small Steam Characteristic System (SSCS) Survey Manual written by Atlantic Salmon Group, there is actually some part of the data we can do it by using sensor automatically. 

Issues:

 Because Aquality project is an external project, and we have no reason to communicate to Centre for Freshwater and Environmental Studies ( CFES ) if we don't decide to take the project as our year 4 project, we actually are building the requirement and functionality by my own thought, and my own imagination that is actually not a real requirement and not realistically needed by CFES.

This is Sprint 1 in our team project, and this means we actually have to build something like concept or prototype depends on our own ideas that is pitch.

Topic:

• Decide Project
• Further research on Idea Pitch

Progress:

Because we have 2 different Idea and both of them have the potential to become our 4th-year project, I decide to split our team into 2 part so I assign Illya to further investigate on his Social-Dev idea, and Zeyan and I start to look at Arduino hardware and insect recognition library which is able to use on recognize the insect under the water which is needed to calculate the score by using the method in SSCS survey manual.

I start to google about the insect recognition library, system.

And also install python on my own laptop, playing around with OpenCV library which supports facial recognition.

• Display For OpenCv FaceRegconition Tutorial Output
• DIsplay For AWS Rekognition Tutorial Output

 And also although everyone is having progress, but they don't have the habit to draw their task on the taskboard on scrumwise, so every night after I asking for their progress in daily meeting, I need to go to scrumwise and drag the task board myself

Issues:

I tried to use the demo on web service provided by Amazon and Microsoft, but the accuracy is not good enough to have done the project, I tried to put the insect image to Amazon Rekognition and it replies me that is a spider. I knew that it is because the web service their provide the usually used on objects more general but not specific on the insect.

After that, I start to find some app on appstore or some open source library online to to the recognition system, but I cannot found any library that is specific for insect and the application on appstore is also doesn't recognize the insect image which is record in SSCS survey manual.

I think about maybe we need to build the AI model ourself, but if we want to do that, we need a huge size of sample image to do that.

Additional:

Looking at the burndown chart for this sprint, actually there is a 10 hour workload disappear from it, that is because at the middle of the sprint, we actually decide to give up social dev idea. I don't know is that a right way, but the decision I made is I just remove the proof of concept building for that Idea from the sprint.

Actually, this should be the first week for our sprint 2, but I decided to put it as sprint 1.5 because our team don't have get the clear information when we planing the sprint and start the sprint. The information now we having is not enough to build the actual prototype for the project, that's why I decide to put it as 1.5 and wait for the meeting with CFES and Aquaweb to clarify what we should do.

Topic:

• Clarify Concept, Requirement, Direction Working On
• Research On Django

Progress:

After we having the meeting with Dr Suzanne from CFES, that is a good news that we actually can use this project as our final year project, and also we know that our project concept for automating it is actually a good direction to implement, but there is still many problems we need to resolve.

There is also good to know that they maybe able to provide the training data to building the AI model.

And also after the meeting with Naill from Aquaweb, we had a clear direction when we will focus on host our own server for our AI model, we thinking about we create a API and able to let Aquality 1.0 application sent request to the API and generate the result back to them.

At this week I trying to do some tutorial on Django framework because it is a python web framework that support rapid development.

Issues:

Because our project is like building at the top of Aquality V1.0, but we only have the view on Aquality V1.0 that is be done 2 years ago, we don't have any picture about how is Aquality currently looking and how it function work. But we know that we will work for hardware stuff and trying to get the data, so maybe having our own server won't be a big problem to linking with them

This is Sprint 2 in our project, this week we need to clarify the concept, and produce our paper prototype.

Topic:

• Setting up Backlog Item CheckPoint
• Plan the Goal For Paper Prototype

Progress:

At this week, I change the daily standup meeting time from 8.30 to morning at the empty slot of class, and also I start to set up the checkpoint of each backlog item so what should be considered as done, but there is any backlog item like "research on something" that actually should be considered as a spike instead of features, so I change them from feature to spike because it is not a feature. And also for some backlog item that I don't know what stage should be considered as a finish, I discuss with teammates to decide what should be considered as done.

For example, the research on the sensor that is in charge by Zeyan, I don't have any experience about it, so I discuss with him and setup the checkpoint.

And also the clear goal for this sprint is to produce our paper prototype and present to the lecturer at end of the sprint.

So after the output from Naill and Suzzane, we have a clear objective on this sprint.

This is sprint 3 in our project, and this week we are starting to build up our application.

Topic:

• Planing Release 1
• Setup Server
• Building Database
• Start First Sprint in First Release

Progress:

• Scrum Master

This is a very busy and full of misery week, at the before we start the first sprint, we had a meeting about what is our first release will be like, so after the discussion and looking at the requirement, we have a result of the first release include is like 

1. Build the Arduino Hardware to Collect

• GPS Location
• Ph value
• Temperature

2. Store All the Historical Data Gather by Hardware
3. Building AI Model For Insect Recognition
4. Build a Mobile Application

• Able User Take Photo and user AI Model Recognize insect
• Showing Historical Data

I had emailed to Suzanne about the plan we have for release 1, and she said that looks fine for the current plan.

And also for the first sprint in this release, the goal we have is 

Building The Initial Project

1. Setup Server
2. Setup Database
3. Blank Template for UI
4. Preparation for the AI Model

So we start to build the server and Ui template, prepare for having an AI model.

• Backend Developer

At this week, I start to build the server and database by using the Django framework, and the database we use is auto-generate by developing the models in Django framework.  

After having the database, I had written the function to get all the waterbody data from epa.ie API, currently we have 4845 rows of river data in hour database.

And this is our database schema current having, it is a bit different from the version we have before

Issues:

• Scrum Master

When we start our first release, I really afraid about I had the story point overestimated or underestimated, I know that we would not have a specific right answer for that, but I afraid my estimation is far away from it. 

And also when our teammate starts to going to code, they found that the react-native learning curve is higher than they thought if they want to use other plugin or template directly, for example, Voon Dic spend a lot of time struggle with the dependency error. I aware that might delay our release process, but I don't know how to help him solve the problem, luckily he solves the problems after.

And also when Illya start to building the AI model because Suzanne is still sourcing the image of the insect, so he need to find most of the image online is category himself, so he spends a lot of time to doing that, and I advise him just gather the image for only first group of insect and try to build the model, let's see will it work before going too far for gathering image.

• Backend Developer

When developing the database for application , the database schema is not same with what we expect at the first, but it didn't affect too much, so I change some field from boolean to Charfield, and most of the field to nullable, because when I gather all the data from epa.ie API, there is some error occur when I trying to store the data as boolean, and also some record are not provided for every single waterbody, there are consider as additional data.

This is sprint 4 in our project, and this week we are starting to build up our application.

Topic:

• UI Testing Framework
• Start to Develop API

Progress:

• Scrum Master

At this sprint, we will start to building our UI testing framework, so Wenyu will start to research into "appium" which is an open source test automation framework for use with native, hybrid and mobile web apps. Since I have some experiance about "Selenium" framework during my workplacement, and appium is speaking same webdriver protocol with selenium, so I have a quick overview with her and introduce to how selenium work.

And this sprint Zeyan successfully done his task and start to research into Pubnub to transfer data. And I have a discuss with him about how to transfer data to our server since the pubnub is just a channel publishing data. I ask him to research into how pubnub publish data and maybe can sent some http request. And he is researching on it.

• Backend Developer

This sprint I am trying to developing the API with return the river data and able to let frontend to connect to. After researching into django restframework. After I build a simple API publishing data, I change the database from mysql to PostgreSQL because django support postgresql-gis maturely and support many function. So after the migratetion, I successfully modified the field longitute and latitute to a location coordinates. 

Issues:

• Scrum Master

During this sprint, I start to release that even though just building 1 group of insect is still a large amount of work to illya, so after a quick discuss, we decide instead of building 1 group of insect, he start to build 1 class insect to display a simple model demo before it going too late.

• Backend Developer

During developing the API , the API should accept locations coordinates or locations name to return the rivers list that is nearby. There is a build in function that support accept locations coordicates and return list of river, but currently got no idea about how to process the locations name, maybe have to research into google maps api.

This is sprint 5 in our project, and this week we are starting to build up our application.

Topic:

• Connect Frontend And Backend

Progress:

• Scrum Master

At this sprint our sprint goal is connecting our frontend stuff with our backend stuff together using API, since we already having our backend part, front end part and also hardware device working so far, we want to putting them together.

So the task in this sprint for us are all about setting up API ,API Testing and also frontend fetching data to server.

And also because the Chrismas break is coming, so we decide to lower down our scope by remove the whole help and navigate helper in our first release scope. Hope that won't be a problem.

• Backend Developer

 After Zeyan setting up the pubnub to publish the data, I start to building the API to collect hardware data and also save it to database. And after discuss when Zeyan we success accept the data from pubnub and store in into database.

And also I modified the river API that make it accept the location string and return the river list. The operation inside it is I fetch the location string I accept into google map API, and that API will return a geojson to the server and it will contain a location, I using that location as the point.

And after setup the local API, we need to host our server online, and the server we decide to use is AWS Elastic Beanstalk.

There is an issue when I trying to host it.

Issues:

• Scrum Master

During this sprint we trying to connect backend api with front end request, but the problem is I wrongly estimate the work progress. So even though Wenyu, Illya, and me done the API that able frontend connect into, but they actually not yet ready or not yet reach that point to connect the API. So the work is delayed.

• Backend Developer

Because we using a gdal package when I setting up local server.And the problem is when we trying to deploy our server to amazon elastic beanstalk, it doesn't support the gdal package. So there is a problem it fail to find the library, and a github issue is opened. I fail to fix it until the end of the sprint, so will further investigate into it on next sprint.

This is sprint 6 in our project, and this week we our sprint goal is apply AI model to our appication.

Topic:

• Deploying AI Model To Our Project
• Setup Hardware Data API

Progress:

• Scrum Master

This is the sprint before the first release sprint, so some of our teammate actually have a bit pressure about the progress about it, but fortunately we had a break during the Chrismas and we have a good time on it. During the Chrismas break we had stop our development task and doing some research during which relate to the issue we met before the break. 

The main task I done in this sprint is keep communicate with Lee and Illya about the format for AI model data and also how the project architecture will be working at that part.

• Backend Developer

After the testing before the Chrismas we know that the hardware data is able to sent in to the backendserver. So at this sprint I communicate with Zeyan and confirm the data format also setup the API that letting the frontend connect to.

For the issue last sprint about GDAL library , I decide to giveup using the gis system provide by it and directly use the function to calculate the distance to filter it ourself instead of using django build-in function.

And also for this week I work with Illya to help him deploy his AI regconition server to AWS.

 Beside that I also build up the Login & Register Authetication System that let client able to validate the account and login.

Issues:

• Backend Developer

The issue is we didn't expect the AI model will consume a very large amount of storage when it trying to process the image, so after we deploy it to our free tier amazon server, it actually overloading our server and make our server down and unusable.

This is sprint 7 in our project, and this week we need to finalize our project and ready for the first release.

Topic:

• Finalize Project

Progress:

• Scrum Master

This is the sprint before the first release, our teammate who in charge of frontend become more pressure and pressure because they are going to show their product at that time, And also we seem a bit behind our progress and they are some of the tasks we might not able to do at the end of this sprint, but after the discussion, we decide to no lower down our scope but let Wenyu go to help the development of the mobile application.

• Backend Developer

Continue from the last sprint issue for the server problem, we decide currently split up the server into our backend server ( process API ), and another AI Model Server (that process image). In this case, our API still running and the AI model server doesn't affect the API server.

Besides that, I working on storing the hardware device data save to the database as well. Since Zeyan finalizes the data format sending in, I produce the API let the client application able to communicate with it.

And also Wenyu and Lee is working on the sample history page, I need to setup the API that able client to save the sample record also fetch the sample record from the server. The API now is filtering the data using the username.

And to help the user get the same insect list, we also build a insect API list that contain all the insect that we concern about during the sampling process and the client side can directly interact with it.

This is sprint 8 in our project, and this week we need to finalize our project and ready for first release.

Topic:

• Fixing Issue

Progress:

• Scrum Master

This is a extra week for our release 1, we actually have a functionally working application but there is some bug and UI is not good enough to deploy yet. So this sprint we actually taking a break and fixing the bug that remains before.

• Backend Developer

At this sprint, there is no legacy issue from last week to me this week, so I am working on setting up remote storage for the webserver by moving the local storage to Amazon S3 remote storage. Now we using AWS S3 bucket to serve the website static & media files.

•  Meeting with Customer

This week we had a meeting with Suzanne at Monday and having some feedback from her about 

This is a transition period between release 1 presentation and release 2. I had sent an email to Suzanne query about the task for our release 2. So while we waiting for her reply, we decide to open a sprint that working on our presentation. 

Topic:

• Preparing Presentation and Slides

Progress:

At this week, we are working on preparing the presentation and preparing about the demo.

We building our own part of demo and having some screenshot and video on group mahara page. Group Mahara Link

Then we have rehearsal about our demo for few times even though Wenyu is now having a flight to China.

We work on the presentations slides we having and also I build up the database schema that we currently having. 

And also I produce a excel file that contain all the API that able access to.

API List Excel File

This is the first sprint in our release 2, luckily we having a meeting with Dr. Suzanne just before the presentation and get some feedback from her, so we having some objectives that we should achieve in release 2.

Topic:

• Become the TCP of the Team
• Setup Our Own Admin Website

Progress:

• Scrum Master

After the feedback from Suzanne, we decided on our Release 2 Plan that will contain 

1. 1. Improvement for AI Model that calculates the insect tail to clarify between mayflies and Baetis.
   2. Adding Group 6 Insect into our application which actually disturbance.
   3. Having the feature allows the user to upload the image for an insect that lets the admin able review it.
   4. Having the feature allows the user to upload the image for the river that lets the admin able review it.
   5. Help Function That allows the user to check the documentation when confusing.
   6. The survey step for the user to take down the river site details
   7. Safety Guide and Term & Condition step to help users keep safe.
   8. Admin Site to review all data.
   9. Report problem function when AI Model goes wrong.
   10. And also some detailed improvements.

And after the user feedback, many of them are talking about it's hard to navigate through our apps, so Lee makes out the decision that we will change the flow of our application in order to help the application.

So in this sprint, I had been keeping talking with everyone and asking about the progress and also the protocol for the API they interact with.

• Backend Developer

Since the last stage of the team project actually, Illya is almost spending all his time on the AI model, and I actually work 80% of the backend stuff including setting up the database, creating the API, and also hosting the website. So in this sprint I trying to give back some tasks to him to make him actually act as a backend lead in this project. I having a discussion with him and talk about I am going to leave most of the API stuff maintenance task to him, and I will focus on working admin site that able admin to review all the data.

At this sprint I had build the activity map that able the admin view on the record using Google Maps API.

This is sprint 11 in our team project, and because we plan to left 2 weeks as our last sprint, we decide to make this sprint become a 1-week sprint.

Topic:

• Let user taking the control
• Showing Details On The Map

Progress:

• Scrum Master

After the discussion about the accuracy of our AI model is quite low, so we decide to let the user to taking the control on the feedback of the AI Model, so allow the user make to modifed the number of insect AI model detect, which can be save directly into the sample. And also we decide having a report problem page to allow user to report the issue when they found that the AI model is wrong.

• Backend Developer

In order to achieve the task I mention just now, we create a API that accept and store the report problem into database.

It accepts the post request in the form with key report_image_path ( which is the actual image), report_problem (this is the title), and also the report_problem_description.

Beside that, I continue building the activity map since last sprint I had showing the marker in clustering on the map, but it not showing the data yet. So this week I modified it and make all the marker can having a infowindow, which showing some of the details on it. After clicking the more info on it, it will show up a modal with more details with it.

This is the sprint for our release 2, we start to finalize our apps and ready for the release. And there is good news that we had a meeting with Dr. Suzanne this sprint then we having some feedback from her.

Topic:

• API for Image Upload By User
• Permission Level On Admin Site

Progress:

• Scrum Master

During this sprint, actually everyone clearly knowing what we want, and what feature we want to implement, but when Illya is developing the API for the image upload, Wenyu notice that the function Illya modified to accept the image is using JSON format but JSON format is not able to accept the image. So I start to research it and ask Illya if he able to modify it to change it to accept base64 format for the image otherwise the mobile application will just only send the location of the image locally.

• Backend Developer 

At this sprint, I start to work on the admin site that able to let the admin access and review the data. Django actually provides a built-in admin site that can access by the user and it provides some basic functionality. I used the built-in website and create user group and user level that limit the user can only access a specific region of data if he is in role staff, but the admin role can access all the function inside the website. After that, I found a framework call simple-ui which is building on Django default admin site which is more beautiful. So I apply it to our project.

Then I start to work on the sample record displaying inside the database since it actually having someone to many relationships inside the database, so I need to include all of the data and make it display at one goal when the user is looking on the sample data. 

This is the first sprint in our release 3 and also a full sprint that is inside the holiday without the school. We decide to take a weekend break, plan and start our release and sprint on Monday.

Topic:

• Planning Release 3
• Function On Admin Site

Progress:

• Scrum Master

When the team project actually going to the end, I start to confuse what is important, and start to worry about is our application is really ready for the expo. But even though I am confuse, but we still need to plan and start the release. So we start to discuss the review from the lecturers and also from user feedback, we pick up the task that we think should able to do during release 3 since I decide to cut down the velocity for our team because I think we are overloading ourselves during the team project.  We decide to fix most of the bug and polish some of the screens and details that in our mobile application and build some additional features on our admin site.

And here is the initial plan.

1. Fix Duplicate Error For Insect Input
2. Modify River Sample Model and make it saving current location
3. Modify the logic calling openweather API when the mobile application startup
4. Modify the review screen after taking the sample
5. Adding apps logo on the mobile application
6. Fix Ai Model 500 Server Error
7. Setup the pipeline to deploy the apps to google play store
8. Admin Site add Filter & Export Data
9. Admin Site add let Creating multiple users in one goal.

• Backend Developer

In this sprint, I working on adding filter and export data on the admin site and also adding the import register function. So after some of the research, the filter data is actually built-in by Django, just need to calling it and put it into the admins.py while export data we need to create our own function and create a custom action on admin site to do it.

And about the import register function, actually I am creating a new page like the activity map, and embed it inside the admin site. Now the import function accept a csv file  which ignore the first line, and assume first column is username, second column is email address. It will return the message about create status and also password that generated.

This is the final sprint for our team project, and we start to finish up the project.

Topic:

• Simulating a Kick Sampling
• Slightly Modified on the API Return

Progress:

• Scrum Master

At this sprint, we still have some small modification on the application but mostly is on UI, like Lee is trying to add our team logo and apps logo in the application, and Wenyu is deploying the application to google play store.

Besides that, we are planning to having a simulation kick sampling to test how our application will perform in the site.

• Backend Developer

At this sprint, as a backend developer, I just modified the API for river to make it return the distance to the river as well when the client query on it. Beside that sort the sample history by the data it captured.