Learnings

  1. Focus on the benefit instead of the technology It is useless when using technology because of the technology. We need to focus on the benefit that technology provides. For examples: If it can reduce the technical complexity and speed up the development process for the team If it can make the system more reliable If it can simplify and automate to reduce cost 2. Design from a service/API perspective instead of resource or technology Design from service/API establishes a top-down approach. This helps to distribute the vision to a wider team and enables collective system improvement over time instead of fragmented. 3. Use mainstream and mature technology Technology needs to be selected wisely so that less effort needs to spend on evolving. Use more mature technology instead of technology just because you know it. Don't reinvent the wheel. All the self-invented wheels or non-popular technologies will eventually get replaced by mature open source software, as they have huge support...

In a developers journey, they usually go through a phase where they are reluctant to use other people’s code. They rather implement it themselves if not under the project timeline’s pressure. This is not because they believe that they can do better, but rather they will not be bounded by the constraints that are introduced by the added dependency. At the end of the day, writing code is usually easier than reading. After this phase, develop may become more goal-oriented and not so focus on building packages themselves. Then selecting an open-source package/project becomes important. Below are some of the principles I apply when selecting an open-source project choose active package/project choose package/project where its lead has good communication choose the package/project that has a single focus Choose active package/project When importing a package/dependency, we are essentially starting a contract with the package. Future features and bug fixes update will inevitably affect the pr...

There are two classic way to handle concurrency in go, WaitGroup and Context What is WaitGroup WaitGroup wait for multiple goroutines to finish content_copy func main() { var wg sync.WaitGroup wg.Add(2) go func() { time.Sleep(2*time.Second) fmt.Println("1st task done") wg.Done() }() go func() { time.Sleep(2*time.Second) fmt.Println("2nd task done") wg.Done() }() wg.Wait() fmt.Println("Finish") } The program only finishes when the 2 goroutine finished. Otherwise, it will wait for it. This is useful when we want a program to wait for all tasks to finish. However, sometimes we want to actively cancel a goroutine instead of wait for it to finish. An example can be monitoring. We want to exit monitoring instead of wait for it to finish (it will never finish). We can use channel for this usecase. Channel we can use channel + select to repeatedly checking on a global variable to notify the end of a process content_copy func main() { stop := ...

https://github.com/robfig/cron Cron entries are stored in an array, sorted by their next activation time. Cron sleeps until the next job is due to be run. Upon waking: it runs each entry that is active on that second it calculates the next run times for the jobs that were run it re-sorts the array of entries by next activation time. it goes to sleep until the soonest job. Example content_copy c := cron.New() c.AddFunc("0 30 * * * *", func() { fmt.Println("Every hour on the half hour") }) c.AddFunc("@hourly", func() { fmt.Println("Every hour") }) c.AddFunc("@every 1h30m", func() { fmt.Println("Every hour thirty") }) c.Start() .. // Funcs are invoked in their own goroutine, asynchronously. ... // Funcs may also be added to a running Cron c.AddFunc("@daily", func() { fmt.Println("Every day") }) .. // Inspect the cron job entries' next and previous run times. inspect(c.Entries()) .. c.Stop() // Stop t...

Deployment Deployment is a process of running an application on a server or pipeline Given a model object, how to make it part of your application. It consist of components: Deploy model Low level solution Off the shelf solution (High level solution) Serving model Batch serving Low latency serving Batch Serving Pros: Very flexible Easily set up, you can put the model object/data anywhere Cons: High latency Process: Read input Run inference Return output Low Latency Serving Pros: Low latency Cons: Rigid schema Infrastructure setup overhead Only handles relatively simple model? Complexity: Communication interface setup Format (grpc/json) Security Multiple model deployment management Hybrid Still have the same complexity as low latency but with More accurate model Feasible infrastructure Input doesn’t need to contain all features Packages To properly productionise deploy and serving model, we may want to consider requirements related to Model versioning Model metadata Model artifact Lo...

  Git Everyday commands and workflow Git internal Everyday commands and workflow I usually keep a file called  command  in the parent directory to remind me of things I don't want to remember content_copy git checkout -b feature/new_branch # create new branch feature/new_branch from current branch git commit -m 'message' # new snapshot git commit --amend # append current change to previous snapshot git pull --rebase origin develop # merge develop branch changes to local but put your changes on top git push # sync local changes to remote git push -f # force sync local changes to remote git reset HEAD~ # undo last commit git merge --continue # after solve conflict git merge --abort # abort merging git stash apply git checkout stash -- . # force stash apply Do the setup once, hopefully only once content_copy git config --global credential.helper store git config --global user.name "Your Name" git config --global user.email "you@email" just use UI for the ...