Learnings

The latest pip 20.3 introduce a breaking change to dependency management https://github.com/pypa/pip/blob/20.3.3/docs/html/user_guide.rst It will  reduce inconsistency : it will  no longer install a combination of packages that is mutually inconsistent . In older versions of pip, it is possible for pip to install a package which does not satisfy the declared requirements of another installed package. For example, in pip 20.0,  pip install "six<1.12" "virtualenv==20.0.2"  does the wrong thing, “successfully” installing  six==1.11 , even though  virtualenv==20.0.2  requires  six>=1.12.0,<2  ( defined here ). The new resolver, instead, outright rejects installing anything if it gets that input. It will be  stricter  - if you ask pip to install two packages with incompatible requirements, it will refuse (rather than installing a broken combination, like it did in previous versions). This is definitely a nice change to make sure...

The field of machine learning engineering is still young. There's so much more to learn from other fields of engineering. Besides trying to make it more accurate, faster and cheaper, we should also pay attention to the 4th management objective: resilience. Resilience is the intrinsic ability of a system to adjust its functioning prior to, during, or following changes and disturbances, so that it can sustain required operations under both expected and unexpected conditions. Since resilience is about being able to function, rather than being impervious to failure, there is no conflict between productivity and safety. fault tolerance, redundancy, unit test, e2e test are some of the terms we think of when it comes to building a resilience pipeline. However,  resilience does not come from the test, it comes from design . With enough tests, we can cover different scenarios, but the pipeline can be still brittle and fragile, and prone to bugs in change. A lot of the failure we observed ca...