Beyond Chaos

Recap AWS re:Invent 2020

2021-03-27T00:00:00+00:00

I’m glad that people like my post How to Become a Google Certified Professional Cloud Architect. Google has sent me a hoodie and a mask for participating in their exam. However, my employer Indeed selects AWS as its Preferred Cloud, so these gifts won’t stop me from pursuing certificates from the cloud leader. Amazon hosts re:Invent annually to announce AWS updates. The just ended 2020 one would be the best chance to catch up with their latest progress.

This is the first time hosting the whole event online for safety. It’s both unlikely and unnecessary to watch all sessions of several thousand minutes in total. Here are my top 7 updates worth noting:

1. S3 supports Strong Read-After-Write Consistency.

In other services, like DynamoDB, AWS already offered strong consistency with a price premium. But this time, Amazon S3 delivers strong read-after-write consistency automatically for all applications. You don’t need to write code to maintain “object”¹ consistency any more. Companies requiring state synchronization like Dropbox could benefit a lot from this. This is nothing revolutionary since Google’s counterpart Cloud Storage already has this feature, but still, engineers could save their brains for other top priorities.

2. EC2 Mac instances.

Mac mini in the chassis

This is a great news for those who want to develop apps for Apple ecosystem on AWS. I have to say Amazon really surprises me with their implementation of Mac instances. They don’t build something like a bigger Mac Pro, but squeeze a Mac mini into their chassis. I think this is Amazon’s practical engineering style different from Google’s shining one.

3. Fully managed container and serverless applications deployment service: Proton.

According to Berkeley’s prediction, serverless would be the future for the next 10 years’ cloud computing. With Proton, serverless and container-based applications could automate and manage infrastructure provisioning and code deployments with lower effort. It’s kind of like a “kubernetes” for more than just containers. Let’s hope it will not be so complex like kubernetes that Google has to make an Autopilot feature to simplify its usage.

4. Babelfish for Aurora database to understand Microsoft SQL Server.

Microsoft Azure, not Google Cloud Platform, is AWS’ main rival. They are very good at transferring their Windows/Office/SQL Server users into their cloud customers. AWS wants to defend their market share, but they don’t have Android or G Suit like Google, so now their potential customers could lift their programs and shift to AWS with no extra cost. They are about to make Babelfish open source so everyone could fix bugs and keep it updated with SQL Server.

5. SageMaker Feature Store.

Machine learning is now literally everywhere. It’s like you can’t say your products use state of the art technology if there is no “learning” in it. AWS announced their “App Store” for reusing machine learning models via SageMaker. I don’t think you have to integrate this into every product, but it could eventually benefit your business if this ecosystem does give birth to some really amazing feature.

6. AWS Fault Injection Simulator.

Netflix is famous for their chaos engineering. The only way to test your system resilience is killing part of your services and see what happens, because this is what reality is. You can now use an official tool instead of some framework from third parties, which is a safer way to keep your system safe.

7. Fully managed Grafana, Prometheus, CloudShell.

“Cloud Native” tools to visualize your operational data, monitor your system and manage your resources. I’m kind of surprised that AWS didn’t have a cloud shell like GCP from the very beginning, maybe this is because they don’t own a web browser like Google.

I hope seven takeaways could fit in your memory since this magic number is said to be our information processing capacity at a time. Some of the products mentioned above are in preview state and not ready for business, and you could follow their links for more details. I will present my journey to AWS certificates in following articles soon.

[1] S3 is object storage for files, images, etc.

Cross-posted on Medium.

The Missing Manual For Github Pages

2021-01-03T00:00:00+00:00

Blog — a living fossil on the internet. Nowadays, people are immersed in short videos, let alone images and tweets. For writers striving for impact, social publishing platforms like Medium would be a better choice. But in the eyes of engineers, their code syntax highlighting sucks, and provide only limited config options. That’s why I can’t give up on a stand-alone blog. However, renting a VPS or using a cloud platform is breaking a butterfly on the wheel. Github Pages is just fine, though its document is a mess.

Launch your blog with a remote theme

First of all, you definitely want beautiful Jekyll themes on Github instead of boring default themes. But the official “Creating a Github Pages site with Jekyll” leads you down the wrong path — You don’t need to create a new site in your repo using Jekyll or bundle because this will generate useless stuff from a default theme. You only need 5 steps:

Create an empty repo named *.github.io* on Github as **your blog repo**. [Don’t fork a theme repo as your blog¹](https://matthiaslischka.at/2018/12/03/github-jekyll-best-practice/).
Choose a Jekyll theme on Github. Remember, it should be a gem-based theme².
Fork it so you won’t be surprised when the author’s done something crazy³. Now you get a */* as **your theme repo**.
Copy _config.yml, Gemfile and index.html from your theme repo to your blog repo. Add remote_theme: <your Github username>/<theme name> in _config.yml; add jekyll-remote-theme to the plugins section of _config.yml; add gem “jekyll-remote-theme” in Gemfile.
Add a _posts folder in your blog repo for articles. You can add markdown files named like 2021–01–03-Hello-World.md.

Now a personal blog is served at *.github.io*. If you want to specify a custom domain, go to your blog repo’s settings on Github.

Customize your blog and test it locally

In general, the above steps would be enough. But every theme may have its particular problems, and you need to customize the blog title and other stuff soon — push to Github every time for testing is time-consuming.

You can clone your blog repo to the local computer, and then install bundler to manage gems locally:

Executing bundle exec jekyll serve in the blog repo could preview the blog on your laptop.
If there is an error, fix them according to the error message.
Edit _config.yml to configure options.
Create folders and files with the same name as in your theme repo to override them as you please.

[1] It will be very hard if you want to change that afterward. And it’s better to separate your content from the presentation.

[2] Usually, this kind of theme has an assets folder to store CSS and Javascript files.

[3] You can always sync updates from the original repo.

Cross-posted on Medium.

How To Become A Google Certified Professional Cloud Architect

2020-12-05T00:00:00+00:00

After about two months’ preparation, I got my Professional Cloud Architect certificate this week. This experience gives me a lot on Google Cloud Platform(a.k.a. GCP) and architecture design principles. The exam is not difficult at all for a veteran engineer, and I haven’t tried GCP before in my life. I’ll explain how to pass it quickly, and how you can get the most of it rather than just a certificate.

Why GCP Cloud Architect

According to Canalys’ report, Google only possesses 7% of the whole cloud market in Q3 this year, comparing to Microsoft’s 19% and Amazon’s 32%. Amazon’s AWS stepped into this arena first, and Microsoft Azure expands its share from Enterprise customers who using Windows and Office. It must be very difficult for Google to catch up, so why choose a GCP certificate now?

Global Knowledge said in 15 Top-Paying IT Certifications for 2020 that GCP Cloud Architects earn an average of $175,761 in the U.S., about $26,000 more than AWS Certified Solutions Architect per year. A possible reason would be the supply-demand relationship: Engineers with Cloud Architect certificate aren’t enough in the market. You may choose AWS if you want to fit in more positions, but a GCP Cloud Architect certificate may take your career to the next level.

I choose GCP for three more reasons. First, it’s Google who published three papers(GFS, MapReduce, Bigtable) opening our minds to large-scale distributed systems. Almost every engineer who cares about scalability would be familiar with these concepts. This makes you feel natural about products on GCP rather than AWS. And Google is always standing on the frontier of technology, you could anticipate that the cutting edge research would soon be integrated into their platform, keeping yourself competent in the industry.

Second, you definitely want to take a remote exam rather than going to a test center spending hours with other candidates, while the coronavirus is still not under control in most of the world. However, Amazon didn’t allow candidates in China, Japan, Slovenia, and South Korea to take online proctoring exams when I decided to get a cloud certificate months ago. In October, they finally increased availability for these people. For candidates now, you can pay $300 to take a 3-hour AWS online test or $200 for a 2-hour GCP one. But be careful, an AWS certificate is valid for 3 years while a GCP one for 2 years. You can take a renewal exam for half a price when they are expired.

Last but not least, GCP Cloud Architect is more on high-level principles while their Cloud Engineer or Cloud Developer is more on platform details. It’s both easier and more meaningful if you can focus on the big picture and skip the command line parameters at first. After all, you can always refer to the manual, why bothering to pay for it.

Which resources are handy

There are millions of search results throwing tons of references to you. However, I think you don’t need to become a library to get a certificate. When you register an account on Google Cloud Certification, they would provide you some materials to prepare. I found the most valuable resource is the ebook named Official Google Cloud Certified Professional Cloud Architect Study Guide. You can purchase it for $19.99 with their coupon. This book covers every aspect of the exam, arming you with exercises after each chapter and an assessment test of 25 sample questions. The content is more concentrated than Google’s own documents so you could get familiar with the exam quickly. Also, you can register a new account on Coursera to access 6 courses for free in the first month. These courses would be more fun than plain text in the ebook, and you can get your hands dirty on Google Cloud Platform to solve the quizzes. Teachers would teach you how to design a system considering the requirements besides details of the platform, which would benefit your career in general.

After finishing the reading and lessons, you will find official sample questions and practice tests on whizlab really helpful before the exam. Use these tests to get familiar with question forms and UI. There are only two types of questions: multiple choice and multiple select. You even get hints of how many answers to be chosen. In my experience, whizlab’s practice questions are more difficult than the real ones in my exam. You can use them to identify your weaknesses and don’t get entangled in specific details or the score you get. I got no more than 80% accuracy in all these tests, which is said to be the pass line, yet still got my certificate.

Which parts are tricky

The most important part is to design a solution meeting both business and technical requirements. This normally involves three sample case studies offered by Google: Mountkirk Games, Dress4Win, TerramEarth. They are also typical architecture models in the current industry: online game craving for scaling both up and down, e-commerce website with high availability and transactional deals, IoT network aiming for high volume stream analysis. There are different ways to look into their problems, which could bring solutions from different domains.

To be specific, you need to consider GCP computing instances, storage services, big data analysis tools, load balancers, CDN, network infrastructures, security measures, legal compliance, data migration plans, etc. For experienced engineers, just get familiar with Google Cloud Products and choose from them based on the situation. If you haven’t made trade-offs that much, it will be very helpful to read the ebook and answer questions to figure out why they are like that.

Among all of these, network, security, legal compliance would be relatively unfamiliar to developers, because we are not even aware of them in our daily coding, testing, deploying and monitoring process. Site reliability engineers, security engineers and lawyers handle them for us, so this may be a good chance for you to know what an Architect needs.

What’s next

Now you are a Professional Cloud Architect, what does it mean? It could mean almost nothing because you don’t even need engineering experience to nail the exam, many salespeople passed this. It’s like you can’t say you are a native speaker after passing some language exam. However, you could become a real architect if you keep using what you learned in real-life engineering. Assessing business and technical requirements, design your architecture for scalability and reliability, choosing or even crafting services to fulfill them, monitoring your system for debugging and performance tuning, working with other colleagues to prevent security and legal problems, are what an architect means, not a digital certificate.

Large scale distributed systems are the only solution for worldwide companies. You need to dive into the Google Cloud Products figuring out why they could face the challenge. CAP Theory, relational databases, NoSQL, stream computing, consistency, etc. Official documents would be your best friends now. Why spanner could be a worldwide relational database and Cloud SQL can’t? Why BigTable could handle intense concurrent writing? Why Cloud Function could be so lightweight making serverless real? You will get the most of this certificate if you keep hungry for these non-trivial problems and answer them. Even better if you can solve problems that Google can’t.

Cross-posted on Medium.