Day 100: Cloud, Cli, And Tools

WHAT DO BROS DRIVING SUBARU WRX'S BLOWING SICK VAPE PENS AND AMAZON HAVE IN COMMON?

The cloud.

THE CLOUD? EVER HEARD OF IT?

Let's just get this out of the way: I'm going to recommend you start with AWS because it's more widely adopted.

Azure and Google Cloud are also good, and I do recommend you familiarize yourself with their cloud native offerings. Once again, I am not picking favorites. (And if I was, I'd tell you that I prefer Azure Data Factory to AWS Glue because I think their orchestration tools are better, but you probably have no idea what that reference means.)

This will get you started on the basics: EC2, S3, RDS, and the essential stuff you'll need to know about provisioning security groups, some DNS basics, etc.

You're not going to stop here. There's a second layer.

Add the course to your Udemy Personal Plan (Intro to Cloud Computing on AWS) (https://www.udemy.com/course/introduction-to-cloud-computing-on-amazon-aws-for-beginners/).

AWS SOLUTIONS ARCHITECT

You're going to be doing big data work, and in a perfect world, there would be a cloud architect in your organization that provisions your environment for you.

Plot twist, we don't live in that world. These people create solutions to big data problems. (I do a lot of this in my professional life.)

You'll need to know how a ton of AWS services work, because your code is going to live in a place that you want to control. That's why you should learn how to make you own end-to-end, production ready solutions.

It is helpful to have AWS Certifications, but it's not 100% essential. They're good to have on your resume, but it does cost money to take the tests. You can't just BS your way through them. AWS wants to make it hard to get those badges, because the integrity of their brand depends upon it.

Add the course to your Udemy Personal Plan (AWS Certified Solutions Architect Associate) (https://www.udemy.com/course/aws-certified-solutions-architect-associate-hands-on/).

GRAPHQL APIS

You're going to want to know this.

REST APIs are like ordering a combo meal and getting way more fries than you wanted, or worse, having to place ten separate orders for every item you want. They over-fetch, under-fetch, and get tangled in versioning nightmares as your app grows. Managing a sprawling mess of endpoints? Oh, joy.

REST APIs are good. You should know them. Because you'll be asked to use them… a lot. But there's a better way.

Enter GraphQL, the smarter cousin. It lets you ask for exactly what you need, nothing more, nothing less. One request, all your data. Efficient, right? Plus, no more versioning chaos — GraphQL evolves gracefully without turning your API into a relic. And the best part? It's strongly typed, so you actually get what you asked for. Imagine that.

Add the course to your Udemy Personal Plan (Building GraphQL APIs with Python) (https://www.udemy.com/course/building-graphql-apis-with-python/).

DEVII — AUTO-GRAPHQL FOR SQL DATABASES

Once you learn GraphQL APIs, I want you to try out Devii.

Devii is a magical tool that automatically turns all your SQL databases into GraphQL APIs instantly.

Do not skip the course on the prior page. You still need to understand how all of this works. Devii is a tool that will make you faster, but if you don't know the basics, you're going to be lost as heck.

A Devii personal plan is free.

(Full disclosure, Anthony, the CEO of Devii is a friend, but we're friends because the software community is small, and people with great ideas tend to hang around each other.)

Get a FREE Devii Personal Plan (https://www.devii.io/).

COMMAND LINE INTERFACE (CLI)

Somehow I forgot command line on Version 1.0. I think it's because I use Linux as my daily driver.

Okay, so hear me out — I forgot CLI (command line interface) on V 1.0 of Internet Menace. And let me be very clear — you don't need to know CLI to create applications, become a data engineer, or make dashboards. It's just really good to know and get comfortable with CLI, because you'll use it… a lot. I use it all the time, but the problem is, I forgot YOU don't use it all the time. (I use Linux for my daily driver computer, so I do CLI work constantly. It's part of my workflow.)

Learning command land means learning the basics of Linux, and this is great. Linux is… 98% of all the servers in the world. Apple and Windows are personal. Linux is business.

Add the course to your Udemy Personal Plan (The Linux Command Line Bootcamp) (https://www.udemy.com/course/the-linux-command-line-bootcamp/).

WARP — THE LAZY WAY TO LEARN CLI

Want to not suck at CLI and learning Linux? Get Warp.

Whether you use Windows, Mac, or Linux… Warp is the lazy way to learn CLI. I still think you should learn command line basics, but Warp is going to be your crutch, because if I'm keeping it real, you're already trying to memorize a lot here, and adding more to it is… going to be a ton.

Warp has this little AI assistant that has made me totally lazy, and I love it.

Get Warp (https://www.warp.dev/).

OTHER TOOLS YOU'LL USE OFTEN

APACHE SPARK (AND PYSPARK)

Spark consumes data the same way you eat an entire bag of tortilla chips. Spark does this with something called in-memory processing, so it doesn't waste time running back and forth to the hard drive. (This kinda goes back to what I wrote in those first slides, in understanding RAM versus hard drive storage.)

Spark stores data in RAM. Think of it like this: RAM is the bowl of tortilla chips and smaller bowl of salsa you keep nearby. The bag of chips and jar of salsa are in the fridge — they represent your hard drive. More RAM is a bigger bowl for the chips and salsa, which means fewer trips to the fridge for refills.

PySpark is an API for Spark, so you give it Python, and it gives you back Spark. Then the Cowboys lose and you eat an entire plate of nachos to drown your sadness.

Spark gets you distributed processing, basically taking really big data and turning it into little pieces. This means you have better memory efficiency, etc. PySpark is the Python API for Spark.

Spark is the processor and PySpark is the API.

Add the course to your Udemy Personal Plan (Taming Big Data with Apache Spark) (https://www.udemy.com/course/taming-big-data-with-apache-spark-hands-on/).

FASTAPI

How to make an API, fast. It's all in the name.

FastAPI is the easy button for API development. Depending on whether you took the Udemy or the YouTube course for Python, you learned about Flask. FastAPI takes all those Flask skills and basically says, "Yeah, we're just going to do this now."

FastAPI comes with Swagger. (This is an automatic documentation service.) With automatic interactive documentation, your API doesn't need a boring old README. You get an automatic Swagger UI to show your boss or clients. It looks cool, and you don't have to think about it.

I use FastAPI personally… like once a week? More? It's awesome.

FastAPI is a web framework that is faster than me in bed. Because clearly, our lives revolve around shaving off a few milliseconds from our API response times. I mean, who wouldn't want to impress their boss by boasting about those negligible speed gains? It's not like we have actual work to do, right?

Add the course to your Udemy Personal Plan (FastAPI — The Complete Course) (https://www.udemy.com/course/fastapi-the-complete-course/).

PLOTLY AND PLOTLY DASH

Real world, easy to deploy dashboard and data visualization tools, all with components that are pre made. You'll get to do front end development, without needing to learn HTML, CSS, or JS.

With Plotly and Plotly Dash, you get a Python library that gives you interactive graphs. You don't need to know HTML, CSS or Javascript, and customizing the look, feel, colors, etc is easy.

Remember learning Pandas earlier in your Python courses? Plotly works with Pandas, so if you need to manipulate your data, you can.

You'll also see a ton of components that are pre-made for Dash, so all the "stuff" like sliders, dropdowns, date pickers etc — those are all done for you. (Common stuff we all see all the time, basically. No need to reinvent the wheel.)

Add the course to your Udemy Personal Plan (https://www.udemy.com/course/taming-big-data-with-apache-spark-hands-on/).

STREAMLIT

What is Streamlit? Streamlit is the easy to use app-maker in the world of data visualization. It turns your complex Python scripts into user-friendly web apps with just a few lines of code.

Streamlit is a bit like Plotly and Plotly Dash… but dare I say even less complex and easier to use. (I got into a discussion about this with my coworker Collin, and we're a little divided on which is "better" but they're both good.)

Like Plotly, you don't need to know HTML, CSS, or anything on the front end. All the widgets are pre-made, so things like columns, sidebars, sliders, etc are done for you.

So here's the thing… the Streamlit tutorials on Udemy were bad. And the YouTube videos weren't much better. The documentation on the Streamlit site is the best way to learn. It's not a video, but you've gotten this far. I know you'll be able to figure this out.

Learn Streamlit and make an app (https://docs.streamlit.io/get-started/tutorials/create-an-app).

APACHE DRUID

What does Druid do? Connect to lots of different data sources, in different places, and then ingest them into a single place where you can manage them. The GUI is brilliantly easy to figure out, and it's the backbone for what many of us call an open source "data mesh."

I love this damn program. It's an open-source, high-performance data store designed for real-time analytics on large-scale datasets.

At its core, Druid excels in providing fast, real-time data ingestion, combined with flexible data querying capabilities. It's great for big pipelines with event-driven data. (Think network logging, live sensors, and triggered architecture like computer vision, etc.)

Druid achieves this performance by combining columnar storage, which is efficient for analytics queries, with an architecture that can handle both streaming (Kafka, pub-sub style) and batch data (CSV, XML, JSON files etc) ingestion, thus offering a hybrid solution for real-time and historical data analysis.

Learn Streamlit and make an app (https://docs.streamlit.io/get-started/tutorials/create-an-app).

SCRAPERS (NOT THE OAKLAND KIND)

You're frequently going to need data that exists on web pages and other sites around the web. When this happens, you automate these tasks, and often run them on a schedule, with a scraping tool.

How does scraping work? You've got three options here. (I admittedly only use BeautifulSoup and Selenium, but it seems like Scrapy is useful, so you're going to learn something I don't know.)

BeautifulSoup takes your website and break it into a parsable "web tree" of the HTML tags contained in a site. When you're picking static stuff off sites, this is my go-to.

Then there's Selenium, which allows you to automate things like button clicks, and generally interacts with web pages like a human. If you've ever wondered how I build web-based play-bots, this is one of the ways I did it. (But I'm not encouraging bad behavior.)

• What is web scraping? (https://www.urbandictionary.com/define.php?term=scraper)
• Learn how to scrape — Udemy course on BS4, Selenium, and Scrapy (https://www.udemy.com/course/web-scraping-course-in-python-bs4-selenium-and-scrapy/)

IT'S TIME TO TALK ABOUT MAGE

When I started making this curriculum, I was going to suggest Apache Airflow. In that time period, Mage became an objectively better tool for the job.

You learned Docker, so you'll easily be able to get Mage up and running in a Docker container with their quick start guide.

Once it's up and running, they have some demos you can try out, but here's what I'd do: Find a data set on Kaggle that speaks to you. Then try running a data pipeline with it.

As this guide evolves, I'll add tools I find useful, and I may remove others. In this case, I think Mage is an objectively better way to build data pipelines.

Mage AI — Quick Start Guide (https://github.com/mage-ai/mage-ai#%EF%B8%8F-install).

MELTANO — HERE'S WHY IT'S GOOD TO KNOW CLI

Meltano is made by this company called Arch, and it allows you to to ELT (extract, load, transform… just ETL in a different order) all in one place with the command line.

It's a good data product, and it's either self hosted, or hosted by them. But you can download the open source version and it's a snap to work with.

Read the documentation on how to get started here (https://docs.meltano.com/getting-started/part1/).

RIVERY — THE "EASY BUTTON" FOR MANAGED DATA

Sometimes, after you learn everything, you want an "easy button" for managed data.

Rivery can make setting up data pipelines a breeze, especially once you've got the basics of Python and SQL down. Think of it as the ultimate "puzzle-block-thing" that takes the heavy lifting out of data integration and management. With Rivery, you can build, manage, and monitor data pipelines without writing endless lines of code. Lots of the boilerplate code is done for you.

Its user-friendly, drag-and-drop interface lets you visually design workflows, transforming and moving data with a few clicks. So, instead of hand-coding every step, you can focus on higher-level logic and let Rivery handle the grunt work.

Check out Rivery (once you've got the basics down) (https://rivery.io/).

PUB-SUB: RABBITMQ

(I find it easier than Kafka.)

There are plenty of pub-sub messaging agents out there, and there's no "wrong" tool. There are limitations, pros and cons to anything you choose.

I'm telling you to learn RabbitMQ because I think it's easier to learn. Should you learn Kafka? Sure, eventually, but learn RabbitMQ first.

Read the documentation on how to get started here (https://www.rabbitmq.com/tutorials/tutorial-one-python).

And there's a Udemy course, if you need it. (Try the documentation first, because it's actually something you may be able to dabble in before you need to watch videos.) — RabbitMQ: The Complete Guide on Udemy (https://www.udemy.com/course/rabbitmq-the-complete-guide-with-software-architecture-applications/).