Concept of Covariance, LLMs, and Insights from Industry Experts

Welcome to this edition of the Data Pragmatist, your dose of all things data science and AI. A warm welcome to the 862 new members who joined our community of over 8,000 data professionals since Friday. Thanks for your interest in data science.

📖 Estimated Reading Time: 4 minutes. Missed our previous editions? Catch up on some insightful reads here:

• Poisson’s Distribution & Data Science Reading List.

• R Squared, SQL learning materials and Apple Data Interview Process

Today we are talking about LLMs and few resources to learn more about them. Covariance — The concept which defines how two variables are related is useful not only in work but also in our daily life. Following that is insightful interview with Ariel Herrera who is data scientist at AppFolio. She has great advices for budding data professionals. Do not miss it. Do follow us on Linkedin and Twitter for more real-time updates.

I want to recommend a newsletter by Jakub Chodounsky with great weekly insights for CTOs, engineering managers and senior engineers to become better leaders. Click here to subscribe to this weekly newsletter for free.

— Arun Chinnachamy

📚 Must Reads

• How Does an LLM Generate Text? — This article provides insights into how Large Language Models (LLMs) generate text. It explains the process of text generation, focusing on the decoding and generation of new text based on context. Various strategies like Greedy Sampling, Beam Search, and different forms of Random Sampling are discussed, each with its unique approach to selecting the next word in a sequence.

• Guide to setup analytics in consumer startup — If you're diving into the exciting world of consumer tech startups and wondering how to nail the analytics setup, here’s a quick guide for you! Start by defining clear goals and key performance indicators to understand your users better. Next, choose the right analytics tools that align with your needs, whether it’s for tracking user behavior, crunching numbers, or visualizing data.

• The state of AI in 2023: Generative AI’s breakout year — Report from McKinsey about the adoption of generative AI and what it means for us. Few highlights are about the rapid adoption though it is still early days and adoption is growing strongly. Read through the report for more details but it is a great news for data community.

• How to prepare of Data Engineer interview in 2023 — Getting ready for interviews? Brush up on Data Modeling and System Design, practice Coding with real-life scenarios, and get a grip on basic ML/Data Science concepts.

• Bayesian Causal inference: why you should be excited [Youtube Video]

• Demystifying NLP: A Beginner’s Guide to Natural Language Processing Basics and Techniques. Learn about different techniques and how they can help you in understanding NLP better.

   

🧠 Statistical Concept: Covariance

A Measure of synchronization in variability

Covariance provides a measure of the degree to which two variables change together. If the variables tend to show similar behavior (i.e., increase or decrease together), the covariance is positive. If one variable tends to go up when the other goes down, the covariance is negative. It shows the relation between two or more variables and how they affect each other.

Why Covariance Matters

Covariance is crucial as it gives us insights into the directional relationship between two variables. It helps in understanding whether an increase in one variable would result in an increase or decrease in another variable. However, it does not provide the strength of the relationship nor is it standardized, which is why correlation is often used along with covariance to understand both the strength and direction of the relationship between two variables.

Real-Life Examples of Covariance

Covariance is a versatile concept used across various domains to understand the relationship between variables. Here are a few real-life examples:

1. Finance: In finance, covariance is used to understand the directional relationship between the returns on two assets, helping in portfolio optimization and risk management.

2. Economics: Economists use covariance to study how different economic variables, such as income and expenditure, move together, aiding in economic modeling and forecasting.

3. Healthcare: In healthcare, covariance can be used to study the relationship between different health variables, such as the relationship between calorie intake and weight gain, to inform health policies and interventions.

4. Environmental Science: Covariance is used to study the relationship between environmental variables, such as temperature and ice melt, to understand and model environmental changes.

Navigating Data Relationships with Covariance

Understanding covariance is like having a compass in the vast sea of data analysis. It doesn’t tell us our exact location (the strength of the relationship) but it does tell us the direction we are headed (the type of relationship). It’s a starting point in exploring the intricate dance between variables, guiding us towards deeper insights and more robust analyses.

🚀 Analytical Voyages #3 - Interview with Ariel Herrera 📊

Today, we are talking to Ariel Herrera, an experienced Data Scientist with a demonstrated history of working in the financial services and SaaS industry. Ariel's passion is deriving insights from large datasets and effectively communicating results to a non-technical audience. Summary of our conversation with her is below. You can read the full interview with Ariel here.

1. The Power of Automation: Ariel's career journey took a significant turn when she discovered the power of automation through coding. This realization, triggered by a VBA course, ignited her passion for programming and set the stage for her transition into the data science field.

2. Continuous Learning: Throughout her journey, Ariel consistently sought opportunities for learning and growth. She pursued a Master's Degree in Information Systems, honing her skills and deepening her knowledge in data science and technology.

3. Consulting and Stakeholder Communication: Ariel's experience in consulting taught her the art of effectively communicating complex technical concepts to senior management. This skill proved instrumental in her ability to propose and implement data-driven projects that impact the business positively.

4. Networking and Collaboration: Building a strong professional network was a key strategy for Ariel. Regularly connecting with colleagues and seeking advice from others in her field allowed her to tap into valuable resources and insights.

5. Real-World Problem Solving: Ariel's advice for aspiring data scientists is to tackle real-world problems they are passionate about. Her own journey into real estate analytics exemplifies this approach, where she leveraged data to automate property assessment and shared her work through a public repository and blog.

Read the full interview here.