Gatsby Starter Blog RSS Feed

The mystery behind the `pycache` folder

Tue, 18 Feb 2025 23:46:37 GMT

If you’ve been diving into Python, you might have stumbled upon a curious directory called __pycache__. Let’s demystify it a bit!

Image source: Reddit

What is `pycache`?

Simply put, __pycache__ is a special folder created by Python. It stores the compiled bytecode files (those ending with .pyc) of your scripts. These are optimized versions of your code, meant to speed up the execution process.

Why is it used?

The main reason for __pycache__ is to boost the performance of your Python programs. When you run a Python script, it gets compiled into bytecode. By storing these bytecode files, Python can skip recompiling on subsequent runs, leading to faster startup times.

What about its naming convention?

Inside __pycache__, you’ll find .pyc files with names like module_name.cpython-version.pyc. This structure ensures compatibility across different Python versions. For instance, example.cpython-39.pyc means it was compiled with CPython version 3.9. There are other Python distributions specific for different use-cases:

CPython

CPython is the Python reference implementation, the standard version that all other Python incarnations look to. CPython is written in C, as implied by the name, and is produced by the same core group of people responsible for making top-level decisions about the Python language.

Anaconda Python

Anaconda, produced by Anaconda Inc. (formerly Continuum Analytics), is designed for Python developers who need a distribution backed by a commercial provider and with support plans for enterprises. The chief use cases for Anaconda Python are math, statistics, engineering, data analysis, machine learning, and related applications.

ActiveState Python

Like Anaconda, ActiveState Python is created and maintained by a for-profit company—in this case, ActiveState, which markets a number of language runtimes along with the multi-language Komodo IDE.

PyPy

A drop-in replacement for the CPython interpreter, PyPy uses JIT compilation to speed up the execution of Python programs. Depending on the task, the performance gains can be dramatic.

and many more.

When does it get created?

__pycache__ pops up automatically when you run a Python script that imports another module. If you’re running a standalone file, it doesn’t show up. If the relevant .pyc files don’t exist yet, or if the source file has changed since the last run, Python will compile the source code and store the bytecode in __pycache__.

Should You Delete pycache?

You may wonder whether it's safe to delete the "pycache" folder. In most cases, it's entirely safe, as Python will recreate it as needed.

However, there are some scenarios where you might want to keep it:

Development: During development, it's convenient to keep "pycache" to benefit from faster module imports. Deleting it won't harm your code, but it might slow down imports temporarily until Python recreates the bytecode files
Deployment: When deploying your Python code, you can safely omit the "pycache" folder. It won't affect your production environment, and it can save space.

So, __pycache__ is a behind-the-scenes helper in Python that enhances the performance and compatibility of your code. It automatically stores compiled bytecode, making your Python programs run faster.

Think next when you write that for loop!

Wed, 21 Feb 2024 23:46:37 GMT

If you've worked with Python before, you've probably come across this code snippet in various forms.

numbers = [1, 2, 3, 4, 5]
sum_result = 0

for num in numbers:
    sum_result += num

print("Sum using for loop:", sum_result)

The for loop is a crucial tool in Python for iterating over sequences, and it's widely used. It's great for cases where you want a clear and easy-to-read iteration structure, and its benefits have made it popular and useful.

Readability: for loops are generally easy to read and understand. The structure of a for loop makes it clear that you are iterating over a sequence (in this case, a list of numbers).
Simplicity : For simple iteration tasks, a for loop provides a straightforward and concise way to express the logic without unnecessary boilerplate code.
Versatility:for loops can iterate over various iterable objects, including lists, tuples, strings, and more. This versatility makes them suitable for a wide range of use cases.
Control Flow: The break and continue statements can be used within a for loop to control the flow of execution, allowing for more complex logic when needed.
Index Access: In addition to iterating over the elements directly, you can use the enumerate() with a for loop to access both the index and the element in each iteration.

The problem

However, even though it's a simple and essential building block for developers, it can sometimes make your code inefficient. For example, let's take a look at this case:

numbers = [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 28, 29, 31, 33, 35, 37, 39]

first_even = None
for num in numbers:
    if num % 2 == 0:
        first_even = num
        break

print("First even number (using for loop):", first_even)

In this specific example, if numbers were a very large list, a for loop would continue iterating over the entire list.

The solution: `next()`

Now, let's find the first even number using next()

first_even = next((num for num in numbers if num % 2 == 0), None)
print("First even number (using next()):", first_even)

Using next() can lead to better performance and resource utilization in situations where you only need the first matching element. In the example provided, next() efficiently finds the first element that satisfies the condition and stops iterating. This can be advantageous when dealing with large datasets or when the iteration process is time-consuming.

Additionally, the use of next() can result in more concise and expressive code, especially when the logic for finding the element is straightforward. It eliminates the need for a separate variable and an explicit break statement, making the code cleaner and easier to read.

Overall, while the for loop is a fundamental tool in Python, considering alternatives like next() can improve code efficiency and readability in specific scenarios.

Conciseness: Using next() often results in more concise and expressive code. It allows you to express the iteration and condition in a single line, which can be easier to read and understand, especially for simple cases.
Efficiency: next() stops iterating as soon as it finds the first element that satisfies the condition. In contrast, a for loop would continue iterating over the entire iterable even after finding the desired element. This can be more efficient, especially with large iterables, as it avoids unnecessary iterations.
Readability: For simple conditions, using next() can make the code more readable by eliminating the need for explicit loop constructs and breaking out of the loop. It can be particularly useful when the goal is to find the first occurrence that satisfies a condition.
Lazy evaluation: The generator expression used with next() allows for lazy evaluation. It generates values on-the-fly and stops as soon as a match is found. This can be advantageous when working with potentially infinite iterables.

Lazy evaluation

Lazy evaluation is a programming concept that involves delaying the evaluation of an expression until its value is required. In the case of Python and using next() with a generator expression, lazy evaluation means that the values are generated dynamically and only when they are actually needed.

With lazy evaluation, the generator expression is not executed immediately upon definition. Instead, it is executed step by step, generating the next value in the sequence as it is requested. This approach is particularly useful when working with iterables that could potentially be infinite or when there is no need to generate all the values upfront.

By employing lazy evaluation, you can optimize resource utilization and improve performance by only generating the values that are necessary at any given moment. This can be especially advantageous when dealing with large datasets or when the generation process is time-consuming.

Overall, lazy evaluation provides a flexible and efficient way to handle sequences of values, allowing for on-demand generation and consumption of data.

To illustrate lazy evaluation, let's consider an example.

Using for loop:

counter = 0
infinite_squares = []

for num in range(1, float('inf')):
    infinite_squares.append(num ** 2)
    counter += 1
    if counter == 5:
        break  # Breaking to prevent an infinite loop; not recommended in practice
print("First 5 squares using for loop:", infinite_squares)

Using next() for lazy evaluation (finite extraction from infinite iterable):

infinite_squares_generator = (num ** 2 for num in range(1, float('inf')))
infinite_squares = [next(infinite_squares_generator) for _ in range(5)]

print("First 5 squares using lazy evaluation:", infinite_squares)

In this case, we have a generator expression called infinite_squares_generator that represents an infinite sequence of squares. We use the next() function to extract the first 5 squares from this infinite sequence.

With lazy evaluation, the generator doesn't attempt to calculate all squares upfront. Instead, it produces values as they are requested, allowing for more controlled and efficient handling of infinite sequences.

Lazy evaluation can be particularly useful when working with large or infinite datasets, as it can be more memory-efficient and avoid unnecessary computations. By computing values only when they are needed in the program's execution flow, it allows for more responsive and efficient use of resources.

Overall, lazy evaluation is a powerful programming concept that can help optimize performance and resource utilization in a variety of scenarios.

The real harm of using ChatGPT that nobody is talking about

Fri, 24 Mar 2023 23:46:37 GMT

Do you find yourself wanting to use ChatGPT for everything?

Consider again!

ChatGPT is a powerful AI tool that can assist you with certain tasks, but it can also impair your thinking, analytical, and creative abilities over time.

This is why:

"Education is not just about finding the right answers to questions in your textbooks but internalising the concepts and learning how to apply them in real-world"
- ChatGPT is not a replacement for learning. It can generate text based on keywords, but it has no understanding of the actual meaning & context.
- It can tell you the answers your questions from the internet but it can't educate you. If you rely solely on ChatGPT for your education & research, you will miss out on learning how to research about a topic, scour through books, internet and learn from experts.
"Googling is an art"
- ChatGPT is not a replacement for thinking. It can generate coherent sentences, but there is no logic or reasoning behind them.
- It is not capable of analysing problems and making decisions out of them. You lose your ability to think critically and analyse things when you expect ChatGPT to give a direct answer.
- You learn when you try different solutions to a problem and analyse the outcomes. And most often than not, there's no one size fits all for our problems.
"Art is an expression of our thoughts, emotions, intuitions, and desires"
- ChatGPT is not a replacement for creating. It can mimic different styles and formats, but it lacks originality and personality.
- It is cannot express your voice, vision, emotions and personality. Once invests time to learn a craft like writing, drawing, painting. Once attaiining a level of skill, one learns to express himself through his craft.
- You never learn a craft to create something unique and meaningful if you rely on ChatGPT for your writing or art.

Don't let ChatGPT rob you off the unique skills that separate humans and AI-machines! Use it sparingly and always double-check its output for accuracy and quality. Keep in mind that nothing can ever replace your own brain and creativity!

**This blog was generated & paraphrased using ChatGPT

Exploring the Power of Python Data Classes

Sat, 28 Jan 2023 23:46:37 GMT

Why and Where to Use Them

In Python, when we think of a class, a structure like this comes to our mind

class Student():
    def __init__(self, first_name, last_name, age, grade):
        self.first_name = first_name
        self.last_name = last_name
        self.age = age
        self.grade = grade

    def function1(arg1, arg2):
        pass

which is not so intuitive. What if we could have a more intuitive & less repeatitive way of doing the same thing?

from dataclasses import dataclass
@dataclass
class Student:
     first_name: str  # attribute 1
     last_name: str   # attribute 2
     age: int         # attribute 3
     job: int         # attribute 4

    def function1(arg1, arg2): # functionality 1
        pass

Yes, we are talking about the Data Classes.

Less biolerplate code
More intuitive
Type Annotations, to help you & your text editor with better type checking & linting

In this blog, we'll focus on why you should use DataClasses & where you should use them. For a deeper dive in DataClasses, I'll attach resources.

REQUIREMENTS

Python 3.7+ 🐍
A zeal to learn 😊

When working with data-oriented classes, that behave like data-containers, you might want to create many instances of them ,compare, sort the objects and do a lot of data centric operations. Such functionalities are not provided right out of the box by regular classes. Data classes can hlp you achieve those functionalities with a more compact & intuitive code.

1. Less code to define a class

from dataclasses import dataclass
@dataclass
class Student:
     first_name: str  # attribute 1
     last_name: str   # attribute 2
     age: int         # attribute 3
     grade: int       # attribute 4

The dataclass decorator @dataclass is actually a code generator that automatically adds other methods under the hood. It adds methods like __init__ , __eq__ and __repr__ and many more to your class. These methods are responsible for setting the attribute values, testing for equality and representing objects in a nice string format.

2. Default values for attributes

from dataclasses import dataclass
@dataclass
class Student:
    first_name: str = "John"
    last_name: str = "Smith"
    age: int = 12
    grade: int = 5

As a general python convention, "fields without default values cannot appear after fields with default values."

3. Customised representation of the objects

The __repr__() can be modified to ovverride the default presentation of objects in the console


from dataclasses import dataclass


@dataclass
class Student:
    first_name: str = "John"
    last_name: str = "Smith"
    age: int = 12
    grade: int = 5

    def __repr__(self):
        return f"{self.first_name} {self.last_name} age {self.age}, studies in grade {self.grade}"


john = Student()
print(john)
# John Smith age 12, studies in grade 5
# instead of
# Student(first_name='John', last_name='Smith', age=12, grade=5)

The same can be achieved by using a __str__() which works with regular classes as well.

def __str__(self) -> str:
    return f"{self.first_name} {self.last_name} age {self.age}, studies in grade {self.grade}"

# John Smith age 12, studies in grade 5

4. Easy conversion to a tuple or a dictionary

Always converting objects to dict or tuple when interacting with other programs that expect these formats? Objects can be easily serialized into dicts or tuples using Data Classes.

from dataclasses import dataclass
from dataclasses import asdict, astuple


@dataclass
class Student:
    first_name: str = "John"
    last_name: str = "Smith"
    age: int = 12
    grade: int = 5

    # def __repr__(self):
    #     return f"{self.first_name} {self.last_name} age {self.age}, studies in grade {self.grade}"


john = Student()
print(john)
print(astuple(john))
# ('John', 'Smith', 12, 5)

print(asdict(john))
# {'first_name': 'John', 'last_name': 'Smith', 'age': 12, 'grade': 5}

5. Create READ-only objects

There are times when you want to prevent anyone from modifying the values of the attributes once the object is instantiated i.e. you want a frozen or an immutable instance.

This can be achieved by using @dataclass(frozen=True)

from dataclasses import dataclass

@dataclass(frozen=True)
class Student:
    first_name: str = "John"
    last_name: str = "Smith"
    age: int = 12
    grade: int = 5

john = Student()
print(john)

john.age = 13

If you set a frozen object’s attribute to a new value, a FrozenInstanceError error will be raised.

Student(first_name='John', last_name='Smith', age=12, grade=5)
Traceback (most recent call last):
  File "student.py", line 15, in <module>
    john.age = 13
  File "<string>", line 4, in __setattr__
dataclasses.FrozenInstanceError: cannot assign to field 'age'

Also a __hash__() will be automatically created by Python when frozen=True

6. Comparison of objects

john_again = Student(first_name='John', last_name='Smith', age=12, grade=5)

print(john == john_again)

In order to compare two objects john & john_again, you’d have to implement the __eq__ method yourself. This method should first check that the two objects are instances of the same class and then test the equality between tuples of attributes.

def __eq__(self, other):
    if other.__class__ is not self.__class__:
        return NotImplemented
    return (self.first_name,
            self.last_name,
            self.age,
            self.grade) == (other.first_name,
                            other.last_name,
                            other.age,
                            other.grade)

Now if you decide to add new attributes to your class, you’d have to update the __eq__() method again. Then do the same for __ge__() , __gt__() ,__le__() and __lt__() if they’re used.

But, with DataClass, it works out of the box

print(john == john_again)
# True

ADVANCED FEATURE: SORTING

It's also possible to sort & compare objects based on an attribute as a default.

from dataclasses import dataclass
from dataclasses import field  # import field


@dataclass(order=True)  # tells dataclass that we want to compare objects
class Student:
    first_name: str = "John"
    last_name: str = "Smith"
    age: int = 12
    grade: int = 5
    sort_index: int = field(init=False, repr=False)
    # init = False, means sort_index will be initialised after 'age' is initialised
    # repr = False, means sort_index will not be printed

    def __post_init__(self):
        self.sort_index = self.age


john = Student(age=15)
print(john)
# Student(first_name='John', last_name='Smith', age=15, grade=5)

sam = Student(age=20)
print(sam)
# Student(first_name='John', last_name='Smith', age=20, grade=5)

print(sam > john)
# True

Here, __gt__() considers age attribute for comparison.

QUESTION❓How'd you sort a frozen set?

Using the __setattr__()

from dataclasses import dataclass
from dataclasses import field  # import field


@dataclass(order=True, frozen=True)  # enable order=True
class Student:
    first_name: str = "John"
    last_name: str = "Smith"
    age: int = 12
    grade: int = 5
    sort_index: int = field(init=False, repr=False)
    # init = False, means sort_index will be initialised after 'age' is initialised
    # repr = False, means sort_index will not be printed

    def __post_init__(self):
        object.__setattr__(self,'sort_index', self.age)


john = Student(age=15)
print(john)
# Student(first_name='John', last_name='Smith', age=15, grade=5)

sam = Student(age=20)
print(sam)
# Student(first_name='John', last_name='Smith', age=20, grade=5)

print(sam > john)
# True

⭐BONUS⭐

If you've read this far, here's an advanced functionality of DataClasses

7. Initialize internal attributes

In some situations, you may need to create an attribute that is only defined internally, not when the class is instantiated. The attribute has a value that can depend on previously-set attributes.
The __post_init__() is called right after the __init__() method is called. By using this function and setting its init=False and repr=True we can create a new field called full_name

from dataclasses import dataclass
from dataclasses import field  # import field

@dataclass
class Student:
    first_name: str = "John"
    last_name: str = "Smith"
    age: int = 12
    grade: int = 5
    full_name: str = field(init=False, repr=True)

    def __post_init__(self):
         self.full_name = self.first_name + " " + self.last_name

john = Student()
print(john)
# Student(first_name='John', last_name='Smith', age=12, grade=5, full_name='John Smith')

print(john.full_name)
# John Smith

We can still instantiate the Student class without setting the full_name attribute. The repr=True makes it visible when the object is printed.

It's all great but where do I use it?

If you have Python 3.7+ 😛
You're working on a data-centric scenario
- It's a data-driven scenario & less behaviour-driven
- You need to create many data-centric classes
- Optionally, you need to ensure a degree of immutability on the objects of this class.

Example from artifacts.py showing how Data Classes are used for data-centric classes in production code.

from dataclasses import dataclass


@dataclass
class DataIngestionArtifact:
    trained_file_path: str
    test_file_path: str


@dataclass
class DataValidationArtifact:
    validation_status: bool
    valid_train_file_path: str
    valid_test_file_path: str
    invalid_train_file_path: str
    invalid_test_file_path: str
    drift_report_file_path: str


@dataclass
class DataTransformationArtifact:
    transformed_object_file_path: str
    transformed_train_file_path: str
    transformed_test_file_path: str

Resources

While learning about dataclasses, I went through many resources, here’s a list of the best I’ve found.

I have tried to be thorough and cover most of the use cases, yet, no man is perfect. Reach out if you find mistakes, or want me to pay attention to relevant use cases.

Thanks for reading 🙏 If you’ve made it this far, I really thank you for your time 😊 That’ll be all for me today. Until next time 👋

Reviving an Old PC 🖥

Sun, 22 Jan 2023 23:46:37 GMT

A Step-by-Step Guide to Installing Linux Mint XFCE with Windows 7 and Zorin OS Lite in Dual Boot

⚠️Caution: Please follow this post cautiously. I will not be responsible for any issue caused to your system.

You can google and find various blogs & videos that will tell you all the standard steps to follow. But this post is specific to my situation. The constraints I had, the errors I faced & the solutions that worked for me.

Requirements

Stable internet & electricity connection
Advanced knowledge about Windows & Linux
1 Old PC
1 Pen Drive

Preparation

⚠️Caution: Take a back up of your data before proceeding.

If Windows 7 is your permanent OS & you don't have any other in dual boot, follow this wonderful guide on how to proceed.

If you already have any other OS in dual boot with Windows 7, you might need to uninstall it, like I did.

Prepare the bootable USB

Follow these steps.
Uninstall Zorin OS

Following steps from this video
- I didn't had a bootable CD for Windows to fix boot menu. So I jumped to uninstalling Zorin.
- Delete Partition was not enabled for me somehow. I tried to fix it but ended up extending the volume.
- EasyBCD or any similar tool is not mandatory as you can fix your BCD using Windows Command Prompt
- I rebooted my system to begin installing Linux Mint
- You can try to enter Windows recovery by following these steps if feasible.
Installing Linux Mint

I got a similar GRUB menu & followed the steps described here After installation is completed. We have to fix boot menu before restarting.
Repair Boot Menu

It is possible to fix any boot issues using Linux. In fact, if you are stuck in a situation try to boot into another OS & fix the boot issues.

Boot Repair is a tool to repair common boot issues on Ubuntu, Debian, Arch, Linux Mint, OpenSUSE, Fedora and other Linux distributions, Windows and Mac OS operating systems. I followed this guide.

You have to
- Install Boot Repair
- Do Recommended Repair
- Backup Partition Table
- Repair File System

Try rebooting & check if both Linux Mint & Windows 7 are working as expected.

👋 You have successfully installed Linux Mint in dual boot with Windows 7 after uninstalling Zorin OS! 🥳

Thanks for reading 🙏 If you’ve made it this far, I really thank you for your time :) That’ll be all for me today. Until next time 👋