A Python list is an ordered and mutable container that stores elements of various types (integers, strings, other lists) in a single variable. This structure relies on a mechanism of dynamically allocated array of pointers: the allocated memory grows in increments with each addition, making common operations fast without special configuration. Understanding this internal functioning allows for better choices between the different methods available for manipulating data on a daily basis.
Dynamic array and over-allocation: what happens in memory
When a script adds elements one by one with append(), Python does not allocate memory element by element. The language pre-allocates a capacity greater than the number of elements actually stored, often by a factor close to double. This mechanism makes each call to append() amortized in constant time.
The downside of this strategy appears during reallocations. When the pre-allocated capacity is reached, Python copies the entire array to a new, larger memory space. In a script that progressively collects lines from a file or measurements, these copy spikes remain imperceptible for a few thousand elements but become measurable with larger volumes.
For this reason, grouping additions with extend() rather than chaining append() in a loop reduces the number of reallocations. If the final size is known in advance, building the list all at once (via a comprehension, for example) remains the most efficient method. A Python guide on Tech Mafia details several of these strategies applied to concrete cases of data processing.
Python list comprehensions: syntax and optimization in 3.12
A list comprehension condenses a transformation loop into a single line. The basic syntax follows this pattern:
result = [expression for element in iterable if condition]
This writing replaces a classic for block followed by an append(), and the gain is not merely cosmetic. In Python 3.12, the Faster CPython project eliminated the creation of a separate internal frame for each comprehension. The result: less overhead and faster processing of common transformations like string cleaning or filtering numbers in a dataset.
A concrete example: extracting ages greater than 30 from a list of dictionaries.
ages = [p["age"] for p in people if p["age"] > 30]
This line does the work of a for loop, a conditional test, and an append in a single readable expression. Comprehensions become less suitable when the internal logic exceeds two conditions or involves side effects (writing to a file, network calls). In this case, reverting to an explicit loop improves code readability.
Slicing and negative indexing: accessing data without a loop
Slicing allows you to extract a sublist without writing a loop. The notation uses three parameters in brackets: start, end, and step.
list[2:5]returns the elements at index 2, 3, and 4 (the end boundary is excluded)list[::-1]reverses the order of the entire list without modifying the originallist[::2]selects every other element, useful for sampling data from sensors or logs
The negative index counts from the end: list[-1] gives the last element, list[-3:] gives the last three. This notation avoids calculating the length of the list with len() before accessing its final elements.
Slicing creates a shallow copy of the extracted portion. Modifying the sublist does not change the original list, which protects the source data during exploratory processing.
Sorting, filtering, and combining lists with native functions
Python provides two sorting approaches. The sort() method modifies the list in place, while the sorted() function returns a new sorted list without touching the original. The distinction matters when the program needs to preserve the original order for further processing.
The key= parameter accepts a function that defines the sorting criterion. Sorting a list of strings by length, for example:
sorted_words = sorted(words, key=len)
To filter, the filter() function applies a condition to each element and returns an iterator. Combined with list(), it produces a new list:
positives = list(filter(lambda x: x > 0, numbers))
Combining two lists can be done using the + operator (concatenation) or with extend(). The difference: + creates a new list, while extend() modifies the existing list. With significant volumes, extend() consumes less memory since it does not duplicate the structure.
append(element)adds a single element to the end of the listextend(iterable)adds each element from an iterable to the existing listinsert(index, element)places an element at a specific position, shifting all subsequent elements (cost proportional to the size of the list)pop(index)removes and returns the element at the given index, or the last one if no index is specified
Nested lists and tabular structures in Python
A list can contain other lists, forming a two-dimensional structure comparable to a table. Each sublist then represents a row of data:
table = [["Alice", 28], ["Bob", 35], ["Clara", 42]]
Accessing Bob’s age is done with table[1][1], which is the second row, second column. This format is suitable for small datasets (a few dozen rows), but shows its limits beyond that: each access via double indexing makes the code less readable, and column-by-column operations (sum, average) require explicit loops.
For more structured tabular processing, converting these nested lists into dictionaries (with named keys) improves clarity. Each row becomes a dictionary with explicit keys like “name” and “age”, making the code self-documenting.
Python lists cover the majority of storage and data transformation needs in everyday scripts. The choice between comprehension, explicit loop, or native function mainly depends on the desired readability and the volume being processed. When the data exceeds a few hundred rows or requires column operations, transitioning to specialized structures like pandas DataFrames becomes a natural extension of the work started with lists.
