Question

Write a program that reads a Python source file and produces an index of all identifiers in the file. For each identifier, print all lines in which it occurs. For simplicity, consider any string consisting only of letters, numbers, and underscores an identificr.

Short answer

Read the file, tokenize by line, check for identifiers, and store line numbers in a dictionary.

Step-by-step solution

Read the File

Begin by opening the Python source file using Python's built-in `open()` function. Use a loop to read the file line by line. This allows processing each line separately and maintaining line numbers which is essential for later steps.

Initialize Data Structures

Create a dictionary to store identifiers as keys and sets as values. Each key will be an identifier, and the set will contain line numbers where the identifier appears. This structure allows for efficient checking and insertion of line numbers.

Define an Identifier Recognition Method

Write a function that checks if a word consists only of letters, numbers, and underscores. Use Python's `str.isidentifier()` function which meets these criteria, or manually check using regular expressions.

Tokenize Each Line

For each line, split the line into words using Python’s `str.split()` method. This helps to access potential identifiers one by one.

Identify and Store Identifiers

Iterate through the list of words from the previous step. Use the identifier recognition method to check if a word is an identifier. If it is, check if it's already in the dictionary. If not, add it with the current line number. If it is already in the dictionary, simply append the line number to the existing set.

Output the Results

After processing all lines, iterate through the dictionary and print each identifier followed by its associated line numbers. This produces the index of identifiers as required.

Key concepts

Identifiers

In Python programming, an identifier is a name used to identify a variable, function, class, module, or other object. Identifiers play a vital role in Python code to reference these objects. They must begin with a letter (A-Z or a-z) or an underscore (_), followed by any combination of letters, numbers, and underscores.
They are case-sensitive, meaning that 'Variable' and 'variable' would be distinct identifiers. Python also reserves a number of keywords that cannot be used as identifiers, such as `for`, `if`, `else`, etc.
Understanding identifiers is crucial as they help make programs readable and maintainable. Choosing meaningful names as identifiers can make your code easier to follow and understand.

File Handling

File handling is an essential aspect of Python programming. It allows you to open, read, write, and manipulate files stored on your computer. Python provides several built-in functions to help with file handling, with `open()` being one of the most important.
When you use `open()`, you can specify the mode in which you want to open the file. For example:

• `'r'` mode allows you to read the file.
• `'w'` mode will open the file for writing and will overwrite the content.
• `'a'` mode opens the file for appending, adding data to the end of the file.
• `'b'` is used to open the file in binary mode.

Reading files line by line is often done using a loop, which helps to maintain line numbers and process each line separately, as illustrated in the solution step for reading the file.

String Manipulation

String manipulation refers to the methods and techniques used to work with strings in Python. Strings are sequences of characters, and Python provides a plethora of methods for manipulating them.
Some basic operations include:

• `str.split(separator, maxsplit)` - Splits the string into a list, using 'separator' as the delimiter.
• `str.isidentifier()` - Checks if a string is a valid identifier.
• `str.isalpha()` - Returns True if the string only contains letters.
• `str.isalnum()` - Returns True if the string contains only letters and numbers.

These methods are highly useful when processing and extracting identifiers from each line of a program, allowing for efficient tokenization and filtering of potential identifiers.

Data Structures

Data structures are vital components in programming and are used to store and organize data efficiently. Python provides several built-in data structures, including lists, sets, dictionaries, and tuples, each with its own unique properties.
In the context of this exercise:

• A "dictionary" is used to store identifiers as keys and line numbers as values. This allows rapid searching and appending line numbers where the identifier appears.
• A "set" is used to store line numbers, ensuring there are no duplicates and insertion is efficient.

Choosing the right data structure is crucial for performance. Python's dictionaries and sets are fast and offer an excellent way to manage keys and unique items, making them perfect for this task.