[doc] os.listdir and os.path.join inconsistent on empty path

[babou]

BPO	17545
Nosy	@ncoghlan, @bitdancer
Files	issue17545.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-03-25.16:26:53.421>
labels = ['easy', 'type-bug', '3.8', '3.9', '3.10', 'library', 'docs']
title = '[doc] os.listdir and os.path.join inconsistent on empty path'
updated_at = <Date 2020-11-25.10:55:07.339>
user = 'https://bugs.python.org/babou'

bugs.python.org fields:

activity = <Date 2020-11-25.10:55:07.339>
actor = 'iritkatriel'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Library (Lib)']
creation = <Date 2013-03-25.16:26:53.421>
creator = 'babou'
dependencies = []
files = ['30367']
hgrepos = []
issue_num = 17545
keywords = ['easy']
message_count = 12.0
messages = ['185207', '185209', '185212', '185297', '185299', '187783', '187788', '189401', '189689', '189773', '189988', '193704']
nosy_count = 6.0
nosy_names = ['ncoghlan', 'r.david.murray', 'docs@python', 'Thomas Fenzl', 'babou', 'woparry']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17545'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-61745: Clarify os.listdir behavior for empty path #145338

[babou]

The empty path '' is considered as an acceptable path in os.path.join, and works as a neutral prefix:
print os.path.join('','aaa') ===> aaa
which seems rather natural.
But it raises an exception when used as a parameter to os.listdir.
Logically, it should then list the current directory.
(the alternative would be to raise an exception when os.path.join gets an empty argument).

The inconsistency became clear to me when I had to write the following function :

def listdirs(path,flist): # Appends to "flist" all paths to files that
            # start with "path". Argument "path" can be empty string ''.
    if path=='' : localist=os.listdir('.')
    else : localist=os.listdir(path)
    for f in localist :
        p=os.path.join(path,f)
        flist.append(p)
        if os.path.isdir(p) :
            listdirs(p,flist)
    return flist

The conditionnal is needed only to avoid the exception, while the code is unique afterwards.

This is related to bpo-818059, but I did not see how that issue was resolved.

Furthermore, the case of the empty path as argument to os.listdir should be documented in http://docs.python.org/2/library/os.html

[bitdancer]

See also bpo-6095.

However:

rdmurray@hey:~>ls ''
ls: cannot access : No such file or directory

So, the behavior is consistent with the shell. The shell has no equivalent to os.path.join.

[bitdancer]

By the way, you could write this as:

  localist = os.listdir(path if path else '.')

[babou]

Reply to R.David.Murray

See also bpo-6095.

You are right. I goofed, this is the issue I meant to point to.

$ ls ''
ls: cannot access : No such file or directory
So, the behavior is consistent with the shell.

This is a fair remark.
But still, giving a meaning to something that has none in the shell would not be an inconsistency either. There are lots of other differences in the Unix shell design, and strings are often used as a syntactic device. You do not have to quote file names or paths, unless they raise syntactic problems.

Now if you look at path manipulation commands in the shell, you have :

$ basename aaa
aaa
$ dirname aaa
.

This is a fair choice for the shell since an empty path would print as nothing. Furthermore, string manipulation is not as convenient with the shell as it is with Python. So the shell is altogether ignoring the empty path, and string manipulations (possibly using the empty string) to build a path representation are not part of the path system.

Python has already made a different choice.

In [4]: os.path.basename('aaa')
Out[4]: 'aaa'

In [5]: os.path.dirname('aaa')
Out[5]: ''

These are the two results of os.path.split('aaa'), which is somewhat the inverse of os.path.join(...) which I initially considered.

So os.path.dirname in Python does return the empty string '' where the shell dirname returns a dot. This could also be seen as an inconsistency between Unix shell and Python.

However, the Unix shell is internally consistent, while taking into account its own specific constraints.

It seems that it is more important for Python to similarly have its own internal consistency, especially when considering that Python is already departing from Unix shell in some minor ways, which are related to the internal consistency issue that was raised.

[bitdancer]

I'm inclined to agree with you. I believe '' represents the current directory in sys.path, for example. It ought to be easy to add this since bpo-6095 is already fixed.

[babou]

Thank you, David.

BTW, I sent a message on april 20 to docs@python.org about a bug in
the documentation regarding os.readlink(path)
on page http://docs.python.org/2/library/os.html

and proposing an alternative text.

I got no reply.

This was not long ago ... should I just wait ?

Sorry for asking you ... I know no one else.

Regards

Bernard

• R. David Murray <report@bugs.python.org>, le 20-04-13, a écrit:

Changes by R. David Murray <rdmurray@bitdance.com>:

----------
keywords: +easy
stage: -> needs patch

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue17545\>

---

--
SVP Ne plus m'écrire à Bernard.Lang@inria.fr mais à l'adresse ci-dessous

Please No longer write to Bernard.Lang@inria.fr but to the address below

Bernard.Lang@datcha.net ,_ /\o \o/ gsm +33 6 6206 1693
http://www.datcha.net/ ^^^^^^^^^^^^^^^^^ tel +33 1 3056 1693
Je n'exprime que mon opinion - I express only my opinion
CAGED BEHIND WINDOWS or FREE WITH LINUX

[bitdancer]

I don't know if the docs team replies to messages or not. I know that issues sometimes get opened up here that refer to emails to that alias. You can open an issue here yourself for the doc bug, if you want to.

[woparry]

I started working on a patch for this, but the more I think about it the less I am convinced it is wanted.

The issue requests that os.listdir('') be equal to os.listdir('.')

The given example of os.path.join doesn't follow this:

>>> os.path.join('','aaa')
'aaa'
>>> os.path.join('.','aaa')
'.\\aaa'

This makes sense: prepending an empty path should be a no-op, while prepending the current directory has a different meaning.

It seems consistent in this case that listing an empty path and listing the current directory should have different meanings.

Is there any other traction/use case for this change?

[ThomasFenzl]

I also looked into creating a patch and now I'm not convinced about the solution.

While os.path.join accepts an empty string, the functions is os (e.g. stat and friends, utime, chown don't. os.walk doesn't throw an Exception, but generates an empty iterator on ''.
So from a module perspective, things wouldn't get more consistent, but less.

Any thoughts about how to proceed?

[woparry]

I suggest that this is a documentation issue.

I have seen three classes of functions is os and os.path
i - those which operate on path names only (os.path.join, os.path.dirname, etc) and do not depend on the state of the file system. Since these are string manipulation functions, they treat '' as the empty string.
ii - those which turn path names into valid paths via knowledge of the state of the file system (os.path.abspath, os.path.normpath, etc). These treat '' as an empty relative path and thus equivalent to the current directory.
iii - those which operate on valid paths only and thus depend on the state of the file system (os.listdir, os.stat, os.path.relpath, os.path.exists, etc). '' is not a valid path so these functions throw an exception.

The documentation should clearly state which class a function belongs to. I will provide a patch sometime this week.

os.walk is an odd duck - it has class iii behaviour except that it returns an empty iterator instead of raising an exception. Note that it still distinguishes between '' (invalid path) and '.' (current directory) and so is not a good reason to change os.listdir:

>>> [x for x in os.walk('')]
[]
>>> [x for x in os.walk('.')]
[('.', [], [])]

in an empty directory.

[woparry]

Patch as described above. Comments appreciated.