python - Why is the subprocess.Popen argument length limit smaller than what the OS reports? -

i running python 3.4.3 on linux 3.16.0. want use subprocess.popen run command long single argument (a complex bash invocation), 200kib.

according getconf , xargs, should within limits:

$ getconf arg_max 2097152 $ xargs --show-limits < /dev/null environment variables take 3364 bytes posix upper limit on argument length (this system): 2091740 posix smallest allowable upper limit on argument length (all systems): 4096 maximum length of command use: 2088376 size of command buffer using: 131072 

however, python fails quite smaller limits:

>>> subprocess.popen('echo %s > /dev/null' % ('a' * (131072-4096)), shell=true, executable='/bin/bash') <subprocess.popen object @ 0x7f4613b58410> >>> subprocess.popen('echo %s > /dev/null' % ('a' * (262144-4096)), shell=true, executable='/bin/bash') traceback (most recent call last):   [...] oserror: [errno 7] argument list long 

note python limit same "actually using" command buffer xargs reports. suggests xargs somehow smart enough start smaller limit , increase needed, python not.

questions:

1. why python limit smaller os limit of 2mib?
2. can increase python limit?
3. if so, how?

the maximum size single string argument limited 131072. has nothing python:

~$ /bin/echo "$(printf "%*s" 131071 "a")">/dev/null ~$ /bin/echo "$(printf "%*s" 131072 "a")">/dev/null bash: /bin/echo: argument list long 

it max_arg_strlen decides max size single string:

and additional limit since 2.6.23, 1 argument must not longer max_arg_strlen (131072). might become relevant if generate long call "sh -c 'generated long arguments'". (pointed out xan lopez , ralf wildenhues)

see this discussion of arg_max, under "number of arguments , maximum length of 1 argument", , this question on unix.stackexchange.

you can see in binfmts.h:

/*  * these maximum length , maximum number of strings passed  * execve() system call.  max_arg_strlen random serves  * prevent kernel being unduly impacted misaddressed pointers.  * max_arg_strings chosen fit in signed 32-bit integer.  */ #define max_arg_strlen (page_size * 32) #define max_arg_strings 0x7fffffff  ~$ echo $(( $(getconf page_size)*32 ))  131072 

you can pass multiple strings of length 131071:

subprocess.check_call(['echo', "a"*131071,"b"*131071], executable='/bin/bash',stdout=open("/dev/null","w")) 

but single string arg cannot longer 131071 bytes.