[ LP comment 1 by: Michele Mattioni, on 2010-01-18 11:54:43.261168+00:00 ]

Expected:

• The window should respond to the command (zoom, rotate...) and not freeze.

• The prompt should be back and be responsive.

[ LP comment 2 by: Michele Mattioni, on 2010-01-18 12:02:03.250363+00:00 ]

Ok, this is weird.

Sometimes the window freeze, sometimes respond to the command (like zoom and rotate).

However I can't get the prompt back and ready to insert a new command.

[ LP comment 3 by: Brian Granger, on 2010-01-18 16:42:52+00:00 ]

Mayavi and pylab are probably doing some funny things with the event
loop that is messing up
things. It will probably require someone looking through the
Mayavi/pylab/traits code carefully
to look for these things.

Cheers,

Brian

On Mon, Jan 18, 2010 at 4:02 AM, Michele Mattioni mattions@gmail.com">mattions@gmail.com wrote:

Ok, this is weird.

Sometimes the window freeze, sometimes respond to the command (like zoom and rotate).

However I can't get the prompt back and ready to insert a new command.

-- Ensure %gui qt works with new Mayavi and pylab https://bugs.launchpad.net/bugs/508953 You received this bug notification because you are a member of IPython Developers, which is subscribed to IPython.

Status in IPython - Enhanced Interactive Python: Confirmed

Bug description: Right now mayavi/qt is only on SVN, but eventually we need to ensure that %gui qt allows mayavi/qt and pylab/qt to work together.

For more details: http://mail.scipy.org/pipermail/ipython-dev/2010-January/005728.html

-- Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com

[ LP comment 4 by: Fernando Perez, on 2010-01-18 23:16:28.353993+00:00 ]

The attached script can be run in ipython trunk-dev, and shows the problem:

maqroll[Desktop]> ip
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
Type "copyright", "credits" or "license" for more information.

IPython 0.11.bzr.r1346 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: gui -a qt
Out[1]: <PyQt4.QtGui.QApplication object at 0x93086ac>

In [2]: run qt_embedding.py

Button pushed

Terminal is frozen here, until we close the mayavi window.

In contrast, if we run the same script in 0.10 with -q4thread:

(ipython-0.10)maqroll[Desktop]> ip -q4thread Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: run qt_embedding.py

In [2]: 1+1 # terminal accepts input though window remains open
Out[2]: 2

In [3]: Button pushed

Thanks to Gael for the feedback to fix the api calls in the example script.

I think this is related to the larger issues we are having with Traits and IPython's new even t loop stuff.

[ LP comment 1 by: Brian Granger, on 2009-04-14 21:38:33.292246+00:00 ]

I have confirmed this. It also occurs for the MultiEngineClient when the controller stop and is restarted underneath a live client. The client should have reconnect logic that is triggered on a DeadReferenceError.

[ LP comment 2 by: Vishal Vatsa, on 2009-04-24 18:22:23+00:00 ]

Hey Brian,

Totally forgot about this bug. Since I have had a look at this before,
feel free to assign it to me.

-vishal

2009/4/14 Brian Granger ellisonbg@gmail.com">ellisonbg@gmail.com:

I have confirmed this.  It also occurs for the MultiEngineClient when the controller stop and is restarted underneath a live client.  The client should have reconnect logic that is triggered on a DeadReferenceError.

** Changed in: ipython   Importance: Undecided => Medium

** Changed in: ipython       Status: New => Confirmed

** Changed in: ipython     Assignee: (unassigned) => Brian Granger (ellisonbg)

-- Getting "DeadReferenceError: Calling Stale Broker" after ipcontroller restart https://bugs.launchpad.net/bugs/308253 You received this bug notification because you are a direct subscriber of the bug.

Status in IPython - Enhanced Interactive Python: Confirmed

Bug description: Not Sure that is a bug or an expected behavior.

I am starting the ipcontoller such that it reuses the ipcontroller-tc.furl between restarts. After the restart, if I invoke a DeadReferenceError with the following traceback: I am using the blocking TaskClient from an ipython shell.

In [8]: tc.spin()

DeadReferenceError                        Traceback (most recent call last)

/home/vishal/ in ()

/usr/lib/python2.5/site-packages/IPython/kernel/taskclient.pyc in spin(self)    119         scheduler is idle for some reason.    120         """ --> 121         return blockingCallFromThread(self.task_controller.spin)    122    123     def queue_status(self, verbose=False):

/usr/lib/python2.5/site-packages/IPython/kernel/twistedutil.pyc in blockingCallFromThread(f, a, **kw)     67         @raise: any error raised during the callback chain.     68         """ ---> 69         return twisted.internet.threads.blockingCallFromThread(reactor, f, a, **kw)     70     71 else:

/usr/lib/python2.5/site-packages/twisted/internet/threads.pyc in blockingCallFromThread(reactor, f, *a, **kw)     81     result = queue.get()     82     if isinstance(result, failure.Failure): ---> 83         result.raiseException()     84     return result     85

/usr/lib/python2.5/site-packages/twisted/python/failure.pyc in raiseException(self)    317         information if available.    318         """ --> 319         raise self.type, self.value, self.tb    320    321

DeadReferenceError: Calling Stale Broker

[ LP comment 3 by: Brian Granger, on 2009-04-27 21:13:32.675484+00:00 ]

Thanks for looking into this!

[ LP comment 1 by: Fernando Perez, on 2010-01-18 01:39:06.370752+00:00 ]

Some notes on the above:

• I talked with William Stein (sage) a few days ago: no major progress from them on using pexpect for windows, instead they are moving away from pexpect as much as possible by building library wrappers for everything they can (like maxima and gap).

• I don't see us abandoning nose anytime soon. Trial has a few critical problems:

• it would make testing depend on twisted for us. Nose is a tiny, self-contained download, but Twisted is a much larger dependency. Too much for something as critical as testing.

• It has no discovery mechanism for doctests.

• My earlier efforts on doing ipython doctests without nose were only partly successful. In the last few weeks that I've been working a lot on this, I realized we really need quite a bit of machinery to be able to validate full ipython sessions, just doing syntax conversion to get a unittest out of a docstring (the part I was able to succeed on) isn't sufficient for the whole test suite.

I don't think it's really possible to expect our test suite to pass without an ipython shell running somewhere. We are more than just plain python, and to test all of that we really need a true ipython somewhere that we can beat on.

Having said all this, I think things now aren't really that bad. Having cleaned up the test machinery a lot recently, it's working quite well, debugging is possible with --pdb or other mechanisms, etc. The only thing that is hard is debugging a big doctest with lots of output, but that's simply how doctests are, not something specific to ipython (and for that reason, a bad idea in general if you want your tests to be debuggable).

I'm all for making our testing machinery as simple/robust as humanly possible, because I fully realize the importance of the test support in the health of a project. But having spent literally upwards of a hundred hours on this problem alone, I'm not sure right now I'm capable of making it much simpler without writing something like nose again, from scratch.

One ray of light may come from python itself: I think for 2.7 they are improving unittest, so perhaps we could find better support there once it's out.

I'm open to revisiting this question when possible, but I'm not terribly optimistic on me making lots of improvements beyond what's right now in my trunk-dev branch (which actually works quite well, it must be said). But I'd love to see someone who comes with a fresh eye on this problem make progress on it!

[ LP comment 1 by: Fernando Perez, on 2010-01-30 05:43:41+00:00 ]

On Fri, Jan 29, 2010 at 8:40 PM, Brian Granger ellisonbg@gmail.com wrote:

Public bug reported:

Currently in IPython.core we sometimes use IPython.utils.io.Term to print and sometimes we just do print.  The advantage of using Term is that it will properly handle colors on Windows.  But we don't use it everywhere.  Really, we need to come up with a uniform way of handling printing to stdout/stderr in IPython.  This is particularly relevant as we refactor the core to be used without a terminal.

Absolutely: we should probably add a little tool that looks for naked
print statements and flags them. It's OK to have one in the middle of
debuggin on your own, but we probably shouldn't leave any in committed
code.

Thanks!

[ LP comment 2 by: Brian Granger, on 2010-01-30 06:06:39+00:00 ]

Thinking about moving towards 2.6/3.0 support...

It would probably be a good idea to unify Term/print into a new API
that closely follows the
print function of python 2.6/3.0. We could even write a simply
version of the print function
that works with 2.5 and would allow us to move to the new API now. We
will need to do this
for the refactor of IPython.core for GUI versions of IPython.

Cheers,

Brian

Absolutely: we should probably add a little tool that looks for naked print statements and flags them.  It's OK to have one in the middle of debuggin on your own, but we probably shouldn't leave any in committed code.

Thanks!

-- Decide when to use Term and when to print https://bugs.launchpad.net/bugs/514613 You received this bug notification because you are a member of IPython Developers, which is subscribed to IPython.

Status in IPython - Enhanced Interactive Python: New

Bug description: Currently in IPython.core we sometimes use IPython.utils.io.Term to print and sometimes we just do print.  The advantage of using Term is that it will properly handle colors on Windows.  But we don't use it everywhere.  Really, we need to come up with a uniform way of handling printing to stdout/stderr in IPython.  This is particularly relevant as we refactor the core to be used without a terminal.

-- Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com

[ LP comment 1 by: Ville M. Vainio, on 2009-07-07 06:32:30+00:00 ]

On Tue, Jul 7, 2009 at 7:13 AM, matt_ipythontmp_ipython@mailinator.com">tmp_ipython@mailinator.com wrote:

The object "compiled_re" does in fact get pickled and is automatically restored when ipython is next launched. But code to print out user feedback is assuming that the object being stored has an attribute "class", which a compiled regular expression does not have.

Yeah, it's trivial to just do the class name lookup in Try/Except.
Currently the code is:

/usr/local/lib/python2.6/dist-packages/IPython/Extensions/pspersistence.pyc in magic_store(self, parameter_s)

178             #pickled = pickle.dumps(obj)

179             self.db[ 'autorestore/' + args[0] ] = obj

--> 180 print "Stored '%s' (%s)" % (args[0], obj.class.name)

181

-- Ville M. Vainio
http://tinyurl.com/vainio

[ LP comment 1 by: anatoly techtonik, on 2009-06-04 12:33:54.822394+00:00 ]

[ LP comment 1 by: Brian Granger, on 2009-04-14 21:00:17.906104+00:00 ]

This is a continuing problem on OS X and we need to come up with a more robust solution!

[ LP comment 1 by: Andrea Riciputi, on 2009-04-16 20:35:54.161954+00:00 ]

[ LP comment 1 by: Vishal Vatsa, on 2010-01-18 09:36:37.944247+00:00 ]

Hi Tony,

Which version of ipython are you using?

Thanks,
-vishal

[ LP comment 2 by: Vishal Vatsa, on 2010-01-18 15:30:05+00:00 ]

Ipython version info from user.

---------- Forwarded message ---------- From: Toby Burnett tburnett@uw.edu
Date: 2010/1/18
Subject: RE: [IPython-user] ipcluster does not start all the engines
To: "vishal.vatsa@gmail.com" vishal.vatsa@gmail.com">vishal.vatsa@gmail.com

HI Vishal,

Thanks for looking at it. I’m using 0.10 with python 2.5.4

Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]

IPython 0.10 -- An enhanced Interactive Python.

--Toby

P.S. I tried to join the bug list, separate from user list I guess. No response.

[ LP comment 3 by: Brian Granger, on 2010-01-18 16:41:12+00:00 ]

Hi, I did not find any previous notes on this. I have a cluster of 4 machines, each with 8 hyperthreaded cores, so I can run 16 engines per machine, or 64 in all. It is amazingly easy and useful, thanks so much for providing this.

Great, glad it is useful to you.

However, when using ipcluster on one of these machines in ssh mode, with this clusterfile, send_furl = False engines = { 'tev1' : 16,           'tev2' : 16,           'tev3' : 16,           'tev4' : 16 }

I typically get about 50 engines to actually start. Since there seems to be no log file for ipcluster (in spite of code that seems like it should record which engines it tried to start), I can't send that. The ipcontroller log file looks fine, except for recording fewer than the 64 engines that I expected.

I have an alternative, very klugy method that starts a controller, then executes 64 ssh commands to the respective machines to simply run ipengine. I found the same problem, which went away when I introduced a 1 second delay after each ssh call, which of course takes more than a minute to run, and leaves all those ssh processes running.

I think I know what the issue is here. We have found that sometimes
the engines startup
so fast that the controller is not yet up and running. The engine
that try to connect
before the controller is running fail. Twisted is fully capable of
handling many simultaneous
connections, so I don't think it is that.

The good news is that all of this is fixed in trunk (ipcluster is much
improved). The bad news
is that I haven't yet gotten the ssh mode cluster working with the new
ipcluster in trunk.
It shouldn't be difficult, and Vishal knows this code as well.

In the mean time, I would suggest looking through ipcluster.py - you
should be able to put a delay
between when the controller is started and when the engines are started.

Cheers,

Brian

So I suspect that the same thing would work in the loop in this method of ipcluster.SSHEngineSet

  def _ssh_engine(self, hostname, count):       exec_engine = "ssh %s sh %s/%s-sshx.sh %s" % (           hostname, self.temp_dir,           os.environ['USER'], self.engine_command       )       cmds = exec_engine.split()       dlist = []       log.msg("about to start engines...")       for i in range(count):           log.msg('Starting engines: %s' % exec_engine)           d = getProcessOutput(cmds[0], cmds[1:], env=os.environ)           dlist.append(d)       return gatherBoth(dlist, consumeErrors=True)

but that would be inelegant, given that the real problem is probably related to the controller not responding properly to multiple requests.

Thanks for looking at this.

--Toby Burnett

** Affects: ipython     Importance: Undecided         Status: New

-- ipcluster does not start all the engines https://bugs.launchpad.net/bugs/509015 You received this bug notification because you are a member of IPython Developers, which is subscribed to IPython.

Status in IPython - Enhanced Interactive Python: New

Bug description: As reported on the mailing list...

---------- Forwarded message ---------- From: Toby Burnett <tburnett@EMAIL-REMOVED> Date: Sat, Jan 16, 2010 at 8:58 AM Subject: [IPython-user] ipcluster does not start all the engines To: "ipython-user@scipy.org" ipython-user@scipy.org

Hi, I did not find any previous notes on this. I have a cluster of 4 machines, each with 8 hyperthreaded cores, so I can run 16 engines per machine, or 64 in all. It is amazingly easy and useful, thanks so much for providing this.

However, when using ipcluster on one of these machines in ssh mode, with this clusterfile, send_furl = False engines = { 'tev1' : 16,           'tev2' : 16,           'tev3' : 16,           'tev4' : 16 }

I typically get about 50 engines to actually start. Since there seems to be no log file for ipcluster (in spite of code that seems like it should record which engines it tried to start), I can't send that. The ipcontroller log file looks fine, except for recording fewer than the 64 engines that I expected.

I have an alternative, very klugy method that starts a controller, then executes 64 ssh commands to the respective machines to simply run ipengine. I found the same problem, which went away when I introduced a 1 second delay after each ssh call, which of course takes more than a minute to run, and leaves all those ssh processes running.

So I suspect that the same thing would work in the loop in this method of ipcluster.SSHEngineSet

  def _ssh_engine(self, hostname, count):       exec_engine = "ssh %s sh %s/%s-sshx.sh %s" % (           hostname, self.temp_dir,           os.environ['USER'], self.engine_command       )       cmds = exec_engine.split()       dlist = []       log.msg("about to start engines...")       for i in range(count):           log.msg('Starting engines: %s' % exec_engine)           d = getProcessOutput(cmds[0], cmds[1:], env=os.environ)           dlist.append(d)       return gatherBoth(dlist, consumeErrors=True)

but that would be inelegant, given that the real problem is probably related to the controller not responding properly to multiple requests.

Thanks for looking at this.

--Toby Burnett

-- Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com

[ LP comment 4 by: Vishal Vatsa, on 2010-01-18 21:21:06+00:00 ]

2010/1/18 Brian Granger ellisonbg@gmail.com">ellisonbg@gmail.com:

I think I know what the issue is here.  We have found that sometimes the engines startup so fast that the controller is not yet up and running.  The engine that try to connect before the controller is running fail.  Twisted is fully capable of handling many simultaneous connections, so I don't think it is that.

Yep this sound about correct.
Though it should not happen as, there is a _delay_start method which
waits for the furl files to be created on disk, which should be the test
that ipcontroller is up and running.

Toby, which OS are you on? could the filesystem semantics be different?
(In theory, it should work on windows/cygwin but I have never tested it)

I have not been able to replicate this so far on NAS backed cluster.
If this continues to be an issue for you, I can try to give you a patch
to insert a delay in the ssh engine start.

Regards,
-vishal

[ LP comment 5 by: Brian Granger, on 2010-01-18 22:38:16+00:00 ]

Yep this sound about correct. Though it should not happen as, there is a _delay_start method which waits for the furl files to be created on disk, which should be the test that ipcontroller is up and running.

Yes, even though that method exists, we have sometimes observed it to fail.

Toby, which OS are you on? could the filesystem semantics be different? (In theory, it should work on windows/cygwin but I have never tested it)

The problems we had with this were on Windows.

I have not been able to replicate this so far on NAS backed cluster. If this continues to be an issue for you, I can try to give you a patch to insert a delay in the ssh engine start.

Regards, -vishal

-- ipcluster does not start all the engines https://bugs.launchpad.net/bugs/509015 You received this bug notification because you are a member of IPython Developers, which is subscribed to IPython.

Status in IPython - Enhanced Interactive Python: New

Bug description: As reported on the mailing list...

---------- Forwarded message ---------- From: Toby Burnett <tburnett@EMAIL-REMOVED> Date: Sat, Jan 16, 2010 at 8:58 AM Subject: [IPython-user] ipcluster does not start all the engines To: "ipython-user@scipy.org" ipython-user@scipy.org

Hi, I did not find any previous notes on this. I have a cluster of 4 machines, each with 8 hyperthreaded cores, so I can run 16 engines per machine, or 64 in all. It is amazingly easy and useful, thanks so much for providing this.

However, when using ipcluster on one of these machines in ssh mode, with this clusterfile, send_furl = False engines = { 'tev1' : 16,           'tev2' : 16,           'tev3' : 16,           'tev4' : 16 }

I typically get about 50 engines to actually start. Since there seems to be no log file for ipcluster (in spite of code that seems like it should record which engines it tried to start), I can't send that. The ipcontroller log file looks fine, except for recording fewer than the 64 engines that I expected.

I have an alternative, very klugy method that starts a controller, then executes 64 ssh commands to the respective machines to simply run ipengine. I found the same problem, which went away when I introduced a 1 second delay after each ssh call, which of course takes more than a minute to run, and leaves all those ssh processes running.

So I suspect that the same thing would work in the loop in this method of ipcluster.SSHEngineSet

  def _ssh_engine(self, hostname, count):       exec_engine = "ssh %s sh %s/%s-sshx.sh %s" % (           hostname, self.temp_dir,           os.environ['USER'], self.engine_command       )       cmds = exec_engine.split()       dlist = []       log.msg("about to start engines...")       for i in range(count):           log.msg('Starting engines: %s' % exec_engine)           d = getProcessOutput(cmds[0], cmds[1:], env=os.environ)           dlist.append(d)       return gatherBoth(dlist, consumeErrors=True)

but that would be inelegant, given that the real problem is probably related to the controller not responding properly to multiple requests.

Thanks for looking at this.

--Toby Burnett

-- Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com

[ LP comment 6 by: Toby Burnett, on 2010-01-18 23:13:58.126879+00:00 ]

This was on linux.
So I guess the only issue is a slight delay after starting the ipcontroller. I'll try that, thanks.

Looking at the ipcluster code, (version 0.10) I see that the ssh mode is not supported for windows to linux, a case that we would like to use. Hopefully with the new version when you get it to work>

[ LP comment 7 by: Brian Granger, on 2010-01-18 23:25:06+00:00 ]

Is there a ssh daemon that is usable on Windows. I now about putty
for client ssh and we could use
that. Not sure if users would have an ssh daemon on Windows though.
For Windows, the best
solution is to use the Windows HPC 2008 OS and its builtin job
scheduler (IPython has very good support for this).
Only downside is the cost of the OS ;(

Brian

On Mon, Jan 18, 2010 at 3:13 PM, Toby Burnett tburnett@uw.edu wrote:

This was on linux. So I guess the only issue is a slight delay after starting the ipcontroller. I'll try that, thanks.

Looking at the ipcluster code, (version 0.10) I see that the ssh mode is not supported for windows to linux, a case that we would like to use. Hopefully with the new version when you get it to work>

-- ipcluster does not start all the engines https://bugs.launchpad.net/bugs/509015 You received this bug notification because you are a member of IPython Developers, which is subscribed to IPython.

Status in IPython - Enhanced Interactive Python: New

Bug description: As reported on the mailing list...

---------- Forwarded message ---------- From: Toby Burnett <tburnett@EMAIL-REMOVED> Date: Sat, Jan 16, 2010 at 8:58 AM Subject: [IPython-user] ipcluster does not start all the engines To: "ipython-user@scipy.org" ipython-user@scipy.org

Hi, I did not find any previous notes on this. I have a cluster of 4 machines, each with 8 hyperthreaded cores, so I can run 16 engines per machine, or 64 in all. It is amazingly easy and useful, thanks so much for providing this.

However, when using ipcluster on one of these machines in ssh mode, with this clusterfile, send_furl = False engines = { 'tev1' : 16,           'tev2' : 16,           'tev3' : 16,           'tev4' : 16 }

I typically get about 50 engines to actually start. Since there seems to be no log file for ipcluster (in spite of code that seems like it should record which engines it tried to start), I can't send that. The ipcontroller log file looks fine, except for recording fewer than the 64 engines that I expected.

I have an alternative, very klugy method that starts a controller, then executes 64 ssh commands to the respective machines to simply run ipengine. I found the same problem, which went away when I introduced a 1 second delay after each ssh call, which of course takes more than a minute to run, and leaves all those ssh processes running.

So I suspect that the same thing would work in the loop in this method of ipcluster.SSHEngineSet

  def _ssh_engine(self, hostname, count):       exec_engine = "ssh %s sh %s/%s-sshx.sh %s" % (           hostname, self.temp_dir,           os.environ['USER'], self.engine_command       )       cmds = exec_engine.split()       dlist = []       log.msg("about to start engines...")       for i in range(count):           log.msg('Starting engines: %s' % exec_engine)           d = getProcessOutput(cmds[0], cmds[1:], env=os.environ)           dlist.append(d)       return gatherBoth(dlist, consumeErrors=True)

but that would be inelegant, given that the real problem is probably related to the controller not responding properly to multiple requests.

Thanks for looking at this.

--Toby Burnett

-- Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com

[ LP comment 8 by: Vishal Vatsa, on 2010-01-18 23:39:09+00:00 ]

I have used cygwin and openssh to get a shell on winXP even X11
portfowarding in cygwin work :)

2010/1/18 Brian Granger ellisonbg@gmail.com">ellisonbg@gmail.com:

Is there a ssh daemon that is usable on Windows.  I now about putty for client ssh and we could use that.  Not sure if users would have an ssh daemon on Windows though. For Windows, the best solution is to use the Windows HPC 2008 OS and its builtin job scheduler (IPython has very good support for this). Only downside is the cost of the OS ;(

Brian

On Mon, Jan 18, 2010 at 3:13 PM, Toby Burnett tburnett@uw.edu">tburnett@uw.edu wrote:

This was on linux. So I guess the only issue is a slight delay after starting the ipcontroller. I'll try that, thanks.

Looking at the ipcluster code, (version 0.10) I see that the ssh mode is not supported for windows to linux, a case that we would like to use. Hopefully with the new version when you get it to work>

-- ipcluster does not start all the engines https://bugs.launchpad.net/bugs/509015 You received this bug notification because you are a member of IPython Developers, which is subscribed to IPython.

Status in IPython - Enhanced Interactive Python: New

Bug description: As reported on the mailing list...

---------- Forwarded message ---------- From: Toby Burnett <tburnett@EMAIL-REMOVED> Date: Sat, Jan 16, 2010 at 8:58 AM Subject: [IPython-user] ipcluster does not start all the engines To: "ipython-user@scipy.org" ipython-user@scipy.org

Hi, I did not find any previous notes on this. I have a cluster of 4 machines, each with 8 hyperthreaded cores, so I can run 16 engines per machine, or 64 in all. It is amazingly easy and useful, thanks so much for providing this.

However, when using ipcluster on one of these machines in ssh mode, with this clusterfile, send_furl = False engines = { 'tev1' : 16,           'tev2' : 16,           'tev3' : 16,           'tev4' : 16 }

I typically get about 50 engines to actually start. Since there seems to be no log file for ipcluster (in spite of code that seems like it should record which engines it tried to start), I can't send that. The ipcontroller log file looks fine, except for recording fewer than the 64 engines that I expected.

I have an alternative, very klugy method that starts a controller, then executes 64 ssh commands to the respective machines to simply run ipengine. I found the same problem, which went away when I introduced a 1 second delay after each ssh call, which of course takes more than a minute to run, and leaves all those ssh processes running.

So I suspect that the same thing would work in the loop in this method of ipcluster.SSHEngineSet

  def _ssh_engine(self, hostname, count):       exec_engine = "ssh %s sh %s/%s-sshx.sh %s" % (           hostname, self.temp_dir,           os.environ['USER'], self.engine_command       )       cmds = exec_engine.split()       dlist = []       log.msg("about to start engines...")       for i in range(count):           log.msg('Starting engines: %s' % exec_engine)           d = getProcessOutput(cmds[0], cmds[1:], env=os.environ)           dlist.append(d)       return gatherBoth(dlist, consumeErrors=True)

but that would be inelegant, given that the real problem is probably related to the controller not responding properly to multiple requests.

Thanks for looking at this.

--Toby Burnett

-- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgranger@calpoly.edu ellisonbg@gmail.com

-- ipcluster does not start all the engines https://bugs.launchpad.net/bugs/509015 You received this bug notification because you are a bug assignee.

Status in IPython - Enhanced Interactive Python: New

Bug description: As reported on the mailing list...

---------- Forwarded message ---------- From: Toby Burnett <tburnett@EMAIL-REMOVED> Date: Sat, Jan 16, 2010 at 8:58 AM Subject: [IPython-user] ipcluster does not start all the engines To: "ipython-user@scipy.org" ipython-user@scipy.org

Hi, I did not find any previous notes on this. I have a cluster of 4 machines, each with 8 hyperthreaded cores, so I can run 16 engines per machine, or 64 in all. It is amazingly easy and useful, thanks so much for providing this.

However, when using ipcluster on one of these machines in ssh mode, with this clusterfile, send_furl = False engines = { 'tev1' : 16,           'tev2' : 16,           'tev3' : 16,           'tev4' : 16 }

I typically get about 50 engines to actually start. Since there seems to be no log file for ipcluster (in spite of code that seems like it should record which engines it tried to start), I can't send that. The ipcontroller log file looks fine, except for recording fewer than the 64 engines that I expected.

I have an alternative, very klugy method that starts a controller, then executes 64 ssh commands to the respective machines to simply run ipengine. I found the same problem, which went away when I introduced a 1 second delay after each ssh call, which of course takes more than a minute to run, and leaves all those ssh processes running.

So I suspect that the same thing would work in the loop in this method of ipcluster.SSHEngineSet

  def _ssh_engine(self, hostname, count):       exec_engine = "ssh %s sh %s/%s-sshx.sh %s" % (           hostname, self.temp_dir,           os.environ['USER'], self.engine_command       )       cmds = exec_engine.split()       dlist = []       log.msg("about to start engines...")       for i in range(count):           log.msg('Starting engines: %s' % exec_engine)           d = getProcessOutput(cmds[0], cmds[1:], env=os.environ)           dlist.append(d)       return gatherBoth(dlist, consumeErrors=True)

but that would be inelegant, given that the real problem is probably related to the controller not responding properly to multiple requests.

Thanks for looking at this.

--Toby Burnett

[ LP comment 9 by: Brian Granger, on 2010-01-19 00:00:11+00:00 ]

What about non-cygwin Windows?

On Mon, Jan 18, 2010 at 3:39 PM, Vishal Vatsa vishal.vatsa@gmail.com wrote:

I have used cygwin and openssh to get a shell on winXP even X11 portfowarding in cygwin work :)

2010/1/18 Brian Granger ellisonbg@gmail.com">ellisonbg@gmail.com:

Is there a ssh daemon that is usable on Windows.  I now about putty for client ssh and we could use that.  Not sure if users would have an ssh daemon on Windows though. For Windows, the best solution is to use the Windows HPC 2008 OS and its builtin job scheduler (IPython has very good support for this). Only downside is the cost of the OS ;(

Brian

On Mon, Jan 18, 2010 at 3:13 PM, Toby Burnett tburnett@uw.edu wrote:

This was on linux. So I guess the only issue is a slight delay after starting the ipcontroller. I'll try that, thanks.

Looking at the ipcluster code, (version 0.10) I see that the ssh mode is not supported for windows to linux, a case that we would like to use. Hopefully with the new version when you get it to work>

-- ipcluster does not start all the engines https://bugs.launchpad.net/bugs/509015 You received this bug notification because you are a member of IPython Developers, which is subscribed to IPython.

Status in IPython - Enhanced Interactive Python: New

Bug description: As reported on the mailing list...

---------- Forwarded message ---------- From: Toby Burnett <tburnett@EMAIL-REMOVED> Date: Sat, Jan 16, 2010 at 8:58 AM Subject: [IPython-user] ipcluster does not start all the engines To: "ipython-user@scipy.org" ipython-user@scipy.org

Hi, I did not find any previous notes on this. I have a cluster of 4 machines, each with 8 hyperthreaded cores, so I can run 16 engines per machine, or 64 in all. It is amazingly easy and useful, thanks so much for providing this.

However, when using ipcluster on one of these machines in ssh mode, with this clusterfile, send_furl = False engines = { 'tev1' : 16,           'tev2' : 16,           'tev3' : 16,           'tev4' : 16 }

I typically get about 50 engines to actually start. Since there seems to be no log file for ipcluster (in spite of code that seems like it should record which engines it tried to start), I can't send that. The ipcontroller log file looks fine, except for recording fewer than the 64 engines that I expected.

I have an alternative, very klugy method that starts a controller, then executes 64 ssh commands to the respective machines to simply run ipengine. I found the same problem, which went away when I introduced a 1 second delay after each ssh call, which of course takes more than a minute to run, and leaves all those ssh processes running.

So I suspect that the same thing would work in the loop in this method of ipcluster.SSHEngineSet

  def _ssh_engine(self, hostname, count):       exec_engine = "ssh %s sh %s/%s-sshx.sh %s" % (           hostname, self.temp_dir,           os.environ['USER'], self.engine_command       )       cmds = exec_engine.split()       dlist = []       log.msg("about to start engines...")       for i in range(count):           log.msg('Starting engines: %s' % exec_engine)           d = getProcessOutput(cmds[0], cmds[1:], env=os.environ)           dlist.append(d)       return gatherBoth(dlist, consumeErrors=True)

but that would be inelegant, given that the real problem is probably related to the controller not responding properly to multiple requests.

Thanks for looking at this.

--Toby Burnett

-- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgranger@calpoly.edu ellisonbg@gmail.com

-- ipcluster does not start all the engines https://bugs.launchpad.net/bugs/509015 You received this bug notification because you are a bug assignee.

Status in IPython - Enhanced Interactive Python: New

Bug description: As reported on the mailing list...

---------- Forwarded message ---------- From: Toby Burnett <tburnett@EMAIL-REMOVED> Date: Sat, Jan 16, 2010 at 8:58 AM Subject: [IPython-user] ipcluster does not start all the engines To: "ipython-user@scipy.org" ipython-user@scipy.org

Hi, I did not find any previous notes on this. I have a cluster of 4 machines, each with 8 hyperthreaded cores, so I can run 16 engines per machine, or 64 in all. It is amazingly easy and useful, thanks so much for providing this.

However, when using ipcluster on one of these machines in ssh mode, with this clusterfile, send_furl = False engines = { 'tev1' : 16,           'tev2' : 16,           'tev3' : 16,           'tev4' : 16 }

I typically get about 50 engines to actually start. Since there seems to be no log file for ipcluster (in spite of code that seems like it should record which engines it tried to start), I can't send that. The ipcontroller log file looks fine, except for recording fewer than the 64 engines that I expected.

I have an alternative, very klugy method that starts a controller, then executes 64 ssh commands to the respective machines to simply run ipengine. I found the same problem, which went away when I introduced a 1 second delay after each ssh call, which of course takes more than a minute to run, and leaves all those ssh processes running.

So I suspect that the same thing would work in the loop in this method of ipcluster.SSHEngineSet

  def _ssh_engine(self, hostname, count):       exec_engine = "ssh %s sh %s/%s-sshx.sh %s" % (           hostname, self.temp_dir,           os.environ['USER'], self.engine_command       )       cmds = exec_engine.split()       dlist = []       log.msg("about to start engines...")       for i in range(count):           log.msg('Starting engines: %s' % exec_engine)           d = getProcessOutput(cmds[0], cmds[1:], env=os.environ)           dlist.append(d)       return gatherBoth(dlist, consumeErrors=True)

but that would be inelegant, given that the real problem is probably related to the controller not responding properly to multiple requests.

Thanks for looking at this.

--Toby Burnett

-- ipcluster does not start all the engines https://bugs.launchpad.net/bugs/509015 You received this bug notification because you are a member of IPython Developers, which is subscribed to IPython.

Status in IPython - Enhanced Interactive Python: New

Bug description: As reported on the mailing list...

---------- Forwarded message ---------- From: Toby Burnett <tburnett@EMAIL-REMOVED> Date: Sat, Jan 16, 2010 at 8:58 AM Subject: [IPython-user] ipcluster does not start all the engines To: "ipython-user@scipy.org" ipython-user@scipy.org

Hi, I did not find any previous notes on this. I have a cluster of 4 machines, each with 8 hyperthreaded cores, so I can run 16 engines per machine, or 64 in all. It is amazingly easy and useful, thanks so much for providing this.

However, when using ipcluster on one of these machines in ssh mode, with this clusterfile, send_furl = False engines = { 'tev1' : 16,           'tev2' : 16,           'tev3' : 16,           'tev4' : 16 }

I typically get about 50 engines to actually start. Since there seems to be no log file for ipcluster (in spite of code that seems like it should record which engines it tried to start), I can't send that. The ipcontroller log file looks fine, except for recording fewer than the 64 engines that I expected.

I have an alternative, very klugy method that starts a controller, then executes 64 ssh commands to the respective machines to simply run ipengine. I found the same problem, which went away when I introduced a 1 second delay after each ssh call, which of course takes more than a minute to run, and leaves all those ssh processes running.

So I suspect that the same thing would work in the loop in this method of ipcluster.SSHEngineSet

  def _ssh_engine(self, hostname, count):       exec_engine = "ssh %s sh %s/%s-sshx.sh %s" % (           hostname, self.temp_dir,           os.environ['USER'], self.engine_command       )       cmds = exec_engine.split()       dlist = []       log.msg("about to start engines...")       for i in range(count):           log.msg('Starting engines: %s' % exec_engine)           d = getProcessOutput(cmds[0], cmds[1:], env=os.environ)           dlist.append(d)       return gatherBoth(dlist, consumeErrors=True)

but that would be inelegant, given that the real problem is probably related to the controller not responding properly to multiple requests.

Thanks for looking at this.

--Toby Burnett

-- Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com

[ LP comment 10 by: Toby Burnett, on 2010-01-19 15:56:55.304633+00:00 ]

About ssh on windows, we use the cygwin version, but not in the cygwin bash shell.
C:\Users\burnett>ssh -V
OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004

I tried a version of my script that starts an ipcontroller, then 16 ipengines on each of the 4 machines. It I delay after the ipcontroller, I have the same problem: 2-3 engines don't get started per machine. However, a 100 ms delay in that loop that creates the engines works just fine.

This seems inconsistent with an assertion about how the engines connect to a controller.

[ LP comment 11 by: Vishal Vatsa, on 2010-01-19 16:30:54+00:00 ]

Would you mind sending me a copy of your script.
Also, is the setup like:
ipcontroller is on windows and ipengines are on linux

-v

2010/1/19 Toby Burnett tburnett@uw.edu:

About ssh on windows, we use the cygwin version, but not in the cygwin bash shell. C:\Users\burnett>ssh -V OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004

I tried a version of my script that starts an ipcontroller, then 16 ipengines on each of the 4 machines. It I delay after the ipcontroller, I have the same problem: 2-3 engines don't get started per machine. However, a 100 ms delay in that loop that creates the engines works just fine.

This seems inconsistent with an assertion about how the engines connect to a controller.

-- ipcluster does not start all the engines https://bugs.launchpad.net/bugs/509015 You received this bug notification because you are a bug assignee.

Status in IPython - Enhanced Interactive Python: New

Bug description: As reported on the mailing list...

---------- Forwarded message ---------- From: Toby Burnett <tburnett@EMAIL-REMOVED> Date: Sat, Jan 16, 2010 at 8:58 AM Subject: [IPython-user] ipcluster does not start all the engines To: "ipython-user@scipy.org" ipython-user@scipy.org

Hi, I did not find any previous notes on this. I have a cluster of 4 machines, each with 8 hyperthreaded cores, so I can run 16 engines per machine, or 64 in all. It is amazingly easy and useful, thanks so much for providing this.

However, when using ipcluster on one of these machines in ssh mode, with this clusterfile, send_furl = False engines = { 'tev1' : 16,           'tev2' : 16,           'tev3' : 16,           'tev4' : 16 }

I typically get about 50 engines to actually start. Since there seems to be no log file for ipcluster (in spite of code that seems like it should record which engines it tried to start), I can't send that. The ipcontroller log file looks fine, except for recording fewer than the 64 engines that I expected.

I have an alternative, very klugy method that starts a controller, then executes 64 ssh commands to the respective machines to simply run ipengine. I found the same problem, which went away when I introduced a 1 second delay after each ssh call, which of course takes more than a minute to run, and leaves all those ssh processes running.

So I suspect that the same thing would work in the loop in this method of ipcluster.SSHEngineSet

  def _ssh_engine(self, hostname, count):       exec_engine = "ssh %s sh %s/%s-sshx.sh %s" % (           hostname, self.temp_dir,           os.environ['USER'], self.engine_command       )       cmds = exec_engine.split()       dlist = []       log.msg("about to start engines...")       for i in range(count):           log.msg('Starting engines: %s' % exec_engine)           d = getProcessOutput(cmds[0], cmds[1:], env=os.environ)           dlist.append(d)       return gatherBoth(dlist, consumeErrors=True)

but that would be inelegant, given that the real problem is probably related to the controller not responding properly to multiple requests.

Thanks for looking at this.

--Toby Burnett

[ LP comment 12 by: Toby Burnett, on 2010-01-19 18:40:49.459167+00:00 ]

Here is the script, with an attempt to use ipcluster ssh commented out. The 64-engine case is run on one of the same nodes.
def setup_mec(engines=None, machines='tev1 tev2 tev3 tev4'.split()):

""" On windows:start cluster and 4 engines on the local machine, in the current directory
    On linux: (our tev cluster) start controller on local machine, 16 engines/machine in all
"""
if os.name=='nt':
    engines = engines or 4
    os.system(r'start /MIN /D %s cmd /K python C:\python25\scripts\ipcluster local -xy -n %d'% (os.getcwd(),engines))
else:
    # on a tev machine
    engines = engines or 16 #default on a tev machine!

    #clusterfile_data='send_furl = False'\
    #    + '\nengines={'\
    #    + '\n'.join(['\t"%s" : %d,'%(m,engines) for m in machines])\
    #    + '\n}'
    #print 'cluster info:\n%s' % clusterfile_data
    #ofile=open('clusterfile', 'w')
    #ofile.writelines(clusterfile_data)
    #ofile.close()
    #os.system('ipcluster ssh -xy --clusterfile clusterfile &')

    # old, klugy way
    os.system('ipcontroller local -xy&')  
    for m in machines:
        for i in range(engines):
            time.sleep(0.1) # make sure the controller is started ?
            os.system('ssh %s ipengine&'% m) # this assumes that the environment is setup with non-interactive login

[ LP comment 1 by: Murkt, on 2009-03-08 20:07:25.828411+00:00 ]

This line in trunk: http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython//iplib.py#L2031

[ LP comment 2 by: Murkt, on 2009-03-08 20:54:18.976119+00:00 ]

This bug was noticed in 2006 year: http://lists.ipython.scipy.org/pipermail/ipython-dev/2006-August/002305.html

[ LP comment 3 by: Sergey Kishchenko, on 2009-03-09 11:57:58.714729+00:00 ]

I confirm this bug. Attached patch fixed the issue for me

[ LP comment 4 by: Laurent Dufrechou, on 2009-03-17 20:20:24.848465+00:00 ]

also related :
https://bugs.launchpad.net/bugs/290677

[ LP comment 5 by: Fernando Perez, on 2009-03-17 20:24:00.727454+00:00 ]

That's indeed a bug, but the patch is removing a line that was put in there explicitly for some reason. So what I'd like to have, before committing this, is a set of tests in a file named test_unicode.py, that encapsulates all of the recent unicode work.

Unfortunately a lot of these unicode fixes have been made in a completely ad-hoc manner, as people report problems, but we don't have a centralized list of cases to check against. His may be a reasonable fix, for all I know, but I'm afraid that if we apply it we'll get back 10 old bugs again. I don't know, maybe not, but there's simply no way to be sure.

I'm one of the most ignorant of our bunch in unicode issues, blissfully living in the stupidity of the ascii world. It would be great if one of us who knows more about this stuff could at least write a set of simple unicode tests that catch many of the recently reported encoding problems. Jorgen, Ville, any chance you guys could take this up at some point? You know about it a lot more than I do...

[ LP comment 6 by: Jörgen Stenarson, on 2009-03-17 20:28:38.963967+00:00 ]

The proposed patch does not work for me on win32 with or without pyreadline

sys.stdin.encoding == "cp1252"

Standard python:

c:\python>python
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

"åäö" '\xe5\xe4\xf6' u"åäö" u'\xe5\xe4\xf6'

IPython from trunk:

c:\python>ipython
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)]
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: "åäö"
Out[1]: '\xe5\xe4\xf6'

In [2]: u"åäö"
Out[2]: u'\xe5\xe4\xf6'

In [3]:
Do you really want to exit ([y]/n)?

IPython with proposed change:

c:\python>ipython
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)]
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: "åäö"
Out[1]: '\xc3\xa5\xc3\xa4\xc3\xb6'

In [2]: u"åäö"
Out[2]: u'\xe5\xe4\xf6'

In [3]:
Do you really want to exit ([y]/n)?

[ LP comment 7 by: Rodrigo Senra, on 2009-03-24 03:46:20.603219+00:00 ]

This bugis still live and kicking.
The problem is in iplib.py: source=source.encode(self.stdin_encoding)

This is wrong whenever there is a unicode string in source.

A simple:

x = u"ação"

with the offending line becomes:

x = u'a\xc3\xa7\xc3\xa3o'

Notice that the encoding is done inplace,and the u"" is kept after the encoding. This is wrong.
I have removed the line, and it is now working for me. Do not know enough of IPython internals to predict side effects. I hope this helps.
regards,
Rod Senra

[ LP comment 8 by: INADA Naoki, on 2009-04-12 00:25:48.950763+00:00 ]

This is another patch that handle encoded byte string literal and unicode literal correctly.

    source=source.encode(self.stdin_encoding)
    if source[:1] in [u' ', u'\t']:
        source = u'if 1:\n%s' % source

•  source = '# coding: %s\n%s' % (self.stdin_encoding, source)
  

[ LP comment 9 by: Fernando Perez, on 2009-04-14 07:20:50+00:00 ]

Can anyone provide a set of tests that we can actually run
automatically for this? Honestly, until we have actual tests, this is
like playing whack-a-mole blind: the problems will just keep
resurfacing... What we need is a test file for unicode that can be
run reliably, by anyone, and that shows the various issues...

As I said earlier, it's quite possible that the various proposed fixes
work for someone, but without actual tests that we can include,
there's no way to know what they may break for someone else (as has
happened in the past).

Sorry to seem like a curmudgeon: I really appreciate people
contributing ideas and even code. But we need to fix these unicode
problems the right way, else we'll be hunting them forever.

[ LP comment 10 by: Brian Granger, on 2009-04-14 17:38:38+00:00 ]

Definitely, I don't like playing whack-a-mole blind. These types of
bug fixes definitely need tests before fixes get commited.

Brian

On Tue, Apr 14, 2009 at 12:20 AM, Fernando Perez fperez.net@gmail.com wrote:

Can anyone provide a set of tests that we can actually run automatically for this? Honestly, until we have actual tests, this is like playing whack-a-mole blind: the problems will just keep resurfacing... What we need is a test file for unicode that can be run reliably, by anyone, and that shows the various issues...

As I said earlier, it's quite possible that the various proposed fixes work for someone, but without actual tests that we can include, there's no way to know what they may break for someone else (as has happened in the past).

Sorry to seem like a curmudgeon: I really appreciate people contributing ideas and even code. But we need to fix these unicode problems the right way, else we'll be hunting them forever.

-- unicode bug - encoding input https://bugs.launchpad.net/bugs/339642 You received this bug notification because you are a member of IPython Developers, which is subscribed to IPython.

Status in IPython - Enhanced Interactive Python: New

Bug description: Default Python shell:

u'абвгд' u'\u0430\u0431\u0432\u0433\u0434'

IPython 0.9.1:

u'абвгд' u'\xd0\xb0\xd0\xb1\xd0\xb2\xd0\xb3\xd0\xb4' 'абвгд'.decode('utf8') u'\u0430\u0431\u0432\u0433\u0434'

sys.stdin.encoding is 'UTF-8'.

How to fix: remove the line No.2022 from IPython/iplib.py (for 0.9.1 release). Here it is:

--- a/iplib.py +++ b/iplib.py @@ -2019,7 +2019,6 @@

    # this allows execution of indented pasted code. It is tempting
    # to add '\n' at the end of source to run commands like ' a=1'
    # directly, but this fails for more complicated scenarios

•  source=source.encode(self.stdin_encoding)
  if source[:1] in [' ', '\t']:
      source = 'if 1:\n%s' % source
  

I didn't find any intoduced bugs by a quick check.

Additionaly, I checked ipython-wx and ipythonx - latter doesn't have this bug.

-- Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com

[ LP comment 11 by: Jörgen Stenarson, on 2009-04-14 18:16:27+00:00 ]

Fernando Perez skrev:

Can anyone provide a set of tests that we can actually run automatically for this? Honestly, until we have actual tests, this is like playing whack-a-mole blind: the problems will just keep resurfacing... What we need is a test file for unicode that can be run reliably, by anyone, and that shows the various issues...

As I said earlier, it's quite possible that the various proposed fixes work for someone, but without actual tests that we can include, there's no way to know what they may break for someone else (as has happened in the past).

Sorry to seem like a curmudgeon: I really appreciate people contributing ideas and even code. But we need to fix these unicode problems the right way, else we'll be hunting them forever.

I agree, but part of the problem here is that part of the problem is to
have the correct visual output in the shell and this may be difficult to
check automatically. I have a feeling that this problem is also platform
dependent making it necessary to run the tests on several platforms as
well to see that the bug has been fixed.

/Jörgen

[ LP comment 12 by: Fernando Perez, on 2009-04-14 21:58:00+00:00 ]

On Tue, Apr 14, 2009 at 11:16 AM, Jörgen Stenarson
jorgen.stenarson@bostream.nu wrote:

Fernando Perez skrev:

I agree, but part of the problem here is that part of the problem is to have the correct visual output in the shell and this may be difficult to check automatically. I have a feeling that this problem is also platform dependent making it necessary to run the tests on several platforms as well to see that the bug has been fixed.

Well, even if we have a special file we need to re-run by hand, that
would be better than little snippets as we have. At least the file
can be run by the test suite automatically and not crashing is a good
start. Core developers can then re-run it by hand (we can put an "if
__name__" main section at the bottom for this) to check visually. This is basically what we are doing now with snippets all over the
mailing list, I'm just suggesting that unless all those checks are:

• collected in one file
• auto-executed

we'll never get anywhere reliable on these unicode problems. We can
then have a note to manually do

%run test_unicode

ourselves for the full visual verification.

Cheers,

f

[ LP comment 13 by: gdamjan, on 2009-05-02 01:04:38.272380+00:00 ]

I can confirm this bug and the sollution given.

Now obviously the bug is in the input handling of ipython .. how do you make test cases for that??

[ LP comment 14 by: Andy Mikhailenko, on 2009-05-14 20:52:29.106176+00:00 ]

Confirming. "UTF-8" in all cases, IPython prints screwed up "unicode" strings and this renders the program almost unusable.

Anyone got ideas about how to test this? I guess IPython developers possess a bit more knowledge of the immense innards of the package than reporters of the bug do, so users could expect at least some guidelines for writing tests, could they?

Maybe we should allow to tune bug-related behaviour in user settings until the bug is finally fixed? This may also help with testing.

[ LP comment 15 by: pawciobiel, on 2009-09-04 00:24:55.923328+00:00 ]

Confirming.

core/iplib.py
2201
--- source=source.encode(self.stdin_encoding)

Apart of the above, shouldn't the input be decoded if it's not unicode?
(Similar issue was in python2.5/code.py) core/iplib.py
2332,2334d2330
< line = raw_input_original(prompt)
< if not isinstance(line, unicode):
< line = line.decode(self.stdin_encoding)

cheers,

[ LP comment 16 by: INADA Naoki, on 2009-10-08 03:52:22.535452+00:00 ]

I manage to fix this bug in Python side: http://bugs.python.org/issue5911
But if the python issue is fixed in Python 2.7, this problem is still in Python 2.6 and lower.

[ LP comment 17 by: t0ster, on 2010-04-26 16:45:21.798105+00:00 ]

Patches worked for me, removing 'source=source.encode(self.stdin_encoding)' helped in Mac OS X 10.6

Thanks

[ LP comment 1 by: avdd, on 2009-07-16 03:48:57.797382+00:00 ]

Has nothing to do with pickle, but the FakeModule that is registered as main:

$ python Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import sys sys.modules[name].foo = 1 foo 1

a$ ipython

In [1]: import sys

In [2]: sys.modules[name].foo = 1

In [3]: foo

NameError Traceback (most recent call last)

/home/avdd/ in ()

NameError: name 'foo' is not defined

In [4]:

[ LP comment 2 by: Matt Hickford, on 2009-07-18 19:36:43.821085+00:00 ]

I have problem that neither pickle nor cPickle work in ipython (but work in python proper).

In my case pickle.dump works but pickle.

This is Jaunty, Python 2.6

My error:
AttributeError: 'FakeModule' object has no attribute 'Dictionary'

(I have a custom class inherited from list called Dictionary)

[ LP comment 3 by: Ville M. Vainio, on 2009-07-18 20:11:20+00:00 ]

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickfordmatt.hickford+launchpad@gmail.com wrote:

I have problem that neither pickle nor cPickle work in ipython (but work in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

-- Ville M. Vainio
http://tinyurl.com/vainio

[ LP comment 4 by: reckoner, on 2009-07-19 23:50:39+00:00 ]

If you get the class using the %run magic instead of importing, you DO
get the pickle failure. I don't know if doing %run is the same as
defining a class interactively, though.

Any guidance appreciated.

thanks.

On Sat, Jul 18, 2009 at 1:11 PM, Ville M. Vainiovivainio@gmail.com wrote:

On Sat, Jul 18, 2009 at 10:36 PM, Matt Hickfordmatt.hickford+launchpad@gmail.com wrote:

I have problem that neither pickle nor cPickle work in ipython (but work in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable object is not declared interactively, but rather imported from module.

-- Ville M. Vainio http://tinyurl.com/vainio

-- cPickle works in standard interpreter, but not in IPython https://bugs.launchpad.net/bugs/363115 You received this bug notification because you are a direct subscriber of the bug.

Status in IPython - Enhanced Interactive Python: Confirmed

Bug description:

for example:

In IPython:

class Mylist(list):       def init(self,x=[]):           list.init(self,x)

from cPickle import dumps w=Mylist([1,2,3]) dumps(w)

PicklingError: Can't pickle : attribute lookup main.Mylist failed

However, using the standard Python interpreter:

Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.

from cPickle import dumps class Mylist(list): ...         def init(self,x=[]): ...             list.init(self,x) ... w=Mylist([1,2,3]) dumps(w) 'ccopy_reg\n_reconstructor\np1\n(cmain\nMylist\np2\ncbuiltin\nlist\np3\n (lp4\nI1\naI2\naI3\natRp5\n.'

[ LP comment 5 by: eteq, on 2009-12-17 01:34:20.367155+00:00 ]

I've encountered this in the past - I always just define any class I want to pickle in a file and then reload the module as necessary instead of pickling outputs from run or interactively-defined objects. That seems safer anyway - otherwise if you restart the interpreter, the methods on the object are forgotten.

It seems to work fine if you pickle from a file and then load that file with an interactively-defined or defined in "run" version, though.

[ LP comment 6 by: Zak Stone, on 2009-12-17 03:02:24.672555+00:00 ]

For me, a similar problem arises when I am simply unpickling dictionaries in ipython -- no custom classes involved.

[ LP comment 1 by: eteq, on 2009-09-04 00:41:46.232025+00:00 ]

This is happening for me as well in 0.10 - it's pretty annoying in that it renders dreload (a very useful function) essentially useless. It doesn't even work to do dreload(somemodulethatusesnumpy,exclude=['numpy','numpy.numpy','numpy.show'])

[ LP comment 1 by: Brian Granger, on 2009-04-24 17:07:44.027943+00:00 ]

For now, we are simply skipping the doctesting of Shell in iptest. When this issue is fixed, we need to start doctesting Shell.py

[ LP comment 1 by: Joe Covalesky, on 2008-06-06 17:39:46.864436+00:00 ]

This workaround didn't fix this problem for me.

WinXP Pro SP2
Emacs 22.1.1
ipython.el 2927
python-mode.el 4.75
pyreadline 1.3 or trunk (post-1.5)

[ LP comment 2 by: Brian Granger, on 2009-04-14 20:21:09.437714+00:00 ]

Any thoughts on this from Windows/Emacs users?