python

A 29-post collection

Conda (and Anaconda) for python package management and virtual envs

Of installing scipy

I have once wrote a way to install scipy on Windows 64 bit here

scipy is notoriously hard to install on Windows. It's not a bug or anythnig, it is by design.

Python is a super dynamic language designed to be compatible with C extensions.

That means, Python is affordable to be a little bit slow while keeping its interface with C extensions easy and fast. The advent of GIL (Global Interpreter Lock) in CPython is the testimony of this.

(Somebody has summarized why Python (CPython) uses GIL even though it renders multi-core CPUs useless, it's a good and short read here)

In this case, Python can be as dynamic as it wants, and let the C extensions do the dirty and fast part.

However, it comes with a cost, C is not a write once run anywhere programming language. Most of the time, you need to compile it on your machine to guarantee the usability. If you have a good C compiler installed this should not be much of the problem. But most of us, especially Windows users, don't.

Many people have been trying to circumvent this limitation, thus far I have concerned, many packages can now easily be installed by pip without much to do with any compiler i.e. numpy, that means somebody has compiled it beforehand for you.

But that doesn't include scipy. You cannot install it (as of early 2017), you cannot install it using pip without proper configured compiler on Windows with Python 64 bit version installed.

There is some other way namely conda (bare conda can be installed via Miniconda) and Anaconda (which is a full-fledged python install for scientific use, also comes with conda you have no need to install both) by one of which you can install numpy

อ่านต่อ »

Python 3 (64 bit) + scipy on Windows

I tried original one, but didn't support scipy

(x64) I just can't install scipy as a matter of fact, scipy doesn't provide pre-compiled solution for Windows in the first place, and installing python in way, I think, I have no hope for it.

So next, I go for another option which I can properly setup my complier, namely cygwin.

I have read somewhere, it will work if I install python (32 bit) instead. But why should I, this is 2017!?

I tried cygwin ... it failed to build scipy

I first installed these packages from the cygwin's installer.

  1. python3
  2. python3-devel (you need this if you want to install packgaes like numpy)
  3. python3-pip

After installation, Python 3 will take the name of python3 to make it work as a default python I did some ln -s and some export PATH to make it present in the default PATH.

I tried to install numpy which I did install successfully. (arduously long install due to the compilation time)

Then, install scipy now it failed with the following error:

numpy.distutils.system_info.NotFoundError: no lapack/blas resources found  

I didn't seem to find any solutions or workaround online ... so I paused there !

So, I give conda a try ... It seems to work

You don't have to install the full-big anaconda to use conda, you can just install the smaller miniconda it will do the job.

I think conda has its own way to install packages, I think it still leverage the use of pip in some point but also with some modifications. So in short, conda is a front-end to multi-platform mulit-language package installation.

When I install numpy scipy it seemed to me that conda also install mkl (I don't know much about this, it's like a math optimization library for Intel CPUs or

อ่านต่อ »

Pushing Python Project to Pypi using Twine

(thread: https://github.com/pypa/twine#why-should-i-use-this)

You should now prefer wheel instead of egg why ? (https://packaging.python.org/wheel_egg/)

And also, with the new pypi.io, you don't really need to register your package, just go to the upload step. (https://github.com/pypa/twine/issues/200)

Build

pip install wheel  
python setup.py sdist bdist_wheel # building wheel  

Register (if needed; not needed for pypi.io)

twine register dist/...  

Upload

twine upload dist/...<file .whl>  
อ่านต่อ »

Python turn async function back to sync

Actually, this is straightforward and those who works with asyncio library use all the time.

It is asyncio.get_event_look().run_until_complete(...)

Now, I will just wrap it into a nicer package, like this:

def force_sync(fn):  
    '''
    turn an async function to sync function
    '''
    import asyncio

    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        res = fn(*args, **kwargs)
        if asyncio.iscoroutine(res):
            return asyncio.get_event_loop().run_until_complete(res)
        return res

    return wrapper

Here is how to use it:

@force_sync
async def run():  
    ... do some async works ...

results = run()  

For those who look for the vice versa, turning a sync to async, I have wrote it here

อ่านต่อ »

Python turn sync functions to async and awaitable using threads

Not all io bound commands are ported to asynchronous reducing somehow the worthy of asyncio library. In fact, we can interpolate the use of both async and sync functions with relative ease of implementation.

I will try using threading interfaces to do the job, in the end I will force_async on a synchronous function and turn it into async one.

def force_async(fn):  
    '''
    turns a sync function to async function using threads
    :param sync function:
    :return async funciton:
    '''
    from concurrent.futures import ThreadPoolExecutor
    import asyncio
    pool = ThreadPoolExecutor()

    def wrapper(*args, **kwargs):
        future = pool.submit(fn, *args, **kwargs)
        return asyncio.wrap_future(future)  # make it awaitable

    return wrapper

Usage:

@force_async
def long_blocking_function():  
    import time
    time.sleep(10)
    return True

import asyncio  
async def run():  
    a = long_blocking_function()
    b = long_blocking_function()
    return await asyncio.gather(a, b) # [True, True] in 10 seconds

The main idea of the trick is to run a sync task in a thread (returning a future object, coroutine.Future) and then turn this future object into a asyncio.Future object. Obivously, this future object will be resolve when the sync function is done, and also is awaitable.

If you're looking for turning async function into sync function, which is much more obivous, see here

อ่านต่อ »

Matplotlib Set Figure Size

import matplotlib.pylot as plt  
fig, axes = plt.subplots(...)  
# ndarray size [w, h] in inches
# with default DPI set to 100
size = fig.get_size_inches() * 2  
fig.set_size_inches(size[0], size[1])  

The code snippet above set the figure size to be 4 times as large as the default 8 x 6 sq.inches, while keeping DPI to stay the same this will make things appear to be smaller.

อ่านต่อ »