Konpat Preechakul

105 posts

Cygwin Problems with CR/LF newline

Upon finding this short post, you might have encountered the notoriously newline issue in Windows. The more you try to be UNIX-like on Windows the more the issue pronounces itself.

For a prominent example, when you are trying to build some certain program on Windows. You will more than likely to encounter the excess \r problem (or in some circumstances will you see ^M). I am not certainly sure how it happens but it is sure that any native Linux program which we runs on Windows will struggle to understand \r and must fail.

So this is where dos2unix and unix2dos commands come in to play. It mitigates the problem if your source of problem comes from the file system itself.

But, it is not the only source of problem.

Some commands just output with a newline and it might have problems, for example:

jupyter --data-dir  

Which should output the data-dir for jupyter in this case it might be C:\Users\<user>\AppData\Roaming\jupyter, but in practices, it is not, it has a newline, and not only \n conventional newline. It comes with \r\n newline. which becomes a problem when you trying to substitute this command in another command like:

$(jupyter --data-dir)/abc

You will not get C:\Users\<user>\AppData\Roaming\jupyter/abc as you might have expected. You will rather get C:\Users\<user>\AppData\Roaming\jupyter^M/abc. This breaks the command !

As you can see, it is really a problem and cannot be solved by simply running dos2unix because it is not a file.

Fortunately, there is a way to make bash completely ignore \r character. It only works on bash.

In .bash_profile

# ignore carriage return
export SHELLOPTS  
set -o igncr  

And then source it source .bash_

อ่านต่อ »

Async Await with Timeout

It's not trivial to put a timeout on an async process.

But, here is how you do it.

Let's say we have some delay function defined like this one:

async function delay(ms) {  
    return new Promise((res, rej) => {
        setTimeout(() => {
        }, ms)

It's quite a handy function, considering using it like await delay(1000)

And extend this function to get a timeout function like this:

async function timeout(ms) {  
    await delay(ms)
    throw new Error('timeout')

Suppose you have some long running function around which you want to have a timeout:

async function longRunning() {  

Obviously, you will call it this way await longRunning().

Now, you can put a timeout around it using Promise.race, which is a handy function to get only the one that resolved earliest, this way:

const res = await Promise.race([  

You now have it ! In a normal case, longRunning() is expected to be resolved earlier than timeout(1000) and the const res is the return value of that longRunning() function, but when the timeout(1000) is resolved earlier, it throws causing await Proimse.race to throw as well, no return value recorded.

อ่านต่อ »

Conda (and Anaconda) for python package management and virtual envs

Of installing scipy

I have once wrote a way to install scipy on Windows 64 bit here

scipy is notoriously hard to install on Windows. It's not a bug or anythnig, it is by design.

Python is a super dynamic language designed to be compatible with C extensions.

That means, Python is affordable to be a little bit slow while keeping its interface with C extensions easy and fast. The advent of GIL (Global Interpreter Lock) in CPython is the testimony of this.

(Somebody has summarized why Python (CPython) uses GIL even though it renders multi-core CPUs useless, it's a good and short read here)

In this case, Python can be as dynamic as it wants, and let the C extensions do the dirty and fast part.

However, it comes with a cost, C is not a write once run anywhere programming language. Most of the time, you need to compile it on your machine to guarantee the usability. If you have a good C compiler installed this should not be much of the problem. But most of us, especially Windows users, don't.

Many people have been trying to circumvent this limitation, thus far I have concerned, many packages can now easily be installed by pip without much to do with any compiler i.e. numpy, that means somebody has compiled it beforehand for you.

But that doesn't include scipy. You cannot install it (as of early 2017), you cannot install it using pip without proper configured compiler on Windows with Python 64 bit version installed.

There is some other way namely conda (bare conda can be installed via Miniconda) and Anaconda (which is a full-fledged python install for scientific use, also comes with conda you have no need to install both) by one of which you can install numpy

อ่านต่อ »

Python 3 (64 bit) + scipy on Windows

I tried original one, but didn't support scipy

(x64) I just can't install scipy as a matter of fact, scipy doesn't provide pre-compiled solution for Windows in the first place, and installing python in way, I think, I have no hope for it.

So next, I go for another option which I can properly setup my complier, namely cygwin.

I have read somewhere, it will work if I install python (32 bit) instead. But why should I, this is 2017!?

I tried cygwin ... it failed to build scipy

I first installed these packages from the cygwin's installer.

  1. python3
  2. python3-devel (you need this if you want to install packgaes like numpy)
  3. python3-pip

After installation, Python 3 will take the name of python3 to make it work as a default python I did some ln -s and some export PATH to make it present in the default PATH.

I tried to install numpy which I did install successfully. (arduously long install due to the compilation time)

Then, install scipy now it failed with the following error:

numpy.distutils.system_info.NotFoundError: no lapack/blas resources found  

I didn't seem to find any solutions or workaround online ... so I paused there !

So, I give conda a try ... It seems to work

You don't have to install the full-big anaconda to use conda, you can just install the smaller miniconda it will do the job.

I think conda has its own way to install packages, I think it still leverage the use of pip in some point but also with some modifications. So in short, conda is a front-end to multi-platform mulit-language package installation.

When I install numpy scipy it seemed to me that conda also install mkl (I don't know much about this, it's like a math optimization library for Intel CPUs or

อ่านต่อ »

Pushing Python Project to Pypi using Twine

(thread: https://github.com/pypa/twine#why-should-i-use-this)

You should now prefer wheel instead of egg why ? (https://packaging.python.org/wheel_egg/)

And also, with the new pypi.io, you don't really need to register your package, just go to the upload step. (https://github.com/pypa/twine/issues/200)


pip install wheel  
python setup.py sdist bdist_wheel # building wheel  

Register (if needed; not needed for pypi.io)

twine register dist/...  


twine upload dist/...<file .whl>  
อ่านต่อ »

Mass upload and download client for all your cloud drives

Not all platforms are supported with native clients, even the simple tasks of downloand and uploading (not even syncing) ... I have gone so far as buliding my own download and upload client with the cloud drive of choice, but to make it reliable one must be very resourceful to do so.

I have found rclone (rsync for cloud drive; link) app just recently, it worked just like I always wanted to do, simple upload and download a lot of files.

Just download, you might put it at /usr/local/bin

rclone config  

Now, to upload

rclone copy <local_path> <remote_name>:<remote_path>  

If you want to upload to the root at remote, you can just leave the blank.

To download, you simple swap the two

rclone copy <remote_name>:<remote_path> <local_path>  

It will try to skip making changes to the existing files, mostly by the file size.

อ่านต่อ »

Run Commands in Docker by Preserving Rights and Ownerships

docker run --user ${UID} ...  

To run commands in a container by using the same user as whoami.

It's not perfect though, it has the very limitation that the container won't know the actual user name of the user ${UID} which, in many cases, is not acceptable.

If you want to make it more realistic, make the container really aware of this user and its username, I'm afraid that you might have to mount the /etc/passwd to the container which is in most ways not favorable.

docker run --user ${UID} -v /etc/passwd:/etc/passwd:ro ...  

But, it managed to work quite well.

Another way that might serve you well is to run the container with root just like always, but run specific commands with hand-curated specific users like:

In Dockerfile:

# run with user id
RUN sudo -u "#<userid>" <command>  
# run with username
RUN sudo -u "<username>" <command>  

Note: you have to apt-get install sudo to do this. You still need to mount the /etc/passwd.

อ่านต่อ »