node

A 4-post collection

You should handle timeout in all the IO callbacks in NodeJS

At first, you might hope that just calls the function and let it tell you just when the job is done and you wait ... and wait. Soon, you realize that the callback you have been waiting will never come.

This is true, and I can pretty much guarantee you that. I have been working with a project involves in heavy IO operations. Even I try to fine tune the speed at which node reads. It is not suffice as I have seen. It soon slows down and stops completely with the cause I have yet to find.

So, I decided that I have to put some timeout, which is not provided by default means, here is my attempt:

export async function delay(ms: number) {  
    return new Promise<void>((res, rej) => {
        setTimeout(() => res(), ms)
    })
}

class Timeout extends Error {}  
export async function delayedError(timeout: number) {  
    await delay(timeout)
    throw new Timeout()
}

export async function withTimeout<T>(promise: Promise<T>, timeout: number) {  
    return await Promise.race<T>([
        promise,
        delayedError(timeout) as any
    ])
}

Let's further say that wrap the fs.readFile with this async:

import fs = require('fs')

export async function readFile(path: string) {  
    return new Promise<Buffer>((res, rej) => {
        fs.readFile(path, (err, content) => {
            if (err) rej(err)
            else res(content)
        })
    })
}

To use it, now you can:

try {  
    await withTimeout(readFile('test.txt'), 1000)
} catch (err) {
    if (err instanceof Timeout) {
        console.error('timeout')
    } else {
        throw err
    }
}
อ่านต่อ »

NodeJS to check whether the file is being written by another process

To me at first, it should have a simple graceful solution, but turned out it has none.

I will try to convince you by the following scenario.

Writer

I have a python writer (to make sure the file will be written by another process rather than the node itself), here is the code:

import os  
import time  
from functools import partial

def copy(a, b):  
    with open(a, 'rb') as r:
        with open(b, 'wb', 0) as w:
            for chunk in iter(partial(r.read, 4 * 1024), b''):
                w.write(chunk)
                time.sleep(0.1)

Basically, it will write a file by copying chunk by chunk of 4 KB to the destination.

I have added some timeout in the code to make the write slower and to emphasize the problem.

Reader

Now, we expect that the reader must be able to know whether the file is being written by the code above.

Things that don't work

  1. fs.access(path, fs.constants.R_OK & fs.constants.W_OK & fs.constants.X_OK, callback)
  2. fs.open(path, 'r+', callback)
  3. fs.open(path, 'a+', callback)
  4. lockfile.lock(path, {}, callback) by using npm install lockfile here. It just cannot lock whether how long.
  5. lockfile.lock(path, callback) by using npm install proper-lockfile here. It always can lock.

Things that do work

  1. Check the file size difference:
    const aSize = await fileSize(path) await delay(100) const bSize = await fileSize(path) return aSize !== bSize Note: It is not reliable though because you cannot guarantee that the delay you put is enough for any writer.

  2. Check whether the file can be moved:
    const tmp = path + '-tmp' try { await rename(path, tmp) await rename(tmp, path) return false } catch (err) { return true }

Conclusion

To what I have to be working, try-moving-the-file wins because it

อ่านต่อ »

Async Await with Timeout

It's not trivial to put a timeout on an async process.

But, here is how you do it.

Let's say we have some delay function defined like this one:

async function delay(ms) {  
    return new Promise((res, rej) => {
        setTimeout(() => {
            res()
        }, ms)
    })
}

It's quite a handy function, considering using it like await delay(1000)

And extend this function to get a timeout function like this:

async function timeout(ms) {  
    await delay(ms)
    throw new Error('timeout')
}

Suppose you have some long running function around which you want to have a timeout:

async function longRunning() {  
    ...
}

Obviously, you will call it this way await longRunning().

Now, you can put a timeout around it using Promise.race, which is a handy function to get only the one that resolved earliest, this way:

const res = await Promise.race([  
    longRunning(),
    timeout(1000)
])

You now have it ! In a normal case, longRunning() is expected to be resolved earlier than timeout(1000) and the const res is the return value of that longRunning() function, but when the timeout(1000) is resolved earlier, it throws causing await Proimse.race to throw as well, no return value recorded.

อ่านต่อ »