Simple Queueing of Requests: a File Lock

We have a system which over the years have scaled up a couple of times. When we first created it, we decided against doing a “real queue” system like ActiveMQ and the likes.

However now we have a lot of servers asking for stuff to do from this system, at the same time. Servers are idling, asking for something to do every 5 minutes - and since they all have synced clocks - they ask the server at exactly the same time - the system just hands the same task to all of the clients.

Instead of reimplementing it - and yeah, doing it right - we decided that a quick - and acceptably elegant - solution would be to wrap the exact point where a job is gotten from the database in a statement which needs a filelock - this means that only one process at a time - and thus one client - can get a job. This eliminated our bottleneck with only 8 lines of (reusable) code.

We are - however - doing so much queue-like functionality in this system, that we may need to do something about that in the near future.

We tried two different implementations a locking version, and a default return value version. The latter is what we are using in production right now. Which one is right for you depends on how you handle it on the remote end, and how much latency you can have per request, and between requests.

We decided to just return a default value telling the caller to try again instead of having a lot of requests waiting for a file lock to be released.

<code class="language-php"><span class="cp"><?php</span>

<span class="k">function</span> <span class="nf">run_with_lock</span><span class="p">(</span><span class="nv">$lock_file_path</span><span class="p">,</span> <span class="nv">$default_out</span><span class="p">,</span> <span class="nv">$block</span><span class="o">=</span><span class="k">false</span><span class="p">)</span> <span class="p">{</span>
    <span class="nv">$lock_block</span> <span class="o">=</span> <span class="nx">LOCK_NB</span><span class="p">;</span>
    <span class="k">if</span><span class="p">(</span><span class="nv">$block</span><span class="p">)</span> <span class="p">{</span>
        <span class="nv">$lock_block</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="nv">$fp</span> <span class="o">=</span> <span class="nb">fopen</span><span class="p">(</span> <span class="nv">$lock_file_path</span> <span class="p">,</span><span class="s2">"w"</span><span class="p">);</span> <span class="c1">// open it for WRITING ("w")</span>
    <span class="k">if</span> <span class="p">(</span><span class="nb">flock</span><span class="p">(</span><span class="nv">$fp</span><span class="p">,</span> <span class="nx">LOCK_EX</span> <span class="o">|</span> <span class="nv">$lock_block</span><span class="p">))</span> <span class="p">{</span>
        <span class="nb">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
        <span class="nb">flock</span><span class="p">(</span><span class="nv">$fp</span><span class="p">,</span> <span class="nx">LOCK_UN</span><span class="p">);</span> <span class="c1">// unlock the file</span>
        <span class="k">return</span> <span class="s1">'ok'</span><span class="p">;</span>
    <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
        <span class="k">return</span> <span class="nv">$default_out</span><span class="p">;</span>
    <span class="p">}</span>
<span class="p">}</span>

<span class="k">echo</span> <span class="nx">run_with_lock</span><span class="p">(</span><span class="s1">'/tmp/test.lck'</span><span class="p">,</span> <span class="s1">'something'</span><span class="p">,</span> <span class="k">true</span><span class="p">);</span></code>

If you call the blocking version from two separate terminals at the same time the one who gets the file lock will return ok after two seconds, and the other one will return ok after another two seconds. Having a lot of requests in this scenario will let the last one to get the lock wait for (number_of_requests * time_per_request). This was not reasonable for us.

We decided to use the other verssion. If this is run (final parameter set to false) the process that gets the lock will return ok after two seconds, the other one will return the default output value (“something” in this instance) as soon as the function is called - ensuring that requests that do not get the lock are exiting as soon as possible).