spielwiese. (Posts about nginx.)https://spielwiese.fontein.de/tag/nginx.atom2024-01-05T07:10:21ZfelixNikolasimple file uploading in nginxhttps://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/2017-04-23T22:03:45+02:002017-04-23T22:03:45+02:00felix<p>assume you want to allow users (or programs) to upload files/data/... to your website without having to write a script/cgi/... which handles the uploading. something very simple, which just stores the files somewhere so you can analyze them later. this is for example very useful to create a <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/report-uri">reporting endpoint</a> for a <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP">content security policy</a> without using specialized software.</p>
<p>if you use <a class="reference external" href="https://en.wikipedia.org/wiki/Nginx">nginx</a> as your webserver, there's a simple solution for this. the idea is to use the <a class="reference external" href="http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_in_file_only"><code class="docutils literal">client_body_in_file_only</code> directive</a>, which allows you to dump uploads to disk to pass the filename to a reverse proxy, instead of asking nginx to cache the upload and pass it on to the reverse proxy, so that it looks like a regular <code class="docutils literal">post</code>/<code class="docutils literal">put</code> to the reverse proxy.</p>
<p>unfortunately, this doesn't work if you use a <code class="docutils literal">return xxx;</code> instead of <code class="docutils literal">proxy_pass yyy;</code>, which would have been my preferred solution. but there's a little trick: you can ask nginx to also listen on another port, say 4000, and simply return a fixed message there. then, for the main listener (on ports 80/443), you use <a class="reference external" href="http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_in_file_only"><code class="docutils literal">client_body_in_file_only</code> directive</a> combined with <code class="docutils literal">proxy_pass <span class="pre">http://127.0.0.1:4000</span></code>. this looks as follows:</p>
<div class="code"><pre class="code nginx"><a id="rest_code_9be97a2b73a146fd9157b346852fa482-1" name="rest_code_9be97a2b73a146fd9157b346852fa482-1" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-1"></a><span class="k">server</span><span class="w"> </span><span class="p">{</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-2" name="rest_code_9be97a2b73a146fd9157b346852fa482-2" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-2"></a><span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="n">127.0.0.1</span><span class="p">:</span><span class="mi">4000</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-3" name="rest_code_9be97a2b73a146fd9157b346852fa482-3" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-3"></a>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-4" name="rest_code_9be97a2b73a146fd9157b346852fa482-4" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-4"></a><span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="s">/</span><span class="w"> </span><span class="p">{</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-5" name="rest_code_9be97a2b73a146fd9157b346852fa482-5" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-5"></a><span class="w"> </span><span class="kn">return</span><span class="w"> </span><span class="mi">200</span><span class="w"> </span><span class="s">"Thank</span><span class="w"> </span><span class="s">you</span><span class="w"> </span><span class="s">for</span><span class="w"> </span><span class="s">your</span><span class="w"> </span><span class="s">report.\n"</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-6" name="rest_code_9be97a2b73a146fd9157b346852fa482-6" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-6"></a><span class="w"> </span><span class="p">}</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-7" name="rest_code_9be97a2b73a146fd9157b346852fa482-7" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-7"></a><span class="p">}</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-8" name="rest_code_9be97a2b73a146fd9157b346852fa482-8" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-8"></a>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-9" name="rest_code_9be97a2b73a146fd9157b346852fa482-9" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-9"></a><span class="k">limit_req_zone</span><span class="w"> </span><span class="nv">$binary_remote_addr</span><span class="w"> </span><span class="s">zone=peripzone:10m</span><span class="w"> </span><span class="s">rate=5r/m</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-10" name="rest_code_9be97a2b73a146fd9157b346852fa482-10" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-10"></a>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-11" name="rest_code_9be97a2b73a146fd9157b346852fa482-11" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-11"></a><span class="k">server</span><span class="w"> </span><span class="p">{</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-12" name="rest_code_9be97a2b73a146fd9157b346852fa482-12" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-12"></a><span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="s">*:80</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-13" name="rest_code_9be97a2b73a146fd9157b346852fa482-13" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-13"></a>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-14" name="rest_code_9be97a2b73a146fd9157b346852fa482-14" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-14"></a><span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">/csp-reporting</span><span class="w"> </span><span class="p">{</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-15" name="rest_code_9be97a2b73a146fd9157b346852fa482-15" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-15"></a><span class="w"> </span><span class="c1"># We just allow POST actions. Add PUT if you want to</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-16" name="rest_code_9be97a2b73a146fd9157b346852fa482-16" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-16"></a><span class="w"> </span><span class="c1"># support PUT as well. (Not needed for CSP reports.)</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-17" name="rest_code_9be97a2b73a146fd9157b346852fa482-17" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-17"></a><span class="w"> </span><span class="kn">limit_except</span><span class="w"> </span><span class="s">POST</span><span class="w"> </span><span class="p">{</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-18" name="rest_code_9be97a2b73a146fd9157b346852fa482-18" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-18"></a><span class="w"> </span><span class="kn">deny</span><span class="w"> </span><span class="s">all</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-19" name="rest_code_9be97a2b73a146fd9157b346852fa482-19" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-19"></a><span class="w"> </span><span class="p">}</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-20" name="rest_code_9be97a2b73a146fd9157b346852fa482-20" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-20"></a>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-21" name="rest_code_9be97a2b73a146fd9157b346852fa482-21" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-21"></a><span class="w"> </span><span class="c1"># Where to store the files on disk</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-22" name="rest_code_9be97a2b73a146fd9157b346852fa482-22" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-22"></a><span class="w"> </span><span class="kn">client_body_temp_path</span><span class="w"> </span><span class="s">/var/www/reports/csp/</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-23" name="rest_code_9be97a2b73a146fd9157b346852fa482-23" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-23"></a><span class="w"> </span><span class="c1"># Store the file on disk, and don't delete it, no matter</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-24" name="rest_code_9be97a2b73a146fd9157b346852fa482-24" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-24"></a><span class="w"> </span><span class="c1"># what the proxy returns.</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-25" name="rest_code_9be97a2b73a146fd9157b346852fa482-25" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-25"></a><span class="w"> </span><span class="kn">client_body_in_file_only</span><span class="w"> </span><span class="no">on</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-26" name="rest_code_9be97a2b73a146fd9157b346852fa482-26" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-26"></a><span class="w"> </span><span class="c1"># Store at most 64k on disk. That should be sufficient</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-27" name="rest_code_9be97a2b73a146fd9157b346852fa482-27" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-27"></a><span class="w"> </span><span class="c1"># for CSP reports.</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-28" name="rest_code_9be97a2b73a146fd9157b346852fa482-28" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-28"></a><span class="w"> </span><span class="kn">client_body_buffer_size</span><span class="w"> </span><span class="s">64K</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-29" name="rest_code_9be97a2b73a146fd9157b346852fa482-29" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-29"></a><span class="w"> </span><span class="kn">client_max_body_size</span><span class="w"> </span><span class="s">64K</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-30" name="rest_code_9be97a2b73a146fd9157b346852fa482-30" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-30"></a><span class="w"> </span><span class="c1"># Give the client 10 seconds to upload.</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-31" name="rest_code_9be97a2b73a146fd9157b346852fa482-31" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-31"></a><span class="w"> </span><span class="kn">client_body_timeout</span><span class="w"> </span><span class="s">10s</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-32" name="rest_code_9be97a2b73a146fd9157b346852fa482-32" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-32"></a>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-33" name="rest_code_9be97a2b73a146fd9157b346852fa482-33" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-33"></a><span class="w"> </span><span class="c1"># Do rate limiting</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-34" name="rest_code_9be97a2b73a146fd9157b346852fa482-34" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-34"></a><span class="w"> </span><span class="kn">limit_req</span><span class="w"> </span><span class="s">zone=peripzone</span><span class="w"> </span><span class="s">burst=20</span><span class="w"> </span><span class="s">nodelay</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-35" name="rest_code_9be97a2b73a146fd9157b346852fa482-35" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-35"></a>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-36" name="rest_code_9be97a2b73a146fd9157b346852fa482-36" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-36"></a><span class="w"> </span><span class="c1"># Now proxy to the small internal server we started</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-37" name="rest_code_9be97a2b73a146fd9157b346852fa482-37" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-37"></a><span class="w"> </span><span class="c1"># above, and don't pass the uploaded file to it.</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-38" name="rest_code_9be97a2b73a146fd9157b346852fa482-38" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-38"></a><span class="w"> </span><span class="kn">proxy_set_body</span><span class="w"> </span><span class="no">off</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-39" name="rest_code_9be97a2b73a146fd9157b346852fa482-39" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-39"></a><span class="w"> </span><span class="kn">proxy_pass</span><span class="w"> </span><span class="s">http://127.0.0.1:4000/</span><span class="p">;</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-40" name="rest_code_9be97a2b73a146fd9157b346852fa482-40" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-40"></a><span class="w"> </span><span class="p">}</span>
<a id="rest_code_9be97a2b73a146fd9157b346852fa482-41" name="rest_code_9be97a2b73a146fd9157b346852fa482-41" href="https://spielwiese.fontein.de/2017/04/23/simple-file-uploading-in-nginx/#rest_code_9be97a2b73a146fd9157b346852fa482-41"></a><span class="p">}</span>
</pre></div>
<p>note that i added rate limiting (see the <a class="reference external" href="http://nginx.org/en/docs/http/ngx_http_limit_req_module.html"><code class="docutils literal">ngx_http_limit_req_module</code> module</a>), allowing on average five uploads per second for remote ips, with bursts up to 10 uploads. see the <code class="docutils literal">ngx_http_limit_req_module</code> module's <a class="reference external" href="http://nginx.org/en/docs/http/ngx_http_limit_req_module.html">documentation</a> for more information on adding more rate limiting. you can for example also add limits per server, instead of just one per remote ip.</p>