This is very old-school webdev, but I recently had reason to create a styled placeholder. Or specifically, a placeholder where part of the placeholder text has a different style. This doesn't seem possible to do reasonably without vendor-prefixed web APIs. Here's how you can do it.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Conforming HTML5 Template</title>
<link rel="shortcut icon" href="favicon.ico" type="image/x-icon">
<style>
#sp-container {
    position: relative;
}

#sp-placeholder {
    position: absolute;

   /* non-essential; mimic default placeholder */
    font-family: sans-serif;
    font-size: 14px;
    color: #777777;

    /* vertically centre */
    top: 50%;
    transform: translate(0%, -50%);
}
</style>
<script>
const PLACEHOLDER_ID = 'sp-placeholder';
const INPUT_ID = 'sp-input';

function onReady() {
    const span = document.getElementById(PLACEHOLDER_ID);
    const input = document.getElementById(INPUT_ID);

    const maybeShow = () => {
        if (input.value === "") {
            span.style.display = 'inline-block';
        }
    };

    const hide = () => {
        span.style.display = 'none';
    };

    // If the value was pre-filled by some query param, don't show the hint.
    const maybeHide = () => {
        if (input.value !== "") {
            hide();
        }
    };

    // To hide on focus:
    //input.addEventListener('focus', hide)
    input.addEventListener('input', hide)
    input.addEventListener('blur', maybeShow);

    maybeHide();
}

document.addEventListener('DOMContentLoaded', onReady);
</script>
</head>
<body>
<form id="sp-container">
<label>
<span id="sp-placeholder">Enter text <strong>here</strong></span>
<input type="text" id="sp-input"/>
</label>
<input type="submit"/>
</form>
</body>
</html>

Several things to note here. This uses the relative-inside-absolute trick to take the placeholder span out of the main flow. Note that if the form is submitted, and the user then presses the back button, the form values will be recalled by default on page load. In this case, you should hide the placeholder span. I'm not totally sure that this solution actually handles that properly, although it attempts to.

The vertical centring can be handled either through the top/translate trick shown above, or it can be handled through tricky margins and calc(). I didn't have much luck with flexbox in this case, although it may also be a possibility.

Posted 2020-07-23

A few arbitrary tips on Omeka-S theming.

Retain existing classes on analogous elements as far as possible, as they are sometimes referred to by internal Omeka CSS classes. Obviously you can imagine edge cases where this isn't desirable. But it's a good first rule to follow. Obviously if you omit some design-element entirely you must also omit the class associated with it.

In a similar vein, be wary of doing less than the requirements of a theme. There's no real verification or way of testing that a theme handles all paths sensibly and in a manner consistent with the expectations of the user.

For example, there are certain site settings your theme should support: "embed media on item page" is a site specific setting that you must handle within your theme, but there is no validation that your theme does in fact handle this.

Do not ever use <p></p> elements in your module markup. Use custom divs or spans with your own classes. Why? The Html blocklayout generates <p> tags, so any attempt to style page text will often inadvertently style your more-irregular p tags.

Config form names should be in snake_case.

Posted 2020-07-10

Patreon seem to find it acceptable to do a whole bunch of security-through-obscurity measures on their site that verge on DRM. This is a symptom of the times, really. They don't provide an API for users, although they do provide an API for creators. Let's try to scrape MP3s.

<a data-tag="post-file-download"
   href="https://c10.patreonusercontent.com/LONGSTRING" class="sc-fzpans kmqqXw">Myfile.mp3</a>

This is what the link for downloads looks like. We are after the URL in the href attribute, but we won't be able to get it without providing our cookie. A query string parameter token-hash and token-time are included in the URL. This means that it's actually possible to download these files without sending any cookie, as the authentication details are already encoded in the URL.

So that means that the problem reduces to the following:

  • Expanding the infinite scroll of the site
  • Find all links and correlate them with some metadata

Expand the infinite scroll

Start out at the URL that has a tags filter that matches what you want.

https://www.patreon.com/somecreator/posts?filters%5Btag%5D=Patron%20Only%20Audio

We look for the 'Load more' button and expand it.

<button class="sc-fzoiQi ivnVZu" tabindex="0" type="button">...</button>

Patreon uses some stupid minified CSS classes, so I don't know how stable this identifier is. In this case, ivnVZu is the relevant class, as you can confirm by testing the output of document.querySelectorAll in the inspector.

function expandScroll() {
    const button = document.querySelector('.ivnVZu');
    if (button) {
        console.log("click");
        button.click();
        window.setTimeout(expandScroll, 30000);
    } else {
        console.log("terminating expansion");
    }
}

The site is so ungodly slow that this can hang Chrome quite easily. I suggest either having an unrealistic amount of patience, or pausing script execution in the "Sources" tab in the inspector after a while.

Scrape the URLs

Scraping the URLs is a bit more interesting. We need two pieces of information:

<a data-tag="post-published-at" href="/posts/some-post-12345" class="sc-fzpans bwkMGo">May 16 at 9:28am</a>

There may be more information in the show title but I don't think that we can guarantee uniqueness. And the previously mentioned URL.

From some experimentation in the inspector, I can tell that the common ancestor of these elements is the following: <div data-tag="post-card">...</div>

This at least is a bit semantic.

Let's scrape:

var postNodes = document.querySelectorAll('[data-tag="post-card"]');
var data = Array.from(postNodes).map(el => {
    const publishedAt = el.querySelector('[data-tag="post-published-at"]');
    const postFileDownload = el.querySelector('[data-tag="post-file-download"]');

    return {
        publishedAt: publishedAt.textContent,
        href: postFileDownload.getAttribute('href')
    };
});

Now you can console.log(JSON.stringify(data)) and paste it into a text editor, you can save it as exported.json. Using that JSON data you can use this Python script to download the files into timestamped files:

import json
import requests
import unicodedata
import re

with open('exported.json', 'r') as f:
    data = json.load(f)

def slugify(value):
    value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode('ascii')
    value = re.sub('[^:\w\s-]', '', value).strip().lower()
    return re.sub('[-\s]+', '-', value)


for rec in data:
    timestamp = rec['publishedAt']
    href = rec['href']
    filename = "{}.mp3".format(slugify(timestamp))
    print(filename)
    r = requests.get(href, stream=True, timeout=30)

    with open(filename, 'wb') as fd:
        for chunk in r.iter_content(chunk_size=128):
            fd.write(chunk)
Posted 2020-05-17

I wanted to make an example getting started post as a reaction to an official vertical list example which I find to be overly complex.

What API surface does it provide? It provides a bunch of things:

DragDropContext, this is a top-level container element. Droppable, which must envelop all draggables. Draggable, which appropriately enough must envelop an individual draggable item.

Each item expects its children prop to be a function, which it calls to get further elements to render. That enables it to pass the context information down the component tree. In some ways this is a good example of the issue I described earlier in the post FP & the Context Problem.

Luckily this library has a good set of types available in the DefinitelyTyped repository. You can find out most of the API from reading the TypeScript definitions. So a well-typed example is below.

import React, {useState} from 'react';
import {
    DragDropContext, Droppable, Draggable,
    DropResult, DroppableProvided, DroppableStateSnapshot, DraggableRubric, 
    DraggableStateSnapshot, DraggableProvided, ResponderProvided
} from 'react-beautiful-dnd';

interface AppProps {
};

function makeDraggableChildren(value: string) {
    return (provided: DraggableProvided,
            snapshot: DraggableStateSnapshot,
            rubric: DraggableRubric) => 
                <li ref={provided.innerRef} 
                     {...provided.dragHandleProps}
                     {...provided.draggableProps}>{value}</li>;
}

function makeDroppableChildren(items: string[]) {
    return (provided: DroppableProvided, 
            snapshot: DroppableStateSnapshot) =>
                <ul ref={provided.innerRef}>
                  {items.map((value, i) => (
                      <Draggable draggableId={i.toString()}
                                 index={i}
                                 key={i}>
                        {makeDraggableChildren(value)}
                      </Draggable>
                  ))}
                  {provided.placeholder}
                </ul>;
}

export function DndDemo(props: AppProps) {
    const [items, setItems] = useState(['fry', 'bender', 'leela']);

    const onDragEnd = (result: DropResult, provided: ResponderProvided) => {
        if (!result.destination)  return;    // invalid drop

        const startIndex = result.source.index;
        const endIndex = result.destination.index;

        // Destructure because we know we always have 1 item.
        const [removed] = items.splice(startIndex, 1);
        items.splice(endIndex, 0, removed);
        setItems(items);

        console.log("start index is %o, end index is %o", startIndex, endIndex);
    };

    return (
        <div>
          <h1>Hello</h1>

          <DragDropContext onDragEnd={onDragEnd}>
            <Droppable droppableId="main">
              {makeDroppableChildren(items)}
            </Droppable>
          </DragDropContext>
        </div>
    );
}

While it's still very ugly, I feel that overall this is a relatively good deal when it comes to avoiding dealing with the HTML5 drag and drop API directly. The abstraction is extremely leaky-to-nonexistent here, as you can see. You're plugging a bunch of mandatory stuff into your HTML and while you can ignore the detailed-contents of the stuff, you can't ignore the presence of the stuff itself. Every innerRef, *props value here is the cruft from the mechanism showing through. I must give this library its due, though: the error messages from this library are extremely good, and the documentation is also decent. Once you commit to introducing this piece of large bit of ugly boilerplate, you do get a good amount for taking on this burden, so I think it's a good bet overall.

I feel like there's probably a much more idiomatic way to do the currying approach above, but I'm not experienced enough with React yet to know what it is.

Posted 2020-05-12

I don't know that there's a good name for this technique, but it's basically what's required when you need to have local database credentials that differ from other developers in a team, and also can't be committed to version control. In React and Vue this feature is called .env.local.

When you need this in Spring, you need to know the following:

  • You can have multiple profiles.
  • But you'll need to end up gitignoring a file pattern at some point anyway.
  • Your 'mainstream' config file is likely stored under your project tree at src/main/resources/application.properties.
  • The default search path for Spring is:
    • file:./config/
    • file:./
    • classpath:/config/
    • classpath:/
  • file:. means the real root of your project, i.e. where your .git directory is, pom.xml, etc.
  • classpath:/ means src/main/resources, broadly speaking.

From this we can reason that the path of least resistance is to add a gitignore rule /config/ (the leading slash is important), and store your custom configuration in config/application.properties in the project root. Of course the more sophisticated way would be to use actual profiles, but this has the benefit of not needing any IDE configuration.

Posted 2019-11-26

I know next-to-nothing about any of these, except for Typescript which I know a bit about. The goal is to produce the most minimal example possible of a JS app with state, i.e. a counter, the classic demo.

Set up the project with npx create-react-app myapp --typescript.

First off: You want both pieces, redux and react-redux. You will use imports from both of these.

To get data into and out of your components, you'll need a store. The store is constructed with the createStore function from redux.

You always need a reducer to construct a store. A reducer is a function from a state and an action to the next state.

This should be the contents of index.tsx:

interface MyState {
    counter: number;
}

const INCREMENT = 'INCREMENT';

interface IncrementAction {
    type: typeof INCREMENT
}

type MyActionTypes = IncrementAction;


function myReducer(state: MyState | undefined, action: MyActionTypes): MyState {
    console.log("Reducer being called.");
    if (state === undefined) {
        return { counter: 0 };
    }

    switch (action.type) {
        case INCREMENT:
            return Object.assign({}, state, { counter: state.counter + 1 });
        default:
            return state;
    }
}

const store = createStore(myReducer);

ReactDOM.render(
    <Provider store={store}>
        <App />
    </Provider>,
    document.getElementById('root')
);

Points to note here:

  • MyActionTypes is a sum type with only one member.
  • Each action has a mandatory property, type. The use of typeof in the definition of IncrementAction ensures that type receives a string literal type. This in turn enables the switch below to be type safe.
  • The structure of the switch will make sure that action is inferred to the correct type within its case branch. That is, even though MyActionTypes may be Foo | Bar, within its matching case branch TS knows that action is FooAction or BarAction.
  • Provider is a wrapper component that will wire up the store to all components that are beneath it in the component tree.
  • The type of the first argument must be MyState | undefined NOT simply MyState. You react to the undefined state by configuring the initial state. You're going to get confusing type errors when you try to call createStore if you type this wrongly.

This should be the contents of App.tsx. Note that you also need the type definitions from above, I recommend extracting them to a file.

function mapStateToProps(state: MyState) {
    return {
        counter: state.counter
    };
}

function incrementActionCreator(): IncrementAction {
    return {
        type: INCREMENT
    };
}

const mapDispatchToProps = {
    increment: incrementActionCreator
};

interface AppProps {
    counter: number;
    increment: () => void;
}


class App2 extends React.Component<AppProps> {
    render() {
        const { counter, increment } = this.props;

        return (
            <div className="App">
                <header className="App-header">
                    <img src={logo} className="App-logo" alt="logo" />

                    <p>Counter value: {counter}</p>

                    <button onClick={increment}>Increment</button>

                    <a
                        className="App-link"
                        href="https://reactjs.org"
                        target="_blank"
                        rel="noopener noreferrer"
                    >
                        Learn React
                </a>
                </header>
            </div>
        );
    }
}


export default connect(
    mapStateToProps, mapDispatchToProps
)(App2);

Things to note here:

  • The connect function lives in react-redux.
  • App2 needs to be converted to the class-based component syntax, it won't work with the functional component syntax, as far as I can see.
  • An action creator is a function returning an action, which in this case just means that it returns an object that has a type property.
  • The argument to mapStateToProps tells react-redux that the prop counter, accessible within App2 as this.props.counter, should reflect the value of the property counter in the store. (The fact that they have the same name is incidental.)
  • The object mapDispatchToProps just says: within App2, the function-valued prop increment will call the action creator incrementActionCreator and dispatch the resulting action... which will end up calling the reducer.
  • To get these props typed, we have to explicitly declare them by making our component a class that extends React.Component, and declare an interface AppProps. I'm not sure that the type of increment is correct, but it's marginally better than the any type, I would suppose.
  • Arguably the declaration of AppProps should be unnecessarily as it should be able to be inferred; here, React is in a rather similar position to the Vuex helper methods such as mapState. Neither provide automatic typing of mapped store props, I was rather hoping that React might have been better here -- but see this question.
  • Note that we have to destructure the mapped stuff from this.props. This is a little bit dangerous because if you used the same name for the mapped prop as the action creator (which you've probably imported into your scope from Some other module), you're shadowing that with the mapped-prop version, meaning that if you FORGET to destructure the props and just end up calling the parent-scope version then you'll get no behaviour and no errors.
Posted 2019-10-29

Just a very short note. Anyone experiencing difficulties with the above hardware. The symptom is complete hard locks in X11. They happen every hour or two, seemingly without any cause. No kernel logs, no errors, no response to anything. If music is playing it will hang and loop the same 2-second snippet. It's exactly like a kernel hang, except without any diagnostics remaining after reboot. Most people suggest hardware RAM issues, but I've run checks on both regular memory and GPU memory.

I don't have a solution, but I do have a workaround that I found in a thread on MXLinux forums. The workaround is to downgrade the 4.19.x kernel to the 4.9 kernel. Horrible, I know, especially as the 4.9 kernel has actually been removed from Buster. I think that I only had it around as the remnants of a previous dist-upgrade from Stretch. Frankly I don't know what the best procedure for dealing with this is, but I can say that this is the only thing that worked, having tried tons of X11 config hacks, nothing made any difference. I just have to hope that this issue will vanish in a future kernel release. :/

Posted 2019-10-10

I wanted to set up a way that my keys could not be intercepted by some rogue who stole my laptop. Realistically, given that I have access to many servers, someone who stole my laptop could potentially wreak untold havoc. For a long time I have avoided using authorized_keys based authentication for this reason, preferring to type my password when prompted. But now I come into contact with several servers that simply flat-out forbid password authentication. So it was time for a change.

The basic change needed is to add a passphrase to the SSH key. You can do this through

ssh-keygen -m PEM -p -f ~/.ssh/id_rsa

PEM format is required for compatibility with certain tools, cough, Paramiko, which do not support RSA format private keys.

The next step is, how do you avoid typing your passphrase at every turn? The answer is usually ssh-agent, which can hold unlocked keys in memory. On Debian, ssh-agent is started by your display manager as part of the Xsession configuration.

I didn't want to be immediately prompted for a key unlock every time I rebooted, so I found this very cool new option from newer SSH versions:

# ~/.ssh/config

# This will make sure that any new keys get added to ssh-agent using ssh-add(1)
# on a lazy basis (whenever the host is first connected to).
AddKeysToAgent yes

So using this mitigates any need for manual calls to ssh-add.

Bear in mind that using this encrypted key will break any noninteractive scripts that you need to use. This will affect rsync, git-annex, etc. For these cases I am not yet sure of the best option. The simplest approach, and the one that I used, is to create two SSH identities, one encrypted and one unencrypted, and deploy the unencrypted private key only to servers. Servers can then use pull-based access, and only if the server is already compromised does an attacker essentially own everything. Someone with only workstation access can't get server access without typing the passphrase first.

But wait, we still need to expire the keys to force the passphrase to be re-entered. Otherwise a clever attacker can take as long as they like to examine the known_hosts file and own everything. Debian deploys a file /etc/X11/Xsession.d/90x11-common_ssh-agent with the package x11-common. This file defined the startup commands that Xsession uses to launch ssh-agent. It includes this line:

SSHAGENTARGS=

Just modify this line and include an argument to specify the lifetime that you want for the identities. I chose one hour, so I wrote the line below.

SSHAGENTARGS="-t 1h"

The rules for specifying times are specified in the manual page sshd_config(5), under the section TIME FORMATS.

Posted 2019-09-11

Upgrading notes:

Posted 2019-08-05

The task is to pipe together two subprocesses, and we also want to be able to get both their exit codes while accumulating their output into memory.

Python:

import subprocess

params = {
    'args': ['gzip', '-dc', 'alice.txt.gz'],
    'stdout': subprocess.PIPE
}

with subprocess.Popen(**params) as p1:
    with subprocess.Popen(args=['tr', '[:upper:]', '[:lower:]'], stdin=p1.stdout, stdout=subprocess.PIPE) as p2:
        p1.stdout.close()
        stdout_data, stderr_data = p2.communicate()
        print("Length of data on pipe:", len(stdout_data))

        e1 = p1.wait()
        print("Return code of p1 was:", e1)

        e2 = p2.wait()
        print("Return code of p2 was:", e2)

Perl:

use IPC::Run qw(harness);
use Data::Dump qw(dump);
use v5.20.2;

my $gzip = ['gzip', '-dc', 'alice.txt.gz'];
my $tr = ['tr', '[:lower:]', '[:upper:]'];

my $buf;
my $h = harness $gzip, '|', $tr, '>', \$buf;

$h->run();
my @results = $h->results();

say "Length is ", length $buf;
say dump(\@results);

The comparison is slightly unfair because subprocess is a standard module, whereas IPC::Run is only available on CPAN.

Posted 2019-07-02
The X3 Wiki Archive
Posted 2019-06-16
Fabric 2 cheat sheet
Posted 2019-03-05
Using comboboxes in Qt5
Posted 2019-02-27
System Puppet, CentOS 7 Client
Posted 2019-02-25
X3 savegames
Posted 2019-02-02
Shadow Tween technique in Vue
Posted 2019-01-06
Width list transition in Vue
Posted 2018-12-18
Emoji Representations
Posted 2018-09-14
Thoughts on Cheesesteak & More
Posted 2018-08-29
Vue + GraphQL + PostgreSQL
Posted 2018-07-20
Neo4j Cypher query to NetworkX
Posted 2018-05-09
FP & the 'Context Problem'
Posted 2018-02-27
Cloake Vegetable Biryani
Posted 2018-02-25
FFXII Builds
Posted 2018-02-02
Custom deployments solution
Posted 2017-12-09
SCons and Google Mock
Posted 2017-11-30
Sunday Lamb Aloo
Posted 2017-11-19
centos 6 debian lxc host
Posted 2017-11-03
Srichacha Noodle Soup
Posted 2017-10-17
Kaeng Kari
Posted 2017-10-13
Ayam Bakar (Sri Owen)
Posted 2017-10-12
Pangek Ikan (Sri Owen)
Posted 2017-10-12
Chicken Tikka Balti Masala
Posted 2017-10-06
Clojure Log Configuration
Posted 2017-09-28
Clojure Idioms: strict-get
Posted 2017-09-28
About
Posted 2017-09-18
Philly Cheesesteak
Posted 2017-09-14
Welcome
Posted 2017-09-13
Srichacha Kaeng Pa
Posted 2017-08-31
Malaidar Aloo
Posted 2017-08-10
BBQ Balti Chicken
Posted 2017-07-19
Sabzi Korma
Posted 2017-07-18
Vegetable Tikka Masala
Posted 2017-07-02
Soto Ayam
Posted 2017-06-08
Bombay Aloo w/Bunjarra
Posted 2017-06-03
Chicken Dopiaza
Posted 2017-06-01
LJ Bunjarra
Posted 2017-05-31
Glasgow Lamb Shoulder Tikka
Posted 2017-05-24
Tofu Char Kway Teow
Posted 2017-05-12
King Prawn Balti
Posted 2017-04-24
Ad-hoc Quorn Rogan Josh
Posted 2017-04-15
Glasgow Vindaloo
Posted 2017-03-28
Rempeyek
Posted 2017-03-26
Toombs Saag Balti
Posted 2017-02-25
Glasgow Bombay Rogan Josh
Posted 2017-02-21
Glasgow Chicken Balti
Posted 2017-02-16
Quorn Balti & Cloake Naan
Posted 2017-02-03
Two Spice Marinades
Posted 2017-01-18

This blog is powered by coffee and ikiwiki.