Specify image in Dockerfile based on building client platform/architecture

I have a somewhat specific, Swift-based problem.
It seems that none of the official Swift images on Dockerhub support the M1, and instead the only seemingly reliable arm64 Swift images are found at swiftarm/swift. This means that manifest querying will always fail for an M1 for something like FROM swift:5.5.2. Additionally, trying to get around this with --platform=x86_64 the particular project I'm trying to run fails to compile due to an error with the Swift compiler in QEMU.
Is there a way to do something like

IF platform == arm64 FROM swiftarm/swiftELSE FROM swift:5.5.2

or should I just build and push my own multiplatform image