QUINDI is a first principles beam diagnostics simulator which calculates the radiative spectrum from a relativistic electron bunch passing through a magnetic array. *QUINDI is in beta, certain things might be buggy*
- Features -
Programmed in C++
Parallelized with MPI
HDF5 output
Postprocessing with Matlab
Internal tracking or use external tracking file (TREDI)
When you run QUINDI, a 'Master' node is selected at random from the cluster, and the other nodes are designated as 'Slave' nodes.
The Master node transmits an individual particle's phasespace to a Slave node.
The Slave node is responsible for calculating everything for that particle over the entire detector grid, then it writes the data to an individual node HDF5 file in 'run/' and requests the next particle from the Master.
After all of the particles are processed, the Slave nodes terminate.
The Master node cycles through the node HDF5 files, adds up or concatinates the data, producing a single merged HDF5 file called 'mergedHDF5.h5' by default.
This file then must be postprocessed in some fashion. We've written a few Matlab scripts, but anything that can read HDF5 could be used.
QUINDI can do it's own tracking based on an input beam file, or it can use a TREDI HDF5 output file for the tracking.
The slave nodes operate in an asymmetrical fashion;
they receive one particle each, at a time, and will calculate the tracking, the fields, and the spectrum before moving on to the next particle.
QUINDI calculates the spectrum along a square detector, which is defined by it's physical size and by how much resolution you desire.
- Hardware/Software Requirements -
QUINDI requires a multiprocessor environment of at least 2 processors.
It was developed on our 24-node Opteron cluster with 24 gigs of memory running Gentoo, and it should work on any similar setup, or even with something as simple as a desktop dual-core CPU.
QUINDI was successfully run in OSX on a MacBook Pro with a Core Duo.
The following software must be installed on all nodes that QUINDI will be run on:
HDF5 - get it here - *NOTE* - QUINDI requires HDF5 v1.8 or higher
- Installation -
We want to make a fancy autoconf for QUINDI, but for now, all you have to do is 'make'. You might need to edit the Makefile if you installed MPI or HDF5 in a strange location. Everything will be compiled into a single executable called 'quindi'.
- How to Run -
The easiest way to run QUINDI is with 'mpirun', which is part of MPICH. Issue a command like:
mpirun -np 24 quindi main.in
The '-np 24' part is the number of nodes you are using. The master node needs to be included in this number. The 'quindi' part is what you want mpirun to run, and the 'main.in' part is the name of the main input file you want to use.
We also use Sun GridEngine for job queuing, but this is not required for QUINDI to work. The included script 'sge.sh' can be used if you change the paths, and this can be submitted via the 'qsub' command.
- Coordinate System -
It is important to understand the coordinate system that QUINDI uses.
The z-direction is the longitudinal direction, the one where gamma is the highest and the primary direction the beam travels.
The y-direction is the bending plane, and the x-direction is the primary direction of the magnetic field.
All of the coordinates for the magnets and detector are in the lab frame, they are not relative to the beam!
- Main Input File -
The main input file contains most of the input parameters you will want to adjust. The main input file can include some or all of the input parameters.
Lines beginning in '//' are ignored.
If a line is omitted, the default value shown in () will be used. The units are in [] where applicable:
r0(x,y,z),r1(x,y,z),r2(x,y,z): Location of the target, as 3D vectors with corresponding cartesian coordinates x, y, and z. Each of the 9 coordinates gets it's own line. [m] roundtarget: Setting this to 1 causes the target to be elliptical with axes given as n1 and n2 (default: 0) n1,n2: Number of detector points on target in two dimensions. Note: SpecGUI postprocessor requires n1=n2. singletarget: Setting this to 1 forces the target to be a single point. (default: 0) singleparticle: Setting this to 1 forces the input beam to be a single reference particle, i.e. beamfile=singleparticle.sdds (default: 0) freq_cut: maximum frequency for spectrum calculation. (default: 1e14) [Hz] low_freq_cut: minimum frequency for spectrum calculation. (default: 0) [Hz] freq_sam: Number of frequency samples for spectrum calculation. (default: 100) scaling: determines how the frequency slices are selected, whether they are spaced evenly ('linear') or if they are spaced logarithmically ('log') (default: linear) dz: Minimum distance step for calculation. When whotracks=tredi, this parameter is the interpolated stepsize. (default: 20e-6) [m] timeToDump: Time when final phase space of bunch is dumped (default: 9e-9) [s] magnetfile: Filename of magnetic lattice input file magdimension: Dimensionality of the magnetic lattice, choose 1, 2 or 3 (default: 2) beamfile: Filename for the input beam distribution beamcap: This will limit the total number of particles in the beam. Setting to 0 will process entire beam (default: 0) trackonly: setting this to 1 will only track the beam, not calculate fields or spectrum (default: 0) fullTrajDump: setting this to 1 will dump full trajectory of every particle, instead of just for one particle (default: 0) totalcharge: This is the total charge of the beam, which is evenly distributed to the particles in the beam (default: 300e-12) [C] windowstart: This defines the z-position where the field begins to be calculated. Setting to 0 will start at beginning (default: 0) [m] windowend: This defines the z-position where the field stops being calculated. Setting to 0 will go to the end (default: 0) [m] lenbefore: This is an additional drift before the magnets (default: 0) [m] lenafter: This is an additional drift after the magnets (default: 0) [m] columns: Number of columns in beamfile, either 6 or 8. (default: 6) outputdir: Path for merged output file. Always include the trailing / and directory must already exist (default: ./) scratchdir: directory where temporary files will be stored for run (default: ./run/) outputfile: filename of output file (default: mergedHDF5.h5) whotracks: Who is doing the tracking, quindi or tredi (default: quindi) trackfile: If whotracks is set to something other than quindi, this is the external file used for tracking operationmode: This determines if QUINDI operates in CER (operationmode=cer) or CTR (=ctr) mode (default: cer) decreaser: If operationmode=ctr, this controls how fast the beam stops. Value should be between 0 and 1, this value is multiplied by the momenta for each step, so lower values kill the beam faster (default: 1) higherOrderSpectrum: if this is set to 1, the 2nd order field is calculated (default: 0)
The parameter 'beamfile' describes the file that contains the input beam distribution (only used when 'whotracks=quindi'). This file should be in Elegant format, having 6 columns like this:
y-position y-momentum/γ x-position x-momentum/γ time γ
The columns should be space delimited, with one line per particle. If you have one of those bastard files with two extra columns for time and charge, you can set the 'columns' parameter to 8 to ignore them.
The parameter 'magnetfile' describes the file with the information for the magnets the beam is sent through. Each line in this file represents a single magnet. QUINDI supports 1-dimensional, 2-dimensional, and 3-dimensional magnetic lattices. The 3-D version has a fringe field which has both x and z components, as opposed to the 1-D and 2-D versions which only have fields with x components. A line in a 1D magnetic lattice file will look like this:
bend: L=0.5; Zpos=1.25; Bmax=0.8; Lfringe=0.025;
The first thing on each line should be the type of magnet, followed by a colon. Right now, the only choice is "bend". After this is four pieces of information, the units in []:
L: Length of magnet, not including the fringe [m] Zpos: Absolute position of the magnet, from the origin to the closest edge [m] Bmax: Peak field strength of magnet [T] Lfringe: Length of the magnet's fringe [m]
Each parameter should have the parameter name, an equal sign, the value, and a semicolon at the end.
For an undulator, the parameters have different meanings:
L: length of one full period of the undulator [m] Zpos: absolute position of the undulator, from the origin to the closest edge [m] Bmax: peak field strengh in bulk of undulator, initial and final half-periods will be half this value [T] Lfringe: this should always be set to 0.0 for undulators [m] NumPeriods: number of full periods of the undulator, initial and final half-periods will be added to this number [+int]
A line in a 2D magnetic lattice file looks like the following:
The 2D lattice has all the information of the 1D, plus the additional parameters:
Offset: Offset of the center of the magnet compared to the incoming beam (y-direction) [m] Height: Height of the magnet [m] Angle1: Angle of the incoming face of the magnet [deg] Angle2: Angle of the outgoing face of the magnet [deg]
Angle1 and Angle2 are the angles formed by the faces compared to the line z=const (the beam moves primarily in the z-direction). A negative angle means the face slopes towards the incident beam, while a positive angle means the face slopes away from the incident beam. This is useful if you are modeling a magnet that is trapezoidal instead of rectangular.
A line in a 3D magnetic lattice file looks like the this:
The 3D lattice has all the information of the 2D, plus the additional parameters:
xOffset: Offset of the center of the magnet in the x-direction [m] Gap: Seperation distance of the plates of the magnet [m]
In the 3D case, Lfringe is ignored and Gap is used as the length of the fringe.
- Who Tracks? -
If the parameter 'whotracks' is set to 'quindi', QUINDI behaves as normal and does it's own tracking. If set to 'tredi', QUINDI tracking will be bypassed, and the tracking data in the TREDI HDF5 output file named by the parameter 'trackfile' is used instead. Additionally, with 'whotracks=tredi', the TREDI trajectory will be smoothed out by cubic spline interpolation before being used for the proceeding calculations. In this case, the parameter 'dz' will be used for the target interpolated stepsize.
You can also set 'whotracks' to 'quindi-ext'. This way, you can use the tracking data from a previous QUINDI run for the field calculation. Just set 'trackfile' to the name of the QUINDI output file you want to use. Please be aware that this external QUINDI file should have been produced with fullTrajDump=1 in order to have tracking for more than one particle.
- QUINDI HDF5 Output File -
Each node will generate it's own HDF5 file, which are merged at the end of the run by the master node into a single HDF5 output file called 'mergedoutput.h5'. This file contains a ton of information, described here:
/info -> various metadata about the run
/info/detector -> coordinates of detector corners (on y-z plane)
/info/magnetic_lattice -> coordinates of magnets (on y-z plane)
/info/input_deck -> input deck of run
/phasespace/final/bx -> final x momentum of particle
/phasespace/final/by -> final y momentum of particle
/phasespace/final/bz -> final z momentum of particle
/phasespace/final/gamma -> gamma of particle
/phasespace/final/x -> final x position of particle
/phasespace/final/y -> final y position of particle
/phasespace/final/z -> final z position of particle
/phasespace/initial/bx -> initial x momentum of particle
/phasespace/initial/by -> initial y momentum of particle
/phasespace/initial/bz -> initial z momentum of particle
/phasespace/initial/gamma -> gamma of particle (same as final)
/phasespace/initial/x -> initial x position of particle
/phasespace/initial/y -> initial y position of particle
/phasespace/initial/z -> initial z position of particle
/target/Gridpoint#/spectralcuts -> frequency cuts for spectum
/target/Gridpoint#/spectrumX/real -> real E-field of spectrum in X
/target/Gridpoint#/spectrumX/imaginary -> imaginary E-field of spectrum in X
/target/Gridpoint#/spectrumY/real -> real E-field of spectrum in Y
/target/Gridpoint#/spectrumY/imaginary -> imaginary E-field of spectrum in Y
/target/Gridpoint#/spectrumZ/real -> real E-field of spectrum in Z
/target/Gridpoint#/spectrumZ/imaginary -> imaginary E-field of spectrum in Z
/target/Gridpoint#/xlocation -> x location of detector point
/target/Gridpoint#/ylocation -> y location of detector point
/target/Gridpoint#/zlocation -> z location of detector point
/trajectory/Particle0000000/bdx -> x acceleration of particle
/trajectory/Particle0000000/bdy -> y acceleration of particle
/trajectory/Particle0000000/bdz -> z acceleration of particle
/trajectory/Particle0000000/bx -> x velocity of particle
/trajectory/Particle0000000/by -> y velocity of particle
/trajectory/Particle0000000/bz -> z velocity of particle
/trajectory/Particle0000000/t -> timesteps of particle
/trajectory/Particle0000000/x -> x position of particle
/trajectory/Particle0000000/y -> y position of particle
/trajectory/Particle0000000/z -> z position of particle
If there are multiple particles in the beam, then they are stored serially as arrays in the various datasets. Trajectory only contains information about the very first particle the first node sees, unless fullTrajDump=1 in which case there will be an entry for each particle.
- Postprocessing -
Postprocessing on the merged HDF5 file can be done with Matlab and the scripts in the "matlab/" directory.
These scripts dissect the merged HDF5 file and create variables for the appropriate things. The scripts have to be modified for the appropriate path to 'mergedHDF5.h5'.
hdf5_ps -- parses the phasespace group
hdf5_traj -- parses the trajectory group
hdf5_info -- parses the info group (mainly for plotting detector and magnets)
geometry -- draws out the beamline as seen on the z-y plane, the corners of the magnets are represented with red dots, the trajectory of the first particle the first slave node sees is in blue, and the detector is in green
There is also a script called SpecGUI in the "matlab/specgui/" directory which is a handy way to view a ton of info about the spectrum using Matlab. In Matlab, the syntax goes like:
specgui('filename')
where filename is the location of the HDF5 merged file (in single quotes). This script only works for square targets!