5
\$\begingroup\$

Edit: I have been able to achieve 560ps uncertainty in simulation by using external PLL feedback through the entire chip. Once I verify in real hardware I will post a complete solution.

I'm trying to synchronize the outputs of two physically separated Spartan-6 FPGAs. Ideally, the skew between the two device's outputs would be less than 1ns.

The outputs are differential pins controlled by an OSERDES2.

One possibility might be to distribute a reference clock to the two FPGAs. The signal would be routed so that it arrives at each FPGA at "exactly the same" time.

enter image description here

The reference clock would be passed through a PLL, with feedback through a BUFIO2_FB device. As shown in the clocking resources example:

enter image description here

Will this feedback mechanism cancel out clock insertion delay, so that the IOCLK is in phase with clock input pin?

The ISERDES2 in the same IOCLK domain would register a reference data signal which simply denotes a particular clock edge to align the output with.

Are there other device variations which will prevent the outputs from being deterministically in phase?

For example I'm concerned the propagation delay from OSERDES2 to output pin could vary greatly between devices. Based on timing datasheet:

enter image description here

Does this specification of 0.94 ns mean that the propagation can be anywhere from 0 to 0.94ns across different devices?

Is there any more sound approach to this problem?

\$\endgroup\$
6
  • \$\begingroup\$ Check out the NTP and PTP protocols. \$\endgroup\$
    – user110971
    Commented Dec 7, 2017 at 7:39
  • \$\begingroup\$ NTP?? Really, for nanosecond timings? Between FPGAs? PTP is better, but is still orders of magnitude wide of the mark, and if that was all the precision required there are far easier ways. \$\endgroup\$
    – Dan Mills
    Commented Dec 7, 2017 at 11:59
  • \$\begingroup\$ @DanMills they can be modified for better precision. I meant it more as an example of how one would approach such a problem. \$\endgroup\$
    – user110971
    Commented Dec 7, 2017 at 18:19
  • \$\begingroup\$ @user110971 thanks for the lead, it's helpful to know the techniques used to solve these types of problems, even if the details differ. \$\endgroup\$
    – Keegan Jay
    Commented Dec 7, 2017 at 18:37
  • \$\begingroup\$ @JayKeegan what I wanted to point your attention to is the feedback mechanisms. I suspect such precision is very hard to implement in an open loop systems. Precision timing systems usually do something like this: measure error -> control -> PLL/DLL. \$\endgroup\$
    – user110971
    Commented Dec 7, 2017 at 18:41

3 Answers 3

Reset to default
3
\$\begingroup\$

Is there any more sound approach to this problem?

Yes!!

Put both of the FPGA functions into a single device.

Trying to maintain determinism to with 1ns for separate devices is close to a pipe dream.

\$\endgroup\$
5
  • \$\begingroup\$ Hm, yes if only... :) \$\endgroup\$
    – Keegan Jay
    Commented Dec 7, 2017 at 4:21
  • \$\begingroup\$ If you have to have separate devices then change your system design architecture to remove the determinism requirement. \$\endgroup\$
    – Michael Karas
    Commented Dec 7, 2017 at 4:22
  • \$\begingroup\$ It's a fine-timing application, so determinism is required somewhere, but I could think of ways to shift it in the meantime. \$\endgroup\$
    – Keegan Jay
    Commented Dec 7, 2017 at 4:24
  • \$\begingroup\$ What in particular is the downfall of the described approach? \$\endgroup\$
    – Keegan Jay
    Commented Dec 7, 2017 at 4:26
  • \$\begingroup\$ Temperature, voltage and signal flight times on wires. Then throw in noise for good measure. \$\endgroup\$
    – Michael Karas
    Commented Dec 7, 2017 at 4:30
2
\$\begingroup\$

You really, really want both signals to be registered in the same FPGA, ideally in the same IO bank, and even then 1ns is pushing it for a general purpose LVDS pair.

You could of course not worry about it and trim cable lengths to cancel the skew, you are looking at ~1ns per foot so trimming lengths of coax is quite a reasonable approach in a one off instrument, possibly rather less so in production volumes....

You could try finding an ECL10K or such dual flipflop and use it to register the two outputs externally to the FPGA? Maybe someone like Pericom has something suitable?

\$\endgroup\$
3
  • \$\begingroup\$ Within the same BUFIO region, clock tree skew is actually 60ps, package skew is 95 ps. Propagation delay variability is an open question... If I can trim coax lengths (or use the tapped delays) are you saying that device variation is static and possible to calibrate out? This worries me less than variation over temperature of > 1ns. Finally, what is the advantage of the external flip flops? More precise timing specs? The Spartan 6 (-3 grade) can register data at 1 Gbps on all I/O pins. \$\endgroup\$
    – Keegan Jay
    Commented Dec 7, 2017 at 17:47
  • 1
    \$\begingroup\$ Device delay is not generally stable over temperature, and usually you have to do some automated lane alignment if running at anything like 1GHz DDR, this is easy when the protocol is designed to support it, but I get the impression we are talking timing application not data link here? Ovenised FPGAs....? The external flipflops provide a register that may have lower skew (and lower variability) then the FPGA itself, you would have to study the datasheets. \$\endgroup\$
    – Dan Mills
    Commented Dec 7, 2017 at 17:53
  • \$\begingroup\$ OK, we understand each other. Thanks for the advice! \$\endgroup\$
    – Keegan Jay
    Commented Dec 7, 2017 at 17:54
0
\$\begingroup\$

First of all: when these signals get deserialized again, is this timing relationship still relevant, or is there a rate matching FIFO with insertion/deletion symbols that will cause and correct more clock skew than 1 ns?

Other than that, your best bet is probably a calibration sequence that gets feedback from the receiver or a phase comparator on the way, and adjusts the PLL phase offset on the sender.

\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.