In this paper, we present a joint end-to-end line segment detection algorithmusing Transformers that is post-processing and heuristics-guided intermediateprocessing (edge/junction/region detection) free. Our method, named LinEsegment TRansformers (LETR), takes advantages of having integrated tokenizedqueries, a self-attention mechanism, and an encoding-decoding strategy withinTransformers by skipping standard heuristic designs for the edge elementdetection and perceptual grouping processes. We equip Transformers with amulti-scale encoder/decoder strategy to perform fine-grained line segmentdetection under a direct endpoint distance loss. This loss term is particularlysuitable for detecting geometric structures such as line segments that are notconveniently represented by the standard bounding box representations. TheTransformers learn to gradually refine line segments through layers ofself-attention. In our experiments, we show state-of-the-art results onWireframe and YorkUrban benchmarks.