The Temple is said to have been founded by Priest Nichizo (1269-1342), one of the Nine Great Disciples of Priest Nichiro (1243-1320), who was one of the Six Great Disciples of Priest Nichiren (1222-1282), the founder of the Nichiren sect. Born in Ibaraki Prefecture, Priest Nichizo took the Buddhist vows joining the sect at the age of 7. He also had opportunities to contact Priest Nichiren directly. As a matter of fact, Priest Nichiren instructed him to spread the Nichiren Sect dogma in Kyoto at the moment of death.

The Temple was originally named as Baireiji {bye-ray-gee} belonging to the Shingon sect, but was not run well. Seeing it in devastation, Priest Nichizo gave support to the Temple and rebuilt it as a Nichiren sect temple. Most probably, he is known as the promoter of the Nichiren sect Buddhism in Kyoto, Japan's long-time metropolis, where the Tendai sect was most dominant back then. The Tendai group tried with every effort to keep the Nichiren followers from coming into Kyoto. While he was in Kyoto for missionary work, the authorities ousted him out for one reason or another. Despite those persecutions, he finally succeeded in founding the Nichiren sect temple called Myokenji in Kyoto 1321. Behind the success was his tactic to worm himself into Emperor's confidence. Through devotional services, he helped Emperor Godaigo (1288-1339), who was then in exile to a small island off the coast of Shimane Prefecture, to get back to Kyoto. Myokenji turned to the stronghold for spreading the Nichiren sect in western Japan.

During the period between 1624 to 1643, the Temple was rebuilt again under the name of Bairyuji {bye-r'yoo-gee}. Though remained the same Nichiren sect, their particular faith was at the heart of controversy. The Nichiren sect had a number of sub-sects and the Temple here was one of the most closed to the eyes of outsiders. This sub-sect was called Fuju Fuse {foo-jew foo-seh} group, meaning "Not receiving, not giving". To be more specific, it took the position that the followers of Nichiren sect should not receive offerings from those of other sects, nor should they give any offerings or religious services to them.

It was a doctrinal view advocated originally by Priest Nichiren, and had been honored for quite some time. As the years went by, however, most of the Nichiren sect followers began to compromise, and took part in religious services held by non-followers so long as the requests were made from the Shogun or the Court authorities. Nevertheless, a few followers adhered to the Fuju Fuse doctrines. When Hideyoshi Toyotomi {he-deh-yo-she toh-yo-toh-me} (1536-1598), a historic hero who unified Japan in 1585, summoned one thousand priests from eight Buddhist sects including the Nichiren in 1595 to hold grand mass requiem for his ancestors at Hokoji, a powerful Tendai sect temple in Kyoto, Priest Nichi-o {nee-chee-oh} (1565-1630) of the Nichiren sect, chief priest of Myokakuji in Kyoto and the advocate of Fuju Fuse doctrine, declined the invitation and did not join the mass. Again in 1599, Ieyasu Tokugawa {e-eh-yah-soo toh-koo-gah-wah} (1542-1616), another hero in the Japan's history and the founder of the Tokugawa Shogunate, held a similar mass, Priest Nichi-o did not accept the request this time either. As a result, he was exiled to Tsushima Island, 150 kilometers northwest of Fukuoka city, where he was jailed for 13 years. Pardoned, he came back to Myokakuji in 1612, but did not change his faith at all, turning down all requests from the authorities to join grand mass. At his death of 1630, the authorities ordered to uncover his buried body and sent it to Tsushima as a punishment. At the same time, the government placed the Fuju Fuse sub-sect under a ban. Persecution against them continued. Since then, the followers of Fuju Fuse went underground. There were two religious groups that had been proscribed during the Edo Period (1603-1868): The others was Christianity.

The Temple here would have been demolished should it have officially remained Fuju Fuse. But, the Temple changed its name again to the present-day Yaku-oji in order to camouflage, and survived. The ban on Christianity as well as Fuju Fuse was lifted in 1873 shortly after the Meiji Imperial Restoration of 1868, but all Buddhists had to undergo hardships anew since the new Imperial Government designated Shinto as the state religion, and employed the "Abolish the Buddha and destroy Sakyamuni" policy. It was only after World War II that all religious people started enjoying freedom of faith with the new Constitution enforced on May 3, 1947 expressly guaranteeing freedom of religion.

The word Yakuoji is literally a Bodhisattva called Bhaisajya-raja in Skt., one of the 15 Bodhisattvas serving Sakyamuni in the Lotus Sutra teachings.

Usually the door is open a fraction or one can peer in through the glass window after worshiping. The hall is gorgeously decorated with Buddhist fittings and kept clean. Installed right on the center of the altar is a colorful statue of Priest Nichiren surrounded with other Buddha statues. The temple's information board tells us that it has a Enbu-Dagon (Jambunada-suvarna in Skt.) statue of Nyorai (Tathagata in Skt.). Enbu-Dagon (or Dangon) is translated as gold dust panned out from the river-sand in the Indian forest covered with imaginary trees. The information board does not clarify which Nyorai it is. Literatures say the main object of the Temple is Yakushi Nyorai or Bhaisajya-guru in Skt., and therefore, this Nyorai might be gold-foiled Yakushi (Bhaisajya-guru in Skt.) statue. Others mention, however, that the statue of Enbu-Dagon Nyorai is that of Shaka.

One photo books featuring Buddha statues in Kamakura run a Kan'non (Avalokitesvara in Skt.) statue owned by the Temple, which is said to have been fashioned in the late 7th century.