The bars were stacked casually on a desk in a back office of Dubai's bustling gold souk, and they immediately caught the eye of the inspectors. Keen to impress his guests, the manager picked one up.

After a lifetime in the trade, Osama Kaloti suggested he could tell the provenance and purity of each bar simply by its look and weight. This one was silver.

Or so it seemed. Kaloti insisted the metal in his hand was gold. To prove his point, he scraped away the outer coating, revealing the yellow treasure beneath. There were about six bars in the stack, which might have weighed about 75kg in total. If gold, at today's prices, they would be worth $3.2m. If silver, the same size bars would be valued at less than $30,000.

The inspectors, from Ernst & Young Dubai, had been hired to audit Osama's family business Kaloti Group, Dubai's largest gold refinery, a process designed to assure important international customers that the company was doing all in its power to avoid conflict gold.

Processing some 300 tonnes a year, Kaloti claims to account for close to half of all refining in Dubai's gold market. Regulators claim gold trading in the emirate reached $70bn in 2012, up 25% on the previous year.

"Dubai is emerging as a gold bullion centre to rival London, Shanghai and others," Osama's father, Munir Kaloti, who founded the business, explained to the Middle East Economic Digest in April last year. "Dubai now accounts for approximately 25% of the world's annual gold trade. The principal challenge [our] company faces is ensuring the responsible sourcing of gold … Our safeguard measures are beyond the minimum set out by trade associations and government bodies."

But back in the souk office, only the previous month, E&Y inspectors could hardly believe what Osama was telling them. According to their minutes: "He took a scanner and showed that the gold content was more than 85% and these bars are … from a Moroccan supplier. He said that it's normal to receive silver coated gold bars especially from Morocco due to the gold export limits imposed by the Moroccan customs."

Gold trading in Dubai is said to have reached $70bn in 2012. Photograph: Karim Sahib/AFP/Getty Images

It also emerged the Moroccan suppliers had brought the shipment in by hand and were paid in cash at Kaloti's office in the souk. The shipment had left North Africa incorrectly labelled silver, and was properly accepted at Dubai airport customs as gold. It sounded like a smuggling scam.

Further investigations found about four tonnes of gold, hidden beneath silver plating, may have come to Kaloti in a similar manner from several suppliers in Morocco. At today's prices, which are lower than those of 2012, four tonnes of gold is worth more than $170m.

After being melted down, the bars would eventually emerge from the refinery as newly minted bullion, of 99.5% purity or greater, destined for the international gold market and stamped with a serial number and the words "Kaloti, Dubai".

Following the E&Y inspection, Kaloti expected to be able to claim that its gold bars had been independently verified as conflict-free and responsibly sourced. But the inspectors were rapidly taking a very different view. They were shocked by refinery admissions that it accepted gold which had probably been smuggled. This was to be just one of the failings they stumbled over but it was enough, they thought, to mean the refinery had already effectively failed its inspection and would receive from E&Y the worst available score for the review period, covering the last seven months of 2012: "Breach of review protocol and zero tolerance".

Serious

This score is reserved for the most serious lapses – such as refineries withholding information from reviewers, using unethical means to influence them, knowingly taking conflict gold, or taking gold from suppliers they know to have used falsified or misrepresented paperwork.

Such a conclusion could even have resulted in the Dubai authorities stripping Kaloti of certification needed to trade easily with international customers. The damage to Dubai's reputation as a gold market would be huge.

Of the gold seemingly smuggled from Morocco, E&Y inspectors initially recorded in confidential reports that there had been "misrepresentation and falsification of documentation by the ... supplier with the knowledge and acceptance of [Kaloti]."

There were other serious shortcomings. Kaloti records showed more than 1,000 transactions with customers walking into the group's office in the souk, off the street, without having to provide paperwork and being paid cash for gold. A total of 2.4 tonnes was accept this way, with company records showing simply "call customer".

The average amount received from these walk-in customers was 2.25kg, but once gold weighing 35kg — the average weight of a 10-year-old boy in the UK — was exchanged for cash from a walk-in customer whose identity was unrecorded. Today this amount of gold would be worth about £850,000.

And these cash deals were just a small fraction of Kaloti transactions taking place outside the banking system. "EY team observed large quantities of cash in the vault," minutes of the inspectors' visit to the gold souk office recorded. "Kaloti team explained that about 40% of the transaction value in Kaloti was carried out in cash in the Gold Souk office." E&Y's eventual finding was that the refinery group's cash-for-gold deals in 2012 amounted to $5.2bn, compared to $6.6bn of transactions involving conventional bank transfers. If the office opened for business all year round that would still be equivalent to more than $14m a day being paid out to customers in cash.

Kaloti told the inspectors clients sometimes preferred cash as payment via a bank transfer involved a charge of 1,000 dirhams (£163). It later added that cash deals had "historically been the typical modus operandi in Dubai's cosmopolitan wholesale gold market".

Again the E&Y inspectors were troubled. Kaloti had declared itself signed up to international guidelines recommending all gold firms should "avoid cash purchases where possible, and ensure that all unavoidable cash purchases are supported by verifiable documentation".

Elsewhere, inspectors also found evidence of Kaloti transactions with suppliers from Sudan, a country linked to conflict gold in the past. Instead of showing heightened vigilance, Kaloti staff were happy to offer cash-in-hand in exchange for gold hand-delivered and originating from artisanal, small-scale mines (ASMs).

Inspectors found Kaloti had no record of mining licences for the shipments, ensuring any audit trail to the source was lost. ASMs are low-tech, labour-intensive mining operations, highly vulnerable to extortion by armed groups.

E&Y inspectors diligently recorded each of the failures. Less than three months into their work, the list had become so concerning that they discreetly contacted the regulator, warning the Dubai Multi Commodities Centre (DMCC) of impending embarrassment.

The response was a surprising one. One DMCC email inquired: "We would like to understand that if we change the review period to Jan-March 2013 ... would there be a material difference that will impact the rating positively?"

This was the first indication that led some at E&Y to believe the Dubai authorities were prepared to tinker with their own rules if the result spared the blushes of important players in the Dubai gold industry – an inference the DMCC says was unfounded.

One internal E&Y email revealed concern that Amjad Rihan, the partner in charge of the inspection team, was "getting some heat from the regulator who is keen to promote the Dubai gold industry on the global stage".

Unrealistic

It added: "They have unrealistic expectations of their industry's level of compliance, which have repeatedly been communicated to them, and there is an undercurrent of them looking to use EY's brand for their advantage, which of course is being managed."

In a statement to the Guardian, the regulator said: "The DMCC strongly refutes any allegation that it pressured EY, that it sought to influence or interfere with the review process or that it softened the review process to favour any member refinery."

Establishing the DMCC 12 years ago, Dubai's ruler Sheikh Mohammed, said: "The world's gold production is about 2,300-2,400 tonnes. Our main objective is to achieve half of the world production in the next few years. We have the will and the determination."

While Dubai's wider economic fortunes faltered in 2009, the DMCC thrived, benefiting from a surging gold price and heavy demand from India and China, the emerging economic engines of global growth.

In typical Dubai style, last year DMCC chairman Ahmed bin Sulayem celebrated this success by announcing the regulator had commissioned the Burj 2020, the world's tallest commercial tower, to be built for Dubai's hosting of the World Expo in six years' time.

Meanwhile, Dubai's largest refinery group Kaloti in December broke ground on what it says will be the world's largest gold refinery, a new $60m plant with capacity to refine as much as 1,400 tonnes of gold a year.

"The construction of our new refinery is part of the company's overall strategy to expand its capacity up to 2,000 tonnes," said Munir Kaloti.

In the end, the DMCC did not elect to alter the inspection period for Kaloti. Instead, between May and June last year, the regulator came up with another, less conspicuous, tweak to the rulebook.

It removed a recommendation that the overall verdict of inspectors be recorded in public documents, instead ordering that any such reference must henceforth be kept confidential. At the time DMCC maintained, as it does now, these rule changes were being introduced "to be consistent with similar global accepted guidance manuals" – a claim verified by London consultancy firm SGS in a detailed report.

Other gold regulators around the world were also fine-tuning rules at this time as they too assimilated international standards drawn up by the OECD.

Nevertheless the DMCC's alteration – which was not to be the last occasion the regulator tinkered with the rules – greatly concerned senior partners at E&Y Dubai, particularly Rihan, who led the division involved out the Kaloti audit.

Behind the scenes he and senior executives at E&Y Dubai contacted the audit group's global office in London asking for urgent advice on how best to treat the damaging findings in refinery inspection reports. "The issue of conflict minerals is both delicate and critical for our business in Dubai and, indeed, the region," said one email to E&Y's global head of climate change and sustainability Juan Costa Climent, a former Spanish finance minister. "We need to take a position urgently and we need the global firm to be fully aware of the facts and the issues … I am sorry to press you on the matter but it is both serious and critical."

Within two weeks the issue had been further escalated to Mark Otty, E&Y's London-based managing partner for Europe, Middle East, India and Africa, who assembled a team of senior figures to deal with the matter, also calling in law firm Linklaters to provide outside advice. "We are taking this issue very seriously," Otty wrote in one of a number of emails on the subject. "I have taken the lead in relation to our investigation of it."

Meanwhile, in Dubai, it became increasingly clear that Kaloti did not see the problem. The refinery group began drafting an official summary of E&Y's inspection – a document it believed would become the only publicly available report. Not only did it make no reference to the inspectors' overall conclusion, in accordance with the DMCC's new rules, but it also contained scant mention of many serious failings found by the independent review team.

This was too much for E&Y, which responded by indicating it would not be able to sign off on such a whitewash report as accurate.

After Kaloti received a quiet word from the DMCC, however, it began to work more collaboratively with E&Y towards a formula of words acceptable to both. Kaloti hired Jeff Rhodes, one of the most experienced executives in the Dubai gold industry who served on DMCC committees, to deal with the matter. By now Rihan, a 39-year-old partner at E&Y, had stepped back from the project, remaining deeply uncomfortable.

The work progressed without him, with both sides exchanging and refining drafts of what both inspectors and refinery staff believed would eventually be published as the official summary of the troublesome inspection.

Leaked draft filings, attached in these email exchanges, show the extent to which E&Y actively advised Kaloti how best to describe its shortcomings publicly.

At one stage the audit firm recommended edits which removed reference to "bars coated with silver", replacing the words with a more generic reference to "an incident in which there were certain documentary irregularities". It was a formula Kaloti was quick to adopt. At a stroke, any suggestion of smuggling was replaced with a more anodyne phrase hinting at what may just have been a clerical error.

Contacted by the Guardian, E&Y was unable to answer several detailed questions because it was under a duty of confidentiality to Kaloti, but said: "EY Dubai refutes entirely the suggestion that we did anything but highly professional work in relation to our engagement with Kaloti." It added that any failings discovered were fully reported to the DMCC. Meanwhile, the Guardian understands it is commonplace for audit firms to use their technical expertise to advise clients on the appropriate wording to be used in official reports.

On 8 September last year official filings were submitted to DMCC, their language carefully polished and signed off by EY and by three managing directors at Kaloti.

The phrase "breach of review protocol – zero tolerance" did appear in paperwork expected to be made public, but there was no indication it was also the inspectors' final conclusion.

Colleagues at E&Y told Rihan they regarded it a successful outcome. Otty urged him to look on the bright side. "The situation has changed dramatically … In particular, we have a team in Dubai who have worked very hard to get us to the point that both client and regulator will report deficiencies," he emailed. "On this basis I suggest that we … achieve all that you were focused on – and more." Joe Murphy, E&Y Dubai managing partner, also suggested Rihan should no longer have cause for concern. "Each of these clients has now issued a compliance report which satisfactorily takes into account the elements of non-compliance," he emailed. "We can now expect that DMCC will publish the compliance report … There is now no tension or concern about reporting."

Murphy may not have know it, but, in truth, tensions had not vanished. Kaloti was fuming.

It felt it had been rushed into the regulatory submission. The company had understood that the controversial shipments from Morocco were going to be rated only as "non-compliant, high risk", but were told the rating would be "breach of protocol and zero tolerance" just days before the submission deadline.

Despite being convinced E&Y was wrong, three top Kaloti executives, signed the compliance report — even though the refinery believed it to be incorrect — and cleared the audit firm to submit it to the DMCC.

Kaloti say this was done reluctantly, and only to meet a regulatory deadline.

According to the DMCC, queries over the findings in the submitted Kaloti report were first raised with it by E&Y, at the request of the refinery group.

If it was unusual for a firm like E&Y to be querying its own regulatory filings after they had been submitted, the move seemed doubly odd given that Otty and other senior figures at E&Y's global head office in London had earlier spent considerable time wrestling over how best to treat the Morocco finding with the help of Linklaters.

EY Dubai had concluded Kaloti's actions merited a rating of "breach of review protocol and zero tolerance". Now, however, they were no longer so sure.

In response the DMCC contacted E&Y giving its opinion of inspectors' assessment. That led the audit firm re reconsider its own view and ultimately to recall its 8 September submission.

A new set of filings were entered one month later, this time with controversial dealings with Moroccan suppliers downgraded to "High risk, non-compliance".

The exact reason for the reclassification is unclear, but confidential paperwork shows that it remained uncontested that Kaloti had told inspectors "gold plated with silver was exported as silver in Morocco however the same is declared as gold at Dubai customs".

Again, the regulator rejected any allegation that it pressured E&Y or in any other way acted improperly. E&Y said it "refutes entirely the suggestion that we did anything but highly professional work."

The unusual circumstances in which the regulatory filings had been retracted was not the end of the matter. Less than three weeks later, the DMCC would change the inspection reporting rules once again, this time in a way that meant Kaloti's second compliance report, with the downgraded finding, would never be made public.

It was a move that surprised some at E&Y. As the weeks had passed, without the anticipated publication, members of the inspection team became increasingly puzzled why no reports had been made public. By this time they were completing a brief follow-up inspection to verify Kaloti had put in place remedial action, and that failings earlier identified by E&Y had not reoccurred.

As part of this process they noted that the failure to publish a compliance report once again put Kaloti on track for further public criticisms. DMCC guidance required refiners to "publicly report ... to generate public confidence [in the inspection process]".

In internal memos E&Y inspectors wrote: "The refiner has not publicly reported ... This disclosure needs to be made prior to issuance of the follow-up review report." Such an omission, E&Y reviewers believed, was another "Non-compliance: High Risk" failing.

What they had not counted on, however, was the DMCC's willingness once again to bend its own inspection rules. This time, without advertising the changes on its website, the regulator quietly changed the public reporting standards, introducing the new concept of a "consolidated compliance report".

This removed any requirement for a refiner to publish a compliance report with embarrassing findings — so long as it later carried out a short follow-up review. The review and initial audit period would then be summerised in a single published report.

These discreet changes led E&Y to advise Kaloti that the refiner was no longer required to publish a separate compliance report covering the earlier inspection. The DMCC consulted with EY before introducing this change, and it said the audit firm confirmed such a move would not compromise the integrity of the ongoing review and publication process.

In confidential papers, E&Y noted that the DMCC rules had been "amended" in November 2013. In documents to be made public, however, E&Y was circumspect. It said it had tested Kaloti compliance against DMCC rules "in issuance as at the date of this report". In one internal email, a sharp-eyed E&Y inspector noted that a date and version number were missing on the regulator's new rules.

When, last December – and six months later than planned – Kaloti for the first time made public its official summary of the troublesome E&Y inspection, this eleventh-hour DMCC rule change was not mentioned. Also hard to discern in this, the only publicly available summary of E&Y's inspection, were full details of the shortcomings at Kaloti, between June and December 2012.

In the publicly released consolidated report Kaloti stressed that by the end of the period it was "fully compliant" with all DMCC responsible gold sourcing standards. There was no mention of "breach of protocol and zero tolerance" or even "non-compliance, high risk" in the report. The report did refer briefly to some past imperfections, labelling them "deviations".

Referencing dealings with gold seemingly smuggled from Morocco, it said: "There was lack of documentation in identifying risk in the supply chain for a small number of transactions although they were in accordance with Dubai Customs import regulations".

That Kaloti routinely offered cash-for-gold is stated clearly, though there is no indication that the sums involved ran into billions of dollars. "During the initial review period we had deviations ... in which [Kaloti] did not identify risks in cash settlement transactions which was the typical modus operandi in the Dubai wholesale market."

There is no suggestion any descriptions in the Kaloti report, signed off as "fair" by E&Y, were out of line with regulatory rules or industry practice. In any event, whatever these elliptically described historic lapses had been, Kaloti's consolidated report said they had been firmly rectified.

Kaloti said: "The final findings of the consolidated report were published in accordance with the requirements of the regulator, and were consistent with global best practices and industry norms of compliance reporting."

Meanwhile, after months trying to persuade senior colleagues in Dubai, and at E&Y's London office, that clear details of failures should appear in Kaloti's published report, Rihan decided to take the full findings, to the Guardian.

He had become increasingly isolated at E&Y and senior figures were growing impatient at his meddlesome interventions. "It is the firm's view ... a satisfactory position was reached with regard to the client engagements and with the DMCC," wrote Murphy.

"We respect, but do not agree with, your views as to how issues with one specific client [Kaloti] require to be presented both to the DMCC and more broadly," wrote one of E&Y's top lawyers in London, claiming Rihan's position was at odds with "internal and external experts".

When independent lawyers for Rihan wrote to E&Y formally urging for the audit firm to ensure the complete picture of inspection findings be revealed, the response was a sharp one from lawyers at Linklaters.

"The position is that EY Dubai agreed to carry out certain assurance work for clients [including Kaloti]. Pursuant to the terms of those engagements, EY Dubai was required, both as a matter of contract ... and as a matter of compliance with the regulatory regime under which it operated, to apply the DMCC guidelines as in force from time to time.

"EY Dubai has done so and delivered the required reports to its clients and to the DMCC setting out the findings made in the course of the work that it undertook.

"EY Dubai has therefore discharged its obligations ... Mr Rihan may have preferred that matters had been handled in a different manner, but that was not his decision to make."