The Robots Exclusion Protocol (REP) or robots.txt celebrates its 25th birthday and therefore it is to become the Internet standard. The protocol enables webmasters, i.e. website operators, to decide for themselves which content and pages and, above all, to what extent crawlers may access them.

What does robots.txt mean?

Robots.txt is a file that determines to which page web crawlers are first directed. This file is usually positioned in the root directory of a website. Web crawlers are search engine technologies that work independently and search for so-called “pings” in order to register them in their own index and thus output them in a meaningful way in search queries- just as self-thinking algorithms. Pins are the messages, i.e. contents, which a webmaster – in the best case SEO specifies – wants to output.

The aim is to fill the protocol with an official validity at the Internet Engineering Task Force (IETF) and thus implement it as the final Internet standard. This should make it easier for webmasters to deal with pings and crawlers, since a uniform language is used that permits less error liability.

robots.txt is not limited to http, but can also be used with FTP or CoAP. However, the document size is decisive for the smooth crawl: A defined maximum document size should ensure shortened connections.

Undefined or unsupported rules can no longer be used for Google in the future.

Google would like to make its own robots.txt parser available as Open Source. A test tool integrated in the open source version enables developers to directly check the embedded and assigned document.

Why does robots.txt make sense?

Make Google more user-friendly and developer-friendly and, above all, make it possible to participate in the design of the world’s most frequently used search engine: This is Google’s goal. The Robots Exclusion Protocol as standard should provide Google with the best possible crawling results and thus make the modern web more controlled.

Since social media has been on the triumphal march, so-called content marketing has played a major role within online marketing. But some companies still need to learn what content marketing is all about. We will introduce you to the most common mistakes you can make in content marketing and explain what content marketing is. First of all: Your customers don’t want advertising, but content with added value.

Mistake 1: Content marketing is not product advertising

This mistake happens above all to companies that are not settled in the world of social media yet and do not understand what users want to see and click. Nobody wants to spend their time watching commercials or reading copies, and that’s why obvious ads aren’t clicked. Instead, users expect a video on YouTube, a post on Instagram or a blog article to add value. Be it entertainment or interesting facts – the content must be in the foreground and the actual product advertising must only flow subtly into the content.

Mistake 2: Content means content

To ensure that the user is actually offered added value, it is very important that content is not created just for the content’s sake. As the name suggests, content should convey content. So, if you set up an upload plan, you should ensure that there is enough content to fill a post office every week, for example. Users notice immediately if the content is only a means to an end and will not click on the empty “content” again.

Mistake 3: Content marketing also needs a strategy

Perhaps because social media is still such a new medium, some companies believe that content marketing doesn’t have to be a serious strategy behind the posts. If you just post what crosses your way, you won’t be successful with content marketing. Just as with more classical marketing methods, it is indispensable to establish a detailed strategy for content marketing. It must be clear which target group is involved, which content makes sense on which platforms and which influencers could fit the target group.

Mistake 4: Social media is also work

Social media is often dismissed as a playground where no real work can take place. But anyone who has ever planned and executed a successful strategy for content marketing knows better. The development of the strategy, the planning and implementation of the content and the associated monitoring are of course a lot of work. A job that absolutely must be planned in, but which can be very worthwhile in times of social media hype.

Mistake 5: The potential must be exploited

As far as monitoring is concerned, we have already reached the next mistake that many companies make. It’s not enough to produce content, upload it and then not waste another thought on it. Monitoring means that the content will continue to be monitored. Which contents are particularly well received and what is not clicked? Only those who react to the taste of the users can really be successful in content marketing. Some companies waste potential by not sufficiently linking to their product. It is not primarily about advertising, but if a user is interested in the product behind the content, then they must have the opportunity with one click to inform himself further and to buy. After all, in times of social media, more effort is no longer accepted.

Mistake 6: Social media is not a legal vacuum

Anyone who posts on the Internet must abide by the same laws as in the classic media. Images and music may only be used if you have the rights to them. If you are on your first try at content marketing, you should allow enough time to familiarize yourself with the legal situation. Or one turns immediately to a marketing partner for whom content marketing is no longer a new world.

Matching the content marketing: Our colleagues from Seokratie GmbH have prepared the topic in a YouTube video.

Surely every single user of the Internet has wondered why this particular advertisement is now being displayed to him or her. Whether it’s cosmetics, food or a new car – sometimes you just don’t feel addressed by the advertising and have the feeling that something must have gone wrong. But some people like to go exactly the other way around and the advertising, which is perfectly tailored to their own interests, seems a bit scary.

Facebook wants to take the transparency of its Facebook ads a big step further, so that such questions can soon be answered with just a few clicks. Every Facebook ad has the function “Why do I see this ad”, which you can click on for more information. This is nothing new, but so far this tool has only hidden very general information about one’s membership of a demographic group. And if you are female and over 25 years old or male and over 50, you usually know that yourself. With the new tools, you can find out much more precisely why you were selected as a potential customer when targeting a certain company.

At a glance: The new tools for more transparency of Facebook ads

  1. If you click on the question “Why do I see this ad”, you will continue to receive information about which demographic group the company wants to reach. It’s all about gender and age.
  2. Sometimes the place of residence is also decisive for the Facebook ads you receive. This is also disclosed here by Facebook.
  3. In addition, you can now also see whether you have ever visited or clicked on the company’s website and therefore received its advertising.
  4. The interests that can be declared on Facebook are now also disclosed as possible reasons for advertising. If, for example, you have indicated cosmetics as an interest and therefore receive advertising for a new makeup product, you can see this in the new tool.
  5. What is particularly interesting is that you not only receive this new information from Facebook, but can also react immediately. If you want to click on and change the corresponding settings. If for example, you are annoyed about the makeup product and Facebook then draws your attention to the fact that years ago you indicated that you were interested in cosmetics, you can change this setting with just a few clicks. But you can also switch off the possibility for companies to pass on their data to Facebook. Because that you were on a certain website and Facebook now uses this information for Facebook Ads is only possible with your consent.
  6. Finally, you can now see which companies have recently uploaded lists of user data that contain their own data. Facebook also explains what happened to this data and what it was used for.
  7. Facebook: Is my data safe now?

The new tools with which Facebook makes Facebook ads more transparent are certainly a step in the right direction. But complete transparency is far from being achieved with these tools. Facebook’s main goal with these innovations is to limit damage, because recently the company’s reputation has suffered greatly and many Facebook users no longer feel secure. We are curious to see how data protection on Facebook will continue in the future and will of course keep you up to date.

For more information and a direct look at the new tools: