Data sur les stack techniques pendant la COVID-19

J’avais publié un ancien article https://blog.zenika.com/2021/07/19/panorama-des-stacks-techniques-de-la-data-de-la-cybersecurite-et-de-lagilite-a-lere-de-la-covid-19 . Cependant il manque les illustrations.


Je remercie :
Gérard Chan pour son aide en python. Les heatmaps sont grâce à son génie.

Norbert Nadir , responsable Qualité & Pédagogie de Zenika Training. Cet article a répondu à la veille des compétences (Qualiopi critère 6 indicateur 24) .

– toutes les personnes qui ont répondu à ce sondage. Ou qui ont mis en place ce sondage.

Voici la publication originale au format pdf .

How to answer any Impromptu speeches

My Toastmasters club recently had a session focused only on Table Topic

It’s often called an ‘elevator pitch’. The simple idea is that you get into an elevator and meet your ideal client. At Apple employees dread to go into an elevator with Steve Jobs. He’d say : “I’m Steve. What are you working on?” . If he didn’t like nor understand the answer , he’d fire the person.

At Toastmasters, the table topics/ Impromptu speeches are random questions thrown at the participant. Then you get 2min max for a mini speech. Example of questions:
– If you want peace, prepare for war. Do you agree or not ?
– How do you deal with conflicts ?
– Does music bring peace ?

You have a short time to pitch your idea to a captive audience.

Sounds easy in theory, but in practice, your pitch is subject to various factors.

Buy yourself time

Instead of jumping straight, try rephrase the question and get additional time. It is better than a silence.

Have as many prepared stories

Darren Tay’s book “Express to Impress” has a chapter dedicated to Table Topics. His advice is have prepared as many stories as possible. Then pick the one closest to the impromptu question.

Use a structure

To have more impact , think of a format to deliver the message. By adopting a structure, your message carries more data in less time.

Some prefer
– sandwich ( intro, meaty development, conclusion )
– STAR (situation, task, action, result )
DIGS
– PREP (Point, Reason, Example, Point)
– Pros vs Cons
I personally prefer the full circle where the conclusion is a callback to the introduction. Re-emphasize the initial question. Or try to end with an open ended question .

How much will it cost to store Google photos on AWS

Google Photos will end its free unlimited storage on June 1st, 2021. The difference with premium Google Photos is that the photos are stored RAW (no compression and be 16 times larger)

I was wondering how much did it really cost for Google to store photos per user.
Since I am more familiar with AWS than Google Cloud. Let’s evaluate the minimal cost to store photos on AWS.

AWS S3 (simple storage service) is the defacto choice to store the photos and is object storage service. In addition, S3 is the most cost effective solution (cheaper than EBS elastic block storage, EFS Elastic File System). Our assumptions are:

  • losing photos is not acceptable (Redundancy is a must)
  • only a few photos will be accessed (infrequent access)
    We can therefore select S3 Standard – Infrequent Access

In this post, we only look at the storage cost to build a minimal viable alternative to Google Photos. Regarding S3 pricing, there are three other factors to account (which won’t be covered here):

– request and data retrieval pricing
– data transfer and transfer acceleration pricing
– data management features pricing

Google Photos Cost

Price (EUR / monthly)Capacity up to (GB)Cost to store
015?
1.99100?
2.99200?
9.992000?
Google One pricing

To find the cost of storage per gigabyte (GB), let’s have a look at AWS calculator.

PS: Later, I’m going to mix USD and EUR pricing.
As I’m in Europe, Google Photos show the pricing only in EUR. In whatever regions, AWS pricing is always in USD. Sorry for the inconvenience.

Estimating the cost per GB

I used the https://calculator.aws/#/createCalculator to estimate the costs. The more you consume, the better cost deal you’ll get.

  1. Press the button “add service”.
  2. Enter in the search “S3”, “Amazon Simple Storage Service (S3)” will appear.
  3. At its bottom right corner, click on the “Configure” button.
  4. Don’t forget to select the region before entering any calculations.

I inputed the maximum storage capacity (99999999 TB) for S3 – Infrequent Access. Here are the following costs in the different Europe regions:

  • 0.0125 USD in Ireland, Stockholm
  • 0.0131 USD in London, Paris, Milan
  • 0.0135 USD in Frankfurt

Let’s take 0.0125 USD/GB as the minimum and 0.0135 USD/GB as the maximum

There are S3  Intelligent-Tiering storage class cheaper. The longer backups and archives are, the cheaper the storage is.
The pricing for Glacier is at 0.004 USD/GB and Glacier Deep Archive is at 0.00099 USD/GB.

Estimated cost of storage

The max storage column was calculate by multiplying the capacity by maximum price per GB. Example: for the capacity of 200 GB, 200 GB * 0.0135 USD/GB = 2.7 USD

The min storage column is a bit trickier. Once you exceed the capacity, the subscription goes to next tiers of pricing ( next row). The formula is (previous_capacity + 1) * min_price. Example: for the capacity of 200 GB, (100+1) GB * 0.0125 USD/GB = 1.2625 USD

Price (EUR / monthly)Capacity up to (GB)Min storage cost (USD)Max cost for S3 Infrequent (USD)Max cost Glacier (USD)Max cost for Glacier Deep Archive (USD)
0150.01250.20250.060.01485
1.991000.21.350.40.099
2.992001.26252.70.80.198
9.9920002.51252781.98

There must be an anomaly, the max storage estimated cost is 27 USD for the 2000GB. While Google charges only 9.99 EUR.

To make more cost effective solution is that of object lifecycle management whereby the photos could be transitioned to a cheaper storage class (like S3 Glacier or Glacier Deep Archive ) after a period of time depending the lifecycle configuration .

Obviously at very high volume, AWS pricing scheme is much lower than its calculator showed us.

Cheaper storage but slower access

The photos could be transitioned to a cheaper storage class (like S3 Glacier) after a period of time depending the lifecycle configuration you put in place.

However the cost of data retrieval is more expensive, the longer archives are.

Conclusion

This exercise was only about storage cost.

To make the service complete, we must account for the outbound data transfer, cost to run the software services on top.

The majority of premium customers use less than half a storage and retrieve a few percent of their photos.
For these type of service, the worst case scenario: reach the maximum storage capacity and retrieve all data.

Last, how could Google afford to make storage free under 15GB ?
In one year, the max storage will cost 0.2025*12 = 2.43 USD.
2.43 USD, a meagre sum compared to the 182 USD revenue per user that Google makes (https://arkenea.com/blog/big-tech-companies-user-worth )

Special thanks

I consulted with AWS business support. Special thanks in particular to Nicholas, the support engineer, who re-verified my calculations and brought new light with references.

Data privacy compliance

In 2018, before GDPR arrived, I worked 2 months full time to who is the data controller vs data processor, the reasons why this data is collected. You read more in plain English.

Two years later, there is no easy proper solution for a simple blog. Most serious budget friendly solutions print their terms & conditions on a .pdf file which is exposed open in the public.

This website only uses Google Analytics and users can post comments.

I got the terms from getterms.io

File Sharing infringements in Germany

In 2014 , I downloaded plenty of films on BitTorrent. I received several copyright warning letters from law firms asking compensation up to 1000€ each and a declaration to cease and desist.

Warning letters concern a formal demand to stop doing a particular activity. The idea behind such warning letters is to settle disputes directly and cheaply, without resorting to taking claims to court. The copyright infringement warning letter is an out-of-court settlement offer.

Worse, all the warning letters must be replied within 7 days. I contacted some lawyers specialised in internet copyright.  I learned a lot about file sharing infringements. If you don’t have much time to read through this, go directly to the last paragraph Summary

What is illegal ?

The music and film industry pursue file sharers. As a general rule of thumb, download is allowed. You are not allowed to upload copyright material publicly.

If the exchange of music to friends via e-mail , chat or cloud storage ( even though this is an upload) takes place within the private sphere, it is then legal.

Streaming is legal as long you only download. How can you that you’re not uploading while watching ? never install any software to watch anything online. Ex : some people thought popcorn-time was uploading on their behalf.

How do law firms track file sharers ?

By default, when you use a peer to peer such as torrent, your IP address is visible. Look below at the screen capture . The IP addresses are displayed and their respective countries.

utorrent peers list . IP addresses are listed

utorrent peers list . IP addresses are listed

Copyright holders ( Hollywood , music labels … ) purse the file sharers of their content. To expand territorial scope  ( e.g. Hollywood is US based) , copyright holders use law firms based in Germany. These law firms target only IP addresses only in Germany and the time ( important for dynamic IP address ).

The name and address of the person accessing the internet cannot be directly obtained from this information.

The internet service provider ( ISP ) is not permitted to simply pass on personal details; but they must do so if presented with a civil court ruling. ( Copyright Act , Urheberrechtsgesetz,  § 101 UrhG )

The ISP is obliged by court order to send the information directly to the copyright holder. In doing so, internet providers may demand payment for providing copyright holders  the information concerning IP addresses. (According to paragraph 113, sub paragraphs 2, sentences 2 and 3 of the Telecommunications Law)

Once the name and addresses are obtained, warning letters are sent.  The requests from rights holders comprise thousands IP-addresses at a time.  The whole process is highly automated.

However the IP Address is not sufficient to prove its owner was the one who committed the crime. It could the children or a neighbor using the wi-fi.

The defense (you or your lawyer) focus on proving you are not the perpetrator but someone else did it.

It could be someone else who was using your internet access via Wi-Fi. That’s why all wifi must be secured.

Unsecure Wi-Fi

The German Federal Supreme Court decided that every person who owns an internet connection is under an obligation to ensure that it is secured.

If the wireless connection was not secured, but the owner can prove that he/she didn’t commit the copyright infringement (e.g. because they were not at home at the time the download took place), then the owner will not necessarily be liable as the perpetrator, but will nevertheless be strictly liable for causing a nuisance.

Cease and Desist letter

Not only does the warning letter ask for a compensation ( the price always inflated ) , but it asks you to sign and return a cease and desist letter (Abmahnung) within the next 7 days.

The initial cease and desist letter is crafted to take advantage of you because :

  • You acknowledge to pay compensation and the opposition side’s legal cost
  • You were the one who committed the copyright infringement.
  • You would be bound by the declaration for 30 years and future changes in the law by Parliament or through the courts would have no effect.

Instead you should submit a modified declaration to cease and desist, which satisfies the other side’s claim. You acknowledge to pay a lower compensation, you don’t know who did it and you will never be involved in copyright infringements.

There are plenty of free “cease and desist” samples online. But the modified declaration can be rejected by the opposition side if the formulation terms are incorrect.

A specialist lawyer in file-sharing case is a great help for the average joe.

How to choose a lawyer

I contacted some lawyers specialised in internet copyright. Each case costs a nominal fee. Most of them have a flat rate price no matter the number of warning letters received. Meaning all the copyright infringements committed before the last warning letter was sent will be ignored but the ones after are not included in the flat rate contract.

But the most important criterion for me was the location of the lawyer. If the case is escalated to Court, it’s better to have your lawyer in the same city.

If you choose one, you will have to mandate the law firm and give the power of attorney (  Vollmacht in German ).

Executive Summary

Now you received a warning letter , it doesn’t matter how you were caught. You read you have to pay a fine.

Remember a warning letter is an out-of-court settlement offer. The copyright holders didn’t bring the case to court. A letter is cheap, the court is expensive. 

After receiving a copyright warning letter, there are several options :

  • Be the stupid guy ( be Ned Stark ) . Sign the cease and desist letter and accept their unfavourable conditions.
  • Be the nice  guy . Return a modified cease and desist letter favorable for you ( lower compensation and you were not the perpetrator) . But never sign the original one.
  • Be bold. Reject the warning letter and reject the cease and desist letter. State no compensation will be paid. The copyright holders will escalate with another out-of-court letter. Maybe the case will reach the court only if the opposition thinks the cost of the Court is worth.
  • Be ignorant. Ignore the warning letter as if you never saw it. There is little chance that the copyright holders will ignore your ignorance. The consequences will be worsened.

A lawyer is not required but can be very helpful for your defence.

Copyright holders ( via their law firms ) bring customers to defence law firms.  The compensation requested by the warning letters is always inflated. Copyright holders have minimum costs to recover because Internet providers can demand payment for providing the names behind IP addresses.

Warning letters based on alleged copyright infringements have become big business for the German content-industry, anti-piracy firms and their affiliated lawyers. And the people who are benefitting the most are the defence lawyers whose demand is only increased by customers who receive warning letters.

Because the whole process of  1 tracking IP addresses/ 2 requesting the name holder of those IP addresses / 3 sending warning letters is highly automated, it has created a nuisance for hundreds of thousands if not millions of honest internet users in Germany. The current implementation of warning letters violates EU law

A major difference with France’s HADOPI , the French government sends some waning letters ( rarely with fine ) while in Germany, private law firms carry on the warning letters.

With growing losses due to piracy , the corporations are suing the content consumers instead of re-inventing the content distribution.

To avoid future warning letters, I suggest to read anonymity on P2P network


 

The following lawyers frequently send out copyright warnings:

Auffenberg, Petzhold, Witte; Baumgarten & Brandt; Bindhart , Fiedler, Zerbe; CSR; Daniel Sebastian; Denecke Haxthausen & Partner; Fareds; Johannes Rübenach; Kornmeier & Patner; Lihl; Lutz Schroeder; Marcus Meier; Marko Schiek; Negele; Nümann und Lang; Paulus; Philipp Marquort; Rainer Munderloh; Rasch; Reichelt, Klute, Aßmann; Sasse und Patner; Schalast & Patner; Scheuermann,Westerhoff, Strittmatter; Schutt, Waetke; SKW Schwarz; U+C; Vahrenwald & Kretschmer; Waldorf Frommer; WeSaveYourCopyrights; Winterstein; Zimmermann & Decker.

Why TOEFL is a money making machine and ETS doesn’t respect You

To start the new year, I will add one more Rant about the TOEFL iBT ( in addition to this rant from an Oxford scholar)  . One week before the test, I took some practice with Notefull videos . Joseph Miranda does a great job at explaining how to score well on the TOEFL iBT. The latter is not exactly measuring English skills , you have to know different kinds of tricks which are not even nearly related to English skills. My weakness is the Speaking part . My English speaking is fluent but I find it unnatural to say everything in 45 sec or 60 sec ( no more , no less). The voice is recorded and later scored by a certified ETS  rater. It is a similar to a rapper contest without the rhythm. Even in my native language ( French ), I cannot do it. I passed the test on December 5 at GLS Sprachenzentrum Berlin. Despite the training from Notefull, I knew I fucked up the speaking part.

@ets.org agents reply like robots

I received my score, 10 days later , I thought I performed better. Only 22 on writing.

I knew I could score better at reading and much more at writing ( I used the right paragraph structure according to Notefull. Wrote 350+ words for the first essay and 500+ for the 2nd => the more the higher score). Surprised , I requested  toeflsupport4germany@ets.org to see my corrected exam. Here is the template answer I received :

 

Thank you for contacting the TOEFL Program.

Dear Mr.,

Thank you for your email.

Please be informed that we do offer a score review service – after receiving your scores, you can request a review of the Writing section and/or the Speaking section of your TOEFL iBT® test. You have 1 month after your test date to request a review. IMPORTANT NOTE – the score review service is not available if you have already requested that your scores be sent to any institutions or agencies.

Keep in mind that you can only request only 1 score review per test administration, and there is a cost of US$80 for each section. Both sections together cost US$160.

If the review results in a change in your scores (either higher or lower), your new scores will be posted to yourTOEFL online account, and you will receive a revised test taker score report if you requested a paper copy before you took the test. The revised scores will become your official scores.

Download the Score Review Request Form at www.ets.org/toefl/ibt/scores/scoring.

Results of the score review will be posted online within 3 weeks of the receipt of your request and payment

If you have any further questions, please don’t hesitate to contact us again.

Kind regards,

Stefania
Agent ID: xxx
Case ID: xxxx

 

US$80 to request one score review ?? It is no secret that paying for an extra review will grant a higher score. Now I never asked to review, I simply requested to see my corrected test.

The  TOEFL test taker are not allowed to display parts of the exam in any public environments. So I understand their refusal to show only the reading and listening parts because they are multiple choice questions. But the answers can be so different on the writing and speaking sections. So why hide it ?

ETS agents only reply with template answer. I don’t know if these agents can properly read emails. After 5 exchanges , I am redirected to toefl@ets.org  to request a correction of my exam. Judging by the template answers the ETS agent reply , I seriously doubt they can get a B1 on the Reading.

 

Thank you for contacting the TOEFL Program.

Dear  ,

It’s great to hear you took the TOEFL® test!

We received your request for a score review – unfortunately, your scores can’t be reviewed because you have already requested that they be sent to at least one institution or agency.

As stated on the TOEFL website at www.ets.org/toefl/ibt/scores/scoring and on the Score Review Request Form, our policy is that scores cannot be reviewed if the test taker has selected any institution or agency as a score recipient.

Furthermore, our Score Review service is not available for the Reading or Listening sections of the TOEFL iBT® test. Those sections are scored by computer and validated by the ETS® Score Quality Control process. The Reading and Listening sections are graded and validated in several stages, and the scores can’t be put through our review process.

If you decide to take the test again to try to improve your scores, be sure to check out all the free and priced test preparation products at www.ets.org/toefl/ibt/prepare.

We’re sorry we can’t be of more help!

ETS is committed to improving your customer service experience. Please take a moment and complete a short Customer Service Survey at: http://surveys1013.websurveyor.net/wsb.dll/bfried/VOTCE.htm?WSB36=TOEFL

Thank you in advance for your participation.
Maria xxxx
Case ID xxxxx

 

How qualified are TOEFL examiners ?

OK by reading above , I guess I will never see my corrected essays. It is no use to waste my time . If the monkeys behind @ets.org email cannot read and answer by their own words , I don’t blame them. What about the people who score the test : writing and speaking ?  The examiners must have at least a higher command of English . I checked the requirements for the scoring job opportunities ( note this is not specific for TOEFL )

The Online Network for Evaluation (ONE) is ETS’s proprietary software application that enables Raters to score test-taker responses and/or other submitted work via secure Internet access. Raters typically score from their homes or offices, in accordance with ETS guidelines. Online scoring sessions may occasionally be conducted at regional sites.

The process for each testing program includes:

……

To certify and score using the ETS Online Scoring system, prospective Raters must have computer equipment that meets the minimum hardware and software requirements.

So the raters are just a bunch of people working from home. There is no need to have a bachelor, no English certification required ( TEFL certification ) . The English teachers I met in China and Germany aren’t the smartest crowd, yet they have at least a bachelor. I have nothing against them, some are very smart ( one was a former investment banker). But these guys/gals recruited by ETS have no fuck*** credentials. There is no requirements for them to speak fluent English. Just have a computer and internet connection, and you can score any TOEFL listening and writing tests. I wonder where are located the ETS scorers ?  If I were the CEO of ETS , I would put the bare minimum for all raters at least 26 out of 30 on the Writing or Speaking sections.

On top of that, ETS charges US$80 to review the speaking or writing part . My guess estimation is that it takes no more that 30 minutes to score each section. Sorry ETS ( Educational Testing Service ), I think the test takers deserve more respect from you. Especially when you charge US$245 for a 4h test in Europe. I got excellent customer service for as cheap as US$10. ETS has the monopoly of the TOEFL, it is just a lobbying and money sucking machine.

So next time, I will look at IELTS . At least the IELTS speaking part is done in the presence of an academic examiner. ETS, don’t count on me to take your GRE. I prefer the GMAT.

IntelliJ bigger Font size

At any meetup, conferences, the speaker shows the code on the large. People in the back struggle to read what is on the screen.

Usually, the advice is Ctrl+plus , or Ctrl + mouse wheel up on Mac or PC. But this applies only to the current file (not a global setting).

Recently (yesterday) I had a hangout call, where I had to show code on my IDE.

The Smart Way to increase Font size

In order to keep the same font size global, go to IntelliJ IDEA > Preferences

Preferences > Editor > Font > Size . Enter the new font size

Don’t forget to press “Apply”or “OK”

After the presentation you can revert to the previous font size.

Tutorial: change Color Scheme on IntelliJ IDE

You found a nice theme packaged in a .jar file (eg. from http://color-themes.com). Now you wonder how to change your IntelliJ based IDE to the new colors.

A long time ago, I wrote how to change the Theme in IntelliJ based IDE . The word “Theme” is misleading for Scheme (read below) and Theme (Preferences > Appearance & Behavior > Appearance > Theme).

In MacOS, the Preferences bar can be accessed at the top left corner under the application name or shortcut “cmd + ,” . Please post in the comment, how to in Windows or Linux.

In order to change the Scheme, go Preferences > Editor > Color Scheme > General.

On the cog icon circled in red, select “Import Scheme” and select the .jar file.

select the color scheme .jar file

And voilà , the look & feel of Android Studio Dark Kotlin

Now if someone has an idea how to import these themes from JetBrain, I am curious.

AWS Summit Paris 2019

AWS ( Amazon Web Services ) dominates the Cloud industry with the breadth of services ( over 100+ ) and is way ahead Google Cloud and Microsoft’s Azure.

If an AWS Summit is in your town and go ahead, it is well worth it and free. The keynotes , sessions are really enriching. It’s great way to network with the local players. The AWS partners range from cloud consulting to service to tooling providers. You can expose your cloud problems to an AWS Solutions Architect.

I went to the keynotes just for Adrian Cockcroft, ex-Netflix , now VP Cloud at AWS .

Alexa

The sessions were enriching especially the “Alexa design” by Benoit Nachawati. The arrival of Dynamic Entities helps to personalize Alexa Skill experiences by adapting the interaction model at runtime without edits, builds, or re-certification. Basically imagine that Alexa can only accept a finite Set of words , say a drink : coffee, latte, tea … in the utterance. The user says “water” , and it is not a defined value, your Alexa Skill can remember this word for later use.

Startup Central

An area was dedicated to startups . I went to see the architecture of 8 finalist startups. Serverless was used by all of the finalists, maybe a condition from AWS to promote this technology.

Static website chenon.app

I started to experiment with static website generators. Why use one instead of sticking to a CMS such as WordPress ?

Static sites response time are faster because there is no database connection. But static websites cannot persist any data, meaning that no one can post comments.
In short, static site features are more limited than WordPress. If you only need a single page application or few pages, go static.

I created a simple website https://chenon.app with Hugo for my portfolio and freelance activities.

staticgen.com is great start which platforms to choose.