May 20 13:50:39.330048 ip-10-0-140-71 systemd[1]: Starting Kubernetes Kubelet...
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.743785    2566 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750857    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750871    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750874    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750878    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750882    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750886    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750889    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:39.767936 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750892    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750894    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750897    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750900    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750902    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750905    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750907    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750910    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750913    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750915    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750918    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750920    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750923    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750925    2566 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750928    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750931    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750934    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750936    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750939    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750942    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:39.768973 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750945    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750947    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750950    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750958    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750961    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750964    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750967    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750969    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750972    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750974    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750977    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750979    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750982    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750984    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750987    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750990    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750992    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.750997    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751007    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:39.769797 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751011    2566 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751014    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751017    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751021    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751024    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751027    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751030    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751032    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751035    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751037    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751040    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751042    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751045    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751048    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751051    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751053    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751057    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751059    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751062    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751065    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:39.770306 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751068    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751070    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751073    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751076    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751078    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751089    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751093    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751096    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751098    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751101    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751108    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751110    2566 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751118    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751122    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751124    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751126    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751129    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751132    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751135    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.751137    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:39.770922 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752312    2566 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752319    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752323    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752326    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752329    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752332    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752334    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752337    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752340    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752345    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752349    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752352    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752355    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752358    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752361    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752364    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752366    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752369    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752372    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:39.771473 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752375    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752377    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752380    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752383    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752385    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752395    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752398    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752400    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752403    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752405    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752408    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752411    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752413    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752416    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752419    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752422    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752424    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752427    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752431    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752433    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:39.772053 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752437    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752439    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752442    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752445    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752447    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752450    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752453    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752455    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752457    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752460    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752463    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752465    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752468    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752470    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752473    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752475    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752478    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752481    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752489    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752493    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:39.772769 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752497    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752500    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752503    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752506    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752509    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752511    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752514    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752516    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752519    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752521    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752524    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752526    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752529    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752531    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752534    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752536    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752538    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752541    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752543    2566 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752546    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:39.773324 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752548    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752551    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752553    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752555    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752558    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752560    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.752563    2566 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.753971    2566 flags.go:64] FLAG: --address="0.0.0.0"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.753981    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.753997    2566 flags.go:64] FLAG: --anonymous-auth="true"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754001    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754014    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754017    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754022    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754027    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754030    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754033    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754037    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754040    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754043    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754046    2566 flags.go:64] FLAG: --cgroup-root=""
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754049    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754052    2566 flags.go:64] FLAG: --client-ca-file=""
May 20 13:50:39.773826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754055    2566 flags.go:64] FLAG: --cloud-config=""
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754057    2566 flags.go:64] FLAG: --cloud-provider="external"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754060    2566 flags.go:64] FLAG: --cluster-dns="[]"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754066    2566 flags.go:64] FLAG: --cluster-domain=""
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754069    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754072    2566 flags.go:64] FLAG: --config-dir=""
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754075    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754078    2566 flags.go:64] FLAG: --container-log-max-files="5"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754082    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754085    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754088    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754091    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754096    2566 flags.go:64] FLAG: --contention-profiling="false"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754099    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754102    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754105    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754107    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754112    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754116    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754119    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754122    2566 flags.go:64] FLAG: --enable-load-reader="false"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754131    2566 flags.go:64] FLAG: --enable-server="true"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754134    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754141    2566 flags.go:64] FLAG: --event-burst="100"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754144    2566 flags.go:64] FLAG: --event-qps="50"
May 20 13:50:39.774386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754147    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754150    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754153    2566 flags.go:64] FLAG: --eviction-hard=""
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754157    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754160    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754162    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754166    2566 flags.go:64] FLAG: --eviction-soft=""
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754169    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754171    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754174    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754177    2566 flags.go:64] FLAG: --experimental-mounter-path=""
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754180    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754183    2566 flags.go:64] FLAG: --fail-swap-on="true"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754186    2566 flags.go:64] FLAG: --feature-gates=""
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754190    2566 flags.go:64] FLAG: --file-check-frequency="20s"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754193    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754196    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754200    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754203    2566 flags.go:64] FLAG: --healthz-port="10248"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754207    2566 flags.go:64] FLAG: --help="false"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754210    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-140-71.ec2.internal"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754213    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754216    2566 flags.go:64] FLAG: --http-check-frequency="20s"
May 20 13:50:39.775044 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754219    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754222    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754227    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754230    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754232    2566 flags.go:64] FLAG: --image-service-endpoint=""
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754235    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754244    2566 flags.go:64] FLAG: --kube-api-burst="100"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754247    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754250    2566 flags.go:64] FLAG: --kube-api-qps="50"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754253    2566 flags.go:64] FLAG: --kube-reserved=""
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754256    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754259    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754262    2566 flags.go:64] FLAG: --kubelet-cgroups=""
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754265    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754268    2566 flags.go:64] FLAG: --lock-file=""
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754271    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754274    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754277    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754282    2566 flags.go:64] FLAG: --log-json-split-stream="false"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754285    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754288    2566 flags.go:64] FLAG: --log-text-split-stream="false"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754291    2566 flags.go:64] FLAG: --logging-format="text"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754293    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754297    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 20 13:50:39.775732 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754299    2566 flags.go:64] FLAG: --manifest-url=""
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754302    2566 flags.go:64] FLAG: --manifest-url-header=""
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754306    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754310    2566 flags.go:64] FLAG: --max-open-files="1000000"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754319    2566 flags.go:64] FLAG: --max-pods="110"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754322    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754325    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754328    2566 flags.go:64] FLAG: --memory-manager-policy="None"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754331    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754334    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754338    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754341    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754349    2566 flags.go:64] FLAG: --node-status-max-images="50"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754352    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754355    2566 flags.go:64] FLAG: --oom-score-adj="-999"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754364    2566 flags.go:64] FLAG: --pod-cidr=""
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754367    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754372    2566 flags.go:64] FLAG: --pod-manifest-path=""
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754375    2566 flags.go:64] FLAG: --pod-max-pids="-1"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754378    2566 flags.go:64] FLAG: --pods-per-core="0"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754381    2566 flags.go:64] FLAG: --port="10250"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754384    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754387    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0227e5e9e7a0d8dae"
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754390    2566 flags.go:64] FLAG: --qos-reserved=""
May 20 13:50:39.776300 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754393    2566 flags.go:64] FLAG: --read-only-port="10255"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754396    2566 flags.go:64] FLAG: --register-node="true"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754399    2566 flags.go:64] FLAG: --register-schedulable="true"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754402    2566 flags.go:64] FLAG: --register-with-taints=""
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754405    2566 flags.go:64] FLAG: --registry-burst="10"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754408    2566 flags.go:64] FLAG: --registry-qps="5"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754411    2566 flags.go:64] FLAG: --reserved-cpus=""
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754413    2566 flags.go:64] FLAG: --reserved-memory=""
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754417    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754420    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754423    2566 flags.go:64] FLAG: --rotate-certificates="false"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754426    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754428    2566 flags.go:64] FLAG: --runonce="false"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754433    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754436    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754439    2566 flags.go:64] FLAG: --seccomp-default="false"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754442    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754445    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754450    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754453    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754456    2566 flags.go:64] FLAG: --storage-driver-password="root"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754459    2566 flags.go:64] FLAG: --storage-driver-secure="false"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754462    2566 flags.go:64] FLAG: --storage-driver-table="stats"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754464    2566 flags.go:64] FLAG: --storage-driver-user="root"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754475    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754478    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
May 20 13:50:39.776891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754481    2566 flags.go:64] FLAG: --system-cgroups=""
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754483    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754489    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754492    2566 flags.go:64] FLAG: --tls-cert-file=""
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754495    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754501    2566 flags.go:64] FLAG: --tls-min-version=""
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754503    2566 flags.go:64] FLAG: --tls-private-key-file=""
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754506    2566 flags.go:64] FLAG: --topology-manager-policy="none"
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754509    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754512    2566 flags.go:64] FLAG: --topology-manager-scope="container"
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754515    2566 flags.go:64] FLAG: --v="2"
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754519    2566 flags.go:64] FLAG: --version="false"
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754523    2566 flags.go:64] FLAG: --vmodule=""
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754527    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754530    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754639    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754642    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754645    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754649    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754654    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754657    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754661    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:39.777521 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754665    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754683    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754689    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754692    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754695    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754698    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754701    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754704    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754707    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754715    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754718    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754721    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754724    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754727    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754729    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754732    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754735    2566 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754737    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754740    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754742    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:39.778063 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754745    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754747    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754750    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754752    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754755    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754758    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754760    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754764    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754766    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754770    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754773    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754775    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754778    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754780    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754785    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754788    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754790    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754793    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754795    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754798    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:39.778609 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754801    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754804    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754812    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754814    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754817    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754820    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754822    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754825    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754827    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754830    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754832    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754835    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754838    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754840    2566 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754843    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754845    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754848    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754850    2566 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754853    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754856    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754858    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:39.779111 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754862    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754864    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754867    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754870    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754872    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754876    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754879    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754881    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754884    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754886    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754889    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754892    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754894    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754897    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754906    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754909    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754912    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:39.779602 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.754915    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.754920    2566 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.761951    2566 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.761968    2566 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762019    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762024    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762028    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762031    2566 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762035    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762037    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762041    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762044    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762047    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762050    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762053    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762056    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:39.780055 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762058    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762061    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762064    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762066    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762069    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762071    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762074    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762077    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762079    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762082    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762084    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762087    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762090    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762094    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762096    2566 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762099    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762103    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762106    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762108    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:39.780438 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762111    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762113    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762116    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762119    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762121    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762124    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762126    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762129    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762131    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762134    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762137    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762139    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762142    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762144    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762147    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762150    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762152    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762155    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762157    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762160    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:39.780921 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762162    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762168    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762172    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762177    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762180    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762183    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762186    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762190    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762192    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762199    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762201    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762204    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762207    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762209    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762212    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762214    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762217    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762220    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762222    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:39.781451 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762225    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762228    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762230    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762232    2566 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762235    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762238    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762240    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762243    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762246    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762248    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762251    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762253    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762256    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762259    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762264    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:39.781932 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762268    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.762273    2566 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762371    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762376    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762379    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762382    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762388    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762391    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762396    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762400    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762403    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762406    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762409    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762411    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762415    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:39.782290 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762418    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762421    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762424    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762427    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762429    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762432    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762435    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762437    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762440    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762443    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762445    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762448    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762451    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762453    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762456    2566 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762458    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762462    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762465    2566 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762467    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762470    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:39.782710 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762472    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762475    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762477    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762480    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762483    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762487    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762490    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762493    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762496    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762498    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762501    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762503    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762506    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762508    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762511    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762514    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762516    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762519    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762522    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:39.783186 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762524    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762527    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762529    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762532    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762535    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762537    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762540    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762542    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762545    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762549    2566 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762552    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762554    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762557    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762559    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762562    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762564    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762567    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762570    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762574    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762577    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:39.783714 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762579    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762582    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762585    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762587    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762590    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762592    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762595    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762597    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762600    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762602    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762605    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762607    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762610    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:39.762612    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.762617    2566 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:39.784225 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.763383    2566 server.go:962] "Client rotation is on, will bootstrap in background"
May 20 13:50:39.784618 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.766660    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 20 13:50:39.784618 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.767574    2566 server.go:1019] "Starting client certificate rotation"
May 20 13:50:39.784618 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.770192    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 13:50:39.784618 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.770233    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 13:50:39.791995 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.791977    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 13:50:39.794649 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.794627    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 13:50:39.808069 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.808050    2566 log.go:25] "Validated CRI v1 runtime API"
May 20 13:50:39.813631 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.813614    2566 log.go:25] "Validated CRI v1 image API"
May 20 13:50:39.815042 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.815025    2566 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 20 13:50:39.820163 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.820140    2566 fs.go:135] Filesystem UUIDs: map[5607c424-763c-4d70-bfff-3efded233490:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8f43dbc4-86f0-4235-aa93-88e4644ec4e7:/dev/nvme0n1p4]
May 20 13:50:39.820220 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.820164    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 20 13:50:39.824832 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.824704    2566 manager.go:217] Machine: {Timestamp:2026-05-20 13:50:39.824194622 +0000 UTC m=+0.378640188 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101396 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec210b60200dc706d63c01d4bdbb96c1 SystemUUID:ec210b60-200d-c706-d63c-01d4bdbb96c1 BootID:87e4679c-05b3-4851-a3c1-607fafd10ee6 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ac:b9:48:e9:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ac:b9:48:e9:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:da:8e:f4:56:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 20 13:50:39.825494 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.825482    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 20 13:50:39.825583 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.825571    2566 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 20 13:50:39.827263 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.827237    2566 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 20 13:50:39.827389 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.827374    2566 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 13:50:39.827446 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.827265    2566 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-71.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 20 13:50:39.827513 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.827461    2566 topology_manager.go:138] "Creating topology manager with none policy"
May 20 13:50:39.827513 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.827475    2566 container_manager_linux.go:306] "Creating device plugin manager"
May 20 13:50:39.827513 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.827486    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 13:50:39.828222 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.828210    2566 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 13:50:39.830936 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.830922    2566 state_mem.go:36] "Initialized new in-memory state store"
May 20 13:50:39.831074 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.831062    2566 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 20 13:50:39.833177 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.833166    2566 kubelet.go:491] "Attempting to sync node with API server"
May 20 13:50:39.833235 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.833183    2566 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 20 13:50:39.833235 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.833201    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 20 13:50:39.833235 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.833215    2566 kubelet.go:397] "Adding apiserver pod source"
May 20 13:50:39.833235 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.833227    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 20 13:50:39.834309 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.834296    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 13:50:39.834383 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.834319    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 13:50:39.836882 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.836866    2566 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 20 13:50:39.839160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.839147    2566 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 20 13:50:39.839937 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.839918    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 20 13:50:39.839980 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.839958    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 20 13:50:39.839980 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.839965    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 20 13:50:39.839980 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.839971    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 20 13:50:39.839980 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.839980    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 20 13:50:39.840092 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.839986    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 20 13:50:39.840092 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.839991    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 20 13:50:39.840092 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.840005    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 20 13:50:39.840092 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.840013    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 20 13:50:39.840092 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.840018    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 20 13:50:39.840092 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.840034    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 20 13:50:39.840265 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.840226    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 20 13:50:39.841001 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.840991    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 20 13:50:39.841001 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.841000    2566 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 20 13:50:39.844275 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.844261    2566 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 20 13:50:39.844312 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.844307    2566 server.go:1295] "Started kubelet"
May 20 13:50:39.844769 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.844751    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 20 13:50:39.845236 ip-10-0-140-71 systemd[1]: Started Kubernetes Kubelet.
May 20 13:50:39.845990 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.845806    2566 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 20 13:50:39.845990 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.845882    2566 server_v1.go:47] "podresources" method="list" useActivePods=true
May 20 13:50:39.847127 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.847113    2566 server.go:317] "Adding debug handlers to kubelet server"
May 20 13:50:39.847519 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.847501    2566 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 20 13:50:39.851460 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.851432    2566 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 20 13:50:39.851460 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.851446    2566 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-71.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 20 13:50:39.851628 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.851605    2566 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 20 13:50:39.854276 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.854257    2566 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 20 13:50:39.854771 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.851499    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-71.ec2.internal.18b14a511215b74d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-71.ec2.internal,UID:ip-10-0-140-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-71.ec2.internal,},FirstTimestamp:2026-05-20 13:50:39.844276045 +0000 UTC m=+0.398721615,LastTimestamp:2026-05-20 13:50:39.844276045 +0000 UTC m=+0.398721615,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-71.ec2.internal,}"
May 20 13:50:39.855564 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.855546    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 20 13:50:39.855564 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.855554    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 20 13:50:39.855723 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.855650    2566 volume_manager.go:295] "The desired_state_of_world populator starts"
May 20 13:50:39.855723 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.855661    2566 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 20 13:50:39.855809 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.855794    2566 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 20 13:50:39.855874 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.855862    2566 reconstruct.go:97] "Volume reconstruction finished"
May 20 13:50:39.855919 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.855876    2566 reconciler.go:26] "Reconciler: start to sync state"
May 20 13:50:39.856381 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.856366    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:39.856454 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.856416    2566 factory.go:55] Registering systemd factory
May 20 13:50:39.856506 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.856466    2566 factory.go:223] Registration of the systemd container factory successfully
May 20 13:50:39.856708 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.856690    2566 factory.go:153] Registering CRI-O factory
May 20 13:50:39.856768 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.856710    2566 factory.go:223] Registration of the crio container factory successfully
May 20 13:50:39.856768 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.856761    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 20 13:50:39.856890 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.856778    2566 factory.go:103] Registering Raw factory
May 20 13:50:39.856890 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.856794    2566 manager.go:1196] Started watching for new ooms in manager
May 20 13:50:39.858077 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.858062    2566 manager.go:319] Starting recovery of all containers
May 20 13:50:39.859056 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.859034    2566 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 20 13:50:39.859239 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.859218    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 20 13:50:39.872499 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.872484    2566 manager.go:324] Recovery completed
May 20 13:50:39.876739 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.876611    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:39.878634 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.878616    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:39.878734 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.878646    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:39.878734 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.878657    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:39.879089 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.879067    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vpbgd"
May 20 13:50:39.879210 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.879194    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 20 13:50:39.879210 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.879206    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 20 13:50:39.879321 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.879221    2566 state_mem.go:36] "Initialized new in-memory state store"
May 20 13:50:39.881983 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.881972    2566 policy_none.go:49] "None policy: Start"
May 20 13:50:39.882036 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.881992    2566 memory_manager.go:186] "Starting memorymanager" policy="None"
May 20 13:50:39.882036 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.882003    2566 state_mem.go:35] "Initializing new in-memory state store"
May 20 13:50:39.889271 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.889254    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vpbgd"
May 20 13:50:39.889610 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.889551    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-71.ec2.internal.18b14a511421f9a8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-71.ec2.internal,UID:ip-10-0-140-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-71.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-71.ec2.internal,},FirstTimestamp:2026-05-20 13:50:39.878633896 +0000 UTC m=+0.433079461,LastTimestamp:2026-05-20 13:50:39.878633896 +0000 UTC m=+0.433079461,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-71.ec2.internal,}"
May 20 13:50:39.924954 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.915584    2566 manager.go:341] "Starting Device Plugin manager"
May 20 13:50:39.924954 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.915620    2566 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 20 13:50:39.924954 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.915642    2566 server.go:85] "Starting device plugin registration server"
May 20 13:50:39.924954 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.916286    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
May 20 13:50:39.924954 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.916297    2566 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 20 13:50:39.924954 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.916399    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 20 13:50:39.924954 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.916822    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 20 13:50:39.924954 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.916835    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 20 13:50:39.924954 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.917520    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 20 13:50:39.924954 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.917560    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:39.985224 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.985197    2566 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 20 13:50:39.986344 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.986330    2566 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 20 13:50:39.986344 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.986347    2566 status_manager.go:230] "Starting to sync pod status with apiserver"
May 20 13:50:39.986467 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.986364    2566 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 20 13:50:39.986467 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.986371    2566 kubelet.go:2451] "Starting kubelet main sync loop"
May 20 13:50:39.986467 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:39.986405    2566 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 20 13:50:39.988962 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:39.988944    2566 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:40.016777 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.016757    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:40.017823 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.017808    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:40.017909 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.017836    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:40.017909 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.017847    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:40.017909 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.017869    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-71.ec2.internal"
May 20 13:50:40.026312 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.026275    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-71.ec2.internal"
May 20 13:50:40.026312 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.026295    2566 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-71.ec2.internal\": node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.051436 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.051415    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.087502 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.087481    2566 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"]
May 20 13:50:40.087563 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.087540    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:40.089512 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.089496    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:40.089595 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.089523    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:40.089595 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.089534    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:40.090656 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.090645    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:40.090803 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.090790    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.090840 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.090817    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:40.091309 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.091295    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:40.091309 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.091317    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:40.091442 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.091333    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:40.091592 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.091576    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:40.091697 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.091603    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:40.091697 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.091627    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:40.092932 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.092917    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.093026 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.092942    2566 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:40.093709 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.093693    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:40.093791 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.093723    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:40.093791 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.093737    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:40.119253 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.119230    2566 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-71.ec2.internal\" not found" node="ip-10-0-140-71.ec2.internal"
May 20 13:50:40.123605 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.123590    2566 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-71.ec2.internal\" not found" node="ip-10-0-140-71.ec2.internal"
May 20 13:50:40.152110 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.152078    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.252511 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.252481    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.257777 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.257759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cf59113a5665c46de4a28abde7daddb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-71.ec2.internal\" (UID: \"2cf59113a5665c46de4a28abde7daddb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.257896 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.257782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcdfa0c75a262c6545872ac1dac38391-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"bcdfa0c75a262c6545872ac1dac38391\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.257896 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.257801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcdfa0c75a262c6545872ac1dac38391-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"bcdfa0c75a262c6545872ac1dac38391\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.353481 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.353453    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.358781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.358766    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcdfa0c75a262c6545872ac1dac38391-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"bcdfa0c75a262c6545872ac1dac38391\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.358848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.358792    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcdfa0c75a262c6545872ac1dac38391-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"bcdfa0c75a262c6545872ac1dac38391\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.358848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.358810    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cf59113a5665c46de4a28abde7daddb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-71.ec2.internal\" (UID: \"2cf59113a5665c46de4a28abde7daddb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.358848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.358838    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cf59113a5665c46de4a28abde7daddb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-71.ec2.internal\" (UID: \"2cf59113a5665c46de4a28abde7daddb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.358939 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.358863    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcdfa0c75a262c6545872ac1dac38391-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"bcdfa0c75a262c6545872ac1dac38391\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.358939 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.358863    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcdfa0c75a262c6545872ac1dac38391-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal\" (UID: \"bcdfa0c75a262c6545872ac1dac38391\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.422926 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.422880    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.426667 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.426649    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
May 20 13:50:40.453768 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.453736    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.554725 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.554684    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.655214 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.655140    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.755609 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.755574    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.767954 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.767933    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 20 13:50:40.768479 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.768064    2566 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 13:50:40.785175 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.785148    2566 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:40.856136 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.856104    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.856299 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.856184    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 20 13:50:40.878206 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.878182    2566 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 13:50:40.891498 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.891466    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 13:45:39 +0000 UTC" deadline="2027-12-26 15:05:30.735393087 +0000 UTC"
May 20 13:50:40.891498 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.891493    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14041h14m49.843902505s"
May 20 13:50:40.896030 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.896011    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7zs4l"
May 20 13:50:40.908195 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.908152    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7zs4l"
May 20 13:50:40.947125 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:40.947092    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdfa0c75a262c6545872ac1dac38391.slice/crio-6b9a1beb14a492425071cde115efed396a7e472f7f4474cd66655f4c5445b4b9 WatchSource:0}: Error finding container 6b9a1beb14a492425071cde115efed396a7e472f7f4474cd66655f4c5445b4b9: Status 404 returned error can't find the container with id 6b9a1beb14a492425071cde115efed396a7e472f7f4474cd66655f4c5445b4b9
May 20 13:50:40.947320 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:40.947299    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf59113a5665c46de4a28abde7daddb.slice/crio-d024277201e40dab4f762f039160d880fa56c91765da50492ef2375f6d334d19 WatchSource:0}: Error finding container d024277201e40dab4f762f039160d880fa56c91765da50492ef2375f6d334d19: Status 404 returned error can't find the container with id d024277201e40dab4f762f039160d880fa56c91765da50492ef2375f6d334d19
May 20 13:50:40.951647 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.951632    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 13:50:40.956265 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:40.956248    2566 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-71.ec2.internal\" not found"
May 20 13:50:40.989846 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.989798    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal" event={"ID":"2cf59113a5665c46de4a28abde7daddb","Type":"ContainerStarted","Data":"d024277201e40dab4f762f039160d880fa56c91765da50492ef2375f6d334d19"}
May 20 13:50:40.990745 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.990722    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal" event={"ID":"bcdfa0c75a262c6545872ac1dac38391","Type":"ContainerStarted","Data":"6b9a1beb14a492425071cde115efed396a7e472f7f4474cd66655f4c5445b4b9"}
May 20 13:50:40.996221 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:40.996204    2566 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:41.026453 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.026430    2566 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:41.056088 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.056054    2566 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal"
May 20 13:50:41.072406 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.072388    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 13:50:41.073056 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.073045    2566 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal"
May 20 13:50:41.083667 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.083655    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 13:50:41.833721 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.833694    2566 apiserver.go:52] "Watching apiserver"
May 20 13:50:41.841957 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.841933    2566 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 20 13:50:41.844424 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.844398    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25","openshift-cluster-node-tuning-operator/tuned-wmtf6","openshift-dns/node-resolver-dxhmp","openshift-multus/multus-dm4bw","openshift-network-diagnostics/network-check-target-v8vbc","openshift-image-registry/node-ca-v27tj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal","openshift-multus/multus-additional-cni-plugins-gfj8n","openshift-multus/network-metrics-daemon-twxjw","openshift-network-operator/iptables-alerter-vpxdx","openshift-ovn-kubernetes/ovnkube-node-rdtkf","kube-system/konnectivity-agent-8qthd"]
May 20 13:50:41.846891 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.846870    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:41.849025 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.848972    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.850106 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.850087    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 20 13:50:41.850492 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.850475    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8snb7\""
May 20 13:50:41.850582 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.850561    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 20 13:50:41.850847 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.850795    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 20 13:50:41.851923 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.851898    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 20 13:50:41.852155 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.852135    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 20 13:50:41.852698 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.852661    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rllbv\""
May 20 13:50:41.852936 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.852920    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 20 13:50:41.853291 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.853273    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.855748 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.855731    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:41.856484 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.856465    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 20 13:50:41.856572 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.856505    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m5jvj\""
May 20 13:50:41.856572 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.856529    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 20 13:50:41.858232 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.858215    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.858415 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.858396    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rltdz\""
May 20 13:50:41.858940 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.858917    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 20 13:50:41.859045 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.858959    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 20 13:50:41.860533 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.860501    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:41.860632 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:41.860577    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:50:41.860842 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.860828    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 20 13:50:41.860925 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.860843    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 20 13:50:41.860925 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.860887    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 20 13:50:41.861029 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.860954    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lckns\""
May 20 13:50:41.861292 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.861274    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 20 13:50:41.863104 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.862847    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:41.865082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865064    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:41.865179 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:41.865139    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:50:41.865286 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865271    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 20 13:50:41.865739 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865723    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 20 13:50:41.865834 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865750    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kcbqn\""
May 20 13:50:41.865834 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-host\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.865834 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db48875c-3279-4fbe-8771-625c9099802f-host\") pod \"node-ca-v27tj\" (UID: \"db48875c-3279-4fbe-8771-625c9099802f\") " pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:41.865989 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-device-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.865989 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.865989 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl5sb\" (UniqueName: \"kubernetes.io/projected/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-kube-api-access-kl5sb\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.865989 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-var-lib-cni-multus\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.865989 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-var-lib-kubelet\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.865989 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.865981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvcbv\" (UniqueName: \"kubernetes.io/projected/db48875c-3279-4fbe-8771-625c9099802f-kube-api-access-rvcbv\") pod \"node-ca-v27tj\" (UID: \"db48875c-3279-4fbe-8771-625c9099802f\") " pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:41.866253 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-lib-modules\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.866253 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpfl\" (UniqueName: \"kubernetes.io/projected/1e092c89-8dac-4254-bbf3-8db3d7058f76-kube-api-access-xlpfl\") pod \"node-resolver-dxhmp\" (UID: \"1e092c89-8dac-4254-bbf3-8db3d7058f76\") " pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:41.866253 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-cnibin\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.866253 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-daemon-config\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.866253 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.866253 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-socket-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.866253 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-sys-fs\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.866253 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-tuned\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.866253 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24d3b560-c9fd-48f2-9967-def69e9bb2f8-cni-binary-copy\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-run-k8s-cni-cncf-io\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-run-netns\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-kubernetes\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-systemd\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3864b0b1-b650-47fc-ba2e-cbbff966797a-tmp\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-etc-kubernetes\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866446    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-modprobe-d\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-cni-dir\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866504    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-os-release\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-socket-dir-parent\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-sysconfig\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-sys\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866611    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-var-lib-kubelet\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e092c89-8dac-4254-bbf3-8db3d7058f76-hosts-file\") pod \"node-resolver-dxhmp\" (UID: \"1e092c89-8dac-4254-bbf3-8db3d7058f76\") " pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e092c89-8dac-4254-bbf3-8db3d7058f76-tmp-dir\") pod \"node-resolver-dxhmp\" (UID: \"1e092c89-8dac-4254-bbf3-8db3d7058f76\") " pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:41.866705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-system-cni-dir\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-var-lib-cni-bin\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-hostroot\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db48875c-3279-4fbe-8771-625c9099802f-serviceca\") pod \"node-ca-v27tj\" (UID: \"db48875c-3279-4fbe-8771-625c9099802f\") " pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-sysctl-conf\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-run\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gs8\" (UniqueName: \"kubernetes.io/projected/3864b0b1-b650-47fc-ba2e-cbbff966797a-kube-api-access-69gs8\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866947    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-conf-dir\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-run-multus-certs\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.866988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bznmb\" (UniqueName: \"kubernetes.io/projected/24d3b560-c9fd-48f2-9967-def69e9bb2f8-kube-api-access-bznmb\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.867041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-registration-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.867360 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.867083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-sysctl-d\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.867885 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.867440    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:41.869921 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.869800    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 20 13:50:41.869921 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.869823    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 20 13:50:41.870069 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.869938    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 20 13:50:41.870294 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.870276    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2xvnw\""
May 20 13:50:41.870979 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.870879    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.873128 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.873107    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8qthd"
May 20 13:50:41.873314 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.873294    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 20 13:50:41.873531 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.873513    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 20 13:50:41.873601 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.873571    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 20 13:50:41.873650 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.873606    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 20 13:50:41.873650 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.873614    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 20 13:50:41.874709 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.874653    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hvvlr\""
May 20 13:50:41.874837 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.874819    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 20 13:50:41.875637 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.875615    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 20 13:50:41.875899 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.875821    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qh7ft\""
May 20 13:50:41.875986 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.875904    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 20 13:50:41.909348 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.909306    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:40 +0000 UTC" deadline="2027-11-12 04:17:31.356823549 +0000 UTC"
May 20 13:50:41.909348 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.909347    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12974h26m49.447480365s"
May 20 13:50:41.957169 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.957143    2566 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 20 13:50:41.967857 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.967822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8959890a-80d9-44d7-a2f1-64b78a493e34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:41.967999 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.967860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8959890a-80d9-44d7-a2f1-64b78a493e34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:41.967999 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.967890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctms\" (UniqueName: \"kubernetes.io/projected/629f62b6-2132-440f-9f2f-e327d44004c9-kube-api-access-cctms\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:41.967999 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.967954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-run-netns\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.967999 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.967997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-node-log\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968175 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968029    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-sysconfig\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.968175 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968056    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-sys\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.968175 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968083    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e092c89-8dac-4254-bbf3-8db3d7058f76-hosts-file\") pod \"node-resolver-dxhmp\" (UID: \"1e092c89-8dac-4254-bbf3-8db3d7058f76\") " pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:41.968175 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968106    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e092c89-8dac-4254-bbf3-8db3d7058f76-tmp-dir\") pod \"node-resolver-dxhmp\" (UID: \"1e092c89-8dac-4254-bbf3-8db3d7058f76\") " pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:41.968175 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968132    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-hostroot\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.968175 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968155    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-sysconfig\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968177    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e092c89-8dac-4254-bbf3-8db3d7058f76-hosts-file\") pod \"node-resolver-dxhmp\" (UID: \"1e092c89-8dac-4254-bbf3-8db3d7058f76\") " pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968175    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-sys\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968158    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bznmb\" (UniqueName: \"kubernetes.io/projected/24d3b560-c9fd-48f2-9967-def69e9bb2f8-kube-api-access-bznmb\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968232    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-sysctl-conf\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968258    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-run\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968281    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-69gs8\" (UniqueName: \"kubernetes.io/projected/3864b0b1-b650-47fc-ba2e-cbbff966797a-kube-api-access-69gs8\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968305    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-conf-dir\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968315    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-run\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968285    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-hostroot\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-run-systemd\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28nf\" (UniqueName: \"kubernetes.io/projected/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-kube-api-access-m28nf\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-kubelet\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968386    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-conf-dir\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.968400 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-etc-openvswitch\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968378    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-sysctl-conf\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-run-ovn\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968441    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e092c89-8dac-4254-bbf3-8db3d7058f76-tmp-dir\") pod \"node-resolver-dxhmp\" (UID: \"1e092c89-8dac-4254-bbf3-8db3d7058f76\") " pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-log-socket\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-ovnkube-config\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-ovnkube-script-lib\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968538    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db48875c-3279-4fbe-8771-625c9099802f-host\") pod \"node-ca-v27tj\" (UID: \"db48875c-3279-4fbe-8771-625c9099802f\") " pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968563    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-device-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968569    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db48875c-3279-4fbe-8771-625c9099802f-host\") pod \"node-ca-v27tj\" (UID: \"db48875c-3279-4fbe-8771-625c9099802f\") " pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968584    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968604    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-var-lib-kubelet\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968613    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-device-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-cni-bin\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-ovn-node-metrics-cert\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968657    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-var-lib-kubelet\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.968993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d87ec45b-44de-41c0-b11f-d927ae91e4d6-konnectivity-ca\") pod \"konnectivity-agent-8qthd\" (UID: \"d87ec45b-44de-41c0-b11f-d927ae91e4d6\") " pod="kube-system/konnectivity-agent-8qthd"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968752    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8959890a-80d9-44d7-a2f1-64b78a493e34-cni-binary-copy\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968746    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968780    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rvcbv\" (UniqueName: \"kubernetes.io/projected/db48875c-3279-4fbe-8771-625c9099802f-kube-api-access-rvcbv\") pod \"node-ca-v27tj\" (UID: \"db48875c-3279-4fbe-8771-625c9099802f\") " pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968806    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-daemon-config\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968830    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqf5\" (UniqueName: \"kubernetes.io/projected/2810ffa5-0463-4a15-9729-c1d04fd38658-kube-api-access-nxqf5\") pod \"iptables-alerter-vpxdx\" (UID: \"2810ffa5-0463-4a15-9729-c1d04fd38658\") " pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968855    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-socket-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968881    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-sys-fs\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968975    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-sys-fs\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.968999    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-socket-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969043    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-run-k8s-cni-cncf-io\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-slash\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969103    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-run-openvswitch\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969169    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-run-k8s-cni-cncf-io\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969198    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-kubernetes\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969214    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-systemd\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-system-cni-dir\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:41.969767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-run-ovn-kubernetes\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969250    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-kubernetes\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-env-overrides\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969292    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-systemd\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969320    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-socket-dir-parent\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-os-release\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969402    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gg52\" (UniqueName: \"kubernetes.io/projected/8959890a-80d9-44d7-a2f1-64b78a493e34-kube-api-access-8gg52\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969418    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-socket-dir-parent\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-cni-netd\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d87ec45b-44de-41c0-b11f-d927ae91e4d6-agent-certs\") pod \"konnectivity-agent-8qthd\" (UID: \"d87ec45b-44de-41c0-b11f-d927ae91e4d6\") " pod="kube-system/konnectivity-agent-8qthd"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969484    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-daemon-config\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969552    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-var-lib-kubelet\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969579    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-system-cni-dir\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969604    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-var-lib-cni-bin\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969621    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-var-lib-kubelet\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969629    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-run-multus-certs\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.970466 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-cnibin\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969687    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-system-cni-dir\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969701    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db48875c-3279-4fbe-8771-625c9099802f-serviceca\") pod \"node-ca-v27tj\" (UID: \"db48875c-3279-4fbe-8771-625c9099802f\") " pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969691    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-var-lib-cni-bin\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969728    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-run-multus-certs\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2810ffa5-0463-4a15-9729-c1d04fd38658-iptables-alerter-script\") pod \"iptables-alerter-vpxdx\" (UID: \"2810ffa5-0463-4a15-9729-c1d04fd38658\") " pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-var-lib-openvswitch\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969786    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-registration-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969845    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-registration-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969864    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-sysctl-d\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969887    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-host\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2810ffa5-0463-4a15-9729-c1d04fd38658-host-slash\") pod \"iptables-alerter-vpxdx\" (UID: \"2810ffa5-0463-4a15-9729-c1d04fd38658\") " pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969931    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5sb\" (UniqueName: \"kubernetes.io/projected/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-kube-api-access-kl5sb\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969954    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-var-lib-cni-multus\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969992    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-sysctl-d\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-systemd-units\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:41.971160 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.969980    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-host\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970034    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-lib-modules\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970080    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db48875c-3279-4fbe-8771-625c9099802f-serviceca\") pod \"node-ca-v27tj\" (UID: \"db48875c-3279-4fbe-8771-625c9099802f\") " pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970085    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpfl\" (UniqueName: \"kubernetes.io/projected/1e092c89-8dac-4254-bbf3-8db3d7058f76-kube-api-access-xlpfl\") pod \"node-resolver-dxhmp\" (UID: \"1e092c89-8dac-4254-bbf3-8db3d7058f76\") " pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970094    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-var-lib-cni-multus\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970195    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-cnibin\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758m8\" (UniqueName: \"kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8\") pod \"network-check-target-v8vbc\" (UID: \"4452f914-4de0-4fbb-bbfe-237b2cc85d2c\") " pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970258    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970278    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-cnibin\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970282    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-tuned\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970309    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-lib-modules\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970316    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970318    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24d3b560-c9fd-48f2-9967-def69e9bb2f8-cni-binary-copy\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970432    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-run-netns\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970454    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3864b0b1-b650-47fc-ba2e-cbbff966797a-tmp\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970475    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-etc-kubernetes\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970495    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-modprobe-d\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970514    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-cni-dir\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.971781 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970538    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-os-release\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.972523 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970555    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-etc-kubernetes\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.972523 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970550    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 20 13:50:41.972523 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970635    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-os-release\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.972523 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970548    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-host-run-netns\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.972523 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970656    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-modprobe-d\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.972523 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970718    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d3b560-c9fd-48f2-9967-def69e9bb2f8-multus-cni-dir\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.972523 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.970767    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24d3b560-c9fd-48f2-9967-def69e9bb2f8-cni-binary-copy\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.973925 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.973885    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3864b0b1-b650-47fc-ba2e-cbbff966797a-etc-tuned\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.973925 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.973915    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3864b0b1-b650-47fc-ba2e-cbbff966797a-tmp\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.978100 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.978054    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl5sb\" (UniqueName: \"kubernetes.io/projected/ecee6bdf-7a9e-42a4-b757-1fafcc70e88a-kube-api-access-kl5sb\") pod \"aws-ebs-csi-driver-node-rnv25\" (UID: \"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:41.978100 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.978056    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bznmb\" (UniqueName: \"kubernetes.io/projected/24d3b560-c9fd-48f2-9967-def69e9bb2f8-kube-api-access-bznmb\") pod \"multus-dm4bw\" (UID: \"24d3b560-c9fd-48f2-9967-def69e9bb2f8\") " pod="openshift-multus/multus-dm4bw"
May 20 13:50:41.978499 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.978475    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gs8\" (UniqueName: \"kubernetes.io/projected/3864b0b1-b650-47fc-ba2e-cbbff966797a-kube-api-access-69gs8\") pod \"tuned-wmtf6\" (UID: \"3864b0b1-b650-47fc-ba2e-cbbff966797a\") " pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:41.978576 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.978478    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvcbv\" (UniqueName: \"kubernetes.io/projected/db48875c-3279-4fbe-8771-625c9099802f-kube-api-access-rvcbv\") pod \"node-ca-v27tj\" (UID: \"db48875c-3279-4fbe-8771-625c9099802f\") " pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:41.978876 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:41.978858    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpfl\" (UniqueName: \"kubernetes.io/projected/1e092c89-8dac-4254-bbf3-8db3d7058f76-kube-api-access-xlpfl\") pod \"node-resolver-dxhmp\" (UID: \"1e092c89-8dac-4254-bbf3-8db3d7058f76\") " pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:42.008608 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.008571    2566 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:42.070992 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.070953    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-758m8\" (UniqueName: \"kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8\") pod \"network-check-target-v8vbc\" (UID: \"4452f914-4de0-4fbb-bbfe-237b2cc85d2c\") " pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:42.070992 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.070996    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8959890a-80d9-44d7-a2f1-64b78a493e34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.071207 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071016    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8959890a-80d9-44d7-a2f1-64b78a493e34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.071207 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071041    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cctms\" (UniqueName: \"kubernetes.io/projected/629f62b6-2132-440f-9f2f-e327d44004c9-kube-api-access-cctms\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:42.071207 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071166    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-run-netns\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071359 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071210    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-node-log\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071359 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071247    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-run-systemd\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071359 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071273    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m28nf\" (UniqueName: \"kubernetes.io/projected/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-kube-api-access-m28nf\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071359 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071303    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-kubelet\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071359 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071305    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-run-netns\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071359 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071327    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-etc-openvswitch\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071359 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071341    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-node-log\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071366    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-kubelet\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071374    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-etc-openvswitch\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071396    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-run-ovn\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071422    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-log-socket\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071448    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071461    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-run-ovn\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071474    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-log-socket\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071477    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-ovnkube-config\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071504    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-ovnkube-script-lib\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071514    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071534    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-cni-bin\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071572    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-ovn-node-metrics-cert\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071591    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-cni-bin\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071597    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d87ec45b-44de-41c0-b11f-d927ae91e4d6-konnectivity-ca\") pod \"konnectivity-agent-8qthd\" (UID: \"d87ec45b-44de-41c0-b11f-d927ae91e4d6\") " pod="kube-system/konnectivity-agent-8qthd"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071622    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8959890a-80d9-44d7-a2f1-64b78a493e34-cni-binary-copy\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071649    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqf5\" (UniqueName: \"kubernetes.io/projected/2810ffa5-0463-4a15-9729-c1d04fd38658-kube-api-access-nxqf5\") pod \"iptables-alerter-vpxdx\" (UID: \"2810ffa5-0463-4a15-9729-c1d04fd38658\") " pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:42.071696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071667    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8959890a-80d9-44d7-a2f1-64b78a493e34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071695    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-slash\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071720    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-run-openvswitch\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071748    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-system-cni-dir\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071771    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-run-ovn-kubernetes\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071796    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-env-overrides\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071821    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-os-release\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071850    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071875    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8gg52\" (UniqueName: \"kubernetes.io/projected/8959890a-80d9-44d7-a2f1-64b78a493e34-kube-api-access-8gg52\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071924    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-cni-netd\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071923    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-system-cni-dir\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071960    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-slash\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071979    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-run-ovn-kubernetes\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071990    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d87ec45b-44de-41c0-b11f-d927ae91e4d6-agent-certs\") pod \"konnectivity-agent-8qthd\" (UID: \"d87ec45b-44de-41c0-b11f-d927ae91e4d6\") " pod="kube-system/konnectivity-agent-8qthd"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072025    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-cnibin\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072056    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2810ffa5-0463-4a15-9729-c1d04fd38658-iptables-alerter-script\") pod \"iptables-alerter-vpxdx\" (UID: \"2810ffa5-0463-4a15-9729-c1d04fd38658\") " pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072081    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-var-lib-openvswitch\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.072449 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072107    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2810ffa5-0463-4a15-9729-c1d04fd38658-host-slash\") pod \"iptables-alerter-vpxdx\" (UID: \"2810ffa5-0463-4a15-9729-c1d04fd38658\") " pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072131    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072148    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-ovnkube-config\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072156    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-systemd-units\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072154    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8959890a-80d9-44d7-a2f1-64b78a493e34-cni-binary-copy\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.071690    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8959890a-80d9-44d7-a2f1-64b78a493e34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072199    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-ovnkube-script-lib\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072226    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d87ec45b-44de-41c0-b11f-d927ae91e4d6-konnectivity-ca\") pod \"konnectivity-agent-8qthd\" (UID: \"d87ec45b-44de-41c0-b11f-d927ae91e4d6\") " pod="kube-system/konnectivity-agent-8qthd"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072244    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-os-release\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072293    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-host-cni-netd\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072321    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-var-lib-openvswitch\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072329    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-cnibin\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072332    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-run-openvswitch\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072357    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2810ffa5-0463-4a15-9729-c1d04fd38658-host-slash\") pod \"iptables-alerter-vpxdx\" (UID: \"2810ffa5-0463-4a15-9729-c1d04fd38658\") " pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072361    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-systemd-units\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072295    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8959890a-80d9-44d7-a2f1-64b78a493e34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072398    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-env-overrides\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.073082 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.072410    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:42.073717 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.072514    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs podName:629f62b6-2132-440f-9f2f-e327d44004c9 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:42.572474305 +0000 UTC m=+3.126919870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs") pod "network-metrics-daemon-twxjw" (UID: "629f62b6-2132-440f-9f2f-e327d44004c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:42.073717 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072548    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-run-systemd\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.073717 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.072646    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2810ffa5-0463-4a15-9729-c1d04fd38658-iptables-alerter-script\") pod \"iptables-alerter-vpxdx\" (UID: \"2810ffa5-0463-4a15-9729-c1d04fd38658\") " pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:42.074570 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.074527    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-ovn-node-metrics-cert\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.074793 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.074776    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d87ec45b-44de-41c0-b11f-d927ae91e4d6-agent-certs\") pod \"konnectivity-agent-8qthd\" (UID: \"d87ec45b-44de-41c0-b11f-d927ae91e4d6\") " pod="kube-system/konnectivity-agent-8qthd"
May 20 13:50:42.079804 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.079785    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:42.079804 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.079806    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:42.079917 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.079836    2566 projected.go:194] Error preparing data for projected volume kube-api-access-758m8 for pod openshift-network-diagnostics/network-check-target-v8vbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:42.079917 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.079910    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8 podName:4452f914-4de0-4fbb-bbfe-237b2cc85d2c nodeName:}" failed. No retries permitted until 2026-05-20 13:50:42.579892462 +0000 UTC m=+3.134338014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-758m8" (UniqueName: "kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8") pod "network-check-target-v8vbc" (UID: "4452f914-4de0-4fbb-bbfe-237b2cc85d2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:42.081338 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.081314    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqf5\" (UniqueName: \"kubernetes.io/projected/2810ffa5-0463-4a15-9729-c1d04fd38658-kube-api-access-nxqf5\") pod \"iptables-alerter-vpxdx\" (UID: \"2810ffa5-0463-4a15-9729-c1d04fd38658\") " pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:42.082000 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.081833    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gg52\" (UniqueName: \"kubernetes.io/projected/8959890a-80d9-44d7-a2f1-64b78a493e34-kube-api-access-8gg52\") pod \"multus-additional-cni-plugins-gfj8n\" (UID: \"8959890a-80d9-44d7-a2f1-64b78a493e34\") " pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.082910 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.082890    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctms\" (UniqueName: \"kubernetes.io/projected/629f62b6-2132-440f-9f2f-e327d44004c9-kube-api-access-cctms\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:42.082990 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.082947    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28nf\" (UniqueName: \"kubernetes.io/projected/85434994-53d3-48f0-beaf-f4b9e3ecdeb7-kube-api-access-m28nf\") pod \"ovnkube-node-rdtkf\" (UID: \"85434994-53d3-48f0-beaf-f4b9e3ecdeb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.159784 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.159750    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v27tj"
May 20 13:50:42.167528 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.167499    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25"
May 20 13:50:42.176124 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.176102    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wmtf6"
May 20 13:50:42.181734 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.181705    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dxhmp"
May 20 13:50:42.188339 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.188313    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dm4bw"
May 20 13:50:42.196021 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.195994    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gfj8n"
May 20 13:50:42.203716 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.203687    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vpxdx"
May 20 13:50:42.210494 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.210466    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:50:42.216271 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.216248    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8qthd"
May 20 13:50:42.544431 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:42.544402    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85434994_53d3_48f0_beaf_f4b9e3ecdeb7.slice/crio-6cb695a04c18d9811fe493918773eb74e02a65051ffe2835414e25978dc26268 WatchSource:0}: Error finding container 6cb695a04c18d9811fe493918773eb74e02a65051ffe2835414e25978dc26268: Status 404 returned error can't find the container with id 6cb695a04c18d9811fe493918773eb74e02a65051ffe2835414e25978dc26268
May 20 13:50:42.545485 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:42.545457    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2810ffa5_0463_4a15_9729_c1d04fd38658.slice/crio-7722b5715fd095161e0953897ba8342dfd3089f377d603e5cf0026cfd52f3ee1 WatchSource:0}: Error finding container 7722b5715fd095161e0953897ba8342dfd3089f377d603e5cf0026cfd52f3ee1: Status 404 returned error can't find the container with id 7722b5715fd095161e0953897ba8342dfd3089f377d603e5cf0026cfd52f3ee1
May 20 13:50:42.546025 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:42.546001    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb48875c_3279_4fbe_8771_625c9099802f.slice/crio-f61956e8185f1f4f0a08d49e1196b279573d420aed094e999572cd2ed53af61d WatchSource:0}: Error finding container f61956e8185f1f4f0a08d49e1196b279573d420aed094e999572cd2ed53af61d: Status 404 returned error can't find the container with id f61956e8185f1f4f0a08d49e1196b279573d420aed094e999572cd2ed53af61d
May 20 13:50:42.550182 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:42.550164    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8959890a_80d9_44d7_a2f1_64b78a493e34.slice/crio-439858541b6a28bb9b5e97c473696eb8f7b65f8cd246ce3a80ffe40abd75a555 WatchSource:0}: Error finding container 439858541b6a28bb9b5e97c473696eb8f7b65f8cd246ce3a80ffe40abd75a555: Status 404 returned error can't find the container with id 439858541b6a28bb9b5e97c473696eb8f7b65f8cd246ce3a80ffe40abd75a555
May 20 13:50:42.550899 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:42.550878    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3864b0b1_b650_47fc_ba2e_cbbff966797a.slice/crio-a206a293f47427bd924f8f53bc29c0473c0bf6dda84a040a1cc6f2909042d9ff WatchSource:0}: Error finding container a206a293f47427bd924f8f53bc29c0473c0bf6dda84a040a1cc6f2909042d9ff: Status 404 returned error can't find the container with id a206a293f47427bd924f8f53bc29c0473c0bf6dda84a040a1cc6f2909042d9ff
May 20 13:50:42.551829 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:42.551811    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87ec45b_44de_41c0_b11f_d927ae91e4d6.slice/crio-bce3596522d7649ef40241ab329b2c7c0ce97a08e61aac0dbb4c6fd043788e95 WatchSource:0}: Error finding container bce3596522d7649ef40241ab329b2c7c0ce97a08e61aac0dbb4c6fd043788e95: Status 404 returned error can't find the container with id bce3596522d7649ef40241ab329b2c7c0ce97a08e61aac0dbb4c6fd043788e95
May 20 13:50:42.558610 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:50:42.558560    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e092c89_8dac_4254_bbf3_8db3d7058f76.slice/crio-78d680533d63fb93605297935750347a9d893b3f5006340d42fc60dd4ad17f87 WatchSource:0}: Error finding container 78d680533d63fb93605297935750347a9d893b3f5006340d42fc60dd4ad17f87: Status 404 returned error can't find the container with id 78d680533d63fb93605297935750347a9d893b3f5006340d42fc60dd4ad17f87
May 20 13:50:42.576151 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.576129    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:42.576288 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.576271    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:42.576341 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.576335    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs podName:629f62b6-2132-440f-9f2f-e327d44004c9 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:43.576318751 +0000 UTC m=+4.130764302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs") pod "network-metrics-daemon-twxjw" (UID: "629f62b6-2132-440f-9f2f-e327d44004c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:42.677417 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.677252    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-758m8\" (UniqueName: \"kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8\") pod \"network-check-target-v8vbc\" (UID: \"4452f914-4de0-4fbb-bbfe-237b2cc85d2c\") " pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:42.677417 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.677410    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:42.677602 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.677434    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:42.677602 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.677446    2566 projected.go:194] Error preparing data for projected volume kube-api-access-758m8 for pod openshift-network-diagnostics/network-check-target-v8vbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:42.677602 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:42.677501    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8 podName:4452f914-4de0-4fbb-bbfe-237b2cc85d2c nodeName:}" failed. No retries permitted until 2026-05-20 13:50:43.677481632 +0000 UTC m=+4.231927201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-758m8" (UniqueName: "kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8") pod "network-check-target-v8vbc" (UID: "4452f914-4de0-4fbb-bbfe-237b2cc85d2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:42.909702 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.909656    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:40 +0000 UTC" deadline="2027-12-28 21:47:52.623009715 +0000 UTC"
May 20 13:50:42.909702 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.909700    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14095h57m9.713312502s"
May 20 13:50:42.994777 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.994746    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wmtf6" event={"ID":"3864b0b1-b650-47fc-ba2e-cbbff966797a","Type":"ContainerStarted","Data":"a206a293f47427bd924f8f53bc29c0473c0bf6dda84a040a1cc6f2909042d9ff"}
May 20 13:50:42.996020 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.995965    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfj8n" event={"ID":"8959890a-80d9-44d7-a2f1-64b78a493e34","Type":"ContainerStarted","Data":"439858541b6a28bb9b5e97c473696eb8f7b65f8cd246ce3a80ffe40abd75a555"}
May 20 13:50:42.997130 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:42.997100    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v27tj" event={"ID":"db48875c-3279-4fbe-8771-625c9099802f","Type":"ContainerStarted","Data":"f61956e8185f1f4f0a08d49e1196b279573d420aed094e999572cd2ed53af61d"}
May 20 13:50:43.000203 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.000179    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal" event={"ID":"2cf59113a5665c46de4a28abde7daddb","Type":"ContainerStarted","Data":"e1c9a955d61dd76944cfcc86155ab62eec91ec47eed439ac53d695654d13956f"}
May 20 13:50:43.003004 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.002974    2566 generic.go:358] "Generic (PLEG): container finished" podID="bcdfa0c75a262c6545872ac1dac38391" containerID="b9dda2f79096a548e2291514bb3abbcdd2f0672fde2cbbb70bc47fd13dc16694" exitCode=0
May 20 13:50:43.003100 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.003050    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal" event={"ID":"bcdfa0c75a262c6545872ac1dac38391","Type":"ContainerDied","Data":"b9dda2f79096a548e2291514bb3abbcdd2f0672fde2cbbb70bc47fd13dc16694"}
May 20 13:50:43.005696 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.005660    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25" event={"ID":"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a","Type":"ContainerStarted","Data":"2f848424771d85030da88794b3ba547e105a6e0b06605d297ca2b561ca303e0b"}
May 20 13:50:43.010804 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.010779    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dm4bw" event={"ID":"24d3b560-c9fd-48f2-9967-def69e9bb2f8","Type":"ContainerStarted","Data":"98a1047fb5fc55820d64d5b3e425034de16831e4573438f2988ab6b461e55eac"}
May 20 13:50:43.013655 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.013593    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vpxdx" event={"ID":"2810ffa5-0463-4a15-9729-c1d04fd38658","Type":"ContainerStarted","Data":"7722b5715fd095161e0953897ba8342dfd3089f377d603e5cf0026cfd52f3ee1"}
May 20 13:50:43.014726 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.014662    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf" event={"ID":"85434994-53d3-48f0-beaf-f4b9e3ecdeb7","Type":"ContainerStarted","Data":"6cb695a04c18d9811fe493918773eb74e02a65051ffe2835414e25978dc26268"}
May 20 13:50:43.016222 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.016165    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dxhmp" event={"ID":"1e092c89-8dac-4254-bbf3-8db3d7058f76","Type":"ContainerStarted","Data":"78d680533d63fb93605297935750347a9d893b3f5006340d42fc60dd4ad17f87"}
May 20 13:50:43.022038 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.021998    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8qthd" event={"ID":"d87ec45b-44de-41c0-b11f-d927ae91e4d6","Type":"ContainerStarted","Data":"bce3596522d7649ef40241ab329b2c7c0ce97a08e61aac0dbb4c6fd043788e95"}
May 20 13:50:43.034603 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.034551    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-71.ec2.internal" podStartSLOduration=2.034535995 podStartE2EDuration="2.034535995s" podCreationTimestamp="2026-05-20 13:50:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:43.016450601 +0000 UTC m=+3.570896173" watchObservedRunningTime="2026-05-20 13:50:43.034535995 +0000 UTC m=+3.588981569"
May 20 13:50:43.586280 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.586193    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:43.586447 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:43.586386    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:43.586500 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:43.586453    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs podName:629f62b6-2132-440f-9f2f-e327d44004c9 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:45.586433935 +0000 UTC m=+6.140879490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs") pod "network-metrics-daemon-twxjw" (UID: "629f62b6-2132-440f-9f2f-e327d44004c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:43.687034 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.686899    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-758m8\" (UniqueName: \"kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8\") pod \"network-check-target-v8vbc\" (UID: \"4452f914-4de0-4fbb-bbfe-237b2cc85d2c\") " pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:43.687221 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:43.687078    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:43.687221 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:43.687103    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:43.687221 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:43.687118    2566 projected.go:194] Error preparing data for projected volume kube-api-access-758m8 for pod openshift-network-diagnostics/network-check-target-v8vbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:43.687221 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:43.687178    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8 podName:4452f914-4de0-4fbb-bbfe-237b2cc85d2c nodeName:}" failed. No retries permitted until 2026-05-20 13:50:45.687158887 +0000 UTC m=+6.241604442 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-758m8" (UniqueName: "kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8") pod "network-check-target-v8vbc" (UID: "4452f914-4de0-4fbb-bbfe-237b2cc85d2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:43.988323 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.987825    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:43.988323 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:43.987962    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:50:43.989126 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:43.988957    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:43.989126 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:43.989053    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:50:44.030778 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:44.030728    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal" event={"ID":"bcdfa0c75a262c6545872ac1dac38391","Type":"ContainerStarted","Data":"f2eb5a7c51323b6578e92b81c658713f3f4f818f7c463ed45d9c6037ccd863f9"}
May 20 13:50:45.605064 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:45.605028    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:45.605531 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:45.605163    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:45.605531 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:45.605222    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs podName:629f62b6-2132-440f-9f2f-e327d44004c9 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:49.605203719 +0000 UTC m=+10.159649274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs") pod "network-metrics-daemon-twxjw" (UID: "629f62b6-2132-440f-9f2f-e327d44004c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:45.706938 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:45.706329    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-758m8\" (UniqueName: \"kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8\") pod \"network-check-target-v8vbc\" (UID: \"4452f914-4de0-4fbb-bbfe-237b2cc85d2c\") " pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:45.706938 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:45.706519    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:45.706938 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:45.706540    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:45.706938 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:45.706552    2566 projected.go:194] Error preparing data for projected volume kube-api-access-758m8 for pod openshift-network-diagnostics/network-check-target-v8vbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:45.706938 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:45.706610    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8 podName:4452f914-4de0-4fbb-bbfe-237b2cc85d2c nodeName:}" failed. No retries permitted until 2026-05-20 13:50:49.70659085 +0000 UTC m=+10.261036417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-758m8" (UniqueName: "kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8") pod "network-check-target-v8vbc" (UID: "4452f914-4de0-4fbb-bbfe-237b2cc85d2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:45.987817 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:45.987725    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:45.987817 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:45.987771    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:45.988023 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:45.987849    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:50:45.988326 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:45.988269    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:50:47.987341 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:47.987307    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:47.987836 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:47.987468    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:50:47.987836 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:47.987528    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:47.987836 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:47.987652    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:50:49.635992 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:49.635953    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:49.636440 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:49.636145    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:49.636440 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:49.636221    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs podName:629f62b6-2132-440f-9f2f-e327d44004c9 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:57.636200003 +0000 UTC m=+18.190645557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs") pod "network-metrics-daemon-twxjw" (UID: "629f62b6-2132-440f-9f2f-e327d44004c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:49.736722 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:49.736530    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-758m8\" (UniqueName: \"kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8\") pod \"network-check-target-v8vbc\" (UID: \"4452f914-4de0-4fbb-bbfe-237b2cc85d2c\") " pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:49.736722 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:49.736708    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:49.736722 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:49.736731    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:49.736997 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:49.736742    2566 projected.go:194] Error preparing data for projected volume kube-api-access-758m8 for pod openshift-network-diagnostics/network-check-target-v8vbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:49.736997 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:49.736801    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8 podName:4452f914-4de0-4fbb-bbfe-237b2cc85d2c nodeName:}" failed. No retries permitted until 2026-05-20 13:50:57.736783189 +0000 UTC m=+18.291228742 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-758m8" (UniqueName: "kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8") pod "network-check-target-v8vbc" (UID: "4452f914-4de0-4fbb-bbfe-237b2cc85d2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:49.988315 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:49.988230    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:49.988476 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:49.988397    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:49.988539 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:49.988491    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:50:49.988898 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:49.988868    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:50:51.986683 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:51.986599    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:51.986683 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:51.986646    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:51.987159 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:51.986758    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:50:51.987159 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:51.986881    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:50:53.986804 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:53.986763    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:53.987270 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:53.986777    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:53.987270 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:53.986910    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:50:53.987270 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:53.986963    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:50:55.986622 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:55.986579    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:55.987146 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:55.986736    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:50:55.987146 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:55.986816    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:55.987146 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:55.986927    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:50:57.690370 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:57.690326    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:57.690910 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:57.690446    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:57.690910 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:57.690559    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs podName:629f62b6-2132-440f-9f2f-e327d44004c9 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:13.69053804 +0000 UTC m=+34.244983606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs") pod "network-metrics-daemon-twxjw" (UID: "629f62b6-2132-440f-9f2f-e327d44004c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:57.791338 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:57.791305    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-758m8\" (UniqueName: \"kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8\") pod \"network-check-target-v8vbc\" (UID: \"4452f914-4de0-4fbb-bbfe-237b2cc85d2c\") " pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:57.791483 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:57.791441    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:57.791483 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:57.791455    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:57.791483 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:57.791464    2566 projected.go:194] Error preparing data for projected volume kube-api-access-758m8 for pod openshift-network-diagnostics/network-check-target-v8vbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:57.791609 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:57.791511    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8 podName:4452f914-4de0-4fbb-bbfe-237b2cc85d2c nodeName:}" failed. No retries permitted until 2026-05-20 13:51:13.791498672 +0000 UTC m=+34.345944224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-758m8" (UniqueName: "kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8") pod "network-check-target-v8vbc" (UID: "4452f914-4de0-4fbb-bbfe-237b2cc85d2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:57.987213 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:57.987131    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:57.987353 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:57.987272    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:50:57.987353 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:57.987291    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:57.987468 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:57.987384    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:50:59.987976 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:59.987942    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:50:59.988434 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:50:59.987985    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:50:59.988434 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:59.988049    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:50:59.988434 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:50:59.988174    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:51:01.068111 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.067760    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf" event={"ID":"85434994-53d3-48f0-beaf-f4b9e3ecdeb7","Type":"ContainerStarted","Data":"144b37279dee184b51fa0b77ae970ab607b08c0d045acd53415fd02c66324d31"}
May 20 13:51:01.068729 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.068130    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf" event={"ID":"85434994-53d3-48f0-beaf-f4b9e3ecdeb7","Type":"ContainerStarted","Data":"babf607d154dbef92a0dbf1a8513a98baaffe7e2367edba6a83243c4a8219cd3"}
May 20 13:51:01.068729 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.068152    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf" event={"ID":"85434994-53d3-48f0-beaf-f4b9e3ecdeb7","Type":"ContainerStarted","Data":"d70b29d1e139ceaca6176cc4ea58a373e748de1a59078f7f551ebc336265f852"}
May 20 13:51:01.068868 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.068844    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dxhmp" event={"ID":"1e092c89-8dac-4254-bbf3-8db3d7058f76","Type":"ContainerStarted","Data":"921816da747e7af8891d7bd01f5d75cbbb8d2d64d0b04648b07f48a104822cc5"}
May 20 13:51:01.069976 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.069954    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8qthd" event={"ID":"d87ec45b-44de-41c0-b11f-d927ae91e4d6","Type":"ContainerStarted","Data":"90a5471d7a8a4885ce2832c4b4da2423b03361238f65f301a2fa7b66e7a5b29a"}
May 20 13:51:01.071062 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.071044    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wmtf6" event={"ID":"3864b0b1-b650-47fc-ba2e-cbbff966797a","Type":"ContainerStarted","Data":"2e6b227919261b91e3e0b3f1a5ef72f3f368aa7f368aacd42d43b7ec10ffc0cf"}
May 20 13:51:01.072355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.072305    2566 generic.go:358] "Generic (PLEG): container finished" podID="8959890a-80d9-44d7-a2f1-64b78a493e34" containerID="a1d991f2a1db86f937d0cada9e9d4021c7afcf01faabb2b88ca1018d3f4df290" exitCode=0
May 20 13:51:01.072429 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.072384    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfj8n" event={"ID":"8959890a-80d9-44d7-a2f1-64b78a493e34","Type":"ContainerDied","Data":"a1d991f2a1db86f937d0cada9e9d4021c7afcf01faabb2b88ca1018d3f4df290"}
May 20 13:51:01.073644 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.073610    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v27tj" event={"ID":"db48875c-3279-4fbe-8771-625c9099802f","Type":"ContainerStarted","Data":"596a6a4b447aca2533fe10b02d0ac644f6bb4ebb4c3ee8e67cea475258f90da5"}
May 20 13:51:01.074869 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.074751    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25" event={"ID":"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a","Type":"ContainerStarted","Data":"218be4f723d61ddf5b3d7c9e3304bc7c8e0f72673b4bdf93b83d5ce1f1fb8d9a"}
May 20 13:51:01.075969 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.075923    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dm4bw" event={"ID":"24d3b560-c9fd-48f2-9967-def69e9bb2f8","Type":"ContainerStarted","Data":"4b405d91cf47fbf6d7cbd3f8bb9ab59c5bef2c82376501f2188eb32ae4168185"}
May 20 13:51:01.084823 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.084791    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-71.ec2.internal" podStartSLOduration=20.08478159 podStartE2EDuration="20.08478159s" podCreationTimestamp="2026-05-20 13:50:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:44.048477378 +0000 UTC m=+4.602922953" watchObservedRunningTime="2026-05-20 13:51:01.08478159 +0000 UTC m=+21.639227163"
May 20 13:51:01.085276 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.085249    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-dxhmp" podStartSLOduration=3.071876454 podStartE2EDuration="21.085240434s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:50:42.561262871 +0000 UTC m=+3.115708426" lastFinishedPulling="2026-05-20 13:51:00.57462685 +0000 UTC m=+21.129072406" observedRunningTime="2026-05-20 13:51:01.084725415 +0000 UTC m=+21.639171008" watchObservedRunningTime="2026-05-20 13:51:01.085240434 +0000 UTC m=+21.639686009"
May 20 13:51:01.104835 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.104804    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-dm4bw" podStartSLOduration=3.029575159 podStartE2EDuration="21.104793288s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:50:42.557633854 +0000 UTC m=+3.112079406" lastFinishedPulling="2026-05-20 13:51:00.632851966 +0000 UTC m=+21.187297535" observedRunningTime="2026-05-20 13:51:01.10442071 +0000 UTC m=+21.658866287" watchObservedRunningTime="2026-05-20 13:51:01.104793288 +0000 UTC m=+21.659238862"
May 20 13:51:01.166308 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.166263    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v27tj" podStartSLOduration=10.808575227 podStartE2EDuration="21.166248646s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:50:42.548029909 +0000 UTC m=+3.102475462" lastFinishedPulling="2026-05-20 13:50:52.905703325 +0000 UTC m=+13.460148881" observedRunningTime="2026-05-20 13:51:01.144450588 +0000 UTC m=+21.698896163" watchObservedRunningTime="2026-05-20 13:51:01.166248646 +0000 UTC m=+21.720694214"
May 20 13:51:01.166828 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.166800    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wmtf6" podStartSLOduration=3.143658943 podStartE2EDuration="21.166792324s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:50:42.552849417 +0000 UTC m=+3.107294971" lastFinishedPulling="2026-05-20 13:51:00.575982783 +0000 UTC m=+21.130428352" observedRunningTime="2026-05-20 13:51:01.166640539 +0000 UTC m=+21.721086115" watchObservedRunningTime="2026-05-20 13:51:01.166792324 +0000 UTC m=+21.721237896"
May 20 13:51:01.185999 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.185966    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8qthd" podStartSLOduration=3.167012234 podStartE2EDuration="21.185954451s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:50:42.556005273 +0000 UTC m=+3.110450826" lastFinishedPulling="2026-05-20 13:51:00.57494749 +0000 UTC m=+21.129393043" observedRunningTime="2026-05-20 13:51:01.185529983 +0000 UTC m=+21.739975557" watchObservedRunningTime="2026-05-20 13:51:01.185954451 +0000 UTC m=+21.740400024"
May 20 13:51:01.977785 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.977727    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 20 13:51:01.987371 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.987316    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:01.987490 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:01.987430    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:51:01.987490 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:01.987483    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:51:01.987613 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:01.987573    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:51:02.081232 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:02.081202    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf" event={"ID":"85434994-53d3-48f0-beaf-f4b9e3ecdeb7","Type":"ContainerStarted","Data":"4284800c86258c7a5dde8fe579dd5d1241a35b1535a95e1c9015c58362553796"}
May 20 13:51:02.081589 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:02.081238    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf" event={"ID":"85434994-53d3-48f0-beaf-f4b9e3ecdeb7","Type":"ContainerStarted","Data":"bcb72050fd19ec071955656e7507f835791407d1c7c5a4d224e48592636436b2"}
May 20 13:51:02.081589 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:02.081249    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf" event={"ID":"85434994-53d3-48f0-beaf-f4b9e3ecdeb7","Type":"ContainerStarted","Data":"b61c91cd9233d4ec95be1a9d7b4ae8e119dc8902825574e8ba4c72738a2f9782"}
May 20 13:51:02.082896 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:02.082872    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25" event={"ID":"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a","Type":"ContainerStarted","Data":"e6fd446cf53c7a15dfd9e7bae7ca780ea18e594a9755f5c5e218f0bce7bdd670"}
May 20 13:51:02.084347 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:02.084317    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vpxdx" event={"ID":"2810ffa5-0463-4a15-9729-c1d04fd38658","Type":"ContainerStarted","Data":"cbea17451f4bdee8101b8820edfdf14323c46ee5855eb493d89c186e0b707958"}
May 20 13:51:02.929852 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:02.929743    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T13:51:01.977749297Z","UUID":"4eda096e-33f7-44f2-90cc-d3859050d6a6","Handler":null,"Name":"","Endpoint":""}
May 20 13:51:02.935171 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:02.935144    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 20 13:51:02.935313 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:02.935177    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 20 13:51:03.140368 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:03.140332    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8qthd"
May 20 13:51:03.987423 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:03.987383    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:03.987636 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:03.987383    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:51:03.987636 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:03.987517    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:51:03.987786 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:03.987625    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:51:04.090139 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:04.090102    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25" event={"ID":"ecee6bdf-7a9e-42a4-b757-1fafcc70e88a","Type":"ContainerStarted","Data":"6ab8f59e4b8f3c44cff834280b93c8ed952927f463fbbefbcba6c1fed97c61a3"}
May 20 13:51:04.093356 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:04.093327    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf" event={"ID":"85434994-53d3-48f0-beaf-f4b9e3ecdeb7","Type":"ContainerStarted","Data":"de3d8275556a4dc0882fa70319f22aeae7f9cffd5840054a3e0f792c800006b2"}
May 20 13:51:04.107463 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:04.107420    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vpxdx" podStartSLOduration=6.080667534 podStartE2EDuration="24.107406643s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:50:42.547944507 +0000 UTC m=+3.102390060" lastFinishedPulling="2026-05-20 13:51:00.574683603 +0000 UTC m=+21.129129169" observedRunningTime="2026-05-20 13:51:03.106587218 +0000 UTC m=+23.661032794" watchObservedRunningTime="2026-05-20 13:51:04.107406643 +0000 UTC m=+24.661852217"
May 20 13:51:04.107775 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:04.107745    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnv25" podStartSLOduration=3.592892698 podStartE2EDuration="24.107737518s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:50:42.559286537 +0000 UTC m=+3.113732089" lastFinishedPulling="2026-05-20 13:51:03.074131355 +0000 UTC m=+23.628576909" observedRunningTime="2026-05-20 13:51:04.107282538 +0000 UTC m=+24.661728113" watchObservedRunningTime="2026-05-20 13:51:04.107737518 +0000 UTC m=+24.662183092"
May 20 13:51:05.439958 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:05.439786    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8qthd"
May 20 13:51:05.440448 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:05.440308    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8qthd"
May 20 13:51:05.987617 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:05.987420    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:51:05.987738 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:05.987421    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:05.987738 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:05.987667    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:51:05.987843 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:05.987781    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:51:06.098686 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:06.098654    2566 generic.go:358] "Generic (PLEG): container finished" podID="8959890a-80d9-44d7-a2f1-64b78a493e34" containerID="20d1b628ead54a7e5f4b876a95e0e03514f8af2d8254654e59c75234ef975762" exitCode=0
May 20 13:51:06.098788 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:06.098705    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfj8n" event={"ID":"8959890a-80d9-44d7-a2f1-64b78a493e34","Type":"ContainerDied","Data":"20d1b628ead54a7e5f4b876a95e0e03514f8af2d8254654e59c75234ef975762"}
May 20 13:51:06.102258 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:06.102235    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf" event={"ID":"85434994-53d3-48f0-beaf-f4b9e3ecdeb7","Type":"ContainerStarted","Data":"30de8827ff159ec6d02c73d67c6f9c29be4d33428c5fcbf72dab01827866e400"}
May 20 13:51:06.102556 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:06.102531    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:51:06.102655 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:06.102564    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:51:06.102655 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:06.102575    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:51:06.103024 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:06.102995    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8qthd"
May 20 13:51:06.118337 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:06.118314    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:51:06.118476 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:06.118397    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:51:06.150589 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:06.150539    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf" podStartSLOduration=8.059160948 podStartE2EDuration="26.150521466s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:50:42.547121265 +0000 UTC m=+3.101566829" lastFinishedPulling="2026-05-20 13:51:00.638481782 +0000 UTC m=+21.192927347" observedRunningTime="2026-05-20 13:51:06.150494449 +0000 UTC m=+26.704940024" watchObservedRunningTime="2026-05-20 13:51:06.150521466 +0000 UTC m=+26.704967035"
May 20 13:51:07.552822 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:07.552753    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v8vbc"]
May 20 13:51:07.553544 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:07.552850    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:07.553544 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:07.552936    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:51:07.556075 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:07.556050    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-twxjw"]
May 20 13:51:07.556199 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:07.556127    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:51:07.556257 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:07.556200    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:51:08.107722 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:08.107687    2566 generic.go:358] "Generic (PLEG): container finished" podID="8959890a-80d9-44d7-a2f1-64b78a493e34" containerID="2d75e5ea4237ad7daeaf018725dabcbc68fb3158e5b6f72d90f8e04930736fb9" exitCode=0
May 20 13:51:08.107907 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:08.107766    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfj8n" event={"ID":"8959890a-80d9-44d7-a2f1-64b78a493e34","Type":"ContainerDied","Data":"2d75e5ea4237ad7daeaf018725dabcbc68fb3158e5b6f72d90f8e04930736fb9"}
May 20 13:51:08.986947 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:08.986874    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:51:08.987273 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:08.986874    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:08.987273 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:08.986981    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:51:08.987273 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:08.987028    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:51:09.111808 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:09.111774    2566 generic.go:358] "Generic (PLEG): container finished" podID="8959890a-80d9-44d7-a2f1-64b78a493e34" containerID="6b221141f40757f26db84547850adcf00c47bc5c2a6cb1bdc2526fde774e3b55" exitCode=0
May 20 13:51:09.111938 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:09.111840    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfj8n" event={"ID":"8959890a-80d9-44d7-a2f1-64b78a493e34","Type":"ContainerDied","Data":"6b221141f40757f26db84547850adcf00c47bc5c2a6cb1bdc2526fde774e3b55"}
May 20 13:51:10.987178 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:10.987138    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:10.987638 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:10.987149    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:51:10.987638 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:10.987262    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:51:10.987638 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:10.987333    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:51:12.987215 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:12.987180    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:12.987861 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:12.987188    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:51:12.987861 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:12.987286    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8vbc" podUID="4452f914-4de0-4fbb-bbfe-237b2cc85d2c"
May 20 13:51:12.987861 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:12.987372    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:51:13.249039 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.248966    2566 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-71.ec2.internal" event="NodeReady"
May 20 13:51:13.249167 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.249097    2566 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 20 13:51:13.295370 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.295338    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pnn8g"]
May 20 13:51:13.301495 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.301466    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g6bws"]
May 20 13:51:13.301659 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.301642    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.304320 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.304285    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 20 13:51:13.304432 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.304393    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jxfgl\""
May 20 13:51:13.304719 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.304699    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:13.304826 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.304722    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 20 13:51:13.308352 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.307579    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 20 13:51:13.308352 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.307721    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vxwgr\""
May 20 13:51:13.308352 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.307884    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 20 13:51:13.308352 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.307930    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 20 13:51:13.308830 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.308808    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pnn8g"]
May 20 13:51:13.324224 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.323032    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g6bws"]
May 20 13:51:13.405122 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.405091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a270e1c-51a9-4ae0-83a1-71ec680550ec-tmp-dir\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.405316 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.405132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttthp\" (UniqueName: \"kubernetes.io/projected/5a270e1c-51a9-4ae0-83a1-71ec680550ec-kube-api-access-ttthp\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.405316 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.405164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97n6r\" (UniqueName: \"kubernetes.io/projected/e6d66e5d-0d38-4279-bb66-2177165a5342-kube-api-access-97n6r\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:13.405316 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.405213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.405316 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.405235    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:13.405316 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.405296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a270e1c-51a9-4ae0-83a1-71ec680550ec-config-volume\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.506478 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.506392    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a270e1c-51a9-4ae0-83a1-71ec680550ec-tmp-dir\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.506478 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.506439    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ttthp\" (UniqueName: \"kubernetes.io/projected/5a270e1c-51a9-4ae0-83a1-71ec680550ec-kube-api-access-ttthp\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.506478 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.506461    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-97n6r\" (UniqueName: \"kubernetes.io/projected/e6d66e5d-0d38-4279-bb66-2177165a5342-kube-api-access-97n6r\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:13.506735 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.506622    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.506735 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.506657    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:13.506735 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.506703    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a270e1c-51a9-4ae0-83a1-71ec680550ec-config-volume\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.506881 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:13.506800    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:13.506881 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:13.506811    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:13.506881 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.506803    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a270e1c-51a9-4ae0-83a1-71ec680550ec-tmp-dir\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.507018 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:13.506884    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert podName:e6d66e5d-0d38-4279-bb66-2177165a5342 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:14.006863202 +0000 UTC m=+34.561308766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert") pod "ingress-canary-g6bws" (UID: "e6d66e5d-0d38-4279-bb66-2177165a5342") : secret "canary-serving-cert" not found
May 20 13:51:13.507018 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:13.506904    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls podName:5a270e1c-51a9-4ae0-83a1-71ec680550ec nodeName:}" failed. No retries permitted until 2026-05-20 13:51:14.006893219 +0000 UTC m=+34.561338772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls") pod "dns-default-pnn8g" (UID: "5a270e1c-51a9-4ae0-83a1-71ec680550ec") : secret "dns-default-metrics-tls" not found
May 20 13:51:13.507180 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.507156    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a270e1c-51a9-4ae0-83a1-71ec680550ec-config-volume\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.518139 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.518114    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttthp\" (UniqueName: \"kubernetes.io/projected/5a270e1c-51a9-4ae0-83a1-71ec680550ec-kube-api-access-ttthp\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:13.518263 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.518139    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97n6r\" (UniqueName: \"kubernetes.io/projected/e6d66e5d-0d38-4279-bb66-2177165a5342-kube-api-access-97n6r\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:13.708270 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.708231    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:51:13.708454 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:13.708387    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:51:13.708514 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:13.708472    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs podName:629f62b6-2132-440f-9f2f-e327d44004c9 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:45.70845691 +0000 UTC m=+66.262902466 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs") pod "network-metrics-daemon-twxjw" (UID: "629f62b6-2132-440f-9f2f-e327d44004c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:51:13.809195 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:13.809112    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-758m8\" (UniqueName: \"kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8\") pod \"network-check-target-v8vbc\" (UID: \"4452f914-4de0-4fbb-bbfe-237b2cc85d2c\") " pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:13.809337 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:13.809285    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:51:13.809337 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:13.809307    2566 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:51:13.809337 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:13.809319    2566 projected.go:194] Error preparing data for projected volume kube-api-access-758m8 for pod openshift-network-diagnostics/network-check-target-v8vbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:51:13.809453 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:13.809381    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8 podName:4452f914-4de0-4fbb-bbfe-237b2cc85d2c nodeName:}" failed. No retries permitted until 2026-05-20 13:51:45.809361444 +0000 UTC m=+66.363807019 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-758m8" (UniqueName: "kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8") pod "network-check-target-v8vbc" (UID: "4452f914-4de0-4fbb-bbfe-237b2cc85d2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:51:14.010993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:14.010961    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:14.010993 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:14.010997    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:14.011756 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:14.011109    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:14.011756 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:14.011117    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:14.011756 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:14.011172    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert podName:e6d66e5d-0d38-4279-bb66-2177165a5342 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:15.011152398 +0000 UTC m=+35.565597974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert") pod "ingress-canary-g6bws" (UID: "e6d66e5d-0d38-4279-bb66-2177165a5342") : secret "canary-serving-cert" not found
May 20 13:51:14.011756 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:14.011186    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls podName:5a270e1c-51a9-4ae0-83a1-71ec680550ec nodeName:}" failed. No retries permitted until 2026-05-20 13:51:15.011180164 +0000 UTC m=+35.565625715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls") pod "dns-default-pnn8g" (UID: "5a270e1c-51a9-4ae0-83a1-71ec680550ec") : secret "dns-default-metrics-tls" not found
May 20 13:51:14.987411 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:14.987387    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:51:14.987500 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:14.987387    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:14.991750 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:14.991730    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 13:51:14.991861 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:14.991753    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtwjf\""
May 20 13:51:14.991861 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:14.991790    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 13:51:14.991861 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:14.991793    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vwf8m\""
May 20 13:51:14.991861 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:14.991849    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 13:51:15.018077 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:15.017982    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:15.018077 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:15.018008    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:15.018456 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:15.018132    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:15.018456 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:15.018178    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:15.018456 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:15.018199    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls podName:5a270e1c-51a9-4ae0-83a1-71ec680550ec nodeName:}" failed. No retries permitted until 2026-05-20 13:51:17.018182717 +0000 UTC m=+37.572628270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls") pod "dns-default-pnn8g" (UID: "5a270e1c-51a9-4ae0-83a1-71ec680550ec") : secret "dns-default-metrics-tls" not found
May 20 13:51:15.018456 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:15.018225    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert podName:e6d66e5d-0d38-4279-bb66-2177165a5342 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:17.018213506 +0000 UTC m=+37.572659057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert") pod "ingress-canary-g6bws" (UID: "e6d66e5d-0d38-4279-bb66-2177165a5342") : secret "canary-serving-cert" not found
May 20 13:51:15.125537 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:15.125509    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfj8n" event={"ID":"8959890a-80d9-44d7-a2f1-64b78a493e34","Type":"ContainerStarted","Data":"03b510cae67cd73d83fd8f82b0998847e69ebae5856e73dbb7bd787c1d3c91c9"}
May 20 13:51:16.129004 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:16.128974    2566 generic.go:358] "Generic (PLEG): container finished" podID="8959890a-80d9-44d7-a2f1-64b78a493e34" containerID="03b510cae67cd73d83fd8f82b0998847e69ebae5856e73dbb7bd787c1d3c91c9" exitCode=0
May 20 13:51:16.129429 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:16.129023    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfj8n" event={"ID":"8959890a-80d9-44d7-a2f1-64b78a493e34","Type":"ContainerDied","Data":"03b510cae67cd73d83fd8f82b0998847e69ebae5856e73dbb7bd787c1d3c91c9"}
May 20 13:51:17.032154 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:17.031933    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:17.032316 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:17.032169    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:17.032316 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:17.032077    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:17.032316 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:17.032240    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:17.032316 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:17.032287    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls podName:5a270e1c-51a9-4ae0-83a1-71ec680550ec nodeName:}" failed. No retries permitted until 2026-05-20 13:51:21.032271681 +0000 UTC m=+41.586717252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls") pod "dns-default-pnn8g" (UID: "5a270e1c-51a9-4ae0-83a1-71ec680550ec") : secret "dns-default-metrics-tls" not found
May 20 13:51:17.032316 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:17.032303    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert podName:e6d66e5d-0d38-4279-bb66-2177165a5342 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:21.032296006 +0000 UTC m=+41.586741557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert") pod "ingress-canary-g6bws" (UID: "e6d66e5d-0d38-4279-bb66-2177165a5342") : secret "canary-serving-cert" not found
May 20 13:51:17.132958 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:17.132926    2566 generic.go:358] "Generic (PLEG): container finished" podID="8959890a-80d9-44d7-a2f1-64b78a493e34" containerID="f0fcd5ad3e450cba81d60001d93d555f91625937edc49d1af3c44a8e387822b8" exitCode=0
May 20 13:51:17.133377 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:17.132984    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfj8n" event={"ID":"8959890a-80d9-44d7-a2f1-64b78a493e34","Type":"ContainerDied","Data":"f0fcd5ad3e450cba81d60001d93d555f91625937edc49d1af3c44a8e387822b8"}
May 20 13:51:18.138092 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:18.138058    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfj8n" event={"ID":"8959890a-80d9-44d7-a2f1-64b78a493e34","Type":"ContainerStarted","Data":"c28b07e23fd9e4254948e0663ecde4cc3dafb8aee3f9820d0bbb86f44ff0a1c5"}
May 20 13:51:18.161431 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:18.161388    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gfj8n" podStartSLOduration=5.726039278 podStartE2EDuration="38.161376577s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:50:42.552459577 +0000 UTC m=+3.106905130" lastFinishedPulling="2026-05-20 13:51:14.987796862 +0000 UTC m=+35.542242429" observedRunningTime="2026-05-20 13:51:18.160183175 +0000 UTC m=+38.714628749" watchObservedRunningTime="2026-05-20 13:51:18.161376577 +0000 UTC m=+38.715822152"
May 20 13:51:21.059525 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:21.059487    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:21.059894 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:21.059539    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:21.059894 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:21.059643    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:21.059894 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:21.059650    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:21.059894 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:21.059718    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls podName:5a270e1c-51a9-4ae0-83a1-71ec680550ec nodeName:}" failed. No retries permitted until 2026-05-20 13:51:29.059700884 +0000 UTC m=+49.614146436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls") pod "dns-default-pnn8g" (UID: "5a270e1c-51a9-4ae0-83a1-71ec680550ec") : secret "dns-default-metrics-tls" not found
May 20 13:51:21.059894 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:21.059734    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert podName:e6d66e5d-0d38-4279-bb66-2177165a5342 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:29.059726287 +0000 UTC m=+49.614171838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert") pod "ingress-canary-g6bws" (UID: "e6d66e5d-0d38-4279-bb66-2177165a5342") : secret "canary-serving-cert" not found
May 20 13:51:27.792729 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.792697    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"]
May 20 13:51:27.812619 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.812595    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"]
May 20 13:51:27.812764 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.812710    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:27.815404 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.815378    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 20 13:51:27.815524 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.815378    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 20 13:51:27.815524 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.815383    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 20 13:51:27.816463 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.816447    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 20 13:51:27.822134 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.822115    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"]
May 20 13:51:27.833578 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.833557    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:27.834758 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.834732    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"]
May 20 13:51:27.836236 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.836215    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 20 13:51:27.836342 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.836264    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 20 13:51:27.836342 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.836272    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 20 13:51:27.836342 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.836272    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 20 13:51:27.909619 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.909601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4894442b-700e-4e3d-ab06-699246e61225-tmp\") pod \"klusterlet-addon-workmgr-7994874b69-82rdt\" (UID: \"4894442b-700e-4e3d-ab06-699246e61225\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:27.909729 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.909656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgt4c\" (UniqueName: \"kubernetes.io/projected/4894442b-700e-4e3d-ab06-699246e61225-kube-api-access-cgt4c\") pod \"klusterlet-addon-workmgr-7994874b69-82rdt\" (UID: \"4894442b-700e-4e3d-ab06-699246e61225\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:27.909779 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:27.909757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4894442b-700e-4e3d-ab06-699246e61225-klusterlet-config\") pod \"klusterlet-addon-workmgr-7994874b69-82rdt\" (UID: \"4894442b-700e-4e3d-ab06-699246e61225\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:28.010795 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.010773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-hub\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.010876 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.010817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94sqz\" (UniqueName: \"kubernetes.io/projected/9a968e98-7e11-43e1-ba33-6819b9ed5b93-kube-api-access-94sqz\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.010876 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.010839    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4894442b-700e-4e3d-ab06-699246e61225-klusterlet-config\") pod \"klusterlet-addon-workmgr-7994874b69-82rdt\" (UID: \"4894442b-700e-4e3d-ab06-699246e61225\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:28.010991 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.010884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.010991 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.010918    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4894442b-700e-4e3d-ab06-699246e61225-tmp\") pod \"klusterlet-addon-workmgr-7994874b69-82rdt\" (UID: \"4894442b-700e-4e3d-ab06-699246e61225\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:28.010991 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.010939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-ca\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.010991 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.010984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.011180 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.011077    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cgt4c\" (UniqueName: \"kubernetes.io/projected/4894442b-700e-4e3d-ab06-699246e61225-kube-api-access-cgt4c\") pod \"klusterlet-addon-workmgr-7994874b69-82rdt\" (UID: \"4894442b-700e-4e3d-ab06-699246e61225\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:28.011180 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.011131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/9a968e98-7e11-43e1-ba33-6819b9ed5b93-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.011248 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.011232    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4894442b-700e-4e3d-ab06-699246e61225-tmp\") pod \"klusterlet-addon-workmgr-7994874b69-82rdt\" (UID: \"4894442b-700e-4e3d-ab06-699246e61225\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:28.014129 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.014109    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4894442b-700e-4e3d-ab06-699246e61225-klusterlet-config\") pod \"klusterlet-addon-workmgr-7994874b69-82rdt\" (UID: \"4894442b-700e-4e3d-ab06-699246e61225\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:28.020051 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.020020    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgt4c\" (UniqueName: \"kubernetes.io/projected/4894442b-700e-4e3d-ab06-699246e61225-kube-api-access-cgt4c\") pod \"klusterlet-addon-workmgr-7994874b69-82rdt\" (UID: \"4894442b-700e-4e3d-ab06-699246e61225\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:28.112306 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.112287    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-ca\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.112386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.112334    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.112386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.112376    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/9a968e98-7e11-43e1-ba33-6819b9ed5b93-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.112448 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.112415    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-hub\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.112490 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.112463    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-94sqz\" (UniqueName: \"kubernetes.io/projected/9a968e98-7e11-43e1-ba33-6819b9ed5b93-kube-api-access-94sqz\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.112528 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.112502    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.113200 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.113175    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/9a968e98-7e11-43e1-ba33-6819b9ed5b93-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.114519 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.114499    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-ca\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.114868 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.114850    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.115037 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.115019    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.115133 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.115115    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9a968e98-7e11-43e1-ba33-6819b9ed5b93-hub\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.120627 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.120606    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94sqz\" (UniqueName: \"kubernetes.io/projected/9a968e98-7e11-43e1-ba33-6819b9ed5b93-kube-api-access-94sqz\") pod \"cluster-proxy-proxy-agent-789f6b54df-fdvm4\" (UID: \"9a968e98-7e11-43e1-ba33-6819b9ed5b93\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.121351 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.121327    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:28.158385 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.158363    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:51:28.283437 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.283401    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"]
May 20 13:51:28.288688 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:51:28.288645    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4894442b_700e_4e3d_ab06_699246e61225.slice/crio-1c6d3b4c645737c51dc62687835d13d84cb261ac16cb16e5c3f5839a537d274d WatchSource:0}: Error finding container 1c6d3b4c645737c51dc62687835d13d84cb261ac16cb16e5c3f5839a537d274d: Status 404 returned error can't find the container with id 1c6d3b4c645737c51dc62687835d13d84cb261ac16cb16e5c3f5839a537d274d
May 20 13:51:28.291487 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:28.291433    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"]
May 20 13:51:28.294954 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:51:28.294928    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a968e98_7e11_43e1_ba33_6819b9ed5b93.slice/crio-ea8c2e07715dcf0abe1e98acd9507f825ff5854f30b45de36eb49f6dd2f19fa4 WatchSource:0}: Error finding container ea8c2e07715dcf0abe1e98acd9507f825ff5854f30b45de36eb49f6dd2f19fa4: Status 404 returned error can't find the container with id ea8c2e07715dcf0abe1e98acd9507f825ff5854f30b45de36eb49f6dd2f19fa4
May 20 13:51:29.120240 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:29.120207    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:29.120609 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:29.120243    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:29.120609 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:29.120340    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:29.120609 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:29.120348    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:29.120609 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:29.120391    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert podName:e6d66e5d-0d38-4279-bb66-2177165a5342 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:45.12037814 +0000 UTC m=+65.674823692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert") pod "ingress-canary-g6bws" (UID: "e6d66e5d-0d38-4279-bb66-2177165a5342") : secret "canary-serving-cert" not found
May 20 13:51:29.120609 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:29.120412    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls podName:5a270e1c-51a9-4ae0-83a1-71ec680550ec nodeName:}" failed. No retries permitted until 2026-05-20 13:51:45.120397969 +0000 UTC m=+65.674843521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls") pod "dns-default-pnn8g" (UID: "5a270e1c-51a9-4ae0-83a1-71ec680550ec") : secret "dns-default-metrics-tls" not found
May 20 13:51:29.159167 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:29.159138    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt" event={"ID":"4894442b-700e-4e3d-ab06-699246e61225","Type":"ContainerStarted","Data":"1c6d3b4c645737c51dc62687835d13d84cb261ac16cb16e5c3f5839a537d274d"}
May 20 13:51:29.160165 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:29.160139    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" event={"ID":"9a968e98-7e11-43e1-ba33-6819b9ed5b93","Type":"ContainerStarted","Data":"ea8c2e07715dcf0abe1e98acd9507f825ff5854f30b45de36eb49f6dd2f19fa4"}
May 20 13:51:34.173724 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:34.173689    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt" event={"ID":"4894442b-700e-4e3d-ab06-699246e61225","Type":"ContainerStarted","Data":"1ae2c3b9ba1f2550994f47bf3dd55208ba94ba9a6417dd62b05351bb2ed6292f"}
May 20 13:51:34.174131 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:34.173913    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:34.175095 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:34.175069    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" event={"ID":"9a968e98-7e11-43e1-ba33-6819b9ed5b93","Type":"ContainerStarted","Data":"470a6203cba03c600b679ac1094c60d1b2a9210c00c5e3036ae7d81512970734"}
May 20 13:51:34.175307 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:34.175293    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt"
May 20 13:51:34.191180 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:34.191140    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7994874b69-82rdt" podStartSLOduration=1.666508657 podStartE2EDuration="7.191128092s" podCreationTimestamp="2026-05-20 13:51:27 +0000 UTC" firstStartedPulling="2026-05-20 13:51:28.290551872 +0000 UTC m=+48.844997433" lastFinishedPulling="2026-05-20 13:51:33.815171313 +0000 UTC m=+54.369616868" observedRunningTime="2026-05-20 13:51:34.190468295 +0000 UTC m=+54.744913868" watchObservedRunningTime="2026-05-20 13:51:34.191128092 +0000 UTC m=+54.745573665"
May 20 13:51:37.181734 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:37.181699    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" event={"ID":"9a968e98-7e11-43e1-ba33-6819b9ed5b93","Type":"ContainerStarted","Data":"a53d2c206f6c15d2f4c6a9dcce87e10624ae53798f63e7fb176550e7eed43d2e"}
May 20 13:51:37.182056 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:37.181740    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" event={"ID":"9a968e98-7e11-43e1-ba33-6819b9ed5b93","Type":"ContainerStarted","Data":"d4f12016eef6f264e6e82e205e68349154171fc2205298c9f244660d791d9e90"}
May 20 13:51:37.202217 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:37.202174    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" podStartSLOduration=1.710320461 podStartE2EDuration="10.202161317s" podCreationTimestamp="2026-05-20 13:51:27 +0000 UTC" firstStartedPulling="2026-05-20 13:51:28.29660754 +0000 UTC m=+48.851053093" lastFinishedPulling="2026-05-20 13:51:36.788448383 +0000 UTC m=+57.342893949" observedRunningTime="2026-05-20 13:51:37.200752202 +0000 UTC m=+57.755197776" watchObservedRunningTime="2026-05-20 13:51:37.202161317 +0000 UTC m=+57.756606890"
May 20 13:51:38.128769 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:38.128742    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rdtkf"
May 20 13:51:45.136460 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:45.136427    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:51:45.136460 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:45.136465    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:51:45.136916 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:45.136575    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:45.136916 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:45.136648    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls podName:5a270e1c-51a9-4ae0-83a1-71ec680550ec nodeName:}" failed. No retries permitted until 2026-05-20 13:52:17.13663101 +0000 UTC m=+97.691076562 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls") pod "dns-default-pnn8g" (UID: "5a270e1c-51a9-4ae0-83a1-71ec680550ec") : secret "dns-default-metrics-tls" not found
May 20 13:51:45.136916 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:45.136580    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:45.136916 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:45.136719    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert podName:e6d66e5d-0d38-4279-bb66-2177165a5342 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:17.136708067 +0000 UTC m=+97.691153619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert") pod "ingress-canary-g6bws" (UID: "e6d66e5d-0d38-4279-bb66-2177165a5342") : secret "canary-serving-cert" not found
May 20 13:51:45.740487 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:45.740452    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:51:45.743485 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:45.743465    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 13:51:45.751632 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:45.751613    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 13:51:45.751733 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:51:45.751666    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs podName:629f62b6-2132-440f-9f2f-e327d44004c9 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:49.751652328 +0000 UTC m=+130.306097880 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs") pod "network-metrics-daemon-twxjw" (UID: "629f62b6-2132-440f-9f2f-e327d44004c9") : secret "metrics-daemon-secret" not found
May 20 13:51:45.841558 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:45.841536    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-758m8\" (UniqueName: \"kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8\") pod \"network-check-target-v8vbc\" (UID: \"4452f914-4de0-4fbb-bbfe-237b2cc85d2c\") " pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:45.844338 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:45.844319    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 13:51:45.854823 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:45.854803    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 13:51:45.865214 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:45.865190    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-758m8\" (UniqueName: \"kubernetes.io/projected/4452f914-4de0-4fbb-bbfe-237b2cc85d2c-kube-api-access-758m8\") pod \"network-check-target-v8vbc\" (UID: \"4452f914-4de0-4fbb-bbfe-237b2cc85d2c\") " pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:45.904865 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:45.904844    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtwjf\""
May 20 13:51:45.912929 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:45.912908    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:46.033451 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:46.033381    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v8vbc"]
May 20 13:51:46.036115 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:51:46.036089    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4452f914_4de0_4fbb_bbfe_237b2cc85d2c.slice/crio-ebc3f0d47f5e1c2f0f9f399da6ec71448a885abfc2fc85fe5745e2b19a7be65a WatchSource:0}: Error finding container ebc3f0d47f5e1c2f0f9f399da6ec71448a885abfc2fc85fe5745e2b19a7be65a: Status 404 returned error can't find the container with id ebc3f0d47f5e1c2f0f9f399da6ec71448a885abfc2fc85fe5745e2b19a7be65a
May 20 13:51:46.198107 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:46.198071    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v8vbc" event={"ID":"4452f914-4de0-4fbb-bbfe-237b2cc85d2c","Type":"ContainerStarted","Data":"ebc3f0d47f5e1c2f0f9f399da6ec71448a885abfc2fc85fe5745e2b19a7be65a"}
May 20 13:51:49.207311 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:49.207248    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v8vbc" event={"ID":"4452f914-4de0-4fbb-bbfe-237b2cc85d2c","Type":"ContainerStarted","Data":"d8ab9f14160d5876fdd92386ea4212c731a40fc6b8802dd6e4bc3d363b9126f1"}
May 20 13:51:49.207597 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:49.207443    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:51:49.224960 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:51:49.224924    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v8vbc" podStartSLOduration=66.32762299 podStartE2EDuration="1m9.224913089s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:51:46.037789523 +0000 UTC m=+66.592235075" lastFinishedPulling="2026-05-20 13:51:48.935079622 +0000 UTC m=+69.489525174" observedRunningTime="2026-05-20 13:51:49.224813432 +0000 UTC m=+69.779259005" watchObservedRunningTime="2026-05-20 13:51:49.224913089 +0000 UTC m=+69.779358663"
May 20 13:52:17.143649 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:52:17.143599    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:52:17.143649 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:52:17.143654    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:52:17.144100 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:52:17.143787    2566 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:52:17.144100 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:52:17.143794    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:52:17.144100 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:52:17.143872    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert podName:e6d66e5d-0d38-4279-bb66-2177165a5342 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:21.143856892 +0000 UTC m=+161.698302444 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert") pod "ingress-canary-g6bws" (UID: "e6d66e5d-0d38-4279-bb66-2177165a5342") : secret "canary-serving-cert" not found
May 20 13:52:17.144100 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:52:17.143886    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls podName:5a270e1c-51a9-4ae0-83a1-71ec680550ec nodeName:}" failed. No retries permitted until 2026-05-20 13:53:21.143880471 +0000 UTC m=+161.698326023 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls") pod "dns-default-pnn8g" (UID: "5a270e1c-51a9-4ae0-83a1-71ec680550ec") : secret "dns-default-metrics-tls" not found
May 20 13:52:20.211525 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:52:20.211492    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v8vbc"
May 20 13:52:49.753950 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:52:49.753914    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:52:49.754391 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:52:49.754030    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 13:52:49.754391 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:52:49.754088    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs podName:629f62b6-2132-440f-9f2f-e327d44004c9 nodeName:}" failed. No retries permitted until 2026-05-20 13:54:51.754072996 +0000 UTC m=+252.308518547 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs") pod "network-metrics-daemon-twxjw" (UID: "629f62b6-2132-440f-9f2f-e327d44004c9") : secret "metrics-daemon-secret" not found
May 20 13:52:54.347278 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:52:54.347251    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dxhmp_1e092c89-8dac-4254-bbf3-8db3d7058f76/dns-node-resolver/0.log"
May 20 13:52:55.547472 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:52:55.547448    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v27tj_db48875c-3279-4fbe-8771-625c9099802f/node-ca/0.log"
May 20 13:53:16.335967 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:53:16.335926    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pnn8g" podUID="5a270e1c-51a9-4ae0-83a1-71ec680550ec"
May 20 13:53:16.343101 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:53:16.343068    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-g6bws" podUID="e6d66e5d-0d38-4279-bb66-2177165a5342"
May 20 13:53:16.361440 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:16.361413    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:53:16.361556 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:16.361414    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnn8g"
May 20 13:53:17.178948 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.178917    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75bbd8854-4b256"]
May 20 13:53:17.180923 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.180908    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.185310 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.185290    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 20 13:53:17.188560 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.188544    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gr7x7\""
May 20 13:53:17.200804 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.200783    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 20 13:53:17.216330 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.216303    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75bbd8854-4b256"]
May 20 13:53:17.216624 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.216609    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 20 13:53:17.216861 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.216845    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 20 13:53:17.230328 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.230304    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kh7mn"]
May 20 13:53:17.232746 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.232727    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.241573 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.241555    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 20 13:53:17.241650 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.241592    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tgnrd\""
May 20 13:53:17.242776 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.242758    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 20 13:53:17.245834 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.245816    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 20 13:53:17.245904 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.245866    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 20 13:53:17.265785 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.265754    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kh7mn"]
May 20 13:53:17.340578 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86144d59-02a9-4ba9-b088-ca38960d39b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.340578 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a20092fe-2492-4f66-ad11-d115e5783408-registry-tls\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.341011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86144d59-02a9-4ba9-b088-ca38960d39b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.341011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mvh\" (UniqueName: \"kubernetes.io/projected/a20092fe-2492-4f66-ad11-d115e5783408-kube-api-access-r7mvh\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.341011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a20092fe-2492-4f66-ad11-d115e5783408-trusted-ca\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.341011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmd7v\" (UniqueName: \"kubernetes.io/projected/86144d59-02a9-4ba9-b088-ca38960d39b1-kube-api-access-cmd7v\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.341011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a20092fe-2492-4f66-ad11-d115e5783408-registry-certificates\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.341011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a20092fe-2492-4f66-ad11-d115e5783408-image-registry-private-configuration\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.341011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86144d59-02a9-4ba9-b088-ca38960d39b1-crio-socket\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.341011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a20092fe-2492-4f66-ad11-d115e5783408-ca-trust-extracted\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.341011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.340954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86144d59-02a9-4ba9-b088-ca38960d39b1-data-volume\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.341011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.341000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a20092fe-2492-4f66-ad11-d115e5783408-installation-pull-secrets\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.341305 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.341020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a20092fe-2492-4f66-ad11-d115e5783408-bound-sa-token\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.442399 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442316    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cmd7v\" (UniqueName: \"kubernetes.io/projected/86144d59-02a9-4ba9-b088-ca38960d39b1-kube-api-access-cmd7v\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.442399 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442352    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a20092fe-2492-4f66-ad11-d115e5783408-registry-certificates\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.442594 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442403    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a20092fe-2492-4f66-ad11-d115e5783408-image-registry-private-configuration\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.442594 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442425    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86144d59-02a9-4ba9-b088-ca38960d39b1-crio-socket\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.442594 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442443    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a20092fe-2492-4f66-ad11-d115e5783408-ca-trust-extracted\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.442594 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442526    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86144d59-02a9-4ba9-b088-ca38960d39b1-crio-socket\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.442594 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442577    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86144d59-02a9-4ba9-b088-ca38960d39b1-data-volume\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.442862 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442605    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a20092fe-2492-4f66-ad11-d115e5783408-installation-pull-secrets\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.442862 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442627    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a20092fe-2492-4f66-ad11-d115e5783408-bound-sa-token\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.442862 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442654    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86144d59-02a9-4ba9-b088-ca38960d39b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.442862 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442698    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a20092fe-2492-4f66-ad11-d115e5783408-registry-tls\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.442862 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442744    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86144d59-02a9-4ba9-b088-ca38960d39b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.442862 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442770    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mvh\" (UniqueName: \"kubernetes.io/projected/a20092fe-2492-4f66-ad11-d115e5783408-kube-api-access-r7mvh\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.442862 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442800    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a20092fe-2492-4f66-ad11-d115e5783408-trusted-ca\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.443197 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442867    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a20092fe-2492-4f66-ad11-d115e5783408-ca-trust-extracted\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.443197 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.442965    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86144d59-02a9-4ba9-b088-ca38960d39b1-data-volume\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.443313 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.443292    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86144d59-02a9-4ba9-b088-ca38960d39b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.443459 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.443439    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a20092fe-2492-4f66-ad11-d115e5783408-registry-certificates\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.443773 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.443746    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a20092fe-2492-4f66-ad11-d115e5783408-trusted-ca\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.445096 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.445073    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a20092fe-2492-4f66-ad11-d115e5783408-installation-pull-secrets\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.445365 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.445350    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a20092fe-2492-4f66-ad11-d115e5783408-registry-tls\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.445452 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.445433    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86144d59-02a9-4ba9-b088-ca38960d39b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.445488 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.445441    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a20092fe-2492-4f66-ad11-d115e5783408-image-registry-private-configuration\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.457904 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.457874    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a20092fe-2492-4f66-ad11-d115e5783408-bound-sa-token\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.458054 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.458037    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mvh\" (UniqueName: \"kubernetes.io/projected/a20092fe-2492-4f66-ad11-d115e5783408-kube-api-access-r7mvh\") pod \"image-registry-75bbd8854-4b256\" (UID: \"a20092fe-2492-4f66-ad11-d115e5783408\") " pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.459526 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.459505    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmd7v\" (UniqueName: \"kubernetes.io/projected/86144d59-02a9-4ba9-b088-ca38960d39b1-kube-api-access-cmd7v\") pod \"insights-runtime-extractor-kh7mn\" (UID: \"86144d59-02a9-4ba9-b088-ca38960d39b1\") " pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.488910 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.488877    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:17.541297 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.541263    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kh7mn"
May 20 13:53:17.639810 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.639778    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75bbd8854-4b256"]
May 20 13:53:17.642633 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:53:17.642604    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20092fe_2492_4f66_ad11_d115e5783408.slice/crio-05c887def31ceee37f44972fafdd0e53bc4ca109dea6b2e6f22b7965ba0fc31a WatchSource:0}: Error finding container 05c887def31ceee37f44972fafdd0e53bc4ca109dea6b2e6f22b7965ba0fc31a: Status 404 returned error can't find the container with id 05c887def31ceee37f44972fafdd0e53bc4ca109dea6b2e6f22b7965ba0fc31a
May 20 13:53:17.680130 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:17.680102    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kh7mn"]
May 20 13:53:17.684234 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:53:17.684210    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86144d59_02a9_4ba9_b088_ca38960d39b1.slice/crio-6eb57bbf86ca9b239c325ea80b11221348d05481b7153fdee5fda6dc4d190a7e WatchSource:0}: Error finding container 6eb57bbf86ca9b239c325ea80b11221348d05481b7153fdee5fda6dc4d190a7e: Status 404 returned error can't find the container with id 6eb57bbf86ca9b239c325ea80b11221348d05481b7153fdee5fda6dc4d190a7e
May 20 13:53:17.997106 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:53:17.997026    2566 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-twxjw" podUID="629f62b6-2132-440f-9f2f-e327d44004c9"
May 20 13:53:18.370352 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:18.370316    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75bbd8854-4b256" event={"ID":"a20092fe-2492-4f66-ad11-d115e5783408","Type":"ContainerStarted","Data":"8c6aa1ecfb63f370f3c04209a44786ba8cca1a102e088e6906f2818acf583327"}
May 20 13:53:18.370703 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:18.370357    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75bbd8854-4b256" event={"ID":"a20092fe-2492-4f66-ad11-d115e5783408","Type":"ContainerStarted","Data":"05c887def31ceee37f44972fafdd0e53bc4ca109dea6b2e6f22b7965ba0fc31a"}
May 20 13:53:18.370703 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:18.370655    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:18.371621 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:18.371584    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kh7mn" event={"ID":"86144d59-02a9-4ba9-b088-ca38960d39b1","Type":"ContainerStarted","Data":"b105f52c5fe6453ea7dab88c2c9ec23754eda153049589f613cee9a9e490a731"}
May 20 13:53:18.371621 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:18.371620    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kh7mn" event={"ID":"86144d59-02a9-4ba9-b088-ca38960d39b1","Type":"ContainerStarted","Data":"6eb57bbf86ca9b239c325ea80b11221348d05481b7153fdee5fda6dc4d190a7e"}
May 20 13:53:18.394561 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:18.394519    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75bbd8854-4b256" podStartSLOduration=1.3945057 podStartE2EDuration="1.3945057s" podCreationTimestamp="2026-05-20 13:53:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:53:18.393587314 +0000 UTC m=+158.948032889" watchObservedRunningTime="2026-05-20 13:53:18.3945057 +0000 UTC m=+158.948951274"
May 20 13:53:19.378357 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:19.378320    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kh7mn" event={"ID":"86144d59-02a9-4ba9-b088-ca38960d39b1","Type":"ContainerStarted","Data":"d2024a783bf54174f33aab6398876fe04b2afb340fe442002ac8aa8621d5c1a8"}
May 20 13:53:20.381728 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:20.381691    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kh7mn" event={"ID":"86144d59-02a9-4ba9-b088-ca38960d39b1","Type":"ContainerStarted","Data":"5fdefc87ffa0d40a33f73da13528c254f2ecf5c5f9344ef621ecec603d496df9"}
May 20 13:53:20.401280 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:20.401232    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kh7mn" podStartSLOduration=1.497000313 podStartE2EDuration="3.40119465s" podCreationTimestamp="2026-05-20 13:53:17 +0000 UTC" firstStartedPulling="2026-05-20 13:53:17.722758909 +0000 UTC m=+158.277204461" lastFinishedPulling="2026-05-20 13:53:19.626953229 +0000 UTC m=+160.181398798" observedRunningTime="2026-05-20 13:53:20.400700184 +0000 UTC m=+160.955145757" watchObservedRunningTime="2026-05-20 13:53:20.40119465 +0000 UTC m=+160.955640223"
May 20 13:53:21.172183 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:21.172147    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:53:21.172183 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:21.172181    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:53:21.174378 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:21.174344    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a270e1c-51a9-4ae0-83a1-71ec680550ec-metrics-tls\") pod \"dns-default-pnn8g\" (UID: \"5a270e1c-51a9-4ae0-83a1-71ec680550ec\") " pod="openshift-dns/dns-default-pnn8g"
May 20 13:53:21.174528 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:21.174394    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6d66e5d-0d38-4279-bb66-2177165a5342-cert\") pod \"ingress-canary-g6bws\" (UID: \"e6d66e5d-0d38-4279-bb66-2177165a5342\") " pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:53:21.465181 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:21.465103    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jxfgl\""
May 20 13:53:21.466174 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:21.466144    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vxwgr\""
May 20 13:53:21.473207 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:21.473191    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g6bws"
May 20 13:53:21.473304 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:21.473218    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnn8g"
May 20 13:53:21.607560 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:21.604799    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g6bws"]
May 20 13:53:21.609191 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:53:21.609161    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d66e5d_0d38_4279_bb66_2177165a5342.slice/crio-8e471140ea8f40c9c46e034f5b182d2f11a3344d23dbade082612942899ad030 WatchSource:0}: Error finding container 8e471140ea8f40c9c46e034f5b182d2f11a3344d23dbade082612942899ad030: Status 404 returned error can't find the container with id 8e471140ea8f40c9c46e034f5b182d2f11a3344d23dbade082612942899ad030
May 20 13:53:21.618011 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:21.617989    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pnn8g"]
May 20 13:53:21.620865 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:53:21.620833    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a270e1c_51a9_4ae0_83a1_71ec680550ec.slice/crio-15c946302146f39661d1ae886c70f8fc5e6defa29e6b205b3b283b9bb1fa5c68 WatchSource:0}: Error finding container 15c946302146f39661d1ae886c70f8fc5e6defa29e6b205b3b283b9bb1fa5c68: Status 404 returned error can't find the container with id 15c946302146f39661d1ae886c70f8fc5e6defa29e6b205b3b283b9bb1fa5c68
May 20 13:53:22.388468 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:22.388424    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g6bws" event={"ID":"e6d66e5d-0d38-4279-bb66-2177165a5342","Type":"ContainerStarted","Data":"8e471140ea8f40c9c46e034f5b182d2f11a3344d23dbade082612942899ad030"}
May 20 13:53:22.389611 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:22.389567    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnn8g" event={"ID":"5a270e1c-51a9-4ae0-83a1-71ec680550ec","Type":"ContainerStarted","Data":"15c946302146f39661d1ae886c70f8fc5e6defa29e6b205b3b283b9bb1fa5c68"}
May 20 13:53:24.397600 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:24.397563    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnn8g" event={"ID":"5a270e1c-51a9-4ae0-83a1-71ec680550ec","Type":"ContainerStarted","Data":"b776f5e539f58e27c17d1cf8db5f1c1071e8ef712e8f5364a4eb9d7fff651b27"}
May 20 13:53:24.398049 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:24.397608    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnn8g" event={"ID":"5a270e1c-51a9-4ae0-83a1-71ec680550ec","Type":"ContainerStarted","Data":"f9fcc868ad3af3ccf1b1333557ad8035d74d2ce5a375451a38477be239708140"}
May 20 13:53:24.398049 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:24.397665    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pnn8g"
May 20 13:53:24.398940 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:24.398911    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g6bws" event={"ID":"e6d66e5d-0d38-4279-bb66-2177165a5342","Type":"ContainerStarted","Data":"ed4624a7b4d6250ace6f53059198ceea102754477ff2d27ed4444878ed534d51"}
May 20 13:53:24.414642 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:24.414602    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-pnn8g" podStartSLOduration=129.596148882 podStartE2EDuration="2m11.414591908s" podCreationTimestamp="2026-05-20 13:51:13 +0000 UTC" firstStartedPulling="2026-05-20 13:53:21.622426263 +0000 UTC m=+162.176871815" lastFinishedPulling="2026-05-20 13:53:23.440869287 +0000 UTC m=+163.995314841" observedRunningTime="2026-05-20 13:53:24.413452604 +0000 UTC m=+164.967898179" watchObservedRunningTime="2026-05-20 13:53:24.414591908 +0000 UTC m=+164.969037482"
May 20 13:53:24.428714 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:24.428654    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g6bws" podStartSLOduration=129.596515738 podStartE2EDuration="2m11.428643386s" podCreationTimestamp="2026-05-20 13:51:13 +0000 UTC" firstStartedPulling="2026-05-20 13:53:21.611057238 +0000 UTC m=+162.165502790" lastFinishedPulling="2026-05-20 13:53:23.443184878 +0000 UTC m=+163.997630438" observedRunningTime="2026-05-20 13:53:24.428188773 +0000 UTC m=+164.982634347" watchObservedRunningTime="2026-05-20 13:53:24.428643386 +0000 UTC m=+164.983088959"
May 20 13:53:29.038182 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.038095    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-47rm7"]
May 20 13:53:29.040987 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.040964    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.043478 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.043454    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 20 13:53:29.043599 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.043484    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 20 13:53:29.043772 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.043756    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 20 13:53:29.044767 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.044746    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 13:53:29.044878 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.044746    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vz65g\""
May 20 13:53:29.044878 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.044781    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 13:53:29.044975 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.044880    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 13:53:29.133268 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.133236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrsf\" (UniqueName: \"kubernetes.io/projected/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-kube-api-access-4xrsf\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.133268 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.133269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.133480 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.133289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-wtmp\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.133480 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.133330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-tls\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.133586 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.133494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-sys\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.133586 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.133535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.133697 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.133596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-root\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.133697 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.133628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-metrics-client-ca\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.133697 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.133690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-textfile\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234266 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234239    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-tls\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234407 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234272    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-sys\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234407 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234291    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234407 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234318    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-root\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234407 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234339    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-metrics-client-ca\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234407 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234349    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-sys\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234407 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234369    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-textfile\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234742 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:53:29.234432    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 20 13:53:29.234742 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:53:29.234570    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-tls podName:8f38fe80-2cd1-4b05-8934-6743f3dac8d8 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:29.734550411 +0000 UTC m=+170.288995963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-tls") pod "node-exporter-47rm7" (UID: "8f38fe80-2cd1-4b05-8934-6743f3dac8d8") : secret "node-exporter-tls" not found
May 20 13:53:29.234742 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234732    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-root\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234925 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234876    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrsf\" (UniqueName: \"kubernetes.io/projected/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-kube-api-access-4xrsf\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234982 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234913    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.234982 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234958    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-wtmp\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.235085 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.234990    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-textfile\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.235140 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.235097    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-wtmp\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.235140 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.235113    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.235259 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.235134    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-metrics-client-ca\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.237283 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.237262    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.245372 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.245354    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrsf\" (UniqueName: \"kubernetes.io/projected/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-kube-api-access-4xrsf\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.739528 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:29.739489    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-tls\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:29.739723 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:53:29.739636    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 20 13:53:29.739723 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:53:29.739721    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-tls podName:8f38fe80-2cd1-4b05-8934-6743f3dac8d8 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:30.739705117 +0000 UTC m=+171.294150668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-tls") pod "node-exporter-47rm7" (UID: "8f38fe80-2cd1-4b05-8934-6743f3dac8d8") : secret "node-exporter-tls" not found
May 20 13:53:30.747971 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:30.747936    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-tls\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:30.750213 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:30.750194    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f38fe80-2cd1-4b05-8934-6743f3dac8d8-node-exporter-tls\") pod \"node-exporter-47rm7\" (UID: \"8f38fe80-2cd1-4b05-8934-6743f3dac8d8\") " pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:30.849431 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:30.849388    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-47rm7"
May 20 13:53:30.857539 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:53:30.857511    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f38fe80_2cd1_4b05_8934_6743f3dac8d8.slice/crio-15bc238babb33b75a0382f8870cd8e5f8f9c1d06691a11f68e8efa1ae15cabf9 WatchSource:0}: Error finding container 15bc238babb33b75a0382f8870cd8e5f8f9c1d06691a11f68e8efa1ae15cabf9: Status 404 returned error can't find the container with id 15bc238babb33b75a0382f8870cd8e5f8f9c1d06691a11f68e8efa1ae15cabf9
May 20 13:53:31.416299 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:31.416261    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47rm7" event={"ID":"8f38fe80-2cd1-4b05-8934-6743f3dac8d8","Type":"ContainerStarted","Data":"15bc238babb33b75a0382f8870cd8e5f8f9c1d06691a11f68e8efa1ae15cabf9"}
May 20 13:53:31.987524 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:31.987492    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:53:32.419542 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:32.419507    2566 generic.go:358] "Generic (PLEG): container finished" podID="8f38fe80-2cd1-4b05-8934-6743f3dac8d8" containerID="ff80c70588fcd019f594ee10d0a35241a45e96b3afaef83a0e0c69566bcb0769" exitCode=0
May 20 13:53:32.419736 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:32.419577    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47rm7" event={"ID":"8f38fe80-2cd1-4b05-8934-6743f3dac8d8","Type":"ContainerDied","Data":"ff80c70588fcd019f594ee10d0a35241a45e96b3afaef83a0e0c69566bcb0769"}
May 20 13:53:33.423971 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:33.423929    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47rm7" event={"ID":"8f38fe80-2cd1-4b05-8934-6743f3dac8d8","Type":"ContainerStarted","Data":"eb6b5ff9bdd220be35a88bd53eb0acb7f695167eed769285c0af0ab5e6ef2827"}
May 20 13:53:33.423971 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:33.423971    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47rm7" event={"ID":"8f38fe80-2cd1-4b05-8934-6743f3dac8d8","Type":"ContainerStarted","Data":"b1c422b5ee09fe1f73b6661663b54df450bbbdb1e158104f22284750216772ee"}
May 20 13:53:33.446817 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:33.446773    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-47rm7" podStartSLOduration=3.552717774 podStartE2EDuration="4.446758739s" podCreationTimestamp="2026-05-20 13:53:29 +0000 UTC" firstStartedPulling="2026-05-20 13:53:30.859648621 +0000 UTC m=+171.414094173" lastFinishedPulling="2026-05-20 13:53:31.753689576 +0000 UTC m=+172.308135138" observedRunningTime="2026-05-20 13:53:33.445298479 +0000 UTC m=+173.999744054" watchObservedRunningTime="2026-05-20 13:53:33.446758739 +0000 UTC m=+174.001204312"
May 20 13:53:34.403433 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:34.403403    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pnn8g"
May 20 13:53:35.442265 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.442227    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:53:35.445825 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.445803    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.449007 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.448984    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 20 13:53:35.449115 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.448985    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 20 13:53:35.449115 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.448986    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 20 13:53:35.449266 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.449251    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 20 13:53:35.449266 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.449260    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 20 13:53:35.449381 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.449365    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 20 13:53:35.449911 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.449895    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 20 13:53:35.449968 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.449932    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 20 13:53:35.450020 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.449968    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7gu25e2mcb187\""
May 20 13:53:35.450273 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.450239    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 20 13:53:35.450662 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.450646    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 20 13:53:35.451367 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.451347    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 20 13:53:35.451706 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.451667    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jp2mg\""
May 20 13:53:35.457897 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.457874    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 20 13:53:35.458259 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.458242    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 20 13:53:35.462915 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.462895    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:53:35.484513 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484635 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484635 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484635 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484776 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484693    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-config-out\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484776 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484776 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484889 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484778    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484889 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484889 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484823    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484889 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-config\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.484889 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484870    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.485070 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-web-config\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.485070 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59cn\" (UniqueName: \"kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-kube-api-access-w59cn\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.485070 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.485070 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.485070 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.484996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.485070 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.485019    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.585386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585348    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.585386 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585388    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-config\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.585609 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585410    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.585609 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585437    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-web-config\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.585609 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585460    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w59cn\" (UniqueName: \"kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-kube-api-access-w59cn\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.585609 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585485    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.585609 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585510    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.585609 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585533    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.585925 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585818    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.585925 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585906    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.586090 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585833    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.586090 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585942    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.586090 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.585967    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.586090 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.586000    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.586090 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.586058    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-config-out\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.586330 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.586094    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.586330 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.586138    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.586330 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.586177    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.586330 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.586211    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.586517 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.586469    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.587628 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.587598    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.587778 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.587724    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.588632 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.588609    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.588959 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.588935    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.589268 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.589216    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.589364 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.589268    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.589691 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.589645    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.589774 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.589749    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.590727 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.590706    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.590994 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.590976    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-config-out\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.591122 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.591100    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-web-config\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.591851 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.591821    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.591939 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.591855    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.592055 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.592034    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.592356 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.592341    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-config\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.594910 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.594891    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59cn\" (UniqueName: \"kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-kube-api-access-w59cn\") pod \"prometheus-k8s-0\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.759806 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.759707    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:35.899539 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:35.899505    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:53:35.904777 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:53:35.904753    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b52a87_234c_45e3_99d6_5a8fa02c846f.slice/crio-bc4807f65ba28d6090a734c2ec3abd81321fb322d4eec9eb5c5cbe876ac30ef4 WatchSource:0}: Error finding container bc4807f65ba28d6090a734c2ec3abd81321fb322d4eec9eb5c5cbe876ac30ef4: Status 404 returned error can't find the container with id bc4807f65ba28d6090a734c2ec3abd81321fb322d4eec9eb5c5cbe876ac30ef4
May 20 13:53:36.432385 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:36.432348    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerStarted","Data":"bc4807f65ba28d6090a734c2ec3abd81321fb322d4eec9eb5c5cbe876ac30ef4"}
May 20 13:53:37.436467 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:37.436434    2566 generic.go:358] "Generic (PLEG): container finished" podID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerID="068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208" exitCode=0
May 20 13:53:37.436850 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:37.436496    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerDied","Data":"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"}
May 20 13:53:39.382903 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:39.382870    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75bbd8854-4b256"
May 20 13:53:40.446556 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:40.446521    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerStarted","Data":"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"}
May 20 13:53:40.446556 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:40.446560    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerStarted","Data":"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"}
May 20 13:53:42.454349 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:42.454311    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerStarted","Data":"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"}
May 20 13:53:42.454349 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:42.454347    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerStarted","Data":"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"}
May 20 13:53:42.454349 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:42.454357    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerStarted","Data":"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"}
May 20 13:53:42.454924 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:42.454365    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerStarted","Data":"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"}
May 20 13:53:42.482915 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:42.482869    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.777547883 podStartE2EDuration="7.482853637s" podCreationTimestamp="2026-05-20 13:53:35 +0000 UTC" firstStartedPulling="2026-05-20 13:53:35.907074261 +0000 UTC m=+176.461519813" lastFinishedPulling="2026-05-20 13:53:41.612380012 +0000 UTC m=+182.166825567" observedRunningTime="2026-05-20 13:53:42.480565905 +0000 UTC m=+183.035011492" watchObservedRunningTime="2026-05-20 13:53:42.482853637 +0000 UTC m=+183.037299215"
May 20 13:53:45.759836 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:45.759800    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:58.162501 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:53:58.162446    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" podUID="9a968e98-7e11-43e1-ba33-6819b9ed5b93" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 20 13:54:08.159986 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:08.159944    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" podUID="9a968e98-7e11-43e1-ba33-6819b9ed5b93" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 20 13:54:18.161554 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:18.161499    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" podUID="9a968e98-7e11-43e1-ba33-6819b9ed5b93" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 20 13:54:18.162032 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:18.161595    2566 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4"
May 20 13:54:18.162363 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:18.162314    2566 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a53d2c206f6c15d2f4c6a9dcce87e10624ae53798f63e7fb176550e7eed43d2e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 20 13:54:18.162434 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:18.162409    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" podUID="9a968e98-7e11-43e1-ba33-6819b9ed5b93" containerName="service-proxy" containerID="cri-o://a53d2c206f6c15d2f4c6a9dcce87e10624ae53798f63e7fb176550e7eed43d2e" gracePeriod=30
May 20 13:54:18.550006 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:18.549106    2566 generic.go:358] "Generic (PLEG): container finished" podID="9a968e98-7e11-43e1-ba33-6819b9ed5b93" containerID="a53d2c206f6c15d2f4c6a9dcce87e10624ae53798f63e7fb176550e7eed43d2e" exitCode=2
May 20 13:54:18.550006 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:18.549190    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" event={"ID":"9a968e98-7e11-43e1-ba33-6819b9ed5b93","Type":"ContainerDied","Data":"a53d2c206f6c15d2f4c6a9dcce87e10624ae53798f63e7fb176550e7eed43d2e"}
May 20 13:54:18.550006 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:18.549227    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-789f6b54df-fdvm4" event={"ID":"9a968e98-7e11-43e1-ba33-6819b9ed5b93","Type":"ContainerStarted","Data":"b42233283e6e15259546b5359d07004e76f76df1ed2fc984389a821edb55e864"}
May 20 13:54:35.760288 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:35.760252    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:35.778244 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:35.778216    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:36.614197 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:36.614170    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:51.833245 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:51.833200    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:54:51.835529 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:51.835505    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629f62b6-2132-440f-9f2f-e327d44004c9-metrics-certs\") pod \"network-metrics-daemon-twxjw\" (UID: \"629f62b6-2132-440f-9f2f-e327d44004c9\") " pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:54:52.092002 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:52.091934    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vwf8m\""
May 20 13:54:52.099574 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:52.099557    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twxjw"
May 20 13:54:52.218128 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:52.218100    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-twxjw"]
May 20 13:54:52.221060 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:54:52.221033    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629f62b6_2132_440f_9f2f_e327d44004c9.slice/crio-17b6b2b7e6ff9f801d692dcc1cc1f5ce87aa9c36c117272ec1f19702506cacd8 WatchSource:0}: Error finding container 17b6b2b7e6ff9f801d692dcc1cc1f5ce87aa9c36c117272ec1f19702506cacd8: Status 404 returned error can't find the container with id 17b6b2b7e6ff9f801d692dcc1cc1f5ce87aa9c36c117272ec1f19702506cacd8
May 20 13:54:52.641844 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:52.641798    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twxjw" event={"ID":"629f62b6-2132-440f-9f2f-e327d44004c9","Type":"ContainerStarted","Data":"17b6b2b7e6ff9f801d692dcc1cc1f5ce87aa9c36c117272ec1f19702506cacd8"}
May 20 13:54:53.645806 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.645767    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twxjw" event={"ID":"629f62b6-2132-440f-9f2f-e327d44004c9","Type":"ContainerStarted","Data":"a1c834a35619cf62e32c4d12d2fdfe9f2bc2d5403edaf0b852fab52a703cc9f5"}
May 20 13:54:53.646209 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.645810    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twxjw" event={"ID":"629f62b6-2132-440f-9f2f-e327d44004c9","Type":"ContainerStarted","Data":"7902b3218048472814ef067c22f97e89f0cd28761cab44f71eabc555ca6231ba"}
May 20 13:54:53.672054 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.672003    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-twxjw" podStartSLOduration=252.732129451 podStartE2EDuration="4m13.671987237s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:54:52.222790016 +0000 UTC m=+252.777235568" lastFinishedPulling="2026-05-20 13:54:53.1626478 +0000 UTC m=+253.717093354" observedRunningTime="2026-05-20 13:54:53.670513539 +0000 UTC m=+254.224959113" watchObservedRunningTime="2026-05-20 13:54:53.671987237 +0000 UTC m=+254.226432812"
May 20 13:54:53.719856 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.719825    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:53.720432 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.720305    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="prometheus" containerID="cri-o://2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c" gracePeriod=600
May 20 13:54:53.720432 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.720316    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3" gracePeriod=600
May 20 13:54:53.720432 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.720334    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="thanos-sidecar" containerID="cri-o://7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3" gracePeriod=600
May 20 13:54:53.720432 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.720383    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy" containerID="cri-o://f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953" gracePeriod=600
May 20 13:54:53.720432 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.720341    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy-web" containerID="cri-o://c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47" gracePeriod=600
May 20 13:54:53.720432 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.720374    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="config-reloader" containerID="cri-o://9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce" gracePeriod=600
May 20 13:54:53.953494 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:53.953470    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.047623 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047597    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.047808 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047630    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-serving-certs-ca-bundle\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.047808 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047657    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-kube-rbac-proxy\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.047808 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047698    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-rulefiles-0\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.047808 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047726    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-tls\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.047808 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047766    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-grpc-tls\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.047808 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047792    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-metrics-client-ca\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047818    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-web-config\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047845    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w59cn\" (UniqueName: \"kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-kube-api-access-w59cn\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047870    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-kubelet-serving-ca-bundle\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047898    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-tls-assets\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047930    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-trusted-ca-bundle\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047957    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-metrics-client-certs\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.047983    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.048022    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-db\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.048063    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-config\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.048073    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:54.048118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.048096    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-config-out\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048643 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.048133    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-thanos-prometheus-http-client-file\") pod \"15b52a87-234c-45e3-99d6-5a8fa02c846f\" (UID: \"15b52a87-234c-45e3-99d6-5a8fa02c846f\") "
May 20 13:54:54.048643 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.048375    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.048643 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.048377    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:54.050310 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.049105    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:54.050310 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.049458    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:54.050310 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.049781    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:54.050310 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.049869    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:54:54.050852 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.050808    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:54.051015 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.050991    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:54.052645 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.052601    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:54.052761 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.052622    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:54.052761 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.052638    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-config" (OuterVolumeSpecName: "config") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:54.052761 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.052651    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:54.053087 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.053058    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:54.053321 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.053281    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:54:54.053321 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.053284    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:54.053455 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.053363    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-config-out" (OuterVolumeSpecName: "config-out") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:54:54.054137 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.054120    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-kube-api-access-w59cn" (OuterVolumeSpecName: "kube-api-access-w59cn") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "kube-api-access-w59cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:54:54.061965 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.061942    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-web-config" (OuterVolumeSpecName: "web-config") pod "15b52a87-234c-45e3-99d6-5a8fa02c846f" (UID: "15b52a87-234c-45e3-99d6-5a8fa02c846f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:54.149520 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149486    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-grpc-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149520 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149516    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-metrics-client-ca\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149520 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149526    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-web-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149535    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w59cn\" (UniqueName: \"kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-kube-api-access-w59cn\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149546    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149557    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15b52a87-234c-45e3-99d6-5a8fa02c846f-tls-assets\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149567    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149576    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-metrics-client-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149585    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149594    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-db\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149606    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-config\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149619    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15b52a87-234c-45e3-99d6-5a8fa02c846f-config-out\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149630    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149638    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149647    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-kube-rbac-proxy\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149657    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15b52a87-234c-45e3-99d6-5a8fa02c846f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.149783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.149667    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15b52a87-234c-45e3-99d6-5a8fa02c846f-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 13:54:54.651282 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651252    2566 generic.go:358] "Generic (PLEG): container finished" podID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerID="906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3" exitCode=0
May 20 13:54:54.651282 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651276    2566 generic.go:358] "Generic (PLEG): container finished" podID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerID="f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953" exitCode=0
May 20 13:54:54.651282 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651284    2566 generic.go:358] "Generic (PLEG): container finished" podID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerID="c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47" exitCode=0
May 20 13:54:54.651282 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651290    2566 generic.go:358] "Generic (PLEG): container finished" podID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerID="7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3" exitCode=0
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651296    2566 generic.go:358] "Generic (PLEG): container finished" podID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerID="9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce" exitCode=0
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651301    2566 generic.go:358] "Generic (PLEG): container finished" podID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerID="2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c" exitCode=0
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651331    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerDied","Data":"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"}
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651356    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651366    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerDied","Data":"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"}
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651380    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerDied","Data":"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"}
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651394    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerDied","Data":"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"}
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651404    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerDied","Data":"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"}
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651414    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerDied","Data":"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"}
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651425    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15b52a87-234c-45e3-99d6-5a8fa02c846f","Type":"ContainerDied","Data":"bc4807f65ba28d6090a734c2ec3abd81321fb322d4eec9eb5c5cbe876ac30ef4"}
May 20 13:54:54.651848 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.651442    2566 scope.go:117] "RemoveContainer" containerID="906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"
May 20 13:54:54.660293 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.659617    2566 scope.go:117] "RemoveContainer" containerID="f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"
May 20 13:54:54.667518 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.667502    2566 scope.go:117] "RemoveContainer" containerID="c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"
May 20 13:54:54.673407 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.673391    2566 scope.go:117] "RemoveContainer" containerID="7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"
May 20 13:54:54.679067 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.679051    2566 scope.go:117] "RemoveContainer" containerID="9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"
May 20 13:54:54.684703 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.684692    2566 scope.go:117] "RemoveContainer" containerID="2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"
May 20 13:54:54.692153 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.692135    2566 scope.go:117] "RemoveContainer" containerID="068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"
May 20 13:54:54.692538 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.692521    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:54.697833 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.697819    2566 scope.go:117] "RemoveContainer" containerID="906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"
May 20 13:54:54.698072 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:54:54.698052    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": container with ID starting with 906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3 not found: ID does not exist" containerID="906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"
May 20 13:54:54.698113 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.698079    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"} err="failed to get container status \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": rpc error: code = NotFound desc = could not find container \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": container with ID starting with 906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3 not found: ID does not exist"
May 20 13:54:54.698113 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.698096    2566 scope.go:117] "RemoveContainer" containerID="f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"
May 20 13:54:54.698311 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:54:54.698295    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": container with ID starting with f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953 not found: ID does not exist" containerID="f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"
May 20 13:54:54.698351 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.698315    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"} err="failed to get container status \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": rpc error: code = NotFound desc = could not find container \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": container with ID starting with f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953 not found: ID does not exist"
May 20 13:54:54.698351 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.698327    2566 scope.go:117] "RemoveContainer" containerID="c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"
May 20 13:54:54.698503 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:54:54.698489    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": container with ID starting with c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47 not found: ID does not exist" containerID="c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"
May 20 13:54:54.698544 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.698505    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"} err="failed to get container status \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": rpc error: code = NotFound desc = could not find container \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": container with ID starting with c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47 not found: ID does not exist"
May 20 13:54:54.698544 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.698517    2566 scope.go:117] "RemoveContainer" containerID="7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"
May 20 13:54:54.698740 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:54:54.698726    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": container with ID starting with 7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3 not found: ID does not exist" containerID="7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"
May 20 13:54:54.698791 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.698743    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"} err="failed to get container status \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": rpc error: code = NotFound desc = could not find container \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": container with ID starting with 7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3 not found: ID does not exist"
May 20 13:54:54.698791 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.698756    2566 scope.go:117] "RemoveContainer" containerID="9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"
May 20 13:54:54.699001 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:54:54.698982    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": container with ID starting with 9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce not found: ID does not exist" containerID="9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"
May 20 13:54:54.699082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.699005    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"} err="failed to get container status \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": rpc error: code = NotFound desc = could not find container \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": container with ID starting with 9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce not found: ID does not exist"
May 20 13:54:54.699082 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.699018    2566 scope.go:117] "RemoveContainer" containerID="2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"
May 20 13:54:54.699339 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:54:54.699314    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": container with ID starting with 2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c not found: ID does not exist" containerID="2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"
May 20 13:54:54.699401 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.699356    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"} err="failed to get container status \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": rpc error: code = NotFound desc = could not find container \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": container with ID starting with 2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c not found: ID does not exist"
May 20 13:54:54.699401 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.699372    2566 scope.go:117] "RemoveContainer" containerID="068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"
May 20 13:54:54.699621 ip-10-0-140-71 kubenswrapper[2566]: E0520 13:54:54.699596    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": container with ID starting with 068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208 not found: ID does not exist" containerID="068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"
May 20 13:54:54.699755 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.699625    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"} err="failed to get container status \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": rpc error: code = NotFound desc = could not find container \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": container with ID starting with 068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208 not found: ID does not exist"
May 20 13:54:54.699755 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.699651    2566 scope.go:117] "RemoveContainer" containerID="906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"
May 20 13:54:54.699914 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.699894    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"} err="failed to get container status \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": rpc error: code = NotFound desc = could not find container \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": container with ID starting with 906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3 not found: ID does not exist"
May 20 13:54:54.699959 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.699917    2566 scope.go:117] "RemoveContainer" containerID="f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"
May 20 13:54:54.700165 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.700143    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"} err="failed to get container status \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": rpc error: code = NotFound desc = could not find container \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": container with ID starting with f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953 not found: ID does not exist"
May 20 13:54:54.700218 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.700167    2566 scope.go:117] "RemoveContainer" containerID="c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"
May 20 13:54:54.700394 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.700377    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"} err="failed to get container status \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": rpc error: code = NotFound desc = could not find container \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": container with ID starting with c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47 not found: ID does not exist"
May 20 13:54:54.700394 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.700394    2566 scope.go:117] "RemoveContainer" containerID="7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"
May 20 13:54:54.700648 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.700627    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"} err="failed to get container status \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": rpc error: code = NotFound desc = could not find container \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": container with ID starting with 7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3 not found: ID does not exist"
May 20 13:54:54.700724 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.700649    2566 scope.go:117] "RemoveContainer" containerID="9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"
May 20 13:54:54.700948 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.700919    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"} err="failed to get container status \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": rpc error: code = NotFound desc = could not find container \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": container with ID starting with 9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce not found: ID does not exist"
May 20 13:54:54.701008 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.700965    2566 scope.go:117] "RemoveContainer" containerID="2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"
May 20 13:54:54.701207 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.701185    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"} err="failed to get container status \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": rpc error: code = NotFound desc = could not find container \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": container with ID starting with 2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c not found: ID does not exist"
May 20 13:54:54.701207 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.701206    2566 scope.go:117] "RemoveContainer" containerID="068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"
May 20 13:54:54.701485 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.701467    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"} err="failed to get container status \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": rpc error: code = NotFound desc = could not find container \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": container with ID starting with 068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208 not found: ID does not exist"
May 20 13:54:54.701485 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.701485    2566 scope.go:117] "RemoveContainer" containerID="906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"
May 20 13:54:54.701706 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.701685    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:54.701794 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.701719    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"} err="failed to get container status \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": rpc error: code = NotFound desc = could not find container \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": container with ID starting with 906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3 not found: ID does not exist"
May 20 13:54:54.701794 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.701737    2566 scope.go:117] "RemoveContainer" containerID="f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"
May 20 13:54:54.701951 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.701935    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"} err="failed to get container status \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": rpc error: code = NotFound desc = could not find container \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": container with ID starting with f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953 not found: ID does not exist"
May 20 13:54:54.701991 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.701952    2566 scope.go:117] "RemoveContainer" containerID="c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"
May 20 13:54:54.702149 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.702133    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"} err="failed to get container status \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": rpc error: code = NotFound desc = could not find container \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": container with ID starting with c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47 not found: ID does not exist"
May 20 13:54:54.702196 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.702150    2566 scope.go:117] "RemoveContainer" containerID="7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"
May 20 13:54:54.702324 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.702304    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"} err="failed to get container status \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": rpc error: code = NotFound desc = could not find container \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": container with ID starting with 7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3 not found: ID does not exist"
May 20 13:54:54.702402 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.702324    2566 scope.go:117] "RemoveContainer" containerID="9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"
May 20 13:54:54.702531 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.702511    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"} err="failed to get container status \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": rpc error: code = NotFound desc = could not find container \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": container with ID starting with 9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce not found: ID does not exist"
May 20 13:54:54.702589 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.702532    2566 scope.go:117] "RemoveContainer" containerID="2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"
May 20 13:54:54.702830 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.702810    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"} err="failed to get container status \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": rpc error: code = NotFound desc = could not find container \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": container with ID starting with 2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c not found: ID does not exist"
May 20 13:54:54.702903 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.702830    2566 scope.go:117] "RemoveContainer" containerID="068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"
May 20 13:54:54.703069 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.703046    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"} err="failed to get container status \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": rpc error: code = NotFound desc = could not find container \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": container with ID starting with 068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208 not found: ID does not exist"
May 20 13:54:54.703136 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.703071    2566 scope.go:117] "RemoveContainer" containerID="906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"
May 20 13:54:54.703344 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.703325    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"} err="failed to get container status \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": rpc error: code = NotFound desc = could not find container \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": container with ID starting with 906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3 not found: ID does not exist"
May 20 13:54:54.703436 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.703345    2566 scope.go:117] "RemoveContainer" containerID="f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"
May 20 13:54:54.703655 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.703624    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"} err="failed to get container status \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": rpc error: code = NotFound desc = could not find container \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": container with ID starting with f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953 not found: ID does not exist"
May 20 13:54:54.703655 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.703654    2566 scope.go:117] "RemoveContainer" containerID="c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"
May 20 13:54:54.703937 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.703917    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"} err="failed to get container status \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": rpc error: code = NotFound desc = could not find container \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": container with ID starting with c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47 not found: ID does not exist"
May 20 13:54:54.704016 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.703939    2566 scope.go:117] "RemoveContainer" containerID="7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"
May 20 13:54:54.704242 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.704221    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"} err="failed to get container status \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": rpc error: code = NotFound desc = could not find container \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": container with ID starting with 7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3 not found: ID does not exist"
May 20 13:54:54.704316 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.704253    2566 scope.go:117] "RemoveContainer" containerID="9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"
May 20 13:54:54.704494 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.704475    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"} err="failed to get container status \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": rpc error: code = NotFound desc = could not find container \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": container with ID starting with 9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce not found: ID does not exist"
May 20 13:54:54.704543 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.704495    2566 scope.go:117] "RemoveContainer" containerID="2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"
May 20 13:54:54.704695 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.704661    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"} err="failed to get container status \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": rpc error: code = NotFound desc = could not find container \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": container with ID starting with 2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c not found: ID does not exist"
May 20 13:54:54.704741 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.704696    2566 scope.go:117] "RemoveContainer" containerID="068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"
May 20 13:54:54.704912 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.704891    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"} err="failed to get container status \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": rpc error: code = NotFound desc = could not find container \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": container with ID starting with 068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208 not found: ID does not exist"
May 20 13:54:54.704958 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.704914    2566 scope.go:117] "RemoveContainer" containerID="906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"
May 20 13:54:54.705128 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.705112    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"} err="failed to get container status \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": rpc error: code = NotFound desc = could not find container \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": container with ID starting with 906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3 not found: ID does not exist"
May 20 13:54:54.705128 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.705128    2566 scope.go:117] "RemoveContainer" containerID="f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"
May 20 13:54:54.705339 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.705317    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"} err="failed to get container status \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": rpc error: code = NotFound desc = could not find container \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": container with ID starting with f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953 not found: ID does not exist"
May 20 13:54:54.705376 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.705342    2566 scope.go:117] "RemoveContainer" containerID="c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"
May 20 13:54:54.705536 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.705521    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"} err="failed to get container status \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": rpc error: code = NotFound desc = could not find container \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": container with ID starting with c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47 not found: ID does not exist"
May 20 13:54:54.705577 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.705536    2566 scope.go:117] "RemoveContainer" containerID="7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"
May 20 13:54:54.705746 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.705729    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"} err="failed to get container status \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": rpc error: code = NotFound desc = could not find container \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": container with ID starting with 7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3 not found: ID does not exist"
May 20 13:54:54.705820 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.705746    2566 scope.go:117] "RemoveContainer" containerID="9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"
May 20 13:54:54.705955 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.705940    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"} err="failed to get container status \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": rpc error: code = NotFound desc = could not find container \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": container with ID starting with 9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce not found: ID does not exist"
May 20 13:54:54.706009 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.705955    2566 scope.go:117] "RemoveContainer" containerID="2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"
May 20 13:54:54.706157 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.706142    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"} err="failed to get container status \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": rpc error: code = NotFound desc = could not find container \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": container with ID starting with 2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c not found: ID does not exist"
May 20 13:54:54.706202 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.706156    2566 scope.go:117] "RemoveContainer" containerID="068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"
May 20 13:54:54.706378 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.706359    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"} err="failed to get container status \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": rpc error: code = NotFound desc = could not find container \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": container with ID starting with 068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208 not found: ID does not exist"
May 20 13:54:54.706422 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.706378    2566 scope.go:117] "RemoveContainer" containerID="906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"
May 20 13:54:54.706571 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.706551    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3"} err="failed to get container status \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": rpc error: code = NotFound desc = could not find container \"906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3\": container with ID starting with 906ee115f13ba7bb33a7aff6a26e6bf3b40969d2cc73796dc644c9ada317ddc3 not found: ID does not exist"
May 20 13:54:54.706612 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.706572    2566 scope.go:117] "RemoveContainer" containerID="f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"
May 20 13:54:54.706785 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.706767    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953"} err="failed to get container status \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": rpc error: code = NotFound desc = could not find container \"f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953\": container with ID starting with f09977308a6f56ae0a0a1b4b4e47a0746931f617d40a11f78e4c1b526bcce953 not found: ID does not exist"
May 20 13:54:54.706832 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.706786    2566 scope.go:117] "RemoveContainer" containerID="c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"
May 20 13:54:54.706983 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.706966    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47"} err="failed to get container status \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": rpc error: code = NotFound desc = could not find container \"c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47\": container with ID starting with c062c517af297f0aa371580f580bc00c697ab01e7742b790d0f175d92e34cb47 not found: ID does not exist"
May 20 13:54:54.707032 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.706983    2566 scope.go:117] "RemoveContainer" containerID="7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"
May 20 13:54:54.707169 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.707153    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3"} err="failed to get container status \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": rpc error: code = NotFound desc = could not find container \"7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3\": container with ID starting with 7608f76a88d228716ac5abc0ed580abdcf4123d2c8f38554d2cc0a8d78aca1b3 not found: ID does not exist"
May 20 13:54:54.707169 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.707169    2566 scope.go:117] "RemoveContainer" containerID="9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"
May 20 13:54:54.707356 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.707342    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce"} err="failed to get container status \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": rpc error: code = NotFound desc = could not find container \"9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce\": container with ID starting with 9f5698012486a42c4c05e7b0eaaca52f99ba839d22b647a636e7bc111c9e22ce not found: ID does not exist"
May 20 13:54:54.707356 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.707355    2566 scope.go:117] "RemoveContainer" containerID="2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"
May 20 13:54:54.707530 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.707513    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c"} err="failed to get container status \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": rpc error: code = NotFound desc = could not find container \"2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c\": container with ID starting with 2a9a06e0634fc580ffd64da3b44f2659ed5ee6baca6a2d33d4be8c09affaba3c not found: ID does not exist"
May 20 13:54:54.707530 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.707530    2566 scope.go:117] "RemoveContainer" containerID="068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"
May 20 13:54:54.707764 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.707749    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208"} err="failed to get container status \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": rpc error: code = NotFound desc = could not find container \"068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208\": container with ID starting with 068840bf22ae5d777391dc8fef47c3bf560cdd51aaf3dfb5522a181e65b55208 not found: ID does not exist"
May 20 13:54:54.770500 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770474    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:54.770730 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770718    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="init-config-reloader"
May 20 13:54:54.770782 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770733    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="init-config-reloader"
May 20 13:54:54.770782 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770740    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="config-reloader"
May 20 13:54:54.770782 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770745    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="config-reloader"
May 20 13:54:54.770782 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770756    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="thanos-sidecar"
May 20 13:54:54.770782 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770763    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="thanos-sidecar"
May 20 13:54:54.770782 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770771    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy"
May 20 13:54:54.770782 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770779    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770786    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="prometheus"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770791    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="prometheus"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770797    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy-web"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770802    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy-web"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770809    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy-thanos"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770814    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy-thanos"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770857    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy-web"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770865    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770872    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="kube-rbac-proxy-thanos"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770879    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="prometheus"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770885    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="config-reloader"
May 20 13:54:54.770970 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.770890    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" containerName="thanos-sidecar"
May 20 13:54:54.775769 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.775754    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.783871 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.783853    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 20 13:54:54.783976 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.783961    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 20 13:54:54.784093 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.784080    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 20 13:54:54.784093 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.784086    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 20 13:54:54.788138 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.788123    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 20 13:54:54.790040 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.790023    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 20 13:54:54.790119 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.790058    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 20 13:54:54.790119 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.790106    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7gu25e2mcb187\""
May 20 13:54:54.790224 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.790147    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jp2mg\""
May 20 13:54:54.790342 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.790323    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 20 13:54:54.791565 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.791549    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 20 13:54:54.791660 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.791644    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 20 13:54:54.791739 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.791690    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 20 13:54:54.794586 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.794572    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 20 13:54:54.803118 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.803101    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 20 13:54:54.834110 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.834088    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:54.854977 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.854954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6ff3414-4a60-46ad-9329-632449a21641-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855067 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.854983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855067 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855067 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855087    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-config\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895jd\" (UniqueName: \"kubernetes.io/projected/d6ff3414-4a60-46ad-9329-632449a21641-kube-api-access-895jd\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6ff3414-4a60-46ad-9329-632449a21641-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855355 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855353    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855740 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855740 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855740 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.855740 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.855461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6ff3414-4a60-46ad-9329-632449a21641-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.955837 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.955773    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.955837 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.955813    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.955837 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.955832    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956054 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.955892    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956054 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.955939    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-config\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956054 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.955961    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-895jd\" (UniqueName: \"kubernetes.io/projected/d6ff3414-4a60-46ad-9329-632449a21641-kube-api-access-895jd\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956054 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.955982    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956248 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956111    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956248 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956144    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6ff3414-4a60-46ad-9329-632449a21641-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956248 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956177    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956248 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956202    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956248 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956224    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956486 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956248    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956486 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956276    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956486 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956312    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6ff3414-4a60-46ad-9329-632449a21641-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956486 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956341    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6ff3414-4a60-46ad-9329-632449a21641-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956486 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956367    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956486 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956392    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.956806 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.956617    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.957396 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.957084    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6ff3414-4a60-46ad-9329-632449a21641-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.957396 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.957343    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.957626 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.957603    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.958015 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.957969    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.959489 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.959463    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.960211 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.960171    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6ff3414-4a60-46ad-9329-632449a21641-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.960302 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.960270    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6ff3414-4a60-46ad-9329-632449a21641-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.960557 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.960521    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.960728 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.960707    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.961109 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.961066    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.961195 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.961112    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.961280 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.961260    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.961425 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.961406    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-config\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.961488 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.961466    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.961710 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.961665    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6ff3414-4a60-46ad-9329-632449a21641-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.961902 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.961885    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6ff3414-4a60-46ad-9329-632449a21641-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:54.966022 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:54.966001    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895jd\" (UniqueName: \"kubernetes.io/projected/d6ff3414-4a60-46ad-9329-632449a21641-kube-api-access-895jd\") pod \"prometheus-k8s-0\" (UID: \"d6ff3414-4a60-46ad-9329-632449a21641\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:55.084097 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:55.084071    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:55.215295 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:55.215270    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:55.217218 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:54:55.217192    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ff3414_4a60_46ad_9329_632449a21641.slice/crio-728b12914b47449332544c6e4bc02bf91975933959007af50daaf94d28e14f9d WatchSource:0}: Error finding container 728b12914b47449332544c6e4bc02bf91975933959007af50daaf94d28e14f9d: Status 404 returned error can't find the container with id 728b12914b47449332544c6e4bc02bf91975933959007af50daaf94d28e14f9d
May 20 13:54:55.655705 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:55.655653    2566 generic.go:358] "Generic (PLEG): container finished" podID="d6ff3414-4a60-46ad-9329-632449a21641" containerID="22c793da2e8a94159d0b24c5c13d96485befe363f5097c02bc1ab0cf9987d1f3" exitCode=0
May 20 13:54:55.656086 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:55.655722    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6ff3414-4a60-46ad-9329-632449a21641","Type":"ContainerDied","Data":"22c793da2e8a94159d0b24c5c13d96485befe363f5097c02bc1ab0cf9987d1f3"}
May 20 13:54:55.656086 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:55.655742    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6ff3414-4a60-46ad-9329-632449a21641","Type":"ContainerStarted","Data":"728b12914b47449332544c6e4bc02bf91975933959007af50daaf94d28e14f9d"}
May 20 13:54:55.991805 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:55.991779    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15b52a87-234c-45e3-99d6-5a8fa02c846f" path="/var/lib/kubelet/pods/15b52a87-234c-45e3-99d6-5a8fa02c846f/volumes"
May 20 13:54:56.663909 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:56.663877    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6ff3414-4a60-46ad-9329-632449a21641","Type":"ContainerStarted","Data":"90760cb24e1200fc314221c466b5e2d3b8c5999f6117a63fb3d1541150ab585d"}
May 20 13:54:56.663909 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:56.663911    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6ff3414-4a60-46ad-9329-632449a21641","Type":"ContainerStarted","Data":"599a58ee876cba12284dd194b5f77732c3703cf1b754706808a1f8f0a2ea71eb"}
May 20 13:54:56.664314 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:56.663921    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6ff3414-4a60-46ad-9329-632449a21641","Type":"ContainerStarted","Data":"2b46a87d1b92de75697abf5a91f6cbff65015321df49a8b4cdec8c5ed9bf9494"}
May 20 13:54:56.664314 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:56.663930    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6ff3414-4a60-46ad-9329-632449a21641","Type":"ContainerStarted","Data":"e5cd59197fefa8fa40d84b317c559cfb65c6422c28849d6298359e5a9b21c6a5"}
May 20 13:54:56.664314 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:56.663938    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6ff3414-4a60-46ad-9329-632449a21641","Type":"ContainerStarted","Data":"c4f39ea514d968696900ef7f2e64e5eddb9a279abff825d6b2e747b09f5aafb0"}
May 20 13:54:56.664314 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:56.663945    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6ff3414-4a60-46ad-9329-632449a21641","Type":"ContainerStarted","Data":"e24ee6c29ca8d754d9faaab7e117500ce7c7f9ba8ee9aa4abb57cd3a2aacc2f0"}
May 20 13:54:56.697125 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:54:56.697069    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.69705329 podStartE2EDuration="2.69705329s" podCreationTimestamp="2026-05-20 13:54:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:54:56.695491547 +0000 UTC m=+257.249937122" watchObservedRunningTime="2026-05-20 13:54:56.69705329 +0000 UTC m=+257.251498865"
May 20 13:55:00.084476 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:55:00.084406    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:55:39.872783 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:55:39.872761    2566 kubelet.go:1628] "Image garbage collection succeeded"
May 20 13:55:55.084520 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:55:55.084481    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:55:55.099485 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:55:55.099461    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:55:55.833933 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:55:55.833905    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:56:34.132538 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.132455    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ctwsd"]
May 20 13:56:34.135477 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.135462    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.137998 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.137983    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 20 13:56:34.143064 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.143043    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ctwsd"]
May 20 13:56:34.282167 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.282130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3823f5fb-0ece-4840-8565-4bfb6334a457-original-pull-secret\") pod \"global-pull-secret-syncer-ctwsd\" (UID: \"3823f5fb-0ece-4840-8565-4bfb6334a457\") " pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.282167 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.282169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3823f5fb-0ece-4840-8565-4bfb6334a457-kubelet-config\") pod \"global-pull-secret-syncer-ctwsd\" (UID: \"3823f5fb-0ece-4840-8565-4bfb6334a457\") " pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.282440 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.282237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3823f5fb-0ece-4840-8565-4bfb6334a457-dbus\") pod \"global-pull-secret-syncer-ctwsd\" (UID: \"3823f5fb-0ece-4840-8565-4bfb6334a457\") " pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.382742 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.382640    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3823f5fb-0ece-4840-8565-4bfb6334a457-original-pull-secret\") pod \"global-pull-secret-syncer-ctwsd\" (UID: \"3823f5fb-0ece-4840-8565-4bfb6334a457\") " pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.382742 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.382709    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3823f5fb-0ece-4840-8565-4bfb6334a457-kubelet-config\") pod \"global-pull-secret-syncer-ctwsd\" (UID: \"3823f5fb-0ece-4840-8565-4bfb6334a457\") " pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.382961 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.382773    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3823f5fb-0ece-4840-8565-4bfb6334a457-dbus\") pod \"global-pull-secret-syncer-ctwsd\" (UID: \"3823f5fb-0ece-4840-8565-4bfb6334a457\") " pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.382961 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.382844    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3823f5fb-0ece-4840-8565-4bfb6334a457-kubelet-config\") pod \"global-pull-secret-syncer-ctwsd\" (UID: \"3823f5fb-0ece-4840-8565-4bfb6334a457\") " pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.382961 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.382931    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3823f5fb-0ece-4840-8565-4bfb6334a457-dbus\") pod \"global-pull-secret-syncer-ctwsd\" (UID: \"3823f5fb-0ece-4840-8565-4bfb6334a457\") " pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.384879 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.384862    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3823f5fb-0ece-4840-8565-4bfb6334a457-original-pull-secret\") pod \"global-pull-secret-syncer-ctwsd\" (UID: \"3823f5fb-0ece-4840-8565-4bfb6334a457\") " pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.444830 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.444773    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctwsd"
May 20 13:56:34.576217 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.576088    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ctwsd"]
May 20 13:56:34.578897 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:56:34.578863    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3823f5fb_0ece_4840_8565_4bfb6334a457.slice/crio-b6ac2d37566c7d2d51d69c64c04d32faf6feb247bca2e00610a538cc3e7388d2 WatchSource:0}: Error finding container b6ac2d37566c7d2d51d69c64c04d32faf6feb247bca2e00610a538cc3e7388d2: Status 404 returned error can't find the container with id b6ac2d37566c7d2d51d69c64c04d32faf6feb247bca2e00610a538cc3e7388d2
May 20 13:56:34.580407 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.580392    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 13:56:34.916343 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:34.916306    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ctwsd" event={"ID":"3823f5fb-0ece-4840-8565-4bfb6334a457","Type":"ContainerStarted","Data":"b6ac2d37566c7d2d51d69c64c04d32faf6feb247bca2e00610a538cc3e7388d2"}
May 20 13:56:38.928481 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:38.928436    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ctwsd" event={"ID":"3823f5fb-0ece-4840-8565-4bfb6334a457","Type":"ContainerStarted","Data":"7f7a7b65fd744b15066eb2814748e0c86229ec9aa1f8b9251777e72400a336e6"}
May 20 13:56:38.944314 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:56:38.944267    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ctwsd" podStartSLOduration=1.361516488 podStartE2EDuration="4.944252944s" podCreationTimestamp="2026-05-20 13:56:34 +0000 UTC" firstStartedPulling="2026-05-20 13:56:34.580519447 +0000 UTC m=+355.134964998" lastFinishedPulling="2026-05-20 13:56:38.163255902 +0000 UTC m=+358.717701454" observedRunningTime="2026-05-20 13:56:38.943818716 +0000 UTC m=+359.498264292" watchObservedRunningTime="2026-05-20 13:56:38.944252944 +0000 UTC m=+359.498698519"
May 20 13:59:45.926131 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.926037    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"]
May 20 13:59:45.929138 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.929122    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:45.933771 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.933751    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 20 13:59:45.933878 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.933827    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 20 13:59:45.933949 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.933936    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 20 13:59:45.934006 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.933989    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 20 13:59:45.934058 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.934030    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 20 13:59:45.934117 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.934086    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nd5s7\""
May 20 13:59:45.939281 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.939260    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"]
May 20 13:59:45.985077 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.985054    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-metrics-cert\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:45.985175 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.985083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtn4\" (UniqueName: \"kubernetes.io/projected/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-kube-api-access-nrtn4\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:45.985175 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.985105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-manager-config\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:45.985245 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:45.985210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-cert\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:46.085836 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.085807    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-cert\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:46.085948 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.085850    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-metrics-cert\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:46.085988 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.085964    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtn4\" (UniqueName: \"kubernetes.io/projected/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-kube-api-access-nrtn4\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:46.086028 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.086005    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-manager-config\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:46.086620 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.086599    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-manager-config\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:46.088258 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.088243    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-cert\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:46.088318 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.088292    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-metrics-cert\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:46.094836 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.094807    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtn4\" (UniqueName: \"kubernetes.io/projected/c1ac38c3-08ab-4d1f-a2cd-d156716a95a0-kube-api-access-nrtn4\") pod \"lws-controller-manager-79d5949c76-m9d7g\" (UID: \"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:46.237623 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.237537    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:46.358532 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.358507    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"]
May 20 13:59:46.360892 ip-10-0-140-71 kubenswrapper[2566]: W0520 13:59:46.360852    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ac38c3_08ab_4d1f_a2cd_d156716a95a0.slice/crio-3f904c1e36f1002f56d69152dc89b2f4b9368a5b7c32eab5942de4b39eb15868 WatchSource:0}: Error finding container 3f904c1e36f1002f56d69152dc89b2f4b9368a5b7c32eab5942de4b39eb15868: Status 404 returned error can't find the container with id 3f904c1e36f1002f56d69152dc89b2f4b9368a5b7c32eab5942de4b39eb15868
May 20 13:59:46.398484 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:46.398457    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g" event={"ID":"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0","Type":"ContainerStarted","Data":"3f904c1e36f1002f56d69152dc89b2f4b9368a5b7c32eab5942de4b39eb15868"}
May 20 13:59:49.410263 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:49.410188    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g" event={"ID":"c1ac38c3-08ab-4d1f-a2cd-d156716a95a0","Type":"ContainerStarted","Data":"2e349731b76b2c23e7e024e1da4e8d53a571644988389421465549a75db72ebf"}
May 20 13:59:49.410727 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:49.410309    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 13:59:49.428259 ip-10-0-140-71 kubenswrapper[2566]: I0520 13:59:49.428219    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g" podStartSLOduration=1.6670632570000001 podStartE2EDuration="4.428203461s" podCreationTimestamp="2026-05-20 13:59:45 +0000 UTC" firstStartedPulling="2026-05-20 13:59:46.362664122 +0000 UTC m=+546.917109674" lastFinishedPulling="2026-05-20 13:59:49.123804327 +0000 UTC m=+549.678249878" observedRunningTime="2026-05-20 13:59:49.427481516 +0000 UTC m=+549.981927091" watchObservedRunningTime="2026-05-20 13:59:49.428203461 +0000 UTC m=+549.982649036"
May 20 14:00:00.415486 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:00.415450    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-m9d7g"
May 20 14:00:38.714853 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:38.714809    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7587b89b76-r95s7"]
May 20 14:00:38.717933 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:38.717916    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7587b89b76-r95s7"
May 20 14:00:38.724096 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:38.723911    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 20 14:00:38.725166 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:38.725147    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 20 14:00:38.725241 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:38.725147    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-998cb\""
May 20 14:00:38.750322 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:38.750297    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7587b89b76-r95s7"]
May 20 14:00:38.895494 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:38.895460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mqfq\" (UniqueName: \"kubernetes.io/projected/e10b8171-6aa5-49b6-be5d-dda923e1a9f3-kube-api-access-2mqfq\") pod \"authorino-operator-7587b89b76-r95s7\" (UID: \"e10b8171-6aa5-49b6-be5d-dda923e1a9f3\") " pod="kuadrant-system/authorino-operator-7587b89b76-r95s7"
May 20 14:00:38.996920 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:38.996829    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2mqfq\" (UniqueName: \"kubernetes.io/projected/e10b8171-6aa5-49b6-be5d-dda923e1a9f3-kube-api-access-2mqfq\") pod \"authorino-operator-7587b89b76-r95s7\" (UID: \"e10b8171-6aa5-49b6-be5d-dda923e1a9f3\") " pod="kuadrant-system/authorino-operator-7587b89b76-r95s7"
May 20 14:00:39.011606 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:39.011583    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mqfq\" (UniqueName: \"kubernetes.io/projected/e10b8171-6aa5-49b6-be5d-dda923e1a9f3-kube-api-access-2mqfq\") pod \"authorino-operator-7587b89b76-r95s7\" (UID: \"e10b8171-6aa5-49b6-be5d-dda923e1a9f3\") " pod="kuadrant-system/authorino-operator-7587b89b76-r95s7"
May 20 14:00:39.027404 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:39.027377    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7587b89b76-r95s7"
May 20 14:00:39.172623 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:39.172486    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7587b89b76-r95s7"]
May 20 14:00:39.175484 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:00:39.175460    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10b8171_6aa5_49b6_be5d_dda923e1a9f3.slice/crio-bbf91f3fe12581d9dee0650ab60999fec39a888062748fe8bdaa3d7e371a566a WatchSource:0}: Error finding container bbf91f3fe12581d9dee0650ab60999fec39a888062748fe8bdaa3d7e371a566a: Status 404 returned error can't find the container with id bbf91f3fe12581d9dee0650ab60999fec39a888062748fe8bdaa3d7e371a566a
May 20 14:00:39.544584 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:39.544547    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7587b89b76-r95s7" event={"ID":"e10b8171-6aa5-49b6-be5d-dda923e1a9f3","Type":"ContainerStarted","Data":"bbf91f3fe12581d9dee0650ab60999fec39a888062748fe8bdaa3d7e371a566a"}
May 20 14:00:42.556043 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:42.555990    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7587b89b76-r95s7" event={"ID":"e10b8171-6aa5-49b6-be5d-dda923e1a9f3","Type":"ContainerStarted","Data":"fb4211067674d58ee97456c820a583d84a6c686fecacdc55f466d0d4fc9f9f0c"}
May 20 14:00:42.556427 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:42.556115    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7587b89b76-r95s7"
May 20 14:00:42.573911 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:42.573851    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7587b89b76-r95s7" podStartSLOduration=2.126793386 podStartE2EDuration="4.573834696s" podCreationTimestamp="2026-05-20 14:00:38 +0000 UTC" firstStartedPulling="2026-05-20 14:00:39.178117104 +0000 UTC m=+599.732562657" lastFinishedPulling="2026-05-20 14:00:41.625158413 +0000 UTC m=+602.179603967" observedRunningTime="2026-05-20 14:00:42.572583145 +0000 UTC m=+603.127028713" watchObservedRunningTime="2026-05-20 14:00:42.573834696 +0000 UTC m=+603.128280270"
May 20 14:00:53.560959 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:00:53.560924    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7587b89b76-r95s7"
May 20 14:01:26.820665 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:26.820631    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79cbc94b89-gsf4x"]
May 20 14:01:26.823810 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:26.823790    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-gsf4x"
May 20 14:01:26.826598 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:26.826581    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qglng\""
May 20 14:01:26.833200 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:26.833178    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-gsf4x"]
May 20 14:01:26.948017 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:26.947984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vwg\" (UniqueName: \"kubernetes.io/projected/36470573-2ee8-474e-bc65-f9f357dac988-kube-api-access-n2vwg\") pod \"authorino-79cbc94b89-gsf4x\" (UID: \"36470573-2ee8-474e-bc65-f9f357dac988\") " pod="kuadrant-system/authorino-79cbc94b89-gsf4x"
May 20 14:01:27.048347 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:27.048312    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vwg\" (UniqueName: \"kubernetes.io/projected/36470573-2ee8-474e-bc65-f9f357dac988-kube-api-access-n2vwg\") pod \"authorino-79cbc94b89-gsf4x\" (UID: \"36470573-2ee8-474e-bc65-f9f357dac988\") " pod="kuadrant-system/authorino-79cbc94b89-gsf4x"
May 20 14:01:27.057600 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:27.057579    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vwg\" (UniqueName: \"kubernetes.io/projected/36470573-2ee8-474e-bc65-f9f357dac988-kube-api-access-n2vwg\") pod \"authorino-79cbc94b89-gsf4x\" (UID: \"36470573-2ee8-474e-bc65-f9f357dac988\") " pod="kuadrant-system/authorino-79cbc94b89-gsf4x"
May 20 14:01:27.132509 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:27.132474    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-gsf4x"
May 20 14:01:27.260374 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:27.260345    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-gsf4x"]
May 20 14:01:27.262967 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:01:27.262942    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36470573_2ee8_474e_bc65_f9f357dac988.slice/crio-14a62ee28cd93af05e134e514bccb2d41171db14e8eb049f5e8954d9cb63c4a4 WatchSource:0}: Error finding container 14a62ee28cd93af05e134e514bccb2d41171db14e8eb049f5e8954d9cb63c4a4: Status 404 returned error can't find the container with id 14a62ee28cd93af05e134e514bccb2d41171db14e8eb049f5e8954d9cb63c4a4
May 20 14:01:27.679180 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:27.679143    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-gsf4x" event={"ID":"36470573-2ee8-474e-bc65-f9f357dac988","Type":"ContainerStarted","Data":"14a62ee28cd93af05e134e514bccb2d41171db14e8eb049f5e8954d9cb63c4a4"}
May 20 14:01:29.686305 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:29.686271    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-gsf4x" event={"ID":"36470573-2ee8-474e-bc65-f9f357dac988","Type":"ContainerStarted","Data":"cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a"}
May 20 14:01:29.703438 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:29.703367    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-79cbc94b89-gsf4x" podStartSLOduration=1.480512206 podStartE2EDuration="3.703352085s" podCreationTimestamp="2026-05-20 14:01:26 +0000 UTC" firstStartedPulling="2026-05-20 14:01:27.264208529 +0000 UTC m=+647.818654081" lastFinishedPulling="2026-05-20 14:01:29.487048392 +0000 UTC m=+650.041493960" observedRunningTime="2026-05-20 14:01:29.702692509 +0000 UTC m=+650.257138078" watchObservedRunningTime="2026-05-20 14:01:29.703352085 +0000 UTC m=+650.257797659"
May 20 14:01:51.505157 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.505121    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68bd676465-4dv2w"]
May 20 14:01:51.508195 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.508175    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-4dv2w"
May 20 14:01:51.510811 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.510792    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
May 20 14:01:51.517547 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.517527    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-4dv2w"]
May 20 14:01:51.623775 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.623736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80129d87-90f4-43ef-90f3-3f16b9be4ec6-tls-cert\") pod \"authorino-68bd676465-4dv2w\" (UID: \"80129d87-90f4-43ef-90f3-3f16b9be4ec6\") " pod="kuadrant-system/authorino-68bd676465-4dv2w"
May 20 14:01:51.623775 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.623769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdtm9\" (UniqueName: \"kubernetes.io/projected/80129d87-90f4-43ef-90f3-3f16b9be4ec6-kube-api-access-sdtm9\") pod \"authorino-68bd676465-4dv2w\" (UID: \"80129d87-90f4-43ef-90f3-3f16b9be4ec6\") " pod="kuadrant-system/authorino-68bd676465-4dv2w"
May 20 14:01:51.724329 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.724294    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80129d87-90f4-43ef-90f3-3f16b9be4ec6-tls-cert\") pod \"authorino-68bd676465-4dv2w\" (UID: \"80129d87-90f4-43ef-90f3-3f16b9be4ec6\") " pod="kuadrant-system/authorino-68bd676465-4dv2w"
May 20 14:01:51.724329 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.724330    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sdtm9\" (UniqueName: \"kubernetes.io/projected/80129d87-90f4-43ef-90f3-3f16b9be4ec6-kube-api-access-sdtm9\") pod \"authorino-68bd676465-4dv2w\" (UID: \"80129d87-90f4-43ef-90f3-3f16b9be4ec6\") " pod="kuadrant-system/authorino-68bd676465-4dv2w"
May 20 14:01:51.726883 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.726857    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/80129d87-90f4-43ef-90f3-3f16b9be4ec6-tls-cert\") pod \"authorino-68bd676465-4dv2w\" (UID: \"80129d87-90f4-43ef-90f3-3f16b9be4ec6\") " pod="kuadrant-system/authorino-68bd676465-4dv2w"
May 20 14:01:51.733286 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.733261    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdtm9\" (UniqueName: \"kubernetes.io/projected/80129d87-90f4-43ef-90f3-3f16b9be4ec6-kube-api-access-sdtm9\") pod \"authorino-68bd676465-4dv2w\" (UID: \"80129d87-90f4-43ef-90f3-3f16b9be4ec6\") " pod="kuadrant-system/authorino-68bd676465-4dv2w"
May 20 14:01:51.816546 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.816473    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-4dv2w"
May 20 14:01:51.934913 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.934888    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-4dv2w"]
May 20 14:01:51.940359 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:01:51.940284    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80129d87_90f4_43ef_90f3_3f16b9be4ec6.slice/crio-1130c9ffb3db54a32cbb3be35d7d76ec4c268829a3880320375a2e17cd4fc4e6 WatchSource:0}: Error finding container 1130c9ffb3db54a32cbb3be35d7d76ec4c268829a3880320375a2e17cd4fc4e6: Status 404 returned error can't find the container with id 1130c9ffb3db54a32cbb3be35d7d76ec4c268829a3880320375a2e17cd4fc4e6
May 20 14:01:51.941262 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:51.941242    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:01:52.749136 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:52.749055    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-4dv2w" event={"ID":"80129d87-90f4-43ef-90f3-3f16b9be4ec6","Type":"ContainerStarted","Data":"52db66c32a9d9e183585fe02528743bebdb2b11ede6867813ba9e5dc56fa2eb1"}
May 20 14:01:52.749136 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:52.749102    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-4dv2w" event={"ID":"80129d87-90f4-43ef-90f3-3f16b9be4ec6","Type":"ContainerStarted","Data":"1130c9ffb3db54a32cbb3be35d7d76ec4c268829a3880320375a2e17cd4fc4e6"}
May 20 14:01:52.764937 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:52.764894    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-68bd676465-4dv2w" podStartSLOduration=1.2631531759999999 podStartE2EDuration="1.76488101s" podCreationTimestamp="2026-05-20 14:01:51 +0000 UTC" firstStartedPulling="2026-05-20 14:01:51.941374559 +0000 UTC m=+672.495820111" lastFinishedPulling="2026-05-20 14:01:52.443102388 +0000 UTC m=+672.997547945" observedRunningTime="2026-05-20 14:01:52.763401131 +0000 UTC m=+673.317846707" watchObservedRunningTime="2026-05-20 14:01:52.76488101 +0000 UTC m=+673.319326585"
May 20 14:01:52.790169 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:52.790136    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-gsf4x"]
May 20 14:01:52.790335 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:52.790316    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-79cbc94b89-gsf4x" podUID="36470573-2ee8-474e-bc65-f9f357dac988" containerName="authorino" containerID="cri-o://cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a" gracePeriod=30
May 20 14:01:53.027903 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.027881    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-gsf4x"
May 20 14:01:53.135899 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.135874    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2vwg\" (UniqueName: \"kubernetes.io/projected/36470573-2ee8-474e-bc65-f9f357dac988-kube-api-access-n2vwg\") pod \"36470573-2ee8-474e-bc65-f9f357dac988\" (UID: \"36470573-2ee8-474e-bc65-f9f357dac988\") "
May 20 14:01:53.138071 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.138032    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36470573-2ee8-474e-bc65-f9f357dac988-kube-api-access-n2vwg" (OuterVolumeSpecName: "kube-api-access-n2vwg") pod "36470573-2ee8-474e-bc65-f9f357dac988" (UID: "36470573-2ee8-474e-bc65-f9f357dac988"). InnerVolumeSpecName "kube-api-access-n2vwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:01:53.236525 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.236504    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vwg\" (UniqueName: \"kubernetes.io/projected/36470573-2ee8-474e-bc65-f9f357dac988-kube-api-access-n2vwg\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:01:53.752909 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.752874    2566 generic.go:358] "Generic (PLEG): container finished" podID="36470573-2ee8-474e-bc65-f9f357dac988" containerID="cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a" exitCode=0
May 20 14:01:53.753290 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.752918    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-gsf4x"
May 20 14:01:53.753290 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.752960    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-gsf4x" event={"ID":"36470573-2ee8-474e-bc65-f9f357dac988","Type":"ContainerDied","Data":"cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a"}
May 20 14:01:53.753290 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.752995    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-gsf4x" event={"ID":"36470573-2ee8-474e-bc65-f9f357dac988","Type":"ContainerDied","Data":"14a62ee28cd93af05e134e514bccb2d41171db14e8eb049f5e8954d9cb63c4a4"}
May 20 14:01:53.753290 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.753010    2566 scope.go:117] "RemoveContainer" containerID="cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a"
May 20 14:01:53.760395 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.760380    2566 scope.go:117] "RemoveContainer" containerID="cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a"
May 20 14:01:53.760638 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:01:53.760615    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a\": container with ID starting with cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a not found: ID does not exist" containerID="cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a"
May 20 14:01:53.760748 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.760642    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a"} err="failed to get container status \"cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a\": rpc error: code = NotFound desc = could not find container \"cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a\": container with ID starting with cee0cf92a4c73e0bd27bc160d03f9c4a489434076e58c42a6af58a62a2d94b4a not found: ID does not exist"
May 20 14:01:53.772331 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.772311    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-gsf4x"]
May 20 14:01:53.774396 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.774377    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-gsf4x"]
May 20 14:01:53.990786 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:01:53.990763    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36470573-2ee8-474e-bc65-f9f357dac988" path="/var/lib/kubelet/pods/36470573-2ee8-474e-bc65-f9f357dac988/volumes"
May 20 14:05:27.137118 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.137054    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"]
May 20 14:05:27.137667 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.137440    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36470573-2ee8-474e-bc65-f9f357dac988" containerName="authorino"
May 20 14:05:27.137667 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.137461    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="36470573-2ee8-474e-bc65-f9f357dac988" containerName="authorino"
May 20 14:05:27.137667 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.137544    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="36470573-2ee8-474e-bc65-f9f357dac988" containerName="authorino"
May 20 14:05:27.140687 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.140647    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.143888 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.143861    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:05:27.144998 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.144973    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:05:27.145136 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.144998    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:05:27.145136 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.144983    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:05:27.145136 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.145039    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
May 20 14:05:27.152230 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.152202    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"]
May 20 14:05:27.238043 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.237997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.238043 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.238051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.238255 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.238079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-model-cache\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.238255 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.238132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-dshm\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.238255 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.238170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-home\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.238255 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.238201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9652d75-fe46-462c-b21c-3be241b7ae7f-tls-certs\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.238255 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.238247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bg4x\" (UniqueName: \"kubernetes.io/projected/e9652d75-fe46-462c-b21c-3be241b7ae7f-kube-api-access-8bg4x\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.339276 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.339233    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8bg4x\" (UniqueName: \"kubernetes.io/projected/e9652d75-fe46-462c-b21c-3be241b7ae7f-kube-api-access-8bg4x\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.339469 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.339297    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.339469 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.339338    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.339469 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.339364    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-model-cache\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.339469 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.339394    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-dshm\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.339709 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.339522    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-home\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.339709 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.339584    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9652d75-fe46-462c-b21c-3be241b7ae7f-tls-certs\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.339849 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.339829    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.340167 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.340138    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-model-cache\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.340167 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.340159    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.340312 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.340214    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-home\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.341951 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.341923    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-dshm\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.342190 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.342174    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9652d75-fe46-462c-b21c-3be241b7ae7f-tls-certs\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.348421 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.348392    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bg4x\" (UniqueName: \"kubernetes.io/projected/e9652d75-fe46-462c-b21c-3be241b7ae7f-kube-api-access-8bg4x\") pod \"scheduler-inline-config-test-kserve-5748858bf9-zzm67\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.453275 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.453168    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:27.590494 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:27.590457    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"]
May 20 14:05:27.593366 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:05:27.593336    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9652d75_fe46_462c_b21c_3be241b7ae7f.slice/crio-d9487488c486d25ae94b189ebc7d0b0cddb3e7d9f3c46520c2a0965a7df665f3 WatchSource:0}: Error finding container d9487488c486d25ae94b189ebc7d0b0cddb3e7d9f3c46520c2a0965a7df665f3: Status 404 returned error can't find the container with id d9487488c486d25ae94b189ebc7d0b0cddb3e7d9f3c46520c2a0965a7df665f3
May 20 14:05:28.321685 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:28.321580    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67" event={"ID":"e9652d75-fe46-462c-b21c-3be241b7ae7f","Type":"ContainerStarted","Data":"d9487488c486d25ae94b189ebc7d0b0cddb3e7d9f3c46520c2a0965a7df665f3"}
May 20 14:05:31.333522 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:31.333427    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67" event={"ID":"e9652d75-fe46-462c-b21c-3be241b7ae7f","Type":"ContainerStarted","Data":"df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3"}
May 20 14:05:43.369767 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:43.369730    2566 generic.go:358] "Generic (PLEG): container finished" podID="e9652d75-fe46-462c-b21c-3be241b7ae7f" containerID="df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3" exitCode=0
May 20 14:05:43.370174 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:43.369790    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67" event={"ID":"e9652d75-fe46-462c-b21c-3be241b7ae7f","Type":"ContainerDied","Data":"df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3"}
May 20 14:05:45.377295 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:45.377257    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67" event={"ID":"e9652d75-fe46-462c-b21c-3be241b7ae7f","Type":"ContainerStarted","Data":"4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b"}
May 20 14:05:45.397776 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:45.397717    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67" podStartSLOduration=0.790841875 podStartE2EDuration="18.397701685s" podCreationTimestamp="2026-05-20 14:05:27 +0000 UTC" firstStartedPulling="2026-05-20 14:05:27.595318732 +0000 UTC m=+888.149764299" lastFinishedPulling="2026-05-20 14:05:45.202178553 +0000 UTC m=+905.756624109" observedRunningTime="2026-05-20 14:05:45.395271427 +0000 UTC m=+905.949717012" watchObservedRunningTime="2026-05-20 14:05:45.397701685 +0000 UTC m=+905.952147259"
May 20 14:05:47.453927 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:47.453885    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:47.453927 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:47.453937    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:47.466832 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:47.466800    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:05:48.397083 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:05:48.397049    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:06:19.496205 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.496171    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"]
May 20 14:06:19.502106 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.496957    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67" podUID="e9652d75-fe46-462c-b21c-3be241b7ae7f" containerName="main" containerID="cri-o://4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b" gracePeriod=30
May 20 14:06:19.739499 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.739475    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:06:19.786638 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.786572    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-dshm\") pod \"e9652d75-fe46-462c-b21c-3be241b7ae7f\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") "
May 20 14:06:19.786638 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.786611    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bg4x\" (UniqueName: \"kubernetes.io/projected/e9652d75-fe46-462c-b21c-3be241b7ae7f-kube-api-access-8bg4x\") pod \"e9652d75-fe46-462c-b21c-3be241b7ae7f\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") "
May 20 14:06:19.786869 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.786648    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-model-cache\") pod \"e9652d75-fe46-462c-b21c-3be241b7ae7f\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") "
May 20 14:06:19.786869 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.786666    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-home\") pod \"e9652d75-fe46-462c-b21c-3be241b7ae7f\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") "
May 20 14:06:19.786869 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.786716    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-kserve-provision-location\") pod \"e9652d75-fe46-462c-b21c-3be241b7ae7f\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") "
May 20 14:06:19.786869 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.786732    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9652d75-fe46-462c-b21c-3be241b7ae7f-tls-certs\") pod \"e9652d75-fe46-462c-b21c-3be241b7ae7f\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") "
May 20 14:06:19.786869 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.786799    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-tmp-dir\") pod \"e9652d75-fe46-462c-b21c-3be241b7ae7f\" (UID: \"e9652d75-fe46-462c-b21c-3be241b7ae7f\") "
May 20 14:06:19.787128 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.786963    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-home" (OuterVolumeSpecName: "home") pod "e9652d75-fe46-462c-b21c-3be241b7ae7f" (UID: "e9652d75-fe46-462c-b21c-3be241b7ae7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:19.787128 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.787079    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:06:19.787128 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.787117    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9652d75-fe46-462c-b21c-3be241b7ae7f" (UID: "e9652d75-fe46-462c-b21c-3be241b7ae7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:19.787281 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.787198    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9652d75-fe46-462c-b21c-3be241b7ae7f" (UID: "e9652d75-fe46-462c-b21c-3be241b7ae7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:19.788960 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.788939    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9652d75-fe46-462c-b21c-3be241b7ae7f-kube-api-access-8bg4x" (OuterVolumeSpecName: "kube-api-access-8bg4x") pod "e9652d75-fe46-462c-b21c-3be241b7ae7f" (UID: "e9652d75-fe46-462c-b21c-3be241b7ae7f"). InnerVolumeSpecName "kube-api-access-8bg4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:06:19.789088 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.789022    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9652d75-fe46-462c-b21c-3be241b7ae7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9652d75-fe46-462c-b21c-3be241b7ae7f" (UID: "e9652d75-fe46-462c-b21c-3be241b7ae7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:06:19.789088 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.789075    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-dshm" (OuterVolumeSpecName: "dshm") pod "e9652d75-fe46-462c-b21c-3be241b7ae7f" (UID: "e9652d75-fe46-462c-b21c-3be241b7ae7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:19.842311 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.842273    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9652d75-fe46-462c-b21c-3be241b7ae7f" (UID: "e9652d75-fe46-462c-b21c-3be241b7ae7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:19.888021 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.887990    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-kserve-provision-location\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:06:19.888021 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.888018    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9652d75-fe46-462c-b21c-3be241b7ae7f-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:06:19.888181 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.888031    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:06:19.888181 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.888040    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:06:19.888181 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.888049    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bg4x\" (UniqueName: \"kubernetes.io/projected/e9652d75-fe46-462c-b21c-3be241b7ae7f-kube-api-access-8bg4x\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:06:19.888181 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:19.888059    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9652d75-fe46-462c-b21c-3be241b7ae7f-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:06:20.474779 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.474744    2566 generic.go:358] "Generic (PLEG): container finished" podID="e9652d75-fe46-462c-b21c-3be241b7ae7f" containerID="4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b" exitCode=0
May 20 14:06:20.474985 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.474834    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67" event={"ID":"e9652d75-fe46-462c-b21c-3be241b7ae7f","Type":"ContainerDied","Data":"4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b"}
May 20 14:06:20.474985 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.474859    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"
May 20 14:06:20.474985 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.474876    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67" event={"ID":"e9652d75-fe46-462c-b21c-3be241b7ae7f","Type":"ContainerDied","Data":"d9487488c486d25ae94b189ebc7d0b0cddb3e7d9f3c46520c2a0965a7df665f3"}
May 20 14:06:20.474985 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.474893    2566 scope.go:117] "RemoveContainer" containerID="4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b"
May 20 14:06:20.482424 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.482404    2566 scope.go:117] "RemoveContainer" containerID="df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3"
May 20 14:06:20.492756 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.492728    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"]
May 20 14:06:20.495397 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.495374    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-5748858bf9-zzm67"]
May 20 14:06:20.541633 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.541465    2566 scope.go:117] "RemoveContainer" containerID="4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b"
May 20 14:06:20.541905 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:06:20.541813    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b\": container with ID starting with 4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b not found: ID does not exist" containerID="4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b"
May 20 14:06:20.541905 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.541839    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b"} err="failed to get container status \"4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b\": rpc error: code = NotFound desc = could not find container \"4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b\": container with ID starting with 4004271e2083a7d45bc7f70d0ab2bb45dbc97633038de80e0a65550c34f4580b not found: ID does not exist"
May 20 14:06:20.541905 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.541858    2566 scope.go:117] "RemoveContainer" containerID="df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3"
May 20 14:06:20.542117 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:06:20.542102    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3\": container with ID starting with df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3 not found: ID does not exist" containerID="df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3"
May 20 14:06:20.542213 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:20.542121    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3"} err="failed to get container status \"df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3\": rpc error: code = NotFound desc = could not find container \"df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3\": container with ID starting with df6579126832c4f9ee08bb36dd9ad01ef4587623102c18c5e7acb167c042b1f3 not found: ID does not exist"
May 20 14:06:21.990234 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:21.990197    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9652d75-fe46-462c-b21c-3be241b7ae7f" path="/var/lib/kubelet/pods/e9652d75-fe46-462c-b21c-3be241b7ae7f/volumes"
May 20 14:06:43.145134 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.145103    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"]
May 20 14:06:43.145498 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.145365    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9652d75-fe46-462c-b21c-3be241b7ae7f" containerName="main"
May 20 14:06:43.145498 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.145376    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9652d75-fe46-462c-b21c-3be241b7ae7f" containerName="main"
May 20 14:06:43.145498 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.145401    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9652d75-fe46-462c-b21c-3be241b7ae7f" containerName="storage-initializer"
May 20 14:06:43.145498 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.145406    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9652d75-fe46-462c-b21c-3be241b7ae7f" containerName="storage-initializer"
May 20 14:06:43.145498 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.145450    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9652d75-fe46-462c-b21c-3be241b7ae7f" containerName="main"
May 20 14:06:43.148648 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.148627    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.151580 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.151555    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:06:43.152755 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.152733    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:06:43.152944 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.152924    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
May 20 14:06:43.153042 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.152939    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:06:43.153042 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.152924    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:06:43.157959 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.157396    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"]
May 20 14:06:43.272412 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.272386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-home\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.272573 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.272423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.272573 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.272459    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.272573 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.272494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-model-cache\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.272573 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.272523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-dshm\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.272794 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.272595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c401db-8257-4685-9bbb-01be0ded1786-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.272794 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.272628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxn5l\" (UniqueName: \"kubernetes.io/projected/f0c401db-8257-4685-9bbb-01be0ded1786-kube-api-access-qxn5l\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.373905 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.373879    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qxn5l\" (UniqueName: \"kubernetes.io/projected/f0c401db-8257-4685-9bbb-01be0ded1786-kube-api-access-qxn5l\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.374014 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.373923    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-home\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.374014 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.373953    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.374014 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.373979    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.374014 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.373997    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-model-cache\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.374215 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.374016    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-dshm\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.374215 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.374049    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c401db-8257-4685-9bbb-01be0ded1786-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.374361 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.374339    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-home\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.374420 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.374355    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.374420 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.374402    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.374496 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.374422    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-model-cache\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.376277 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.376256    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-dshm\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.376486 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.376468    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c401db-8257-4685-9bbb-01be0ded1786-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.382901 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.382879    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxn5l\" (UniqueName: \"kubernetes.io/projected/f0c401db-8257-4685-9bbb-01be0ded1786-kube-api-access-qxn5l\") pod \"scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.461085 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.461009    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:43.587506 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:43.587480    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"]
May 20 14:06:43.590150 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:06:43.590122    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c401db_8257_4685_9bbb_01be0ded1786.slice/crio-9a9f9f2a2594b6dd76723cdc3a9d4a2ae2e6f57596800d6c694da5807bfe2262 WatchSource:0}: Error finding container 9a9f9f2a2594b6dd76723cdc3a9d4a2ae2e6f57596800d6c694da5807bfe2262: Status 404 returned error can't find the container with id 9a9f9f2a2594b6dd76723cdc3a9d4a2ae2e6f57596800d6c694da5807bfe2262
May 20 14:06:44.542254 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:44.542209    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb" event={"ID":"f0c401db-8257-4685-9bbb-01be0ded1786","Type":"ContainerStarted","Data":"8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a"}
May 20 14:06:44.542254 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:44.542252    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb" event={"ID":"f0c401db-8257-4685-9bbb-01be0ded1786","Type":"ContainerStarted","Data":"9a9f9f2a2594b6dd76723cdc3a9d4a2ae2e6f57596800d6c694da5807bfe2262"}
May 20 14:06:47.551067 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:47.551041    2566 generic.go:358] "Generic (PLEG): container finished" podID="f0c401db-8257-4685-9bbb-01be0ded1786" containerID="8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a" exitCode=0
May 20 14:06:47.551380 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:47.551112    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb" event={"ID":"f0c401db-8257-4685-9bbb-01be0ded1786","Type":"ContainerDied","Data":"8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a"}
May 20 14:06:48.555128 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:48.555094    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb" event={"ID":"f0c401db-8257-4685-9bbb-01be0ded1786","Type":"ContainerStarted","Data":"3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268"}
May 20 14:06:48.574834 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:48.574783    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb" podStartSLOduration=5.574766167 podStartE2EDuration="5.574766167s" podCreationTimestamp="2026-05-20 14:06:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:06:48.574004363 +0000 UTC m=+969.128449960" watchObservedRunningTime="2026-05-20 14:06:48.574766167 +0000 UTC m=+969.129211742"
May 20 14:06:53.462092 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:53.462062    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:53.462506 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:53.462215    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:53.474828 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:53.474800    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:06:53.581089 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:06:53.581066    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:07:15.985691 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:15.985602    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"]
May 20 14:07:15.986065 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:15.985931    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb" podUID="f0c401db-8257-4685-9bbb-01be0ded1786" containerName="main" containerID="cri-o://3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268" gracePeriod=30
May 20 14:07:16.227742 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.227719    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:07:16.306028 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.305960    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-tmp-dir\") pod \"f0c401db-8257-4685-9bbb-01be0ded1786\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") "
May 20 14:07:16.306028 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306007    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-home\") pod \"f0c401db-8257-4685-9bbb-01be0ded1786\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") "
May 20 14:07:16.306214 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306032    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c401db-8257-4685-9bbb-01be0ded1786-tls-certs\") pod \"f0c401db-8257-4685-9bbb-01be0ded1786\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") "
May 20 14:07:16.306214 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306065    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-dshm\") pod \"f0c401db-8257-4685-9bbb-01be0ded1786\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") "
May 20 14:07:16.306214 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306091    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxn5l\" (UniqueName: \"kubernetes.io/projected/f0c401db-8257-4685-9bbb-01be0ded1786-kube-api-access-qxn5l\") pod \"f0c401db-8257-4685-9bbb-01be0ded1786\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") "
May 20 14:07:16.306214 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306130    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-model-cache\") pod \"f0c401db-8257-4685-9bbb-01be0ded1786\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") "
May 20 14:07:16.306214 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306190    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-kserve-provision-location\") pod \"f0c401db-8257-4685-9bbb-01be0ded1786\" (UID: \"f0c401db-8257-4685-9bbb-01be0ded1786\") "
May 20 14:07:16.306458 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306223    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0c401db-8257-4685-9bbb-01be0ded1786" (UID: "f0c401db-8257-4685-9bbb-01be0ded1786"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:16.306458 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306340    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-home" (OuterVolumeSpecName: "home") pod "f0c401db-8257-4685-9bbb-01be0ded1786" (UID: "f0c401db-8257-4685-9bbb-01be0ded1786"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:16.306458 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306426    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0c401db-8257-4685-9bbb-01be0ded1786" (UID: "f0c401db-8257-4685-9bbb-01be0ded1786"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:16.306609 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306454    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:07:16.306609 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.306481    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:07:16.308224 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.308197    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0c401db-8257-4685-9bbb-01be0ded1786-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0c401db-8257-4685-9bbb-01be0ded1786" (UID: "f0c401db-8257-4685-9bbb-01be0ded1786"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:07:16.308224 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.308198    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c401db-8257-4685-9bbb-01be0ded1786-kube-api-access-qxn5l" (OuterVolumeSpecName: "kube-api-access-qxn5l") pod "f0c401db-8257-4685-9bbb-01be0ded1786" (UID: "f0c401db-8257-4685-9bbb-01be0ded1786"). InnerVolumeSpecName "kube-api-access-qxn5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:07:16.308362 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.308208    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-dshm" (OuterVolumeSpecName: "dshm") pod "f0c401db-8257-4685-9bbb-01be0ded1786" (UID: "f0c401db-8257-4685-9bbb-01be0ded1786"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:16.366486 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.366460    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0c401db-8257-4685-9bbb-01be0ded1786" (UID: "f0c401db-8257-4685-9bbb-01be0ded1786"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:16.407462 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.407444    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-kserve-provision-location\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:07:16.407556 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.407463    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c401db-8257-4685-9bbb-01be0ded1786-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:07:16.407556 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.407475    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:07:16.407556 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.407484    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxn5l\" (UniqueName: \"kubernetes.io/projected/f0c401db-8257-4685-9bbb-01be0ded1786-kube-api-access-qxn5l\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:07:16.407556 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.407493    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c401db-8257-4685-9bbb-01be0ded1786-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:07:16.634401 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.634377    2566 generic.go:358] "Generic (PLEG): container finished" podID="f0c401db-8257-4685-9bbb-01be0ded1786" containerID="3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268" exitCode=0
May 20 14:07:16.634506 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.634446    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"
May 20 14:07:16.634547 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.634447    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb" event={"ID":"f0c401db-8257-4685-9bbb-01be0ded1786","Type":"ContainerDied","Data":"3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268"}
May 20 14:07:16.634584 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.634550    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb" event={"ID":"f0c401db-8257-4685-9bbb-01be0ded1786","Type":"ContainerDied","Data":"9a9f9f2a2594b6dd76723cdc3a9d4a2ae2e6f57596800d6c694da5807bfe2262"}
May 20 14:07:16.634584 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.634565    2566 scope.go:117] "RemoveContainer" containerID="3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268"
May 20 14:07:16.645287 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.643793    2566 scope.go:117] "RemoveContainer" containerID="8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a"
May 20 14:07:16.659265 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.659246    2566 scope.go:117] "RemoveContainer" containerID="3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268"
May 20 14:07:16.659562 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:07:16.659545    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268\": container with ID starting with 3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268 not found: ID does not exist" containerID="3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268"
May 20 14:07:16.659630 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.659569    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268"} err="failed to get container status \"3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268\": rpc error: code = NotFound desc = could not find container \"3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268\": container with ID starting with 3d088d30ae9c37f069bf02beb3d574c63aa8c100a949437ab6987bb340b02268 not found: ID does not exist"
May 20 14:07:16.659630 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.659588    2566 scope.go:117] "RemoveContainer" containerID="8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a"
May 20 14:07:16.659841 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.659821    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"]
May 20 14:07:16.659903 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:07:16.659851    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a\": container with ID starting with 8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a not found: ID does not exist" containerID="8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a"
May 20 14:07:16.659903 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.659884    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a"} err="failed to get container status \"8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a\": rpc error: code = NotFound desc = could not find container \"8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a\": container with ID starting with 8889e4c145acc1ed2d4b36a5883324bdd3da42bef5a9715bb1374606ec73705a not found: ID does not exist"
May 20 14:07:16.663302 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:16.663280    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-59c9cf8475-gjmqb"]
May 20 14:07:17.990519 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:17.990485    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0c401db-8257-4685-9bbb-01be0ded1786" path="/var/lib/kubelet/pods/f0c401db-8257-4685-9bbb-01be0ded1786/volumes"
May 20 14:07:20.674172 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.674142    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"]
May 20 14:07:20.674575 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.674372    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c401db-8257-4685-9bbb-01be0ded1786" containerName="storage-initializer"
May 20 14:07:20.674575 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.674382    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c401db-8257-4685-9bbb-01be0ded1786" containerName="storage-initializer"
May 20 14:07:20.674575 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.674396    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c401db-8257-4685-9bbb-01be0ded1786" containerName="main"
May 20 14:07:20.674575 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.674402    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c401db-8257-4685-9bbb-01be0ded1786" containerName="main"
May 20 14:07:20.674575 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.674448    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="f0c401db-8257-4685-9bbb-01be0ded1786" containerName="main"
May 20 14:07:20.677647 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.677626    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.680573 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.680551    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:07:20.680833 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.680816    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:07:20.680937 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.680922    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:07:20.681000 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.680922    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:07:20.681941 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.681925    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
May 20 14:07:20.688049 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.688030    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"]
May 20 14:07:20.734621 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.734598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.734761 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.734624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48lqp\" (UniqueName: \"kubernetes.io/projected/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kube-api-access-48lqp\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.734761 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.734652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-dshm\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.734761 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.734740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-model-cache\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.734872 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.734764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.734872 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.734782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.734872 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.734802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-home\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.835705 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.835657    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.835791 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.835729    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.835791 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.835748    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-home\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.835791 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.835774    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.835909 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.835791    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-48lqp\" (UniqueName: \"kubernetes.io/projected/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kube-api-access-48lqp\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.835909 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.835837    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-dshm\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.835909 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.835877    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-model-cache\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.836044 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.835961    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.836165 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.836142    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.836165 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.836161    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-model-cache\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.836281 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.836226    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-home\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.837834 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.837815    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-dshm\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.838000 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.837985    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.843592 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.843571    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48lqp\" (UniqueName: \"kubernetes.io/projected/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kube-api-access-48lqp\") pod \"scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.987598 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.987537    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:07:20.998097 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:20.998074    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"]
May 20 14:07:21.001100 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.001083    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.004123 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.004105    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-l4hpq\""
May 20 14:07:21.016944 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.016918    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"]
May 20 14:07:21.037786 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.037335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.037786 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.037473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2dxs\" (UniqueName: \"kubernetes.io/projected/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kube-api-access-f2dxs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.037786 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.037515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.037786 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.037560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.037786 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.037622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.037786 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.037693    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.111126 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.111105    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"]
May 20 14:07:21.113504 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:07:21.113480    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb7a9c9_f66d_4f96_b24d_eb70cd18bea4.slice/crio-0b2ae8d5f7151dab7294af5a8e6ae10d6f23a84a3c51d9edacebc42a36a1ec00 WatchSource:0}: Error finding container 0b2ae8d5f7151dab7294af5a8e6ae10d6f23a84a3c51d9edacebc42a36a1ec00: Status 404 returned error can't find the container with id 0b2ae8d5f7151dab7294af5a8e6ae10d6f23a84a3c51d9edacebc42a36a1ec00
May 20 14:07:21.115182 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.115164    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:07:21.139080 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.139057    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.139172 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.139102    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.139172 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.139152    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.139280 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.139190    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dxs\" (UniqueName: \"kubernetes.io/projected/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kube-api-access-f2dxs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.139280 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.139227    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.139280 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.139269    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.139413 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.139385    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.139476 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.139454    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.139534 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.139497    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.139586 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.139551    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.141441 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.141421    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.148431 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.148412    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dxs\" (UniqueName: \"kubernetes.io/projected/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kube-api-access-f2dxs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.312586 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.312521    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:21.450262 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.450233    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"]
May 20 14:07:21.452050 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:07:21.452020    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c01ecf4_b17c_4ac3_aff3_fc7783615098.slice/crio-a7c5833428467d1a5e90abf352eff4d55a92757374eed326e0bcbbe6b60d4b6e WatchSource:0}: Error finding container a7c5833428467d1a5e90abf352eff4d55a92757374eed326e0bcbbe6b60d4b6e: Status 404 returned error can't find the container with id a7c5833428467d1a5e90abf352eff4d55a92757374eed326e0bcbbe6b60d4b6e
May 20 14:07:21.649446 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.649410    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" event={"ID":"9c01ecf4-b17c-4ac3-aff3-fc7783615098","Type":"ContainerStarted","Data":"add0f50d9615a08365638123e8d63bced4286489a69f610b243303c925087184"}
May 20 14:07:21.649621 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.649455    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" event={"ID":"9c01ecf4-b17c-4ac3-aff3-fc7783615098","Type":"ContainerStarted","Data":"a7c5833428467d1a5e90abf352eff4d55a92757374eed326e0bcbbe6b60d4b6e"}
May 20 14:07:21.650818 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.650784    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx" event={"ID":"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4","Type":"ContainerStarted","Data":"65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2"}
May 20 14:07:21.650818 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:21.650820    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx" event={"ID":"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4","Type":"ContainerStarted","Data":"0b2ae8d5f7151dab7294af5a8e6ae10d6f23a84a3c51d9edacebc42a36a1ec00"}
May 20 14:07:22.654953 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:22.654918    2566 generic.go:358] "Generic (PLEG): container finished" podID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerID="add0f50d9615a08365638123e8d63bced4286489a69f610b243303c925087184" exitCode=0
May 20 14:07:22.655326 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:22.655009    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" event={"ID":"9c01ecf4-b17c-4ac3-aff3-fc7783615098","Type":"ContainerDied","Data":"add0f50d9615a08365638123e8d63bced4286489a69f610b243303c925087184"}
May 20 14:07:23.659133 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:23.659102    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" event={"ID":"9c01ecf4-b17c-4ac3-aff3-fc7783615098","Type":"ContainerStarted","Data":"f383a6248f9624c4879663eba4c45eac14e24799bcd585a100eba52d62fdd5fc"}
May 20 14:07:52.754073 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:52.754038    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" event={"ID":"9c01ecf4-b17c-4ac3-aff3-fc7783615098","Type":"ContainerStarted","Data":"8d25f40de1493e6dcbf06d3f2b6511c66f4ddea00065910526c60a6167900bce"}
May 20 14:07:52.754556 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:52.754206    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:07:52.778208 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:52.778160    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" podStartSLOduration=3.236372387 podStartE2EDuration="32.778145356s" podCreationTimestamp="2026-05-20 14:07:20 +0000 UTC" firstStartedPulling="2026-05-20 14:07:22.656252981 +0000 UTC m=+1003.210698534" lastFinishedPulling="2026-05-20 14:07:52.198025951 +0000 UTC m=+1032.752471503" observedRunningTime="2026-05-20 14:07:52.776742101 +0000 UTC m=+1033.331187674" watchObservedRunningTime="2026-05-20 14:07:52.778145356 +0000 UTC m=+1033.332590930"
May 20 14:07:53.759547 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:07:53.759506    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 14:08:01.312928 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:01.312892    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:08:01.313364 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:01.312939    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:08:01.314460 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:01.314428    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 14:08:01.314571 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:01.314547    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:08:01.777359 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:01.777316    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 14:08:01.777529 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:01.777478    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:08:02.780164 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:02.780128    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 14:08:12.780907 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:12.780873    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 14:08:16.816896 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:16.816862    2566 generic.go:358] "Generic (PLEG): container finished" podID="7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" containerID="65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2" exitCode=0
May 20 14:08:16.817202 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:16.816933    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx" event={"ID":"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4","Type":"ContainerDied","Data":"65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2"}
May 20 14:08:17.821479 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:17.821447    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx" event={"ID":"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4","Type":"ContainerStarted","Data":"286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24"}
May 20 14:08:17.844969 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:17.844928    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx" podStartSLOduration=57.844918651 podStartE2EDuration="57.844918651s" podCreationTimestamp="2026-05-20 14:07:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:08:17.84357388 +0000 UTC m=+1058.398019452" watchObservedRunningTime="2026-05-20 14:08:17.844918651 +0000 UTC m=+1058.399364221"
May 20 14:08:20.988566 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:20.988530    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:08:20.988566 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:20.988569    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:08:21.001282 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:21.001262    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:08:21.842687 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:21.842653    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:08:22.781007 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:22.780968    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 14:08:22.819417 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:22.819372    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"]
May 20 14:08:22.820638 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:22.820612    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"]
May 20 14:08:22.820969 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:22.820943    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main" containerID="cri-o://f383a6248f9624c4879663eba4c45eac14e24799bcd585a100eba52d62fdd5fc" gracePeriod=30
May 20 14:08:22.821057 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:22.821016    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="tokenizer" containerID="cri-o://8d25f40de1493e6dcbf06d3f2b6511c66f4ddea00065910526c60a6167900bce" gracePeriod=30
May 20 14:08:22.822799 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:22.822772    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 14:08:22.947383 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:08:22.947352    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
May 20 14:08:22.947530 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:08:22.947421    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tls-certs podName:7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4 nodeName:}" failed. No retries permitted until 2026-05-20 14:08:23.447403641 +0000 UTC m=+1064.001849195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tls-certs") pod "scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx" (UID: "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
May 20 14:08:23.452690 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:08:23.452628    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
May 20 14:08:23.452898 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:08:23.452725    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tls-certs podName:7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4 nodeName:}" failed. No retries permitted until 2026-05-20 14:08:24.452710518 +0000 UTC m=+1065.007156071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tls-certs") pod "scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx" (UID: "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
May 20 14:08:23.838906 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:23.838877    2566 generic.go:358] "Generic (PLEG): container finished" podID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerID="8d25f40de1493e6dcbf06d3f2b6511c66f4ddea00065910526c60a6167900bce" exitCode=0
May 20 14:08:23.838906 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:23.838901    2566 generic.go:358] "Generic (PLEG): container finished" podID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerID="f383a6248f9624c4879663eba4c45eac14e24799bcd585a100eba52d62fdd5fc" exitCode=0
May 20 14:08:23.839274 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:23.838949    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" event={"ID":"9c01ecf4-b17c-4ac3-aff3-fc7783615098","Type":"ContainerDied","Data":"8d25f40de1493e6dcbf06d3f2b6511c66f4ddea00065910526c60a6167900bce"}
May 20 14:08:23.839274 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:23.838982    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" event={"ID":"9c01ecf4-b17c-4ac3-aff3-fc7783615098","Type":"ContainerDied","Data":"f383a6248f9624c4879663eba4c45eac14e24799bcd585a100eba52d62fdd5fc"}
May 20 14:08:23.839274 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:23.839171    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx" podUID="7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" containerName="main" containerID="cri-o://286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24" gracePeriod=30
May 20 14:08:23.956938 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:23.956912    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:08:24.057792 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.057761    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-cache\") pod \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") "
May 20 14:08:24.057931 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.057805    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-uds\") pod \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") "
May 20 14:08:24.057931 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.057846    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-tmp\") pod \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") "
May 20 14:08:24.057931 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.057871    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2dxs\" (UniqueName: \"kubernetes.io/projected/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kube-api-access-f2dxs\") pod \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") "
May 20 14:08:24.057931 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.057908    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tls-certs\") pod \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") "
May 20 14:08:24.058180 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.057937    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kserve-provision-location\") pod \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\" (UID: \"9c01ecf4-b17c-4ac3-aff3-fc7783615098\") "
May 20 14:08:24.058180 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.058061    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "9c01ecf4-b17c-4ac3-aff3-fc7783615098" (UID: "9c01ecf4-b17c-4ac3-aff3-fc7783615098"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.058180 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.058151    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9c01ecf4-b17c-4ac3-aff3-fc7783615098" (UID: "9c01ecf4-b17c-4ac3-aff3-fc7783615098"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.058325 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.058266    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "9c01ecf4-b17c-4ac3-aff3-fc7783615098" (UID: "9c01ecf4-b17c-4ac3-aff3-fc7783615098"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.058325 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.058315    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.058441 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.058335    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-uds\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.058792 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.058744    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c01ecf4-b17c-4ac3-aff3-fc7783615098" (UID: "9c01ecf4-b17c-4ac3-aff3-fc7783615098"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.060028 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.060007    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kube-api-access-f2dxs" (OuterVolumeSpecName: "kube-api-access-f2dxs") pod "9c01ecf4-b17c-4ac3-aff3-fc7783615098" (UID: "9c01ecf4-b17c-4ac3-aff3-fc7783615098"). InnerVolumeSpecName "kube-api-access-f2dxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:08:24.060103 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.060087    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c01ecf4-b17c-4ac3-aff3-fc7783615098" (UID: "9c01ecf4-b17c-4ac3-aff3-fc7783615098"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:08:24.096030 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.096007    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:08:24.159220 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159196    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kserve-provision-location\") pod \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") "
May 20 14:08:24.159350 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159229    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-home\") pod \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") "
May 20 14:08:24.159350 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159254    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tmp-dir\") pod \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") "
May 20 14:08:24.159350 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159283    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-model-cache\") pod \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") "
May 20 14:08:24.159350 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159306    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-dshm\") pod \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") "
May 20 14:08:24.159350 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159337    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48lqp\" (UniqueName: \"kubernetes.io/projected/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kube-api-access-48lqp\") pod \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") "
May 20 14:08:24.159604 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159380    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tls-certs\") pod \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\" (UID: \"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4\") "
May 20 14:08:24.159604 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159502    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-home" (OuterVolumeSpecName: "home") pod "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" (UID: "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.159604 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159514    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" (UID: "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.159604 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159540    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" (UID: "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.159824 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159624    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tokenizer-tmp\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.159824 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159654    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2dxs\" (UniqueName: \"kubernetes.io/projected/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kube-api-access-f2dxs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.159824 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159696    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c01ecf4-b17c-4ac3-aff3-fc7783615098-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.159824 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159714    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.159824 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159728    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c01ecf4-b17c-4ac3-aff3-fc7783615098-kserve-provision-location\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.159824 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159744    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.159824 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.159757    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.161356 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.161327    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kube-api-access-48lqp" (OuterVolumeSpecName: "kube-api-access-48lqp") pod "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" (UID: "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4"). InnerVolumeSpecName "kube-api-access-48lqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:08:24.161473 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.161426    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-dshm" (OuterVolumeSpecName: "dshm") pod "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" (UID: "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.161656 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.161639    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" (UID: "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:08:24.214422 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.214392    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" (UID: "7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.260645 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.260586    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.260645 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.260607    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48lqp\" (UniqueName: \"kubernetes.io/projected/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kube-api-access-48lqp\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.260645 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.260617    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.260645 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.260626    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4-kserve-provision-location\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:08:24.848992 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.848951    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl" event={"ID":"9c01ecf4-b17c-4ac3-aff3-fc7783615098","Type":"ContainerDied","Data":"a7c5833428467d1a5e90abf352eff4d55a92757374eed326e0bcbbe6b60d4b6e"}
May 20 14:08:24.849446 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.849012    2566 scope.go:117] "RemoveContainer" containerID="8d25f40de1493e6dcbf06d3f2b6511c66f4ddea00065910526c60a6167900bce"
May 20 14:08:24.849446 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.849014    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"
May 20 14:08:24.850459 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.850434    2566 generic.go:358] "Generic (PLEG): container finished" podID="7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" containerID="286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24" exitCode=0
May 20 14:08:24.850547 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.850482    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx" event={"ID":"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4","Type":"ContainerDied","Data":"286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24"}
May 20 14:08:24.850547 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.850509    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx" event={"ID":"7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4","Type":"ContainerDied","Data":"0b2ae8d5f7151dab7294af5a8e6ae10d6f23a84a3c51d9edacebc42a36a1ec00"}
May 20 14:08:24.850547 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.850525    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"
May 20 14:08:24.861424 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.861408    2566 scope.go:117] "RemoveContainer" containerID="f383a6248f9624c4879663eba4c45eac14e24799bcd585a100eba52d62fdd5fc"
May 20 14:08:24.868123 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.868107    2566 scope.go:117] "RemoveContainer" containerID="add0f50d9615a08365638123e8d63bced4286489a69f610b243303c925087184"
May 20 14:08:24.874976 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.874957    2566 scope.go:117] "RemoveContainer" containerID="286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24"
May 20 14:08:24.875367 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.875345    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"]
May 20 14:08:24.878825 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.878803    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8smmhl"]
May 20 14:08:24.881710 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.881693    2566 scope.go:117] "RemoveContainer" containerID="65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2"
May 20 14:08:24.892546 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.892527    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"]
May 20 14:08:24.895704 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.895664    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-865c7d9d55-dgjrx"]
May 20 14:08:24.941107 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.941089    2566 scope.go:117] "RemoveContainer" containerID="286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24"
May 20 14:08:24.941353 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:08:24.941336    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24\": container with ID starting with 286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24 not found: ID does not exist" containerID="286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24"
May 20 14:08:24.941400 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.941362    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24"} err="failed to get container status \"286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24\": rpc error: code = NotFound desc = could not find container \"286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24\": container with ID starting with 286c6e0b63ad704969eac8f7a6b83ef534cc7137c4885a62d5f61c185ad67f24 not found: ID does not exist"
May 20 14:08:24.941400 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.941380    2566 scope.go:117] "RemoveContainer" containerID="65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2"
May 20 14:08:24.941647 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:08:24.941624    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2\": container with ID starting with 65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2 not found: ID does not exist" containerID="65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2"
May 20 14:08:24.941738 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:24.941651    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2"} err="failed to get container status \"65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2\": rpc error: code = NotFound desc = could not find container \"65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2\": container with ID starting with 65f35efad424f6cee81a019611a46c78cc7ce24bba76f1240917915905806ff2 not found: ID does not exist"
May 20 14:08:25.990152 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:25.990118    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" path="/var/lib/kubelet/pods/7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4/volumes"
May 20 14:08:25.990522 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:25.990505    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" path="/var/lib/kubelet/pods/9c01ecf4-b17c-4ac3-aff3-fc7783615098/volumes"
May 20 14:08:55.141853 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.141815    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"]
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142098    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="tokenizer"
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142109    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="tokenizer"
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142118    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" containerName="storage-initializer"
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142124    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" containerName="storage-initializer"
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142133    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="storage-initializer"
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142139    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="storage-initializer"
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142147    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" containerName="main"
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142155    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" containerName="main"
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142180    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main"
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142188    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main"
May 20 14:08:55.142228 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142234    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="7fb7a9c9-f66d-4f96-b24d-eb70cd18bea4" containerName="main"
May 20 14:08:55.142577 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142242    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="tokenizer"
May 20 14:08:55.142577 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.142251    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="9c01ecf4-b17c-4ac3-aff3-fc7783615098" containerName="main"
May 20 14:08:55.146371 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.146352    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.149769 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.149654    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:08:55.150967 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.150947    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:08:55.151181 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.150949    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:08:55.151304 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.150956    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
May 20 14:08:55.151304 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.150965    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:08:55.157775 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.157753    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"]
May 20 14:08:55.296296 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.296271    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-home\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.296416 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.296304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.296416 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.296326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-model-cache\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.296416 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.296368    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-dshm\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.296416 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.296403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6x7r\" (UniqueName: \"kubernetes.io/projected/370408d6-ff67-4924-bbbb-ef87b813a6ed-kube-api-access-w6x7r\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.296577 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.296421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.296577 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.296475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370408d6-ff67-4924-bbbb-ef87b813a6ed-tls-certs\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397295 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397243    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370408d6-ff67-4924-bbbb-ef87b813a6ed-tls-certs\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397295 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397274    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-home\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397418 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397297    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397457 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397422    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-model-cache\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397498 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397455    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-dshm\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397498 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397481    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6x7r\" (UniqueName: \"kubernetes.io/projected/370408d6-ff67-4924-bbbb-ef87b813a6ed-kube-api-access-w6x7r\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397594 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397553    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397648 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397619    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-home\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397734 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397668    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397802 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397756    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-model-cache\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.397969 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.397941    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.399645 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.399621    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-dshm\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.399813 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.399795    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370408d6-ff67-4924-bbbb-ef87b813a6ed-tls-certs\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.406932 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.406907    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6x7r\" (UniqueName: \"kubernetes.io/projected/370408d6-ff67-4924-bbbb-ef87b813a6ed-kube-api-access-w6x7r\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-smcw5\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.456533 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.456512    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:08:55.484263 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.484233    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"]
May 20 14:08:55.489353 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.489331    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.492208 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.492188    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-w5jl7\""
May 20 14:08:55.500150 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.500128    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"]
May 20 14:08:55.587981 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.587951    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"]
May 20 14:08:55.591207 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:08:55.591182    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370408d6_ff67_4924_bbbb_ef87b813a6ed.slice/crio-2355e14a62f3c14b0fa3947d202e162bfb319ce542d6871c531576ac87f6202b WatchSource:0}: Error finding container 2355e14a62f3c14b0fa3947d202e162bfb319ce542d6871c531576ac87f6202b: Status 404 returned error can't find the container with id 2355e14a62f3c14b0fa3947d202e162bfb319ce542d6871c531576ac87f6202b
May 20 14:08:55.599114 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.599094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.599190 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.599127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5fe103-001d-46ec-af60-a5f380f0e11e-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.599190 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.599154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.599279 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.599215    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.599335 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.599303    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngk8\" (UniqueName: \"kubernetes.io/projected/da5fe103-001d-46ec-af60-a5f380f0e11e-kube-api-access-jngk8\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.599426 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.599410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.700354 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.700289    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.700354 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.700331    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.700354 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.700349    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5fe103-001d-46ec-af60-a5f380f0e11e-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.700572 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.700365    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.700572 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.700394    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.700572 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.700432    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jngk8\" (UniqueName: \"kubernetes.io/projected/da5fe103-001d-46ec-af60-a5f380f0e11e-kube-api-access-jngk8\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.700727 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.700702    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.700787 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.700734    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.700834 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.700779    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.700873 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.700847    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.702731 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.702709    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5fe103-001d-46ec-af60-a5f380f0e11e-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.708386 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.708364    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngk8\" (UniqueName: \"kubernetes.io/projected/da5fe103-001d-46ec-af60-a5f380f0e11e-kube-api-access-jngk8\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.799824 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.799794    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:55.937247 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.936545    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5" event={"ID":"370408d6-ff67-4924-bbbb-ef87b813a6ed","Type":"ContainerStarted","Data":"a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8"}
May 20 14:08:55.937247 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.936591    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5" event={"ID":"370408d6-ff67-4924-bbbb-ef87b813a6ed","Type":"ContainerStarted","Data":"2355e14a62f3c14b0fa3947d202e162bfb319ce542d6871c531576ac87f6202b"}
May 20 14:08:55.939635 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:55.939581    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"]
May 20 14:08:55.943290 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:08:55.943262    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5fe103_001d_46ec_af60_a5f380f0e11e.slice/crio-b973beea4901afa539ce0b4f6416af7f51c3f2d6c83c1f2a77f0d8a1d5963345 WatchSource:0}: Error finding container b973beea4901afa539ce0b4f6416af7f51c3f2d6c83c1f2a77f0d8a1d5963345: Status 404 returned error can't find the container with id b973beea4901afa539ce0b4f6416af7f51c3f2d6c83c1f2a77f0d8a1d5963345
May 20 14:08:56.941610 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:56.941569    2566 generic.go:358] "Generic (PLEG): container finished" podID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerID="64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de" exitCode=0
May 20 14:08:56.942130 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:56.941696    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" event={"ID":"da5fe103-001d-46ec-af60-a5f380f0e11e","Type":"ContainerDied","Data":"64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de"}
May 20 14:08:56.942130 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:56.941743    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" event={"ID":"da5fe103-001d-46ec-af60-a5f380f0e11e","Type":"ContainerStarted","Data":"b973beea4901afa539ce0b4f6416af7f51c3f2d6c83c1f2a77f0d8a1d5963345"}
May 20 14:08:57.947344 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:57.947306    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" event={"ID":"da5fe103-001d-46ec-af60-a5f380f0e11e","Type":"ContainerStarted","Data":"253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65"}
May 20 14:08:57.947344 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:57.947344    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" event={"ID":"da5fe103-001d-46ec-af60-a5f380f0e11e","Type":"ContainerStarted","Data":"59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e"}
May 20 14:08:57.947927 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:57.947480    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:08:57.970314 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:57.970253    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" podStartSLOduration=2.9702372219999997 podStartE2EDuration="2.970237222s" podCreationTimestamp="2026-05-20 14:08:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:08:57.968339828 +0000 UTC m=+1098.522785403" watchObservedRunningTime="2026-05-20 14:08:57.970237222 +0000 UTC m=+1098.524682795"
May 20 14:08:59.961382 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:59.961345    2566 generic.go:358] "Generic (PLEG): container finished" podID="370408d6-ff67-4924-bbbb-ef87b813a6ed" containerID="a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8" exitCode=0
May 20 14:08:59.961787 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:08:59.961416    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5" event={"ID":"370408d6-ff67-4924-bbbb-ef87b813a6ed","Type":"ContainerDied","Data":"a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8"}
May 20 14:09:00.969437 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:00.969403    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5" event={"ID":"370408d6-ff67-4924-bbbb-ef87b813a6ed","Type":"ContainerStarted","Data":"ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2"}
May 20 14:09:00.989602 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:00.989553    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5" podStartSLOduration=5.989539611 podStartE2EDuration="5.989539611s" podCreationTimestamp="2026-05-20 14:08:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:09:00.988299987 +0000 UTC m=+1101.542745564" watchObservedRunningTime="2026-05-20 14:09:00.989539611 +0000 UTC m=+1101.543985184"
May 20 14:09:05.457289 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:05.457248    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:09:05.457289 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:05.457293    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:09:05.469728 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:05.469708    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:09:05.800481 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:05.800387    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:09:05.800481 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:05.800428    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:09:05.801683 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:09:05.801657    2566 logging.go:55] [core] [Channel #47 SubChannel #48]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.23:9003", ServerName: "10.134.0.23:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.23:9003: connect: connection refused"
May 20 14:09:05.802973 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:05.802954    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:09:05.983436 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:05.983408    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:09:05.993778 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:05.993758    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:09:06.801212 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:06.801162    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.23:9003\" within 1s: context deadline exceeded"
May 20 14:09:15.801111 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:09:15.801079    2566 logging.go:55] [core] [Channel #49 SubChannel #50]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.23:9003", ServerName: "10.134.0.23:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.23:9003: connect: connection refused"
May 20 14:09:16.800819 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:16.800774    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.23:9003\" within 1s: context deadline exceeded"
May 20 14:09:26.986387 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:26.986357    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:09:28.017722 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.017664    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"]
May 20 14:09:28.018197 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.018079    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="main" containerID="cri-o://59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e" gracePeriod=30
May 20 14:09:28.018197 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.018143    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="tokenizer" containerID="cri-o://253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65" gracePeriod=30
May 20 14:09:28.025290 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.025255    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"]
May 20 14:09:28.025659 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.025606    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5" podUID="370408d6-ff67-4924-bbbb-ef87b813a6ed" containerName="main" containerID="cri-o://ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2" gracePeriod=30
May 20 14:09:28.267059 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.267037    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:09:28.354782 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.354758    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-dshm\") pod \"370408d6-ff67-4924-bbbb-ef87b813a6ed\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") "
May 20 14:09:28.354943 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.354833    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6x7r\" (UniqueName: \"kubernetes.io/projected/370408d6-ff67-4924-bbbb-ef87b813a6ed-kube-api-access-w6x7r\") pod \"370408d6-ff67-4924-bbbb-ef87b813a6ed\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") "
May 20 14:09:28.354943 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.354873    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-model-cache\") pod \"370408d6-ff67-4924-bbbb-ef87b813a6ed\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") "
May 20 14:09:28.354943 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.354911    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-home\") pod \"370408d6-ff67-4924-bbbb-ef87b813a6ed\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") "
May 20 14:09:28.354943 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.354935    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-kserve-provision-location\") pod \"370408d6-ff67-4924-bbbb-ef87b813a6ed\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") "
May 20 14:09:28.355177 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.354964    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-tmp-dir\") pod \"370408d6-ff67-4924-bbbb-ef87b813a6ed\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") "
May 20 14:09:28.355177 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.354996    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370408d6-ff67-4924-bbbb-ef87b813a6ed-tls-certs\") pod \"370408d6-ff67-4924-bbbb-ef87b813a6ed\" (UID: \"370408d6-ff67-4924-bbbb-ef87b813a6ed\") "
May 20 14:09:28.355177 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.355151    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-home" (OuterVolumeSpecName: "home") pod "370408d6-ff67-4924-bbbb-ef87b813a6ed" (UID: "370408d6-ff67-4924-bbbb-ef87b813a6ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:28.355329 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.355188    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "370408d6-ff67-4924-bbbb-ef87b813a6ed" (UID: "370408d6-ff67-4924-bbbb-ef87b813a6ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:28.355329 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.355286    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-model-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:28.355329 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.355328    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-home\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:28.355650 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.355617    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "370408d6-ff67-4924-bbbb-ef87b813a6ed" (UID: "370408d6-ff67-4924-bbbb-ef87b813a6ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:28.357062 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.357031    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370408d6-ff67-4924-bbbb-ef87b813a6ed-kube-api-access-w6x7r" (OuterVolumeSpecName: "kube-api-access-w6x7r") pod "370408d6-ff67-4924-bbbb-ef87b813a6ed" (UID: "370408d6-ff67-4924-bbbb-ef87b813a6ed"). InnerVolumeSpecName "kube-api-access-w6x7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:09:28.357161 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.357106    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/370408d6-ff67-4924-bbbb-ef87b813a6ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "370408d6-ff67-4924-bbbb-ef87b813a6ed" (UID: "370408d6-ff67-4924-bbbb-ef87b813a6ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:09:28.357229 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.357208    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-dshm" (OuterVolumeSpecName: "dshm") pod "370408d6-ff67-4924-bbbb-ef87b813a6ed" (UID: "370408d6-ff67-4924-bbbb-ef87b813a6ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:28.409817 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.409783    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "370408d6-ff67-4924-bbbb-ef87b813a6ed" (UID: "370408d6-ff67-4924-bbbb-ef87b813a6ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:28.456449 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.456410    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-kserve-provision-location\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:28.456449 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.456442    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-tmp-dir\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:28.456449 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.456452    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370408d6-ff67-4924-bbbb-ef87b813a6ed-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:28.456690 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.456462    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/370408d6-ff67-4924-bbbb-ef87b813a6ed-dshm\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:28.456690 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:28.456474    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6x7r\" (UniqueName: \"kubernetes.io/projected/370408d6-ff67-4924-bbbb-ef87b813a6ed-kube-api-access-w6x7r\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:29.047509 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.047479    2566 generic.go:358] "Generic (PLEG): container finished" podID="370408d6-ff67-4924-bbbb-ef87b813a6ed" containerID="ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2" exitCode=0
May 20 14:09:29.047964 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.047548    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5" event={"ID":"370408d6-ff67-4924-bbbb-ef87b813a6ed","Type":"ContainerDied","Data":"ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2"}
May 20 14:09:29.047964 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.047564    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"
May 20 14:09:29.047964 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.047583    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5" event={"ID":"370408d6-ff67-4924-bbbb-ef87b813a6ed","Type":"ContainerDied","Data":"2355e14a62f3c14b0fa3947d202e162bfb319ce542d6871c531576ac87f6202b"}
May 20 14:09:29.047964 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.047614    2566 scope.go:117] "RemoveContainer" containerID="ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2"
May 20 14:09:29.049552 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.049528    2566 generic.go:358] "Generic (PLEG): container finished" podID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerID="59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e" exitCode=0
May 20 14:09:29.049634 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.049584    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" event={"ID":"da5fe103-001d-46ec-af60-a5f380f0e11e","Type":"ContainerDied","Data":"59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e"}
May 20 14:09:29.055552 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.055535    2566 scope.go:117] "RemoveContainer" containerID="a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8"
May 20 14:09:29.069804 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.069781    2566 scope.go:117] "RemoveContainer" containerID="ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2"
May 20 14:09:29.070128 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:09:29.070100    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2\": container with ID starting with ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2 not found: ID does not exist" containerID="ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2"
May 20 14:09:29.070222 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.070137    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2"} err="failed to get container status \"ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2\": rpc error: code = NotFound desc = could not find container \"ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2\": container with ID starting with ff91ce4772cbc6be8097f43511f91a77434020779355040c7830a228055a29a2 not found: ID does not exist"
May 20 14:09:29.070222 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.070160    2566 scope.go:117] "RemoveContainer" containerID="a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8"
May 20 14:09:29.070487 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:09:29.070469    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8\": container with ID starting with a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8 not found: ID does not exist" containerID="a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8"
May 20 14:09:29.070558 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.070495    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8"} err="failed to get container status \"a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8\": rpc error: code = NotFound desc = could not find container \"a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8\": container with ID starting with a31bc1f18bff8c63c195b92f55b0db167d950500b0517f07035239732e753df8 not found: ID does not exist"
May 20 14:09:29.071410 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.071391    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"]
May 20 14:09:29.078701 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.076813    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-smcw5"]
May 20 14:09:29.360749 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.360729    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:09:29.464214 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.464188    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-kserve-provision-location\") pod \"da5fe103-001d-46ec-af60-a5f380f0e11e\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") "
May 20 14:09:29.464323 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.464235    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-tmp\") pod \"da5fe103-001d-46ec-af60-a5f380f0e11e\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") "
May 20 14:09:29.464323 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.464262    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-uds\") pod \"da5fe103-001d-46ec-af60-a5f380f0e11e\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") "
May 20 14:09:29.464323 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.464300    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5fe103-001d-46ec-af60-a5f380f0e11e-tls-certs\") pod \"da5fe103-001d-46ec-af60-a5f380f0e11e\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") "
May 20 14:09:29.464323 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.464316    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jngk8\" (UniqueName: \"kubernetes.io/projected/da5fe103-001d-46ec-af60-a5f380f0e11e-kube-api-access-jngk8\") pod \"da5fe103-001d-46ec-af60-a5f380f0e11e\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") "
May 20 14:09:29.464514 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.464334    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-cache\") pod \"da5fe103-001d-46ec-af60-a5f380f0e11e\" (UID: \"da5fe103-001d-46ec-af60-a5f380f0e11e\") "
May 20 14:09:29.464602 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.464567    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "da5fe103-001d-46ec-af60-a5f380f0e11e" (UID: "da5fe103-001d-46ec-af60-a5f380f0e11e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:29.464739 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.464632    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "da5fe103-001d-46ec-af60-a5f380f0e11e" (UID: "da5fe103-001d-46ec-af60-a5f380f0e11e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:29.464806 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.464731    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "da5fe103-001d-46ec-af60-a5f380f0e11e" (UID: "da5fe103-001d-46ec-af60-a5f380f0e11e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:29.464952 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.464929    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da5fe103-001d-46ec-af60-a5f380f0e11e" (UID: "da5fe103-001d-46ec-af60-a5f380f0e11e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:29.466319 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.466298    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5fe103-001d-46ec-af60-a5f380f0e11e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da5fe103-001d-46ec-af60-a5f380f0e11e" (UID: "da5fe103-001d-46ec-af60-a5f380f0e11e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:09:29.466371 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.466333    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5fe103-001d-46ec-af60-a5f380f0e11e-kube-api-access-jngk8" (OuterVolumeSpecName: "kube-api-access-jngk8") pod "da5fe103-001d-46ec-af60-a5f380f0e11e" (UID: "da5fe103-001d-46ec-af60-a5f380f0e11e"). InnerVolumeSpecName "kube-api-access-jngk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:09:29.565448 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.565424    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-kserve-provision-location\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:29.565448 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.565445    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-tmp\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:29.565578 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.565455    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-uds\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:29.565578 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.565484    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5fe103-001d-46ec-af60-a5f380f0e11e-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:29.565578 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.565492    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngk8\" (UniqueName: \"kubernetes.io/projected/da5fe103-001d-46ec-af60-a5f380f0e11e-kube-api-access-jngk8\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:29.565578 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.565501    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5fe103-001d-46ec-af60-a5f380f0e11e-tokenizer-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:09:29.991017 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:29.990985    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="370408d6-ff67-4924-bbbb-ef87b813a6ed" path="/var/lib/kubelet/pods/370408d6-ff67-4924-bbbb-ef87b813a6ed/volumes"
May 20 14:09:30.054207 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.054176    2566 generic.go:358] "Generic (PLEG): container finished" podID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerID="253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65" exitCode=0
May 20 14:09:30.054507 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.054242    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" event={"ID":"da5fe103-001d-46ec-af60-a5f380f0e11e","Type":"ContainerDied","Data":"253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65"}
May 20 14:09:30.054507 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.054250    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"
May 20 14:09:30.054507 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.054269    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7" event={"ID":"da5fe103-001d-46ec-af60-a5f380f0e11e","Type":"ContainerDied","Data":"b973beea4901afa539ce0b4f6416af7f51c3f2d6c83c1f2a77f0d8a1d5963345"}
May 20 14:09:30.054507 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.054289    2566 scope.go:117] "RemoveContainer" containerID="253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65"
May 20 14:09:30.061777 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.061750    2566 scope.go:117] "RemoveContainer" containerID="59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e"
May 20 14:09:30.068348 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.068332    2566 scope.go:117] "RemoveContainer" containerID="64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de"
May 20 14:09:30.074324 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.074303    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"]
May 20 14:09:30.075403 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.075385    2566 scope.go:117] "RemoveContainer" containerID="253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65"
May 20 14:09:30.075758 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:09:30.075664    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65\": container with ID starting with 253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65 not found: ID does not exist" containerID="253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65"
May 20 14:09:30.075758 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.075712    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65"} err="failed to get container status \"253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65\": rpc error: code = NotFound desc = could not find container \"253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65\": container with ID starting with 253de186db1a108b867adbe53230bc86375a1ac6929dd9a921aaf49d4f955f65 not found: ID does not exist"
May 20 14:09:30.075758 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.075731    2566 scope.go:117] "RemoveContainer" containerID="59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e"
May 20 14:09:30.076126 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:09:30.075999    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e\": container with ID starting with 59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e not found: ID does not exist" containerID="59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e"
May 20 14:09:30.076272 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.076247    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e"} err="failed to get container status \"59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e\": rpc error: code = NotFound desc = could not find container \"59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e\": container with ID starting with 59a2a7a7f40e8091b9b70a9caaa970e93431f3b09d42566c9fb895a2019bda6e not found: ID does not exist"
May 20 14:09:30.076344 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.076275    2566 scope.go:117] "RemoveContainer" containerID="64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de"
May 20 14:09:30.076583 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:09:30.076558    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de\": container with ID starting with 64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de not found: ID does not exist" containerID="64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de"
May 20 14:09:30.076583 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.076590    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de"} err="failed to get container status \"64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de\": rpc error: code = NotFound desc = could not find container \"64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de\": container with ID starting with 64c7117b8e79abf266ef2dcacc72366166da4d5d549f36bb169caed51cbef1de not found: ID does not exist"
May 20 14:09:30.077250 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:30.077232    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7dbf97ff6l2d7"]
May 20 14:09:31.991355 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:31.991321    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" path="/var/lib/kubelet/pods/da5fe103-001d-46ec-af60-a5f380f0e11e/volumes"
May 20 14:09:52.545492 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545456    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"]
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545767    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370408d6-ff67-4924-bbbb-ef87b813a6ed" containerName="main"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545779    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="370408d6-ff67-4924-bbbb-ef87b813a6ed" containerName="main"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545792    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370408d6-ff67-4924-bbbb-ef87b813a6ed" containerName="storage-initializer"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545798    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="370408d6-ff67-4924-bbbb-ef87b813a6ed" containerName="storage-initializer"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545804    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="tokenizer"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545809    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="tokenizer"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545817    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="storage-initializer"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545823    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="storage-initializer"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545829    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="main"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545834    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="main"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545875    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="tokenizer"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545883    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="da5fe103-001d-46ec-af60-a5f380f0e11e" containerName="main"
May 20 14:09:52.546011 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.545892    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="370408d6-ff67-4924-bbbb-ef87b813a6ed" containerName="main"
May 20 14:09:52.548631 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.548611    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.551433 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.551414    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-qf8fj\""
May 20 14:09:52.552419 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.552401    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 20 14:09:52.552501 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.552414    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:09:52.552501 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.552422    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:09:52.552611 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.552504    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:09:52.552611 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.552539    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:09:52.560095 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.560068    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"]
May 20 14:09:52.635218 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.635191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.635322 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.635231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.635322 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.635274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.635393 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.635326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6f29\" (UniqueName: \"kubernetes.io/projected/73ebf0c6-5919-4340-a64e-a78b31becf55-kube-api-access-n6f29\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.635393 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.635346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.635393 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.635370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ebf0c6-5919-4340-a64e-a78b31becf55-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.736415 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.736392    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.736490 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.736425    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.736490 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.736449    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.736490 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.736484    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n6f29\" (UniqueName: \"kubernetes.io/projected/73ebf0c6-5919-4340-a64e-a78b31becf55-kube-api-access-n6f29\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.736607 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.736510    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.736660 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.736602    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ebf0c6-5919-4340-a64e-a78b31becf55-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.736850 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.736831    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.736899 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.736880    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.736935 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.736927    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.736971 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.736943    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.739018 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.738999    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ebf0c6-5919-4340-a64e-a78b31becf55-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.745080 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.745054    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6f29\" (UniqueName: \"kubernetes.io/projected/73ebf0c6-5919-4340-a64e-a78b31becf55-kube-api-access-n6f29\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-f4h99\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.857377 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.857347    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:52.986513 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:52.986489    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"]
May 20 14:09:52.988926 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:09:52.988893    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ebf0c6_5919_4340_a64e_a78b31becf55.slice/crio-87f414c0608a559c076506549a9fda6ed9559cd25284b3d5b463a4beb4ed3f6c WatchSource:0}: Error finding container 87f414c0608a559c076506549a9fda6ed9559cd25284b3d5b463a4beb4ed3f6c: Status 404 returned error can't find the container with id 87f414c0608a559c076506549a9fda6ed9559cd25284b3d5b463a4beb4ed3f6c
May 20 14:09:53.117249 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:53.117164    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" event={"ID":"73ebf0c6-5919-4340-a64e-a78b31becf55","Type":"ContainerStarted","Data":"2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0"}
May 20 14:09:53.117249 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:53.117205    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" event={"ID":"73ebf0c6-5919-4340-a64e-a78b31becf55","Type":"ContainerStarted","Data":"87f414c0608a559c076506549a9fda6ed9559cd25284b3d5b463a4beb4ed3f6c"}
May 20 14:09:54.121176 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:54.121140    2566 generic.go:358] "Generic (PLEG): container finished" podID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerID="2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0" exitCode=0
May 20 14:09:54.121641 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:54.121193    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" event={"ID":"73ebf0c6-5919-4340-a64e-a78b31becf55","Type":"ContainerDied","Data":"2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0"}
May 20 14:09:55.127497 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:55.127464    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" event={"ID":"73ebf0c6-5919-4340-a64e-a78b31becf55","Type":"ContainerStarted","Data":"ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a"}
May 20 14:09:55.127497 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:55.127498    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" event={"ID":"73ebf0c6-5919-4340-a64e-a78b31becf55","Type":"ContainerStarted","Data":"2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a"}
May 20 14:09:55.128030 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:55.127624    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:09:55.153223 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:09:55.153177    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" podStartSLOduration=3.153163801 podStartE2EDuration="3.153163801s" podCreationTimestamp="2026-05-20 14:09:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:09:55.151657989 +0000 UTC m=+1155.706103559" watchObservedRunningTime="2026-05-20 14:09:55.153163801 +0000 UTC m=+1155.707609374"
May 20 14:10:02.857953 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:10:02.857877    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:10:02.857953 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:10:02.857922    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:10:02.860437 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:10:02.860387    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:10:03.153728 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:10:03.153651    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:10:24.157487 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:10:24.157458    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:12:17.645779 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:17.645747    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"]
May 20 14:12:17.646222 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:17.646059    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerName="main" containerID="cri-o://2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a" gracePeriod=30
May 20 14:12:17.646222 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:17.646109    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerName="tokenizer" containerID="cri-o://ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a" gracePeriod=30
May 20 14:12:18.522174 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.522134    2566 generic.go:358] "Generic (PLEG): container finished" podID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerID="2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a" exitCode=0
May 20 14:12:18.522358 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.522187    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" event={"ID":"73ebf0c6-5919-4340-a64e-a78b31becf55","Type":"ContainerDied","Data":"2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a"}
May 20 14:12:18.884773 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.884748    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:12:18.935416 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.935378    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-uds\") pod \"73ebf0c6-5919-4340-a64e-a78b31becf55\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") "
May 20 14:12:18.935602 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.935428    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-kserve-provision-location\") pod \"73ebf0c6-5919-4340-a64e-a78b31becf55\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") "
May 20 14:12:18.935602 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.935457    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ebf0c6-5919-4340-a64e-a78b31becf55-tls-certs\") pod \"73ebf0c6-5919-4340-a64e-a78b31becf55\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") "
May 20 14:12:18.935602 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.935502    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-tmp\") pod \"73ebf0c6-5919-4340-a64e-a78b31becf55\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") "
May 20 14:12:18.935602 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.935562    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6f29\" (UniqueName: \"kubernetes.io/projected/73ebf0c6-5919-4340-a64e-a78b31becf55-kube-api-access-n6f29\") pod \"73ebf0c6-5919-4340-a64e-a78b31becf55\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") "
May 20 14:12:18.935847 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.935606    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-cache\") pod \"73ebf0c6-5919-4340-a64e-a78b31becf55\" (UID: \"73ebf0c6-5919-4340-a64e-a78b31becf55\") "
May 20 14:12:18.935847 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.935724    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "73ebf0c6-5919-4340-a64e-a78b31becf55" (UID: "73ebf0c6-5919-4340-a64e-a78b31becf55"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:18.935945 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.935861    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-uds\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:12:18.935945 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.935889    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "73ebf0c6-5919-4340-a64e-a78b31becf55" (UID: "73ebf0c6-5919-4340-a64e-a78b31becf55"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:18.936010 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.935967    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "73ebf0c6-5919-4340-a64e-a78b31becf55" (UID: "73ebf0c6-5919-4340-a64e-a78b31becf55"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:18.936223 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.936202    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73ebf0c6-5919-4340-a64e-a78b31becf55" (UID: "73ebf0c6-5919-4340-a64e-a78b31becf55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:18.937815 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.937788    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ebf0c6-5919-4340-a64e-a78b31becf55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73ebf0c6-5919-4340-a64e-a78b31becf55" (UID: "73ebf0c6-5919-4340-a64e-a78b31becf55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:12:18.938017 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:18.937996    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ebf0c6-5919-4340-a64e-a78b31becf55-kube-api-access-n6f29" (OuterVolumeSpecName: "kube-api-access-n6f29") pod "73ebf0c6-5919-4340-a64e-a78b31becf55" (UID: "73ebf0c6-5919-4340-a64e-a78b31becf55"). InnerVolumeSpecName "kube-api-access-n6f29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:12:19.036252 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.036214    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-tmp\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:12:19.036252 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.036245    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6f29\" (UniqueName: \"kubernetes.io/projected/73ebf0c6-5919-4340-a64e-a78b31becf55-kube-api-access-n6f29\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:12:19.036252 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.036255    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-tokenizer-cache\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:12:19.036252 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.036265    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ebf0c6-5919-4340-a64e-a78b31becf55-kserve-provision-location\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:12:19.036498 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.036274    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ebf0c6-5919-4340-a64e-a78b31becf55-tls-certs\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:12:19.527206 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.527166    2566 generic.go:358] "Generic (PLEG): container finished" podID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerID="ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a" exitCode=0
May 20 14:12:19.527391 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.527240    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" event={"ID":"73ebf0c6-5919-4340-a64e-a78b31becf55","Type":"ContainerDied","Data":"ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a"}
May 20 14:12:19.527391 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.527269    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99" event={"ID":"73ebf0c6-5919-4340-a64e-a78b31becf55","Type":"ContainerDied","Data":"87f414c0608a559c076506549a9fda6ed9559cd25284b3d5b463a4beb4ed3f6c"}
May 20 14:12:19.527391 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.527285    2566 scope.go:117] "RemoveContainer" containerID="ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a"
May 20 14:12:19.527391 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.527247    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"
May 20 14:12:19.534798 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.534772    2566 scope.go:117] "RemoveContainer" containerID="2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a"
May 20 14:12:19.541830 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.541812    2566 scope.go:117] "RemoveContainer" containerID="2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0"
May 20 14:12:19.548717 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.548661    2566 scope.go:117] "RemoveContainer" containerID="ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a"
May 20 14:12:19.549016 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:12:19.548989    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a\": container with ID starting with ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a not found: ID does not exist" containerID="ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a"
May 20 14:12:19.549103 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.549029    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a"} err="failed to get container status \"ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a\": rpc error: code = NotFound desc = could not find container \"ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a\": container with ID starting with ba967476243572341a5f0267ce65e94ab4a9fe7e8c2a2f71fbaa4af3df74f52a not found: ID does not exist"
May 20 14:12:19.549103 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.549054    2566 scope.go:117] "RemoveContainer" containerID="2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a"
May 20 14:12:19.549366 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:12:19.549348    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a\": container with ID starting with 2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a not found: ID does not exist" containerID="2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a"
May 20 14:12:19.549437 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.549376    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a"} err="failed to get container status \"2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a\": rpc error: code = NotFound desc = could not find container \"2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a\": container with ID starting with 2a0b8fdb58b30cfc7bacdf06b8db33a20d2c2167fd1b60caec113852d03a4d6a not found: ID does not exist"
May 20 14:12:19.549437 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.549398    2566 scope.go:117] "RemoveContainer" containerID="2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0"
May 20 14:12:19.549664 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:12:19.549643    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0\": container with ID starting with 2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0 not found: ID does not exist" containerID="2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0"
May 20 14:12:19.549749 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.549668    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0"} err="failed to get container status \"2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0\": rpc error: code = NotFound desc = could not find container \"2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0\": container with ID starting with 2cf46defcf3376d732a8d5d6ce89d69e2cf6d8133f88bd95ef75d53a02d5d9a0 not found: ID does not exist"
May 20 14:12:19.550626 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.550606    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"]
May 20 14:12:19.554105 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.554085    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-f4h99"]
May 20 14:12:19.991291 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:19.991259    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" path="/var/lib/kubelet/pods/73ebf0c6-5919-4340-a64e-a78b31becf55/volumes"
May 20 14:12:22.155402 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.155365    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"]
May 20 14:12:22.155908 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.155701    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerName="main"
May 20 14:12:22.155908 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.155718    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerName="main"
May 20 14:12:22.155908 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.155731    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerName="tokenizer"
May 20 14:12:22.155908 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.155739    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerName="tokenizer"
May 20 14:12:22.155908 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.155753    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerName="storage-initializer"
May 20 14:12:22.155908 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.155762    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerName="storage-initializer"
May 20 14:12:22.155908 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.155849    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerName="tokenizer"
May 20 14:12:22.155908 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.155861    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="73ebf0c6-5919-4340-a64e-a78b31becf55" containerName="main"
May 20 14:12:22.160918 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.160891    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.163963 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.163930    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:12:22.164118 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.163934    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:12:22.164337 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.164317    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-tlrm5\""
May 20 14:12:22.164389 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.164361    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:12:22.165033 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.165011    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 20 14:12:22.165162 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.165040    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:12:22.171826 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.171802    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"]
May 20 14:12:22.262190 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.262142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.262407 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.262208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.262407 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.262313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2599832a-faa8-42b9-918b-ed3ce46444c6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.262407 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.262369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.262407 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.262404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbdc\" (UniqueName: \"kubernetes.io/projected/2599832a-faa8-42b9-918b-ed3ce46444c6-kube-api-access-zpbdc\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.262589 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.262438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.363354 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.363312    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbdc\" (UniqueName: \"kubernetes.io/projected/2599832a-faa8-42b9-918b-ed3ce46444c6-kube-api-access-zpbdc\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.363354 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.363361    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.363599 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.363396    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.363599 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.363453    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.363599 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.363494    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2599832a-faa8-42b9-918b-ed3ce46444c6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.363599 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.363530    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.363919 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.363895    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.363981 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.363918    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.363981 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.363957    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.364046 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.363996    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2599832a-faa8-42b9-918b-ed3ce46444c6-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.366042 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.366017    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2599832a-faa8-42b9-918b-ed3ce46444c6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.372919 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.372895    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbdc\" (UniqueName: \"kubernetes.io/projected/2599832a-faa8-42b9-918b-ed3ce46444c6-kube-api-access-zpbdc\") pod \"stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb\" (UID: \"2599832a-faa8-42b9-918b-ed3ce46444c6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.472627 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.472548    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:22.603684 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.603633    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"]
May 20 14:12:22.606549 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:12:22.606520    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2599832a_faa8_42b9_918b_ed3ce46444c6.slice/crio-5dd33ef0907bf6aa246a227723415b3c7f521f0125be94559797fbac72d6021f WatchSource:0}: Error finding container 5dd33ef0907bf6aa246a227723415b3c7f521f0125be94559797fbac72d6021f: Status 404 returned error can't find the container with id 5dd33ef0907bf6aa246a227723415b3c7f521f0125be94559797fbac72d6021f
May 20 14:12:22.608346 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:22.608331    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:12:23.540653 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:23.540619    2566 generic.go:358] "Generic (PLEG): container finished" podID="2599832a-faa8-42b9-918b-ed3ce46444c6" containerID="049f756cfbd266bfca92c060c2246c7881c09cf93a36f6db6ab1e9d19fc6abe0" exitCode=0
May 20 14:12:23.541043 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:23.540703    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb" event={"ID":"2599832a-faa8-42b9-918b-ed3ce46444c6","Type":"ContainerDied","Data":"049f756cfbd266bfca92c060c2246c7881c09cf93a36f6db6ab1e9d19fc6abe0"}
May 20 14:12:23.541043 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:23.540736    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb" event={"ID":"2599832a-faa8-42b9-918b-ed3ce46444c6","Type":"ContainerStarted","Data":"5dd33ef0907bf6aa246a227723415b3c7f521f0125be94559797fbac72d6021f"}
May 20 14:12:24.546296 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:24.546253    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb" event={"ID":"2599832a-faa8-42b9-918b-ed3ce46444c6","Type":"ContainerStarted","Data":"ca411513a9313b573ee229b9e2d6d2fe077177df0c0c6e01218fe732dbe4a428"}
May 20 14:12:24.546296 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:24.546298    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb" event={"ID":"2599832a-faa8-42b9-918b-ed3ce46444c6","Type":"ContainerStarted","Data":"de1ead830c6da787d81c464b7da8d6ae4ecc9ec4e07599df0cf086b07ef13948"}
May 20 14:12:24.546743 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:24.546353    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:24.572235 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:24.572186    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb" podStartSLOduration=2.57217051 podStartE2EDuration="2.57217051s" podCreationTimestamp="2026-05-20 14:12:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:12:24.571951899 +0000 UTC m=+1305.126397473" watchObservedRunningTime="2026-05-20 14:12:24.57217051 +0000 UTC m=+1305.126616084"
May 20 14:12:32.473287 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:32.473244    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:32.473287 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:32.473297    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:32.475964 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:32.475941    2566 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:32.570804 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:32.570775    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:12:53.574637 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:12:53.574606    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb"
May 20 14:27:08.449457 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.449423    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"]
May 20 14:27:08.452575 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.452560    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"
May 20 14:27:08.456313 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.456288    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 20 14:27:08.456458 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.456323    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 14:27:08.456458 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.456335    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-jqwfs\""
May 20 14:27:08.456458 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.456350    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 14:27:08.459577 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.459528    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"]
May 20 14:27:08.568561 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.568525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af15db72-0136-46a1-9ab7-613b8c2a1829-cert\") pod \"llmisvc-controller-manager-6bfc7d65cd-k5vsm\" (UID: \"af15db72-0136-46a1-9ab7-613b8c2a1829\") " pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"
May 20 14:27:08.568744 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.568601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgw9\" (UniqueName: \"kubernetes.io/projected/af15db72-0136-46a1-9ab7-613b8c2a1829-kube-api-access-nkgw9\") pod \"llmisvc-controller-manager-6bfc7d65cd-k5vsm\" (UID: \"af15db72-0136-46a1-9ab7-613b8c2a1829\") " pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"
May 20 14:27:08.669526 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.669494    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af15db72-0136-46a1-9ab7-613b8c2a1829-cert\") pod \"llmisvc-controller-manager-6bfc7d65cd-k5vsm\" (UID: \"af15db72-0136-46a1-9ab7-613b8c2a1829\") " pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"
May 20 14:27:08.669661 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.669554    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgw9\" (UniqueName: \"kubernetes.io/projected/af15db72-0136-46a1-9ab7-613b8c2a1829-kube-api-access-nkgw9\") pod \"llmisvc-controller-manager-6bfc7d65cd-k5vsm\" (UID: \"af15db72-0136-46a1-9ab7-613b8c2a1829\") " pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"
May 20 14:27:08.671804 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.671787    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af15db72-0136-46a1-9ab7-613b8c2a1829-cert\") pod \"llmisvc-controller-manager-6bfc7d65cd-k5vsm\" (UID: \"af15db72-0136-46a1-9ab7-613b8c2a1829\") " pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"
May 20 14:27:08.677969 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.677947    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgw9\" (UniqueName: \"kubernetes.io/projected/af15db72-0136-46a1-9ab7-613b8c2a1829-kube-api-access-nkgw9\") pod \"llmisvc-controller-manager-6bfc7d65cd-k5vsm\" (UID: \"af15db72-0136-46a1-9ab7-613b8c2a1829\") " pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"
May 20 14:27:08.762944 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.762874    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"
May 20 14:27:08.884551 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.884528    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"]
May 20 14:27:08.886438 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:27:08.886411    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaf15db72_0136_46a1_9ab7_613b8c2a1829.slice/crio-21199bfd4766b55a7fcb1e3bb1e1c576034cb709c5c2ea4d46f1749fab279462 WatchSource:0}: Error finding container 21199bfd4766b55a7fcb1e3bb1e1c576034cb709c5c2ea4d46f1749fab279462: Status 404 returned error can't find the container with id 21199bfd4766b55a7fcb1e3bb1e1c576034cb709c5c2ea4d46f1749fab279462
May 20 14:27:08.887745 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.887724    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:27:08.924167 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:08.924141    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm" event={"ID":"af15db72-0136-46a1-9ab7-613b8c2a1829","Type":"ContainerStarted","Data":"21199bfd4766b55a7fcb1e3bb1e1c576034cb709c5c2ea4d46f1749fab279462"}
May 20 14:27:12.937742 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:12.937704    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm" event={"ID":"af15db72-0136-46a1-9ab7-613b8c2a1829","Type":"ContainerStarted","Data":"5406eed651518316bdb5545d395a2f77d1bb9319590fb548e3d8aef724298614"}
May 20 14:27:12.938077 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:12.937806    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"
May 20 14:27:12.955093 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:12.955049    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm" podStartSLOduration=1.761395217 podStartE2EDuration="4.955037153s" podCreationTimestamp="2026-05-20 14:27:08 +0000 UTC" firstStartedPulling="2026-05-20 14:27:08.887841734 +0000 UTC m=+2189.442287285" lastFinishedPulling="2026-05-20 14:27:12.08148367 +0000 UTC m=+2192.635929221" observedRunningTime="2026-05-20 14:27:12.953394255 +0000 UTC m=+2193.507839835" watchObservedRunningTime="2026-05-20 14:27:12.955037153 +0000 UTC m=+2193.509482727"
May 20 14:27:43.942377 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:27:43.942349    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6bfc7d65cd-k5vsm"
May 20 14:28:48.977386 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:48.977352    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bh8l2/must-gather-w864p"]
May 20 14:28:48.980624 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:48.980607    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bh8l2/must-gather-w864p"
May 20 14:28:48.983354 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:48.983332    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bh8l2\"/\"kube-root-ca.crt\""
May 20 14:28:48.983459 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:48.983366    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bh8l2\"/\"openshift-service-ca.crt\""
May 20 14:28:48.984504 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:48.984486    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bh8l2\"/\"default-dockercfg-wv84k\""
May 20 14:28:48.988041 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:48.988023    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bh8l2/must-gather-w864p"]
May 20 14:28:49.004152 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:49.004127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93a208da-25fd-4114-8ee4-58b4b586bcfe-must-gather-output\") pod \"must-gather-w864p\" (UID: \"93a208da-25fd-4114-8ee4-58b4b586bcfe\") " pod="openshift-must-gather-bh8l2/must-gather-w864p"
May 20 14:28:49.004269 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:49.004194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlpn9\" (UniqueName: \"kubernetes.io/projected/93a208da-25fd-4114-8ee4-58b4b586bcfe-kube-api-access-nlpn9\") pod \"must-gather-w864p\" (UID: \"93a208da-25fd-4114-8ee4-58b4b586bcfe\") " pod="openshift-must-gather-bh8l2/must-gather-w864p"
May 20 14:28:49.104966 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:49.104939    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93a208da-25fd-4114-8ee4-58b4b586bcfe-must-gather-output\") pod \"must-gather-w864p\" (UID: \"93a208da-25fd-4114-8ee4-58b4b586bcfe\") " pod="openshift-must-gather-bh8l2/must-gather-w864p"
May 20 14:28:49.105079 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:49.104973    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nlpn9\" (UniqueName: \"kubernetes.io/projected/93a208da-25fd-4114-8ee4-58b4b586bcfe-kube-api-access-nlpn9\") pod \"must-gather-w864p\" (UID: \"93a208da-25fd-4114-8ee4-58b4b586bcfe\") " pod="openshift-must-gather-bh8l2/must-gather-w864p"
May 20 14:28:49.105315 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:49.105296    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93a208da-25fd-4114-8ee4-58b4b586bcfe-must-gather-output\") pod \"must-gather-w864p\" (UID: \"93a208da-25fd-4114-8ee4-58b4b586bcfe\") " pod="openshift-must-gather-bh8l2/must-gather-w864p"
May 20 14:28:49.113468 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:49.113445    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlpn9\" (UniqueName: \"kubernetes.io/projected/93a208da-25fd-4114-8ee4-58b4b586bcfe-kube-api-access-nlpn9\") pod \"must-gather-w864p\" (UID: \"93a208da-25fd-4114-8ee4-58b4b586bcfe\") " pod="openshift-must-gather-bh8l2/must-gather-w864p"
May 20 14:28:49.290097 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:49.290036    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bh8l2/must-gather-w864p"
May 20 14:28:49.409107 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:49.409076    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bh8l2/must-gather-w864p"]
May 20 14:28:49.412317 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:28:49.412291    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93a208da_25fd_4114_8ee4_58b4b586bcfe.slice/crio-6023a10d9916655807390a10782791f92b2bd78ae38ec928edf80f7f24ade333 WatchSource:0}: Error finding container 6023a10d9916655807390a10782791f92b2bd78ae38ec928edf80f7f24ade333: Status 404 returned error can't find the container with id 6023a10d9916655807390a10782791f92b2bd78ae38ec928edf80f7f24ade333
May 20 14:28:50.199110 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:50.199075    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh8l2/must-gather-w864p" event={"ID":"93a208da-25fd-4114-8ee4-58b4b586bcfe","Type":"ContainerStarted","Data":"6023a10d9916655807390a10782791f92b2bd78ae38ec928edf80f7f24ade333"}
May 20 14:28:56.219511 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:56.219475    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh8l2/must-gather-w864p" event={"ID":"93a208da-25fd-4114-8ee4-58b4b586bcfe","Type":"ContainerStarted","Data":"62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70"}
May 20 14:28:56.219511 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:56.219519    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh8l2/must-gather-w864p" event={"ID":"93a208da-25fd-4114-8ee4-58b4b586bcfe","Type":"ContainerStarted","Data":"b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047"}
May 20 14:28:56.235356 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:28:56.235293    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-bh8l2/must-gather-w864p" podStartSLOduration=2.357256926 podStartE2EDuration="8.235276704s" podCreationTimestamp="2026-05-20 14:28:48 +0000 UTC" firstStartedPulling="2026-05-20 14:28:49.414084911 +0000 UTC m=+2289.968530465" lastFinishedPulling="2026-05-20 14:28:55.292104684 +0000 UTC m=+2295.846550243" observedRunningTime="2026-05-20 14:28:56.234047807 +0000 UTC m=+2296.788493382" watchObservedRunningTime="2026-05-20 14:28:56.235276704 +0000 UTC m=+2296.789722277"
May 20 14:29:04.047193 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:04.047162    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:04.072485 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:04.072462    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:04.081145 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:04.081127    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:05.063878 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:05.063837    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:05.075425 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:05.075402    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:05.082960 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:05.082939    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:06.069251 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:06.069225    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:06.079909 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:06.079883    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:06.088511 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:06.088496    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:07.042123 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:07.042098    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:07.052931 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:07.052907    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:07.060004 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:07.059983    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:08.050544 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:08.050512    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:08.088030 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:08.088003    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:08.119639 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:08.119618    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:09.073142 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:09.073116    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:09.084305 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:09.084283    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:09.092232 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:09.092212    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:10.043617 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:10.043592    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:10.055173 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:10.055151    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:10.062935 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:10.062919    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:11.010322 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:11.010291    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:11.022313 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:11.022287    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:11.030543 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:11.030525    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:11.987803 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:11.987777    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:11.998947 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:11.998926    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:12.007480 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:12.007462    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:12.951921 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:12.951880    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:12.962341 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:12.962324    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:12.971095 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:12.971061    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:13.926122 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:13.926097    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:13.939498 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:13.939472    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:13.950215 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:13.950189    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:14.899378 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:14.899351    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:14.910303 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:14.910280    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:14.920712 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:14.920696    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:15.893824 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:15.893799    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:15.905195 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:15.905170    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:15.913081 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:15.913064    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:16.901887 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:16.901858    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/main/0.log"
May 20 14:29:16.913133 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:16.913105    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/tokenizer/0.log"
May 20 14:29:16.920886 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:16.920865    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-router-scheduler-6944855df4-q7mvb_2599832a-faa8-42b9-918b-ed3ce46444c6/storage-initializer/0.log"
May 20 14:29:20.418938 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:20.418905    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-4dv2w_80129d87-90f4-43ef-90f3-3f16b9be4ec6/authorino/0.log"
May 20 14:29:20.431622 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:20.431597    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7587b89b76-r95s7_e10b8171-6aa5-49b6-be5d-dda923e1a9f3/manager/0.log"
May 20 14:29:21.305688 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:21.305648    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-4dv2w_80129d87-90f4-43ef-90f3-3f16b9be4ec6/authorino/0.log"
May 20 14:29:21.317875 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:21.317846    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7587b89b76-r95s7_e10b8171-6aa5-49b6-be5d-dda923e1a9f3/manager/0.log"
May 20 14:29:22.230493 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:22.230452    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-4dv2w_80129d87-90f4-43ef-90f3-3f16b9be4ec6/authorino/0.log"
May 20 14:29:22.243184 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:22.243163    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7587b89b76-r95s7_e10b8171-6aa5-49b6-be5d-dda923e1a9f3/manager/0.log"
May 20 14:29:23.154652 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:23.154626    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-4dv2w_80129d87-90f4-43ef-90f3-3f16b9be4ec6/authorino/0.log"
May 20 14:29:23.168112 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:23.168087    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7587b89b76-r95s7_e10b8171-6aa5-49b6-be5d-dda923e1a9f3/manager/0.log"
May 20 14:29:24.049435 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:24.049403    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-4dv2w_80129d87-90f4-43ef-90f3-3f16b9be4ec6/authorino/0.log"
May 20 14:29:24.061334 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:24.061311    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7587b89b76-r95s7_e10b8171-6aa5-49b6-be5d-dda923e1a9f3/manager/0.log"
May 20 14:29:25.307086 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:25.307055    2566 generic.go:358] "Generic (PLEG): container finished" podID="93a208da-25fd-4114-8ee4-58b4b586bcfe" containerID="b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047" exitCode=0
May 20 14:29:25.307454 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:25.307130    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh8l2/must-gather-w864p" event={"ID":"93a208da-25fd-4114-8ee4-58b4b586bcfe","Type":"ContainerDied","Data":"b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047"}
May 20 14:29:25.307454 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:25.307435    2566 scope.go:117] "RemoveContainer" containerID="b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047"
May 20 14:29:25.786552 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:25.786524    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bh8l2_must-gather-w864p_93a208da-25fd-4114-8ee4-58b4b586bcfe/gather/0.log"
May 20 14:29:29.220635 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:29.220557    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ctwsd_3823f5fb-0ece-4840-8565-4bfb6334a457/global-pull-secret-syncer/0.log"
May 20 14:29:29.312578 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:29.312551    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8qthd_d87ec45b-44de-41c0-b11f-d927ae91e4d6/konnectivity-agent/0.log"
May 20 14:29:29.436639 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:29.436606    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-71.ec2.internal_2cf59113a5665c46de4a28abde7daddb/haproxy/0.log"
May 20 14:29:31.260986 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.260948    2566 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bh8l2/must-gather-w864p"]
May 20 14:29:31.261440 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.261154    2566 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-bh8l2/must-gather-w864p" podUID="93a208da-25fd-4114-8ee4-58b4b586bcfe" containerName="copy" containerID="cri-o://62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70" gracePeriod=2
May 20 14:29:31.263884 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.263838    2566 status_manager.go:895] "Failed to get status for pod" podUID="93a208da-25fd-4114-8ee4-58b4b586bcfe" pod="openshift-must-gather-bh8l2/must-gather-w864p" err="pods \"must-gather-w864p\" is forbidden: User \"system:node:ip-10-0-140-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bh8l2\": no relationship found between node 'ip-10-0-140-71.ec2.internal' and this object"
May 20 14:29:31.264267 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.264246    2566 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bh8l2/must-gather-w864p"]
May 20 14:29:31.492295 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.492271    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bh8l2_must-gather-w864p_93a208da-25fd-4114-8ee4-58b4b586bcfe/copy/0.log"
May 20 14:29:31.492621 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.492607    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bh8l2/must-gather-w864p"
May 20 14:29:31.495413 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.495391    2566 status_manager.go:895] "Failed to get status for pod" podUID="93a208da-25fd-4114-8ee4-58b4b586bcfe" pod="openshift-must-gather-bh8l2/must-gather-w864p" err="pods \"must-gather-w864p\" is forbidden: User \"system:node:ip-10-0-140-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bh8l2\": no relationship found between node 'ip-10-0-140-71.ec2.internal' and this object"
May 20 14:29:31.584601 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.584516    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlpn9\" (UniqueName: \"kubernetes.io/projected/93a208da-25fd-4114-8ee4-58b4b586bcfe-kube-api-access-nlpn9\") pod \"93a208da-25fd-4114-8ee4-58b4b586bcfe\" (UID: \"93a208da-25fd-4114-8ee4-58b4b586bcfe\") "
May 20 14:29:31.584601 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.584581    2566 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93a208da-25fd-4114-8ee4-58b4b586bcfe-must-gather-output\") pod \"93a208da-25fd-4114-8ee4-58b4b586bcfe\" (UID: \"93a208da-25fd-4114-8ee4-58b4b586bcfe\") "
May 20 14:29:31.586732 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.586696    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a208da-25fd-4114-8ee4-58b4b586bcfe-kube-api-access-nlpn9" (OuterVolumeSpecName: "kube-api-access-nlpn9") pod "93a208da-25fd-4114-8ee4-58b4b586bcfe" (UID: "93a208da-25fd-4114-8ee4-58b4b586bcfe"). InnerVolumeSpecName "kube-api-access-nlpn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:29:31.590305 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.590284    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a208da-25fd-4114-8ee4-58b4b586bcfe-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "93a208da-25fd-4114-8ee4-58b4b586bcfe" (UID: "93a208da-25fd-4114-8ee4-58b4b586bcfe"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:29:31.685844 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.685797    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93a208da-25fd-4114-8ee4-58b4b586bcfe-must-gather-output\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:29:31.685844 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.685839    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlpn9\" (UniqueName: \"kubernetes.io/projected/93a208da-25fd-4114-8ee4-58b4b586bcfe-kube-api-access-nlpn9\") on node \"ip-10-0-140-71.ec2.internal\" DevicePath \"\""
May 20 14:29:31.991328 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:31.991296    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a208da-25fd-4114-8ee4-58b4b586bcfe" path="/var/lib/kubelet/pods/93a208da-25fd-4114-8ee4-58b4b586bcfe/volumes"
May 20 14:29:32.326044 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:32.325971    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bh8l2_must-gather-w864p_93a208da-25fd-4114-8ee4-58b4b586bcfe/copy/0.log"
May 20 14:29:32.326456 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:32.326309    2566 generic.go:358] "Generic (PLEG): container finished" podID="93a208da-25fd-4114-8ee4-58b4b586bcfe" containerID="62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70" exitCode=143
May 20 14:29:32.326456 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:32.326346    2566 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bh8l2/must-gather-w864p"
May 20 14:29:32.326456 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:32.326399    2566 scope.go:117] "RemoveContainer" containerID="62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70"
May 20 14:29:32.333377 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:32.333362    2566 scope.go:117] "RemoveContainer" containerID="b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047"
May 20 14:29:32.346269 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:32.346250    2566 scope.go:117] "RemoveContainer" containerID="62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70"
May 20 14:29:32.346511 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:29:32.346491    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70\": container with ID starting with 62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70 not found: ID does not exist" containerID="62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70"
May 20 14:29:32.346555 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:32.346519    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70"} err="failed to get container status \"62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70\": rpc error: code = NotFound desc = could not find container \"62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70\": container with ID starting with 62d0751917b9fd564576e54d2f2e12828065fbf12d9f04d3f2e893cf89349d70 not found: ID does not exist"
May 20 14:29:32.346555 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:32.346538    2566 scope.go:117] "RemoveContainer" containerID="b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047"
May 20 14:29:32.346783 ip-10-0-140-71 kubenswrapper[2566]: E0520 14:29:32.346769    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047\": container with ID starting with b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047 not found: ID does not exist" containerID="b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047"
May 20 14:29:32.346831 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:32.346788    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047"} err="failed to get container status \"b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047\": rpc error: code = NotFound desc = could not find container \"b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047\": container with ID starting with b7a76ef8f3cfff2143d13b9aebd8ad77c95697f702b8a1b9e0736833d7fdc047 not found: ID does not exist"
May 20 14:29:33.727625 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:33.727595    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-4dv2w_80129d87-90f4-43ef-90f3-3f16b9be4ec6/authorino/0.log"
May 20 14:29:33.751328 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:33.751306    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7587b89b76-r95s7_e10b8171-6aa5-49b6-be5d-dda923e1a9f3/manager/0.log"
May 20 14:29:35.276860 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:35.276830    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-47rm7_8f38fe80-2cd1-4b05-8934-6743f3dac8d8/node-exporter/0.log"
May 20 14:29:35.297109 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:35.297085    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-47rm7_8f38fe80-2cd1-4b05-8934-6743f3dac8d8/kube-rbac-proxy/0.log"
May 20 14:29:35.320332 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:35.320314    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-47rm7_8f38fe80-2cd1-4b05-8934-6743f3dac8d8/init-textfile/0.log"
May 20 14:29:35.586856 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:35.586786    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6ff3414-4a60-46ad-9329-632449a21641/prometheus/0.log"
May 20 14:29:35.607760 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:35.607739    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6ff3414-4a60-46ad-9329-632449a21641/config-reloader/0.log"
May 20 14:29:35.639380 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:35.639360    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6ff3414-4a60-46ad-9329-632449a21641/thanos-sidecar/0.log"
May 20 14:29:35.660585 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:35.660567    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6ff3414-4a60-46ad-9329-632449a21641/kube-rbac-proxy-web/0.log"
May 20 14:29:35.681807 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:35.681783    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6ff3414-4a60-46ad-9329-632449a21641/kube-rbac-proxy/0.log"
May 20 14:29:35.705439 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:35.705414    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6ff3414-4a60-46ad-9329-632449a21641/kube-rbac-proxy-thanos/0.log"
May 20 14:29:35.726602 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:35.726586    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d6ff3414-4a60-46ad-9329-632449a21641/init-config-reloader/0.log"
May 20 14:29:38.572720 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.572688    2566 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"]
May 20 14:29:38.573126 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.572955    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a208da-25fd-4114-8ee4-58b4b586bcfe" containerName="gather"
May 20 14:29:38.573126 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.572966    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a208da-25fd-4114-8ee4-58b4b586bcfe" containerName="gather"
May 20 14:29:38.573126 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.572980    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a208da-25fd-4114-8ee4-58b4b586bcfe" containerName="copy"
May 20 14:29:38.573126 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.572985    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a208da-25fd-4114-8ee4-58b4b586bcfe" containerName="copy"
May 20 14:29:38.573126 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.573042    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="93a208da-25fd-4114-8ee4-58b4b586bcfe" containerName="gather"
May 20 14:29:38.573126 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.573049    2566 memory_manager.go:356] "RemoveStaleState removing state" podUID="93a208da-25fd-4114-8ee4-58b4b586bcfe" containerName="copy"
May 20 14:29:38.579426 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.579403    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.582944 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.582924    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cpmvk\"/\"kube-root-ca.crt\""
May 20 14:29:38.583050 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.582924    2566 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cpmvk\"/\"openshift-service-ca.crt\""
May 20 14:29:38.584162 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.584143    2566 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cpmvk\"/\"default-dockercfg-rh55f\""
May 20 14:29:38.586293 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.586272    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"]
May 20 14:29:38.639875 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.639831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-sys\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.640045 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.639945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-proc\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.640045 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.639969    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj229\" (UniqueName: \"kubernetes.io/projected/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-kube-api-access-rj229\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.640116 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.640045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-podres\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.640116 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.640077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-lib-modules\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.741226 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.741191    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-lib-modules\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.741226 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.741229    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-sys\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.741425 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.741278    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-proc\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.741425 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.741294    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rj229\" (UniqueName: \"kubernetes.io/projected/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-kube-api-access-rj229\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.741425 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.741317    2566 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-podres\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.741425 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.741375    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-proc\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.741425 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.741378    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-lib-modules\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.741595 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.741418    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-sys\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.741595 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.741429    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-podres\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.749646 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.749613    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj229\" (UniqueName: \"kubernetes.io/projected/1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac-kube-api-access-rj229\") pod \"perf-node-gather-daemonset-ntv9x\" (UID: \"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:38.889489 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:38.889459    2566 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:39.008989 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:39.008968    2566 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"]
May 20 14:29:39.011292 ip-10-0-140-71 kubenswrapper[2566]: W0520 14:29:39.011262    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1a073ef0_3769_4c97_a5e4_1eac2bf8f4ac.slice/crio-d29fbde7bb2c0de893ec241978ffcdfba29aeb1e5159914310ae8be004a2d7b7 WatchSource:0}: Error finding container d29fbde7bb2c0de893ec241978ffcdfba29aeb1e5159914310ae8be004a2d7b7: Status 404 returned error can't find the container with id d29fbde7bb2c0de893ec241978ffcdfba29aeb1e5159914310ae8be004a2d7b7
May 20 14:29:39.346916 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:39.346826    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x" event={"ID":"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac","Type":"ContainerStarted","Data":"678044743fd8e969f2a5b066221791bc79660ff65f99d1e4650d7e635ca9f6f3"}
May 20 14:29:39.346916 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:39.346865    2566 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x" event={"ID":"1a073ef0-3769-4c97-a5e4-1eac2bf8f4ac","Type":"ContainerStarted","Data":"d29fbde7bb2c0de893ec241978ffcdfba29aeb1e5159914310ae8be004a2d7b7"}
May 20 14:29:39.346916 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:39.346892    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:39.364005 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:39.363965    2566 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x" podStartSLOduration=1.363952429 podStartE2EDuration="1.363952429s" podCreationTimestamp="2026-05-20 14:29:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:29:39.362604576 +0000 UTC m=+2339.917050151" watchObservedRunningTime="2026-05-20 14:29:39.363952429 +0000 UTC m=+2339.918398003"
May 20 14:29:39.472758 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:39.472731    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pnn8g_5a270e1c-51a9-4ae0-83a1-71ec680550ec/dns/0.log"
May 20 14:29:39.491644 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:39.491601    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pnn8g_5a270e1c-51a9-4ae0-83a1-71ec680550ec/kube-rbac-proxy/0.log"
May 20 14:29:39.534272 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:39.534246    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dxhmp_1e092c89-8dac-4254-bbf3-8db3d7058f76/dns-node-resolver/0.log"
May 20 14:29:40.003717 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:40.003689    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-75bbd8854-4b256_a20092fe-2492-4f66-ad11-d115e5783408/registry/0.log"
May 20 14:29:40.068075 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:40.068048    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v27tj_db48875c-3279-4fbe-8771-625c9099802f/node-ca/0.log"
May 20 14:29:41.394027 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:41.393999    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g6bws_e6d66e5d-0d38-4279-bb66-2177165a5342/serve-healthcheck-canary/0.log"
May 20 14:29:41.901869 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:41.901839    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kh7mn_86144d59-02a9-4ba9-b088-ca38960d39b1/kube-rbac-proxy/0.log"
May 20 14:29:41.920375 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:41.920354    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kh7mn_86144d59-02a9-4ba9-b088-ca38960d39b1/exporter/0.log"
May 20 14:29:41.940054 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:41.940031    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kh7mn_86144d59-02a9-4ba9-b088-ca38960d39b1/extractor/0.log"
May 20 14:29:44.496004 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:44.495976    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79d5949c76-m9d7g_c1ac38c3-08ab-4d1f-a2cd-d156716a95a0/manager/0.log"
May 20 14:29:45.070467 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:45.070442    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6bfc7d65cd-k5vsm_af15db72-0136-46a1-9ab7-613b8c2a1829/manager/0.log"
May 20 14:29:45.358832 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:45.358806    2566 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-ntv9x"
May 20 14:29:51.616268 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:51.616242    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfj8n_8959890a-80d9-44d7-a2f1-64b78a493e34/kube-multus-additional-cni-plugins/0.log"
May 20 14:29:51.636627 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:51.636598    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfj8n_8959890a-80d9-44d7-a2f1-64b78a493e34/egress-router-binary-copy/0.log"
May 20 14:29:51.656962 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:51.656942    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfj8n_8959890a-80d9-44d7-a2f1-64b78a493e34/cni-plugins/0.log"
May 20 14:29:51.675213 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:51.675194    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfj8n_8959890a-80d9-44d7-a2f1-64b78a493e34/bond-cni-plugin/0.log"
May 20 14:29:51.694225 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:51.694205    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfj8n_8959890a-80d9-44d7-a2f1-64b78a493e34/routeoverride-cni/0.log"
May 20 14:29:51.712523 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:51.712502    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfj8n_8959890a-80d9-44d7-a2f1-64b78a493e34/whereabouts-cni-bincopy/0.log"
May 20 14:29:51.733064 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:51.733043    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfj8n_8959890a-80d9-44d7-a2f1-64b78a493e34/whereabouts-cni/0.log"
May 20 14:29:51.909930 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:51.909857    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dm4bw_24d3b560-c9fd-48f2-9967-def69e9bb2f8/kube-multus/0.log"
May 20 14:29:52.054729 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:52.054692    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-twxjw_629f62b6-2132-440f-9f2f-e327d44004c9/network-metrics-daemon/0.log"
May 20 14:29:52.072480 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:52.072446    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-twxjw_629f62b6-2132-440f-9f2f-e327d44004c9/kube-rbac-proxy/0.log"
May 20 14:29:53.210092 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:53.210067    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rdtkf_85434994-53d3-48f0-beaf-f4b9e3ecdeb7/ovn-controller/0.log"
May 20 14:29:53.237113 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:53.237081    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rdtkf_85434994-53d3-48f0-beaf-f4b9e3ecdeb7/ovn-acl-logging/0.log"
May 20 14:29:53.254233 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:53.254194    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rdtkf_85434994-53d3-48f0-beaf-f4b9e3ecdeb7/kube-rbac-proxy-node/0.log"
May 20 14:29:53.272487 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:53.272468    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rdtkf_85434994-53d3-48f0-beaf-f4b9e3ecdeb7/kube-rbac-proxy-ovn-metrics/0.log"
May 20 14:29:53.290065 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:53.290048    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rdtkf_85434994-53d3-48f0-beaf-f4b9e3ecdeb7/northd/0.log"
May 20 14:29:53.309330 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:53.309310    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rdtkf_85434994-53d3-48f0-beaf-f4b9e3ecdeb7/nbdb/0.log"
May 20 14:29:53.327604 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:53.327583    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rdtkf_85434994-53d3-48f0-beaf-f4b9e3ecdeb7/sbdb/0.log"
May 20 14:29:53.424726 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:53.424703    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rdtkf_85434994-53d3-48f0-beaf-f4b9e3ecdeb7/ovnkube-controller/0.log"
May 20 14:29:54.854469 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:54.854440    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-v8vbc_4452f914-4de0-4fbb-bbfe-237b2cc85d2c/network-check-target-container/0.log"
May 20 14:29:55.865008 ip-10-0-140-71 kubenswrapper[2566]: I0520 14:29:55.864977    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-vpxdx_2810ffa5-0463-4a15-9729-c1d04fd38658/iptables-alerter/0.log"