Jul 16 16:49:21.259862 ip-10-0-133-149 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 16:49:21.259871 ip-10-0-133-149 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 16:49:21.259878 ip-10-0-133-149 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 16:49:21.260116 ip-10-0-133-149 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 16:49:31.510034 ip-10-0-133-149 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 16:49:31.510052 ip-10-0-133-149 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ee2fc1fca469425f8fa854519bdbea19 -- Jul 16 16:51:08.311280 ip-10-0-133-149 systemd[1]: Starting Kubernetes Kubelet... Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.760588 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765312 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765324 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765327 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765330 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765341 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765345 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765348 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:08.767338 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765350 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765354 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765357 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765360 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765363 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765366 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765368 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765371 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765374 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765376 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765379 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765381 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765384 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765386 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765389 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765392 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765394 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765399 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765403 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765406 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:08.771368 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765409 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765412 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765415 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765418 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765420 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765423 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765425 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765428 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765430 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765433 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765435 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765438 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765440 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765442 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765445 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765447 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765450 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765453 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765455 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:08.788691 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765457 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765470 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765474 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765477 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765479 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765482 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765485 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765487 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765490 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765493 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765496 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765498 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765507 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765510 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765512 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765515 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765517 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765519 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765522 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765524 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:08.790842 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765527 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765531 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765534 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765536 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765538 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765541 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765543 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765546 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765549 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765551 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765553 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765556 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765558 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765560 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765563 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765565 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765568 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765570 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765573 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765576 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:08.794274 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765578 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765580 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765583 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765587 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765590 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765600 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765603 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765606 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765609 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765611 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765614 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765616 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.765618 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.769970 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.769984 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.769988 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.769992 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.769995 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.769998 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:08.804682 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770001 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770004 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770007 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770010 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770027 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770034 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770038 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770042 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770046 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770056 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770060 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770063 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770066 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770068 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770071 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770076 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770080 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770086 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770089 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:08.806909 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770092 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770095 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770097 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770100 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770102 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770106 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770110 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770113 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770124 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770127 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770132 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770135 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770138 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770140 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770143 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770147 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770149 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770152 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770155 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770158 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:08.840978 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770160 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770163 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770168 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770170 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770173 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770175 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770178 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770181 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770183 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770186 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770188 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770191 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770193 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770196 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770198 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770206 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770211 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770220 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770223 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770226 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:08.843251 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770229 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770238 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770241 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770243 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770246 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770251 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770261 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770264 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770267 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770270 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770273 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770276 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770279 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770282 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770284 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770287 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770289 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770292 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770295 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:08.908606 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770300 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:08.866692 ip-10-0-133-149 systemd[1]: Started Kubernetes Kubelet. Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770303 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770305 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770307 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770310 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770313 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770315 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770318 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770320 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770323 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770325 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770328 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770333 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770336 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.770345 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771077 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771084 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771086 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771089 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:08.911196 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771092 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771095 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771098 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771103 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771107 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771110 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771115 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771118 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771123 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771126 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771129 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771132 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771135 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771138 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771141 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771144 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771147 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771150 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771153 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:09.113681 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771158 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771162 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771166 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771169 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771172 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771175 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771178 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771182 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771187 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771198 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771202 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771205 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771211 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771214 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771219 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771221 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771224 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771227 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771229 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:09.135336 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771232 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771235 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771237 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771240 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771242 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771248 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771250 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771253 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771255 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771258 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771260 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771264 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771267 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771270 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771272 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771275 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771277 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771284 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771286 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771289 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:09.136688 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771291 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771294 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771296 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771299 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771302 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771305 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771307 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771310 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771312 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771317 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771320 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771322 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771325 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771328 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771330 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771333 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771335 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771337 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771341 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771345 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:09.138963 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771349 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771468 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771490 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771494 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771497 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771501 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771503 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771506 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771509 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771512 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771514 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771517 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771520 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771523 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771525 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771528 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.771532 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771629 2556 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771643 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771656 2556 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771662 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 16:51:09.139997 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771666 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771669 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771673 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771677 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771680 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771683 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771687 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771690 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771693 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771696 2556 flags.go:64] FLAG: --cgroup-root="" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771699 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771702 2556 flags.go:64] FLAG: --client-ca-file="" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771705 2556 flags.go:64] FLAG: --cloud-provider="external" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771708 2556 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771713 2556 flags.go:64] FLAG: --cluster-domain="" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771716 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771719 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771723 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771726 2556 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771729 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771732 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771735 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771739 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771742 2556 flags.go:64] FLAG: --contention-profiling="false" Jul 16 16:51:09.141447 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771744 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771747 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771751 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771754 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771758 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771761 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771764 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771766 2556 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771769 2556 flags.go:64] FLAG: --enable-server="true" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771772 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771782 2556 flags.go:64] FLAG: --event-burst="100" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771785 2556 flags.go:64] FLAG: --event-qps="50" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771788 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771790 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771793 2556 flags.go:64] FLAG: --eviction-hard="" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771797 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771799 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771802 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771805 2556 flags.go:64] FLAG: --eviction-soft="" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771808 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771811 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771815 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771817 2556 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771820 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771824 2556 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 16:51:09.142888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771827 2556 flags.go:64] FLAG: --feature-gates="" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771830 2556 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771833 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771836 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771839 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771842 2556 flags.go:64] FLAG: --healthz-port="10248" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771845 2556 flags.go:64] FLAG: --help="false" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771848 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771851 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771853 2556 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771857 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771861 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771864 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.771866 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772841 2556 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772847 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772850 2556 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772853 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772857 2556 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772860 2556 flags.go:64] FLAG: --kube-reserved="" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772863 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772866 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772869 2556 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772872 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 16:51:09.144805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772875 2556 flags.go:64] FLAG: --lock-file="" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.772877 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774140 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774147 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774153 2556 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774160 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774163 2556 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774167 2556 flags.go:64] FLAG: --logging-format="text" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774172 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774179 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774183 2556 flags.go:64] FLAG: --manifest-url="" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774188 2556 flags.go:64] FLAG: --manifest-url-header="" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774202 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774206 2556 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774213 2556 flags.go:64] FLAG: --max-pods="110" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774218 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774222 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774227 2556 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774232 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774237 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774241 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774246 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774257 2556 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774262 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774267 2556 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 16:51:09.147634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774272 2556 flags.go:64] FLAG: --pod-cidr="" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774276 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774285 2556 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774289 2556 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774294 2556 flags.go:64] FLAG: --pods-per-core="0" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774299 2556 flags.go:64] FLAG: --port="10250" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774304 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774308 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08eac2034d4ab47e8" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774313 2556 flags.go:64] FLAG: --qos-reserved="" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774318 2556 flags.go:64] FLAG: --read-only-port="10255" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774322 2556 flags.go:64] FLAG: --register-node="true" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774327 2556 flags.go:64] FLAG: --register-with-taints="" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774332 2556 flags.go:64] FLAG: --registry-burst="10" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774338 2556 flags.go:64] FLAG: --registry-qps="5" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774342 2556 flags.go:64] FLAG: --reserved-cpus="" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774347 2556 flags.go:64] FLAG: --reserved-memory="" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774352 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774357 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774362 2556 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774366 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774371 2556 flags.go:64] FLAG: --runonce="false" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774376 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774381 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774386 2556 flags.go:64] FLAG: --seccomp-default="false" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774390 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 16:51:09.148715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774395 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774399 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774404 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774409 2556 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774414 2556 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774419 2556 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774424 2556 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774429 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774433 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774438 2556 flags.go:64] FLAG: --system-cgroups="" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774443 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774452 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774457 2556 flags.go:64] FLAG: --tls-cert-file="" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774462 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774473 2556 flags.go:64] FLAG: --tls-min-version="" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774477 2556 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774482 2556 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774486 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774491 2556 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774496 2556 flags.go:64] FLAG: --v="2" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774502 2556 flags.go:64] FLAG: --version="false" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774509 2556 flags.go:64] FLAG: --vmodule="" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774515 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.774520 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774709 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:09.149785 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774715 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774721 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774726 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774729 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774734 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774738 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774742 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774746 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774751 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774755 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774759 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774763 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774767 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774772 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774776 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774780 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774785 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774789 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774793 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:09.150797 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774797 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774801 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774805 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774809 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774813 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774817 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774821 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774826 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774829 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774833 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774838 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774842 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774846 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774850 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774854 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774862 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774866 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774870 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774874 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774877 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:09.151541 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774882 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774885 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774890 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774894 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774898 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774902 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774907 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774911 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774915 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774919 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774924 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774930 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774936 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774942 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774947 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774951 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774955 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774959 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774963 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:09.152427 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774967 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774971 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774975 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774979 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774983 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774987 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774992 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.774996 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775002 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775006 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775010 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775032 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775037 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775041 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775045 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775049 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775053 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775057 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775061 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775067 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:09.153172 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775072 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775076 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775080 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775084 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775088 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775092 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775096 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775100 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775105 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775109 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775112 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775116 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775120 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775124 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775128 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775134 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775139 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775144 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775148 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:09.153948 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.775152 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.775965 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.783636 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.783657 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.783676 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.783684 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783759 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783766 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783771 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783775 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783780 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783785 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783791 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783796 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783800 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:09.154692 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783804 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783808 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783812 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783816 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783821 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783825 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783829 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783833 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783838 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783841 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783845 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783849 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783854 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783858 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783862 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783866 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783869 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783873 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783877 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783883 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:09.155295 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783888 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783893 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783896 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783900 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783904 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783909 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783913 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783917 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783921 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783926 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783930 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783934 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783938 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783942 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783946 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783950 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783954 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783958 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783962 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:09.156081 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783966 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783970 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783974 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783978 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783982 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783987 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783993 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.783998 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784002 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784007 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784011 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784032 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784038 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784042 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784048 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784052 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784056 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784061 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784065 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:09.156782 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784068 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784072 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784076 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784082 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784087 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784092 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784098 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784102 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784107 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784111 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784115 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784119 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784123 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784128 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784132 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784136 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784140 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784144 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784148 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:09.157536 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784152 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784158 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784164 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784169 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784174 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784178 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784184 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784188 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784192 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784196 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784201 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784204 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784208 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.784216 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784373 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:09.158269 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784380 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784386 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784390 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784394 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784399 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784403 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784408 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784411 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784416 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784419 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784424 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784427 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784432 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784436 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784440 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784444 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784447 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784451 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784455 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784459 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:09.158849 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784464 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784468 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784472 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784476 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784482 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784486 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784490 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784495 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784500 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784503 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784507 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784512 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784516 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784522 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784526 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784530 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784534 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784538 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784543 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784547 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:09.159631 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784551 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784555 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784559 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784563 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784568 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784571 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784575 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784579 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784583 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784587 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784592 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784596 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784600 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784603 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784607 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784611 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784616 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784621 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784627 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:09.160388 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784632 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784645 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784650 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784654 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784658 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784661 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784667 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784671 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784675 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784679 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784682 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784686 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784690 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784694 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784698 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784702 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784706 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784710 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784714 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784718 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:09.161134 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784722 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784729 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784733 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784740 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784745 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784749 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784754 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784758 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784762 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784768 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784772 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784776 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784780 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784784 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784795 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784799 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784803 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784808 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:09.161885 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:08.784814 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.784820 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.785003 2556 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.787660 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.789323 2556 server.go:1019] "Starting client certificate rotation" Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.789429 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.789459 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.792143 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.793178 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.810884 2556 log.go:25] "Validated CRI v1 runtime API" Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.820147 2556 log.go:25] "Validated CRI v1 image API" Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.821326 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 16:51:09.162633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.823967 2556 fs.go:135] Filesystem UUIDs: map[6bc4a746-19a7-4eea-83c5-b2a7f078ce49:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ab4dcbdd-d290-41b6-934e-5e985d98e31b:/dev/nvme0n1p4] Jul 16 16:51:09.163098 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.823985 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 16:51:09.163098 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.824707 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.830429 2556 manager.go:216] Machine: {Timestamp:2026-07-16 16:51:08.829046818 +0000 UTC m=+0.388599269 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3197970 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c2a24e5a633d9dd736f464e6b316d SystemUUID:ec2c2a24-e5a6-33d9-dd73-6f464e6b316d BootID:ee2fc1fc-a469-425f-8fa8-54519bdbea19 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:11:82:af:66:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:11:82:af:66:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:9f:0f:24:51:bc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.830590 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.830764 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.834062 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.834085 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-149.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.834875 2556 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.834885 2556 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.834892 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.836627 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.838294 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.842470 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.842580 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.842606 2556 kubelet.go:486] "Attempting to sync node with API server" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.842615 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.842629 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.842638 2556 kubelet.go:398] "Adding apiserver pod source" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.842648 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 16:51:09.163195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.842903 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.842916 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.852386 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.860445 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.860472 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.860631 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.860644 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.860650 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.860655 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.860665 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.860674 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.861225 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.861232 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.861239 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.861245 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.861253 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.861261 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.862102 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.862112 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.864596 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.865246 2556 server.go:1268] "Started kubelet" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.865348 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.865449 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.865490 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.867183 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.868121 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.873670 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18c2d33ec09d0cff default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:ip-10-0-133-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-07-16 16:51:08.865088767 +0000 UTC m=+0.424641273,LastTimestamp:2026-07-16 16:51:08.865088767 +0000 UTC m=+0.424641273,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.877865 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.877900 2556 server.go:310] "Adding debug handlers to kubelet server" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.882734 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.883107 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.883970 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.883979 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.884199 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.884596 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.884682 2556 reconstruct.go:97] "Volume reconstruction finished" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.884688 2556 reconciler.go:29] "Reconciler: start to sync state" Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.885669 2556 factory.go:55] Registering systemd factory Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.885680 2556 factory.go:223] Registration of the systemd container factory successfully Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.885885 2556 factory.go:153] Registering CRI-O factory Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.885901 2556 factory.go:223] Registration of the crio container factory successfully Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.885946 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 16:51:09.163927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.885971 2556 factory.go:103] Registering Raw factory Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.885987 2556 manager.go:1196] Started watching for new ooms in manager Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.886412 2556 manager.go:349] Starting recovery of all containers Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.887568 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.887663 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-149.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.887764 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.897488 2556 manager.go:354] Recovery completed Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.902891 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.904829 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.908583 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.908600 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.909102 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.909116 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.909127 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.909130 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.909975 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.909991 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.910000 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.910762 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18c2d33ec3346c64 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:ip-10-0-133-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-07-16 16:51:08.908563556 +0000 UTC m=+0.468116013,LastTimestamp:2026-07-16 16:51:08.908563556 +0000 UTC m=+0.468116013,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.911154 2556 policy_none.go:49] "None policy: Start" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.911166 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.911175 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.913086 2556 policy_none.go:47] "Start" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.927444 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18c2d33ec334d672 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:ip-10-0-133-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-07-16 16:51:08.908590706 +0000 UTC m=+0.468143162,LastTimestamp:2026-07-16 16:51:08.908590706 +0000 UTC m=+0.468143162,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Jul 16 16:51:09.165620 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.935809 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18c2d33ec33513ec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:ip-10-0-133-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-07-16 16:51:08.908606444 +0000 UTC m=+0.468158896,LastTimestamp:2026-07-16 16:51:08.908606444 +0000 UTC m=+0.468158896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.940045 2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-149.ec2.internal.18c2d33ec3346c64\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18c2d33ec3346c64 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:ip-10-0-133-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-07-16 16:51:08.908563556 +0000 UTC m=+0.468116013,LastTimestamp:2026-07-16 16:51:08.909983621 +0000 UTC m=+0.469536071,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.948511 2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-149.ec2.internal.18c2d33ec334d672\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18c2d33ec334d672 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:ip-10-0-133-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-07-16 16:51:08.908590706 +0000 UTC m=+0.468143162,LastTimestamp:2026-07-16 16:51:08.909994782 +0000 UTC m=+0.469547233,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.950681 2556 manager.go:337] "Starting Device Plugin manager" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.950707 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.950719 2556 server.go:87] "Starting device plugin registration server" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.950933 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.950943 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.951040 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.951126 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.951135 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.951455 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.951546 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.952429 2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-149.ec2.internal.18c2d33ec33513ec\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18c2d33ec33513ec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:ip-10-0-133-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-07-16 16:51:08.908606444 +0000 UTC m=+0.468158896,LastTimestamp:2026-07-16 16:51:08.910003731 +0000 UTC m=+0.469556182,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:08.962449 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18c2d33ec64b6486 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:ip-10-0-133-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-07-16 16:51:08.960400518 +0000 UTC m=+0.519952960,LastTimestamp:2026-07-16 16:51:08.960400518 +0000 UTC m=+0.519952960,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:08.996247 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5vv2r" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.006103 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5vv2r" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.051644 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:09.166584 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.052437 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.052464 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.052476 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.052502 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.059496 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.060726 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.060757 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.060776 2556 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:09.060806 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.063172 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.069503 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:09.069523 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-149.ec2.internal\": node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:09.086285 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.161667 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal"] Jul 16 16:51:09.167370 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.161851 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:09.173024 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.172996 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:09.173104 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.173038 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:09.173104 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.173049 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:09.175175 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.175164 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:09.175267 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.175254 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.175297 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.175280 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:09.176040 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.176009 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:09.176115 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.176050 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:09.176115 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.176065 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:09.176115 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.176069 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:09.176115 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.176085 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:09.176115 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.176096 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:09.178247 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.178232 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.178290 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.178265 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:09.179364 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.179346 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:09.179428 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.179365 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:09.179428 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.179378 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:09.186323 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.186308 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44e0d5f2b8dc87c8745e7edd3a441ecf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"44e0d5f2b8dc87c8745e7edd3a441ecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.186323 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:09.186319 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:09.186438 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.186329 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44e0d5f2b8dc87c8745e7edd3a441ecf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"44e0d5f2b8dc87c8745e7edd3a441ecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.186438 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.186355 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/556eca662b4e5f951dd2eca461c87322-config\") pod \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" (UID: \"556eca662b4e5f951dd2eca461c87322\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.215579 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:09.215558 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-149.ec2.internal\" not found" node="ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.219808 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:09.219793 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-149.ec2.internal\" not found" node="ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.286600 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:09.286576 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:09.286701 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.286595 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44e0d5f2b8dc87c8745e7edd3a441ecf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"44e0d5f2b8dc87c8745e7edd3a441ecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.286701 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.286655 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44e0d5f2b8dc87c8745e7edd3a441ecf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"44e0d5f2b8dc87c8745e7edd3a441ecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.286701 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.286676 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44e0d5f2b8dc87c8745e7edd3a441ecf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"44e0d5f2b8dc87c8745e7edd3a441ecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.286798 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.286709 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/556eca662b4e5f951dd2eca461c87322-config\") pod \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" (UID: \"556eca662b4e5f951dd2eca461c87322\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.286798 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.286737 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/556eca662b4e5f951dd2eca461c87322-config\") pod \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" (UID: \"556eca662b4e5f951dd2eca461c87322\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.286798 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:09.286758 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44e0d5f2b8dc87c8745e7edd3a441ecf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"44e0d5f2b8dc87c8745e7edd3a441ecf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Jul 16 16:51:09.387510 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:09.387421 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:11.263279 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:11.263261 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:11.292510 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.292467 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Jul 16 16:51:11.298253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.297634 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Jul 16 16:51:11.363499 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:11.363465 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:11.463877 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:11.463814 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:11.564215 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:11.564186 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:11.564215 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.564209 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 16:51:11.564699 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.564396 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 16:51:11.664718 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:11.664688 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:11.765177 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:11.765115 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Jul 16 16:51:11.785922 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.785888 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 16:46:10 +0000 UTC" deadline="2027-12-15 19:12:27.788653875 +0000 UTC" Jul 16 16:51:11.785922 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.785918 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12410h21m16.002740341s" Jul 16 16:51:11.786296 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.786281 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:11.860099 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.860073 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Jul 16 16:51:11.874583 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.874563 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 16:51:11.876793 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.876775 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Jul 16 16:51:11.886295 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.886274 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 16:51:11.951752 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.951724 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:11.983153 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:11.983117 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e0d5f2b8dc87c8745e7edd3a441ecf.slice/crio-4d7141770de373b6a47275f3b97519630901bb555705830659fa694b578adeba WatchSource:0}: Error finding container 4d7141770de373b6a47275f3b97519630901bb555705830659fa694b578adeba: Status 404 returned error can't find the container with id 4d7141770de373b6a47275f3b97519630901bb555705830659fa694b578adeba Jul 16 16:51:11.983489 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:11.983464 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556eca662b4e5f951dd2eca461c87322.slice/crio-1aece23dad506b7bb33383346a2b3072bec01158a3153141efec81ac75f6ea05 WatchSource:0}: Error finding container 1aece23dad506b7bb33383346a2b3072bec01158a3153141efec81ac75f6ea05: Status 404 returned error can't find the container with id 1aece23dad506b7bb33383346a2b3072bec01158a3153141efec81ac75f6ea05 Jul 16 16:51:11.988086 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:11.988073 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 16:51:12.156879 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.156800 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:12.542001 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.541926 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:12.619406 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.619136 2556 apiserver.go:52] "Watching apiserver" Jul 16 16:51:12.624752 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.624717 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 16:51:12.625242 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.625173 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-cl2mq","openshift-multus/multus-cjqtv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal","openshift-multus/network-metrics-daemon-sw5rt","openshift-network-diagnostics/network-check-target-stcqc","openshift-network-operator/iptables-alerter-kqz8h","openshift-ovn-kubernetes/ovnkube-node-x84hq","kube-system/global-pull-secret-syncer-gj948","kube-system/konnectivity-agent-kf8nk","kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4kstz","openshift-dns/node-resolver-nh2lf","openshift-image-registry/node-ca-dgrxk"] Jul 16 16:51:12.627761 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.627503 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.629904 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.629883 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.629993 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.629924 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 16:51:12.630383 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.630183 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lhmpg\"" Jul 16 16:51:12.630494 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.630473 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 16:51:12.633069 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.631686 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 16:51:12.633069 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.632082 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.633069 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.632252 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 16:51:12.633069 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.632508 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p6zgn\"" Jul 16 16:51:12.633069 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.632741 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 16:51:12.633069 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.632912 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 16:51:12.634298 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.634239 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jjd89\"" Jul 16 16:51:12.634513 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.634448 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 16:51:12.635457 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.634880 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 16:51:12.635457 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.634938 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.635457 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.635114 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 16:51:12.637832 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.637695 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 16:51:12.639599 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.638489 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 16:51:12.639599 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.638030 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qbkqs\"" Jul 16 16:51:12.639599 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.639030 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:12.639599 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.639186 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:12.642326 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.642106 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:12.642326 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.642265 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:12.644922 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.644903 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.646970 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.646945 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 16:51:12.647325 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.647306 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 16:51:12.647982 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.647545 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 16:51:12.648376 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.648359 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.649242 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.648864 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xmqvz\"" Jul 16 16:51:12.650493 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.650473 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:12.651103 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.651088 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 16:51:12.651349 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.651328 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 16:51:12.651528 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.651513 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 16:51:12.651703 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.651664 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 16:51:12.652040 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.652022 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c8b42\"" Jul 16 16:51:12.652358 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.652321 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 16:51:12.652578 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.652562 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 16:51:12.653697 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.652845 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:12.653697 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.653449 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 16:51:12.653697 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.653463 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 16:51:12.653697 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.653615 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7c876\"" Jul 16 16:51:12.655926 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.654993 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 16:51:12.655926 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.655362 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 16:51:12.655926 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.655620 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5btxt\"" Jul 16 16:51:12.655926 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.655761 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:12.655926 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.655809 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:12.658741 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.657684 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:12.660096 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.660080 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kzknz\"" Jul 16 16:51:12.660493 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.660104 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 16:51:12.660493 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.660212 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 16:51:12.660493 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.660167 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 16:51:12.660493 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.660474 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 16:51:12.662021 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.661692 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 16:51:12.669355 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.669336 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-cni-dir\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.669535 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.669519 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-daemon-config\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.669678 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.669661 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt76r\" (UniqueName: \"kubernetes.io/projected/9ad7a19b-83c8-404e-b28b-464de3f17e16-kube-api-access-lt76r\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.669829 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.669812 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-run\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.669956 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.669942 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.670049 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.670037 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtqll\" (UniqueName: \"kubernetes.io/projected/51008c04-b060-46e6-86e7-07ee832e8aba-kube-api-access-qtqll\") pod \"iptables-alerter-kqz8h\" (UID: \"51008c04-b060-46e6-86e7-07ee832e8aba\") " pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:12.670182 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.670165 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-os-release\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.670281 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.670268 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-kubelet\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.670433 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.670399 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-var-lib-openvswitch\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.670540 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.670527 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a69b989f-c7db-4193-ae08-0686fcc49150-hosts-file\") pod \"node-resolver-nh2lf\" (UID: \"a69b989f-c7db-4193-ae08-0686fcc49150\") " pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:12.670668 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.670653 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-sys\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.670785 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.670771 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-lib-modules\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.670912 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.670898 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26e859cc-9613-4ff0-885d-167beeed43aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.671026 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.671010 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz8s8\" (UniqueName: \"kubernetes.io/projected/5367035f-cfe2-45b8-a2f3-f462d3f043b3-kube-api-access-zz8s8\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.671565 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.671537 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ad7a19b-83c8-404e-b28b-464de3f17e16-ovn-node-metrics-cert\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.671739 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.671723 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxb4p\" (UniqueName: \"kubernetes.io/projected/26e859cc-9613-4ff0-885d-167beeed43aa-kube-api-access-bxb4p\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.671878 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.671859 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:12.672023 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.672008 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-var-lib-kubelet\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.672805 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.672786 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-registration-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.673806 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ad7a19b-83c8-404e-b28b-464de3f17e16-env-overrides\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.673834 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x97f\" (UniqueName: \"kubernetes.io/projected/a69b989f-c7db-4193-ae08-0686fcc49150-kube-api-access-2x97f\") pod \"node-resolver-nh2lf\" (UID: \"a69b989f-c7db-4193-ae08-0686fcc49150\") " pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.673856 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-var-lib-kubelet\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.673890 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-conf-dir\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.673913 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-run-multus-certs\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.673939 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-device-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.673972 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59c160d1-15f8-45b6-86cf-a2ed1fce715c-serviceca\") pod \"node-ca-dgrxk\" (UID: \"59c160d1-15f8-45b6-86cf-a2ed1fce715c\") " pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.673993 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-node-log\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674015 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-run-ovn\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674041 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.673452 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674062 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c582a6c2-65b9-40e6-8382-a7c3133552cd-dbus\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674092 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-var-lib-cni-multus\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674114 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgxh9\" (UniqueName: \"kubernetes.io/projected/59c160d1-15f8-45b6-86cf-a2ed1fce715c-kube-api-access-fgxh9\") pod \"node-ca-dgrxk\" (UID: \"59c160d1-15f8-45b6-86cf-a2ed1fce715c\") " pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674135 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-run-openvswitch\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674160 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-os-release\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.674538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674183 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b89bf64-dbd2-4b29-95e1-25a45fc84481-konnectivity-ca\") pod \"konnectivity-agent-kf8nk\" (UID: \"0b89bf64-dbd2-4b29-95e1-25a45fc84481\") " pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674226 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-run-k8s-cni-cncf-io\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674248 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-sys-fs\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674274 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-log-socket\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-cni-netd\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674409 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69b989f-c7db-4193-ae08-0686fcc49150-tmp-dir\") pod \"node-resolver-nh2lf\" (UID: \"a69b989f-c7db-4193-ae08-0686fcc49150\") " pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674482 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-hostroot\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674524 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674554 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-sysctl-conf\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674580 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-host\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674607 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-cnibin\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674632 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-var-lib-cni-bin\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674656 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674683 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-etc-openvswitch\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674706 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-systemd\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674730 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-tuned\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674756 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c582a6c2-65b9-40e6-8382-a7c3133552cd-kubelet-config\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:12.675329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674781 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-etc-kubernetes\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674807 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-systemd-units\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674839 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-kubernetes\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674863 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674889 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-sysconfig\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674921 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-run-netns\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674945 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ad7a19b-83c8-404e-b28b-464de3f17e16-ovnkube-config\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.674967 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-sysctl-d\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.675001 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a95ad29c-98cf-4c52-af59-05aa91ffe20e-tmp\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.675032 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxsvk\" (UniqueName: \"kubernetes.io/projected/a95ad29c-98cf-4c52-af59-05aa91ffe20e-kube-api-access-dxsvk\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.675080 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-system-cni-dir\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.675109 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-socket-dir-parent\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.675283 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ad7a19b-83c8-404e-b28b-464de3f17e16-ovnkube-script-lib\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.675315 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-modprobe-d\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676037 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-system-cni-dir\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676068 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-run-netns\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.676094 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676097 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmd78\" (UniqueName: \"kubernetes.io/projected/e87197a5-760c-4729-90d5-3a021851f948-kube-api-access-bmd78\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676134 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktwzj\" (UniqueName: \"kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj\") pod \"network-check-target-stcqc\" (UID: \"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c\") " pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676169 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-cni-bin\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676195 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b89bf64-dbd2-4b29-95e1-25a45fc84481-agent-certs\") pod \"konnectivity-agent-kf8nk\" (UID: \"0b89bf64-dbd2-4b29-95e1-25a45fc84481\") " pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676218 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-cnibin\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676240 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-socket-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676280 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6ft\" (UniqueName: \"kubernetes.io/projected/51875cc1-104f-460c-b55f-17a7340ce099-kube-api-access-md6ft\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-run-systemd\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676330 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26e859cc-9613-4ff0-885d-167beeed43aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676356 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51008c04-b060-46e6-86e7-07ee832e8aba-iptables-alerter-script\") pod \"iptables-alerter-kqz8h\" (UID: \"51008c04-b060-46e6-86e7-07ee832e8aba\") " pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676387 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5367035f-cfe2-45b8-a2f3-f462d3f043b3-cni-binary-copy\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676409 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59c160d1-15f8-45b6-86cf-a2ed1fce715c-host\") pod \"node-ca-dgrxk\" (UID: \"59c160d1-15f8-45b6-86cf-a2ed1fce715c\") " pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676459 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-slash\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676483 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-run-ovn-kubernetes\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676507 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26e859cc-9613-4ff0-885d-167beeed43aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.676911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.676538 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51008c04-b060-46e6-86e7-07ee832e8aba-host-slash\") pod \"iptables-alerter-kqz8h\" (UID: \"51008c04-b060-46e6-86e7-07ee832e8aba\") " pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:12.727319 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.727294 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tlsqw" Jul 16 16:51:12.737126 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.737056 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tlsqw" Jul 16 16:51:12.777634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777491 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-systemd\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.777634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777541 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-tuned\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.777634 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777570 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c582a6c2-65b9-40e6-8382-a7c3133552cd-kubelet-config\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:12.777892 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777656 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-systemd\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.777892 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777706 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-etc-kubernetes\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.777892 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777737 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-systemd-units\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.777892 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-kubernetes\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.777892 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777786 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:12.777892 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777808 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-sysconfig\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.777892 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777831 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-run-netns\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.777892 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777853 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ad7a19b-83c8-404e-b28b-464de3f17e16-ovnkube-config\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.777892 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777876 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-sysctl-d\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777898 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a95ad29c-98cf-4c52-af59-05aa91ffe20e-tmp\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777902 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777921 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxsvk\" (UniqueName: \"kubernetes.io/projected/a95ad29c-98cf-4c52-af59-05aa91ffe20e-kube-api-access-dxsvk\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777944 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-system-cni-dir\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777967 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-socket-dir-parent\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777967 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-kubernetes\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.777990 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ad7a19b-83c8-404e-b28b-464de3f17e16-ovnkube-script-lib\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778012 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-modprobe-d\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778032 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-etc-kubernetes\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778035 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-system-cni-dir\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-run-netns\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778071 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-systemd-units\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778085 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmd78\" (UniqueName: \"kubernetes.io/projected/e87197a5-760c-4729-90d5-3a021851f948-kube-api-access-bmd78\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778097 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-system-cni-dir\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778121 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwzj\" (UniqueName: \"kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj\") pod \"network-check-target-stcqc\" (UID: \"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c\") " pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778146 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-cni-bin\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778168 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b89bf64-dbd2-4b29-95e1-25a45fc84481-agent-certs\") pod \"konnectivity-agent-kf8nk\" (UID: \"0b89bf64-dbd2-4b29-95e1-25a45fc84481\") " pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:12.778282 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.778176 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778189 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-cnibin\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778222 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-socket-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.778244 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs podName:51875cc1-104f-460c-b55f-17a7340ce099 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:13.278225607 +0000 UTC m=+3.062380652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs") pod "network-metrics-daemon-sw5rt" (UID: "51875cc1-104f-460c-b55f-17a7340ce099") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778264 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6ft\" (UniqueName: \"kubernetes.io/projected/51875cc1-104f-460c-b55f-17a7340ce099-kube-api-access-md6ft\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778282 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-run-systemd\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778306 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26e859cc-9613-4ff0-885d-167beeed43aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778330 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51008c04-b060-46e6-86e7-07ee832e8aba-iptables-alerter-script\") pod \"iptables-alerter-kqz8h\" (UID: \"51008c04-b060-46e6-86e7-07ee832e8aba\") " pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778351 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-socket-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778359 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5367035f-cfe2-45b8-a2f3-f462d3f043b3-cni-binary-copy\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778382 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59c160d1-15f8-45b6-86cf-a2ed1fce715c-host\") pod \"node-ca-dgrxk\" (UID: \"59c160d1-15f8-45b6-86cf-a2ed1fce715c\") " pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778400 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-run-netns\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778407 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-slash\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778447 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-run-ovn-kubernetes\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778473 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26e859cc-9613-4ff0-885d-167beeed43aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778497 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51008c04-b060-46e6-86e7-07ee832e8aba-host-slash\") pod \"iptables-alerter-kqz8h\" (UID: \"51008c04-b060-46e6-86e7-07ee832e8aba\") " pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:12.779025 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778512 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-socket-dir-parent\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778010 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c582a6c2-65b9-40e6-8382-a7c3133552cd-kubelet-config\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778521 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-cni-dir\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778714 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-run-systemd\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778744 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-daemon-config\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778765 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-run-ovn-kubernetes\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778772 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt76r\" (UniqueName: \"kubernetes.io/projected/9ad7a19b-83c8-404e-b28b-464de3f17e16-kube-api-access-lt76r\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778786 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-cni-dir\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778819 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-modprobe-d\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778890 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-slash\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778920 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59c160d1-15f8-45b6-86cf-a2ed1fce715c-host\") pod \"node-ca-dgrxk\" (UID: \"59c160d1-15f8-45b6-86cf-a2ed1fce715c\") " pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778961 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-run-netns\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779003 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-sysconfig\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779431 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-daemon-config\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779477 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26e859cc-9613-4ff0-885d-167beeed43aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.778475 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-cni-bin\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779497 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ad7a19b-83c8-404e-b28b-464de3f17e16-ovnkube-script-lib\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779503 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ad7a19b-83c8-404e-b28b-464de3f17e16-ovnkube-config\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.779776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779551 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51008c04-b060-46e6-86e7-07ee832e8aba-host-slash\") pod \"iptables-alerter-kqz8h\" (UID: \"51008c04-b060-46e6-86e7-07ee832e8aba\") " pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779569 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-run\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779588 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-sysctl-d\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779644 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779671 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtqll\" (UniqueName: \"kubernetes.io/projected/51008c04-b060-46e6-86e7-07ee832e8aba-kube-api-access-qtqll\") pod \"iptables-alerter-kqz8h\" (UID: \"51008c04-b060-46e6-86e7-07ee832e8aba\") " pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779698 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-os-release\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779709 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-system-cni-dir\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779722 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-kubelet\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779747 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-var-lib-openvswitch\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779753 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-cnibin\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779894 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779944 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-run\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779971 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a69b989f-c7db-4193-ae08-0686fcc49150-hosts-file\") pod \"node-resolver-nh2lf\" (UID: \"a69b989f-c7db-4193-ae08-0686fcc49150\") " pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779981 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-kubelet\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.779981 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26e859cc-9613-4ff0-885d-167beeed43aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780035 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a69b989f-c7db-4193-ae08-0686fcc49150-hosts-file\") pod \"node-resolver-nh2lf\" (UID: \"a69b989f-c7db-4193-ae08-0686fcc49150\") " pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-sys\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780160 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-var-lib-openvswitch\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.780597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-lib-modules\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780201 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26e859cc-9613-4ff0-885d-167beeed43aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780233 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-os-release\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780245 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz8s8\" (UniqueName: \"kubernetes.io/projected/5367035f-cfe2-45b8-a2f3-f462d3f043b3-kube-api-access-zz8s8\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780270 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ad7a19b-83c8-404e-b28b-464de3f17e16-ovn-node-metrics-cert\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780278 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-sys\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780296 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxb4p\" (UniqueName: \"kubernetes.io/projected/26e859cc-9613-4ff0-885d-167beeed43aa-kube-api-access-bxb4p\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780440 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5367035f-cfe2-45b8-a2f3-f462d3f043b3-cni-binary-copy\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780490 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-lib-modules\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780542 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780575 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51008c04-b060-46e6-86e7-07ee832e8aba-iptables-alerter-script\") pod \"iptables-alerter-kqz8h\" (UID: \"51008c04-b060-46e6-86e7-07ee832e8aba\") " pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780578 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-var-lib-kubelet\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780636 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-var-lib-kubelet\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780638 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-registration-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.780678 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780685 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ad7a19b-83c8-404e-b28b-464de3f17e16-env-overrides\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780690 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-registration-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.781478 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.780747 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret podName:c582a6c2-65b9-40e6-8382-a7c3133552cd nodeName:}" failed. No retries permitted until 2026-07-16 16:51:13.28072081 +0000 UTC m=+3.064875856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret") pod "global-pull-secret-syncer-gj948" (UID: "c582a6c2-65b9-40e6-8382-a7c3133552cd") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.780790 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x97f\" (UniqueName: \"kubernetes.io/projected/a69b989f-c7db-4193-ae08-0686fcc49150-kube-api-access-2x97f\") pod \"node-resolver-nh2lf\" (UID: \"a69b989f-c7db-4193-ae08-0686fcc49150\") " pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781149 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ad7a19b-83c8-404e-b28b-464de3f17e16-env-overrides\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781205 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-var-lib-kubelet\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781243 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26e859cc-9613-4ff0-885d-167beeed43aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781317 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-conf-dir\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781347 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-run-multus-certs\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781375 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-device-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781386 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-multus-conf-dir\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781401 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59c160d1-15f8-45b6-86cf-a2ed1fce715c-serviceca\") pod \"node-ca-dgrxk\" (UID: \"59c160d1-15f8-45b6-86cf-a2ed1fce715c\") " pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781471 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-device-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781507 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-var-lib-kubelet\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781532 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-node-log\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781535 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-run-multus-certs\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781569 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-run-ovn\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781602 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781653 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c582a6c2-65b9-40e6-8382-a7c3133552cd-dbus\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:12.782253 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781662 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-run-ovn\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781698 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-node-log\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781710 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-var-lib-cni-multus\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781730 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781749 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgxh9\" (UniqueName: \"kubernetes.io/projected/59c160d1-15f8-45b6-86cf-a2ed1fce715c-kube-api-access-fgxh9\") pod \"node-ca-dgrxk\" (UID: \"59c160d1-15f8-45b6-86cf-a2ed1fce715c\") " pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781765 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-var-lib-cni-multus\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781765 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c582a6c2-65b9-40e6-8382-a7c3133552cd-dbus\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781804 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-run-openvswitch\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781829 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-os-release\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781849 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59c160d1-15f8-45b6-86cf-a2ed1fce715c-serviceca\") pod \"node-ca-dgrxk\" (UID: \"59c160d1-15f8-45b6-86cf-a2ed1fce715c\") " pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781860 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-run-openvswitch\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781853 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b89bf64-dbd2-4b29-95e1-25a45fc84481-konnectivity-ca\") pod \"konnectivity-agent-kf8nk\" (UID: \"0b89bf64-dbd2-4b29-95e1-25a45fc84481\") " pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781898 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-run-k8s-cni-cncf-io\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781920 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-os-release\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781928 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-sys-fs\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781962 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-run-k8s-cni-cncf-io\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.781964 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-log-socket\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782012 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-log-socket\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782033 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-cni-netd\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69b989f-c7db-4193-ae08-0686fcc49150-tmp-dir\") pod \"node-resolver-nh2lf\" (UID: \"a69b989f-c7db-4193-ae08-0686fcc49150\") " pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782069 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-host-cni-netd\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782069 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-sys-fs\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782080 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-hostroot\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782108 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-sysctl-conf\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782154 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-host\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782177 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-cnibin\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782202 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-var-lib-cni-bin\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782211 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782226 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782251 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-hostroot\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782251 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-etc-openvswitch\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782288 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ad7a19b-83c8-404e-b28b-464de3f17e16-etc-openvswitch\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782315 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26e859cc-9613-4ff0-885d-167beeed43aa-cnibin\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782331 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5367035f-cfe2-45b8-a2f3-f462d3f043b3-host-var-lib-cni-bin\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.783888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782344 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69b989f-c7db-4193-ae08-0686fcc49150-tmp-dir\") pod \"node-resolver-nh2lf\" (UID: \"a69b989f-c7db-4193-ae08-0686fcc49150\") " pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:12.784777 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782376 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-sysctl-conf\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.784777 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782360 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a95ad29c-98cf-4c52-af59-05aa91ffe20e-host\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.784777 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782386 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e87197a5-760c-4729-90d5-3a021851f948-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.784777 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782447 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b89bf64-dbd2-4b29-95e1-25a45fc84481-konnectivity-ca\") pod \"konnectivity-agent-kf8nk\" (UID: \"0b89bf64-dbd2-4b29-95e1-25a45fc84481\") " pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:12.784777 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782750 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a95ad29c-98cf-4c52-af59-05aa91ffe20e-tmp\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.784777 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.782929 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a95ad29c-98cf-4c52-af59-05aa91ffe20e-etc-tuned\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.784777 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.784615 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ad7a19b-83c8-404e-b28b-464de3f17e16-ovn-node-metrics-cert\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.784777 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.784668 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b89bf64-dbd2-4b29-95e1-25a45fc84481-agent-certs\") pod \"konnectivity-agent-kf8nk\" (UID: \"0b89bf64-dbd2-4b29-95e1-25a45fc84481\") " pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:12.786302 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.786280 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:12.786407 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.786307 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:12.786407 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.786322 2556 projected.go:196] Error preparing data for projected volume kube-api-access-ktwzj for pod openshift-network-diagnostics/network-check-target-stcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:12.786518 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:12.786447 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj podName:5771265b-9fe3-43a2-b9ca-aa2c4a352f2c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:13.2864285 +0000 UTC m=+3.070583552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ktwzj" (UniqueName: "kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj") pod "network-check-target-stcqc" (UID: "5771265b-9fe3-43a2-b9ca-aa2c4a352f2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:12.789658 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.789547 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6ft\" (UniqueName: \"kubernetes.io/projected/51875cc1-104f-460c-b55f-17a7340ce099-kube-api-access-md6ft\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:12.790920 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.790748 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxsvk\" (UniqueName: \"kubernetes.io/projected/a95ad29c-98cf-4c52-af59-05aa91ffe20e-kube-api-access-dxsvk\") pod \"tuned-4kstz\" (UID: \"a95ad29c-98cf-4c52-af59-05aa91ffe20e\") " pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.790920 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.790808 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmd78\" (UniqueName: \"kubernetes.io/projected/e87197a5-760c-4729-90d5-3a021851f948-kube-api-access-bmd78\") pod \"aws-ebs-csi-driver-node-f7h9t\" (UID: \"e87197a5-760c-4729-90d5-3a021851f948\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.791897 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.791843 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz8s8\" (UniqueName: \"kubernetes.io/projected/5367035f-cfe2-45b8-a2f3-f462d3f043b3-kube-api-access-zz8s8\") pod \"multus-cjqtv\" (UID: \"5367035f-cfe2-45b8-a2f3-f462d3f043b3\") " pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.792682 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.792607 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt76r\" (UniqueName: \"kubernetes.io/projected/9ad7a19b-83c8-404e-b28b-464de3f17e16-kube-api-access-lt76r\") pod \"ovnkube-node-x84hq\" (UID: \"9ad7a19b-83c8-404e-b28b-464de3f17e16\") " pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.794539 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.794500 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgxh9\" (UniqueName: \"kubernetes.io/projected/59c160d1-15f8-45b6-86cf-a2ed1fce715c-kube-api-access-fgxh9\") pod \"node-ca-dgrxk\" (UID: \"59c160d1-15f8-45b6-86cf-a2ed1fce715c\") " pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.795012 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.794973 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x97f\" (UniqueName: \"kubernetes.io/projected/a69b989f-c7db-4193-ae08-0686fcc49150-kube-api-access-2x97f\") pod \"node-resolver-nh2lf\" (UID: \"a69b989f-c7db-4193-ae08-0686fcc49150\") " pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:12.795878 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.795833 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtqll\" (UniqueName: \"kubernetes.io/projected/51008c04-b060-46e6-86e7-07ee832e8aba-kube-api-access-qtqll\") pod \"iptables-alerter-kqz8h\" (UID: \"51008c04-b060-46e6-86e7-07ee832e8aba\") " pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:12.796645 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.796624 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxb4p\" (UniqueName: \"kubernetes.io/projected/26e859cc-9613-4ff0-885d-167beeed43aa-kube-api-access-bxb4p\") pod \"multus-additional-cni-plugins-cl2mq\" (UID: \"26e859cc-9613-4ff0-885d-167beeed43aa\") " pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.845983 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.845856 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" event={"ID":"556eca662b4e5f951dd2eca461c87322","Type":"ContainerStarted","Data":"1aece23dad506b7bb33383346a2b3072bec01158a3153141efec81ac75f6ea05"} Jul 16 16:51:12.847463 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.847389 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" event={"ID":"44e0d5f2b8dc87c8745e7edd3a441ecf","Type":"ContainerStarted","Data":"4d7141770de373b6a47275f3b97519630901bb555705830659fa694b578adeba"} Jul 16 16:51:12.941721 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.941685 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4kstz" Jul 16 16:51:12.950832 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.950804 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cjqtv" Jul 16 16:51:12.960687 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.960663 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" Jul 16 16:51:12.969738 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.969712 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" Jul 16 16:51:12.978408 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.978389 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dgrxk" Jul 16 16:51:12.986722 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.986696 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:12.994353 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:12.994334 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nh2lf" Jul 16 16:51:13.001973 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:13.001955 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:13.009566 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:13.009546 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kqz8h" Jul 16 16:51:13.285667 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:13.285580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:13.285667 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:13.285649 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:13.285871 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:13.285764 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:13.285871 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:13.285776 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:13.285871 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:13.285843 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs podName:51875cc1-104f-460c-b55f-17a7340ce099 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:14.285825231 +0000 UTC m=+4.069980285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs") pod "network-metrics-daemon-sw5rt" (UID: "51875cc1-104f-460c-b55f-17a7340ce099") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:13.285871 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:13.285862 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret podName:c582a6c2-65b9-40e6-8382-a7c3133552cd nodeName:}" failed. No retries permitted until 2026-07-16 16:51:14.285852229 +0000 UTC m=+4.070007271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret") pod "global-pull-secret-syncer-gj948" (UID: "c582a6c2-65b9-40e6-8382-a7c3133552cd") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:13.386912 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:13.386867 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwzj\" (UniqueName: \"kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj\") pod \"network-check-target-stcqc\" (UID: \"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c\") " pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:13.387100 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:13.387023 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:13.387100 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:13.387047 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:13.387100 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:13.387058 2556 projected.go:196] Error preparing data for projected volume kube-api-access-ktwzj for pod openshift-network-diagnostics/network-check-target-stcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:13.387261 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:13.387121 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj podName:5771265b-9fe3-43a2-b9ca-aa2c4a352f2c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:14.387102625 +0000 UTC m=+4.171257664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktwzj" (UniqueName: "kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj") pod "network-check-target-stcqc" (UID: "5771265b-9fe3-43a2-b9ca-aa2c4a352f2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:13.741045 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:13.740963 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 16:46:12 +0000 UTC" deadline="2028-03-07 18:44:09.689124222 +0000 UTC" Jul 16 16:51:13.741045 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:13.740994 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14401h52m55.948133119s" Jul 16 16:51:13.836425 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:13.836381 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:13.836588 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:13.836553 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:13.921599 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:13.921559 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c160d1_15f8_45b6_86cf_a2ed1fce715c.slice/crio-80ebdeddd357c0fbe184a67ae35ee208193fb9a7c369126bb85b32d8cecd1082 WatchSource:0}: Error finding container 80ebdeddd357c0fbe184a67ae35ee208193fb9a7c369126bb85b32d8cecd1082: Status 404 returned error can't find the container with id 80ebdeddd357c0fbe184a67ae35ee208193fb9a7c369126bb85b32d8cecd1082 Jul 16 16:51:13.926912 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:13.926363 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69b989f_c7db_4193_ae08_0686fcc49150.slice/crio-a586e394229aa8a70b87564a9ef91958cbd7196670f4c530f321d85cfccc1449 WatchSource:0}: Error finding container a586e394229aa8a70b87564a9ef91958cbd7196670f4c530f321d85cfccc1449: Status 404 returned error can't find the container with id a586e394229aa8a70b87564a9ef91958cbd7196670f4c530f321d85cfccc1449 Jul 16 16:51:13.928560 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:13.928539 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5367035f_cfe2_45b8_a2f3_f462d3f043b3.slice/crio-45fe3568b8a57fe547a9042618e8659c43ccf1f61a124d7cfa18dbd02e88c83b WatchSource:0}: Error finding container 45fe3568b8a57fe547a9042618e8659c43ccf1f61a124d7cfa18dbd02e88c83b: Status 404 returned error can't find the container with id 45fe3568b8a57fe547a9042618e8659c43ccf1f61a124d7cfa18dbd02e88c83b Jul 16 16:51:13.929482 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:13.929461 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87197a5_760c_4729_90d5_3a021851f948.slice/crio-134754e02d388f2b85a00f696ca0a73fcb3c7b63410fbefaf38686d94c263366 WatchSource:0}: Error finding container 134754e02d388f2b85a00f696ca0a73fcb3c7b63410fbefaf38686d94c263366: Status 404 returned error can't find the container with id 134754e02d388f2b85a00f696ca0a73fcb3c7b63410fbefaf38686d94c263366 Jul 16 16:51:13.930430 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:13.930378 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad7a19b_83c8_404e_b28b_464de3f17e16.slice/crio-f84cce483d2c9644e9e9a626fb5e6c91e3a0e09b80f0792c6c63b8672949bb0f WatchSource:0}: Error finding container f84cce483d2c9644e9e9a626fb5e6c91e3a0e09b80f0792c6c63b8672949bb0f: Status 404 returned error can't find the container with id f84cce483d2c9644e9e9a626fb5e6c91e3a0e09b80f0792c6c63b8672949bb0f Jul 16 16:51:13.932241 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:13.932219 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b89bf64_dbd2_4b29_95e1_25a45fc84481.slice/crio-b0443350b8baa361fad1577eea180ba19686eb7268c1d9b14ebb7fa906f3ac6e WatchSource:0}: Error finding container b0443350b8baa361fad1577eea180ba19686eb7268c1d9b14ebb7fa906f3ac6e: Status 404 returned error can't find the container with id b0443350b8baa361fad1577eea180ba19686eb7268c1d9b14ebb7fa906f3ac6e Jul 16 16:51:13.933639 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:13.933616 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e859cc_9613_4ff0_885d_167beeed43aa.slice/crio-dd565b90aad37b1fe37bdb1110033274b798046203cf3c90bde15c306e07e538 WatchSource:0}: Error finding container dd565b90aad37b1fe37bdb1110033274b798046203cf3c90bde15c306e07e538: Status 404 returned error can't find the container with id dd565b90aad37b1fe37bdb1110033274b798046203cf3c90bde15c306e07e538 Jul 16 16:51:13.934120 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:13.934034 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51008c04_b060_46e6_86e7_07ee832e8aba.slice/crio-b5c9a8d8115e2760f823d957c8ee43c9d8c869f112e0395db309d2e67a8341c3 WatchSource:0}: Error finding container b5c9a8d8115e2760f823d957c8ee43c9d8c869f112e0395db309d2e67a8341c3: Status 404 returned error can't find the container with id b5c9a8d8115e2760f823d957c8ee43c9d8c869f112e0395db309d2e67a8341c3 Jul 16 16:51:13.957146 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:51:13.957127 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95ad29c_98cf_4c52_af59_05aa91ffe20e.slice/crio-634d910a3b2a0a6ec36c8c14c87117c84629cea0b8cf98d5233fd6d5734ed06e WatchSource:0}: Error finding container 634d910a3b2a0a6ec36c8c14c87117c84629cea0b8cf98d5233fd6d5734ed06e: Status 404 returned error can't find the container with id 634d910a3b2a0a6ec36c8c14c87117c84629cea0b8cf98d5233fd6d5734ed06e Jul 16 16:51:14.293087 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.292783 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:14.293087 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.293056 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:14.293296 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:14.292966 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:14.293296 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:14.293189 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret podName:c582a6c2-65b9-40e6-8382-a7c3133552cd nodeName:}" failed. No retries permitted until 2026-07-16 16:51:16.293165418 +0000 UTC m=+6.077320470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret") pod "global-pull-secret-syncer-gj948" (UID: "c582a6c2-65b9-40e6-8382-a7c3133552cd") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:14.293296 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:14.293194 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:14.293296 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:14.293242 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs podName:51875cc1-104f-460c-b55f-17a7340ce099 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:16.293226593 +0000 UTC m=+6.077381649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs") pod "network-metrics-daemon-sw5rt" (UID: "51875cc1-104f-460c-b55f-17a7340ce099") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:14.395195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.394715 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwzj\" (UniqueName: \"kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj\") pod \"network-check-target-stcqc\" (UID: \"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c\") " pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:14.395195 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:14.394924 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:14.395195 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:14.394945 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:14.395195 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:14.394958 2556 projected.go:196] Error preparing data for projected volume kube-api-access-ktwzj for pod openshift-network-diagnostics/network-check-target-stcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:14.395195 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:14.395024 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj podName:5771265b-9fe3-43a2-b9ca-aa2c4a352f2c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:16.395006916 +0000 UTC m=+6.179161976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktwzj" (UniqueName: "kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj") pod "network-check-target-stcqc" (UID: "5771265b-9fe3-43a2-b9ca-aa2c4a352f2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:14.742019 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.741949 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 16:46:12 +0000 UTC" deadline="2028-04-29 17:55:38.297224604 +0000 UTC" Jul 16 16:51:14.742019 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.741987 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15673h4m23.555242846s" Jul 16 16:51:14.837396 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.837364 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:14.837567 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:14.837511 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:14.837992 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.837975 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:14.838100 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:14.838081 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:14.862247 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.862211 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dgrxk" event={"ID":"59c160d1-15f8-45b6-86cf-a2ed1fce715c","Type":"ContainerStarted","Data":"80ebdeddd357c0fbe184a67ae35ee208193fb9a7c369126bb85b32d8cecd1082"} Jul 16 16:51:14.873803 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.873771 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" event={"ID":"556eca662b4e5f951dd2eca461c87322","Type":"ContainerStarted","Data":"606c2788293e6ec7569281b6a7acd0266c6ff1f7079f753eef4b360daf0a165c"} Jul 16 16:51:14.895504 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.895469 2556 generic.go:356] "Generic (PLEG): container finished" podID="44e0d5f2b8dc87c8745e7edd3a441ecf" containerID="c7d1cc0114c76610f996c396031e0c0ab2bed8562502417cd99617bde6119029" exitCode=0 Jul 16 16:51:14.895686 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.895556 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" event={"ID":"44e0d5f2b8dc87c8745e7edd3a441ecf","Type":"ContainerDied","Data":"c7d1cc0114c76610f996c396031e0c0ab2bed8562502417cd99617bde6119029"} Jul 16 16:51:14.901553 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.901514 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" event={"ID":"e87197a5-760c-4729-90d5-3a021851f948","Type":"ContainerStarted","Data":"134754e02d388f2b85a00f696ca0a73fcb3c7b63410fbefaf38686d94c263366"} Jul 16 16:51:14.904041 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.904017 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" event={"ID":"26e859cc-9613-4ff0-885d-167beeed43aa","Type":"ContainerStarted","Data":"dd565b90aad37b1fe37bdb1110033274b798046203cf3c90bde15c306e07e538"} Jul 16 16:51:14.911429 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.911348 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" podStartSLOduration=3.9113284249999998 podStartE2EDuration="3.911328425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:14.890264917 +0000 UTC m=+4.674419976" watchObservedRunningTime="2026-07-16 16:51:14.911328425 +0000 UTC m=+4.695483489" Jul 16 16:51:14.912998 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.912972 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kf8nk" event={"ID":"0b89bf64-dbd2-4b29-95e1-25a45fc84481","Type":"ContainerStarted","Data":"b0443350b8baa361fad1577eea180ba19686eb7268c1d9b14ebb7fa906f3ac6e"} Jul 16 16:51:14.938096 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.937980 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" event={"ID":"9ad7a19b-83c8-404e-b28b-464de3f17e16","Type":"ContainerStarted","Data":"f84cce483d2c9644e9e9a626fb5e6c91e3a0e09b80f0792c6c63b8672949bb0f"} Jul 16 16:51:14.945373 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.945296 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cjqtv" event={"ID":"5367035f-cfe2-45b8-a2f3-f462d3f043b3","Type":"ContainerStarted","Data":"45fe3568b8a57fe547a9042618e8659c43ccf1f61a124d7cfa18dbd02e88c83b"} Jul 16 16:51:14.948578 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.948546 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nh2lf" event={"ID":"a69b989f-c7db-4193-ae08-0686fcc49150","Type":"ContainerStarted","Data":"a586e394229aa8a70b87564a9ef91958cbd7196670f4c530f321d85cfccc1449"} Jul 16 16:51:14.960228 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.960192 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kqz8h" event={"ID":"51008c04-b060-46e6-86e7-07ee832e8aba","Type":"ContainerStarted","Data":"b5c9a8d8115e2760f823d957c8ee43c9d8c869f112e0395db309d2e67a8341c3"} Jul 16 16:51:14.981985 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:14.981945 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4kstz" event={"ID":"a95ad29c-98cf-4c52-af59-05aa91ffe20e","Type":"ContainerStarted","Data":"634d910a3b2a0a6ec36c8c14c87117c84629cea0b8cf98d5233fd6d5734ed06e"} Jul 16 16:51:15.837032 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:15.836954 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:15.837580 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:15.837091 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:16.025400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:16.025365 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" event={"ID":"44e0d5f2b8dc87c8745e7edd3a441ecf","Type":"ContainerStarted","Data":"01c0934d206df759490220f3f444f33ef44a9365cbde3cbe583daad048bcffe4"} Jul 16 16:51:16.041970 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:16.041915 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" podStartSLOduration=5.041894823 podStartE2EDuration="5.041894823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:16.04170399 +0000 UTC m=+5.825859067" watchObservedRunningTime="2026-07-16 16:51:16.041894823 +0000 UTC m=+5.826049886" Jul 16 16:51:16.314073 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:16.313720 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:16.314073 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:16.313795 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:16.314073 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:16.313880 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:16.314073 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:16.313928 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:16.314073 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:16.313953 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret podName:c582a6c2-65b9-40e6-8382-a7c3133552cd nodeName:}" failed. No retries permitted until 2026-07-16 16:51:20.313932693 +0000 UTC m=+10.098087743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret") pod "global-pull-secret-syncer-gj948" (UID: "c582a6c2-65b9-40e6-8382-a7c3133552cd") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:16.314073 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:16.313980 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs podName:51875cc1-104f-460c-b55f-17a7340ce099 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:20.313964223 +0000 UTC m=+10.098119277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs") pod "network-metrics-daemon-sw5rt" (UID: "51875cc1-104f-460c-b55f-17a7340ce099") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:16.414790 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:16.414752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwzj\" (UniqueName: \"kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj\") pod \"network-check-target-stcqc\" (UID: \"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c\") " pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:16.414968 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:16.414940 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:16.414968 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:16.414960 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:16.415067 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:16.414972 2556 projected.go:196] Error preparing data for projected volume kube-api-access-ktwzj for pod openshift-network-diagnostics/network-check-target-stcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:16.415067 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:16.415025 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj podName:5771265b-9fe3-43a2-b9ca-aa2c4a352f2c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:20.415008509 +0000 UTC m=+10.199163568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktwzj" (UniqueName: "kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj") pod "network-check-target-stcqc" (UID: "5771265b-9fe3-43a2-b9ca-aa2c4a352f2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:16.839351 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:16.839323 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:16.839798 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:16.839472 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:16.839923 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:16.839905 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:16.840047 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:16.840008 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:17.836977 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:17.836943 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:17.837225 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:17.837068 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:18.840175 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:18.840039 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:18.840175 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:18.840170 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:18.840809 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:18.840187 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:18.840809 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:18.840301 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:19.837381 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:19.837289 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:19.837573 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:19.837451 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:20.352131 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:20.351498 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:20.352131 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:20.351591 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:20.352131 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:20.351717 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:20.352131 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:20.351771 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret podName:c582a6c2-65b9-40e6-8382-a7c3133552cd nodeName:}" failed. No retries permitted until 2026-07-16 16:51:28.351757605 +0000 UTC m=+18.135912649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret") pod "global-pull-secret-syncer-gj948" (UID: "c582a6c2-65b9-40e6-8382-a7c3133552cd") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:20.352131 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:20.352072 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:20.352131 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:20.352106 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs podName:51875cc1-104f-460c-b55f-17a7340ce099 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:28.352095359 +0000 UTC m=+18.136250399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs") pod "network-metrics-daemon-sw5rt" (UID: "51875cc1-104f-460c-b55f-17a7340ce099") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:20.452806 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:20.452765 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwzj\" (UniqueName: \"kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj\") pod \"network-check-target-stcqc\" (UID: \"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c\") " pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:20.452999 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:20.452959 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:20.452999 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:20.452984 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:20.452999 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:20.452998 2556 projected.go:196] Error preparing data for projected volume kube-api-access-ktwzj for pod openshift-network-diagnostics/network-check-target-stcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:20.453152 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:20.453108 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj podName:5771265b-9fe3-43a2-b9ca-aa2c4a352f2c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:28.45308788 +0000 UTC m=+18.237242936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktwzj" (UniqueName: "kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj") pod "network-check-target-stcqc" (UID: "5771265b-9fe3-43a2-b9ca-aa2c4a352f2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:20.837890 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:20.837676 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:20.837890 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:20.837758 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:20.837890 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:20.837765 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:20.838152 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:20.837917 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:21.836573 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:21.836510 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:21.837049 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:21.836652 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:22.837033 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:22.836995 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:22.837525 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:22.837128 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:22.837525 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:22.837340 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:22.837525 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:22.837460 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:23.836537 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:23.836503 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:23.836732 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:23.836635 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:24.836621 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:24.836522 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:24.836621 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:24.836531 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:24.837064 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:24.836684 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:24.837064 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:24.836791 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:25.837238 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:25.837209 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:25.837653 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:25.837310 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:26.836892 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:26.836860 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:26.837054 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:26.836860 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:26.837054 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:26.836998 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:26.837170 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:26.837054 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:27.837241 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:27.837209 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:27.837724 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:27.837329 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:28.414071 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:28.414030 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:28.414071 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:28.414091 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:28.414292 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:28.414201 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:28.414292 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:28.414208 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:28.414292 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:28.414268 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs podName:51875cc1-104f-460c-b55f-17a7340ce099 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:44.414249746 +0000 UTC m=+34.198404808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs") pod "network-metrics-daemon-sw5rt" (UID: "51875cc1-104f-460c-b55f-17a7340ce099") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:28.414292 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:28.414282 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret podName:c582a6c2-65b9-40e6-8382-a7c3133552cd nodeName:}" failed. No retries permitted until 2026-07-16 16:51:44.41427603 +0000 UTC m=+34.198431072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret") pod "global-pull-secret-syncer-gj948" (UID: "c582a6c2-65b9-40e6-8382-a7c3133552cd") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:28.515170 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:28.515078 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwzj\" (UniqueName: \"kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj\") pod \"network-check-target-stcqc\" (UID: \"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c\") " pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:28.515377 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:28.515267 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:28.515377 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:28.515289 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:28.515377 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:28.515300 2556 projected.go:196] Error preparing data for projected volume kube-api-access-ktwzj for pod openshift-network-diagnostics/network-check-target-stcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:28.515377 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:28.515366 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj podName:5771265b-9fe3-43a2-b9ca-aa2c4a352f2c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:44.515350172 +0000 UTC m=+34.299505229 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktwzj" (UniqueName: "kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj") pod "network-check-target-stcqc" (UID: "5771265b-9fe3-43a2-b9ca-aa2c4a352f2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:28.837407 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:28.837334 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:28.837407 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:28.837380 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:28.837897 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:28.837500 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:28.837897 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:28.837610 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:29.836524 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:29.836496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:29.836694 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:29.836596 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:30.837326 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:30.837296 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:30.837774 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:30.837398 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:30.837774 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:30.837503 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:30.837774 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:30.837625 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:31.837377 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:31.837343 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:31.837792 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:31.837489 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:32.836632 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:32.836466 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:32.836824 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:32.836479 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:32.836824 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:32.836731 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:32.836913 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:32.836826 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:33.059186 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.059153 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4kstz" event={"ID":"a95ad29c-98cf-4c52-af59-05aa91ffe20e","Type":"ContainerStarted","Data":"98aa5b127c92c2735fa432a3697971b3698066b78d14046ccf78d24e28c3359e"} Jul 16 16:51:33.060751 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.060728 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dgrxk" event={"ID":"59c160d1-15f8-45b6-86cf-a2ed1fce715c","Type":"ContainerStarted","Data":"b0caf426c481866b5a10962546f21fad7a7f44cfdfc9bbc5bfde6849363992cc"} Jul 16 16:51:33.062240 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.062219 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" event={"ID":"e87197a5-760c-4729-90d5-3a021851f948","Type":"ContainerStarted","Data":"872f6785fb198977666280e41f09c01b800e8fd01bdb686d1855216e51db7364"} Jul 16 16:51:33.063683 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.063663 2556 generic.go:356] "Generic (PLEG): container finished" podID="26e859cc-9613-4ff0-885d-167beeed43aa" containerID="df4fca040d6d8511e0ddc1214ba07cb3ff691b8633eeb14f9a766653d3f319d8" exitCode=0 Jul 16 16:51:33.063764 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.063744 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" event={"ID":"26e859cc-9613-4ff0-885d-167beeed43aa","Type":"ContainerDied","Data":"df4fca040d6d8511e0ddc1214ba07cb3ff691b8633eeb14f9a766653d3f319d8"} Jul 16 16:51:33.077174 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.077148 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kf8nk" event={"ID":"0b89bf64-dbd2-4b29-95e1-25a45fc84481","Type":"ContainerStarted","Data":"753fb1da9982d6ca3bb9abcdf49b359649a56fe90bddfae9898680d17c639c92"} Jul 16 16:51:33.079313 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.079297 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 16:51:33.079685 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.079666 2556 generic.go:356] "Generic (PLEG): container finished" podID="9ad7a19b-83c8-404e-b28b-464de3f17e16" containerID="2d897d560eefdd25c7d48c462a30fd207d3456cd08fe3b093fa495cfd630bb9a" exitCode=1 Jul 16 16:51:33.079759 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.079724 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" event={"ID":"9ad7a19b-83c8-404e-b28b-464de3f17e16","Type":"ContainerStarted","Data":"785e54a4b139cbeb59a664e679590218f16b67c584279e59488e66b60253eb04"} Jul 16 16:51:33.079810 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.079781 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" event={"ID":"9ad7a19b-83c8-404e-b28b-464de3f17e16","Type":"ContainerDied","Data":"2d897d560eefdd25c7d48c462a30fd207d3456cd08fe3b093fa495cfd630bb9a"} Jul 16 16:51:33.079852 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.079818 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" event={"ID":"9ad7a19b-83c8-404e-b28b-464de3f17e16","Type":"ContainerStarted","Data":"73e876db76170cee1c6ea6e83304adf2eb33cffee1a24c6b5efeedda908e4c93"} Jul 16 16:51:33.081093 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.081072 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cjqtv" event={"ID":"5367035f-cfe2-45b8-a2f3-f462d3f043b3","Type":"ContainerStarted","Data":"e6d085864411d55e502e92171dd90fcf1fb5b6e3ae81ca0c47eab0b5d1b3a177"} Jul 16 16:51:33.082861 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.082842 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nh2lf" event={"ID":"a69b989f-c7db-4193-ae08-0686fcc49150","Type":"ContainerStarted","Data":"7c413a901c7fc4318aa986bc67d3cd043d8ca45974eda086a91e83c620ff34ab"} Jul 16 16:51:33.101822 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.101782 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4kstz" podStartSLOduration=4.56004172 podStartE2EDuration="23.101770112s" totalImagesPullingTime="18.541728392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:33.084835828 +0000 UTC m=+22.868990888" watchObservedRunningTime="2026-07-16 16:51:33.101770112 +0000 UTC m=+22.885925172" Jul 16 16:51:33.101932 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.101913 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dgrxk" podStartSLOduration=4.535908094 podStartE2EDuration="23.101908822s" totalImagesPullingTime="18.566000728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:33.101339622 +0000 UTC m=+22.885494683" watchObservedRunningTime="2026-07-16 16:51:33.101908822 +0000 UTC m=+22.886063910" Jul 16 16:51:33.147088 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.147006 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kf8nk" podStartSLOduration=8.845459985 podStartE2EDuration="23.146984108s" totalImagesPullingTime="14.301524123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:33.146278398 +0000 UTC m=+22.930433460" watchObservedRunningTime="2026-07-16 16:51:33.146984108 +0000 UTC m=+22.931139169" Jul 16 16:51:33.166164 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.166123 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cjqtv" podStartSLOduration=4.543316046 podStartE2EDuration="23.166110382s" totalImagesPullingTime="18.622794336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:33.165628968 +0000 UTC m=+22.949784029" watchObservedRunningTime="2026-07-16 16:51:33.166110382 +0000 UTC m=+22.950265443" Jul 16 16:51:33.183620 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.183581 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nh2lf" podStartSLOduration=4.621314584 podStartE2EDuration="23.18356748s" totalImagesPullingTime="18.562252896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:33.183294319 +0000 UTC m=+22.967449379" watchObservedRunningTime="2026-07-16 16:51:33.18356748 +0000 UTC m=+22.967722539" Jul 16 16:51:33.836843 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:33.836659 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:33.837074 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:33.836923 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:34.088991 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:34.088924 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 16:51:34.089588 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:34.089443 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" event={"ID":"9ad7a19b-83c8-404e-b28b-464de3f17e16","Type":"ContainerStarted","Data":"fea1e5166398c10bb71e025b92cd1abdbb7e08f135d34fe435c665efa13044be"} Jul 16 16:51:34.089588 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:34.089530 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" event={"ID":"9ad7a19b-83c8-404e-b28b-464de3f17e16","Type":"ContainerStarted","Data":"907daedf2ce3fb7279a5432561003cc6d18ed58be9ad2abfe863208b870e0a8c"} Jul 16 16:51:34.089588 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:34.089581 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" event={"ID":"9ad7a19b-83c8-404e-b28b-464de3f17e16","Type":"ContainerStarted","Data":"8333b48ef4a7829ebcde2b0619fcad2a9020ae39b7396cecc4e3b1436ebdf2c1"} Jul 16 16:51:34.090984 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:34.090958 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kqz8h" event={"ID":"51008c04-b060-46e6-86e7-07ee832e8aba","Type":"ContainerStarted","Data":"3b9576e768cfa1d80a76cf53263ec98d4d633dfcb1551c14035c7408450876f7"} Jul 16 16:51:34.106853 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:34.106790 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kqz8h" podStartSLOduration=4.56173014 podStartE2EDuration="23.10677559s" totalImagesPullingTime="18.54504545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:34.106576558 +0000 UTC m=+23.890731620" watchObservedRunningTime="2026-07-16 16:51:34.10677559 +0000 UTC m=+23.890930651" Jul 16 16:51:34.375747 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:34.375663 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:34.376471 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:34.376439 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:34.836570 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:34.836492 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:34.836747 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:34.836637 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:34.836747 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:34.836653 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:34.836747 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:34.836729 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:35.093552 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:35.093473 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:35.093979 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:35.093968 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kf8nk" Jul 16 16:51:35.836884 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:35.836809 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:35.837137 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:35.836912 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:36.099300 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:36.099066 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 16:51:36.099723 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:36.099626 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" event={"ID":"9ad7a19b-83c8-404e-b28b-464de3f17e16","Type":"ContainerStarted","Data":"cc0c82491bf7b2354c1c5efa2f5825f787dbf7b838cb8e187f4c3ffbb775f4f2"} Jul 16 16:51:36.836926 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:36.836887 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:36.837128 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:36.837023 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:36.837128 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:36.837074 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:36.837236 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:36.837209 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:37.837046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:37.836752 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:37.838233 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:37.837092 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:37.998445 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:37.998409 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 16:51:38.107389 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.107301 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 16:51:38.107734 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.107708 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" event={"ID":"9ad7a19b-83c8-404e-b28b-464de3f17e16","Type":"ContainerStarted","Data":"a87bc40fafadf10795988f2566e51d04265628b4441849d99694a7648ebca79a"} Jul 16 16:51:38.107977 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.107943 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:38.108052 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.108007 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:38.108112 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.108054 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:38.108165 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.108130 2556 scope.go:117] "RemoveContainer" containerID="2d897d560eefdd25c7d48c462a30fd207d3456cd08fe3b093fa495cfd630bb9a" Jul 16 16:51:38.109698 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.109669 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" event={"ID":"e87197a5-760c-4729-90d5-3a021851f948","Type":"ContainerStarted","Data":"7157fe10c2a04834dd8c2e2b50961d29377a6f6b7b6c41e70588ff8b39dd5657"} Jul 16 16:51:38.111695 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.111672 2556 generic.go:356] "Generic (PLEG): container finished" podID="26e859cc-9613-4ff0-885d-167beeed43aa" containerID="827e199b4e8ddc3689560a1865e19add69a03e1d83e4cf06431568356bf53d33" exitCode=0 Jul 16 16:51:38.111772 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.111702 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" event={"ID":"26e859cc-9613-4ff0-885d-167beeed43aa","Type":"ContainerDied","Data":"827e199b4e8ddc3689560a1865e19add69a03e1d83e4cf06431568356bf53d33"} Jul 16 16:51:38.124353 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.124326 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:38.125406 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.125384 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:51:38.742621 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.742486 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T16:51:37.998440515Z","UUID":"4ab9f4ab-a310-440e-b6d3-cd81c8a26df6","Handler":null,"Name":"","Endpoint":""} Jul 16 16:51:38.744214 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.744187 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 16:51:38.744345 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.744231 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 16:51:38.836481 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.836452 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:38.836481 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:38.836472 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:38.836696 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:38.836567 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:38.836696 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:38.836684 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:39.117993 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:39.117930 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 16:51:39.118381 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:39.118279 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" event={"ID":"9ad7a19b-83c8-404e-b28b-464de3f17e16","Type":"ContainerStarted","Data":"78a1d98965a936576a02587ac133998aedd6f3f987a12fe0fe1c4faf90005e4f"} Jul 16 16:51:39.148617 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:39.148575 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" podStartSLOduration=10.331305586 podStartE2EDuration="29.148561061s" totalImagesPullingTime="18.817255475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:39.147263007 +0000 UTC m=+28.931418093" watchObservedRunningTime="2026-07-16 16:51:39.148561061 +0000 UTC m=+28.932716122" Jul 16 16:51:39.837232 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:39.837112 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:39.837379 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:39.837295 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:40.001235 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:40.001205 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sw5rt"] Jul 16 16:51:40.001372 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:40.001310 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:40.001434 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:40.001403 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:40.004228 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:40.004206 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gj948"] Jul 16 16:51:40.004364 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:40.004321 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:40.004457 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:40.004435 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:40.004980 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:40.004954 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-stcqc"] Jul 16 16:51:40.122684 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:40.122611 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" event={"ID":"e87197a5-760c-4729-90d5-3a021851f948","Type":"ContainerStarted","Data":"4960b041f8bd1320a760f92f1a5a721b471a63db1c6982ee2c0292fb2a21f9b6"} Jul 16 16:51:40.124719 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:40.124689 2556 generic.go:356] "Generic (PLEG): container finished" podID="26e859cc-9613-4ff0-885d-167beeed43aa" containerID="41ff1bf88f9f8e4d76b0947e598914753d28c68ce96b57cb280472d5c1c4a04e" exitCode=0 Jul 16 16:51:40.124831 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:40.124771 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" event={"ID":"26e859cc-9613-4ff0-885d-167beeed43aa","Type":"ContainerDied","Data":"41ff1bf88f9f8e4d76b0947e598914753d28c68ce96b57cb280472d5c1c4a04e"} Jul 16 16:51:40.124831 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:40.124820 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:40.125206 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:40.125178 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:40.142762 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:40.142724 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7h9t" podStartSLOduration=5.005582203 podStartE2EDuration="30.142709769s" totalImagesPullingTime="25.137127566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:40.142338244 +0000 UTC m=+29.926493304" watchObservedRunningTime="2026-07-16 16:51:40.142709769 +0000 UTC m=+29.926864829" Jul 16 16:51:41.836937 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:41.836907 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:41.837578 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:41.836947 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:41.837578 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:41.836917 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:41.837578 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:41.837047 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:41.837578 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:41.837122 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:41.837578 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:41.837196 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:42.130170 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:42.130092 2556 generic.go:356] "Generic (PLEG): container finished" podID="26e859cc-9613-4ff0-885d-167beeed43aa" containerID="74d8a2944d2b878b77ccf5be335408d18f9391d0efb9b5362d18fcc8b2020dbd" exitCode=0 Jul 16 16:51:42.130170 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:42.130141 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" event={"ID":"26e859cc-9613-4ff0-885d-167beeed43aa","Type":"ContainerDied","Data":"74d8a2944d2b878b77ccf5be335408d18f9391d0efb9b5362d18fcc8b2020dbd"} Jul 16 16:51:43.836951 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:43.836717 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:43.837430 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:43.836728 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:43.837430 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:43.837045 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:43.837430 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:43.837150 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:43.837430 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:43.836773 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:43.837430 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:43.837251 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:44.379446 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:44.379405 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nh2lf_a69b989f-c7db-4193-ae08-0686fcc49150/dns-node-resolver/0.log" Jul 16 16:51:44.434401 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:44.434357 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:44.434592 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:44.434457 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:44.434592 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:44.434531 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:44.434726 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:44.434591 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:44.434726 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:44.434614 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs podName:51875cc1-104f-460c-b55f-17a7340ce099 nodeName:}" failed. No retries permitted until 2026-07-16 16:52:16.434598278 +0000 UTC m=+66.218753321 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs") pod "network-metrics-daemon-sw5rt" (UID: "51875cc1-104f-460c-b55f-17a7340ce099") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:44.434726 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:44.434668 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret podName:c582a6c2-65b9-40e6-8382-a7c3133552cd nodeName:}" failed. No retries permitted until 2026-07-16 16:52:16.434648875 +0000 UTC m=+66.218803918 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret") pod "global-pull-secret-syncer-gj948" (UID: "c582a6c2-65b9-40e6-8382-a7c3133552cd") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:44.534841 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:44.534801 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwzj\" (UniqueName: \"kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj\") pod \"network-check-target-stcqc\" (UID: \"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c\") " pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:44.535062 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:44.534981 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:44.535062 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:44.535004 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:44.535062 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:44.535014 2556 projected.go:196] Error preparing data for projected volume kube-api-access-ktwzj for pod openshift-network-diagnostics/network-check-target-stcqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:44.535202 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:44.535076 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj podName:5771265b-9fe3-43a2-b9ca-aa2c4a352f2c nodeName:}" failed. No retries permitted until 2026-07-16 16:52:16.535058206 +0000 UTC m=+66.319213250 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktwzj" (UniqueName: "kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj") pod "network-check-target-stcqc" (UID: "5771265b-9fe3-43a2-b9ca-aa2c4a352f2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:44.945015 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:44.944989 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dgrxk_59c160d1-15f8-45b6-86cf-a2ed1fce715c/node-ca/0.log" Jul 16 16:51:45.836810 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:45.836782 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:45.836989 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:45.836782 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:45.836989 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:45.836882 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:45.836989 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:45.836951 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:45.836989 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:45.836782 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:45.837184 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:45.837046 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:47.144808 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:47.144778 2556 generic.go:356] "Generic (PLEG): container finished" podID="26e859cc-9613-4ff0-885d-167beeed43aa" containerID="8ce5c6e56a1d460eabefe3d5239f7a0d3495777d3cc6e3067b5d5d3ffb1504af" exitCode=0 Jul 16 16:51:47.145162 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:47.144827 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" event={"ID":"26e859cc-9613-4ff0-885d-167beeed43aa","Type":"ContainerDied","Data":"8ce5c6e56a1d460eabefe3d5239f7a0d3495777d3cc6e3067b5d5d3ffb1504af"} Jul 16 16:51:47.836516 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:47.836466 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:47.836516 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:47.836506 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:47.836738 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:47.836466 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:47.836738 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:47.836619 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:47.836738 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:47.836725 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:47.836835 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:47.836793 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:48.149195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:48.149107 2556 generic.go:356] "Generic (PLEG): container finished" podID="26e859cc-9613-4ff0-885d-167beeed43aa" containerID="183e36a217af3d42db165cc4b39a5527028407140585f44543ae2ccd757154f6" exitCode=0 Jul 16 16:51:48.149195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:48.149174 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" event={"ID":"26e859cc-9613-4ff0-885d-167beeed43aa","Type":"ContainerDied","Data":"183e36a217af3d42db165cc4b39a5527028407140585f44543ae2ccd757154f6"} Jul 16 16:51:49.154372 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:49.154337 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" event={"ID":"26e859cc-9613-4ff0-885d-167beeed43aa","Type":"ContainerStarted","Data":"33cd660235f396fdea205cf68e0d8bf7a2773fa49cf584fefe10c5d5ad7908d3"} Jul 16 16:51:49.182369 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:49.182318 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cl2mq" podStartSLOduration=9.819450066 podStartE2EDuration="39.182300678s" totalImagesPullingTime="29.362850612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:49.182109702 +0000 UTC m=+38.966264764" watchObservedRunningTime="2026-07-16 16:51:49.182300678 +0000 UTC m=+38.966455739" Jul 16 16:51:49.837060 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:49.837027 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:49.837212 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:49.837035 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:49.837212 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:49.837125 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:49.837212 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:49.837150 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:49.837331 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:49.837227 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:49.837331 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:49.837297 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:51.837174 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:51.837142 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:51.837569 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:51.837234 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:51.837569 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:51.837240 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:51.837569 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:51.837258 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:51.837569 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:51.837316 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:51.837569 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:51.837381 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:53.837397 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:53.837355 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:53.837783 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:53.837407 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:53.837783 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:53.837377 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:53.837783 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:53.837488 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:53.837783 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:53.837577 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:53.837783 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:53.837646 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:55.836798 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:55.836768 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:55.836798 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:55.836796 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:55.837345 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:55.836853 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:55.837345 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:55.836889 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:55.837345 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:55.836973 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:55.837345 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:55.837062 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:57.836797 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:57.836768 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:57.837161 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:57.836865 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:57.837161 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:57.836883 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:57.837161 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:57.836933 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:57.837161 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:57.836977 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:57.837161 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:57.837051 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:51:59.836326 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:59.836293 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:51:59.836700 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:59.836392 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:51:59.836700 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:59.836404 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:51:59.836700 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:51:59.836434 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:51:59.836700 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:59.836498 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:51:59.836700 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:51:59.836563 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:52:01.836621 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:01.836587 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:52:01.836621 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:01.836611 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:52:01.837096 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:01.836675 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:52:01.837096 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:01.836679 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:52:01.837096 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:01.836793 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:52:01.837096 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:01.836878 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:52:03.837352 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:03.837319 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:52:03.837824 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:03.837319 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:52:03.837824 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:03.837463 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:52:03.837824 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:03.837319 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:52:03.837824 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:03.837565 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:52:03.837824 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:03.837502 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:52:05.836368 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:05.836341 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:52:05.836793 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:05.836375 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:52:05.836793 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:05.836444 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:52:05.836793 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:05.836481 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:52:05.836793 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:05.836552 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:52:05.836793 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:05.836613 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:52:07.836653 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:07.836619 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:52:07.837114 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:07.836646 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:52:07.837114 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:07.836922 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gj948" podUID="c582a6c2-65b9-40e6-8382-a7c3133552cd" Jul 16 16:52:07.837114 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:07.836664 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:52:07.837114 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:07.836964 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sw5rt" podUID="51875cc1-104f-460c-b55f-17a7340ce099" Jul 16 16:52:07.837114 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:07.837028 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stcqc" podUID="5771265b-9fe3-43a2-b9ca-aa2c4a352f2c" Jul 16 16:52:08.459162 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.459134 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeReady" Jul 16 16:52:08.459328 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.459249 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 16:52:08.517737 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.517705 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c77n8"] Jul 16 16:52:08.539803 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.539771 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4q2xx"] Jul 16 16:52:08.539946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.539898 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.542337 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.542316 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 16:52:08.542450 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.542360 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtf2c\"" Jul 16 16:52:08.542450 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.542365 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 16:52:08.561575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.561553 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c77n8"] Jul 16 16:52:08.561665 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.561585 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4q2xx"] Jul 16 16:52:08.561709 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.561665 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.564069 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.564050 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 16:52:08.564166 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.564069 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rbpsf\"" Jul 16 16:52:08.564166 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.564088 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 16:52:08.564404 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.564391 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 16:52:08.564627 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.564612 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 16:52:08.609012 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.608992 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/173c3533-efe5-4d88-998b-33d87f347c0c-data-volume\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.609146 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.609017 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/173c3533-efe5-4d88-998b-33d87f347c0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.609146 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.609035 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89g4l\" (UniqueName: \"kubernetes.io/projected/173c3533-efe5-4d88-998b-33d87f347c0c-kube-api-access-89g4l\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.609146 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.609053 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5124801-48ed-45ec-8405-0c1b43c560f0-config-volume\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.609146 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.609070 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p26z\" (UniqueName: \"kubernetes.io/projected/a5124801-48ed-45ec-8405-0c1b43c560f0-kube-api-access-8p26z\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.609338 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.609168 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/173c3533-efe5-4d88-998b-33d87f347c0c-crio-socket\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.609338 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.609203 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5124801-48ed-45ec-8405-0c1b43c560f0-metrics-tls\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.609338 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.609259 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/173c3533-efe5-4d88-998b-33d87f347c0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.609467 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.609330 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5124801-48ed-45ec-8405-0c1b43c560f0-tmp-dir\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.618984 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.618958 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mjrrq"] Jul 16 16:52:08.640529 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.640502 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mjrrq"] Jul 16 16:52:08.640637 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.640603 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mjrrq" Jul 16 16:52:08.643208 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.643189 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 16:52:08.643820 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.643804 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lnc6v\"" Jul 16 16:52:08.664545 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.664524 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 16:52:08.664640 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.664555 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 16:52:08.709968 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.709900 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/173c3533-efe5-4d88-998b-33d87f347c0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.709968 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.709940 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8v5k\" (UniqueName: \"kubernetes.io/projected/65af6ca4-eedb-453d-b6d3-3c293d46fd3c-kube-api-access-z8v5k\") pod \"ingress-canary-mjrrq\" (UID: \"65af6ca4-eedb-453d-b6d3-3c293d46fd3c\") " pod="openshift-ingress-canary/ingress-canary-mjrrq" Jul 16 16:52:08.710104 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5124801-48ed-45ec-8405-0c1b43c560f0-tmp-dir\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.710104 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710095 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/173c3533-efe5-4d88-998b-33d87f347c0c-data-volume\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.710199 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710110 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/173c3533-efe5-4d88-998b-33d87f347c0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.710199 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710147 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89g4l\" (UniqueName: \"kubernetes.io/projected/173c3533-efe5-4d88-998b-33d87f347c0c-kube-api-access-89g4l\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.710199 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710185 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5124801-48ed-45ec-8405-0c1b43c560f0-config-volume\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.710337 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710213 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65af6ca4-eedb-453d-b6d3-3c293d46fd3c-cert\") pod \"ingress-canary-mjrrq\" (UID: \"65af6ca4-eedb-453d-b6d3-3c293d46fd3c\") " pod="openshift-ingress-canary/ingress-canary-mjrrq" Jul 16 16:52:08.710337 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710331 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/173c3533-efe5-4d88-998b-33d87f347c0c-data-volume\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.710464 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710441 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p26z\" (UniqueName: \"kubernetes.io/projected/a5124801-48ed-45ec-8405-0c1b43c560f0-kube-api-access-8p26z\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.710520 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710464 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5124801-48ed-45ec-8405-0c1b43c560f0-tmp-dir\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.710520 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710474 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/173c3533-efe5-4d88-998b-33d87f347c0c-crio-socket\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.710520 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710493 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5124801-48ed-45ec-8405-0c1b43c560f0-metrics-tls\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.710658 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710535 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/173c3533-efe5-4d88-998b-33d87f347c0c-crio-socket\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.710658 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710491 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/173c3533-efe5-4d88-998b-33d87f347c0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.710813 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.710783 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5124801-48ed-45ec-8405-0c1b43c560f0-config-volume\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.713971 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.713944 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5124801-48ed-45ec-8405-0c1b43c560f0-metrics-tls\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.714055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.713945 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/173c3533-efe5-4d88-998b-33d87f347c0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.719828 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.719809 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89g4l\" (UniqueName: \"kubernetes.io/projected/173c3533-efe5-4d88-998b-33d87f347c0c-kube-api-access-89g4l\") pod \"insights-runtime-extractor-4q2xx\" (UID: \"173c3533-efe5-4d88-998b-33d87f347c0c\") " pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.720225 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.720208 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p26z\" (UniqueName: \"kubernetes.io/projected/a5124801-48ed-45ec-8405-0c1b43c560f0-kube-api-access-8p26z\") pod \"dns-default-c77n8\" (UID: \"a5124801-48ed-45ec-8405-0c1b43c560f0\") " pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.811221 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.811184 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65af6ca4-eedb-453d-b6d3-3c293d46fd3c-cert\") pod \"ingress-canary-mjrrq\" (UID: \"65af6ca4-eedb-453d-b6d3-3c293d46fd3c\") " pod="openshift-ingress-canary/ingress-canary-mjrrq" Jul 16 16:52:08.811445 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.811238 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8v5k\" (UniqueName: \"kubernetes.io/projected/65af6ca4-eedb-453d-b6d3-3c293d46fd3c-kube-api-access-z8v5k\") pod \"ingress-canary-mjrrq\" (UID: \"65af6ca4-eedb-453d-b6d3-3c293d46fd3c\") " pod="openshift-ingress-canary/ingress-canary-mjrrq" Jul 16 16:52:08.813591 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.813569 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65af6ca4-eedb-453d-b6d3-3c293d46fd3c-cert\") pod \"ingress-canary-mjrrq\" (UID: \"65af6ca4-eedb-453d-b6d3-3c293d46fd3c\") " pod="openshift-ingress-canary/ingress-canary-mjrrq" Jul 16 16:52:08.819282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.819260 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8v5k\" (UniqueName: \"kubernetes.io/projected/65af6ca4-eedb-453d-b6d3-3c293d46fd3c-kube-api-access-z8v5k\") pod \"ingress-canary-mjrrq\" (UID: \"65af6ca4-eedb-453d-b6d3-3c293d46fd3c\") " pod="openshift-ingress-canary/ingress-canary-mjrrq" Jul 16 16:52:08.848978 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.848949 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:08.870142 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.870114 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4q2xx" Jul 16 16:52:08.948855 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:08.948825 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mjrrq" Jul 16 16:52:09.002739 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.002550 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p"] Jul 16 16:52:09.010379 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.010353 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" Jul 16 16:52:09.013165 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.012846 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 16:52:09.013165 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.013049 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2qrgb\"" Jul 16 16:52:09.019349 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.017566 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p"] Jul 16 16:52:09.056947 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.056921 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4q2xx"] Jul 16 16:52:09.061095 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:09.061070 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173c3533_efe5_4d88_998b_33d87f347c0c.slice/crio-9c65411a3bb4ab4dabb0e39b1a413e80a74fb3e47c74d9e47a368f21508fac95 WatchSource:0}: Error finding container 9c65411a3bb4ab4dabb0e39b1a413e80a74fb3e47c74d9e47a368f21508fac95: Status 404 returned error can't find the container with id 9c65411a3bb4ab4dabb0e39b1a413e80a74fb3e47c74d9e47a368f21508fac95 Jul 16 16:52:09.072005 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.071979 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c77n8"] Jul 16 16:52:09.075041 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:09.075013 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5124801_48ed_45ec_8405_0c1b43c560f0.slice/crio-59ce69d6feeea76a777dd73ac736a54cce910f58ab33bb9cd10c1beee682c234 WatchSource:0}: Error finding container 59ce69d6feeea76a777dd73ac736a54cce910f58ab33bb9cd10c1beee682c234: Status 404 returned error can't find the container with id 59ce69d6feeea76a777dd73ac736a54cce910f58ab33bb9cd10c1beee682c234 Jul 16 16:52:09.112309 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.112278 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mjrrq"] Jul 16 16:52:09.113591 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.113570 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba43dc25-33f6-4880-b244-7f9ac519c4ec-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-9ck4p\" (UID: \"ba43dc25-33f6-4880-b244-7f9ac519c4ec\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" Jul 16 16:52:09.115314 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:09.115292 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65af6ca4_eedb_453d_b6d3_3c293d46fd3c.slice/crio-0e8bed5761a781a98d036734621e15c51a109d039c94050d53f524de47830188 WatchSource:0}: Error finding container 0e8bed5761a781a98d036734621e15c51a109d039c94050d53f524de47830188: Status 404 returned error can't find the container with id 0e8bed5761a781a98d036734621e15c51a109d039c94050d53f524de47830188 Jul 16 16:52:09.198903 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.198862 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mjrrq" event={"ID":"65af6ca4-eedb-453d-b6d3-3c293d46fd3c","Type":"ContainerStarted","Data":"0e8bed5761a781a98d036734621e15c51a109d039c94050d53f524de47830188"} Jul 16 16:52:09.199887 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.199862 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q2xx" event={"ID":"173c3533-efe5-4d88-998b-33d87f347c0c","Type":"ContainerStarted","Data":"9c65411a3bb4ab4dabb0e39b1a413e80a74fb3e47c74d9e47a368f21508fac95"} Jul 16 16:52:09.200824 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.200804 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c77n8" event={"ID":"a5124801-48ed-45ec-8405-0c1b43c560f0","Type":"ContainerStarted","Data":"59ce69d6feeea76a777dd73ac736a54cce910f58ab33bb9cd10c1beee682c234"} Jul 16 16:52:09.214224 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.214169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba43dc25-33f6-4880-b244-7f9ac519c4ec-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-9ck4p\" (UID: \"ba43dc25-33f6-4880-b244-7f9ac519c4ec\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" Jul 16 16:52:09.214294 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:09.214283 2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 16 16:52:09.214348 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:09.214338 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba43dc25-33f6-4880-b244-7f9ac519c4ec-tls-certificates podName:ba43dc25-33f6-4880-b244-7f9ac519c4ec nodeName:}" failed. No retries permitted until 2026-07-16 16:52:09.714320048 +0000 UTC m=+59.498475098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ba43dc25-33f6-4880-b244-7f9ac519c4ec-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-9ck4p" (UID: "ba43dc25-33f6-4880-b244-7f9ac519c4ec") : secret "prometheus-operator-admission-webhook-tls" not found Jul 16 16:52:09.719028 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.718501 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba43dc25-33f6-4880-b244-7f9ac519c4ec-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-9ck4p\" (UID: \"ba43dc25-33f6-4880-b244-7f9ac519c4ec\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" Jul 16 16:52:09.729015 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.728970 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba43dc25-33f6-4880-b244-7f9ac519c4ec-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-9ck4p\" (UID: \"ba43dc25-33f6-4880-b244-7f9ac519c4ec\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" Jul 16 16:52:09.837440 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.836770 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:52:09.837440 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.837214 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:52:09.837662 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.837586 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:52:09.839819 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.839795 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h8r6p\"" Jul 16 16:52:09.840675 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.840653 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 16:52:09.841191 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.840793 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 16:52:09.841191 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.840943 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s2b72\"" Jul 16 16:52:09.841191 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.841032 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 16:52:09.841191 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.841153 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 16:52:09.924168 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:09.923946 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" Jul 16 16:52:10.105285 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:10.105250 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p"] Jul 16 16:52:10.112112 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:10.112081 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba43dc25_33f6_4880_b244_7f9ac519c4ec.slice/crio-6598d2d585b0cdcd00e64b5b1ff8d315ce212aa7e39f4967240a26921d71b0b4 WatchSource:0}: Error finding container 6598d2d585b0cdcd00e64b5b1ff8d315ce212aa7e39f4967240a26921d71b0b4: Status 404 returned error can't find the container with id 6598d2d585b0cdcd00e64b5b1ff8d315ce212aa7e39f4967240a26921d71b0b4 Jul 16 16:52:10.142574 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:10.142230 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x84hq" Jul 16 16:52:10.206709 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:10.206675 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" event={"ID":"ba43dc25-33f6-4880-b244-7f9ac519c4ec","Type":"ContainerStarted","Data":"6598d2d585b0cdcd00e64b5b1ff8d315ce212aa7e39f4967240a26921d71b0b4"} Jul 16 16:52:15.223218 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.223125 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c77n8" event={"ID":"a5124801-48ed-45ec-8405-0c1b43c560f0","Type":"ContainerStarted","Data":"fa179399553dc56d3689809e8937a7a054abb5b90c34f47605f3a5136f88436d"} Jul 16 16:52:15.223699 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.223224 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c77n8" event={"ID":"a5124801-48ed-45ec-8405-0c1b43c560f0","Type":"ContainerStarted","Data":"8e786f6feb8200bced6233b57ea8e6e18a194d909013c8713c1ac201e439686c"} Jul 16 16:52:15.223699 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.223294 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:15.225317 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.225272 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q2xx" event={"ID":"173c3533-efe5-4d88-998b-33d87f347c0c","Type":"ContainerStarted","Data":"b5d96049e3222edb1fa4b9ad17375f5da907ebcfe5e59d5cbea010a8c60b1c86"} Jul 16 16:52:15.227913 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.227678 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mjrrq" event={"ID":"65af6ca4-eedb-453d-b6d3-3c293d46fd3c","Type":"ContainerStarted","Data":"ef9124646a692b40300bd590b694420d729aa2c63b1c05b0a5b32d7919d8221f"} Jul 16 16:52:15.229793 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.229769 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" event={"ID":"ba43dc25-33f6-4880-b244-7f9ac519c4ec","Type":"ContainerStarted","Data":"951714ffbb323a19636996b8a35bc9748d937a6c990c0f349eccae89b00769f0"} Jul 16 16:52:15.230000 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.229979 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" Jul 16 16:52:15.235467 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.235444 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" Jul 16 16:52:15.243252 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.243206 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c77n8" podStartSLOduration=1.600850076 podStartE2EDuration="7.243194254s" totalImagesPullingTime="5.642344178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:15.241574816 +0000 UTC m=+65.025729877" watchObservedRunningTime="2026-07-16 16:52:15.243194254 +0000 UTC m=+65.027349349" Jul 16 16:52:15.257351 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.257300 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mjrrq" podStartSLOduration=1.654175123 podStartE2EDuration="7.257283871s" totalImagesPullingTime="5.603108748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:15.256530453 +0000 UTC m=+65.040685516" watchObservedRunningTime="2026-07-16 16:52:15.257283871 +0000 UTC m=+65.041438933" Jul 16 16:52:15.272799 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:15.272010 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-9ck4p" podStartSLOduration=2.669782135 podStartE2EDuration="7.27195851s" totalImagesPullingTime="4.602176375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:15.271203733 +0000 UTC m=+65.055358793" watchObservedRunningTime="2026-07-16 16:52:15.27195851 +0000 UTC m=+65.056113565" Jul 16 16:52:16.087707 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.087671 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-dwzn8"] Jul 16 16:52:16.089689 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.089656 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.092134 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.092110 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 16:52:16.092300 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.092177 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 16:52:16.092610 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.092581 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 16:52:16.093407 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.093384 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-z4qjf\"" Jul 16 16:52:16.093515 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.093432 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 16:52:16.093556 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.093536 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 16:52:16.100157 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.100134 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-dwzn8"] Jul 16 16:52:16.175812 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.175782 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3204b7fb-1086-4d9e-9109-c89961227be3-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.175976 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.175849 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xbcz\" (UniqueName: \"kubernetes.io/projected/3204b7fb-1086-4d9e-9109-c89961227be3-kube-api-access-6xbcz\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.175976 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.175932 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3204b7fb-1086-4d9e-9109-c89961227be3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.175976 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.175968 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3204b7fb-1086-4d9e-9109-c89961227be3-metrics-client-ca\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.276988 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.276950 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xbcz\" (UniqueName: \"kubernetes.io/projected/3204b7fb-1086-4d9e-9109-c89961227be3-kube-api-access-6xbcz\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.277407 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.277015 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3204b7fb-1086-4d9e-9109-c89961227be3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.277407 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.277047 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3204b7fb-1086-4d9e-9109-c89961227be3-metrics-client-ca\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.277407 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.277077 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3204b7fb-1086-4d9e-9109-c89961227be3-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.277868 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.277845 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3204b7fb-1086-4d9e-9109-c89961227be3-metrics-client-ca\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.279708 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.279680 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3204b7fb-1086-4d9e-9109-c89961227be3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.279809 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.279760 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3204b7fb-1086-4d9e-9109-c89961227be3-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.287087 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.287062 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xbcz\" (UniqueName: \"kubernetes.io/projected/3204b7fb-1086-4d9e-9109-c89961227be3-kube-api-access-6xbcz\") pod \"prometheus-operator-64785b6675-dwzn8\" (UID: \"3204b7fb-1086-4d9e-9109-c89961227be3\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.401219 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.401137 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" Jul 16 16:52:16.478564 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.478528 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:52:16.478720 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.478600 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:52:16.481169 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.481140 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 16:52:16.481169 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.481140 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 16:52:16.491684 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.491661 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51875cc1-104f-460c-b55f-17a7340ce099-metrics-certs\") pod \"network-metrics-daemon-sw5rt\" (UID: \"51875cc1-104f-460c-b55f-17a7340ce099\") " pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:52:16.491847 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.491706 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c582a6c2-65b9-40e6-8382-a7c3133552cd-original-pull-secret\") pod \"global-pull-secret-syncer-gj948\" (UID: \"c582a6c2-65b9-40e6-8382-a7c3133552cd\") " pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:52:16.579734 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.579694 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwzj\" (UniqueName: \"kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj\") pod \"network-check-target-stcqc\" (UID: \"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c\") " pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:52:16.582282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.582247 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 16:52:16.592494 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.592465 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 16:52:16.602874 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.602846 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktwzj\" (UniqueName: \"kubernetes.io/projected/5771265b-9fe3-43a2-b9ca-aa2c4a352f2c-kube-api-access-ktwzj\") pod \"network-check-target-stcqc\" (UID: \"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c\") " pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:52:16.755638 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.755604 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h8r6p\"" Jul 16 16:52:16.762266 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.762232 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gj948" Jul 16 16:52:16.763301 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.763278 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:52:16.771452 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.771424 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s2b72\"" Jul 16 16:52:16.779472 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:16.779441 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sw5rt" Jul 16 16:52:18.180045 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:18.180016 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sw5rt"] Jul 16 16:52:18.184041 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:18.184014 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51875cc1_104f_460c_b55f_17a7340ce099.slice/crio-5d3570faf5c4f6a176250298c1b14eecca94b4bb8fc2af8f81765ce77e3d9a30 WatchSource:0}: Error finding container 5d3570faf5c4f6a176250298c1b14eecca94b4bb8fc2af8f81765ce77e3d9a30: Status 404 returned error can't find the container with id 5d3570faf5c4f6a176250298c1b14eecca94b4bb8fc2af8f81765ce77e3d9a30 Jul 16 16:52:18.231945 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:18.231911 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gj948"] Jul 16 16:52:18.234918 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:18.234886 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc582a6c2_65b9_40e6_8382_a7c3133552cd.slice/crio-b56dd658c4ad5efca8196981b2ed611d97c8afe497357f420925102f6a17655d WatchSource:0}: Error finding container b56dd658c4ad5efca8196981b2ed611d97c8afe497357f420925102f6a17655d: Status 404 returned error can't find the container with id b56dd658c4ad5efca8196981b2ed611d97c8afe497357f420925102f6a17655d Jul 16 16:52:18.242466 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:18.242439 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q2xx" event={"ID":"173c3533-efe5-4d88-998b-33d87f347c0c","Type":"ContainerStarted","Data":"3c0c3f730851ce2fc30e2de3fb80f0693eac4c7979ff154ae37c7931dc9a1ea6"} Jul 16 16:52:18.243463 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:18.243438 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gj948" event={"ID":"c582a6c2-65b9-40e6-8382-a7c3133552cd","Type":"ContainerStarted","Data":"b56dd658c4ad5efca8196981b2ed611d97c8afe497357f420925102f6a17655d"} Jul 16 16:52:18.244371 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:18.244352 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sw5rt" event={"ID":"51875cc1-104f-460c-b55f-17a7340ce099","Type":"ContainerStarted","Data":"5d3570faf5c4f6a176250298c1b14eecca94b4bb8fc2af8f81765ce77e3d9a30"} Jul 16 16:52:18.405068 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:18.404993 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-stcqc"] Jul 16 16:52:18.407615 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:18.407581 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-dwzn8"] Jul 16 16:52:18.408284 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:18.408256 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5771265b_9fe3_43a2_b9ca_aa2c4a352f2c.slice/crio-26f50e04326d40a002b845ec42447eb501cbeac4f905d35228308f37343a6779 WatchSource:0}: Error finding container 26f50e04326d40a002b845ec42447eb501cbeac4f905d35228308f37343a6779: Status 404 returned error can't find the container with id 26f50e04326d40a002b845ec42447eb501cbeac4f905d35228308f37343a6779 Jul 16 16:52:18.411953 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:18.411926 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3204b7fb_1086_4d9e_9109_c89961227be3.slice/crio-31d1b272e6c29dcfc196b620f523ca9cf2a06fd21ef5439ce77ed19eedb00ad4 WatchSource:0}: Error finding container 31d1b272e6c29dcfc196b620f523ca9cf2a06fd21ef5439ce77ed19eedb00ad4: Status 404 returned error can't find the container with id 31d1b272e6c29dcfc196b620f523ca9cf2a06fd21ef5439ce77ed19eedb00ad4 Jul 16 16:52:19.249467 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.249437 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-stcqc" event={"ID":"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c","Type":"ContainerStarted","Data":"26f50e04326d40a002b845ec42447eb501cbeac4f905d35228308f37343a6779"} Jul 16 16:52:19.250740 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.250703 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" event={"ID":"3204b7fb-1086-4d9e-9109-c89961227be3","Type":"ContainerStarted","Data":"31d1b272e6c29dcfc196b620f523ca9cf2a06fd21ef5439ce77ed19eedb00ad4"} Jul 16 16:52:19.264176 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.264150 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-p5495"] Jul 16 16:52:19.266622 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.266603 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-p5495" Jul 16 16:52:19.269166 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.269149 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 16:52:19.269685 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.269493 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 16:52:19.269685 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.269570 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-khcnn\"" Jul 16 16:52:19.278884 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.278600 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-p5495"] Jul 16 16:52:19.399237 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.399209 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgpzv\" (UniqueName: \"kubernetes.io/projected/eff21e0b-f006-4ed0-98ed-12ae223f5149-kube-api-access-kgpzv\") pod \"downloads-68c9cbb497-p5495\" (UID: \"eff21e0b-f006-4ed0-98ed-12ae223f5149\") " pod="openshift-console/downloads-68c9cbb497-p5495" Jul 16 16:52:19.499567 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.499531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgpzv\" (UniqueName: \"kubernetes.io/projected/eff21e0b-f006-4ed0-98ed-12ae223f5149-kube-api-access-kgpzv\") pod \"downloads-68c9cbb497-p5495\" (UID: \"eff21e0b-f006-4ed0-98ed-12ae223f5149\") " pod="openshift-console/downloads-68c9cbb497-p5495" Jul 16 16:52:19.513832 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.513806 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgpzv\" (UniqueName: \"kubernetes.io/projected/eff21e0b-f006-4ed0-98ed-12ae223f5149-kube-api-access-kgpzv\") pod \"downloads-68c9cbb497-p5495\" (UID: \"eff21e0b-f006-4ed0-98ed-12ae223f5149\") " pod="openshift-console/downloads-68c9cbb497-p5495" Jul 16 16:52:19.584874 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.584829 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-p5495" Jul 16 16:52:19.738733 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:19.738699 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-p5495"] Jul 16 16:52:19.743617 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:19.743588 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff21e0b_f006_4ed0_98ed_12ae223f5149.slice/crio-28c8cd86deecbbe12eff1a9d390aec4cacd76d87a4826f44b47cb0e9aad8777e WatchSource:0}: Error finding container 28c8cd86deecbbe12eff1a9d390aec4cacd76d87a4826f44b47cb0e9aad8777e: Status 404 returned error can't find the container with id 28c8cd86deecbbe12eff1a9d390aec4cacd76d87a4826f44b47cb0e9aad8777e Jul 16 16:52:20.257290 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:20.257211 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-p5495" event={"ID":"eff21e0b-f006-4ed0-98ed-12ae223f5149","Type":"ContainerStarted","Data":"28c8cd86deecbbe12eff1a9d390aec4cacd76d87a4826f44b47cb0e9aad8777e"} Jul 16 16:52:20.259633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:20.259401 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sw5rt" event={"ID":"51875cc1-104f-460c-b55f-17a7340ce099","Type":"ContainerStarted","Data":"5a74d0db6ad7616580a257e77c4af52e512eeafcdc400caf5ceacad4d4ed48b7"} Jul 16 16:52:20.259633 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:20.259479 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sw5rt" event={"ID":"51875cc1-104f-460c-b55f-17a7340ce099","Type":"ContainerStarted","Data":"3b2829134226e414977dba9166a9b4e22aceefcbdcab988272f4ea8f0c0fb0a7"} Jul 16 16:52:20.280893 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:20.280818 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sw5rt" podStartSLOduration=69.30099323 podStartE2EDuration="1m10.280795791s" totalImagesPullingTime="979.802561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:20.280220787 +0000 UTC m=+70.064375851" watchObservedRunningTime="2026-07-16 16:52:20.280795791 +0000 UTC m=+70.064950853" Jul 16 16:52:25.236524 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:25.236486 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c77n8" Jul 16 16:52:26.289910 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:26.289111 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-stcqc" event={"ID":"5771265b-9fe3-43a2-b9ca-aa2c4a352f2c","Type":"ContainerStarted","Data":"def813fb712dd9d50aa58a5a6f64ccba40eec732d1e001f924f1fee6e5c3f949"} Jul 16 16:52:26.289910 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:26.289812 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:52:26.297380 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:26.296842 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q2xx" event={"ID":"173c3533-efe5-4d88-998b-33d87f347c0c","Type":"ContainerStarted","Data":"dc8e94b5e494794525f61fd8f741a8fa80e9cc1cdfffa97d524fb35a409251b0"} Jul 16 16:52:26.299782 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:26.299752 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gj948" event={"ID":"c582a6c2-65b9-40e6-8382-a7c3133552cd","Type":"ContainerStarted","Data":"aa6c72c174e679b6950c5f0c1aa671861fbfdf21f400f3cb4c4caeb68ea5b07e"} Jul 16 16:52:26.303578 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:26.303090 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" event={"ID":"3204b7fb-1086-4d9e-9109-c89961227be3","Type":"ContainerStarted","Data":"95473a67126cacc88a9ea15e739c2eb842a2953d0cd22d138339ff3f5ec2aec4"} Jul 16 16:52:26.303578 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:26.303136 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" event={"ID":"3204b7fb-1086-4d9e-9109-c89961227be3","Type":"ContainerStarted","Data":"b0c20525edd18bbb73bd3b89bfdc5d4b179fc463f8c183c9386d997e9129841d"} Jul 16 16:52:26.308098 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:26.308057 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-stcqc" podStartSLOduration=68.721009104 podStartE2EDuration="1m16.308043786s" totalImagesPullingTime="7.587034682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:26.306942557 +0000 UTC m=+76.091097618" watchObservedRunningTime="2026-07-16 16:52:26.308043786 +0000 UTC m=+76.092198845" Jul 16 16:52:26.327476 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:26.327385 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4q2xx" podStartSLOduration=1.551208224 podStartE2EDuration="18.327364483s" totalImagesPullingTime="16.776156259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:52:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:26.326155806 +0000 UTC m=+76.110310867" watchObservedRunningTime="2026-07-16 16:52:26.327364483 +0000 UTC m=+76.111519582" Jul 16 16:52:26.343090 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:26.342910 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gj948" podStartSLOduration=67.568297234 podStartE2EDuration="1m15.342894523s" totalImagesPullingTime="7.774597289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:26.342252664 +0000 UTC m=+76.126407737" watchObservedRunningTime="2026-07-16 16:52:26.342894523 +0000 UTC m=+76.127049583" Jul 16 16:52:26.360325 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:26.360093 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-dwzn8" podStartSLOduration=2.7769967700000002 podStartE2EDuration="10.360076063s" totalImagesPullingTime="7.583079293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:26.359602811 +0000 UTC m=+76.143757873" watchObservedRunningTime="2026-07-16 16:52:26.360076063 +0000 UTC m=+76.144231123" Jul 16 16:52:28.436187 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.436153 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv"] Jul 16 16:52:28.439640 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.439611 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.441722 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.441693 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-lmxk5"] Jul 16 16:52:28.442078 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.442055 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 16:52:28.442189 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.442055 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 16:52:28.442559 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.442543 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-b8kb6\"" Jul 16 16:52:28.444878 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.444859 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.447381 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.447292 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-psgr6\"" Jul 16 16:52:28.447381 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.447388 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 16:52:28.447563 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.447444 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 16:52:28.447905 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.447888 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 16:52:28.452978 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.452580 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv"] Jul 16 16:52:28.454379 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.454357 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vq2kf"] Jul 16 16:52:28.468792 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.467641 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-lmxk5"] Jul 16 16:52:28.468792 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.467776 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.473375 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.471920 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 16:52:28.473375 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.472151 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 16:52:28.473375 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.472321 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2tl75\"" Jul 16 16:52:28.473375 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.472548 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 16:52:28.573838 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.573803 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.573838 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.573843 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0e8127-3e57-4679-b5b5-0f6df7811c19-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.574070 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.573866 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-accelerators-collector-config\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.574070 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.573918 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/323be6e6-4068-442e-ad56-01e7cda84457-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.574070 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.573947 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-tls\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.574070 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.573980 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.574070 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3d0e8127-3e57-4679-b5b5-0f6df7811c19-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.574070 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574041 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.574070 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574066 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-wtmp\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.574336 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574093 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgz2\" (UniqueName: \"kubernetes.io/projected/201ad1e8-9e03-4355-b116-7141950f9997-kube-api-access-9xgz2\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.574336 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574175 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/201ad1e8-9e03-4355-b116-7141950f9997-root\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.574336 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574201 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/201ad1e8-9e03-4355-b116-7141950f9997-metrics-client-ca\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.574336 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574235 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhw5w\" (UniqueName: \"kubernetes.io/projected/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-api-access-xhw5w\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.574336 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574261 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/201ad1e8-9e03-4355-b116-7141950f9997-sys\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.574336 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574302 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c68x7\" (UniqueName: \"kubernetes.io/projected/323be6e6-4068-442e-ad56-01e7cda84457-kube-api-access-c68x7\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.574336 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574330 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.574632 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574372 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/323be6e6-4068-442e-ad56-01e7cda84457-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.574632 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574403 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/323be6e6-4068-442e-ad56-01e7cda84457-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.574632 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.574470 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-textfile\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.675550 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675517 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/323be6e6-4068-442e-ad56-01e7cda84457-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.675719 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675559 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-textfile\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.675719 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675607 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.675719 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675634 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0e8127-3e57-4679-b5b5-0f6df7811c19-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.675719 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675661 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-accelerators-collector-config\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.675719 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675689 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/323be6e6-4068-442e-ad56-01e7cda84457-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.675719 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675712 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-tls\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675741 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675767 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3d0e8127-3e57-4679-b5b5-0f6df7811c19-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675793 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675817 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-wtmp\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675837 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgz2\" (UniqueName: \"kubernetes.io/projected/201ad1e8-9e03-4355-b116-7141950f9997-kube-api-access-9xgz2\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675864 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/201ad1e8-9e03-4355-b116-7141950f9997-root\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675890 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/201ad1e8-9e03-4355-b116-7141950f9997-metrics-client-ca\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675927 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhw5w\" (UniqueName: \"kubernetes.io/projected/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-api-access-xhw5w\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675948 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/201ad1e8-9e03-4355-b116-7141950f9997-sys\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675975 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68x7\" (UniqueName: \"kubernetes.io/projected/323be6e6-4068-442e-ad56-01e7cda84457-kube-api-access-c68x7\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.676003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.675998 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.676553 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.676028 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/323be6e6-4068-442e-ad56-01e7cda84457-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.676553 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.676030 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-textfile\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676553 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.676167 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-wtmp\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676553 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.676318 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/323be6e6-4068-442e-ad56-01e7cda84457-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.676553 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.676346 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0e8127-3e57-4679-b5b5-0f6df7811c19-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.676553 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.676470 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/201ad1e8-9e03-4355-b116-7141950f9997-root\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676553 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:28.676488 2556 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 16 16:52:28.676553 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:28.676559 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-tls podName:3d0e8127-3e57-4679-b5b5-0f6df7811c19 nodeName:}" failed. No retries permitted until 2026-07-16 16:52:29.176542042 +0000 UTC m=+78.960697094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-lmxk5" (UID: "3d0e8127-3e57-4679-b5b5-0f6df7811c19") : secret "kube-state-metrics-tls" not found Jul 16 16:52:28.676966 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.676631 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/201ad1e8-9e03-4355-b116-7141950f9997-sys\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676966 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.676714 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-accelerators-collector-config\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.676966 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.676954 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/201ad1e8-9e03-4355-b116-7141950f9997-metrics-client-ca\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.677288 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.677265 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3d0e8127-3e57-4679-b5b5-0f6df7811c19-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.677378 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.677342 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.678870 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.678832 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/323be6e6-4068-442e-ad56-01e7cda84457-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.679649 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.679625 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.679733 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.679627 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/323be6e6-4068-442e-ad56-01e7cda84457-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.679801 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.679759 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/201ad1e8-9e03-4355-b116-7141950f9997-node-exporter-tls\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.679853 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.679794 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.686534 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.686302 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhw5w\" (UniqueName: \"kubernetes.io/projected/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-api-access-xhw5w\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:28.686534 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.686467 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68x7\" (UniqueName: \"kubernetes.io/projected/323be6e6-4068-442e-ad56-01e7cda84457-kube-api-access-c68x7\") pod \"openshift-state-metrics-84bd68b896-f5gnv\" (UID: \"323be6e6-4068-442e-ad56-01e7cda84457\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.686674 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.686621 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgz2\" (UniqueName: \"kubernetes.io/projected/201ad1e8-9e03-4355-b116-7141950f9997-kube-api-access-9xgz2\") pod \"node-exporter-vq2kf\" (UID: \"201ad1e8-9e03-4355-b116-7141950f9997\") " pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.753048 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.753017 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" Jul 16 16:52:28.787796 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.787768 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vq2kf" Jul 16 16:52:28.800942 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:28.800780 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201ad1e8_9e03_4355_b116_7141950f9997.slice/crio-b7cd23d940702701dae1b1a348b2a75c0f3310f0eeced365b62e9748af27887e WatchSource:0}: Error finding container b7cd23d940702701dae1b1a348b2a75c0f3310f0eeced365b62e9748af27887e: Status 404 returned error can't find the container with id b7cd23d940702701dae1b1a348b2a75c0f3310f0eeced365b62e9748af27887e Jul 16 16:52:28.884951 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:28.884922 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv"] Jul 16 16:52:28.888004 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:28.887977 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323be6e6_4068_442e_ad56_01e7cda84457.slice/crio-80a0497906acf2722c30e14ceabceafac6f9255d1dc814065712c2040d25ac22 WatchSource:0}: Error finding container 80a0497906acf2722c30e14ceabceafac6f9255d1dc814065712c2040d25ac22: Status 404 returned error can't find the container with id 80a0497906acf2722c30e14ceabceafac6f9255d1dc814065712c2040d25ac22 Jul 16 16:52:29.179304 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.179264 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:29.182038 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.182013 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0e8127-3e57-4679-b5b5-0f6df7811c19-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-lmxk5\" (UID: \"3d0e8127-3e57-4679-b5b5-0f6df7811c19\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:29.315010 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.314970 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vq2kf" event={"ID":"201ad1e8-9e03-4355-b116-7141950f9997","Type":"ContainerStarted","Data":"b7cd23d940702701dae1b1a348b2a75c0f3310f0eeced365b62e9748af27887e"} Jul 16 16:52:29.317027 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.316998 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" event={"ID":"323be6e6-4068-442e-ad56-01e7cda84457","Type":"ContainerStarted","Data":"535fc00f5102df878942b37ceec3fb6da71573583fe18c2c7d396f730c2dc9c6"} Jul 16 16:52:29.317128 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.317049 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" event={"ID":"323be6e6-4068-442e-ad56-01e7cda84457","Type":"ContainerStarted","Data":"6e577ccda6cfd44da0514c9d025e74ee8ef1e5b1a3a795aa81510854eef8ef14"} Jul 16 16:52:29.317128 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.317068 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" event={"ID":"323be6e6-4068-442e-ad56-01e7cda84457","Type":"ContainerStarted","Data":"80a0497906acf2722c30e14ceabceafac6f9255d1dc814065712c2040d25ac22"} Jul 16 16:52:29.368505 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.368218 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" Jul 16 16:52:29.545058 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.545025 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-lmxk5"] Jul 16 16:52:29.549015 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:29.548989 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0e8127_3e57_4679_b5b5_0f6df7811c19.slice/crio-d362d33829fbfc1485df95fa6c1bb7bf8e1bbd4334cfe0241431037185c9bb71 WatchSource:0}: Error finding container d362d33829fbfc1485df95fa6c1bb7bf8e1bbd4334cfe0241431037185c9bb71: Status 404 returned error can't find the container with id d362d33829fbfc1485df95fa6c1bb7bf8e1bbd4334cfe0241431037185c9bb71 Jul 16 16:52:29.570132 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.568878 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:52:29.573843 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.573818 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.582112 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.582082 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 16:52:29.582357 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.582340 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 16:52:29.582667 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.582649 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 16:52:29.582875 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.582859 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-74cpr\"" Jul 16 16:52:29.583047 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.583032 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 16:52:29.583235 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.583220 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 16:52:29.583397 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.583382 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 16:52:29.583644 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.583628 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 16:52:29.583814 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.583798 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 16:52:29.584199 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.584172 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 16:52:29.594817 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.594719 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:52:29.685882 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.685846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686032 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.685899 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686032 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.685934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686156 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.686038 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686156 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.686081 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2j9\" (UniqueName: \"kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-kube-api-access-lc2j9\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686156 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.686129 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-config-volume\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686357 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.686170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686357 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.686221 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-config-out\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686357 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.686263 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686357 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.686289 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686357 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.686320 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-web-config\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686621 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.686363 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.686621 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.686385 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.787828 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.787881 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.787911 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.787941 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.787965 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2j9\" (UniqueName: \"kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-kube-api-access-lc2j9\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.787998 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-config-volume\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.788023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.788047 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-config-out\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.788073 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.788097 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.788127 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-web-config\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.788167 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.788575 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.788191 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.789296 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.789025 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.789781 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.789685 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.790636 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.790500 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.794881 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.794812 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.794881 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.794836 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-config-out\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.795060 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.794927 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.795699 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.795681 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.795939 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.795864 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.796961 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.796939 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.797282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.797259 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-config-volume\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.798212 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.797525 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-web-config\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.798405 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.798346 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.801371 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.801327 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2j9\" (UniqueName: \"kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-kube-api-access-lc2j9\") pod \"alertmanager-main-0\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:29.894799 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:29.894718 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:52:30.335187 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.334573 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" event={"ID":"323be6e6-4068-442e-ad56-01e7cda84457","Type":"ContainerStarted","Data":"fa9cc796267841291a3cb81e8aefc44a42e0da2ebec701d4bbc9bb5008f2140e"} Jul 16 16:52:30.338993 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.338438 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vq2kf" event={"ID":"201ad1e8-9e03-4355-b116-7141950f9997","Type":"ContainerStarted","Data":"23b4a16ffcde607f5c988e6b8310b579b108c83e600c22ce4605d0e03dc51b21"} Jul 16 16:52:30.343273 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.343195 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" event={"ID":"3d0e8127-3e57-4679-b5b5-0f6df7811c19","Type":"ContainerStarted","Data":"d362d33829fbfc1485df95fa6c1bb7bf8e1bbd4334cfe0241431037185c9bb71"} Jul 16 16:52:30.356993 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.356896 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f5gnv" podStartSLOduration=1.150498462 podStartE2EDuration="2.356870507s" totalImagesPullingTime="1.206372045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:30.355181375 +0000 UTC m=+80.139336482" watchObservedRunningTime="2026-07-16 16:52:30.356870507 +0000 UTC m=+80.141025567" Jul 16 16:52:30.399497 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.399466 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:52:30.401836 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:30.401804 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda245bb98_ef13_4f03_b793_53b541bfa380.slice/crio-906535864d5c6ce26cfcfda04c2a1bc092c4b7aad3edc39274f10acd2da1de58 WatchSource:0}: Error finding container 906535864d5c6ce26cfcfda04c2a1bc092c4b7aad3edc39274f10acd2da1de58: Status 404 returned error can't find the container with id 906535864d5c6ce26cfcfda04c2a1bc092c4b7aad3edc39274f10acd2da1de58 Jul 16 16:52:30.548465 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.548114 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6958bc8f99-4sd92"] Jul 16 16:52:30.552174 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.552152 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.554448 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.554397 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 16 16:52:30.554699 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.554682 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 16 16:52:30.555271 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.554962 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 16 16:52:30.555271 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.555058 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 16 16:52:30.555271 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.555077 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b6ipq7dnbfudh\"" Jul 16 16:52:30.555271 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.555156 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fh4jg\"" Jul 16 16:52:30.555271 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.555234 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 16 16:52:30.563036 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.563009 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6958bc8f99-4sd92"] Jul 16 16:52:30.696589 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.696553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e4c168d-3ee6-4e64-87d0-730dd83180e5-metrics-client-ca\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.696745 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.696603 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.696745 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.696708 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfmr\" (UniqueName: \"kubernetes.io/projected/7e4c168d-3ee6-4e64-87d0-730dd83180e5-kube-api-access-kvfmr\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.696905 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.696762 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-tls\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.696905 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.696810 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.696905 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.696846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.696905 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.696896 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-grpc-tls\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.697064 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.696938 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.797589 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.797555 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfmr\" (UniqueName: \"kubernetes.io/projected/7e4c168d-3ee6-4e64-87d0-730dd83180e5-kube-api-access-kvfmr\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.797765 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.797612 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-tls\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.797765 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.797654 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.797765 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.797688 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.797765 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.797717 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-grpc-tls\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.797765 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.797751 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.798012 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.797785 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e4c168d-3ee6-4e64-87d0-730dd83180e5-metrics-client-ca\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.798012 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.797815 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.798899 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.798801 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e4c168d-3ee6-4e64-87d0-730dd83180e5-metrics-client-ca\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.800710 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.800666 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.800821 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.800739 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-tls\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.800821 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.800796 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.801073 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.801054 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-grpc-tls\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.801185 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.801163 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.801247 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.801173 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e4c168d-3ee6-4e64-87d0-730dd83180e5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.810754 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.810731 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfmr\" (UniqueName: \"kubernetes.io/projected/7e4c168d-3ee6-4e64-87d0-730dd83180e5-kube-api-access-kvfmr\") pod \"thanos-querier-6958bc8f99-4sd92\" (UID: \"7e4c168d-3ee6-4e64-87d0-730dd83180e5\") " pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.864731 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.864700 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:30.987487 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.987450 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-696d77865d-2852v"] Jul 16 16:52:30.990792 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.990767 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:30.993354 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.993331 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hfqgv\"" Jul 16 16:52:30.993515 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.993357 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 16:52:30.993515 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.993380 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 16:52:30.993515 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.993495 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 16:52:30.993515 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.993331 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 16:52:30.993725 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:30.993691 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 16:52:31.003276 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.003255 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-696d77865d-2852v"] Jul 16 16:52:31.099621 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.099538 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-oauth-config\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.099621 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.099586 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-oauth-serving-cert\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.099621 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.099619 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-service-ca\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.099863 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.099688 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbxp\" (UniqueName: \"kubernetes.io/projected/841c4a67-2843-440e-bdb9-e73cd0b0ac42-kube-api-access-cwbxp\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.099863 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.099753 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-serving-cert\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.099863 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.099797 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-config\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.201061 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.201020 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-oauth-config\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.201237 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.201074 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-oauth-serving-cert\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.201237 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.201101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-service-ca\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.201237 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.201130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbxp\" (UniqueName: \"kubernetes.io/projected/841c4a67-2843-440e-bdb9-e73cd0b0ac42-kube-api-access-cwbxp\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.201237 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.201186 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-serving-cert\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.201237 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.201228 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-config\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.201951 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.201926 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-service-ca\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.202050 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.202039 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-oauth-serving-cert\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.202112 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.202037 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-config\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.204224 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.204203 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-oauth-config\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.204329 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.204310 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-serving-cert\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.210911 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.210893 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbxp\" (UniqueName: \"kubernetes.io/projected/841c4a67-2843-440e-bdb9-e73cd0b0ac42-kube-api-access-cwbxp\") pod \"console-696d77865d-2852v\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.303083 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.303047 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:31.350601 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.350514 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerStarted","Data":"906535864d5c6ce26cfcfda04c2a1bc092c4b7aad3edc39274f10acd2da1de58"} Jul 16 16:52:31.352171 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.352143 2556 generic.go:356] "Generic (PLEG): container finished" podID="201ad1e8-9e03-4355-b116-7141950f9997" containerID="23b4a16ffcde607f5c988e6b8310b579b108c83e600c22ce4605d0e03dc51b21" exitCode=0 Jul 16 16:52:31.352295 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:31.352235 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vq2kf" event={"ID":"201ad1e8-9e03-4355-b116-7141950f9997","Type":"ContainerDied","Data":"23b4a16ffcde607f5c988e6b8310b579b108c83e600c22ce4605d0e03dc51b21"} Jul 16 16:52:32.753353 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.753319 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb"] Jul 16 16:52:32.758632 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.758611 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:32.761157 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.761136 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 16:52:32.761891 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.761866 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lh28p\"" Jul 16 16:52:32.762109 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.762093 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1592ra5opnrj8\"" Jul 16 16:52:32.762176 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.762153 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 16:52:32.762439 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.762404 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 16:52:32.762518 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.762426 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 16:52:32.767498 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.767476 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb"] Jul 16 16:52:32.915826 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.915705 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd3adc6-6f09-4128-911b-155204c58cb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:32.915826 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.915752 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0bd3adc6-6f09-4128-911b-155204c58cb0-client-ca-bundle\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:32.916023 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.915833 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0bd3adc6-6f09-4128-911b-155204c58cb0-metrics-server-audit-profiles\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:32.916023 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.915878 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0bd3adc6-6f09-4128-911b-155204c58cb0-secret-metrics-server-tls\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:32.916023 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.915914 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0bd3adc6-6f09-4128-911b-155204c58cb0-audit-log\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:32.916023 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.915955 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2fx\" (UniqueName: \"kubernetes.io/projected/0bd3adc6-6f09-4128-911b-155204c58cb0-kube-api-access-9x2fx\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:32.916167 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:32.916023 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0bd3adc6-6f09-4128-911b-155204c58cb0-secret-metrics-server-client-certs\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.016978 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.016906 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd3adc6-6f09-4128-911b-155204c58cb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.016978 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.016944 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0bd3adc6-6f09-4128-911b-155204c58cb0-client-ca-bundle\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.016978 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.016971 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0bd3adc6-6f09-4128-911b-155204c58cb0-metrics-server-audit-profiles\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.017231 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.016997 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0bd3adc6-6f09-4128-911b-155204c58cb0-secret-metrics-server-tls\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.017231 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.017029 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0bd3adc6-6f09-4128-911b-155204c58cb0-audit-log\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.017231 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.017065 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2fx\" (UniqueName: \"kubernetes.io/projected/0bd3adc6-6f09-4128-911b-155204c58cb0-kube-api-access-9x2fx\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.017231 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.017087 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0bd3adc6-6f09-4128-911b-155204c58cb0-secret-metrics-server-client-certs\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.017826 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.017682 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0bd3adc6-6f09-4128-911b-155204c58cb0-audit-log\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.017826 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.017747 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd3adc6-6f09-4128-911b-155204c58cb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.018332 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.018287 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0bd3adc6-6f09-4128-911b-155204c58cb0-metrics-server-audit-profiles\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.019794 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.019770 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0bd3adc6-6f09-4128-911b-155204c58cb0-client-ca-bundle\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.019961 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.019939 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0bd3adc6-6f09-4128-911b-155204c58cb0-secret-metrics-server-client-certs\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.020069 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.020054 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0bd3adc6-6f09-4128-911b-155204c58cb0-secret-metrics-server-tls\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.026136 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.026113 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2fx\" (UniqueName: \"kubernetes.io/projected/0bd3adc6-6f09-4128-911b-155204c58cb0-kube-api-access-9x2fx\") pod \"metrics-server-6d78b8c7b8-fvcsb\" (UID: \"0bd3adc6-6f09-4128-911b-155204c58cb0\") " pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.070368 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.070339 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:33.216600 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.216557 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq"] Jul 16 16:52:33.223229 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.223204 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:33.225623 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.225592 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 16 16:52:33.225849 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.225834 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kxhzf\"" Jul 16 16:52:33.230428 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.230386 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq"] Jul 16 16:52:33.319875 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.319796 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/84e0b799-0e15-4b9c-a6d0-c13fc6b418f4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r8pwq\" (UID: \"84e0b799-0e15-4b9c-a6d0-c13fc6b418f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:33.320012 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.319916 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72v4\" (UniqueName: \"kubernetes.io/projected/84e0b799-0e15-4b9c-a6d0-c13fc6b418f4-kube-api-access-x72v4\") pod \"monitoring-plugin-5f68df7f58-r8pwq\" (UID: \"84e0b799-0e15-4b9c-a6d0-c13fc6b418f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:33.420472 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.420434 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/84e0b799-0e15-4b9c-a6d0-c13fc6b418f4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r8pwq\" (UID: \"84e0b799-0e15-4b9c-a6d0-c13fc6b418f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:33.420639 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.420540 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72v4\" (UniqueName: \"kubernetes.io/projected/84e0b799-0e15-4b9c-a6d0-c13fc6b418f4-kube-api-access-x72v4\") pod \"monitoring-plugin-5f68df7f58-r8pwq\" (UID: \"84e0b799-0e15-4b9c-a6d0-c13fc6b418f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:33.420639 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:33.420607 2556 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 16 16:52:33.420713 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:52:33.420691 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e0b799-0e15-4b9c-a6d0-c13fc6b418f4-monitoring-plugin-cert podName:84e0b799-0e15-4b9c-a6d0-c13fc6b418f4 nodeName:}" failed. No retries permitted until 2026-07-16 16:52:33.920669218 +0000 UTC m=+83.704824260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/84e0b799-0e15-4b9c-a6d0-c13fc6b418f4-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-r8pwq" (UID: "84e0b799-0e15-4b9c-a6d0-c13fc6b418f4") : secret "monitoring-plugin-cert" not found Jul 16 16:52:33.432610 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.432588 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72v4\" (UniqueName: \"kubernetes.io/projected/84e0b799-0e15-4b9c-a6d0-c13fc6b418f4-kube-api-access-x72v4\") pod \"monitoring-plugin-5f68df7f58-r8pwq\" (UID: \"84e0b799-0e15-4b9c-a6d0-c13fc6b418f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:33.925724 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.925690 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/84e0b799-0e15-4b9c-a6d0-c13fc6b418f4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r8pwq\" (UID: \"84e0b799-0e15-4b9c-a6d0-c13fc6b418f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:33.928387 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:33.928363 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/84e0b799-0e15-4b9c-a6d0-c13fc6b418f4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r8pwq\" (UID: \"84e0b799-0e15-4b9c-a6d0-c13fc6b418f4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:34.135166 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.135129 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:34.743009 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.742975 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:52:34.810390 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.810352 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:52:34.810583 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.810536 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.813115 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.813090 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 16:52:34.813279 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.813214 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 16:52:34.813351 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.813320 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 16:52:34.813408 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.813350 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 16:52:34.813828 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.813805 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 16:52:34.814116 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.813993 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sqq2l\"" Jul 16 16:52:34.814116 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.814085 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 16:52:34.814677 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.814654 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 16:52:34.815087 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.814945 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 16:52:34.815087 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.814988 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2pqm43lh4gpae\"" Jul 16 16:52:34.815087 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.815083 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 16:52:34.818774 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.815167 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 16:52:34.818774 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.816556 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 16:52:34.818774 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.816672 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 16:52:34.824487 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.824467 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 16:52:34.825741 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.825710 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 16:52:34.935984 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.935936 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.935984 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.935985 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936016 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936108 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936185 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936212 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936243 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-config\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936280 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936326 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936347 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936444 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936469 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936495 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.936522 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936518 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.937096 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936551 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrpcx\" (UniqueName: \"kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-kube-api-access-zrpcx\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.937096 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936574 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.937096 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936600 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.937096 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936620 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.937096 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936639 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:34.937096 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:34.936653 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.037742 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037659 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrpcx\" (UniqueName: \"kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-kube-api-access-zrpcx\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.037742 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037710 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.037742 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037741 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037769 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037796 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037820 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037849 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037873 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037903 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037928 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037970 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.037996 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-config\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038060 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038595 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038105 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038595 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038595 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038595 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038191 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038595 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038595 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038249 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038595 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038367 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038986 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038665 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.038986 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.038693 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.040149 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.039238 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.040149 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.039283 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.040149 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.039872 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.041863 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.041541 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.044533 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.044481 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.044642 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.044603 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.045238 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.045216 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.045647 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.045621 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.046607 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.046582 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.048855 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.048822 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.049136 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.049116 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.049307 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.049266 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.049769 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.049701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.050350 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.050307 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.050636 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.050613 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrpcx\" (UniqueName: \"kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-kube-api-access-zrpcx\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.051323 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.051300 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-config\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.051579 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.051562 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:35.127827 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:35.127789 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:38.570762 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:38.569292 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-696d77865d-2852v"] Jul 16 16:52:38.575720 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:38.575649 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841c4a67_2843_440e_bdb9_e73cd0b0ac42.slice/crio-f0948fd3dbe972f4305de23326a4771e59d720472947b426a65b11e9f8f72bc3 WatchSource:0}: Error finding container f0948fd3dbe972f4305de23326a4771e59d720472947b426a65b11e9f8f72bc3: Status 404 returned error can't find the container with id f0948fd3dbe972f4305de23326a4771e59d720472947b426a65b11e9f8f72bc3 Jul 16 16:52:38.594080 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:38.594052 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq"] Jul 16 16:52:38.632313 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:38.632283 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:52:38.635972 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:38.635947 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8255767a_f1d8_4d76_8867_9ede2df130cc.slice/crio-07c6cf1c709086ece804a6e849f3a37d9310a93ada8f2d2547919d282bb7e091 WatchSource:0}: Error finding container 07c6cf1c709086ece804a6e849f3a37d9310a93ada8f2d2547919d282bb7e091: Status 404 returned error can't find the container with id 07c6cf1c709086ece804a6e849f3a37d9310a93ada8f2d2547919d282bb7e091 Jul 16 16:52:38.871474 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:38.871386 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb"] Jul 16 16:52:38.873672 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:38.873059 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6958bc8f99-4sd92"] Jul 16 16:52:39.227805 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:39.227766 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd3adc6_6f09_4128_911b_155204c58cb0.slice/crio-f087d8fa830f2033a0a2a61d7560ff4686b636399cb05a5aadc5b05eaea83723 WatchSource:0}: Error finding container f087d8fa830f2033a0a2a61d7560ff4686b636399cb05a5aadc5b05eaea83723: Status 404 returned error can't find the container with id f087d8fa830f2033a0a2a61d7560ff4686b636399cb05a5aadc5b05eaea83723 Jul 16 16:52:39.228775 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:52:39.228747 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4c168d_3ee6_4e64_87d0_730dd83180e5.slice/crio-2d1e635ed95e042a5ccfd9a35b77510813f88a2737ce22a815c6b19f2d2183bc WatchSource:0}: Error finding container 2d1e635ed95e042a5ccfd9a35b77510813f88a2737ce22a815c6b19f2d2183bc: Status 404 returned error can't find the container with id 2d1e635ed95e042a5ccfd9a35b77510813f88a2737ce22a815c6b19f2d2183bc Jul 16 16:52:39.390243 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.390208 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vq2kf" event={"ID":"201ad1e8-9e03-4355-b116-7141950f9997","Type":"ContainerStarted","Data":"a40ee08b7473d7920ae6911a18f7ee20f4170bd2e4e929c6e6dbc311f94354fe"} Jul 16 16:52:39.390444 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.390277 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vq2kf" event={"ID":"201ad1e8-9e03-4355-b116-7141950f9997","Type":"ContainerStarted","Data":"8407bc771dfb149716b40c98b8fec959dad6331902a0f84261f5229d31ad13f4"} Jul 16 16:52:39.393022 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.392992 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" event={"ID":"3d0e8127-3e57-4679-b5b5-0f6df7811c19","Type":"ContainerStarted","Data":"79065860386bc51e643494d30d280167c15bbfb8368097cecbb581cad0d1f287"} Jul 16 16:52:39.393132 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.393050 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" event={"ID":"3d0e8127-3e57-4679-b5b5-0f6df7811c19","Type":"ContainerStarted","Data":"008e0f6109cbb0857bf78c0200b01b13bfe34e0930d86a58e509a951c0875ca7"} Jul 16 16:52:39.393132 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.393079 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" event={"ID":"3d0e8127-3e57-4679-b5b5-0f6df7811c19","Type":"ContainerStarted","Data":"8c4c683056edc08060131513a6f7c95d99860e1a63e0b69544259efb3b42e0b0"} Jul 16 16:52:39.394537 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.394511 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696d77865d-2852v" event={"ID":"841c4a67-2843-440e-bdb9-e73cd0b0ac42","Type":"ContainerStarted","Data":"f0948fd3dbe972f4305de23326a4771e59d720472947b426a65b11e9f8f72bc3"} Jul 16 16:52:39.395972 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.395944 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerStarted","Data":"07c6cf1c709086ece804a6e849f3a37d9310a93ada8f2d2547919d282bb7e091"} Jul 16 16:52:39.397739 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.397685 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-p5495" event={"ID":"eff21e0b-f006-4ed0-98ed-12ae223f5149","Type":"ContainerStarted","Data":"1db77dd955804f12161cfa9838dffb841e8f48dcb21afd732953f0e642081a06"} Jul 16 16:52:39.397888 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.397867 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-p5495" Jul 16 16:52:39.399170 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.399145 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" event={"ID":"84e0b799-0e15-4b9c-a6d0-c13fc6b418f4","Type":"ContainerStarted","Data":"48e9fd3d222d8238c1f89a33bbf12668ea38920f89a8382f58070ecd401054a6"} Jul 16 16:52:39.400625 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.400591 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" event={"ID":"7e4c168d-3ee6-4e64-87d0-730dd83180e5","Type":"ContainerStarted","Data":"2d1e635ed95e042a5ccfd9a35b77510813f88a2737ce22a815c6b19f2d2183bc"} Jul 16 16:52:39.401750 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.401725 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" event={"ID":"0bd3adc6-6f09-4128-911b-155204c58cb0","Type":"ContainerStarted","Data":"f087d8fa830f2033a0a2a61d7560ff4686b636399cb05a5aadc5b05eaea83723"} Jul 16 16:52:39.410723 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.410679 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vq2kf" podStartSLOduration=9.897521031 podStartE2EDuration="11.410662615s" totalImagesPullingTime="1.423543453s" totalInitContainerRuntime="89.598131ms" isStatefulPod=true podCreationTimestamp="2026-07-16 16:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:39.409330092 +0000 UTC m=+89.193485154" watchObservedRunningTime="2026-07-16 16:52:39.410662615 +0000 UTC m=+89.194817674" Jul 16 16:52:39.411544 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.411523 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-p5495" Jul 16 16:52:39.432372 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.432314 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-lmxk5" podStartSLOduration=2.6263459559999998 podStartE2EDuration="11.432298995s" totalImagesPullingTime="8.805953039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:39.431075016 +0000 UTC m=+89.215230078" watchObservedRunningTime="2026-07-16 16:52:39.432298995 +0000 UTC m=+89.216454056" Jul 16 16:52:39.449611 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:39.449560 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-p5495" podStartSLOduration=1.756451984 podStartE2EDuration="20.449539223s" totalImagesPullingTime="18.693087239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:39.449287064 +0000 UTC m=+89.233442127" watchObservedRunningTime="2026-07-16 16:52:39.449539223 +0000 UTC m=+89.233694305" Jul 16 16:52:40.416004 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:40.415529 2556 generic.go:356] "Generic (PLEG): container finished" podID="a245bb98-ef13-4f03-b793-53b541bfa380" containerID="16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2" exitCode=0 Jul 16 16:52:40.416464 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:40.416387 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerDied","Data":"16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2"} Jul 16 16:52:40.419359 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:40.419315 2556 generic.go:356] "Generic (PLEG): container finished" podID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerID="44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0" exitCode=0 Jul 16 16:52:40.419759 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:40.419534 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerDied","Data":"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0"} Jul 16 16:52:47.457251 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:47.457176 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerStarted","Data":"ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f"} Jul 16 16:52:47.459187 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:47.459158 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696d77865d-2852v" event={"ID":"841c4a67-2843-440e-bdb9-e73cd0b0ac42","Type":"ContainerStarted","Data":"3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d"} Jul 16 16:52:47.472333 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:47.472305 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" event={"ID":"84e0b799-0e15-4b9c-a6d0-c13fc6b418f4","Type":"ContainerStarted","Data":"3cb6128083c6353c7bd2fa39eaa40bb4cf6de50a39c02c434d25c1dbd964cbff"} Jul 16 16:52:47.473347 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:47.473295 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:47.475209 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:47.475176 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" event={"ID":"7e4c168d-3ee6-4e64-87d0-730dd83180e5","Type":"ContainerStarted","Data":"0dcd8d1f26f2e706bdd3c82a4ef9550c64a1a56fff7884a926505f8075711383"} Jul 16 16:52:47.477288 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:47.476856 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" event={"ID":"0bd3adc6-6f09-4128-911b-155204c58cb0","Type":"ContainerStarted","Data":"0133ec20fdce73a73c7a80297be3916795060b300d7e11aa74f351eed13716f9"} Jul 16 16:52:47.478609 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:47.478560 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" Jul 16 16:52:47.483622 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:47.483552 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-696d77865d-2852v" podStartSLOduration=8.843912976 podStartE2EDuration="17.483534729s" totalImagesPullingTime="8.639621753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:47.481336964 +0000 UTC m=+97.265492027" watchObservedRunningTime="2026-07-16 16:52:47.483534729 +0000 UTC m=+97.267689791" Jul 16 16:52:47.508050 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:47.508006 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" podStartSLOduration=7.542588206 podStartE2EDuration="15.507991911s" totalImagesPullingTime="7.965403705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:47.506822029 +0000 UTC m=+97.290977101" watchObservedRunningTime="2026-07-16 16:52:47.507991911 +0000 UTC m=+97.292146974" Jul 16 16:52:47.525532 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:47.525483 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r8pwq" podStartSLOduration=7.589715447 podStartE2EDuration="14.525463818s" totalImagesPullingTime="6.935748371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:47.524504517 +0000 UTC m=+97.308659597" watchObservedRunningTime="2026-07-16 16:52:47.525463818 +0000 UTC m=+97.309618881" Jul 16 16:52:48.483045 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:48.483019 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" event={"ID":"7e4c168d-3ee6-4e64-87d0-730dd83180e5","Type":"ContainerStarted","Data":"578ef3c2ce58b64da4b429d5c64cea9ba6e3824bc9aea74b15fcae547b93febe"} Jul 16 16:52:48.483371 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:48.483103 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" event={"ID":"7e4c168d-3ee6-4e64-87d0-730dd83180e5","Type":"ContainerStarted","Data":"a2cd678f192577c0e30d74e71caff41b1c2c0bf4403aeb0f38dbff559bd0c651"} Jul 16 16:52:48.485845 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:48.485829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerStarted","Data":"8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785"} Jul 16 16:52:48.485932 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:48.485880 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerStarted","Data":"8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4"} Jul 16 16:52:48.485932 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:48.485910 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerStarted","Data":"5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d"} Jul 16 16:52:48.486004 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:48.485950 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerStarted","Data":"bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe"} Jul 16 16:52:48.488335 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:48.488317 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerStarted","Data":"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271"} Jul 16 16:52:48.488408 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:48.488377 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerStarted","Data":"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef"} Jul 16 16:52:48.488408 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:48.488403 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerStarted","Data":"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2"} Jul 16 16:52:48.488512 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:48.488441 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerStarted","Data":"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73"} Jul 16 16:52:49.495319 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:49.495286 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerStarted","Data":"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e"} Jul 16 16:52:49.495717 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:49.495400 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerStarted","Data":"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9"} Jul 16 16:52:49.497577 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:49.497550 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" event={"ID":"7e4c168d-3ee6-4e64-87d0-730dd83180e5","Type":"ContainerStarted","Data":"dddadda025ab79e5fe0f85530e4b0fa5b64603400e978324de1a33e50a79e8cc"} Jul 16 16:52:49.541019 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:49.540965 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.17186465 podStartE2EDuration="15.540942506s" totalImagesPullingTime="8.321927734s" totalInitContainerRuntime="47.150122ms" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:49.539479957 +0000 UTC m=+99.323635019" watchObservedRunningTime="2026-07-16 16:52:49.540942506 +0000 UTC m=+99.325097567" Jul 16 16:52:50.128404 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:50.128365 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:52:50.504378 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:50.504347 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" event={"ID":"7e4c168d-3ee6-4e64-87d0-730dd83180e5","Type":"ContainerStarted","Data":"3b21b24991af1d48d3b3aefaccb9bf7e3e3c133bfe37dd35451d4dac4e6f48f4"} Jul 16 16:52:50.504802 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:50.504440 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" event={"ID":"7e4c168d-3ee6-4e64-87d0-730dd83180e5","Type":"ContainerStarted","Data":"d41dc09a183f3d295a903a243a57a9d4cd78dce658bbab9b9546e1c991281832"} Jul 16 16:52:50.504802 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:50.504577 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:50.507434 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:50.507392 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerStarted","Data":"7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f"} Jul 16 16:52:50.534292 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:50.534245 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" podStartSLOduration=11.126638366 podStartE2EDuration="20.534230261s" totalImagesPullingTime="9.407591895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:50.53244329 +0000 UTC m=+100.316598351" watchObservedRunningTime="2026-07-16 16:52:50.534230261 +0000 UTC m=+100.318385321" Jul 16 16:52:50.559760 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:50.559698 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.244313641 podStartE2EDuration="21.559675516s" totalImagesPullingTime="17.276646452s" totalInitContainerRuntime="38.715423ms" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:50.558763751 +0000 UTC m=+100.342918811" watchObservedRunningTime="2026-07-16 16:52:50.559675516 +0000 UTC m=+100.343830577" Jul 16 16:52:51.304193 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:51.304157 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:51.304364 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:51.304220 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:51.308864 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:51.308841 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:51.426452 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:51.426423 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-696d77865d-2852v"] Jul 16 16:52:51.515567 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:51.515542 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-696d77865d-2852v" Jul 16 16:52:53.071164 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:53.071132 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:53.071640 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:53.071210 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:52:56.517669 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:56.517642 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6958bc8f99-4sd92" Jul 16 16:52:58.312264 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:52:58.312233 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-stcqc" Jul 16 16:53:13.075757 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:13.075725 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:53:13.079467 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:13.079445 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d78b8c7b8-fvcsb" Jul 16 16:53:17.534882 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.534815 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-696d77865d-2852v" podUID="841c4a67-2843-440e-bdb9-e73cd0b0ac42" containerName="console" containerID="cri-o://3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d" gracePeriod=15 Jul 16 16:53:17.820760 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.820712 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696d77865d-2852v" Jul 16 16:53:17.948653 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.948612 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-service-ca\") pod \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " Jul 16 16:53:17.948839 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.948671 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwbxp\" (UniqueName: \"kubernetes.io/projected/841c4a67-2843-440e-bdb9-e73cd0b0ac42-kube-api-access-cwbxp\") pod \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " Jul 16 16:53:17.948839 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.948716 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-oauth-config\") pod \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " Jul 16 16:53:17.948839 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.948792 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-serving-cert\") pod \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " Jul 16 16:53:17.949023 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.948840 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-oauth-serving-cert\") pod \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " Jul 16 16:53:17.949023 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.948896 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-config\") pod \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\" (UID: \"841c4a67-2843-440e-bdb9-e73cd0b0ac42\") " Jul 16 16:53:17.949214 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.949185 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-service-ca" (OuterVolumeSpecName: "service-ca") pod "841c4a67-2843-440e-bdb9-e73cd0b0ac42" (UID: "841c4a67-2843-440e-bdb9-e73cd0b0ac42"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:17.949293 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.949205 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "841c4a67-2843-440e-bdb9-e73cd0b0ac42" (UID: "841c4a67-2843-440e-bdb9-e73cd0b0ac42"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:17.949378 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.949316 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-config" (OuterVolumeSpecName: "console-config") pod "841c4a67-2843-440e-bdb9-e73cd0b0ac42" (UID: "841c4a67-2843-440e-bdb9-e73cd0b0ac42"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:17.951075 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.951047 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "841c4a67-2843-440e-bdb9-e73cd0b0ac42" (UID: "841c4a67-2843-440e-bdb9-e73cd0b0ac42"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:17.951191 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.951151 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "841c4a67-2843-440e-bdb9-e73cd0b0ac42" (UID: "841c4a67-2843-440e-bdb9-e73cd0b0ac42"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:17.951237 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:17.951193 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841c4a67-2843-440e-bdb9-e73cd0b0ac42-kube-api-access-cwbxp" (OuterVolumeSpecName: "kube-api-access-cwbxp") pod "841c4a67-2843-440e-bdb9-e73cd0b0ac42" (UID: "841c4a67-2843-440e-bdb9-e73cd0b0ac42"). InnerVolumeSpecName "kube-api-access-cwbxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:53:18.050613 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.050531 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:18.050613 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.050561 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbxp\" (UniqueName: \"kubernetes.io/projected/841c4a67-2843-440e-bdb9-e73cd0b0ac42-kube-api-access-cwbxp\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:18.050613 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.050570 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-oauth-config\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:18.050613 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.050579 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-serving-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:18.050613 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.050587 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-oauth-serving-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:18.050613 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.050596 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841c4a67-2843-440e-bdb9-e73cd0b0ac42-console-config\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:18.628609 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.628572 2556 generic.go:356] "Generic (PLEG): container finished" podID="841c4a67-2843-440e-bdb9-e73cd0b0ac42" containerID="3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d" exitCode=0 Jul 16 16:53:18.629041 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.628633 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696d77865d-2852v" Jul 16 16:53:18.629041 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.628654 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696d77865d-2852v" event={"ID":"841c4a67-2843-440e-bdb9-e73cd0b0ac42","Type":"ContainerDied","Data":"3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d"} Jul 16 16:53:18.629041 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.628730 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696d77865d-2852v" event={"ID":"841c4a67-2843-440e-bdb9-e73cd0b0ac42","Type":"ContainerDied","Data":"f0948fd3dbe972f4305de23326a4771e59d720472947b426a65b11e9f8f72bc3"} Jul 16 16:53:18.629041 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.628753 2556 scope.go:117] "RemoveContainer" containerID="3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d" Jul 16 16:53:18.638109 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.638094 2556 scope.go:117] "RemoveContainer" containerID="3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d" Jul 16 16:53:18.638355 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:18.638335 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d\": container with ID starting with 3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d not found: ID does not exist" containerID="3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d" Jul 16 16:53:18.638405 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.638363 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d"} err="failed to get container status \"3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d\": rpc error: code = NotFound desc = could not find container \"3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d\": container with ID starting with 3bd9f1ae5829e3342a83e90e1c7c86ad9f1439b7c358bf7de3972f2aa9298e4d not found: ID does not exist" Jul 16 16:53:18.668718 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.668694 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-696d77865d-2852v"] Jul 16 16:53:18.672181 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.672162 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-696d77865d-2852v"] Jul 16 16:53:18.841579 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:18.841549 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="841c4a67-2843-440e-bdb9-e73cd0b0ac42" path="/var/lib/kubelet/pods/841c4a67-2843-440e-bdb9-e73cd0b0ac42/volumes" Jul 16 16:53:35.129215 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:35.129177 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:35.147909 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:35.147886 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:35.716137 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:35.716112 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:48.845197 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:48.845165 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:53:48.846008 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:48.845471 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="alertmanager" containerID="cri-o://ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f" gracePeriod=120 Jul 16 16:53:48.846008 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:48.845527 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy-web" containerID="cri-o://5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d" gracePeriod=120 Jul 16 16:53:48.846008 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:48.845525 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy-metric" containerID="cri-o://8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785" gracePeriod=120 Jul 16 16:53:48.846008 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:48.845561 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="prom-label-proxy" containerID="cri-o://7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f" gracePeriod=120 Jul 16 16:53:48.846008 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:48.845657 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy" containerID="cri-o://8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4" gracePeriod=120 Jul 16 16:53:48.846008 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:48.845848 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="config-reloader" containerID="cri-o://bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe" gracePeriod=120 Jul 16 16:53:49.758032 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:49.757993 2556 generic.go:356] "Generic (PLEG): container finished" podID="a245bb98-ef13-4f03-b793-53b541bfa380" containerID="7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f" exitCode=0 Jul 16 16:53:49.758032 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:49.758020 2556 generic.go:356] "Generic (PLEG): container finished" podID="a245bb98-ef13-4f03-b793-53b541bfa380" containerID="8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785" exitCode=0 Jul 16 16:53:49.758032 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:49.758030 2556 generic.go:356] "Generic (PLEG): container finished" podID="a245bb98-ef13-4f03-b793-53b541bfa380" containerID="8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4" exitCode=0 Jul 16 16:53:49.758032 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:49.758037 2556 generic.go:356] "Generic (PLEG): container finished" podID="a245bb98-ef13-4f03-b793-53b541bfa380" containerID="bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe" exitCode=0 Jul 16 16:53:49.758288 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:49.758044 2556 generic.go:356] "Generic (PLEG): container finished" podID="a245bb98-ef13-4f03-b793-53b541bfa380" containerID="ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f" exitCode=0 Jul 16 16:53:49.758288 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:49.758064 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerDied","Data":"7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f"} Jul 16 16:53:49.758288 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:49.758129 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerDied","Data":"8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785"} Jul 16 16:53:49.758288 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:49.758154 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerDied","Data":"8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4"} Jul 16 16:53:49.758288 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:49.758179 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerDied","Data":"bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe"} Jul 16 16:53:49.758288 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:49.758206 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerDied","Data":"ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f"} Jul 16 16:53:50.082793 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.082771 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.214003 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.213976 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-metrics-client-ca\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214185 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214013 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214185 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214036 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-config-out\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214185 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214058 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-main-db\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214185 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214090 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc2j9\" (UniqueName: \"kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-kube-api-access-lc2j9\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214185 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214115 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-trusted-ca-bundle\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214185 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214146 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-cluster-tls-config\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214185 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214180 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214583 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214207 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-tls-assets\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214583 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214254 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-web-config\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214583 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214296 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-config-volume\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214583 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214322 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-main-tls\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214583 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214359 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-web\") pod \"a245bb98-ef13-4f03-b793-53b541bfa380\" (UID: \"a245bb98-ef13-4f03-b793-53b541bfa380\") " Jul 16 16:53:50.214583 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214464 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:53:50.214872 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214618 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:50.214872 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214648 2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-main-db\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.214872 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.214766 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:50.217398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.217358 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-config-out" (OuterVolumeSpecName: "config-out") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:53:50.217538 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.217447 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-kube-api-access-lc2j9" (OuterVolumeSpecName: "kube-api-access-lc2j9") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "kube-api-access-lc2j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:53:50.217850 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.217729 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:50.217850 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.217803 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-config-volume" (OuterVolumeSpecName: "config-volume") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:50.217850 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.217819 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:50.218040 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.217956 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:50.218318 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.218298 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:50.218585 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.218561 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:53:50.221154 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.221129 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:50.227927 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.227908 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-web-config" (OuterVolumeSpecName: "web-config") pod "a245bb98-ef13-4f03-b793-53b541bfa380" (UID: "a245bb98-ef13-4f03-b793-53b541bfa380"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:50.315398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315297 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315336 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-tls-assets\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315345 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-web-config\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315354 2556 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315363 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-main-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315371 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315381 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-metrics-client-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315392 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315402 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a245bb98-ef13-4f03-b793-53b541bfa380-config-out\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315437 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc2j9\" (UniqueName: \"kubernetes.io/projected/a245bb98-ef13-4f03-b793-53b541bfa380-kube-api-access-lc2j9\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315843 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315446 2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a245bb98-ef13-4f03-b793-53b541bfa380-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.315843 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.315455 2556 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a245bb98-ef13-4f03-b793-53b541bfa380-cluster-tls-config\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:50.764457 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.764429 2556 generic.go:356] "Generic (PLEG): container finished" podID="a245bb98-ef13-4f03-b793-53b541bfa380" containerID="5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d" exitCode=0 Jul 16 16:53:50.764603 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.764492 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerDied","Data":"5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d"} Jul 16 16:53:50.764603 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.764505 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.764603 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.764547 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a245bb98-ef13-4f03-b793-53b541bfa380","Type":"ContainerDied","Data":"906535864d5c6ce26cfcfda04c2a1bc092c4b7aad3edc39274f10acd2da1de58"} Jul 16 16:53:50.764603 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.764580 2556 scope.go:117] "RemoveContainer" containerID="7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f" Jul 16 16:53:50.772679 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.772660 2556 scope.go:117] "RemoveContainer" containerID="8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785" Jul 16 16:53:50.783004 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.782979 2556 scope.go:117] "RemoveContainer" containerID="8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4" Jul 16 16:53:50.790846 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.790820 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:53:50.792607 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.792593 2556 scope.go:117] "RemoveContainer" containerID="5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d" Jul 16 16:53:50.799705 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.799682 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:53:50.800184 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.800169 2556 scope.go:117] "RemoveContainer" containerID="bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe" Jul 16 16:53:50.807019 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.806997 2556 scope.go:117] "RemoveContainer" containerID="ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f" Jul 16 16:53:50.813752 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.813733 2556 scope.go:117] "RemoveContainer" containerID="16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2" Jul 16 16:53:50.820213 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.820195 2556 scope.go:117] "RemoveContainer" containerID="7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f" Jul 16 16:53:50.820460 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:50.820438 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f\": container with ID starting with 7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f not found: ID does not exist" containerID="7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f" Jul 16 16:53:50.820499 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.820469 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f"} err="failed to get container status \"7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f\": rpc error: code = NotFound desc = could not find container \"7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f\": container with ID starting with 7bfc5066a703db3bf10d30cc24486df87babcc938892041d60a3ef7c7c03253f not found: ID does not exist" Jul 16 16:53:50.820499 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.820489 2556 scope.go:117] "RemoveContainer" containerID="8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785" Jul 16 16:53:50.820706 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:50.820693 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785\": container with ID starting with 8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785 not found: ID does not exist" containerID="8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785" Jul 16 16:53:50.820768 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.820709 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785"} err="failed to get container status \"8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785\": rpc error: code = NotFound desc = could not find container \"8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785\": container with ID starting with 8bbf586594cdcf38e14e85faf32bf471301d343279cf25c23807300376105785 not found: ID does not exist" Jul 16 16:53:50.820768 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.820719 2556 scope.go:117] "RemoveContainer" containerID="8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4" Jul 16 16:53:50.820935 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:50.820916 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4\": container with ID starting with 8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4 not found: ID does not exist" containerID="8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4" Jul 16 16:53:50.820968 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.820941 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4"} err="failed to get container status \"8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4\": rpc error: code = NotFound desc = could not find container \"8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4\": container with ID starting with 8091f67e741cb7ccc00b056a3a322f068e99132b4f89e4323ac4baeb6aecd7d4 not found: ID does not exist" Jul 16 16:53:50.820968 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.820958 2556 scope.go:117] "RemoveContainer" containerID="5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d" Jul 16 16:53:50.821148 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:50.821132 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d\": container with ID starting with 5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d not found: ID does not exist" containerID="5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d" Jul 16 16:53:50.821186 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.821152 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d"} err="failed to get container status \"5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d\": rpc error: code = NotFound desc = could not find container \"5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d\": container with ID starting with 5199ce3a610ccde296cbd566bac8008351ad71ac4cd55f7d398777ff8c63912d not found: ID does not exist" Jul 16 16:53:50.821186 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.821164 2556 scope.go:117] "RemoveContainer" containerID="bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe" Jul 16 16:53:50.821366 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:50.821354 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe\": container with ID starting with bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe not found: ID does not exist" containerID="bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe" Jul 16 16:53:50.821405 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.821369 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe"} err="failed to get container status \"bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe\": rpc error: code = NotFound desc = could not find container \"bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe\": container with ID starting with bd8f550bbdfd0031f2410d38394f1f61b52ea19bf174cc702129bf5378568bfe not found: ID does not exist" Jul 16 16:53:50.821405 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.821379 2556 scope.go:117] "RemoveContainer" containerID="ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f" Jul 16 16:53:50.821620 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:50.821602 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f\": container with ID starting with ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f not found: ID does not exist" containerID="ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f" Jul 16 16:53:50.821715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.821623 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f"} err="failed to get container status \"ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f\": rpc error: code = NotFound desc = could not find container \"ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f\": container with ID starting with ec72752dddf6dca2392a6e0f62e988404c92cd4b50de5422148a49f0a411c65f not found: ID does not exist" Jul 16 16:53:50.821715 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.821636 2556 scope.go:117] "RemoveContainer" containerID="16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2" Jul 16 16:53:50.821882 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:50.821868 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2\": container with ID starting with 16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2 not found: ID does not exist" containerID="16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2" Jul 16 16:53:50.821926 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.821884 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2"} err="failed to get container status \"16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2\": rpc error: code = NotFound desc = could not find container \"16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2\": container with ID starting with 16670de16946ea4f5182978ce9fb9cb8e309013bc0a744b2a0df34efb6ff09a2 not found: ID does not exist" Jul 16 16:53:50.834609 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834584 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:53:50.834910 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834898 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="prom-label-proxy" Jul 16 16:53:50.834949 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834911 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="prom-label-proxy" Jul 16 16:53:50.834949 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834921 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="alertmanager" Jul 16 16:53:50.834949 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834927 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="alertmanager" Jul 16 16:53:50.834949 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834936 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841c4a67-2843-440e-bdb9-e73cd0b0ac42" containerName="console" Jul 16 16:53:50.834949 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834942 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="841c4a67-2843-440e-bdb9-e73cd0b0ac42" containerName="console" Jul 16 16:53:50.834949 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834949 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy-metric" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834955 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy-metric" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834965 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy-web" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834971 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy-web" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834979 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="init-config-reloader" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834984 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="init-config-reloader" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.834993 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="config-reloader" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.835000 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="config-reloader" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.835010 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.835015 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.835067 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="841c4a67-2843-440e-bdb9-e73cd0b0ac42" containerName="console" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.835075 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy-web" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.835080 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="prom-label-proxy" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.835086 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="config-reloader" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.835092 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.835098 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="alertmanager" Jul 16 16:53:50.835133 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.835105 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" containerName="kube-rbac-proxy-metric" Jul 16 16:53:50.838640 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.838621 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.841316 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.841180 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-74cpr\"" Jul 16 16:53:50.841316 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.841192 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 16:53:50.841641 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.841622 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 16:53:50.841769 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.841751 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 16:53:50.841903 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.841649 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 16:53:50.842029 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.841908 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 16:53:50.842097 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.841713 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 16:53:50.842205 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.841967 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 16:53:50.842612 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.842500 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 16:53:50.845853 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.845830 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a245bb98-ef13-4f03-b793-53b541bfa380" path="/var/lib/kubelet/pods/a245bb98-ef13-4f03-b793-53b541bfa380/volumes" Jul 16 16:53:50.847160 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.847140 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 16:53:50.859650 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.859619 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:53:50.920640 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920604 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8tx7\" (UniqueName: \"kubernetes.io/projected/05b2d1f1-275e-4f60-81c6-afc36861060a-kube-api-access-j8tx7\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.920640 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920642 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05b2d1f1-275e-4f60-81c6-afc36861060a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.920872 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920662 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.920872 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920688 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.920872 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920749 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.920872 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920788 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-web-config\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.920872 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920823 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-config-volume\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.920872 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920858 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.921079 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920884 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.921079 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920903 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05b2d1f1-275e-4f60-81c6-afc36861060a-config-out\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.921079 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920971 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/05b2d1f1-275e-4f60-81c6-afc36861060a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.921079 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.920990 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05b2d1f1-275e-4f60-81c6-afc36861060a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:50.921079 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:50.921006 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b2d1f1-275e-4f60-81c6-afc36861060a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022016 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.021921 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-config-volume\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022016 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.021976 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022016 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022010 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022305 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022039 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05b2d1f1-275e-4f60-81c6-afc36861060a-config-out\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022305 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022078 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/05b2d1f1-275e-4f60-81c6-afc36861060a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022305 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022102 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05b2d1f1-275e-4f60-81c6-afc36861060a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022305 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b2d1f1-275e-4f60-81c6-afc36861060a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022305 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022284 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8tx7\" (UniqueName: \"kubernetes.io/projected/05b2d1f1-275e-4f60-81c6-afc36861060a-kube-api-access-j8tx7\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022574 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022315 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05b2d1f1-275e-4f60-81c6-afc36861060a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022574 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022346 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022574 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022378 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022574 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022437 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022574 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022474 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-web-config\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.022574 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.022526 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/05b2d1f1-275e-4f60-81c6-afc36861060a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.023483 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.023460 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05b2d1f1-275e-4f60-81c6-afc36861060a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.023650 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.023631 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b2d1f1-275e-4f60-81c6-afc36861060a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.024919 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.024889 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05b2d1f1-275e-4f60-81c6-afc36861060a-config-out\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.025158 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.025058 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-config-volume\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.025158 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.025146 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-web-config\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.025265 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.025207 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.025496 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.025476 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.025761 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.025739 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.026390 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.026371 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.026610 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.026587 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/05b2d1f1-275e-4f60-81c6-afc36861060a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.026885 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.026868 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05b2d1f1-275e-4f60-81c6-afc36861060a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.032938 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.032917 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8tx7\" (UniqueName: \"kubernetes.io/projected/05b2d1f1-275e-4f60-81c6-afc36861060a-kube-api-access-j8tx7\") pod \"alertmanager-main-0\" (UID: \"05b2d1f1-275e-4f60-81c6-afc36861060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.149985 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.149950 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:53:51.286812 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.286730 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:53:51.289487 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:53:51.289463 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b2d1f1_275e_4f60_81c6_afc36861060a.slice/crio-58fc59f23dafa46a91f3fd979663b4dff023cf3e399f410783c6ab1591849ce9 WatchSource:0}: Error finding container 58fc59f23dafa46a91f3fd979663b4dff023cf3e399f410783c6ab1591849ce9: Status 404 returned error can't find the container with id 58fc59f23dafa46a91f3fd979663b4dff023cf3e399f410783c6ab1591849ce9 Jul 16 16:53:51.769760 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.769725 2556 generic.go:356] "Generic (PLEG): container finished" podID="05b2d1f1-275e-4f60-81c6-afc36861060a" containerID="793cc90bc587e5bd7885a9737a856a00634a14f03c0fab9bc818bc654a2a7555" exitCode=0 Jul 16 16:53:51.769938 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.769812 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"05b2d1f1-275e-4f60-81c6-afc36861060a","Type":"ContainerDied","Data":"793cc90bc587e5bd7885a9737a856a00634a14f03c0fab9bc818bc654a2a7555"} Jul 16 16:53:51.769938 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:51.769912 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"05b2d1f1-275e-4f60-81c6-afc36861060a","Type":"ContainerStarted","Data":"58fc59f23dafa46a91f3fd979663b4dff023cf3e399f410783c6ab1591849ce9"} Jul 16 16:53:52.778471 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.778429 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"05b2d1f1-275e-4f60-81c6-afc36861060a","Type":"ContainerStarted","Data":"5b66edf925579b6e39f199982b5615150db58b93fef848b075967ed0c7767e2d"} Jul 16 16:53:52.778881 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.778526 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"05b2d1f1-275e-4f60-81c6-afc36861060a","Type":"ContainerStarted","Data":"a5b8fe37cddff7634c783c6323070209897b332761ecf339ab5cc05a8c20abbb"} Jul 16 16:53:52.778881 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.778568 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"05b2d1f1-275e-4f60-81c6-afc36861060a","Type":"ContainerStarted","Data":"f2a6b51ecc896e855b23eee5f70dde99c8a8ce6976df065e83e1cdba0192d338"} Jul 16 16:53:52.778881 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.778597 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"05b2d1f1-275e-4f60-81c6-afc36861060a","Type":"ContainerStarted","Data":"837d04c7629d560834824d0a508c2f172426f82b9918cf63d7b7bfaa4e315c15"} Jul 16 16:53:52.778881 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.778619 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"05b2d1f1-275e-4f60-81c6-afc36861060a","Type":"ContainerStarted","Data":"134ec7fb78942c58b60e5c9f80bac8bdc62226bb996d98acfd58f98f2f94386b"} Jul 16 16:53:52.778881 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.778654 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"05b2d1f1-275e-4f60-81c6-afc36861060a","Type":"ContainerStarted","Data":"6dcb5e1cb103e7d3376f2555fc97683952036f6709cbdec69d7748033814ff14"} Jul 16 16:53:52.810257 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.810190 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.793382366 podStartE2EDuration="2.810168435s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.786069ms" isStatefulPod=false podCreationTimestamp="2026-07-16 16:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:53:52.808203239 +0000 UTC m=+162.592358312" watchObservedRunningTime="2026-07-16 16:53:52.810168435 +0000 UTC m=+162.594323497" Jul 16 16:53:52.887616 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.887584 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-846bd64b9c-67cc9"] Jul 16 16:53:52.890951 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.890923 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:52.893295 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.893270 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 16:53:52.893453 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.893315 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vknlp\"" Jul 16 16:53:52.893453 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.893270 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 16:53:52.893675 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.893657 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 16:53:52.893718 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.893696 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 16:53:52.893760 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.893748 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 16:53:52.899274 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.899214 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 16:53:52.903968 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.903945 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-846bd64b9c-67cc9"] Jul 16 16:53:52.941167 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.941127 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-secret-telemeter-client\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:52.941321 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.941175 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6081993-be91-452d-af67-2671d0631574-metrics-client-ca\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:52.941321 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.941239 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgcb\" (UniqueName: \"kubernetes.io/projected/f6081993-be91-452d-af67-2671d0631574-kube-api-access-5rgcb\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:52.941506 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.941323 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6081993-be91-452d-af67-2671d0631574-telemeter-trusted-ca-bundle\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:52.941506 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.941360 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-federate-client-tls\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:52.941506 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.941400 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-telemeter-client-tls\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:52.941506 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.941468 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:52.941649 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:52.941511 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6081993-be91-452d-af67-2671d0631574-serving-certs-ca-bundle\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.043093 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.043004 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.043093 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.043064 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6081993-be91-452d-af67-2671d0631574-serving-certs-ca-bundle\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.043279 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.043093 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-secret-telemeter-client\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.043279 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.043132 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6081993-be91-452d-af67-2671d0631574-metrics-client-ca\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.043279 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.043168 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgcb\" (UniqueName: \"kubernetes.io/projected/f6081993-be91-452d-af67-2671d0631574-kube-api-access-5rgcb\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.043279 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.043222 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6081993-be91-452d-af67-2671d0631574-telemeter-trusted-ca-bundle\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.043279 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.043256 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-federate-client-tls\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.043546 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.043298 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-telemeter-client-tls\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.044248 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.044216 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6081993-be91-452d-af67-2671d0631574-serving-certs-ca-bundle\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.044248 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.044243 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6081993-be91-452d-af67-2671d0631574-metrics-client-ca\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.044454 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.044243 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6081993-be91-452d-af67-2671d0631574-telemeter-trusted-ca-bundle\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.045703 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.045678 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.045855 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.045839 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-telemeter-client-tls\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.046107 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.046086 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-federate-client-tls\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.046159 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.046086 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6081993-be91-452d-af67-2671d0631574-secret-telemeter-client\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.053164 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.053136 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgcb\" (UniqueName: \"kubernetes.io/projected/f6081993-be91-452d-af67-2671d0631574-kube-api-access-5rgcb\") pod \"telemeter-client-846bd64b9c-67cc9\" (UID: \"f6081993-be91-452d-af67-2671d0631574\") " pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.201829 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.201790 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" Jul 16 16:53:53.249184 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.247868 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:53:53.249184 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.248207 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="prometheus" containerID="cri-o://e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73" gracePeriod=600 Jul 16 16:53:53.249184 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.248592 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e" gracePeriod=600 Jul 16 16:53:53.249184 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.248698 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy" containerID="cri-o://18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9" gracePeriod=600 Jul 16 16:53:53.249184 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.248781 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy-web" containerID="cri-o://2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271" gracePeriod=600 Jul 16 16:53:53.249184 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.248883 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="thanos-sidecar" containerID="cri-o://adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef" gracePeriod=600 Jul 16 16:53:53.249184 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.248952 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="config-reloader" containerID="cri-o://57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2" gracePeriod=600 Jul 16 16:53:53.364301 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.364232 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-846bd64b9c-67cc9"] Jul 16 16:53:53.370537 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:53:53.370511 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6081993_be91_452d_af67_2671d0631574.slice/crio-7c652c6c22c25af034074838fe67745f742958c955929f8f9701309b00d2d455 WatchSource:0}: Error finding container 7c652c6c22c25af034074838fe67745f742958c955929f8f9701309b00d2d455: Status 404 returned error can't find the container with id 7c652c6c22c25af034074838fe67745f742958c955929f8f9701309b00d2d455 Jul 16 16:53:53.499829 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.499803 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.648378 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648296 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-thanos-prometheus-http-client-file\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648378 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648351 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-serving-certs-ca-bundle\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648616 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648380 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-0\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648616 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648401 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-web-config\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648616 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648460 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-config-out\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648616 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648485 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-kube-rbac-proxy\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648616 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648508 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-1\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648616 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648563 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-metrics-client-ca\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648616 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648591 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-kubelet-serving-ca-bundle\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648618 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648651 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-grpc-tls\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648700 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-trusted-ca-bundle\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648752 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-tls\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648776 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-config\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648803 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-metrics-client-certs\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648850 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrpcx\" (UniqueName: \"kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-kube-api-access-zrpcx\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648873 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-2\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648906 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-db\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.648946 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648934 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-tls-assets\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.649453 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.648970 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8255767a-f1d8-4d76-8867-9ede2df130cc\" (UID: \"8255767a-f1d8-4d76-8867-9ede2df130cc\") " Jul 16 16:53:53.650479 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.650405 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:53.650611 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.650498 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-config-out" (OuterVolumeSpecName: "config-out") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:53:53.650681 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.650616 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:53.650848 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.650827 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:53.650917 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.650886 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:53.650980 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.650932 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:53.651153 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.651130 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:53.651754 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.651723 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:53.651851 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.651812 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:53.652125 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.651624 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:53.652237 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.652168 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:53:53.652719 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.652472 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:53.652719 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.652510 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-config" (OuterVolumeSpecName: "config") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:53.653292 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.653266 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:53.653489 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.653462 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:53.653893 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.653869 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:53:53.654078 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.654045 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-kube-api-access-zrpcx" (OuterVolumeSpecName: "kube-api-access-zrpcx") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "kube-api-access-zrpcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:53:53.654147 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.654079 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:53.654398 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.654375 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:53.661809 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.661786 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-web-config" (OuterVolumeSpecName: "web-config") pod "8255767a-f1d8-4d76-8867-9ede2df130cc" (UID: "8255767a-f1d8-4d76-8867-9ede2df130cc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:53:53.749776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749742 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.749776 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749772 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749787 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-web-config\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749800 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-config-out\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749811 2556 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-kube-rbac-proxy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749824 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749836 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-metrics-client-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749848 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749860 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749871 2556 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-grpc-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749883 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749895 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749907 2556 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-config\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749919 2556 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-metrics-client-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749931 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrpcx\" (UniqueName: \"kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-kube-api-access-zrpcx\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749944 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749957 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8255767a-f1d8-4d76-8867-9ede2df130cc-prometheus-k8s-db\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749969 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8255767a-f1d8-4d76-8867-9ede2df130cc-tls-assets\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749980 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.750046 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.749992 2556 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8255767a-f1d8-4d76-8867-9ede2df130cc-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 16:53:53.787708 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787672 2556 generic.go:356] "Generic (PLEG): container finished" podID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerID="18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e" exitCode=0 Jul 16 16:53:53.787708 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787698 2556 generic.go:356] "Generic (PLEG): container finished" podID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerID="18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9" exitCode=0 Jul 16 16:53:53.787708 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787707 2556 generic.go:356] "Generic (PLEG): container finished" podID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerID="2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271" exitCode=0 Jul 16 16:53:53.787708 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787701 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerDied","Data":"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e"} Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787766 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787787 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerDied","Data":"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9"} Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787716 2556 generic.go:356] "Generic (PLEG): container finished" podID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerID="adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef" exitCode=0 Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787810 2556 generic.go:356] "Generic (PLEG): container finished" podID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerID="57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2" exitCode=0 Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787820 2556 generic.go:356] "Generic (PLEG): container finished" podID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerID="e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73" exitCode=0 Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787831 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerDied","Data":"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271"} Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787869 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerDied","Data":"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef"} Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787884 2556 scope.go:117] "RemoveContainer" containerID="18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e" Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787912 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerDied","Data":"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2"} Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.787953 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerDied","Data":"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73"} Jul 16 16:53:53.788245 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.788015 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8255767a-f1d8-4d76-8867-9ede2df130cc","Type":"ContainerDied","Data":"07c6cf1c709086ece804a6e849f3a37d9310a93ada8f2d2547919d282bb7e091"} Jul 16 16:53:53.789586 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.789556 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" event={"ID":"f6081993-be91-452d-af67-2671d0631574","Type":"ContainerStarted","Data":"7c652c6c22c25af034074838fe67745f742958c955929f8f9701309b00d2d455"} Jul 16 16:53:53.802104 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.802082 2556 scope.go:117] "RemoveContainer" containerID="18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9" Jul 16 16:53:53.810480 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.810456 2556 scope.go:117] "RemoveContainer" containerID="2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271" Jul 16 16:53:53.818529 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.818506 2556 scope.go:117] "RemoveContainer" containerID="adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef" Jul 16 16:53:53.818613 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.818570 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:53:53.825495 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.825470 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:53:53.827509 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.827487 2556 scope.go:117] "RemoveContainer" containerID="57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2" Jul 16 16:53:53.834775 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.834757 2556 scope.go:117] "RemoveContainer" containerID="e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73" Jul 16 16:53:53.842824 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.842810 2556 scope.go:117] "RemoveContainer" containerID="44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0" Jul 16 16:53:53.849355 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.849338 2556 scope.go:117] "RemoveContainer" containerID="18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e" Jul 16 16:53:53.849610 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:53.849592 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": container with ID starting with 18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e not found: ID does not exist" containerID="18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e" Jul 16 16:53:53.849657 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.849619 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e"} err="failed to get container status \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": rpc error: code = NotFound desc = could not find container \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": container with ID starting with 18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e not found: ID does not exist" Jul 16 16:53:53.849657 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.849644 2556 scope.go:117] "RemoveContainer" containerID="18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9" Jul 16 16:53:53.849860 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:53.849843 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": container with ID starting with 18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9 not found: ID does not exist" containerID="18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9" Jul 16 16:53:53.849906 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.849866 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9"} err="failed to get container status \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": rpc error: code = NotFound desc = could not find container \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": container with ID starting with 18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9 not found: ID does not exist" Jul 16 16:53:53.849906 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.849883 2556 scope.go:117] "RemoveContainer" containerID="2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271" Jul 16 16:53:53.850104 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:53.850089 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": container with ID starting with 2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271 not found: ID does not exist" containerID="2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271" Jul 16 16:53:53.850149 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.850108 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271"} err="failed to get container status \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": rpc error: code = NotFound desc = could not find container \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": container with ID starting with 2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271 not found: ID does not exist" Jul 16 16:53:53.850149 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.850120 2556 scope.go:117] "RemoveContainer" containerID="adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef" Jul 16 16:53:53.850314 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:53.850301 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": container with ID starting with adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef not found: ID does not exist" containerID="adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef" Jul 16 16:53:53.850363 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.850317 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef"} err="failed to get container status \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": rpc error: code = NotFound desc = could not find container \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": container with ID starting with adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef not found: ID does not exist" Jul 16 16:53:53.850363 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.850327 2556 scope.go:117] "RemoveContainer" containerID="57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2" Jul 16 16:53:53.850577 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:53.850553 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": container with ID starting with 57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2 not found: ID does not exist" containerID="57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2" Jul 16 16:53:53.850623 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.850588 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2"} err="failed to get container status \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": rpc error: code = NotFound desc = could not find container \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": container with ID starting with 57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2 not found: ID does not exist" Jul 16 16:53:53.850623 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.850609 2556 scope.go:117] "RemoveContainer" containerID="e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73" Jul 16 16:53:53.850852 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:53.850820 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": container with ID starting with e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73 not found: ID does not exist" containerID="e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73" Jul 16 16:53:53.850852 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.850845 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73"} err="failed to get container status \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": rpc error: code = NotFound desc = could not find container \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": container with ID starting with e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73 not found: ID does not exist" Jul 16 16:53:53.850981 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.850861 2556 scope.go:117] "RemoveContainer" containerID="44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0" Jul 16 16:53:53.851146 ip-10-0-133-149 kubenswrapper[2556]: E0716 16:53:53.851118 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": container with ID starting with 44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0 not found: ID does not exist" containerID="44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0" Jul 16 16:53:53.851279 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.851153 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0"} err="failed to get container status \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": rpc error: code = NotFound desc = could not find container \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": container with ID starting with 44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0 not found: ID does not exist" Jul 16 16:53:53.851279 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.851173 2556 scope.go:117] "RemoveContainer" containerID="18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e" Jul 16 16:53:53.851564 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.851517 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e"} err="failed to get container status \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": rpc error: code = NotFound desc = could not find container \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": container with ID starting with 18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e not found: ID does not exist" Jul 16 16:53:53.851564 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.851540 2556 scope.go:117] "RemoveContainer" containerID="18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9" Jul 16 16:53:53.851823 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.851802 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9"} err="failed to get container status \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": rpc error: code = NotFound desc = could not find container \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": container with ID starting with 18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9 not found: ID does not exist" Jul 16 16:53:53.851895 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.851825 2556 scope.go:117] "RemoveContainer" containerID="2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271" Jul 16 16:53:53.852087 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.852068 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271"} err="failed to get container status \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": rpc error: code = NotFound desc = could not find container \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": container with ID starting with 2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271 not found: ID does not exist" Jul 16 16:53:53.852158 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.852093 2556 scope.go:117] "RemoveContainer" containerID="adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef" Jul 16 16:53:53.852773 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.852731 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef"} err="failed to get container status \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": rpc error: code = NotFound desc = could not find container \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": container with ID starting with adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef not found: ID does not exist" Jul 16 16:53:53.852773 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.852773 2556 scope.go:117] "RemoveContainer" containerID="57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2" Jul 16 16:53:53.853223 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.853194 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2"} err="failed to get container status \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": rpc error: code = NotFound desc = could not find container \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": container with ID starting with 57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2 not found: ID does not exist" Jul 16 16:53:53.853309 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.853224 2556 scope.go:117] "RemoveContainer" containerID="e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73" Jul 16 16:53:53.853566 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.853542 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73"} err="failed to get container status \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": rpc error: code = NotFound desc = could not find container \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": container with ID starting with e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73 not found: ID does not exist" Jul 16 16:53:53.853656 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.853569 2556 scope.go:117] "RemoveContainer" containerID="44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0" Jul 16 16:53:53.853920 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.853815 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0"} err="failed to get container status \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": rpc error: code = NotFound desc = could not find container \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": container with ID starting with 44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0 not found: ID does not exist" Jul 16 16:53:53.853920 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.853842 2556 scope.go:117] "RemoveContainer" containerID="18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e" Jul 16 16:53:53.854151 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.854115 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e"} err="failed to get container status \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": rpc error: code = NotFound desc = could not find container \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": container with ID starting with 18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e not found: ID does not exist" Jul 16 16:53:53.854151 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.854135 2556 scope.go:117] "RemoveContainer" containerID="18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9" Jul 16 16:53:53.854374 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.854355 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9"} err="failed to get container status \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": rpc error: code = NotFound desc = could not find container \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": container with ID starting with 18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9 not found: ID does not exist" Jul 16 16:53:53.854462 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.854374 2556 scope.go:117] "RemoveContainer" containerID="2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271" Jul 16 16:53:53.854614 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.854591 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271"} err="failed to get container status \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": rpc error: code = NotFound desc = could not find container \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": container with ID starting with 2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271 not found: ID does not exist" Jul 16 16:53:53.854689 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.854616 2556 scope.go:117] "RemoveContainer" containerID="adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef" Jul 16 16:53:53.854899 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.854875 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef"} err="failed to get container status \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": rpc error: code = NotFound desc = could not find container \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": container with ID starting with adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef not found: ID does not exist" Jul 16 16:53:53.854975 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.854902 2556 scope.go:117] "RemoveContainer" containerID="57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2" Jul 16 16:53:53.855157 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.855139 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2"} err="failed to get container status \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": rpc error: code = NotFound desc = could not find container \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": container with ID starting with 57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2 not found: ID does not exist" Jul 16 16:53:53.855157 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.855157 2556 scope.go:117] "RemoveContainer" containerID="e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73" Jul 16 16:53:53.855540 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.855516 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73"} err="failed to get container status \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": rpc error: code = NotFound desc = could not find container \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": container with ID starting with e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73 not found: ID does not exist" Jul 16 16:53:53.855626 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.855543 2556 scope.go:117] "RemoveContainer" containerID="44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0" Jul 16 16:53:53.855803 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.855779 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0"} err="failed to get container status \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": rpc error: code = NotFound desc = could not find container \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": container with ID starting with 44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0 not found: ID does not exist" Jul 16 16:53:53.855860 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.855806 2556 scope.go:117] "RemoveContainer" containerID="18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e" Jul 16 16:53:53.855928 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.855907 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:53:53.856113 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856094 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e"} err="failed to get container status \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": rpc error: code = NotFound desc = could not find container \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": container with ID starting with 18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e not found: ID does not exist" Jul 16 16:53:53.856179 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856116 2556 scope.go:117] "RemoveContainer" containerID="18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9" Jul 16 16:53:53.856355 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856321 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9"} err="failed to get container status \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": rpc error: code = NotFound desc = could not find container \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": container with ID starting with 18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9 not found: ID does not exist" Jul 16 16:53:53.856355 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856348 2556 scope.go:117] "RemoveContainer" containerID="2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271" Jul 16 16:53:53.856626 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856559 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy-thanos" Jul 16 16:53:53.856626 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856579 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy-thanos" Jul 16 16:53:53.856626 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856596 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="prometheus" Jul 16 16:53:53.856626 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856604 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="prometheus" Jul 16 16:53:53.856626 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856602 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271"} err="failed to get container status \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": rpc error: code = NotFound desc = could not find container \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": container with ID starting with 2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271 not found: ID does not exist" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856632 2556 scope.go:117] "RemoveContainer" containerID="adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856612 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy-web" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856681 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy-web" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856709 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="config-reloader" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856719 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="config-reloader" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856729 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="init-config-reloader" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856734 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="init-config-reloader" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856752 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856759 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856769 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="thanos-sidecar" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856774 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="thanos-sidecar" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856844 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856854 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy-thanos" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856861 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="thanos-sidecar" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856866 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="prometheus" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856874 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="kube-rbac-proxy-web" Jul 16 16:53:53.856916 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856883 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" containerName="config-reloader" Jul 16 16:53:53.857769 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856928 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef"} err="failed to get container status \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": rpc error: code = NotFound desc = could not find container \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": container with ID starting with adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef not found: ID does not exist" Jul 16 16:53:53.857769 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.856950 2556 scope.go:117] "RemoveContainer" containerID="57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2" Jul 16 16:53:53.857769 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.857185 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2"} err="failed to get container status \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": rpc error: code = NotFound desc = could not find container \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": container with ID starting with 57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2 not found: ID does not exist" Jul 16 16:53:53.857769 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.857207 2556 scope.go:117] "RemoveContainer" containerID="e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73" Jul 16 16:53:53.857769 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.857485 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73"} err="failed to get container status \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": rpc error: code = NotFound desc = could not find container \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": container with ID starting with e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73 not found: ID does not exist" Jul 16 16:53:53.857769 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.857520 2556 scope.go:117] "RemoveContainer" containerID="44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0" Jul 16 16:53:53.858017 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.857787 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0"} err="failed to get container status \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": rpc error: code = NotFound desc = could not find container \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": container with ID starting with 44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0 not found: ID does not exist" Jul 16 16:53:53.858017 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.857808 2556 scope.go:117] "RemoveContainer" containerID="18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e" Jul 16 16:53:53.858399 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.858243 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e"} err="failed to get container status \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": rpc error: code = NotFound desc = could not find container \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": container with ID starting with 18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e not found: ID does not exist" Jul 16 16:53:53.858399 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.858272 2556 scope.go:117] "RemoveContainer" containerID="18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9" Jul 16 16:53:53.858709 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.858584 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9"} err="failed to get container status \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": rpc error: code = NotFound desc = could not find container \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": container with ID starting with 18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9 not found: ID does not exist" Jul 16 16:53:53.858709 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.858610 2556 scope.go:117] "RemoveContainer" containerID="2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271" Jul 16 16:53:53.858963 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.858910 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271"} err="failed to get container status \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": rpc error: code = NotFound desc = could not find container \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": container with ID starting with 2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271 not found: ID does not exist" Jul 16 16:53:53.858963 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.858937 2556 scope.go:117] "RemoveContainer" containerID="adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef" Jul 16 16:53:53.859163 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.859143 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef"} err="failed to get container status \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": rpc error: code = NotFound desc = could not find container \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": container with ID starting with adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef not found: ID does not exist" Jul 16 16:53:53.859163 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.859162 2556 scope.go:117] "RemoveContainer" containerID="57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2" Jul 16 16:53:53.859467 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.859437 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2"} err="failed to get container status \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": rpc error: code = NotFound desc = could not find container \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": container with ID starting with 57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2 not found: ID does not exist" Jul 16 16:53:53.859467 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.859460 2556 scope.go:117] "RemoveContainer" containerID="e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73" Jul 16 16:53:53.859744 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.859694 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73"} err="failed to get container status \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": rpc error: code = NotFound desc = could not find container \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": container with ID starting with e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73 not found: ID does not exist" Jul 16 16:53:53.859744 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.859722 2556 scope.go:117] "RemoveContainer" containerID="44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0" Jul 16 16:53:53.860000 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.859983 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0"} err="failed to get container status \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": rpc error: code = NotFound desc = could not find container \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": container with ID starting with 44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0 not found: ID does not exist" Jul 16 16:53:53.860055 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.860001 2556 scope.go:117] "RemoveContainer" containerID="18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e" Jul 16 16:53:53.860296 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.860278 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e"} err="failed to get container status \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": rpc error: code = NotFound desc = could not find container \"18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e\": container with ID starting with 18919b5022f801677d2d75d935cc2bd74def6c1c44a5ace1ff4c265fabf54b0e not found: ID does not exist" Jul 16 16:53:53.860374 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.860348 2556 scope.go:117] "RemoveContainer" containerID="18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9" Jul 16 16:53:53.860666 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.860645 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9"} err="failed to get container status \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": rpc error: code = NotFound desc = could not find container \"18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9\": container with ID starting with 18f36967e8ce15282527dee2e67563cd2326218fb0ce9a15a57f80907aa32fd9 not found: ID does not exist" Jul 16 16:53:53.860798 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.860667 2556 scope.go:117] "RemoveContainer" containerID="2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271" Jul 16 16:53:53.860918 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.860896 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271"} err="failed to get container status \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": rpc error: code = NotFound desc = could not find container \"2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271\": container with ID starting with 2d85a128b3a8650d587e48d14c3b02335dcb664d78c8e9ab3652c4cddc539271 not found: ID does not exist" Jul 16 16:53:53.860979 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.860921 2556 scope.go:117] "RemoveContainer" containerID="adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef" Jul 16 16:53:53.861202 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.861172 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef"} err="failed to get container status \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": rpc error: code = NotFound desc = could not find container \"adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef\": container with ID starting with adde6652204369e8336c4c414c8ca2f5a404235bc858a99092382164385208ef not found: ID does not exist" Jul 16 16:53:53.861202 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.861201 2556 scope.go:117] "RemoveContainer" containerID="57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2" Jul 16 16:53:53.861374 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.861338 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.861494 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.861473 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2"} err="failed to get container status \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": rpc error: code = NotFound desc = could not find container \"57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2\": container with ID starting with 57046abdd55a2e2e5b7bb59778324ea2c8528ff7a255f49bfa75d82118622dd2 not found: ID does not exist" Jul 16 16:53:53.861533 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.861497 2556 scope.go:117] "RemoveContainer" containerID="e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73" Jul 16 16:53:53.862050 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.861918 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73"} err="failed to get container status \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": rpc error: code = NotFound desc = could not find container \"e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73\": container with ID starting with e8213c09914058084b1fec5f67c7c99ce24cc38780c9829017688ce04f4d9c73 not found: ID does not exist" Jul 16 16:53:53.862050 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.861945 2556 scope.go:117] "RemoveContainer" containerID="44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0" Jul 16 16:53:53.862308 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.862281 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0"} err="failed to get container status \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": rpc error: code = NotFound desc = could not find container \"44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0\": container with ID starting with 44f311bb601d84e3de766b36bc5148f07a0fbbcb00073b2679760897a2b1bbf0 not found: ID does not exist" Jul 16 16:53:53.863707 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.863665 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 16:53:53.864138 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.863725 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 16:53:53.864138 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.863725 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 16:53:53.864138 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.863998 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 16:53:53.864456 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.864356 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 16:53:53.864456 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.864368 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 16:53:53.864456 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.864402 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 16:53:53.864745 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.864729 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 16:53:53.864878 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.864748 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 16:53:53.864878 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.864794 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 16:53:53.864878 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.864808 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2pqm43lh4gpae\"" Jul 16 16:53:53.864878 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.864730 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 16:53:53.864878 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.864734 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 16:53:53.865441 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.865399 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sqq2l\"" Jul 16 16:53:53.867446 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.867406 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 16:53:53.870407 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.870382 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 16:53:53.879045 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.879027 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:53:53.951688 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.951603 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.951688 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.951651 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-config\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.951688 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.951677 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7de148e9-e075-423d-9099-42add4521e2a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952015 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.951780 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952015 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.951813 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952015 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.951845 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952015 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.951900 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stzj\" (UniqueName: \"kubernetes.io/projected/7de148e9-e075-423d-9099-42add4521e2a-kube-api-access-2stzj\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952015 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.951934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-web-config\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952015 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.951981 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7de148e9-e075-423d-9099-42add4521e2a-config-out\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952283 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952015 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952283 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952048 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952283 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952088 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952283 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952118 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952283 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952147 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952283 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952172 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952283 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952198 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952283 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952237 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952283 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952261 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952595 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952312 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:53.952595 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:53.952337 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.053722 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.053766 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.053795 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.053823 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stzj\" (UniqueName: \"kubernetes.io/projected/7de148e9-e075-423d-9099-42add4521e2a-kube-api-access-2stzj\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.053851 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-web-config\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.053875 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7de148e9-e075-423d-9099-42add4521e2a-config-out\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.053900 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.053927 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.053953 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.053976 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054015 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054039 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054064 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054102 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054136 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054162 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.054400 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054194 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.055287 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054234 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.055287 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054272 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-config\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.055287 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054297 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7de148e9-e075-423d-9099-42add4521e2a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.055287 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.054839 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.055503 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.055456 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.059739 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.058302 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.059739 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.058857 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.059739 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.059020 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.059739 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.059691 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.059739 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.059724 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7de148e9-e075-423d-9099-42add4521e2a-config-out\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.060051 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.059851 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.060536 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.060496 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7de148e9-e075-423d-9099-42add4521e2a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.061502 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.060908 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.061502 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.061453 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7de148e9-e075-423d-9099-42add4521e2a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.062433 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.062039 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-web-config\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.062433 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.062095 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.062433 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.062134 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.062433 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.062237 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.062622 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.062434 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-config\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.062675 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.062654 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.063591 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.063561 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.064083 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.064065 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7de148e9-e075-423d-9099-42add4521e2a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.071429 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.071400 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stzj\" (UniqueName: \"kubernetes.io/projected/7de148e9-e075-423d-9099-42add4521e2a-kube-api-access-2stzj\") pod \"prometheus-k8s-0\" (UID: \"7de148e9-e075-423d-9099-42add4521e2a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.173747 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.173694 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:53:54.342952 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.342783 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:53:54.759764 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:53:54.759733 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de148e9_e075_423d_9099_42add4521e2a.slice/crio-0ffbbc93efbbbf81969b2e387cecef13a85f510bb6a60e0d61fcb885ada8a560 WatchSource:0}: Error finding container 0ffbbc93efbbbf81969b2e387cecef13a85f510bb6a60e0d61fcb885ada8a560: Status 404 returned error can't find the container with id 0ffbbc93efbbbf81969b2e387cecef13a85f510bb6a60e0d61fcb885ada8a560 Jul 16 16:53:54.794075 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.794050 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de148e9-e075-423d-9099-42add4521e2a","Type":"ContainerStarted","Data":"0ffbbc93efbbbf81969b2e387cecef13a85f510bb6a60e0d61fcb885ada8a560"} Jul 16 16:53:54.842048 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:54.842002 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8255767a-f1d8-4d76-8867-9ede2df130cc" path="/var/lib/kubelet/pods/8255767a-f1d8-4d76-8867-9ede2df130cc/volumes" Jul 16 16:53:55.802855 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:55.802812 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" event={"ID":"f6081993-be91-452d-af67-2671d0631574","Type":"ContainerStarted","Data":"a40928a374b9b9ac1c8382ed5333873a22feee05fdda25f8b844f567c356baa2"} Jul 16 16:53:55.803307 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:55.802888 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" event={"ID":"f6081993-be91-452d-af67-2671d0631574","Type":"ContainerStarted","Data":"a39d7276b1782cc8df949fa007949574a52fc5fba6831184a2da0c31f18e7c14"} Jul 16 16:53:55.803307 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:55.802915 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" event={"ID":"f6081993-be91-452d-af67-2671d0631574","Type":"ContainerStarted","Data":"237fe41342b21efaaca0e4146b61148e5d6fd43d31e4ba0c73192eb9f8edbd18"} Jul 16 16:53:55.804320 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:55.804292 2556 generic.go:356] "Generic (PLEG): container finished" podID="7de148e9-e075-423d-9099-42add4521e2a" containerID="8fef37f52f6b27048e976036cbb3eabd6a66290595e24e3e12cb85f7433a411b" exitCode=0 Jul 16 16:53:55.804454 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:55.804329 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de148e9-e075-423d-9099-42add4521e2a","Type":"ContainerDied","Data":"8fef37f52f6b27048e976036cbb3eabd6a66290595e24e3e12cb85f7433a411b"} Jul 16 16:53:55.854401 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:55.854356 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-846bd64b9c-67cc9" podStartSLOduration=2.441871126 podStartE2EDuration="3.854339068s" totalImagesPullingTime="1.412467942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:53:55.825364324 +0000 UTC m=+165.609519384" watchObservedRunningTime="2026-07-16 16:53:55.854339068 +0000 UTC m=+165.638494128" Jul 16 16:53:56.812218 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:56.812177 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de148e9-e075-423d-9099-42add4521e2a","Type":"ContainerStarted","Data":"6ddbcae93a7b3c65e1d205029ed928e7d7ea7f10b512d5addee7f8ff1415fd4a"} Jul 16 16:53:56.812580 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:56.812252 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de148e9-e075-423d-9099-42add4521e2a","Type":"ContainerStarted","Data":"75b35c4ac9b62179976b7260d806f4c4d01d456d15a9d013ad56a5a8ea86bb97"} Jul 16 16:53:56.812580 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:56.812281 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de148e9-e075-423d-9099-42add4521e2a","Type":"ContainerStarted","Data":"2fd2c734ce19f756c163f0a0bf35884f2365a8ff07fe3fd1b24cbd925c62166a"} Jul 16 16:53:56.812580 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:56.812312 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de148e9-e075-423d-9099-42add4521e2a","Type":"ContainerStarted","Data":"3643fa89881de609cf40eb7d07b3af479c863a048f0e8d6c7e18288237158883"} Jul 16 16:53:56.812580 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:56.812337 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de148e9-e075-423d-9099-42add4521e2a","Type":"ContainerStarted","Data":"fa883ddb8e9f92bca95ec54b7e8a8f20b71e9cbc6e2214a70b74cfb3795e29ba"} Jul 16 16:53:56.812580 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:56.812360 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de148e9-e075-423d-9099-42add4521e2a","Type":"ContainerStarted","Data":"a94e1bd68464501082b4ce2f8fb282beb62b77d05ad085dc2252b42f2f193900"} Jul 16 16:53:56.843714 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:56.843664 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.816906285 podStartE2EDuration="3.843643554s" totalImagesPullingTime="0s" totalInitContainerRuntime="26.737269ms" isStatefulPod=false podCreationTimestamp="2026-07-16 16:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:53:56.842382451 +0000 UTC m=+166.626537511" watchObservedRunningTime="2026-07-16 16:53:56.843643554 +0000 UTC m=+166.627798616" Jul 16 16:53:59.174294 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:53:59.174248 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:54:54.174790 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:54:54.174754 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:54:54.190208 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:54:54.190178 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:54:55.055561 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:54:55.055532 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:56:10.681352 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:10.681321 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 16:56:10.685615 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:10.685592 2556 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 16:56:10.688948 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:10.688931 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 16:56:46.846292 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:46.846254 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf"] Jul 16 16:56:46.849594 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:46.849573 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" Jul 16 16:56:46.853326 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:46.853304 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 16:56:46.854121 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:46.854103 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-r5kmx\"" Jul 16 16:56:46.854199 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:46.854130 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 16:56:46.861568 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:46.861540 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf"] Jul 16 16:56:46.925440 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:46.925381 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjjgh\" (UniqueName: \"kubernetes.io/projected/30e72cfd-bca4-4bc4-85a4-8b06d0c44135-kube-api-access-kjjgh\") pod \"cert-manager-cainjector-559cdd8c87-4wvmf\" (UID: \"30e72cfd-bca4-4bc4-85a4-8b06d0c44135\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" Jul 16 16:56:46.925606 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:46.925514 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30e72cfd-bca4-4bc4-85a4-8b06d0c44135-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4wvmf\" (UID: \"30e72cfd-bca4-4bc4-85a4-8b06d0c44135\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" Jul 16 16:56:47.026395 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:47.026361 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30e72cfd-bca4-4bc4-85a4-8b06d0c44135-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4wvmf\" (UID: \"30e72cfd-bca4-4bc4-85a4-8b06d0c44135\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" Jul 16 16:56:47.026585 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:47.026440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjjgh\" (UniqueName: \"kubernetes.io/projected/30e72cfd-bca4-4bc4-85a4-8b06d0c44135-kube-api-access-kjjgh\") pod \"cert-manager-cainjector-559cdd8c87-4wvmf\" (UID: \"30e72cfd-bca4-4bc4-85a4-8b06d0c44135\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" Jul 16 16:56:47.037676 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:47.037653 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30e72cfd-bca4-4bc4-85a4-8b06d0c44135-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4wvmf\" (UID: \"30e72cfd-bca4-4bc4-85a4-8b06d0c44135\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" Jul 16 16:56:47.037792 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:47.037660 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjjgh\" (UniqueName: \"kubernetes.io/projected/30e72cfd-bca4-4bc4-85a4-8b06d0c44135-kube-api-access-kjjgh\") pod \"cert-manager-cainjector-559cdd8c87-4wvmf\" (UID: \"30e72cfd-bca4-4bc4-85a4-8b06d0c44135\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" Jul 16 16:56:47.167469 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:47.167351 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" Jul 16 16:56:47.310113 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:47.310091 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf"] Jul 16 16:56:47.312549 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:56:47.312519 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30e72cfd_bca4_4bc4_85a4_8b06d0c44135.slice/crio-2d53cfe52246eb9390b0fb51adac2aaae872868db0439b06e4d9ba165d40a097 WatchSource:0}: Error finding container 2d53cfe52246eb9390b0fb51adac2aaae872868db0439b06e4d9ba165d40a097: Status 404 returned error can't find the container with id 2d53cfe52246eb9390b0fb51adac2aaae872868db0439b06e4d9ba165d40a097 Jul 16 16:56:47.314687 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:47.314667 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 16:56:47.487688 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:47.487655 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" event={"ID":"30e72cfd-bca4-4bc4-85a4-8b06d0c44135","Type":"ContainerStarted","Data":"2d53cfe52246eb9390b0fb51adac2aaae872868db0439b06e4d9ba165d40a097"} Jul 16 16:56:50.502551 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:50.502515 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" event={"ID":"30e72cfd-bca4-4bc4-85a4-8b06d0c44135","Type":"ContainerStarted","Data":"a311765de95e2ff8ceb90b368d77536662006344e0d3c0e47b856c89b27d524a"} Jul 16 16:56:50.526160 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:50.526115 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4wvmf" podStartSLOduration=1.701360534 podStartE2EDuration="4.526095672s" totalImagesPullingTime="2.824735138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:56:50.52487692 +0000 UTC m=+340.309031982" watchObservedRunningTime="2026-07-16 16:56:50.526095672 +0000 UTC m=+340.310250733" Jul 16 16:56:56.493907 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.493871 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw"] Jul 16 16:56:56.496795 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.496778 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" Jul 16 16:56:56.502984 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.502958 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 16:56:56.503674 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.503655 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-8277q\"" Jul 16 16:56:56.503743 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.503683 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 16:56:56.520126 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.520105 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw"] Jul 16 16:56:56.611472 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.611436 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdjbm\" (UniqueName: \"kubernetes.io/projected/ae77840b-dc97-4a24-993c-7fdc9a0727e8-kube-api-access-mdjbm\") pod \"openshift-lws-operator-fd8ccff4c-d78pw\" (UID: \"ae77840b-dc97-4a24-993c-7fdc9a0727e8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" Jul 16 16:56:56.611626 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.611502 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae77840b-dc97-4a24-993c-7fdc9a0727e8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d78pw\" (UID: \"ae77840b-dc97-4a24-993c-7fdc9a0727e8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" Jul 16 16:56:56.712039 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.712003 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdjbm\" (UniqueName: \"kubernetes.io/projected/ae77840b-dc97-4a24-993c-7fdc9a0727e8-kube-api-access-mdjbm\") pod \"openshift-lws-operator-fd8ccff4c-d78pw\" (UID: \"ae77840b-dc97-4a24-993c-7fdc9a0727e8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" Jul 16 16:56:56.712204 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.712069 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae77840b-dc97-4a24-993c-7fdc9a0727e8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d78pw\" (UID: \"ae77840b-dc97-4a24-993c-7fdc9a0727e8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" Jul 16 16:56:56.712510 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.712492 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae77840b-dc97-4a24-993c-7fdc9a0727e8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d78pw\" (UID: \"ae77840b-dc97-4a24-993c-7fdc9a0727e8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" Jul 16 16:56:56.723890 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.723841 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdjbm\" (UniqueName: \"kubernetes.io/projected/ae77840b-dc97-4a24-993c-7fdc9a0727e8-kube-api-access-mdjbm\") pod \"openshift-lws-operator-fd8ccff4c-d78pw\" (UID: \"ae77840b-dc97-4a24-993c-7fdc9a0727e8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" Jul 16 16:56:56.805700 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.805610 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" Jul 16 16:56:56.946722 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:56.946678 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw"] Jul 16 16:56:56.951246 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:56:56.951219 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae77840b_dc97_4a24_993c_7fdc9a0727e8.slice/crio-7000963bd64671aa53121e1879540ac0742622627ea58144cc0f07b79cd8c61d WatchSource:0}: Error finding container 7000963bd64671aa53121e1879540ac0742622627ea58144cc0f07b79cd8c61d: Status 404 returned error can't find the container with id 7000963bd64671aa53121e1879540ac0742622627ea58144cc0f07b79cd8c61d Jul 16 16:56:57.532868 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:56:57.532826 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" event={"ID":"ae77840b-dc97-4a24-993c-7fdc9a0727e8","Type":"ContainerStarted","Data":"7000963bd64671aa53121e1879540ac0742622627ea58144cc0f07b79cd8c61d"} Jul 16 16:57:00.549027 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:00.548991 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" event={"ID":"ae77840b-dc97-4a24-993c-7fdc9a0727e8","Type":"ContainerStarted","Data":"6f00f9df18ea6fb2bd23ba1e5f0dee02575d07940016cba87bde0826a46b9da4"} Jul 16 16:57:00.568912 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:00.568854 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d78pw" podStartSLOduration=2.001351414 podStartE2EDuration="4.568840987s" totalImagesPullingTime="2.567489573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:00.567598837 +0000 UTC m=+350.351753901" watchObservedRunningTime="2026-07-16 16:57:00.568840987 +0000 UTC m=+350.352996049" Jul 16 16:57:16.550597 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.550518 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd"] Jul 16 16:57:16.554130 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.554108 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.556756 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.556737 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 16:57:16.556883 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.556866 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sdg7k\"" Jul 16 16:57:16.557163 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.557149 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 16:57:16.557247 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.557151 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 16:57:16.557571 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.557553 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 16:57:16.576492 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.576467 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd"] Jul 16 16:57:16.684905 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.684868 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/407e49f7-6cf3-4600-adda-6e96474beaf6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.685092 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.684926 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/407e49f7-6cf3-4600-adda-6e96474beaf6-tmp\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.685092 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.684969 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f5tl\" (UniqueName: \"kubernetes.io/projected/407e49f7-6cf3-4600-adda-6e96474beaf6-kube-api-access-8f5tl\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.685092 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.684990 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/407e49f7-6cf3-4600-adda-6e96474beaf6-webhook-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.685243 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.685087 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/407e49f7-6cf3-4600-adda-6e96474beaf6-manifests\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.785743 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.785712 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/407e49f7-6cf3-4600-adda-6e96474beaf6-tmp\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.785915 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.785756 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f5tl\" (UniqueName: \"kubernetes.io/projected/407e49f7-6cf3-4600-adda-6e96474beaf6-kube-api-access-8f5tl\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.785915 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.785780 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/407e49f7-6cf3-4600-adda-6e96474beaf6-webhook-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.785915 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.785822 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/407e49f7-6cf3-4600-adda-6e96474beaf6-manifests\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.785915 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.785867 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/407e49f7-6cf3-4600-adda-6e96474beaf6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.786195 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.786172 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/407e49f7-6cf3-4600-adda-6e96474beaf6-manifests\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.788192 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.788168 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/407e49f7-6cf3-4600-adda-6e96474beaf6-tmp\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.788284 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.788239 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/407e49f7-6cf3-4600-adda-6e96474beaf6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.788284 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.788269 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/407e49f7-6cf3-4600-adda-6e96474beaf6-webhook-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.797178 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.797158 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f5tl\" (UniqueName: \"kubernetes.io/projected/407e49f7-6cf3-4600-adda-6e96474beaf6-kube-api-access-8f5tl\") pod \"opendatahub-operator-controller-manager-c7f465c76-6d4qd\" (UID: \"407e49f7-6cf3-4600-adda-6e96474beaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:16.864526 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:16.864456 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:17.020521 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:57:17.020490 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407e49f7_6cf3_4600_adda_6e96474beaf6.slice/crio-1cc21c2b4d0acc718850aad533f93eea699c5c712d889c9edff70c94e27d66b7 WatchSource:0}: Error finding container 1cc21c2b4d0acc718850aad533f93eea699c5c712d889c9edff70c94e27d66b7: Status 404 returned error can't find the container with id 1cc21c2b4d0acc718850aad533f93eea699c5c712d889c9edff70c94e27d66b7 Jul 16 16:57:17.023468 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:17.023447 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd"] Jul 16 16:57:17.631231 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:17.631126 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" event={"ID":"407e49f7-6cf3-4600-adda-6e96474beaf6","Type":"ContainerStarted","Data":"1cc21c2b4d0acc718850aad533f93eea699c5c712d889c9edff70c94e27d66b7"} Jul 16 16:57:20.653407 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:20.653369 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" event={"ID":"407e49f7-6cf3-4600-adda-6e96474beaf6","Type":"ContainerStarted","Data":"8774b7d89a33d70a3a779ac99744aa48916d740e910874525564aadb069423eb"} Jul 16 16:57:21.659699 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:21.659663 2556 generic.go:356] "Generic (PLEG): container finished" podID="407e49f7-6cf3-4600-adda-6e96474beaf6" containerID="8774b7d89a33d70a3a779ac99744aa48916d740e910874525564aadb069423eb" exitCode=0 Jul 16 16:57:21.660148 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:21.659726 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" event={"ID":"407e49f7-6cf3-4600-adda-6e96474beaf6","Type":"ContainerDied","Data":"8774b7d89a33d70a3a779ac99744aa48916d740e910874525564aadb069423eb"} Jul 16 16:57:22.666494 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:22.666454 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" event={"ID":"407e49f7-6cf3-4600-adda-6e96474beaf6","Type":"ContainerStarted","Data":"7a4ac7caf6185e2e5e689169fdabf5f35f37d1f2bdebb9a105834980a6279d87"} Jul 16 16:57:22.666861 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:22.666528 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:22.705279 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:22.705230 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" podStartSLOduration=2.382644168 podStartE2EDuration="6.705213527s" totalImagesPullingTime="3.258194404s" totalInitContainerRuntime="1.064374955s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:57:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:22.699311919 +0000 UTC m=+372.483466981" watchObservedRunningTime="2026-07-16 16:57:22.705213527 +0000 UTC m=+372.489368588" Jul 16 16:57:33.672873 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:33.672840 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-6d4qd" Jul 16 16:57:34.028221 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.028186 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f"] Jul 16 16:57:34.031699 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.031678 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.034850 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.034831 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6v6ql\"" Jul 16 16:57:34.034955 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.034830 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 16:57:34.034955 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.034831 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 16:57:34.035047 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.034830 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 16:57:34.035860 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.035833 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6741004c-e9a1-4234-ae48-aea8047a8545-metrics-cert\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.035977 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.035961 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrgm5\" (UniqueName: \"kubernetes.io/projected/6741004c-e9a1-4234-ae48-aea8047a8545-kube-api-access-vrgm5\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.036037 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.036026 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6741004c-e9a1-4234-ae48-aea8047a8545-cert\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.036132 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.036110 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6741004c-e9a1-4234-ae48-aea8047a8545-manager-config\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.042160 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.042139 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f"] Jul 16 16:57:34.136623 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.136578 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrgm5\" (UniqueName: \"kubernetes.io/projected/6741004c-e9a1-4234-ae48-aea8047a8545-kube-api-access-vrgm5\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.136803 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.136641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6741004c-e9a1-4234-ae48-aea8047a8545-cert\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.136803 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.136749 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6741004c-e9a1-4234-ae48-aea8047a8545-manager-config\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.136923 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.136802 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6741004c-e9a1-4234-ae48-aea8047a8545-metrics-cert\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.137271 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.137247 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6741004c-e9a1-4234-ae48-aea8047a8545-manager-config\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.139063 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.139043 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6741004c-e9a1-4234-ae48-aea8047a8545-cert\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.139167 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.139110 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6741004c-e9a1-4234-ae48-aea8047a8545-metrics-cert\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.145747 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.145724 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrgm5\" (UniqueName: \"kubernetes.io/projected/6741004c-e9a1-4234-ae48-aea8047a8545-kube-api-access-vrgm5\") pod \"lws-controller-manager-7c59ccc8d7-4ld8f\" (UID: \"6741004c-e9a1-4234-ae48-aea8047a8545\") " pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.342078 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.341990 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:34.474081 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.474049 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f"] Jul 16 16:57:34.477312 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:57:34.477287 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6741004c_e9a1_4234_ae48_aea8047a8545.slice/crio-e5a870a6984f176cd53a3dffd64cfb31750750ae62ed35f1a10d15f2c9e581e8 WatchSource:0}: Error finding container e5a870a6984f176cd53a3dffd64cfb31750750ae62ed35f1a10d15f2c9e581e8: Status 404 returned error can't find the container with id e5a870a6984f176cd53a3dffd64cfb31750750ae62ed35f1a10d15f2c9e581e8 Jul 16 16:57:34.727780 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:34.727740 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" event={"ID":"6741004c-e9a1-4234-ae48-aea8047a8545","Type":"ContainerStarted","Data":"e5a870a6984f176cd53a3dffd64cfb31750750ae62ed35f1a10d15f2c9e581e8"} Jul 16 16:57:36.193714 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.193681 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd"] Jul 16 16:57:36.198168 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.198151 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.201977 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.201960 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 16:57:36.202717 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.202697 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 16:57:36.202866 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.202745 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dv5l9\"" Jul 16 16:57:36.202866 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.202762 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 16:57:36.203589 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.203571 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 16:57:36.215725 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.215702 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd"] Jul 16 16:57:36.253486 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.253456 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb42b64-0b44-46db-a4ff-8e0155c17500-tls-certs\") pod \"kube-auth-proxy-5d8b857c67-v4shd\" (UID: \"abb42b64-0b44-46db-a4ff-8e0155c17500\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.253486 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.253487 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvz5w\" (UniqueName: \"kubernetes.io/projected/abb42b64-0b44-46db-a4ff-8e0155c17500-kube-api-access-xvz5w\") pod \"kube-auth-proxy-5d8b857c67-v4shd\" (UID: \"abb42b64-0b44-46db-a4ff-8e0155c17500\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.253698 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.253678 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abb42b64-0b44-46db-a4ff-8e0155c17500-tmp\") pod \"kube-auth-proxy-5d8b857c67-v4shd\" (UID: \"abb42b64-0b44-46db-a4ff-8e0155c17500\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.354496 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.354458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb42b64-0b44-46db-a4ff-8e0155c17500-tls-certs\") pod \"kube-auth-proxy-5d8b857c67-v4shd\" (UID: \"abb42b64-0b44-46db-a4ff-8e0155c17500\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.354496 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.354499 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvz5w\" (UniqueName: \"kubernetes.io/projected/abb42b64-0b44-46db-a4ff-8e0155c17500-kube-api-access-xvz5w\") pod \"kube-auth-proxy-5d8b857c67-v4shd\" (UID: \"abb42b64-0b44-46db-a4ff-8e0155c17500\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.354741 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.354575 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abb42b64-0b44-46db-a4ff-8e0155c17500-tmp\") pod \"kube-auth-proxy-5d8b857c67-v4shd\" (UID: \"abb42b64-0b44-46db-a4ff-8e0155c17500\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.356733 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.356703 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abb42b64-0b44-46db-a4ff-8e0155c17500-tmp\") pod \"kube-auth-proxy-5d8b857c67-v4shd\" (UID: \"abb42b64-0b44-46db-a4ff-8e0155c17500\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.356959 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.356941 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb42b64-0b44-46db-a4ff-8e0155c17500-tls-certs\") pod \"kube-auth-proxy-5d8b857c67-v4shd\" (UID: \"abb42b64-0b44-46db-a4ff-8e0155c17500\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.363554 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.363527 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvz5w\" (UniqueName: \"kubernetes.io/projected/abb42b64-0b44-46db-a4ff-8e0155c17500-kube-api-access-xvz5w\") pod \"kube-auth-proxy-5d8b857c67-v4shd\" (UID: \"abb42b64-0b44-46db-a4ff-8e0155c17500\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.508175 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.508116 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" Jul 16 16:57:36.644180 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.644145 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd"] Jul 16 16:57:36.647076 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:57:36.647046 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb42b64_0b44_46db_a4ff_8e0155c17500.slice/crio-1fee79720e7bc0c373b0249e451cf5690fca583788f4da016c90141ed4def7d3 WatchSource:0}: Error finding container 1fee79720e7bc0c373b0249e451cf5690fca583788f4da016c90141ed4def7d3: Status 404 returned error can't find the container with id 1fee79720e7bc0c373b0249e451cf5690fca583788f4da016c90141ed4def7d3 Jul 16 16:57:36.738099 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:36.738064 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" event={"ID":"abb42b64-0b44-46db-a4ff-8e0155c17500","Type":"ContainerStarted","Data":"1fee79720e7bc0c373b0249e451cf5690fca583788f4da016c90141ed4def7d3"} Jul 16 16:57:40.764678 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:40.764644 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" event={"ID":"abb42b64-0b44-46db-a4ff-8e0155c17500","Type":"ContainerStarted","Data":"e9e4dbff76a9f0b6d58c5ad69224d4a335f1fac23c43ff4642112268dd98d43c"} Jul 16 16:57:40.766120 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:40.766097 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" event={"ID":"6741004c-e9a1-4234-ae48-aea8047a8545","Type":"ContainerStarted","Data":"deb157ecff9d81ea2aa08941b162169d09f7a13f546d3045f1b82d7f188f9eef"} Jul 16 16:57:40.766191 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:40.766151 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:57:40.784282 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:40.784237 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d8b857c67-v4shd" podStartSLOduration=1.6771810409999999 podStartE2EDuration="4.784221293s" totalImagesPullingTime="3.107040252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:40.782386864 +0000 UTC m=+390.566541928" watchObservedRunningTime="2026-07-16 16:57:40.784221293 +0000 UTC m=+390.568376391" Jul 16 16:57:40.800539 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:40.800478 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" podStartSLOduration=1.5246590960000002 podStartE2EDuration="6.800457076s" totalImagesPullingTime="5.27579798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:40.80001224 +0000 UTC m=+390.584167305" watchObservedRunningTime="2026-07-16 16:57:40.800457076 +0000 UTC m=+390.584612142" Jul 16 16:57:51.773906 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:57:51.773874 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c59ccc8d7-4ld8f" Jul 16 16:59:24.465594 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.465564 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh"] Jul 16 16:59:24.468645 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.468628 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.471118 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.471096 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 16 16:59:24.471819 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.471801 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 16 16:59:24.472028 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.472015 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 16:59:24.472096 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.472080 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 16:59:24.472154 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.472113 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-chkj9\"" Jul 16 16:59:24.488934 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.488908 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/da3d79ee-4cc6-44f7-b9ff-bb81897c2102-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-t9bhh\" (UID: \"da3d79ee-4cc6-44f7-b9ff-bb81897c2102\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.489068 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.488997 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r47nx\" (UniqueName: \"kubernetes.io/projected/da3d79ee-4cc6-44f7-b9ff-bb81897c2102-kube-api-access-r47nx\") pod \"kuadrant-console-plugin-66b5774985-t9bhh\" (UID: \"da3d79ee-4cc6-44f7-b9ff-bb81897c2102\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.489132 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.489100 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/da3d79ee-4cc6-44f7-b9ff-bb81897c2102-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-t9bhh\" (UID: \"da3d79ee-4cc6-44f7-b9ff-bb81897c2102\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.491463 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.491440 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh"] Jul 16 16:59:24.589764 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.589738 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r47nx\" (UniqueName: \"kubernetes.io/projected/da3d79ee-4cc6-44f7-b9ff-bb81897c2102-kube-api-access-r47nx\") pod \"kuadrant-console-plugin-66b5774985-t9bhh\" (UID: \"da3d79ee-4cc6-44f7-b9ff-bb81897c2102\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.589907 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.589818 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/da3d79ee-4cc6-44f7-b9ff-bb81897c2102-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-t9bhh\" (UID: \"da3d79ee-4cc6-44f7-b9ff-bb81897c2102\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.589907 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.589853 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/da3d79ee-4cc6-44f7-b9ff-bb81897c2102-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-t9bhh\" (UID: \"da3d79ee-4cc6-44f7-b9ff-bb81897c2102\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.590446 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.590400 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/da3d79ee-4cc6-44f7-b9ff-bb81897c2102-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-t9bhh\" (UID: \"da3d79ee-4cc6-44f7-b9ff-bb81897c2102\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.592164 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.592140 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/da3d79ee-4cc6-44f7-b9ff-bb81897c2102-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-t9bhh\" (UID: \"da3d79ee-4cc6-44f7-b9ff-bb81897c2102\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.604733 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.604707 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r47nx\" (UniqueName: \"kubernetes.io/projected/da3d79ee-4cc6-44f7-b9ff-bb81897c2102-kube-api-access-r47nx\") pod \"kuadrant-console-plugin-66b5774985-t9bhh\" (UID: \"da3d79ee-4cc6-44f7-b9ff-bb81897c2102\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.782364 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.782342 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" Jul 16 16:59:24.927913 ip-10-0-133-149 kubenswrapper[2556]: W0716 16:59:24.927872 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda3d79ee_4cc6_44f7_b9ff_bb81897c2102.slice/crio-afcc45d9ddc0bfbafd920b05364a75030cd13a1033ce6e1c8a0af48bb1842494 WatchSource:0}: Error finding container afcc45d9ddc0bfbafd920b05364a75030cd13a1033ce6e1c8a0af48bb1842494: Status 404 returned error can't find the container with id afcc45d9ddc0bfbafd920b05364a75030cd13a1033ce6e1c8a0af48bb1842494 Jul 16 16:59:24.928123 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:24.928102 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh"] Jul 16 16:59:25.240512 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:25.240485 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" event={"ID":"da3d79ee-4cc6-44f7-b9ff-bb81897c2102","Type":"ContainerStarted","Data":"afcc45d9ddc0bfbafd920b05364a75030cd13a1033ce6e1c8a0af48bb1842494"} Jul 16 16:59:48.360241 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:48.360204 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" event={"ID":"da3d79ee-4cc6-44f7-b9ff-bb81897c2102","Type":"ContainerStarted","Data":"8b7ea7aba1ec92bd185f0deb0c04962c71c6be508e1c0ab567fe762403aaa705"} Jul 16 16:59:48.402072 ip-10-0-133-149 kubenswrapper[2556]: I0716 16:59:48.402024 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-t9bhh" podStartSLOduration=1.629909872 podStartE2EDuration="24.402011012s" totalImagesPullingTime="22.77210114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:59:48.396480452 +0000 UTC m=+518.180635513" watchObservedRunningTime="2026-07-16 16:59:48.402011012 +0000 UTC m=+518.186166071" Jul 16 17:00:12.202508 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.202393 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d5qmt"] Jul 16 17:00:12.259403 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.259367 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d5qmt"] Jul 16 17:00:12.259579 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.259485 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:12.262866 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.262839 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 17:00:12.313217 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.313188 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d5qmt"] Jul 16 17:00:12.334873 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.334845 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwkq\" (UniqueName: \"kubernetes.io/projected/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-kube-api-access-2jwkq\") pod \"limitador-limitador-64c94fd64f-d5qmt\" (UID: \"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:12.334990 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.334914 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-config-file\") pod \"limitador-limitador-64c94fd64f-d5qmt\" (UID: \"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:12.435574 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.435536 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwkq\" (UniqueName: \"kubernetes.io/projected/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-kube-api-access-2jwkq\") pod \"limitador-limitador-64c94fd64f-d5qmt\" (UID: \"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:12.435736 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.435618 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-config-file\") pod \"limitador-limitador-64c94fd64f-d5qmt\" (UID: \"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:12.436292 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.436269 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-config-file\") pod \"limitador-limitador-64c94fd64f-d5qmt\" (UID: \"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:12.445403 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.445374 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwkq\" (UniqueName: \"kubernetes.io/projected/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-kube-api-access-2jwkq\") pod \"limitador-limitador-64c94fd64f-d5qmt\" (UID: \"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:12.570112 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.570076 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:12.715074 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:12.715037 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d5qmt"] Jul 16 17:00:12.718344 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:00:12.718314 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f5c5dd_dc93_4bcb_9a00_c3be434e237f.slice/crio-07a6b77c24a6a281cadce96ee4ff156837650875fd4c6043caedceb67f424c70 WatchSource:0}: Error finding container 07a6b77c24a6a281cadce96ee4ff156837650875fd4c6043caedceb67f424c70: Status 404 returned error can't find the container with id 07a6b77c24a6a281cadce96ee4ff156837650875fd4c6043caedceb67f424c70 Jul 16 17:00:13.403017 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:13.402986 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-mj724"] Jul 16 17:00:13.407546 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:13.407519 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mj724" Jul 16 17:00:13.411153 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:13.411128 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2srxg\"" Jul 16 17:00:13.416383 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:13.416361 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mj724"] Jul 16 17:00:13.476392 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:13.476357 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" event={"ID":"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f","Type":"ContainerStarted","Data":"07a6b77c24a6a281cadce96ee4ff156837650875fd4c6043caedceb67f424c70"} Jul 16 17:00:13.545104 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:13.545074 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sv7f\" (UniqueName: \"kubernetes.io/projected/8e3917e5-8ba1-44ca-9694-0d774e823e76-kube-api-access-8sv7f\") pod \"authorino-75d556bd77-mj724\" (UID: \"8e3917e5-8ba1-44ca-9694-0d774e823e76\") " pod="kuadrant-system/authorino-75d556bd77-mj724" Jul 16 17:00:13.646354 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:13.646319 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sv7f\" (UniqueName: \"kubernetes.io/projected/8e3917e5-8ba1-44ca-9694-0d774e823e76-kube-api-access-8sv7f\") pod \"authorino-75d556bd77-mj724\" (UID: \"8e3917e5-8ba1-44ca-9694-0d774e823e76\") " pod="kuadrant-system/authorino-75d556bd77-mj724" Jul 16 17:00:13.657576 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:13.657513 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sv7f\" (UniqueName: \"kubernetes.io/projected/8e3917e5-8ba1-44ca-9694-0d774e823e76-kube-api-access-8sv7f\") pod \"authorino-75d556bd77-mj724\" (UID: \"8e3917e5-8ba1-44ca-9694-0d774e823e76\") " pod="kuadrant-system/authorino-75d556bd77-mj724" Jul 16 17:00:13.717430 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:13.717387 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mj724" Jul 16 17:00:13.846672 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:13.846648 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mj724"] Jul 16 17:00:13.848507 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:00:13.848482 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e3917e5_8ba1_44ca_9694_0d774e823e76.slice/crio-2d1b4816df0e758e47520ea4150779bac92c582dcbf172b646970ad8f590d080 WatchSource:0}: Error finding container 2d1b4816df0e758e47520ea4150779bac92c582dcbf172b646970ad8f590d080: Status 404 returned error can't find the container with id 2d1b4816df0e758e47520ea4150779bac92c582dcbf172b646970ad8f590d080 Jul 16 17:00:14.483796 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:14.483753 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mj724" event={"ID":"8e3917e5-8ba1-44ca-9694-0d774e823e76","Type":"ContainerStarted","Data":"2d1b4816df0e758e47520ea4150779bac92c582dcbf172b646970ad8f590d080"} Jul 16 17:00:16.498001 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:16.497846 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" event={"ID":"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f","Type":"ContainerStarted","Data":"aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea"} Jul 16 17:00:16.498454 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:16.498367 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:16.519750 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:16.519689 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" podStartSLOduration=1.219573552 podStartE2EDuration="4.519667657s" totalImagesPullingTime="3.300094105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:00:16.517455371 +0000 UTC m=+546.301610469" watchObservedRunningTime="2026-07-16 17:00:16.519667657 +0000 UTC m=+546.303822720" Jul 16 17:00:18.511117 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:18.511087 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mj724" event={"ID":"8e3917e5-8ba1-44ca-9694-0d774e823e76","Type":"ContainerStarted","Data":"c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4"} Jul 16 17:00:18.529640 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:18.529598 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-mj724" podStartSLOduration=0.948069354 podStartE2EDuration="5.529581621s" totalImagesPullingTime="4.581512267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:00:18.527959436 +0000 UTC m=+548.312114497" watchObservedRunningTime="2026-07-16 17:00:18.529581621 +0000 UTC m=+548.313736682" Jul 16 17:00:27.154946 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:27.154909 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d5qmt"] Jul 16 17:00:27.155443 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:27.155165 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" podUID="e4f5c5dd-dc93-4bcb-9a00-c3be434e237f" containerName="limitador" containerID="cri-o://aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea" gracePeriod=30 Jul 16 17:00:27.155848 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:27.155778 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:27.705933 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:27.705905 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:27.884956 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:27.884913 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-config-file\") pod \"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f\" (UID: \"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f\") " Jul 16 17:00:27.884956 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:27.884963 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jwkq\" (UniqueName: \"kubernetes.io/projected/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-kube-api-access-2jwkq\") pod \"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f\" (UID: \"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f\") " Jul 16 17:00:27.885400 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:27.885373 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-config-file" (OuterVolumeSpecName: "config-file") pod "e4f5c5dd-dc93-4bcb-9a00-c3be434e237f" (UID: "e4f5c5dd-dc93-4bcb-9a00-c3be434e237f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:00:27.887187 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:27.887164 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-kube-api-access-2jwkq" (OuterVolumeSpecName: "kube-api-access-2jwkq") pod "e4f5c5dd-dc93-4bcb-9a00-c3be434e237f" (UID: "e4f5c5dd-dc93-4bcb-9a00-c3be434e237f"). InnerVolumeSpecName "kube-api-access-2jwkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:00:27.985822 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:27.985782 2556 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-config-file\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:00:27.985822 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:27.985815 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jwkq\" (UniqueName: \"kubernetes.io/projected/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f-kube-api-access-2jwkq\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:00:28.564246 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.564214 2556 generic.go:356] "Generic (PLEG): container finished" podID="e4f5c5dd-dc93-4bcb-9a00-c3be434e237f" containerID="aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea" exitCode=0 Jul 16 17:00:28.564764 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.564258 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" Jul 16 17:00:28.564764 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.564306 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" event={"ID":"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f","Type":"ContainerDied","Data":"aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea"} Jul 16 17:00:28.564764 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.564369 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-d5qmt" event={"ID":"e4f5c5dd-dc93-4bcb-9a00-c3be434e237f","Type":"ContainerDied","Data":"07a6b77c24a6a281cadce96ee4ff156837650875fd4c6043caedceb67f424c70"} Jul 16 17:00:28.564764 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.564403 2556 scope.go:117] "RemoveContainer" containerID="aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea" Jul 16 17:00:28.573615 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.573597 2556 scope.go:117] "RemoveContainer" containerID="aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea" Jul 16 17:00:28.573870 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:00:28.573851 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea\": container with ID starting with aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea not found: ID does not exist" containerID="aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea" Jul 16 17:00:28.573933 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.573883 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea"} err="failed to get container status \"aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea\": rpc error: code = NotFound desc = could not find container \"aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea\": container with ID starting with aa632c5d86fcf0ea1d531b1299724acd9eb9f84b164f76644b80a7953e216bea not found: ID does not exist" Jul 16 17:00:28.592911 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.592880 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d5qmt"] Jul 16 17:00:28.596018 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.595995 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-d5qmt"] Jul 16 17:00:28.842353 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.842269 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4f5c5dd-dc93-4bcb-9a00-c3be434e237f" path="/var/lib/kubelet/pods/e4f5c5dd-dc93-4bcb-9a00-c3be434e237f/volumes" Jul 16 17:00:28.905451 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.905406 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4"] Jul 16 17:00:28.905798 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.905785 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f5c5dd-dc93-4bcb-9a00-c3be434e237f" containerName="limitador" Jul 16 17:00:28.905845 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.905800 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f5c5dd-dc93-4bcb-9a00-c3be434e237f" containerName="limitador" Jul 16 17:00:28.905889 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.905880 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f5c5dd-dc93-4bcb-9a00-c3be434e237f" containerName="limitador" Jul 16 17:00:28.910298 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.910279 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" Jul 16 17:00:28.913527 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.913505 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-cqmp5\"" Jul 16 17:00:28.913632 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.913508 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 16 17:00:28.913632 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.913608 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 17:00:28.913730 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.913672 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:00:28.920001 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.919978 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4"] Jul 16 17:00:28.994808 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.994774 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmtfv\" (UniqueName: \"kubernetes.io/projected/a0cd2061-169d-4f40-8f6c-86398cacd566-kube-api-access-xmtfv\") pod \"postgres-69fb4594cf-gqhs4\" (UID: \"a0cd2061-169d-4f40-8f6c-86398cacd566\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" Jul 16 17:00:28.994974 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:28.994860 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a0cd2061-169d-4f40-8f6c-86398cacd566-data\") pod \"postgres-69fb4594cf-gqhs4\" (UID: \"a0cd2061-169d-4f40-8f6c-86398cacd566\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" Jul 16 17:00:29.095717 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:29.095641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a0cd2061-169d-4f40-8f6c-86398cacd566-data\") pod \"postgres-69fb4594cf-gqhs4\" (UID: \"a0cd2061-169d-4f40-8f6c-86398cacd566\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" Jul 16 17:00:29.095717 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:29.095705 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmtfv\" (UniqueName: \"kubernetes.io/projected/a0cd2061-169d-4f40-8f6c-86398cacd566-kube-api-access-xmtfv\") pod \"postgres-69fb4594cf-gqhs4\" (UID: \"a0cd2061-169d-4f40-8f6c-86398cacd566\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" Jul 16 17:00:29.096023 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:29.096004 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a0cd2061-169d-4f40-8f6c-86398cacd566-data\") pod \"postgres-69fb4594cf-gqhs4\" (UID: \"a0cd2061-169d-4f40-8f6c-86398cacd566\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" Jul 16 17:00:29.112022 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:29.111989 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmtfv\" (UniqueName: \"kubernetes.io/projected/a0cd2061-169d-4f40-8f6c-86398cacd566-kube-api-access-xmtfv\") pod \"postgres-69fb4594cf-gqhs4\" (UID: \"a0cd2061-169d-4f40-8f6c-86398cacd566\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" Jul 16 17:00:29.220540 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:29.220510 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" Jul 16 17:00:29.353670 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:29.353646 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4"] Jul 16 17:00:29.355722 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:00:29.355691 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cd2061_169d_4f40_8f6c_86398cacd566.slice/crio-0db17d168cb41da7d4733053e964022ecdbaadb930079fabfa9d8a3c5d6321f2 WatchSource:0}: Error finding container 0db17d168cb41da7d4733053e964022ecdbaadb930079fabfa9d8a3c5d6321f2: Status 404 returned error can't find the container with id 0db17d168cb41da7d4733053e964022ecdbaadb930079fabfa9d8a3c5d6321f2 Jul 16 17:00:29.571481 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:29.571444 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" event={"ID":"a0cd2061-169d-4f40-8f6c-86398cacd566","Type":"ContainerStarted","Data":"0db17d168cb41da7d4733053e964022ecdbaadb930079fabfa9d8a3c5d6321f2"} Jul 16 17:00:34.601186 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:34.601148 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" event={"ID":"a0cd2061-169d-4f40-8f6c-86398cacd566","Type":"ContainerStarted","Data":"767ce7116df63c33e6e1bc5ee18f66f5608bf885dabe5770125b06ed604b68db"} Jul 16 17:00:34.601615 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:34.601243 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" Jul 16 17:00:34.622361 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:34.622318 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" podStartSLOduration=1.934512052 podStartE2EDuration="6.622298289s" totalImagesPullingTime="4.687786237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:00:34.621277861 +0000 UTC m=+564.405432922" watchObservedRunningTime="2026-07-16 17:00:34.622298289 +0000 UTC m=+564.406453356" Jul 16 17:00:40.636202 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:40.636171 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gqhs4" Jul 16 17:00:43.590074 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:43.590033 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mj724"] Jul 16 17:00:43.590616 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:43.590259 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-mj724" podUID="8e3917e5-8ba1-44ca-9694-0d774e823e76" containerName="authorino" containerID="cri-o://c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4" gracePeriod=30 Jul 16 17:00:43.842503 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:43.842435 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mj724" Jul 16 17:00:43.933955 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:43.933913 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sv7f\" (UniqueName: \"kubernetes.io/projected/8e3917e5-8ba1-44ca-9694-0d774e823e76-kube-api-access-8sv7f\") pod \"8e3917e5-8ba1-44ca-9694-0d774e823e76\" (UID: \"8e3917e5-8ba1-44ca-9694-0d774e823e76\") " Jul 16 17:00:43.936024 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:43.935991 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3917e5-8ba1-44ca-9694-0d774e823e76-kube-api-access-8sv7f" (OuterVolumeSpecName: "kube-api-access-8sv7f") pod "8e3917e5-8ba1-44ca-9694-0d774e823e76" (UID: "8e3917e5-8ba1-44ca-9694-0d774e823e76"). InnerVolumeSpecName "kube-api-access-8sv7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:00:44.035367 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.035316 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sv7f\" (UniqueName: \"kubernetes.io/projected/8e3917e5-8ba1-44ca-9694-0d774e823e76-kube-api-access-8sv7f\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:00:44.653067 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.653023 2556 generic.go:356] "Generic (PLEG): container finished" podID="8e3917e5-8ba1-44ca-9694-0d774e823e76" containerID="c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4" exitCode=0 Jul 16 17:00:44.653661 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.653116 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mj724" event={"ID":"8e3917e5-8ba1-44ca-9694-0d774e823e76","Type":"ContainerDied","Data":"c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4"} Jul 16 17:00:44.653661 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.653156 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mj724" event={"ID":"8e3917e5-8ba1-44ca-9694-0d774e823e76","Type":"ContainerDied","Data":"2d1b4816df0e758e47520ea4150779bac92c582dcbf172b646970ad8f590d080"} Jul 16 17:00:44.653661 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.653181 2556 scope.go:117] "RemoveContainer" containerID="c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4" Jul 16 17:00:44.653661 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.653351 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mj724" Jul 16 17:00:44.675146 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.675119 2556 scope.go:117] "RemoveContainer" containerID="c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4" Jul 16 17:00:44.675526 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:00:44.675499 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4\": container with ID starting with c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4 not found: ID does not exist" containerID="c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4" Jul 16 17:00:44.675610 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.675535 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4"} err="failed to get container status \"c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4\": rpc error: code = NotFound desc = could not find container \"c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4\": container with ID starting with c2d4c4aa220e3b21f0b6e0402ce411f8022ab2d8d2f9d35548e77897aee491d4 not found: ID does not exist" Jul 16 17:00:44.683591 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.683559 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mj724"] Jul 16 17:00:44.689926 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.689894 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mj724"] Jul 16 17:00:44.843874 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:44.843830 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e3917e5-8ba1-44ca-9694-0d774e823e76" path="/var/lib/kubelet/pods/8e3917e5-8ba1-44ca-9694-0d774e823e76/volumes" Jul 16 17:00:48.665210 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.665165 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6df74f7ffb-j6qgp"] Jul 16 17:00:48.665717 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.665698 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3917e5-8ba1-44ca-9694-0d774e823e76" containerName="authorino" Jul 16 17:00:48.665765 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.665719 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3917e5-8ba1-44ca-9694-0d774e823e76" containerName="authorino" Jul 16 17:00:48.665805 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.665797 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3917e5-8ba1-44ca-9694-0d774e823e76" containerName="authorino" Jul 16 17:00:48.840825 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.840789 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6df74f7ffb-j6qgp"] Jul 16 17:00:48.840825 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.840833 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-758884f967-2n557"] Jul 16 17:00:48.841050 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.840971 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:00:48.844486 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.844459 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 16 17:00:48.844652 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.844639 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5bw72\"" Jul 16 17:00:48.845237 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.845221 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 16 17:00:48.857457 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.857425 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:00:48.866879 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.866849 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-758884f967-2n557"] Jul 16 17:00:48.877099 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.877064 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msv4p\" (UniqueName: \"kubernetes.io/projected/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-kube-api-access-msv4p\") pod \"maas-controller-6df74f7ffb-j6qgp\" (UID: \"39b55aae-14a5-47f1-8b69-9fdfcb4d4904\") " pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:00:48.877284 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.877114 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-webhook-cert\") pod \"maas-controller-6df74f7ffb-j6qgp\" (UID: \"39b55aae-14a5-47f1-8b69-9fdfcb4d4904\") " pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:00:48.978090 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.978046 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-webhook-cert\") pod \"maas-controller-758884f967-2n557\" (UID: \"d71b8602-eb9e-483e-a749-e20a4ac2c5ae\") " pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:00:48.978305 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.978101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msv4p\" (UniqueName: \"kubernetes.io/projected/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-kube-api-access-msv4p\") pod \"maas-controller-6df74f7ffb-j6qgp\" (UID: \"39b55aae-14a5-47f1-8b69-9fdfcb4d4904\") " pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:00:48.978305 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.978138 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-webhook-cert\") pod \"maas-controller-6df74f7ffb-j6qgp\" (UID: \"39b55aae-14a5-47f1-8b69-9fdfcb4d4904\") " pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:00:48.978305 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.978159 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcxwv\" (UniqueName: \"kubernetes.io/projected/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-kube-api-access-bcxwv\") pod \"maas-controller-758884f967-2n557\" (UID: \"d71b8602-eb9e-483e-a749-e20a4ac2c5ae\") " pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:00:48.980698 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.980666 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-webhook-cert\") pod \"maas-controller-6df74f7ffb-j6qgp\" (UID: \"39b55aae-14a5-47f1-8b69-9fdfcb4d4904\") " pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:00:48.988407 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:48.988373 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msv4p\" (UniqueName: \"kubernetes.io/projected/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-kube-api-access-msv4p\") pod \"maas-controller-6df74f7ffb-j6qgp\" (UID: \"39b55aae-14a5-47f1-8b69-9fdfcb4d4904\") " pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:00:49.078716 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:49.078675 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-webhook-cert\") pod \"maas-controller-758884f967-2n557\" (UID: \"d71b8602-eb9e-483e-a749-e20a4ac2c5ae\") " pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:00:49.078897 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:49.078740 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcxwv\" (UniqueName: \"kubernetes.io/projected/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-kube-api-access-bcxwv\") pod \"maas-controller-758884f967-2n557\" (UID: \"d71b8602-eb9e-483e-a749-e20a4ac2c5ae\") " pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:00:49.081394 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:49.081366 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-webhook-cert\") pod \"maas-controller-758884f967-2n557\" (UID: \"d71b8602-eb9e-483e-a749-e20a4ac2c5ae\") " pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:00:49.092192 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:49.092159 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcxwv\" (UniqueName: \"kubernetes.io/projected/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-kube-api-access-bcxwv\") pod \"maas-controller-758884f967-2n557\" (UID: \"d71b8602-eb9e-483e-a749-e20a4ac2c5ae\") " pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:00:49.153087 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:49.153044 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:00:49.168217 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:49.168182 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:00:49.328481 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:49.328452 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6df74f7ffb-j6qgp"] Jul 16 17:00:49.331208 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:00:49.331178 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b55aae_14a5_47f1_8b69_9fdfcb4d4904.slice/crio-948f9308b767b0b63ae33dff9b8672bfde01cb0b1f58a82a1bf5cad5e70aed8c WatchSource:0}: Error finding container 948f9308b767b0b63ae33dff9b8672bfde01cb0b1f58a82a1bf5cad5e70aed8c: Status 404 returned error can't find the container with id 948f9308b767b0b63ae33dff9b8672bfde01cb0b1f58a82a1bf5cad5e70aed8c Jul 16 17:00:49.357115 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:49.357089 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-758884f967-2n557"] Jul 16 17:00:49.359608 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:00:49.359574 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71b8602_eb9e_483e_a749_e20a4ac2c5ae.slice/crio-8c3245d2fd50d08b9395634cecf468c86702313be9072f772d78f2d52677cfa6 WatchSource:0}: Error finding container 8c3245d2fd50d08b9395634cecf468c86702313be9072f772d78f2d52677cfa6: Status 404 returned error can't find the container with id 8c3245d2fd50d08b9395634cecf468c86702313be9072f772d78f2d52677cfa6 Jul 16 17:00:49.679908 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:49.679799 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-758884f967-2n557" event={"ID":"d71b8602-eb9e-483e-a749-e20a4ac2c5ae","Type":"ContainerStarted","Data":"8c3245d2fd50d08b9395634cecf468c86702313be9072f772d78f2d52677cfa6"} Jul 16 17:00:49.680994 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:49.680967 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" event={"ID":"39b55aae-14a5-47f1-8b69-9fdfcb4d4904","Type":"ContainerStarted","Data":"948f9308b767b0b63ae33dff9b8672bfde01cb0b1f58a82a1bf5cad5e70aed8c"} Jul 16 17:00:52.701304 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:52.701263 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-758884f967-2n557" event={"ID":"d71b8602-eb9e-483e-a749-e20a4ac2c5ae","Type":"ContainerStarted","Data":"a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc"} Jul 16 17:00:52.701795 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:52.701434 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:00:52.702874 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:52.702848 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" event={"ID":"39b55aae-14a5-47f1-8b69-9fdfcb4d4904","Type":"ContainerStarted","Data":"dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e"} Jul 16 17:00:52.702979 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:52.702913 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:00:52.733581 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:52.733520 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-758884f967-2n557" podStartSLOduration=2.227098582 podStartE2EDuration="4.733501811s" totalImagesPullingTime="2.506403229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:00:52.732165112 +0000 UTC m=+582.516320190" watchObservedRunningTime="2026-07-16 17:00:52.733501811 +0000 UTC m=+582.517656900" Jul 16 17:00:52.758825 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:00:52.758766 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" podStartSLOduration=2.2231981259999998 podStartE2EDuration="4.758745142s" totalImagesPullingTime="2.535547016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:00:52.757482145 +0000 UTC m=+582.541637208" watchObservedRunningTime="2026-07-16 17:00:52.758745142 +0000 UTC m=+582.542900203" Jul 16 17:01:03.710343 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:03.710307 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:01:03.710818 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:03.710427 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:01:03.816929 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:03.816895 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6df74f7ffb-j6qgp"] Jul 16 17:01:03.817164 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:03.817138 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" podUID="39b55aae-14a5-47f1-8b69-9fdfcb4d4904" containerName="manager" containerID="cri-o://dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e" gracePeriod=30 Jul 16 17:01:04.072994 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.072962 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:01:04.224881 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.224854 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-webhook-cert\") pod \"39b55aae-14a5-47f1-8b69-9fdfcb4d4904\" (UID: \"39b55aae-14a5-47f1-8b69-9fdfcb4d4904\") " Jul 16 17:01:04.225063 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.224896 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msv4p\" (UniqueName: \"kubernetes.io/projected/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-kube-api-access-msv4p\") pod \"39b55aae-14a5-47f1-8b69-9fdfcb4d4904\" (UID: \"39b55aae-14a5-47f1-8b69-9fdfcb4d4904\") " Jul 16 17:01:04.226944 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.226902 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-kube-api-access-msv4p" (OuterVolumeSpecName: "kube-api-access-msv4p") pod "39b55aae-14a5-47f1-8b69-9fdfcb4d4904" (UID: "39b55aae-14a5-47f1-8b69-9fdfcb4d4904"). InnerVolumeSpecName "kube-api-access-msv4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:01:04.226944 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.226941 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "39b55aae-14a5-47f1-8b69-9fdfcb4d4904" (UID: "39b55aae-14a5-47f1-8b69-9fdfcb4d4904"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:01:04.326321 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.326245 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-webhook-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:01:04.326321 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.326272 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msv4p\" (UniqueName: \"kubernetes.io/projected/39b55aae-14a5-47f1-8b69-9fdfcb4d4904-kube-api-access-msv4p\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:01:04.764812 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.764780 2556 generic.go:356] "Generic (PLEG): container finished" podID="39b55aae-14a5-47f1-8b69-9fdfcb4d4904" containerID="dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e" exitCode=0 Jul 16 17:01:04.765221 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.764827 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" Jul 16 17:01:04.765221 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.764846 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" event={"ID":"39b55aae-14a5-47f1-8b69-9fdfcb4d4904","Type":"ContainerDied","Data":"dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e"} Jul 16 17:01:04.765221 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.764889 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df74f7ffb-j6qgp" event={"ID":"39b55aae-14a5-47f1-8b69-9fdfcb4d4904","Type":"ContainerDied","Data":"948f9308b767b0b63ae33dff9b8672bfde01cb0b1f58a82a1bf5cad5e70aed8c"} Jul 16 17:01:04.765221 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.764912 2556 scope.go:117] "RemoveContainer" containerID="dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e" Jul 16 17:01:04.774187 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.774169 2556 scope.go:117] "RemoveContainer" containerID="dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e" Jul 16 17:01:04.774447 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:01:04.774398 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e\": container with ID starting with dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e not found: ID does not exist" containerID="dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e" Jul 16 17:01:04.774509 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.774449 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e"} err="failed to get container status \"dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e\": rpc error: code = NotFound desc = could not find container \"dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e\": container with ID starting with dc85395aa62e9d895cd161383e67c3278257ef300b053e85d28175db43f5206e not found: ID does not exist" Jul 16 17:01:04.790111 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.790076 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6df74f7ffb-j6qgp"] Jul 16 17:01:04.795950 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.795926 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6df74f7ffb-j6qgp"] Jul 16 17:01:04.842927 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:04.842898 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b55aae-14a5-47f1-8b69-9fdfcb4d4904" path="/var/lib/kubelet/pods/39b55aae-14a5-47f1-8b69-9fdfcb4d4904/volumes" Jul 16 17:01:10.718362 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:10.718323 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:01:10.723014 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:10.722990 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:01:24.756603 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.756566 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2"] Jul 16 17:01:24.757034 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.756943 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b55aae-14a5-47f1-8b69-9fdfcb4d4904" containerName="manager" Jul 16 17:01:24.757034 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.756955 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b55aae-14a5-47f1-8b69-9fdfcb4d4904" containerName="manager" Jul 16 17:01:24.757034 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.757028 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b55aae-14a5-47f1-8b69-9fdfcb4d4904" containerName="manager" Jul 16 17:01:24.762142 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.762119 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:24.767773 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.767744 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bff62\"" Jul 16 17:01:24.769103 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.769036 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 16 17:01:24.793006 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.792926 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2"] Jul 16 17:01:24.818063 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.818022 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e-openshift-service-ca\") pod \"maas-api-7b7d967754-7qkq2\" (UID: \"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e\") " pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:24.818262 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.818136 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e-maas-api-tls\") pod \"maas-api-7b7d967754-7qkq2\" (UID: \"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e\") " pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:24.818262 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.818182 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w279p\" (UniqueName: \"kubernetes.io/projected/d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e-kube-api-access-w279p\") pod \"maas-api-7b7d967754-7qkq2\" (UID: \"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e\") " pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:24.918927 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.918885 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e-maas-api-tls\") pod \"maas-api-7b7d967754-7qkq2\" (UID: \"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e\") " pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:24.919154 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.918937 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w279p\" (UniqueName: \"kubernetes.io/projected/d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e-kube-api-access-w279p\") pod \"maas-api-7b7d967754-7qkq2\" (UID: \"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e\") " pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:24.919154 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.918995 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e-openshift-service-ca\") pod \"maas-api-7b7d967754-7qkq2\" (UID: \"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e\") " pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:24.919733 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.919707 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e-openshift-service-ca\") pod \"maas-api-7b7d967754-7qkq2\" (UID: \"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e\") " pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:24.921307 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.921288 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e-maas-api-tls\") pod \"maas-api-7b7d967754-7qkq2\" (UID: \"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e\") " pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:24.944772 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:24.944735 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w279p\" (UniqueName: \"kubernetes.io/projected/d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e-kube-api-access-w279p\") pod \"maas-api-7b7d967754-7qkq2\" (UID: \"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e\") " pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:25.073086 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.072982 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:25.225134 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:01:25.225104 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ae5d9a_456c_4d1c_94f8_41eb3f40d63e.slice/crio-4b6eb3597080e8bd98953750e0a0d6fb272248ba03d25d2cf2dfde1673c6bef1 WatchSource:0}: Error finding container 4b6eb3597080e8bd98953750e0a0d6fb272248ba03d25d2cf2dfde1673c6bef1: Status 404 returned error can't find the container with id 4b6eb3597080e8bd98953750e0a0d6fb272248ba03d25d2cf2dfde1673c6bef1 Jul 16 17:01:25.227056 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.227029 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2"] Jul 16 17:01:25.695761 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.695721 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p"] Jul 16 17:01:25.698983 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.698957 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:25.704856 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:01:25.704810 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-74584bb9f8-5vj4p\" is forbidden: User \"system:node:ip-10-0-133-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-149.ec2.internal' and this object" podUID="125e9af0-f9ce-4f4e-9e09-a1168e9ab748" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:25.704856 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.704846 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 16 17:01:25.730053 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.730016 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p"] Jul 16 17:01:25.829226 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.829172 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e9af0-f9ce-4f4e-9e09-a1168e9ab748-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-5vj4p\" (UID: \"125e9af0-f9ce-4f4e-9e09-a1168e9ab748\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:25.829607 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.829314 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn28w\" (UniqueName: \"kubernetes.io/projected/125e9af0-f9ce-4f4e-9e09-a1168e9ab748-kube-api-access-bn28w\") pod \"payload-pre-processing-74584bb9f8-5vj4p\" (UID: \"125e9af0-f9ce-4f4e-9e09-a1168e9ab748\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:25.870482 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.870442 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" event={"ID":"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e","Type":"ContainerStarted","Data":"4b6eb3597080e8bd98953750e0a0d6fb272248ba03d25d2cf2dfde1673c6bef1"} Jul 16 17:01:25.930161 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.930115 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e9af0-f9ce-4f4e-9e09-a1168e9ab748-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-5vj4p\" (UID: \"125e9af0-f9ce-4f4e-9e09-a1168e9ab748\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:25.930348 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.930218 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn28w\" (UniqueName: \"kubernetes.io/projected/125e9af0-f9ce-4f4e-9e09-a1168e9ab748-kube-api-access-bn28w\") pod \"payload-pre-processing-74584bb9f8-5vj4p\" (UID: \"125e9af0-f9ce-4f4e-9e09-a1168e9ab748\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:25.930879 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.930860 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e9af0-f9ce-4f4e-9e09-a1168e9ab748-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-5vj4p\" (UID: \"125e9af0-f9ce-4f4e-9e09-a1168e9ab748\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:25.958090 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:25.958020 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn28w\" (UniqueName: \"kubernetes.io/projected/125e9af0-f9ce-4f4e-9e09-a1168e9ab748-kube-api-access-bn28w\") pod \"payload-pre-processing-74584bb9f8-5vj4p\" (UID: \"125e9af0-f9ce-4f4e-9e09-a1168e9ab748\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:26.010026 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:26.009984 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:26.189446 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:01:26.189394 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125e9af0_f9ce_4f4e_9e09_a1168e9ab748.slice/crio-f3de05a24a7613814985d5a0fe18524acdf25f7ce861cc7a1bef0952ae609ccb WatchSource:0}: Error finding container f3de05a24a7613814985d5a0fe18524acdf25f7ce861cc7a1bef0952ae609ccb: Status 404 returned error can't find the container with id f3de05a24a7613814985d5a0fe18524acdf25f7ce861cc7a1bef0952ae609ccb Jul 16 17:01:26.194707 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:26.194678 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p"] Jul 16 17:01:26.877754 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:26.877713 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" event={"ID":"125e9af0-f9ce-4f4e-9e09-a1168e9ab748","Type":"ContainerStarted","Data":"f3de05a24a7613814985d5a0fe18524acdf25f7ce861cc7a1bef0952ae609ccb"} Jul 16 17:01:27.884047 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:27.884006 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" event={"ID":"d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e","Type":"ContainerStarted","Data":"32be03b1dba014819f5f749115dddc023f09d7abf2592995c2c32358213afb3d"} Jul 16 17:01:27.884481 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:27.884097 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:27.911896 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:27.911848 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" podStartSLOduration=2.114056825 podStartE2EDuration="3.911832653s" totalImagesPullingTime="1.797775828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:01:27.910058285 +0000 UTC m=+617.694213347" watchObservedRunningTime="2026-07-16 17:01:27.911832653 +0000 UTC m=+617.695987731" Jul 16 17:01:28.891456 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:28.891351 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" event={"ID":"125e9af0-f9ce-4f4e-9e09-a1168e9ab748","Type":"ContainerStarted","Data":"f3dffcfa23f4370aeb689bf1c8b7cfbc8fc158f03de4ddb0388fed6376cd96d5"} Jul 16 17:01:28.912819 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:28.912761 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" podStartSLOduration=1.532014124 podStartE2EDuration="3.912743627s" totalImagesPullingTime="2.380729503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:01:28.9102322 +0000 UTC m=+618.694387262" watchObservedRunningTime="2026-07-16 17:01:28.912743627 +0000 UTC m=+618.696898688" Jul 16 17:01:29.896092 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:29.896059 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:33.897631 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:33.897550 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b7d967754-7qkq2" Jul 16 17:01:40.902232 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:40.902185 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5vj4p" Jul 16 17:01:41.077179 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.077137 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9"] Jul 16 17:01:41.079678 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.079654 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.082721 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.082698 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 17:01:41.082965 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.082934 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 17:01:41.083047 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.082976 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tgg58\"" Jul 16 17:01:41.083101 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.083067 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 16 17:01:41.097026 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.096996 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9"] Jul 16 17:01:41.178428 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.178322 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.178428 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.178368 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwnlh\" (UniqueName: \"kubernetes.io/projected/b069bf99-265b-49fc-b2f5-39d29242707d-kube-api-access-cwnlh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.178428 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.178392 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.178653 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.178449 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b069bf99-265b-49fc-b2f5-39d29242707d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.178653 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.178568 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.178653 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.178625 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.279288 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.279243 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.279288 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.279288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwnlh\" (UniqueName: \"kubernetes.io/projected/b069bf99-265b-49fc-b2f5-39d29242707d-kube-api-access-cwnlh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.279564 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.279308 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.279564 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.279331 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b069bf99-265b-49fc-b2f5-39d29242707d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.279564 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.279464 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.279564 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.279513 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.279779 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.279754 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.279841 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.279766 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.279897 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.279879 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.281780 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.281755 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b069bf99-265b-49fc-b2f5-39d29242707d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.281987 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.281966 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b069bf99-265b-49fc-b2f5-39d29242707d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.294123 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.294088 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwnlh\" (UniqueName: \"kubernetes.io/projected/b069bf99-265b-49fc-b2f5-39d29242707d-kube-api-access-cwnlh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9\" (UID: \"b069bf99-265b-49fc-b2f5-39d29242707d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.390102 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.390062 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:41.537507 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.537481 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9"] Jul 16 17:01:41.539641 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:01:41.539609 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb069bf99_265b_49fc_b2f5_39d29242707d.slice/crio-3ee7ebb2b85d5d1ca06108799dfdbbe113308209b3ecc51d22834832c77447aa WatchSource:0}: Error finding container 3ee7ebb2b85d5d1ca06108799dfdbbe113308209b3ecc51d22834832c77447aa: Status 404 returned error can't find the container with id 3ee7ebb2b85d5d1ca06108799dfdbbe113308209b3ecc51d22834832c77447aa Jul 16 17:01:41.956533 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:41.956494 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" event={"ID":"b069bf99-265b-49fc-b2f5-39d29242707d","Type":"ContainerStarted","Data":"3ee7ebb2b85d5d1ca06108799dfdbbe113308209b3ecc51d22834832c77447aa"} Jul 16 17:01:43.967377 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:43.967339 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" event={"ID":"b069bf99-265b-49fc-b2f5-39d29242707d","Type":"ContainerStarted","Data":"7ef623b90de20486438ff448fd669956fac8cf5e60c62c57049a3f335817b0a1"} Jul 16 17:01:44.008623 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:44.008576 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" podStartSLOduration=1.641609401 podStartE2EDuration="3.008558894s" totalImagesPullingTime="1.366949493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:01:44.007093811 +0000 UTC m=+633.791248873" watchObservedRunningTime="2026-07-16 17:01:44.008558894 +0000 UTC m=+633.792713955" Jul 16 17:01:47.322006 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.321967 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk"] Jul 16 17:01:47.324501 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.324482 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.328593 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.328566 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 16 17:01:47.361919 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.361884 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk"] Jul 16 17:01:47.441127 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.441088 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klwkr\" (UniqueName: \"kubernetes.io/projected/15758bdd-a304-43da-8ed4-2843f625bd04-kube-api-access-klwkr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.441299 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.441134 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.441299 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.441164 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.441453 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.441300 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.441453 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.441387 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.441453 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.441433 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15758bdd-a304-43da-8ed4-2843f625bd04-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.542163 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.542125 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.542163 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.542168 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.542431 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.542184 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15758bdd-a304-43da-8ed4-2843f625bd04-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.542431 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.542220 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klwkr\" (UniqueName: \"kubernetes.io/projected/15758bdd-a304-43da-8ed4-2843f625bd04-kube-api-access-klwkr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.542431 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.542253 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.542431 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.542280 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.542687 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.542659 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.542756 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.542689 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.542756 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.542736 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.544624 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.544601 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15758bdd-a304-43da-8ed4-2843f625bd04-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.544858 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.544839 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15758bdd-a304-43da-8ed4-2843f625bd04-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.555971 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.555946 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klwkr\" (UniqueName: \"kubernetes.io/projected/15758bdd-a304-43da-8ed4-2843f625bd04-kube-api-access-klwkr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zzssk\" (UID: \"15758bdd-a304-43da-8ed4-2843f625bd04\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.635488 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.635367 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:47.776313 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.776279 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk"] Jul 16 17:01:47.778598 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:01:47.778564 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15758bdd_a304_43da_8ed4_2843f625bd04.slice/crio-072c17ea86220f7b21b3fe9010a63c9467feec3ae1c81f29bf242ddce329d08c WatchSource:0}: Error finding container 072c17ea86220f7b21b3fe9010a63c9467feec3ae1c81f29bf242ddce329d08c: Status 404 returned error can't find the container with id 072c17ea86220f7b21b3fe9010a63c9467feec3ae1c81f29bf242ddce329d08c Jul 16 17:01:47.780121 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.780098 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:01:47.988858 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:47.988821 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" event={"ID":"15758bdd-a304-43da-8ed4-2843f625bd04","Type":"ContainerStarted","Data":"072c17ea86220f7b21b3fe9010a63c9467feec3ae1c81f29bf242ddce329d08c"} Jul 16 17:01:48.995301 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:48.995268 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" event={"ID":"15758bdd-a304-43da-8ed4-2843f625bd04","Type":"ContainerStarted","Data":"9d90371f6295c500522f57a83820008f3f9acbcab12a4ba31c8aa192bc4fb52b"} Jul 16 17:01:51.391284 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:51.391224 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:51.391794 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:51.391321 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:51.403968 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:51.403929 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:51.430199 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:51.430141 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" podStartSLOduration=4.248768368 podStartE2EDuration="4.430122159s" totalImagesPullingTime="181.353791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:01:49.038755908 +0000 UTC m=+638.822910971" watchObservedRunningTime="2026-07-16 17:01:51.430122159 +0000 UTC m=+641.214277221" Jul 16 17:01:52.020590 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:52.020557 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9" Jul 16 17:01:57.635508 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:57.635475 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:57.635909 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:57.635529 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:57.647715 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:57.647684 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:01:58.049307 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:01:58.049269 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zzssk" Jul 16 17:02:03.376671 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.376612 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb"] Jul 16 17:02:03.380061 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.380038 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.382566 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.382544 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 16 17:02:03.392483 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.392461 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb"] Jul 16 17:02:03.487892 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.487859 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf2xd\" (UniqueName: \"kubernetes.io/projected/37e5eecc-f207-49d9-bf13-888ac1e6d94e-kube-api-access-mf2xd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.488080 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.487905 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.488080 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.487938 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.488080 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.488004 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.488217 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.488154 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e5eecc-f207-49d9-bf13-888ac1e6d94e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.488217 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.488190 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.589581 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.589539 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e5eecc-f207-49d9-bf13-888ac1e6d94e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.589762 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.589595 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.589762 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.589624 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf2xd\" (UniqueName: \"kubernetes.io/projected/37e5eecc-f207-49d9-bf13-888ac1e6d94e-kube-api-access-mf2xd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.589762 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.589679 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.589762 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.589720 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.589986 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.589763 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.590096 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.590070 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.590190 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.590169 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.590246 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.590223 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.592267 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.592239 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e5eecc-f207-49d9-bf13-888ac1e6d94e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.592267 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.592267 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e5eecc-f207-49d9-bf13-888ac1e6d94e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.603146 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.603117 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf2xd\" (UniqueName: \"kubernetes.io/projected/37e5eecc-f207-49d9-bf13-888ac1e6d94e-kube-api-access-mf2xd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hbhhb\" (UID: \"37e5eecc-f207-49d9-bf13-888ac1e6d94e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.691802 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.691706 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:03.840829 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:03.840795 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb"] Jul 16 17:02:03.843542 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:02:03.843512 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e5eecc_f207_49d9_bf13_888ac1e6d94e.slice/crio-a9be5d00ff3511c315235e094e2f524791b1115db953939bc5cfa215e4a84159 WatchSource:0}: Error finding container a9be5d00ff3511c315235e094e2f524791b1115db953939bc5cfa215e4a84159: Status 404 returned error can't find the container with id a9be5d00ff3511c315235e094e2f524791b1115db953939bc5cfa215e4a84159 Jul 16 17:02:04.066918 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:04.066888 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" event={"ID":"37e5eecc-f207-49d9-bf13-888ac1e6d94e","Type":"ContainerStarted","Data":"a9be5d00ff3511c315235e094e2f524791b1115db953939bc5cfa215e4a84159"} Jul 16 17:02:05.073371 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:05.073327 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" event={"ID":"37e5eecc-f207-49d9-bf13-888ac1e6d94e","Type":"ContainerStarted","Data":"cbf146a9cfeb9498c98fb70220880c77469b0227da81fb6bd35dd1016b8f97dd"} Jul 16 17:02:05.126736 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:05.126673 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" podStartSLOduration=1.934586882 podStartE2EDuration="2.126654779s" totalImagesPullingTime="192.067897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:02:05.12361334 +0000 UTC m=+654.907768403" watchObservedRunningTime="2026-07-16 17:02:05.126654779 +0000 UTC m=+654.910809841" Jul 16 17:02:13.692334 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:13.692294 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:13.692995 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:13.692380 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:13.705035 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:13.705016 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:02:14.127117 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:02:14.127089 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hbhhb" Jul 16 17:04:53.497114 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.497075 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-758884f967-2n557"] Jul 16 17:04:53.497656 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.497318 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-758884f967-2n557" podUID="d71b8602-eb9e-483e-a749-e20a4ac2c5ae" containerName="manager" containerID="cri-o://a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc" gracePeriod=30 Jul 16 17:04:53.753209 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.753146 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:04:53.890589 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.890558 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcxwv\" (UniqueName: \"kubernetes.io/projected/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-kube-api-access-bcxwv\") pod \"d71b8602-eb9e-483e-a749-e20a4ac2c5ae\" (UID: \"d71b8602-eb9e-483e-a749-e20a4ac2c5ae\") " Jul 16 17:04:53.890757 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.890680 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-webhook-cert\") pod \"d71b8602-eb9e-483e-a749-e20a4ac2c5ae\" (UID: \"d71b8602-eb9e-483e-a749-e20a4ac2c5ae\") " Jul 16 17:04:53.892686 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.892646 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-kube-api-access-bcxwv" (OuterVolumeSpecName: "kube-api-access-bcxwv") pod "d71b8602-eb9e-483e-a749-e20a4ac2c5ae" (UID: "d71b8602-eb9e-483e-a749-e20a4ac2c5ae"). InnerVolumeSpecName "kube-api-access-bcxwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:04:53.892796 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.892701 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d71b8602-eb9e-483e-a749-e20a4ac2c5ae" (UID: "d71b8602-eb9e-483e-a749-e20a4ac2c5ae"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:04:53.911590 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.911562 2556 generic.go:356] "Generic (PLEG): container finished" podID="d71b8602-eb9e-483e-a749-e20a4ac2c5ae" containerID="a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc" exitCode=0 Jul 16 17:04:53.911722 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.911601 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-758884f967-2n557" Jul 16 17:04:53.911722 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.911668 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-758884f967-2n557" event={"ID":"d71b8602-eb9e-483e-a749-e20a4ac2c5ae","Type":"ContainerDied","Data":"a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc"} Jul 16 17:04:53.911722 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.911705 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-758884f967-2n557" event={"ID":"d71b8602-eb9e-483e-a749-e20a4ac2c5ae","Type":"ContainerDied","Data":"8c3245d2fd50d08b9395634cecf468c86702313be9072f772d78f2d52677cfa6"} Jul 16 17:04:53.911829 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.911725 2556 scope.go:117] "RemoveContainer" containerID="a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc" Jul 16 17:04:53.920967 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.920947 2556 scope.go:117] "RemoveContainer" containerID="a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc" Jul 16 17:04:53.921238 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:04:53.921216 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc\": container with ID starting with a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc not found: ID does not exist" containerID="a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc" Jul 16 17:04:53.921297 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.921245 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc"} err="failed to get container status \"a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc\": rpc error: code = NotFound desc = could not find container \"a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc\": container with ID starting with a3167bcd0835bfd244806ff5ff890591f0d6e0a7095f3c91f94239f880ff95cc not found: ID does not exist" Jul 16 17:04:53.937825 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.937802 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-758884f967-2n557"] Jul 16 17:04:53.942950 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.942926 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-758884f967-2n557"] Jul 16 17:04:53.991814 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.991787 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-webhook-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:04:53.991814 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:53.991812 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcxwv\" (UniqueName: \"kubernetes.io/projected/d71b8602-eb9e-483e-a749-e20a4ac2c5ae-kube-api-access-bcxwv\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:04:54.710263 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.710209 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-758884f967-2n557" podUID="d71b8602-eb9e-483e-a749-e20a4ac2c5ae" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 17:04:54.785954 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.785909 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-758884f967-bb8k6"] Jul 16 17:04:54.786408 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.786390 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71b8602-eb9e-483e-a749-e20a4ac2c5ae" containerName="manager" Jul 16 17:04:54.786408 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.786425 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71b8602-eb9e-483e-a749-e20a4ac2c5ae" containerName="manager" Jul 16 17:04:54.786621 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.786607 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d71b8602-eb9e-483e-a749-e20a4ac2c5ae" containerName="manager" Jul 16 17:04:54.789246 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.789226 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-758884f967-bb8k6" Jul 16 17:04:54.792790 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.792769 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5bw72\"" Jul 16 17:04:54.792892 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.792799 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 16 17:04:54.792892 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.792769 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 16 17:04:54.800696 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.800675 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-758884f967-bb8k6"] Jul 16 17:04:54.843521 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.843485 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d71b8602-eb9e-483e-a749-e20a4ac2c5ae" path="/var/lib/kubelet/pods/d71b8602-eb9e-483e-a749-e20a4ac2c5ae/volumes" Jul 16 17:04:54.899871 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.899833 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35e2ecd9-785b-48e2-b26f-be44881d505f-webhook-cert\") pod \"maas-controller-758884f967-bb8k6\" (UID: \"35e2ecd9-785b-48e2-b26f-be44881d505f\") " pod="opendatahub/maas-controller-758884f967-bb8k6" Jul 16 17:04:54.900071 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:54.899904 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8vmv\" (UniqueName: \"kubernetes.io/projected/35e2ecd9-785b-48e2-b26f-be44881d505f-kube-api-access-l8vmv\") pod \"maas-controller-758884f967-bb8k6\" (UID: \"35e2ecd9-785b-48e2-b26f-be44881d505f\") " pod="opendatahub/maas-controller-758884f967-bb8k6" Jul 16 17:04:55.001119 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:55.001083 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35e2ecd9-785b-48e2-b26f-be44881d505f-webhook-cert\") pod \"maas-controller-758884f967-bb8k6\" (UID: \"35e2ecd9-785b-48e2-b26f-be44881d505f\") " pod="opendatahub/maas-controller-758884f967-bb8k6" Jul 16 17:04:55.001274 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:55.001143 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8vmv\" (UniqueName: \"kubernetes.io/projected/35e2ecd9-785b-48e2-b26f-be44881d505f-kube-api-access-l8vmv\") pod \"maas-controller-758884f967-bb8k6\" (UID: \"35e2ecd9-785b-48e2-b26f-be44881d505f\") " pod="opendatahub/maas-controller-758884f967-bb8k6" Jul 16 17:04:55.003542 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:55.003509 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35e2ecd9-785b-48e2-b26f-be44881d505f-webhook-cert\") pod \"maas-controller-758884f967-bb8k6\" (UID: \"35e2ecd9-785b-48e2-b26f-be44881d505f\") " pod="opendatahub/maas-controller-758884f967-bb8k6" Jul 16 17:04:55.010105 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:55.010082 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8vmv\" (UniqueName: \"kubernetes.io/projected/35e2ecd9-785b-48e2-b26f-be44881d505f-kube-api-access-l8vmv\") pod \"maas-controller-758884f967-bb8k6\" (UID: \"35e2ecd9-785b-48e2-b26f-be44881d505f\") " pod="opendatahub/maas-controller-758884f967-bb8k6" Jul 16 17:04:55.100571 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:55.100539 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-758884f967-bb8k6" Jul 16 17:04:55.239606 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:55.239581 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-758884f967-bb8k6"] Jul 16 17:04:55.241670 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:04:55.241637 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e2ecd9_785b_48e2_b26f_be44881d505f.slice/crio-1388da09eb7a3c71a3daeec08abd0f7a5419a62785eab83f979269ff4692dbef WatchSource:0}: Error finding container 1388da09eb7a3c71a3daeec08abd0f7a5419a62785eab83f979269ff4692dbef: Status 404 returned error can't find the container with id 1388da09eb7a3c71a3daeec08abd0f7a5419a62785eab83f979269ff4692dbef Jul 16 17:04:55.923716 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:55.923679 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-758884f967-bb8k6" event={"ID":"35e2ecd9-785b-48e2-b26f-be44881d505f","Type":"ContainerStarted","Data":"82131486fe4d9849bc80159f6b5f8fe25de235a8d768f11a1c97b479d77f257d"} Jul 16 17:04:55.924107 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:55.923740 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-758884f967-bb8k6" event={"ID":"35e2ecd9-785b-48e2-b26f-be44881d505f","Type":"ContainerStarted","Data":"1388da09eb7a3c71a3daeec08abd0f7a5419a62785eab83f979269ff4692dbef"} Jul 16 17:04:55.924107 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:55.923770 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-758884f967-bb8k6" Jul 16 17:04:55.950147 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:04:55.950053 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-758884f967-bb8k6" podStartSLOduration=1.5068534869999999 podStartE2EDuration="1.95003883s" totalImagesPullingTime="443.185343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:04:55.948730756 +0000 UTC m=+825.732885817" watchObservedRunningTime="2026-07-16 17:04:55.95003883 +0000 UTC m=+825.734193892" Jul 16 17:05:06.931858 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:05:06.931826 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-758884f967-bb8k6" Jul 16 17:06:10.754597 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:06:10.754530 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:06:10.756031 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:06:10.756014 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:11:10.789088 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:11:10.789059 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:11:10.790906 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:11:10.790885 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:16:10.823658 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:16:10.823631 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:16:10.825005 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:16:10.824977 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:21:10.856723 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:21:10.856696 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:21:10.858619 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:21:10.858599 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:23:26.110670 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.110633 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg"] Jul 16 17:23:26.114225 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.114197 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.117773 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.117750 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-901aa3\"/\"test-model-901aa3-kserve-self-signed-certs\"" Jul 16 17:23:26.117916 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.117835 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-901aa3\"/\"openshift-service-ca.crt\"" Jul 16 17:23:26.117916 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.117855 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-901aa3\"/\"default-dockercfg-r456n\"" Jul 16 17:23:26.117916 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.117904 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-901aa3\"/\"kube-root-ca.crt\"" Jul 16 17:23:26.126899 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.126875 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg"] Jul 16 17:23:26.293104 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.293071 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-dshm\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.293275 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.293149 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-model-cache\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.293275 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.293187 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pw94\" (UniqueName: \"kubernetes.io/projected/6ce808cf-b6c6-4ae8-830f-f4159ea62732-kube-api-access-5pw94\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.293275 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.293225 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tls-certs\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.293275 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.293249 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-home\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.293275 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.293268 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tmp-dir\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.394589 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.394504 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-dshm\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.394589 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.394575 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-model-cache\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.394771 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.394609 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pw94\" (UniqueName: \"kubernetes.io/projected/6ce808cf-b6c6-4ae8-830f-f4159ea62732-kube-api-access-5pw94\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.394771 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.394639 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tls-certs\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.394771 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.394659 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-home\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.394771 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.394676 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tmp-dir\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.395073 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.395052 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-model-cache\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.395130 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.395083 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tmp-dir\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.395130 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.395100 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-home\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.396747 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.396725 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-dshm\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.397006 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.396990 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tls-certs\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.404980 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.404955 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pw94\" (UniqueName: \"kubernetes.io/projected/6ce808cf-b6c6-4ae8-830f-f4159ea62732-kube-api-access-5pw94\") pod \"test-model-901aa3-kserve-c584df699-fbnkg\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.425583 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.425559 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:26.580908 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:26.580865 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg"] Jul 16 17:23:26.582091 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:23:26.582065 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ce808cf_b6c6_4ae8_830f_f4159ea62732.slice/crio-d7529e90291e05c3e952026e6c73a6842fdc524cb6eaa43dbed8657adea67ea8 WatchSource:0}: Error finding container d7529e90291e05c3e952026e6c73a6842fdc524cb6eaa43dbed8657adea67ea8: Status 404 returned error can't find the container with id d7529e90291e05c3e952026e6c73a6842fdc524cb6eaa43dbed8657adea67ea8 Jul 16 17:23:27.395906 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:27.395869 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" event={"ID":"6ce808cf-b6c6-4ae8-830f-f4159ea62732","Type":"ContainerStarted","Data":"89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123"} Jul 16 17:23:27.395906 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:27.395912 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" event={"ID":"6ce808cf-b6c6-4ae8-830f-f4159ea62732","Type":"ContainerStarted","Data":"d7529e90291e05c3e952026e6c73a6842fdc524cb6eaa43dbed8657adea67ea8"} Jul 16 17:23:27.421625 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:27.421574 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" podStartSLOduration=1.421558302 podStartE2EDuration="1.421558302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:23:27.42039397 +0000 UTC m=+1937.204549048" watchObservedRunningTime="2026-07-16 17:23:27.421558302 +0000 UTC m=+1937.205713362" Jul 16 17:23:36.426268 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:36.426230 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:36.426750 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:36.426289 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:36.438118 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:36.438095 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:23:36.452658 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:23:36.452638 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:24:17.928906 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:17.928824 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg"] Jul 16 17:24:32.948531 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:32.948477 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" podUID="6ce808cf-b6c6-4ae8-830f-f4159ea62732" containerName="main" containerID="cri-o://89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123" gracePeriod=45 Jul 16 17:24:33.202743 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.202685 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:24:33.323937 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.323905 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-dshm\") pod \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " Jul 16 17:24:33.324082 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.323975 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pw94\" (UniqueName: \"kubernetes.io/projected/6ce808cf-b6c6-4ae8-830f-f4159ea62732-kube-api-access-5pw94\") pod \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " Jul 16 17:24:33.324082 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.324021 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tmp-dir\") pod \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " Jul 16 17:24:33.324168 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.324083 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-model-cache\") pod \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " Jul 16 17:24:33.324168 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.324099 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tls-certs\") pod \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " Jul 16 17:24:33.324168 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.324143 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-home\") pod \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\" (UID: \"6ce808cf-b6c6-4ae8-830f-f4159ea62732\") " Jul 16 17:24:33.324322 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.324289 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ce808cf-b6c6-4ae8-830f-f4159ea62732" (UID: "6ce808cf-b6c6-4ae8-830f-f4159ea62732"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:24:33.324389 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.324322 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ce808cf-b6c6-4ae8-830f-f4159ea62732" (UID: "6ce808cf-b6c6-4ae8-830f-f4159ea62732"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:24:33.324494 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.324470 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-home" (OuterVolumeSpecName: "home") pod "6ce808cf-b6c6-4ae8-830f-f4159ea62732" (UID: "6ce808cf-b6c6-4ae8-830f-f4159ea62732"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:24:33.324548 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.324484 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:24:33.324548 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.324515 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:24:33.326138 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.326117 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce808cf-b6c6-4ae8-830f-f4159ea62732-kube-api-access-5pw94" (OuterVolumeSpecName: "kube-api-access-5pw94") pod "6ce808cf-b6c6-4ae8-830f-f4159ea62732" (UID: "6ce808cf-b6c6-4ae8-830f-f4159ea62732"). InnerVolumeSpecName "kube-api-access-5pw94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:24:33.326221 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.326141 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ce808cf-b6c6-4ae8-830f-f4159ea62732" (UID: "6ce808cf-b6c6-4ae8-830f-f4159ea62732"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:24:33.326221 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.326207 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-dshm" (OuterVolumeSpecName: "dshm") pod "6ce808cf-b6c6-4ae8-830f-f4159ea62732" (UID: "6ce808cf-b6c6-4ae8-830f-f4159ea62732"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:24:33.425351 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.425327 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:24:33.425351 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.425347 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce808cf-b6c6-4ae8-830f-f4159ea62732-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:24:33.425521 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.425355 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pw94\" (UniqueName: \"kubernetes.io/projected/6ce808cf-b6c6-4ae8-830f-f4159ea62732-kube-api-access-5pw94\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:24:33.425521 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.425366 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce808cf-b6c6-4ae8-830f-f4159ea62732-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:24:33.727610 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.727578 2556 generic.go:356] "Generic (PLEG): container finished" podID="6ce808cf-b6c6-4ae8-830f-f4159ea62732" containerID="89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123" exitCode=0 Jul 16 17:24:33.727764 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.727617 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" event={"ID":"6ce808cf-b6c6-4ae8-830f-f4159ea62732","Type":"ContainerDied","Data":"89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123"} Jul 16 17:24:33.727764 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.727619 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" Jul 16 17:24:33.727764 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.727652 2556 scope.go:117] "RemoveContainer" containerID="89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123" Jul 16 17:24:33.727764 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.727643 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg" event={"ID":"6ce808cf-b6c6-4ae8-830f-f4159ea62732","Type":"ContainerDied","Data":"d7529e90291e05c3e952026e6c73a6842fdc524cb6eaa43dbed8657adea67ea8"} Jul 16 17:24:33.738696 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.738675 2556 scope.go:117] "RemoveContainer" containerID="89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123" Jul 16 17:24:33.738982 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:24:33.738959 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123\": container with ID starting with 89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123 not found: ID does not exist" containerID="89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123" Jul 16 17:24:33.739028 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.738994 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123"} err="failed to get container status \"89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123\": rpc error: code = NotFound desc = could not find container \"89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123\": container with ID starting with 89d9d4190df22282e6b7be220357bc244bd9202e001ec4f3d2196a2354775123 not found: ID does not exist" Jul 16 17:24:33.759157 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.759138 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg"] Jul 16 17:24:33.762431 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:33.762396 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-901aa3/test-model-901aa3-kserve-c584df699-fbnkg"] Jul 16 17:24:34.842526 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:24:34.842494 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ce808cf-b6c6-4ae8-830f-f4159ea62732" path="/var/lib/kubelet/pods/6ce808cf-b6c6-4ae8-830f-f4159ea62732/volumes" Jul 16 17:26:10.891245 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:26:10.891210 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:26:10.894130 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:26:10.894101 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:27:00.578947 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.578910 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh"] Jul 16 17:27:00.579364 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.579333 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce808cf-b6c6-4ae8-830f-f4159ea62732" containerName="main" Jul 16 17:27:00.579364 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.579345 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce808cf-b6c6-4ae8-830f-f4159ea62732" containerName="main" Jul 16 17:27:00.579467 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.579400 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce808cf-b6c6-4ae8-830f-f4159ea62732" containerName="main" Jul 16 17:27:00.582733 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.582715 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.591722 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.591695 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b44c0f\"/\"auth-test-model-b44c0f-kserve-self-signed-certs\"" Jul 16 17:27:00.591864 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.591705 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b44c0f\"/\"kube-root-ca.crt\"" Jul 16 17:27:00.591864 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.591760 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b44c0f\"/\"default-dockercfg-2r7cx\"" Jul 16 17:27:00.591864 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.591765 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b44c0f\"/\"openshift-service-ca.crt\"" Jul 16 17:27:00.600369 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.600344 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh"] Jul 16 17:27:00.622927 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.622899 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a935aad9-8398-416a-871f-6cd3b91f7fbf-tls-certs\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.623087 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.622937 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-home\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.623087 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.622973 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pkmj\" (UniqueName: \"kubernetes.io/projected/a935aad9-8398-416a-871f-6cd3b91f7fbf-kube-api-access-6pkmj\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.623087 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.623035 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-tmp-dir\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.623087 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.623075 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-model-cache\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.623250 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.623145 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-dshm\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.724464 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.724433 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-tmp-dir\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.724666 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.724486 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-model-cache\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.724666 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.724525 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-dshm\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.724666 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.724604 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a935aad9-8398-416a-871f-6cd3b91f7fbf-tls-certs\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.724666 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.724635 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-home\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.724666 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.724663 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkmj\" (UniqueName: \"kubernetes.io/projected/a935aad9-8398-416a-871f-6cd3b91f7fbf-kube-api-access-6pkmj\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.724946 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.724877 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-tmp-dir\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.725003 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.724961 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-home\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.725003 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.724971 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-model-cache\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.726923 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.726897 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-dshm\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.727198 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.727176 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a935aad9-8398-416a-871f-6cd3b91f7fbf-tls-certs\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.741071 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.741042 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkmj\" (UniqueName: \"kubernetes.io/projected/a935aad9-8398-416a-871f-6cd3b91f7fbf-kube-api-access-6pkmj\") pod \"auth-test-model-b44c0f-kserve-84bb4755cf-688lh\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:00.891681 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:00.891585 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:01.041485 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:01.041454 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh"] Jul 16 17:27:01.043810 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:27:01.043766 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda935aad9_8398_416a_871f_6cd3b91f7fbf.slice/crio-a4ea419d6287625f191da53b3ff3a6b5f121032db9694c88f3a4f6fade5af048 WatchSource:0}: Error finding container a4ea419d6287625f191da53b3ff3a6b5f121032db9694c88f3a4f6fade5af048: Status 404 returned error can't find the container with id a4ea419d6287625f191da53b3ff3a6b5f121032db9694c88f3a4f6fade5af048 Jul 16 17:27:01.446392 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:01.446354 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" event={"ID":"a935aad9-8398-416a-871f-6cd3b91f7fbf","Type":"ContainerStarted","Data":"2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a"} Jul 16 17:27:01.446392 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:01.446397 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" event={"ID":"a935aad9-8398-416a-871f-6cd3b91f7fbf","Type":"ContainerStarted","Data":"a4ea419d6287625f191da53b3ff3a6b5f121032db9694c88f3a4f6fade5af048"} Jul 16 17:27:01.474405 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:01.474358 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" podStartSLOduration=1.474343161 podStartE2EDuration="1.474343161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:27:01.473761778 +0000 UTC m=+2151.257916840" watchObservedRunningTime="2026-07-16 17:27:01.474343161 +0000 UTC m=+2151.258498222" Jul 16 17:27:10.892735 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:10.892643 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:10.892735 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:10.892702 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:10.906182 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:10.906156 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:27:11.509463 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:27:11.509407 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:29:57.899375 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:29:57.899331 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh"] Jul 16 17:30:00.154123 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:00.154083 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq"] Jul 16 17:30:00.157890 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:00.157870 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq" Jul 16 17:30:00.160350 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:00.160328 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t6smk\"" Jul 16 17:30:00.170699 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:00.170675 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq" Jul 16 17:30:00.174835 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:00.174804 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq"] Jul 16 17:30:00.325573 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:00.325540 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq"] Jul 16 17:30:00.328641 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:30:00.328608 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123c50df_6a9a_4824_b04e_bd62f09d1824.slice/crio-de9e90b953de81bb180c673b06eedbc15cfcbaac8bc8bc19fe47f822d038f755 WatchSource:0}: Error finding container de9e90b953de81bb180c673b06eedbc15cfcbaac8bc8bc19fe47f822d038f755: Status 404 returned error can't find the container with id de9e90b953de81bb180c673b06eedbc15cfcbaac8bc8bc19fe47f822d038f755 Jul 16 17:30:00.330659 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:00.330640 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:30:00.344054 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:00.344023 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq" event={"ID":"123c50df-6a9a-4824-b04e-bd62f09d1824","Type":"ContainerStarted","Data":"de9e90b953de81bb180c673b06eedbc15cfcbaac8bc8bc19fe47f822d038f755"} Jul 16 17:30:03.361037 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:03.360994 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq" event={"ID":"123c50df-6a9a-4824-b04e-bd62f09d1824","Type":"ContainerStarted","Data":"b1d5335f4f735c90cf95c0c0338ad5b73f7144f4884fbf846e2215f0b13ddb9e"} Jul 16 17:30:03.390465 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:03.390392 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq" podStartSLOduration=0.726053257 podStartE2EDuration="3.390374519s" totalImagesPullingTime="2.664321262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:03.389719351 +0000 UTC m=+2333.173874414" watchObservedRunningTime="2026-07-16 17:30:03.390374519 +0000 UTC m=+2333.174529581" Jul 16 17:30:04.367631 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:04.367589 2556 generic.go:356] "Generic (PLEG): container finished" podID="123c50df-6a9a-4824-b04e-bd62f09d1824" containerID="b1d5335f4f735c90cf95c0c0338ad5b73f7144f4884fbf846e2215f0b13ddb9e" exitCode=0 Jul 16 17:30:04.368023 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:04.367671 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq" event={"ID":"123c50df-6a9a-4824-b04e-bd62f09d1824","Type":"ContainerDied","Data":"b1d5335f4f735c90cf95c0c0338ad5b73f7144f4884fbf846e2215f0b13ddb9e"} Jul 16 17:30:05.512732 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:05.512701 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq" Jul 16 17:30:06.380238 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:06.380210 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq" Jul 16 17:30:06.380435 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:06.380250 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq" event={"ID":"123c50df-6a9a-4824-b04e-bd62f09d1824","Type":"ContainerDied","Data":"de9e90b953de81bb180c673b06eedbc15cfcbaac8bc8bc19fe47f822d038f755"} Jul 16 17:30:06.380435 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:06.380297 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de9e90b953de81bb180c673b06eedbc15cfcbaac8bc8bc19fe47f822d038f755" Jul 16 17:30:12.921646 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:12.921583 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" podUID="a935aad9-8398-416a-871f-6cd3b91f7fbf" containerName="main" containerID="cri-o://2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a" gracePeriod=45 Jul 16 17:30:13.178011 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.177946 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:30:13.284125 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.284082 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-home\") pod \"a935aad9-8398-416a-871f-6cd3b91f7fbf\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " Jul 16 17:30:13.284125 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.284116 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-dshm\") pod \"a935aad9-8398-416a-871f-6cd3b91f7fbf\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " Jul 16 17:30:13.284384 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.284153 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a935aad9-8398-416a-871f-6cd3b91f7fbf-tls-certs\") pod \"a935aad9-8398-416a-871f-6cd3b91f7fbf\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " Jul 16 17:30:13.284384 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.284172 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-tmp-dir\") pod \"a935aad9-8398-416a-871f-6cd3b91f7fbf\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " Jul 16 17:30:13.284384 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.284205 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pkmj\" (UniqueName: \"kubernetes.io/projected/a935aad9-8398-416a-871f-6cd3b91f7fbf-kube-api-access-6pkmj\") pod \"a935aad9-8398-416a-871f-6cd3b91f7fbf\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " Jul 16 17:30:13.284384 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.284222 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-model-cache\") pod \"a935aad9-8398-416a-871f-6cd3b91f7fbf\" (UID: \"a935aad9-8398-416a-871f-6cd3b91f7fbf\") " Jul 16 17:30:13.284633 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.284448 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-home" (OuterVolumeSpecName: "home") pod "a935aad9-8398-416a-871f-6cd3b91f7fbf" (UID: "a935aad9-8398-416a-871f-6cd3b91f7fbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:30:13.284633 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.284563 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a935aad9-8398-416a-871f-6cd3b91f7fbf" (UID: "a935aad9-8398-416a-871f-6cd3b91f7fbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:30:13.284633 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.284610 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:30:13.284797 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.284649 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-model-cache" (OuterVolumeSpecName: "model-cache") pod "a935aad9-8398-416a-871f-6cd3b91f7fbf" (UID: "a935aad9-8398-416a-871f-6cd3b91f7fbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:30:13.286682 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.286649 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-dshm" (OuterVolumeSpecName: "dshm") pod "a935aad9-8398-416a-871f-6cd3b91f7fbf" (UID: "a935aad9-8398-416a-871f-6cd3b91f7fbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:30:13.286800 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.286679 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a935aad9-8398-416a-871f-6cd3b91f7fbf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a935aad9-8398-416a-871f-6cd3b91f7fbf" (UID: "a935aad9-8398-416a-871f-6cd3b91f7fbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:30:13.286800 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.286694 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a935aad9-8398-416a-871f-6cd3b91f7fbf-kube-api-access-6pkmj" (OuterVolumeSpecName: "kube-api-access-6pkmj") pod "a935aad9-8398-416a-871f-6cd3b91f7fbf" (UID: "a935aad9-8398-416a-871f-6cd3b91f7fbf"). InnerVolumeSpecName "kube-api-access-6pkmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:30:13.385112 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.385076 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:30:13.385112 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.385106 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a935aad9-8398-416a-871f-6cd3b91f7fbf-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:30:13.385315 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.385136 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:30:13.385315 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.385145 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pkmj\" (UniqueName: \"kubernetes.io/projected/a935aad9-8398-416a-871f-6cd3b91f7fbf-kube-api-access-6pkmj\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:30:13.385315 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.385154 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a935aad9-8398-416a-871f-6cd3b91f7fbf-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:30:13.418356 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.418316 2556 generic.go:356] "Generic (PLEG): container finished" podID="a935aad9-8398-416a-871f-6cd3b91f7fbf" containerID="2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a" exitCode=0 Jul 16 17:30:13.418555 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.418363 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" Jul 16 17:30:13.418555 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.418377 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" event={"ID":"a935aad9-8398-416a-871f-6cd3b91f7fbf","Type":"ContainerDied","Data":"2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a"} Jul 16 17:30:13.418555 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.418438 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh" event={"ID":"a935aad9-8398-416a-871f-6cd3b91f7fbf","Type":"ContainerDied","Data":"a4ea419d6287625f191da53b3ff3a6b5f121032db9694c88f3a4f6fade5af048"} Jul 16 17:30:13.418555 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.418459 2556 scope.go:117] "RemoveContainer" containerID="2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a" Jul 16 17:30:13.427744 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.427723 2556 scope.go:117] "RemoveContainer" containerID="2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a" Jul 16 17:30:13.428076 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:30:13.428031 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a\": container with ID starting with 2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a not found: ID does not exist" containerID="2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a" Jul 16 17:30:13.428076 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.428059 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a"} err="failed to get container status \"2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a\": rpc error: code = NotFound desc = could not find container \"2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a\": container with ID starting with 2c7b3e6194fbeb511deaa19d3b4ca3e05f40cc5ba2742c2378ddea123028258a not found: ID does not exist" Jul 16 17:30:13.446772 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.446737 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh"] Jul 16 17:30:13.456154 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:13.456122 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/auth-test-model-b44c0f-kserve-84bb4755cf-688lh"] Jul 16 17:30:14.843234 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:14.843197 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a935aad9-8398-416a-871f-6cd3b91f7fbf" path="/var/lib/kubelet/pods/a935aad9-8398-416a-871f-6cd3b91f7fbf/volumes" Jul 16 17:30:17.176150 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.176110 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z"] Jul 16 17:30:17.176569 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.176551 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a935aad9-8398-416a-871f-6cd3b91f7fbf" containerName="main" Jul 16 17:30:17.176569 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.176564 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a935aad9-8398-416a-871f-6cd3b91f7fbf" containerName="main" Jul 16 17:30:17.176650 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.176573 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123c50df-6a9a-4824-b04e-bd62f09d1824" containerName="cleanup" Jul 16 17:30:17.176650 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.176582 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="123c50df-6a9a-4824-b04e-bd62f09d1824" containerName="cleanup" Jul 16 17:30:17.176650 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.176639 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="123c50df-6a9a-4824-b04e-bd62f09d1824" containerName="cleanup" Jul 16 17:30:17.176741 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.176652 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a935aad9-8398-416a-871f-6cd3b91f7fbf" containerName="main" Jul 16 17:30:17.179724 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.179706 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.192709 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.192685 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b44c0f\"/\"default-dockercfg-2r7cx\"" Jul 16 17:30:17.192865 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.192742 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b44c0f\"/\"openshift-service-ca.crt\"" Jul 16 17:30:17.193016 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.192999 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b44c0f\"/\"kube-root-ca.crt\"" Jul 16 17:30:17.197369 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.197345 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z"] Jul 16 17:30:17.197513 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.197384 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b44c0f\"/\"sub-test-model-b44c0f-kserve-self-signed-certs\"" Jul 16 17:30:17.221932 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.221900 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjckw\" (UniqueName: \"kubernetes.io/projected/2dbe070f-149b-4b17-ad50-a6b512f07e8e-kube-api-access-tjckw\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.222119 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.221946 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-model-cache\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.222119 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.222077 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tmp-dir\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.222119 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.222110 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tls-certs\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.222242 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.222173 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-dshm\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.222242 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.222215 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-home\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.323340 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.323297 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-dshm\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.323563 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.323351 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-home\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.323563 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.323381 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjckw\" (UniqueName: \"kubernetes.io/projected/2dbe070f-149b-4b17-ad50-a6b512f07e8e-kube-api-access-tjckw\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.323563 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.323479 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-model-cache\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.323686 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.323612 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tmp-dir\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.323686 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.323650 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tls-certs\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.323929 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.323896 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-home\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.324052 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.323936 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-model-cache\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.324052 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.324025 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tmp-dir\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.325819 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.325799 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-dshm\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.325987 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.325970 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tls-certs\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.339139 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.339102 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjckw\" (UniqueName: \"kubernetes.io/projected/2dbe070f-149b-4b17-ad50-a6b512f07e8e-kube-api-access-tjckw\") pod \"sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.489964 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.489912 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:17.659562 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:17.659540 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z"] Jul 16 17:30:17.661857 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:30:17.661823 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dbe070f_149b_4b17_ad50_a6b512f07e8e.slice/crio-847addd00cf160fbd54adcc0606b152eb22847cbf2bb2f5973e4b4652bd6c7a3 WatchSource:0}: Error finding container 847addd00cf160fbd54adcc0606b152eb22847cbf2bb2f5973e4b4652bd6c7a3: Status 404 returned error can't find the container with id 847addd00cf160fbd54adcc0606b152eb22847cbf2bb2f5973e4b4652bd6c7a3 Jul 16 17:30:18.446788 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:18.446749 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" event={"ID":"2dbe070f-149b-4b17-ad50-a6b512f07e8e","Type":"ContainerStarted","Data":"9aa75a2cd1f8837094c977f1fdc025593c84f00b4bd34a2a18ad15f6c1a54c7e"} Jul 16 17:30:18.446788 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:18.446792 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" event={"ID":"2dbe070f-149b-4b17-ad50-a6b512f07e8e","Type":"ContainerStarted","Data":"847addd00cf160fbd54adcc0606b152eb22847cbf2bb2f5973e4b4652bd6c7a3"} Jul 16 17:30:18.479763 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:18.479704 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" podStartSLOduration=1.479685227 podStartE2EDuration="1.479685227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:18.476522659 +0000 UTC m=+2348.260677724" watchObservedRunningTime="2026-07-16 17:30:18.479685227 +0000 UTC m=+2348.263840288" Jul 16 17:30:27.490688 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:27.490653 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:27.491129 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:27.490710 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:27.503780 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:27.503754 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:28.510955 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:28.510923 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:30:38.106726 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.106692 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr"] Jul 16 17:30:38.110990 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.110965 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.116623 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.116479 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8d3a22\"/\"default-dockercfg-4xt5g\"" Jul 16 17:30:38.116623 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.116479 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8d3a22\"/\"sub-test-model-8d3a22-kserve-self-signed-certs\"" Jul 16 17:30:38.116623 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.116483 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8d3a22\"/\"kube-root-ca.crt\"" Jul 16 17:30:38.116883 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.116745 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8d3a22\"/\"openshift-service-ca.crt\"" Jul 16 17:30:38.123312 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.123280 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr"] Jul 16 17:30:38.232070 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.232010 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-dshm\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.232070 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.232077 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-home\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.232289 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.232170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k259n\" (UniqueName: \"kubernetes.io/projected/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-kube-api-access-k259n\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.232289 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.232239 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tls-certs\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.232367 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.232291 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tmp-dir\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.232367 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.232313 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-model-cache\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.332969 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.332915 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tmp-dir\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.332969 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.332971 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-model-cache\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.333239 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.333014 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-dshm\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.333239 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.333047 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-home\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.333239 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.333099 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k259n\" (UniqueName: \"kubernetes.io/projected/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-kube-api-access-k259n\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.333239 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.333160 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tls-certs\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.333482 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.333339 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tmp-dir\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.333482 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.333366 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-model-cache\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.333482 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.333469 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-home\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.335243 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.335218 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-dshm\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.335731 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.335710 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tls-certs\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.343830 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.343809 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k259n\" (UniqueName: \"kubernetes.io/projected/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-kube-api-access-k259n\") pod \"sub-test-model-8d3a22-kserve-777b679d4c-zqrnr\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.423390 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.423288 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:38.566998 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:38.566973 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr"] Jul 16 17:30:38.569311 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:30:38.569279 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3907ee7_7607_4497_bf7b_02d8bfe46eaf.slice/crio-f27c14415ff6d6786666c198f3e80b52a3fb95babbf3382a724a43a9ffddf51d WatchSource:0}: Error finding container f27c14415ff6d6786666c198f3e80b52a3fb95babbf3382a724a43a9ffddf51d: Status 404 returned error can't find the container with id f27c14415ff6d6786666c198f3e80b52a3fb95babbf3382a724a43a9ffddf51d Jul 16 17:30:39.566790 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:39.566754 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" event={"ID":"c3907ee7-7607-4497-bf7b-02d8bfe46eaf","Type":"ContainerStarted","Data":"3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8"} Jul 16 17:30:39.567164 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:39.566804 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" event={"ID":"c3907ee7-7607-4497-bf7b-02d8bfe46eaf","Type":"ContainerStarted","Data":"f27c14415ff6d6786666c198f3e80b52a3fb95babbf3382a724a43a9ffddf51d"} Jul 16 17:30:39.587236 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:39.587182 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" podStartSLOduration=1.5871648299999999 podStartE2EDuration="1.58716483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:39.585757293 +0000 UTC m=+2369.369912357" watchObservedRunningTime="2026-07-16 17:30:39.58716483 +0000 UTC m=+2369.371319892" Jul 16 17:30:48.424097 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:48.424058 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:48.424682 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:48.424120 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:48.436881 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:48.436853 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:30:48.625230 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:30:48.625200 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:31:10.924841 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:31:10.924710 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:31:10.929845 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:31:10.929821 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x84hq_9ad7a19b-83c8-404e-b28b-464de3f17e16/ovn-acl-logging/0.log" Jul 16 17:31:44.804332 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:31:44.804294 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z"] Jul 16 17:31:45.653979 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:31:45.653942 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr"] Jul 16 17:31:59.824674 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:31:59.824627 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" podUID="2dbe070f-149b-4b17-ad50-a6b512f07e8e" containerName="main" containerID="cri-o://9aa75a2cd1f8837094c977f1fdc025593c84f00b4bd34a2a18ad15f6c1a54c7e" gracePeriod=45 Jul 16 17:31:59.991907 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:31:59.991872 2556 generic.go:356] "Generic (PLEG): container finished" podID="2dbe070f-149b-4b17-ad50-a6b512f07e8e" containerID="9aa75a2cd1f8837094c977f1fdc025593c84f00b4bd34a2a18ad15f6c1a54c7e" exitCode=0 Jul 16 17:31:59.992092 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:31:59.991948 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" event={"ID":"2dbe070f-149b-4b17-ad50-a6b512f07e8e","Type":"ContainerDied","Data":"9aa75a2cd1f8837094c977f1fdc025593c84f00b4bd34a2a18ad15f6c1a54c7e"} Jul 16 17:32:00.082006 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.081936 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:32:00.141787 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.141740 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tmp-dir\") pod \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " Jul 16 17:32:00.142006 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.141805 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tls-certs\") pod \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " Jul 16 17:32:00.142006 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.141851 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-dshm\") pod \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " Jul 16 17:32:00.142006 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.141877 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-home\") pod \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " Jul 16 17:32:00.142006 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.141907 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjckw\" (UniqueName: \"kubernetes.io/projected/2dbe070f-149b-4b17-ad50-a6b512f07e8e-kube-api-access-tjckw\") pod \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " Jul 16 17:32:00.142006 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.141949 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-model-cache\") pod \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\" (UID: \"2dbe070f-149b-4b17-ad50-a6b512f07e8e\") " Jul 16 17:32:00.142265 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.142089 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2dbe070f-149b-4b17-ad50-a6b512f07e8e" (UID: "2dbe070f-149b-4b17-ad50-a6b512f07e8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:32:00.142265 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.142140 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-home" (OuterVolumeSpecName: "home") pod "2dbe070f-149b-4b17-ad50-a6b512f07e8e" (UID: "2dbe070f-149b-4b17-ad50-a6b512f07e8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:32:00.142367 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.142312 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:00.142367 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.142343 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:00.142367 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.142308 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2dbe070f-149b-4b17-ad50-a6b512f07e8e" (UID: "2dbe070f-149b-4b17-ad50-a6b512f07e8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:32:00.144217 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.144185 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbe070f-149b-4b17-ad50-a6b512f07e8e-kube-api-access-tjckw" (OuterVolumeSpecName: "kube-api-access-tjckw") pod "2dbe070f-149b-4b17-ad50-a6b512f07e8e" (UID: "2dbe070f-149b-4b17-ad50-a6b512f07e8e"). InnerVolumeSpecName "kube-api-access-tjckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:32:00.144323 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.144215 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-dshm" (OuterVolumeSpecName: "dshm") pod "2dbe070f-149b-4b17-ad50-a6b512f07e8e" (UID: "2dbe070f-149b-4b17-ad50-a6b512f07e8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:32:00.144359 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.144327 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2dbe070f-149b-4b17-ad50-a6b512f07e8e" (UID: "2dbe070f-149b-4b17-ad50-a6b512f07e8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:32:00.243066 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.243024 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbe070f-149b-4b17-ad50-a6b512f07e8e-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:00.243066 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.243058 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:00.243066 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.243068 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjckw\" (UniqueName: \"kubernetes.io/projected/2dbe070f-149b-4b17-ad50-a6b512f07e8e-kube-api-access-tjckw\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:00.243066 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.243078 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbe070f-149b-4b17-ad50-a6b512f07e8e-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:00.674744 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.674666 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" podUID="c3907ee7-7607-4497-bf7b-02d8bfe46eaf" containerName="main" containerID="cri-o://3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8" gracePeriod=45 Jul 16 17:32:00.934461 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.934371 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:32:00.999016 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.998981 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" Jul 16 17:32:00.999213 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.999031 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z" event={"ID":"2dbe070f-149b-4b17-ad50-a6b512f07e8e","Type":"ContainerDied","Data":"847addd00cf160fbd54adcc0606b152eb22847cbf2bb2f5973e4b4652bd6c7a3"} Jul 16 17:32:00.999213 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:00.999092 2556 scope.go:117] "RemoveContainer" containerID="9aa75a2cd1f8837094c977f1fdc025593c84f00b4bd34a2a18ad15f6c1a54c7e" Jul 16 17:32:01.000565 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.000541 2556 generic.go:356] "Generic (PLEG): container finished" podID="c3907ee7-7607-4497-bf7b-02d8bfe46eaf" containerID="3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8" exitCode=0 Jul 16 17:32:01.000653 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.000574 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" event={"ID":"c3907ee7-7607-4497-bf7b-02d8bfe46eaf","Type":"ContainerDied","Data":"3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8"} Jul 16 17:32:01.000653 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.000618 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" event={"ID":"c3907ee7-7607-4497-bf7b-02d8bfe46eaf","Type":"ContainerDied","Data":"f27c14415ff6d6786666c198f3e80b52a3fb95babbf3382a724a43a9ffddf51d"} Jul 16 17:32:01.000653 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.000621 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr" Jul 16 17:32:01.008353 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.008331 2556 scope.go:117] "RemoveContainer" containerID="3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8" Jul 16 17:32:01.019448 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.019404 2556 scope.go:117] "RemoveContainer" containerID="3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8" Jul 16 17:32:01.019779 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:32:01.019758 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8\": container with ID starting with 3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8 not found: ID does not exist" containerID="3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8" Jul 16 17:32:01.019830 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.019791 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8"} err="failed to get container status \"3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8\": rpc error: code = NotFound desc = could not find container \"3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8\": container with ID starting with 3216d2319a8b14855cef35e532708ca28a8185ed4a94586b70f1f7fdf2bab0b8 not found: ID does not exist" Jul 16 17:32:01.021055 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.021025 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z"] Jul 16 17:32:01.028869 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.028836 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b44c0f/sub-test-model-b44c0f-kserve-5c5b9775c7-pwc4z"] Jul 16 17:32:01.051254 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051207 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tls-certs\") pod \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " Jul 16 17:32:01.051455 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051287 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-model-cache\") pod \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " Jul 16 17:32:01.051455 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051330 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k259n\" (UniqueName: \"kubernetes.io/projected/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-kube-api-access-k259n\") pod \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " Jul 16 17:32:01.051455 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051448 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tmp-dir\") pod \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " Jul 16 17:32:01.051640 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051496 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-dshm\") pod \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " Jul 16 17:32:01.051640 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051565 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-home\") pod \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\" (UID: \"c3907ee7-7607-4497-bf7b-02d8bfe46eaf\") " Jul 16 17:32:01.051640 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051577 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3907ee7-7607-4497-bf7b-02d8bfe46eaf" (UID: "c3907ee7-7607-4497-bf7b-02d8bfe46eaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:32:01.051776 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051742 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3907ee7-7607-4497-bf7b-02d8bfe46eaf" (UID: "c3907ee7-7607-4497-bf7b-02d8bfe46eaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:32:01.051874 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051848 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-home" (OuterVolumeSpecName: "home") pod "c3907ee7-7607-4497-bf7b-02d8bfe46eaf" (UID: "c3907ee7-7607-4497-bf7b-02d8bfe46eaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:32:01.052000 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051977 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:01.052000 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.051996 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:01.052094 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.052011 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:01.053688 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.053661 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-dshm" (OuterVolumeSpecName: "dshm") pod "c3907ee7-7607-4497-bf7b-02d8bfe46eaf" (UID: "c3907ee7-7607-4497-bf7b-02d8bfe46eaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:32:01.053992 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.053966 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3907ee7-7607-4497-bf7b-02d8bfe46eaf" (UID: "c3907ee7-7607-4497-bf7b-02d8bfe46eaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:32:01.053992 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.053978 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-kube-api-access-k259n" (OuterVolumeSpecName: "kube-api-access-k259n") pod "c3907ee7-7607-4497-bf7b-02d8bfe46eaf" (UID: "c3907ee7-7607-4497-bf7b-02d8bfe46eaf"). InnerVolumeSpecName "kube-api-access-k259n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:32:01.153089 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.153042 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:01.153089 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.153075 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k259n\" (UniqueName: \"kubernetes.io/projected/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-kube-api-access-k259n\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:01.153089 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.153087 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3907ee7-7607-4497-bf7b-02d8bfe46eaf-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:32:01.327974 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.327938 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr"] Jul 16 17:32:01.330676 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:01.330649 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8d3a22/sub-test-model-8d3a22-kserve-777b679d4c-zqrnr"] Jul 16 17:32:02.843522 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:02.843481 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbe070f-149b-4b17-ad50-a6b512f07e8e" path="/var/lib/kubelet/pods/2dbe070f-149b-4b17-ad50-a6b512f07e8e/volumes" Jul 16 17:32:02.843937 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:32:02.843923 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3907ee7-7607-4497-bf7b-02d8bfe46eaf" path="/var/lib/kubelet/pods/c3907ee7-7607-4497-bf7b-02d8bfe46eaf/volumes" Jul 16 17:33:06.181079 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.180997 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7596-140535669548976"] Jul 16 17:33:06.181573 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.181424 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbe070f-149b-4b17-ad50-a6b512f07e8e" containerName="main" Jul 16 17:33:06.181573 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.181437 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbe070f-149b-4b17-ad50-a6b512f07e8e" containerName="main" Jul 16 17:33:06.181573 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.181468 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3907ee7-7607-4497-bf7b-02d8bfe46eaf" containerName="main" Jul 16 17:33:06.181573 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.181474 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3907ee7-7607-4497-bf7b-02d8bfe46eaf" containerName="main" Jul 16 17:33:06.181573 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.181540 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3907ee7-7607-4497-bf7b-02d8bfe46eaf" containerName="main" Jul 16 17:33:06.181573 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.181548 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbe070f-149b-4b17-ad50-a6b512f07e8e" containerName="main" Jul 16 17:33:06.183404 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.183387 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535669548976" Jul 16 17:33:06.186750 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.186726 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lnlsz\"" Jul 16 17:33:06.193858 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.193834 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-140535669548976"] Jul 16 17:33:06.212270 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.212237 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxfm\" (UniqueName: \"kubernetes.io/projected/2d554b13-8ceb-44bf-9983-9409c635e0b2-kube-api-access-4wxfm\") pod \"test-curl-7596-140535669548976\" (UID: \"2d554b13-8ceb-44bf-9983-9409c635e0b2\") " pod="opendatahub/test-curl-7596-140535669548976" Jul 16 17:33:06.313075 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.313033 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxfm\" (UniqueName: \"kubernetes.io/projected/2d554b13-8ceb-44bf-9983-9409c635e0b2-kube-api-access-4wxfm\") pod \"test-curl-7596-140535669548976\" (UID: \"2d554b13-8ceb-44bf-9983-9409c635e0b2\") " pod="opendatahub/test-curl-7596-140535669548976" Jul 16 17:33:06.323822 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.323783 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxfm\" (UniqueName: \"kubernetes.io/projected/2d554b13-8ceb-44bf-9983-9409c635e0b2-kube-api-access-4wxfm\") pod \"test-curl-7596-140535669548976\" (UID: \"2d554b13-8ceb-44bf-9983-9409c635e0b2\") " pod="opendatahub/test-curl-7596-140535669548976" Jul 16 17:33:06.493301 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.493270 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535669548976" Jul 16 17:33:06.637108 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:06.637081 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-140535669548976"] Jul 16 17:33:06.642820 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:33:06.642785 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d554b13_8ceb_44bf_9983_9409c635e0b2.slice/crio-efc9b5334d0295e5effd3feac94ee609a4bffcb4a303efbfd5da8c3ca3d3e99a WatchSource:0}: Error finding container efc9b5334d0295e5effd3feac94ee609a4bffcb4a303efbfd5da8c3ca3d3e99a: Status 404 returned error can't find the container with id efc9b5334d0295e5effd3feac94ee609a4bffcb4a303efbfd5da8c3ca3d3e99a Jul 16 17:33:07.348259 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:07.348219 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-140535669548976" event={"ID":"2d554b13-8ceb-44bf-9983-9409c635e0b2","Type":"ContainerStarted","Data":"efc9b5334d0295e5effd3feac94ee609a4bffcb4a303efbfd5da8c3ca3d3e99a"} Jul 16 17:33:08.354526 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:08.354481 2556 generic.go:356] "Generic (PLEG): container finished" podID="2d554b13-8ceb-44bf-9983-9409c635e0b2" containerID="91aba5b5ca7fe3248213d2981a78690757044a6278a579fab69dc4fbadf0cad9" exitCode=0 Jul 16 17:33:08.354951 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:08.354573 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-140535669548976" event={"ID":"2d554b13-8ceb-44bf-9983-9409c635e0b2","Type":"ContainerDied","Data":"91aba5b5ca7fe3248213d2981a78690757044a6278a579fab69dc4fbadf0cad9"} Jul 16 17:33:09.495596 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.495567 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535669548976" Jul 16 17:33:09.547445 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.547393 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7596-140535669548976_2d554b13-8ceb-44bf-9983-9409c635e0b2/test-curl-7596-140535669548976/0.log" Jul 16 17:33:09.549611 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.549585 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wxfm\" (UniqueName: \"kubernetes.io/projected/2d554b13-8ceb-44bf-9983-9409c635e0b2-kube-api-access-4wxfm\") pod \"2d554b13-8ceb-44bf-9983-9409c635e0b2\" (UID: \"2d554b13-8ceb-44bf-9983-9409c635e0b2\") " Jul 16 17:33:09.551730 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.551702 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d554b13-8ceb-44bf-9983-9409c635e0b2-kube-api-access-4wxfm" (OuterVolumeSpecName: "kube-api-access-4wxfm") pod "2d554b13-8ceb-44bf-9983-9409c635e0b2" (UID: "2d554b13-8ceb-44bf-9983-9409c635e0b2"). InnerVolumeSpecName "kube-api-access-4wxfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:09.579126 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.579071 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7596-140535669548976"] Jul 16 17:33:09.581005 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.580981 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7596-140535669548976"] Jul 16 17:33:09.650632 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.650536 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wxfm\" (UniqueName: \"kubernetes.io/projected/2d554b13-8ceb-44bf-9983-9409c635e0b2-kube-api-access-4wxfm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:33:09.705848 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.705809 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7596-140535664475056"] Jul 16 17:33:09.706260 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.706244 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d554b13-8ceb-44bf-9983-9409c635e0b2" containerName="test-curl-7596-140535669548976" Jul 16 17:33:09.706260 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.706262 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d554b13-8ceb-44bf-9983-9409c635e0b2" containerName="test-curl-7596-140535669548976" Jul 16 17:33:09.706407 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.706350 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d554b13-8ceb-44bf-9983-9409c635e0b2" containerName="test-curl-7596-140535669548976" Jul 16 17:33:09.708265 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.708247 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535664475056" Jul 16 17:33:09.718115 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.718088 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-140535664475056"] Jul 16 17:33:09.751409 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.751370 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg26c\" (UniqueName: \"kubernetes.io/projected/8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb-kube-api-access-vg26c\") pod \"test-curl-7596-140535664475056\" (UID: \"8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb\") " pod="opendatahub/test-curl-7596-140535664475056" Jul 16 17:33:09.852591 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.852538 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg26c\" (UniqueName: \"kubernetes.io/projected/8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb-kube-api-access-vg26c\") pod \"test-curl-7596-140535664475056\" (UID: \"8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb\") " pod="opendatahub/test-curl-7596-140535664475056" Jul 16 17:33:09.863000 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:09.862951 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg26c\" (UniqueName: \"kubernetes.io/projected/8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb-kube-api-access-vg26c\") pod \"test-curl-7596-140535664475056\" (UID: \"8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb\") " pod="opendatahub/test-curl-7596-140535664475056" Jul 16 17:33:10.018791 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:10.018740 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535664475056" Jul 16 17:33:10.167621 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:10.167584 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-140535664475056"] Jul 16 17:33:10.170522 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:33:10.170490 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a1bac90_8544_4a8b_80f6_d54b3b1fa4cb.slice/crio-6499ceb0f056a8c60f63a6ed146cbfc4b52ab87b84af098f157f94df6ca89139 WatchSource:0}: Error finding container 6499ceb0f056a8c60f63a6ed146cbfc4b52ab87b84af098f157f94df6ca89139: Status 404 returned error can't find the container with id 6499ceb0f056a8c60f63a6ed146cbfc4b52ab87b84af098f157f94df6ca89139 Jul 16 17:33:10.366382 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:10.366290 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535669548976" Jul 16 17:33:10.366382 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:10.366325 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc9b5334d0295e5effd3feac94ee609a4bffcb4a303efbfd5da8c3ca3d3e99a" Jul 16 17:33:10.367542 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:10.367520 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-140535664475056" event={"ID":"8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb","Type":"ContainerStarted","Data":"6499ceb0f056a8c60f63a6ed146cbfc4b52ab87b84af098f157f94df6ca89139"} Jul 16 17:33:10.378707 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:33:10.378675 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7596-140535669548976\" is forbidden: User \"system:node:ip-10-0-133-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-149.ec2.internal' and this object" podUID="2d554b13-8ceb-44bf-9983-9409c635e0b2" pod="opendatahub/test-curl-7596-140535669548976" Jul 16 17:33:10.848230 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:33:10.848195 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7596-140535669548976\" is forbidden: User \"system:node:ip-10-0-133-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-149.ec2.internal' and this object" podUID="2d554b13-8ceb-44bf-9983-9409c635e0b2" pod="opendatahub/test-curl-7596-140535669548976" Jul 16 17:33:10.850527 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:10.850499 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d554b13-8ceb-44bf-9983-9409c635e0b2" path="/var/lib/kubelet/pods/2d554b13-8ceb-44bf-9983-9409c635e0b2/volumes" Jul 16 17:33:11.373598 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:11.373562 2556 generic.go:356] "Generic (PLEG): container finished" podID="8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb" containerID="a455fb27501b0e87e8b10ea93b044920b7ba79e5c7f0cb4c1789c989428d1e20" exitCode=0 Jul 16 17:33:11.373816 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:11.373654 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-140535664475056" event={"ID":"8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb","Type":"ContainerDied","Data":"a455fb27501b0e87e8b10ea93b044920b7ba79e5c7f0cb4c1789c989428d1e20"} Jul 16 17:33:12.522352 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.522322 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535664475056" Jul 16 17:33:12.565513 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.565485 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7596-140535664475056_8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb/test-curl-7596-140535664475056/0.log" Jul 16 17:33:12.577896 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.577868 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg26c\" (UniqueName: \"kubernetes.io/projected/8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb-kube-api-access-vg26c\") pod \"8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb\" (UID: \"8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb\") " Jul 16 17:33:12.580152 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.580116 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb-kube-api-access-vg26c" (OuterVolumeSpecName: "kube-api-access-vg26c") pod "8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb" (UID: "8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb"). InnerVolumeSpecName "kube-api-access-vg26c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:12.614183 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.614145 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7596-140535664475056"] Jul 16 17:33:12.621447 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.621397 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7596-140535664475056"] Jul 16 17:33:12.679728 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.679642 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg26c\" (UniqueName: \"kubernetes.io/projected/8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb-kube-api-access-vg26c\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:33:12.798022 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.797986 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7596-140535666953504"] Jul 16 17:33:12.798454 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.798439 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb" containerName="test-curl-7596-140535664475056" Jul 16 17:33:12.798503 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.798456 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb" containerName="test-curl-7596-140535664475056" Jul 16 17:33:12.798542 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.798518 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb" containerName="test-curl-7596-140535664475056" Jul 16 17:33:12.800634 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.800618 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:12.811690 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.811662 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-140535666953504"] Jul 16 17:33:12.843397 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.843352 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb" path="/var/lib/kubelet/pods/8a1bac90-8544-4a8b-80f6-d54b3b1fa4cb/volumes" Jul 16 17:33:12.882346 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.882308 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkzwf\" (UniqueName: \"kubernetes.io/projected/af773b95-8ef3-4698-9454-e0f97f9fb3e4-kube-api-access-fkzwf\") pod \"test-curl-7596-140535666953504\" (UID: \"af773b95-8ef3-4698-9454-e0f97f9fb3e4\") " pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:12.983687 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.983651 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzwf\" (UniqueName: \"kubernetes.io/projected/af773b95-8ef3-4698-9454-e0f97f9fb3e4-kube-api-access-fkzwf\") pod \"test-curl-7596-140535666953504\" (UID: \"af773b95-8ef3-4698-9454-e0f97f9fb3e4\") " pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:12.996455 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:12.996400 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkzwf\" (UniqueName: \"kubernetes.io/projected/af773b95-8ef3-4698-9454-e0f97f9fb3e4-kube-api-access-fkzwf\") pod \"test-curl-7596-140535666953504\" (UID: \"af773b95-8ef3-4698-9454-e0f97f9fb3e4\") " pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:13.109741 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:13.109701 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:13.249838 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:13.249811 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-140535666953504"] Jul 16 17:33:13.251875 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:33:13.251844 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf773b95_8ef3_4698_9454_e0f97f9fb3e4.slice/crio-5037d4dd3850796a3796d28704c0ec592e6db980eb1d7c7e75cc874fa473c6f5 WatchSource:0}: Error finding container 5037d4dd3850796a3796d28704c0ec592e6db980eb1d7c7e75cc874fa473c6f5: Status 404 returned error can't find the container with id 5037d4dd3850796a3796d28704c0ec592e6db980eb1d7c7e75cc874fa473c6f5 Jul 16 17:33:13.386372 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:13.386330 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-140535666953504" event={"ID":"af773b95-8ef3-4698-9454-e0f97f9fb3e4","Type":"ContainerStarted","Data":"5037d4dd3850796a3796d28704c0ec592e6db980eb1d7c7e75cc874fa473c6f5"} Jul 16 17:33:13.387618 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:13.387596 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535664475056" Jul 16 17:33:13.387752 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:13.387646 2556 scope.go:117] "RemoveContainer" containerID="a455fb27501b0e87e8b10ea93b044920b7ba79e5c7f0cb4c1789c989428d1e20" Jul 16 17:33:14.394323 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:14.394283 2556 generic.go:356] "Generic (PLEG): container finished" podID="af773b95-8ef3-4698-9454-e0f97f9fb3e4" containerID="436e1a871fc8b8738a6478c4d3f6e95c2f99f78e6b62fe062a8ec2dbfde1e77f" exitCode=0 Jul 16 17:33:14.394774 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:14.394371 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-140535666953504" event={"ID":"af773b95-8ef3-4698-9454-e0f97f9fb3e4","Type":"ContainerDied","Data":"436e1a871fc8b8738a6478c4d3f6e95c2f99f78e6b62fe062a8ec2dbfde1e77f"} Jul 16 17:33:15.547554 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.547528 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:15.576272 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.576239 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7596-140535666953504_af773b95-8ef3-4698-9454-e0f97f9fb3e4/test-curl-7596-140535666953504/0.log" Jul 16 17:33:15.604595 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.604559 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7596-140535666953504"] Jul 16 17:33:15.607159 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.607128 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7596-140535666953504"] Jul 16 17:33:15.608539 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.608512 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkzwf\" (UniqueName: \"kubernetes.io/projected/af773b95-8ef3-4698-9454-e0f97f9fb3e4-kube-api-access-fkzwf\") pod \"af773b95-8ef3-4698-9454-e0f97f9fb3e4\" (UID: \"af773b95-8ef3-4698-9454-e0f97f9fb3e4\") " Jul 16 17:33:15.610622 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.610585 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af773b95-8ef3-4698-9454-e0f97f9fb3e4-kube-api-access-fkzwf" (OuterVolumeSpecName: "kube-api-access-fkzwf") pod "af773b95-8ef3-4698-9454-e0f97f9fb3e4" (UID: "af773b95-8ef3-4698-9454-e0f97f9fb3e4"). InnerVolumeSpecName "kube-api-access-fkzwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:15.710226 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.710129 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkzwf\" (UniqueName: \"kubernetes.io/projected/af773b95-8ef3-4698-9454-e0f97f9fb3e4-kube-api-access-fkzwf\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:33:15.733005 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.732969 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7596-140535666953504"] Jul 16 17:33:15.733462 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.733446 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af773b95-8ef3-4698-9454-e0f97f9fb3e4" containerName="test-curl-7596-140535666953504" Jul 16 17:33:15.733462 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.733463 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af773b95-8ef3-4698-9454-e0f97f9fb3e4" containerName="test-curl-7596-140535666953504" Jul 16 17:33:15.733547 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.733521 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="af773b95-8ef3-4698-9454-e0f97f9fb3e4" containerName="test-curl-7596-140535666953504" Jul 16 17:33:15.735627 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.735610 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:15.746593 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.746560 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-140535666953504"] Jul 16 17:33:15.810725 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.810688 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frsn8\" (UniqueName: \"kubernetes.io/projected/c3afbd22-e635-40ac-ab62-a1f4498711f8-kube-api-access-frsn8\") pod \"test-curl-7596-140535666953504\" (UID: \"c3afbd22-e635-40ac-ab62-a1f4498711f8\") " pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:15.912063 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.912010 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frsn8\" (UniqueName: \"kubernetes.io/projected/c3afbd22-e635-40ac-ab62-a1f4498711f8-kube-api-access-frsn8\") pod \"test-curl-7596-140535666953504\" (UID: \"c3afbd22-e635-40ac-ab62-a1f4498711f8\") " pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:15.922999 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:15.922966 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frsn8\" (UniqueName: \"kubernetes.io/projected/c3afbd22-e635-40ac-ab62-a1f4498711f8-kube-api-access-frsn8\") pod \"test-curl-7596-140535666953504\" (UID: \"c3afbd22-e635-40ac-ab62-a1f4498711f8\") " pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:16.045285 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:16.045245 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:16.197858 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:16.197775 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-140535666953504"] Jul 16 17:33:16.200511 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:33:16.200471 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3afbd22_e635_40ac_ab62_a1f4498711f8.slice/crio-cbccfdd07d30f532392548c5ddf7c4c2394c93bf5165a4ac6fd7d003d9a87ba5 WatchSource:0}: Error finding container cbccfdd07d30f532392548c5ddf7c4c2394c93bf5165a4ac6fd7d003d9a87ba5: Status 404 returned error can't find the container with id cbccfdd07d30f532392548c5ddf7c4c2394c93bf5165a4ac6fd7d003d9a87ba5 Jul 16 17:33:16.407394 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:16.407288 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-140535666953504" event={"ID":"c3afbd22-e635-40ac-ab62-a1f4498711f8","Type":"ContainerStarted","Data":"cbccfdd07d30f532392548c5ddf7c4c2394c93bf5165a4ac6fd7d003d9a87ba5"} Jul 16 17:33:16.408539 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:16.408513 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:16.408539 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:16.408535 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5037d4dd3850796a3796d28704c0ec592e6db980eb1d7c7e75cc874fa473c6f5" Jul 16 17:33:16.421657 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:16.421615 2556 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7596-140535666953504" oldPodUID="af773b95-8ef3-4698-9454-e0f97f9fb3e4" podUID="c3afbd22-e635-40ac-ab62-a1f4498711f8" Jul 16 17:33:16.843294 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:16.843255 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af773b95-8ef3-4698-9454-e0f97f9fb3e4" path="/var/lib/kubelet/pods/af773b95-8ef3-4698-9454-e0f97f9fb3e4/volumes" Jul 16 17:33:17.414703 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:17.414666 2556 generic.go:356] "Generic (PLEG): container finished" podID="c3afbd22-e635-40ac-ab62-a1f4498711f8" containerID="4bf6efb2ecdf1f154451613757e4449523beb5a0d2d2bce737ce2b235ffe2129" exitCode=0 Jul 16 17:33:17.414883 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:17.414760 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-140535666953504" event={"ID":"c3afbd22-e635-40ac-ab62-a1f4498711f8","Type":"ContainerDied","Data":"4bf6efb2ecdf1f154451613757e4449523beb5a0d2d2bce737ce2b235ffe2129"} Jul 16 17:33:18.556315 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.556283 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:18.598159 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.598131 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7596-140535666953504_c3afbd22-e635-40ac-ab62-a1f4498711f8/test-curl-7596-140535666953504/0.log" Jul 16 17:33:18.639746 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.639706 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frsn8\" (UniqueName: \"kubernetes.io/projected/c3afbd22-e635-40ac-ab62-a1f4498711f8-kube-api-access-frsn8\") pod \"c3afbd22-e635-40ac-ab62-a1f4498711f8\" (UID: \"c3afbd22-e635-40ac-ab62-a1f4498711f8\") " Jul 16 17:33:18.640666 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.640640 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7596-140535666953504"] Jul 16 17:33:18.641972 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.641951 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3afbd22-e635-40ac-ab62-a1f4498711f8-kube-api-access-frsn8" (OuterVolumeSpecName: "kube-api-access-frsn8") pod "c3afbd22-e635-40ac-ab62-a1f4498711f8" (UID: "c3afbd22-e635-40ac-ab62-a1f4498711f8"). InnerVolumeSpecName "kube-api-access-frsn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:18.651768 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.651739 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7596-140535666953504"] Jul 16 17:33:18.741132 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.741090 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frsn8\" (UniqueName: \"kubernetes.io/projected/c3afbd22-e635-40ac-ab62-a1f4498711f8-kube-api-access-frsn8\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:33:18.798505 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.798461 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7596-140535666843984"] Jul 16 17:33:18.798972 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.798952 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3afbd22-e635-40ac-ab62-a1f4498711f8" containerName="test-curl-7596-140535666953504" Jul 16 17:33:18.799062 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.798974 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3afbd22-e635-40ac-ab62-a1f4498711f8" containerName="test-curl-7596-140535666953504" Jul 16 17:33:18.799133 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.799118 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3afbd22-e635-40ac-ab62-a1f4498711f8" containerName="test-curl-7596-140535666953504" Jul 16 17:33:18.802076 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.802055 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666843984" Jul 16 17:33:18.814914 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.814888 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-140535666843984"] Jul 16 17:33:18.844201 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.844165 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3afbd22-e635-40ac-ab62-a1f4498711f8" path="/var/lib/kubelet/pods/c3afbd22-e635-40ac-ab62-a1f4498711f8/volumes" Jul 16 17:33:18.942829 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:18.942775 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnx86\" (UniqueName: \"kubernetes.io/projected/0bc31456-cbb2-49c1-8a66-92d513c0d364-kube-api-access-qnx86\") pod \"test-curl-7596-140535666843984\" (UID: \"0bc31456-cbb2-49c1-8a66-92d513c0d364\") " pod="opendatahub/test-curl-7596-140535666843984" Jul 16 17:33:19.043766 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:19.043657 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnx86\" (UniqueName: \"kubernetes.io/projected/0bc31456-cbb2-49c1-8a66-92d513c0d364-kube-api-access-qnx86\") pod \"test-curl-7596-140535666843984\" (UID: \"0bc31456-cbb2-49c1-8a66-92d513c0d364\") " pod="opendatahub/test-curl-7596-140535666843984" Jul 16 17:33:19.057805 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:19.057775 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnx86\" (UniqueName: \"kubernetes.io/projected/0bc31456-cbb2-49c1-8a66-92d513c0d364-kube-api-access-qnx86\") pod \"test-curl-7596-140535666843984\" (UID: \"0bc31456-cbb2-49c1-8a66-92d513c0d364\") " pod="opendatahub/test-curl-7596-140535666843984" Jul 16 17:33:19.111070 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:19.111018 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666843984" Jul 16 17:33:19.269693 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:19.269663 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-140535666843984"] Jul 16 17:33:19.270631 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:33:19.270605 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bc31456_cbb2_49c1_8a66_92d513c0d364.slice/crio-0d097ac63e7f93f690fd494e8d1af18f3e07d18128574a368196c11fc291e031 WatchSource:0}: Error finding container 0d097ac63e7f93f690fd494e8d1af18f3e07d18128574a368196c11fc291e031: Status 404 returned error can't find the container with id 0d097ac63e7f93f690fd494e8d1af18f3e07d18128574a368196c11fc291e031 Jul 16 17:33:19.427339 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:19.427289 2556 scope.go:117] "RemoveContainer" containerID="4bf6efb2ecdf1f154451613757e4449523beb5a0d2d2bce737ce2b235ffe2129" Jul 16 17:33:19.427578 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:19.427300 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666953504" Jul 16 17:33:19.428862 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:19.428832 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-140535666843984" event={"ID":"0bc31456-cbb2-49c1-8a66-92d513c0d364","Type":"ContainerStarted","Data":"0d097ac63e7f93f690fd494e8d1af18f3e07d18128574a368196c11fc291e031"} Jul 16 17:33:20.439474 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:20.439431 2556 generic.go:356] "Generic (PLEG): container finished" podID="0bc31456-cbb2-49c1-8a66-92d513c0d364" containerID="d36eb72a1a1fb2938af1b3d90bd696fd2c4ffd938c0cc36f3405854b52404af3" exitCode=0 Jul 16 17:33:20.439884 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:20.439514 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-140535666843984" event={"ID":"0bc31456-cbb2-49c1-8a66-92d513c0d364","Type":"ContainerDied","Data":"d36eb72a1a1fb2938af1b3d90bd696fd2c4ffd938c0cc36f3405854b52404af3"} Jul 16 17:33:21.598136 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:21.598108 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666843984" Jul 16 17:33:21.624183 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:21.624154 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7596-140535666843984_0bc31456-cbb2-49c1-8a66-92d513c0d364/test-curl-7596-140535666843984/0.log" Jul 16 17:33:21.650587 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:21.650549 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7596-140535666843984"] Jul 16 17:33:21.655079 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:21.655046 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7596-140535666843984"] Jul 16 17:33:21.668399 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:21.668371 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnx86\" (UniqueName: \"kubernetes.io/projected/0bc31456-cbb2-49c1-8a66-92d513c0d364-kube-api-access-qnx86\") pod \"0bc31456-cbb2-49c1-8a66-92d513c0d364\" (UID: \"0bc31456-cbb2-49c1-8a66-92d513c0d364\") " Jul 16 17:33:21.670553 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:21.670528 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc31456-cbb2-49c1-8a66-92d513c0d364-kube-api-access-qnx86" (OuterVolumeSpecName: "kube-api-access-qnx86") pod "0bc31456-cbb2-49c1-8a66-92d513c0d364" (UID: "0bc31456-cbb2-49c1-8a66-92d513c0d364"). InnerVolumeSpecName "kube-api-access-qnx86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:21.770065 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:21.770021 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnx86\" (UniqueName: \"kubernetes.io/projected/0bc31456-cbb2-49c1-8a66-92d513c0d364-kube-api-access-qnx86\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Jul 16 17:33:22.449748 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:22.449711 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-140535666843984" Jul 16 17:33:22.449939 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:22.449756 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d097ac63e7f93f690fd494e8d1af18f3e07d18128574a368196c11fc291e031" Jul 16 17:33:22.463301 ip-10-0-133-149 kubenswrapper[2556]: E0716 17:33:22.463265 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7596-140535666843984\" is forbidden: User \"system:node:ip-10-0-133-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-149.ec2.internal' and this object" podUID="0bc31456-cbb2-49c1-8a66-92d513c0d364" pod="opendatahub/test-curl-7596-140535666843984" Jul 16 17:33:22.842820 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:22.842784 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc31456-cbb2-49c1-8a66-92d513c0d364" path="/var/lib/kubelet/pods/0bc31456-cbb2-49c1-8a66-92d513c0d364/volumes" Jul 16 17:33:48.126650 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:48.126556 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq"] Jul 16 17:33:48.131348 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:48.131311 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d3a22-29737050-lhjfq"] Jul 16 17:33:48.843396 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:33:48.843359 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="123c50df-6a9a-4824-b04e-bd62f09d1824" path="/var/lib/kubelet/pods/123c50df-6a9a-4824-b04e-bd62f09d1824/volumes" Jul 16 17:34:01.924916 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:01.924875 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-758884f967-bb8k6_35e2ecd9-785b-48e2-b26f-be44881d505f/manager/0.log" Jul 16 17:34:02.281850 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:02.281814 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7f465c76-6d4qd_407e49f7-6cf3-4600-adda-6e96474beaf6/copy-manifests/0.log" Jul 16 17:34:02.291930 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:02.291900 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7f465c76-6d4qd_407e49f7-6cf3-4600-adda-6e96474beaf6/manager/0.log" Jul 16 17:34:02.953727 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:02.953696 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b7d967754-7qkq2_d7ae5d9a-456c-4d1c-94f8-41eb3f40d63e/maas-api/0.log" Jul 16 17:34:03.337619 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:03.337573 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-gqhs4_a0cd2061-169d-4f40-8f6c-86398cacd566/postgres/0.log" Jul 16 17:34:05.104866 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:05.104830 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-t9bhh_da3d79ee-4cc6-44f7-b9ff-bb81897c2102/kuadrant-console-plugin/0.log" Jul 16 17:34:06.219655 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:06.219616 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8b857c67-v4shd_abb42b64-0b44-46db-a4ff-8e0155c17500/kube-auth-proxy/0.log" Jul 16 17:34:06.627896 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:06.627846 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5vj4p_125e9af0-f9ce-4f4e-9e09-a1168e9ab748/payload-pre-processing/0.log" Jul 16 17:34:07.301839 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:07.301799 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-flxh9_b069bf99-265b-49fc-b2f5-39d29242707d/main/0.log" Jul 16 17:34:07.439210 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:07.439174 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-zzssk_15758bdd-a304-43da-8ed4-2843f625bd04/main/0.log" Jul 16 17:34:07.834117 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:07.834078 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-hbhhb_37e5eecc-f207-49d9-bf13-888ac1e6d94e/main/0.log" Jul 16 17:34:17.052776 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:17.052738 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gj948_c582a6c2-65b9-40e6-8382-a7c3133552cd/global-pull-secret-syncer/0.log" Jul 16 17:34:17.385781 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:17.385702 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kf8nk_0b89bf64-dbd2-4b29-95e1-25a45fc84481/konnectivity-agent/0.log" Jul 16 17:34:17.558130 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:17.558102 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-149.ec2.internal_556eca662b4e5f951dd2eca461c87322/haproxy/0.log" Jul 16 17:34:22.295914 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:22.295884 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-t9bhh_da3d79ee-4cc6-44f7-b9ff-bb81897c2102/kuadrant-console-plugin/0.log" Jul 16 17:34:24.177002 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.176920 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_05b2d1f1-275e-4f60-81c6-afc36861060a/alertmanager/0.log" Jul 16 17:34:24.206402 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.206373 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_05b2d1f1-275e-4f60-81c6-afc36861060a/config-reloader/0.log" Jul 16 17:34:24.237063 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.237026 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_05b2d1f1-275e-4f60-81c6-afc36861060a/kube-rbac-proxy-web/0.log" Jul 16 17:34:24.264023 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.263990 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_05b2d1f1-275e-4f60-81c6-afc36861060a/kube-rbac-proxy/0.log" Jul 16 17:34:24.292499 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.292473 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_05b2d1f1-275e-4f60-81c6-afc36861060a/kube-rbac-proxy-metric/0.log" Jul 16 17:34:24.332143 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.332123 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_05b2d1f1-275e-4f60-81c6-afc36861060a/prom-label-proxy/0.log" Jul 16 17:34:24.369772 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.369739 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_05b2d1f1-275e-4f60-81c6-afc36861060a/init-config-reloader/0.log" Jul 16 17:34:24.468965 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.468884 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-lmxk5_3d0e8127-3e57-4679-b5b5-0f6df7811c19/kube-state-metrics/0.log" Jul 16 17:34:24.496736 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.496705 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-lmxk5_3d0e8127-3e57-4679-b5b5-0f6df7811c19/kube-rbac-proxy-main/0.log" Jul 16 17:34:24.525558 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.525528 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-lmxk5_3d0e8127-3e57-4679-b5b5-0f6df7811c19/kube-rbac-proxy-self/0.log" Jul 16 17:34:24.557868 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.557837 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d78b8c7b8-fvcsb_0bd3adc6-6f09-4128-911b-155204c58cb0/metrics-server/0.log" Jul 16 17:34:24.595857 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.595823 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-r8pwq_84e0b799-0e15-4b9c-a6d0-c13fc6b418f4/monitoring-plugin/0.log" Jul 16 17:34:24.875167 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.875138 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vq2kf_201ad1e8-9e03-4355-b116-7141950f9997/node-exporter/0.log" Jul 16 17:34:24.907503 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.907466 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vq2kf_201ad1e8-9e03-4355-b116-7141950f9997/kube-rbac-proxy/0.log" Jul 16 17:34:24.940953 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.940901 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vq2kf_201ad1e8-9e03-4355-b116-7141950f9997/init-textfile/0.log" Jul 16 17:34:24.980653 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:24.980623 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-f5gnv_323be6e6-4068-442e-ad56-01e7cda84457/kube-rbac-proxy-main/0.log" Jul 16 17:34:25.025611 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.025586 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-f5gnv_323be6e6-4068-442e-ad56-01e7cda84457/kube-rbac-proxy-self/0.log" Jul 16 17:34:25.058047 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.058005 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-f5gnv_323be6e6-4068-442e-ad56-01e7cda84457/openshift-state-metrics/0.log" Jul 16 17:34:25.104656 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.104623 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de148e9-e075-423d-9099-42add4521e2a/prometheus/0.log" Jul 16 17:34:25.140471 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.140389 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de148e9-e075-423d-9099-42add4521e2a/config-reloader/0.log" Jul 16 17:34:25.179618 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.179588 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de148e9-e075-423d-9099-42add4521e2a/thanos-sidecar/0.log" Jul 16 17:34:25.211434 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.211394 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de148e9-e075-423d-9099-42add4521e2a/kube-rbac-proxy-web/0.log" Jul 16 17:34:25.252194 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.252166 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de148e9-e075-423d-9099-42add4521e2a/kube-rbac-proxy/0.log" Jul 16 17:34:25.289007 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.288980 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de148e9-e075-423d-9099-42add4521e2a/kube-rbac-proxy-thanos/0.log" Jul 16 17:34:25.323595 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.323550 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de148e9-e075-423d-9099-42add4521e2a/init-config-reloader/0.log" Jul 16 17:34:25.375540 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.375507 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-dwzn8_3204b7fb-1086-4d9e-9109-c89961227be3/prometheus-operator/0.log" Jul 16 17:34:25.406780 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.406704 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-dwzn8_3204b7fb-1086-4d9e-9109-c89961227be3/kube-rbac-proxy/0.log" Jul 16 17:34:25.458101 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.458067 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-9ck4p_ba43dc25-33f6-4880-b244-7f9ac519c4ec/prometheus-operator-admission-webhook/0.log" Jul 16 17:34:25.500872 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.500835 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-846bd64b9c-67cc9_f6081993-be91-452d-af67-2671d0631574/telemeter-client/0.log" Jul 16 17:34:25.541175 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.541145 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-846bd64b9c-67cc9_f6081993-be91-452d-af67-2671d0631574/reload/0.log" Jul 16 17:34:25.592114 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.592088 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-846bd64b9c-67cc9_f6081993-be91-452d-af67-2671d0631574/kube-rbac-proxy/0.log" Jul 16 17:34:25.659126 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.659034 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6958bc8f99-4sd92_7e4c168d-3ee6-4e64-87d0-730dd83180e5/thanos-query/0.log" Jul 16 17:34:25.691698 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.691672 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6958bc8f99-4sd92_7e4c168d-3ee6-4e64-87d0-730dd83180e5/kube-rbac-proxy-web/0.log" Jul 16 17:34:25.726761 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.726731 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6958bc8f99-4sd92_7e4c168d-3ee6-4e64-87d0-730dd83180e5/kube-rbac-proxy/0.log" Jul 16 17:34:25.758971 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.758942 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6958bc8f99-4sd92_7e4c168d-3ee6-4e64-87d0-730dd83180e5/prom-label-proxy/0.log" Jul 16 17:34:25.788080 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.788046 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6958bc8f99-4sd92_7e4c168d-3ee6-4e64-87d0-730dd83180e5/kube-rbac-proxy-rules/0.log" Jul 16 17:34:25.818084 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:25.818055 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6958bc8f99-4sd92_7e4c168d-3ee6-4e64-87d0-730dd83180e5/kube-rbac-proxy-metrics/0.log" Jul 16 17:34:27.351574 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.351538 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl"] Jul 16 17:34:27.351988 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.351955 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc31456-cbb2-49c1-8a66-92d513c0d364" containerName="test-curl-7596-140535666843984" Jul 16 17:34:27.351988 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.351968 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc31456-cbb2-49c1-8a66-92d513c0d364" containerName="test-curl-7596-140535666843984" Jul 16 17:34:27.352069 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.352060 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc31456-cbb2-49c1-8a66-92d513c0d364" containerName="test-curl-7596-140535666843984" Jul 16 17:34:27.357089 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.357054 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.360804 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.360756 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p8fjr\"/\"openshift-service-ca.crt\"" Jul 16 17:34:27.361432 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.361391 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p8fjr\"/\"default-dockercfg-nbfnt\"" Jul 16 17:34:27.361542 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.361465 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p8fjr\"/\"kube-root-ca.crt\"" Jul 16 17:34:27.367028 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.367005 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl"] Jul 16 17:34:27.391146 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.391105 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-sys\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.391313 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.391167 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-podres\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.391313 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.391243 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-lib-modules\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.391313 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.391294 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnjjg\" (UniqueName: \"kubernetes.io/projected/f16f84e2-fab2-4fff-888e-3aee9f2ee810-kube-api-access-gnjjg\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.391448 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.391328 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-proc\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.492178 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.492130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-lib-modules\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.492378 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.492195 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnjjg\" (UniqueName: \"kubernetes.io/projected/f16f84e2-fab2-4fff-888e-3aee9f2ee810-kube-api-access-gnjjg\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.492378 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.492232 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-proc\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.492378 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.492262 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-sys\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.492378 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.492278 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-podres\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.492378 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.492331 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-lib-modules\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.492378 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.492350 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-proc\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.492378 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.492368 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-sys\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.492667 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.492394 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f16f84e2-fab2-4fff-888e-3aee9f2ee810-podres\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.503003 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.502970 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnjjg\" (UniqueName: \"kubernetes.io/projected/f16f84e2-fab2-4fff-888e-3aee9f2ee810-kube-api-access-gnjjg\") pod \"perf-node-gather-daemonset-v74gl\" (UID: \"f16f84e2-fab2-4fff-888e-3aee9f2ee810\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.669488 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.669386 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:27.812756 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:27.812734 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl"] Jul 16 17:34:27.814770 ip-10-0-133-149 kubenswrapper[2556]: W0716 17:34:27.814739 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf16f84e2_fab2_4fff_888e_3aee9f2ee810.slice/crio-f0d9d3fb4e1558ef6c7679742976a26d240b87c58c75dac4692ceb955dfab5ce WatchSource:0}: Error finding container f0d9d3fb4e1558ef6c7679742976a26d240b87c58c75dac4692ceb955dfab5ce: Status 404 returned error can't find the container with id f0d9d3fb4e1558ef6c7679742976a26d240b87c58c75dac4692ceb955dfab5ce Jul 16 17:34:28.442373 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:28.442337 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-p5495_eff21e0b-f006-4ed0-98ed-12ae223f5149/download-server/0.log" Jul 16 17:34:28.815111 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:28.815076 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" event={"ID":"f16f84e2-fab2-4fff-888e-3aee9f2ee810","Type":"ContainerStarted","Data":"49eb26befe29eff14aa6f0ddcbb73f72ad5f7372f58a1862aff71ce41456129a"} Jul 16 17:34:28.815301 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:28.815136 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" event={"ID":"f16f84e2-fab2-4fff-888e-3aee9f2ee810","Type":"ContainerStarted","Data":"f0d9d3fb4e1558ef6c7679742976a26d240b87c58c75dac4692ceb955dfab5ce"} Jul 16 17:34:28.815301 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:28.815207 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:28.835743 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:28.835687 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" podStartSLOduration=1.8356685499999998 podStartE2EDuration="1.83566855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:28.834881617 +0000 UTC m=+2598.619036680" watchObservedRunningTime="2026-07-16 17:34:28.83566855 +0000 UTC m=+2598.619823633" Jul 16 17:34:30.437281 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:30.437243 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c77n8_a5124801-48ed-45ec-8405-0c1b43c560f0/dns/0.log" Jul 16 17:34:30.464669 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:30.464636 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c77n8_a5124801-48ed-45ec-8405-0c1b43c560f0/kube-rbac-proxy/0.log" Jul 16 17:34:30.561226 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:30.561190 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nh2lf_a69b989f-c7db-4193-ae08-0686fcc49150/dns-node-resolver/0.log" Jul 16 17:34:31.231641 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:31.231611 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dgrxk_59c160d1-15f8-45b6-86cf-a2ed1fce715c/node-ca/0.log" Jul 16 17:34:32.495655 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:32.495611 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8b857c67-v4shd_abb42b64-0b44-46db-a4ff-8e0155c17500/kube-auth-proxy/0.log" Jul 16 17:34:32.659650 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:32.659615 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5vj4p_125e9af0-f9ce-4f4e-9e09-a1168e9ab748/payload-pre-processing/0.log" Jul 16 17:34:33.469256 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:33.469211 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mjrrq_65af6ca4-eedb-453d-b6d3-3c293d46fd3c/serve-healthcheck-canary/0.log" Jul 16 17:34:34.084077 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:34.084043 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q2xx_173c3533-efe5-4d88-998b-33d87f347c0c/kube-rbac-proxy/0.log" Jul 16 17:34:34.111936 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:34.111909 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q2xx_173c3533-efe5-4d88-998b-33d87f347c0c/exporter/0.log" Jul 16 17:34:34.141335 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:34.141302 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q2xx_173c3533-efe5-4d88-998b-33d87f347c0c/extractor/0.log" Jul 16 17:34:34.831237 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:34.831205 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-v74gl" Jul 16 17:34:37.288044 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:37.288005 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8b857c67-v4shd_abb42b64-0b44-46db-a4ff-8e0155c17500/kube-auth-proxy/0.log" Jul 16 17:34:37.403866 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:37.403823 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5vj4p_125e9af0-f9ce-4f4e-9e09-a1168e9ab748/payload-pre-processing/0.log" Jul 16 17:34:37.443917 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:37.443884 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-758884f967-bb8k6_35e2ecd9-785b-48e2-b26f-be44881d505f/manager/0.log" Jul 16 17:34:37.576286 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:37.576251 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7f465c76-6d4qd_407e49f7-6cf3-4600-adda-6e96474beaf6/manager/0.log" Jul 16 17:34:37.601088 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:37.601060 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7f465c76-6d4qd_407e49f7-6cf3-4600-adda-6e96474beaf6/copy-manifests/0.log" Jul 16 17:34:38.776290 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:38.776257 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8b857c67-v4shd_abb42b64-0b44-46db-a4ff-8e0155c17500/kube-auth-proxy/0.log" Jul 16 17:34:38.880633 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:38.880549 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5vj4p_125e9af0-f9ce-4f4e-9e09-a1168e9ab748/payload-pre-processing/0.log" Jul 16 17:34:39.175286 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:39.175205 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c59ccc8d7-4ld8f_6741004c-e9a1-4234-ae48-aea8047a8545/manager/0.log" Jul 16 17:34:39.251358 ip-10-0-133-149 kubenswrapper[2556]: I0716 17:34:39.251317 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-d78pw_ae77840b-dc97-4a24-993c-7fdc9a0727e8/openshift-lws-operator/0.log"